Use Case: Control: Automatic Operations Mode

Automatic Operations Use Case for the ALMA Control software based on ALMA SW Memo 11, Science Requirements and Use Cases, modified to include detailed nominal control events.

Goal:   Automatically control ALMA telescopes (e.g. keep a current snapshot of the array status, allocate sub-arrays, execute scheduling blocks (SBs), collect monitor data from all instrumentation, collect scientific data from all instrumentation except the correlator). ).

Contact Authors:   J. Mangum, R. Marson, D. Shepherd

Role(s)/Actor(s):
Primaries:

• Control subsystem
  
• Executive subsystem - activates, monitors and shuts down Control.
  
• Correlator subsystem - many things destined for the correlator come through Control first.
  
• All computers - Control subsystem connects to the masers & GPSs which provide time to all networked computers at the OSF or AOS .
  
• Operator - Control provides status screens, full control interfaces, and a command line interface.
  
• Telcal Subsystem - Analyses data collected by the control subsystem and provides results of on-line calibration to the Control subsystem.
  
• Quick-Look Pipeline - Analyses data collected by the control subsystem & listens for scan start/stop events so that scan can be processed.
  

• Archive Subsystem - repository for, e.g., SB Execution block, monitor point data, event records (scan start/stop, SB start/stop). Also provides unique IDs for the ExecBlock.
  

Priority:   Critical

Performance:   Must run in real time.

Frequency of Use:   Once the Control subsystem is initialized in Automatic Mode it continues operating until told to stop. Automatic mode is expected to be the primary mode for ALMA (e.g. distinct from restricted operations in which one or more critical subsystems are not on-line or when the array is operated in Manual Mode (where only the operator can initiate observations and all sub-arrays are created in manual mode).

Preconditions:

1. Verified scheduling blocks (SBs) associated with a project have been placed in the Archive and selected by a Subordinate Scheduler for execution.
2. One or more antennas, including receivers and associated electronics, are powered up. If WVR data will be used, then WVR systems are installed on each antenna.
3. If needed, weather stations are active and providing weather data.
4. The maser time is accurate and synchronized across all the computers in the real-time computers (this includes the ACC and the correlator computers).
5. The Archive permissions are set to allow Control to write to the Archive.
6. The configuration database (e.g. hardware ID, hardware control limits, monitor point IDs) is available somewhere and provided by ACS - Will this be stored in the Archive?
7. Operator control interfaces must be available (GUI & CLI).
8. Video cameras, if available, should be on and transmitting images to Control.
9. Executive initializes all subsystems. It is not clear what order Exec will start subsystems. If a subsystem is not active when Control starts up, then Control will restrict operations to exclude or bypass the missing subsystem (e.g., if Archive is off-line, nothing is written to it; if the Correlator is off-line, no correlations are written (but Total Power is)). Thus, the initialization order is critical to Control operations. This Use Case assumes that all critical subsystems are initialized before Control so full, nominal operations exist.

Basic Course:

1. Control reads the configuration database (on disk or in the Archive).
2. Control surveys hardware on the array. It determines what is there and what the current status of that hardware is (e.g. IFs, correlator configuration, antennas, backends, LO's, masers). The information is stored in ArrayStatus().
3. Control connects to the masers and the GPS to initialize the time on the ARTM and the Network Time Protocol (NTP) server. The real-time computers synchronize their time using using a timing synchronization pulse which generates an interrupt signal and initiates a request for the current time to ARTM. The ARTM responds with the current time and if the round-trip communication takes less than 48 ms, then the real-time computer assumes it is in sync. If the communication takes more than 48 ms, then the computer re-tries. If it cannot establish sync, the real-time computer issues a failure event which the Operator sees on the Control interface (eventually, a Maintenance crew is sent out).
4. For non-real-time computers, the NTP server broadcasts timing values which are picked up by these computers. The computers keep track of the timing events and develop a statistical understanding of the best-guess time. Clients operating in this mode include the Pipeline, Offline, Archive, TelCal.
5. Control collects Monitor Point values across the array and sends the results to the Archive to be stored in a special area reserved for Monitor Point data. (Maximum write speed = 21 times/sec.)
6. TelCal subsystem sends an event saying that calibration parameters are available (e.g., gain & baseline calibration, focus & pointing values). Control 'hears' the event and requests TelCal data (this will include calibration parameters along with information on when the calibration was determined).
7. If video cameras are installed on the array, Control will provide the link to the video feeds and the Operator control interface so the Operator can control which video is selected.

1. A Subordinate Scheduler requests that a subarray be created. If this matches with the available antennas, Control creates the subarray definition. Normally the same scheduler will ask the control subsystem to destroy the subarray.
2. A Subordinate Scheduler requests that a specific SB on a subarray be started.
3. Control retrieves the selected SB from the Archive.
4. Control requests an execution_block_ID from the Archive. The Archive generates an ID that is related to the project and sends this back to Control to be attached to the ExecBlock.
5. Control sets up an ExecBlock (execution block of data which stores header & setup information and all scientifically-relavent information needed to process the data). The ExecBlock includes Total Power values (Four 2-4 GHz chunks per integration (IF 1/2, Pol 1/2); these are not processed by the Correlator). Also, all Monitor Point values critical to scientific processing are also copied to the ExecBlock (as well as put in the Monitor Point Archive space). Issue: if a new Monitor Point is added to the ExecBlock, and a project gets another SB observed, then earlier SBs won't have access to the Monitor Point data unless it is found and retrieved from the Archive. How will this be handled so it is backward compatible?
6. Control extracts correlator setup information from the SB and sends this to the Correlator.
7. Control generates an execution_block_ID and sends this to the Correlator so the data stored by the Correlator can be identified with an ExecBlock (that is attached to a SB that is part of a particular Project).
8. Control begins execution of the SB. Event records about scan start/stop and SB start/stop are broadcast to all active subsystems during SB execution and embedded within the ExecBlock.
9. WVR data is collected from each active antenna and put in the ExecBlock. It is expected to be collected every 1/2 second. Question for Jim Pisano: If sub-integrations come in faster (up to every 48 ms), is the WVR data interpolated or will a single WVR integration be applied to to all data in the next 1/2 second?
10. Control calculates delay values for each antenna based on what is in the configuration database and observational data from the most recent 'delay determination' SB (observations of bright quasars observed at a range of elevations and hour angles). The delays are broken into a rough estimate (delays of one sample or more) which is sent to the Correlator to be applied to the data and a second order estimate (fine adjustment less than the sampler time) which is sent to the samplers in each antenna for real-time correction.
11. Control receives data blanking requirements from Monitor Points which are out-of-limits (e.g. an IF is bad, blank data on that IF) and from the Operator (e.g. specify that data should be blanked if the antenna position encoders show that the phase center of the antenna is greater than some fraction of the primary beam at a given frequency). Blanking information is provided to the Correlator subsystem in real-time so the bad data will not be included in the integrations.
12. Once an SB completes execution, Control will flush the execution block to the archive and broadcast an event saying that it is complete.

1. The Control subsystem will provide a list of commands that it recognizes and can interpret as control sequences. Essentially, this is a document that Control will keep up-to-date. These commands can be used by a PI to write observing scripts for non-standard observations.
2. If requested by the Operator to plot the historical values of a Monitor Point, then Control retrieves the historical data from the Archive and provides it to the Operator in some requested display format (e.g. plot, list of numbers).
3. If the Executive subsystem issues a getControlStatus() command, then the Control subsystem will provide a status summary.
4. If the Executive subsystem issues an Abort command, Control stops everything and waits for further commands. The control software can be separately shutdown by the Executive subsystem.

Postconditions:

1. After initialization, the Control subsystem continually executes the basic course described above. It will stop and wait if the Executive subsystem issues a stop() command.
2. Once an SB completes execution, several things happen:
   • All information about the ExecBlock will be held in memory for some TBD period of time in case a status request is made by a subsystem.
   • The Quick-look Pipeline will finish up processing the SB data and store the results on disk for some period of time.
   • The Project Manager on the Scheduler will start the Science Pipeline if so defined in the Project Definition structure.

Issues to be Determined or Resolved:  

1. How long will the TelCal data be available for retrieval after notification is issued that the data is available?
   
2. It is not clear what order Exec will start subsystems. If a subsystem is not active when Control starts up, then Control will restrict operations to exclude or bypass the missing subsystem. Thus, the initialization order is critical to Control operations.
   
3. Right now, the configuration database resides on a disk file. It is part of Control and provided to the subsystem by ACS. Where will this reside in the long term? In the Archive?
   
4. If a new Monitor Point is added to the ExecBlock, and a project gets another SB observed, then earlier SBs won't have access to the Monitor Point data unless it is found and retrieved from the Archive. How will this be handled so it is backward compatible?
   
5. It is not clear how many video feeds there are, or whether they can be actively controlled (e.g. movable, can be zoomed).
   
6. It has not been determined how long the ExecBlock information about SB processing shall be stored in virtual memory after an SB is complete. (This is required to provide post-execution status to subsystems who run in near-real time and thus lag the real-time operations of the Control subsystem.)
   

Notes:  

1. This Use Case was created by D. Shepherd based on conversations with Ralph Marson and Jeff Mangum to help define the Control subsystem test plan. Relevant SSR Use Cases from SSR Memo 11 are 4.4 (Execute SB), 4.8.5 (Delay Calibration).
   

   Last modified: 21aug03