Use Case: Scheduler: Dynamic Mode Operations

Dynamic Mode Operations Use Case for the ALMA Scheduler software based on ALMA SW Memo 11, Science Requirements and Use Cases, modified to include detailed Dynamic mode scheduler events (e.g. when the scheduling software runs in automatic mode and decides which SB to run).

Goal:   Automatically schedule ALMA SBs for observation (rank available SBs, create a master "look-ahead" schedule, initiate SB execution, update project status, and start Science Pipeline).

Contact Authors:   M. Wright, D. Shepherd, A. Farris

Role(s)/Actor(s):
Primaries:

• Scheduling subsystem
  
• Executive subsystem - activates Scheduler.
  

• Operator, Staff Astronomer - monitors scheduler activities, examines master "look-ahead" schedule.
  
• Archive Subsystem - repository for verified SBs ready for scheduling, receives project status update from Scheduler.
  
• Telcal Subsystem - Provides results of on-line calibration to the Scheduler.
  
• Control subsystem - executes a scheduling block.
  
• Science Pipeline Subsystem - Initiated by Scheduler according to "Project Definition Structure."
  
• PI - the PI is informed of observing results for an SB by the Scheduler "Project Manager" software.
  

Priority:   Critical

Performance:   Must run in near-real time (within seconds).

Frequency of Use:   Once the Scheduler is started in Dynamic Mode, it continues operating until told to stop. Dynamic Mode operations is expected to be the primary operational mode for ALMA (e.g. after commissioning, outside of maintenance blocks, and when the telescope is not in interactive mode).

Preconditions:

1. Verified scheduling blocks (SBs) associated with a project have been placed in the Archive by the Observing Tool. A Project consists of "Obs Unit Sets" which link 1 or more SBs into a group (e.g. an Obs Unit Set defines SB ordering) and a "Project Definition Structure" which identifies, e.g., science priority, breakpoints, data processing requirements, project status, PI notification requests. The Scheduler must be able to read the Project Definition Structure and associate project constraints with each SB.
2. The Project Definition Structure must identify that the SBs available for consideration by the Scheduler are suitable for Dynamic Mode operations.
3. SBs have a required initial setup.
4. SBs in the Archive will identify certain antennas for scheduled maintenance or moves. The Master Scheduler can handle the situation in which some subarrays are running automatically while others are in interactive mode.
   
5. The Executive subsystem has supplied a scheduling policy (contents TBD) which defines the factors to be taken into account when deciding which SB to schedule. The "policy" resides in the archive. Note, Exec can change the policy on-the-fly by telling the scheduling subsystem to stop and restart with a different scheduling policy.
6. The Control subsystem is active and ready to execute an SB when requested by the Scheduler. The Control subsystem also provides information needed by the Scheduler upon startup.
7. The QuickLook Pipeline is active and ready to process data.
8. The Science Pipeline may be active and ready to process data from a completed SB when requested by the Scheduler. If it is not taking requests, then the Scheduler will queue its processing requests in the Archive and the Science Pipeline will process the queue when it is ready.
9. The Archive permissions are set to allow the Scheduler/Project Manager to write to the Archive and to link Science Pipeline results to the Project Definition Structure.
10. The Executive subsystem activates the Scheduler by issuing a startScheduling() command.

Basic Course:

1. The Master Scheduler polls the Archive for new SBs, identifies which ones are valid, and examines SB constraints, new SBs are added to the current list of possible SBs to schedule.
2. The Master Scheduler monitors calibration events and weather conditions provided by TelCal (e.g. PhaseCalEvents, coherence, pointing, focus, wind speed) and obtains a snapshot of the array status from the Control subsystem (e.g. how many antennas there are in what configuration). These data are compared to observing band frequencies and observing constraints to determine likelihood of obtaining good data.
3. The Master Scheduler creates a sub-program called the "Project Manager" which monitors and updates the status of all projects and sends out notification events. The Proj Mgr must run independent of the Subordinate Schedulers.
4. The Master Scheduler creates Subordinate Schedulers that will handle a queue of SBs. There will be one Subordinate Scheduler for each subarray required (e.g. VLBA antennas, Total Power subarray, one or more interferometric subarrays). How the decision is made as to how many subarrays will be needed is TBD.
5. Each Subordinate Scheduler examines its SB priorities and constraints. Based on the current scheduling Policy, and SB constraints, the Scheduler calculates a rank for each SB and uses the rank-ordering to decide which SB(s) to run for its subarray.
6. Each Subordinate Scheduler displays the rank-ordered list of SBs for the Operator to view. SBs chosen for execution are identified.
   • Alternate course: The Operator alters priorities. Subordinate Schedulers must receive a reply from the Operator saying that manual intervention is desired, else it will simply display the list and continue scheduling events.
   • Alternate course: If no SBs can be executed (e.g. all sources are below the horizon, available SBs can't be run due to poor weather conditions), then the Operator is informed and takes necessary action.
7. The Subordinate Scheduler sends notification to the Control subsystem to begin execution of an SB. The Control subsystem writes execution records to the Archive (these are linked to the Project Definition).
8. While the SB is running, the Subordinate Scheduler sends logger messages to the Archive.
9. When the Proj Mgr detects a completion event on an SB from the Control subsystem, the Proj Mgr does the following things:
   • Updates the Project Status in the Archive.
   • A Science Pipeline processing request is sent to the Archive and the Sci Pipeline (if the Project Definition Structure calls for this).
   • When the Sci Pipeline sends out a completion event, then the Proj Mgr links the resulting data/processing history to the Project Definition Structure.
   • The Proj Mgr notifies the PI, requests breakpoint response if needed (if the Project Definition Structure calls for this).
10. Note: The Master Scheduler can terminate a Subordinate Scheduler or the Project Manager at any time.
11. At some point, the Master Scheduler uses the scheduling Simulator tool to create a "look-ahead" schedule for the operations team. The look-ahead schedule will take the current list ranking of SBs, assume current weather conditions(?), and create a master list of likely scheduled projects that will be run in the next 24-48 hours.
12. If the Executive subsystem issues a getStatus() command, then the Master Scheduler will provide a status summary.

Postconditions:

1. The Project Definition is updated in the Archive (includes completion status and link to data & Sci Pipeline results).
2. The system logs are available in the Archive. This includes the results of the SB ranking determined by the Master Scheduler.
3. The Master Scheduler repeats the basic course described above in a continual loop. It will stop and wait if the Executive subsystem issues a stopScheduling() command. If the Control subsystem issues an ExecBlockEnd event (e.g. an SB has ended), then each Subordinate Scheduler will look to see if their subarray is free, if so, the Subordinate Scheduler will schedule another SB.
4. The Master Scheduler will poll the Archive to look for break point responses from the PI. If a response is received, then it will consider this in determining future SBs to schedule.

Issues to be Determined or Resolved:  

1. The set of priorities (Policy) used to calculate SB ranks is TBD.
   
2. How the decision is made as to how many subarrays will be needed is TBD.
   
3. The issue of how to deal with idle antennas is TBD: If antennas are left idle, there may be filler programs available. The Subordinate Schedulers should tell the Master Scheduler about any idle antennas and the Master Scheduler should tell the Operator of the condition. Decisions about how the idle antennas will be used are TBD.
   
4. The SSR Use Cases specify "Pre-amble" and "Post-amble" text associated with each SB. This is no longer correct and the Use Cases should be modified to reflect the current requirements.
   
5. It has not been determined whether the look-ahead schedule should be created automatically every X-hours, should be generated upon request (who's request?), or whether the Simulator inputs can be modified (e.g. Operator knows a storm is approaching and requests a look-ahead schedule for the next 12 hours assuming the weather conditions get 50% worse than current conditions in 2 hours).
   

Notes:  

1. This Use Case was created by D. Shepherd based on conversations with Allen Farris and Mel Wright to help define the Scheduling test plan. Relevant SSR Use Cases from SSR Memo 11 are 4.3 (Schedule SB) & 4.4 (Execute SB)
   

   Last modified: 15sep03