Use Case: Quick Look Pipeline: Data Processing: Interferometric Data

Quick Look Data Processing Use Case for ALMA (based on ALMA SW Memo 11, Science Requirements and Use Cases, modified to include detailed Quick Look Data Processing requirements). Quick Look Data Processing reduces data in quasi-real time, in an automatic way, taking input data from the Pipeline temporary database which has been fed by the correlator data stream, the control subsystem, and the TelCal subsystem via the Quick Look Pipeline. Results are stored in a temporary archive storage area. The Quick Look data reduction should not be a bottleneck for the array operations.

Quick Look data processing will be activated automatically under specified conditions (i.e. after each scheduling block, at the end of each observing session, possibly after each calibration scan). Quick Look data processing operations shall be activated, at a minimum, after each scheduling block and, at a maximum, after each calibration scan. The frequency of Quick Look data processing shall be specifiable in the observing tool. It must also be possible to start these operations whenever requested by the Staff Astronomer or Operator.

Goal:   Provide quick reduction of the data taken in an observing session, including dirty maps of specified channel averages.

Contact Authors:   C. Wilson, R. Lucas

Role(s)/Actor(s):
Primary: Pipeline Subsystem, Quick Look Data Processor; Array Observing System.
Secondaries:

Priority:   Major (may turn out to be a high priority from operations point of view)

Performance:   from seconds to a few minutes for imaging

Frequency of Use:   Minutes.

Preconditions:

  1. Data are available from the Correlator in temporary storage.
  2. Calibration data are available from the Calibration Pipeline running in parallel, in temporary storage.
  3. Simple Quick Look Reduction scripts are available, as defined in the Observing program.
  4. Control activates the Quick Look Data Processor at the specified points in the observing session via ObsEnd and TelCal events.
    Alternate course: activation by Operator, Staff Astronomer, or Interactive Observer.

Basic Course:

  1. The Quick-Look Data Processor retrieves data for most recent time period from the pipeline temporary database.
  2. The Quick Look Data Processor calibrates the visibilities observed on any target source, using the results of the on-line Calibration Operations and the appropriate reduction script:
    Exception course: the Operator is notified if the script fails.
  3. The Quick Look Data Processor displays the spectra observed on an astronomical target with various options to be selected by the Operator, Staff Astronomer, or Interactive Observer:
    • the most recent scan or a time integration of all scans in the current session
    • choice of the baseline(s)
    • baselines summation, with and without shifting phases to a specified position
    • intensity (amplitude and/or phase) as a function of baseline and time (for a selected frequency), or as a function of time and frequency (for a selected baseline) for data from the current scheduling block.
    • phase and amplitude closure for calibrators as a function of time
  4. For Mosaic observations, spectra will be displayed for the mosaic center.
    Alternate course: spectra are displayed for a mosaic field specified by the Operator, Staff Astronomer, or Interactive Observer.
  5. For snapshot observations, spectra will be displayed for the target observed in the most recent scan.
    Alternate course: spectra are displayed for a target specified by the Operator, Staff Astronomer, or Interactive Observer.
  6. The Quick Look Data Processor will compute the dirty image of the astronomical target using all the data available so far from the current session. The resulting maps will be displayed.
    Exception course: the Operator is notified if the script fails.
  7. For snapshot observations, dirty images will be computed and displayed for each target observed since the previous time the Quick Look Data Processor was activated.

Postconditions:

  1. The Quick Look Data Processor shall store the average spectrum (amplitude and phase versus frequency and baseline) and its corresponding weight for each astronomical target integrated over all scans in the current session for each baseline. For mosaic observations, the spectrum will be stored for the mosaic center. It shall also store closure measurements on the calibrators for the duration of the current session.
  2. The Quick Look Data Processor will store the calibrated uv data corresponding to the continuum and the line-averaged images (and their corresponding weights) for each scan of a target source in a temporary database until the end of the observing session. This database will be emptied at the end of the observing session.
  3. The Quick Look Data Processor will store the calibrated spectra (as a function of baseline, time, and frequency) for each target source in a temporary database until the end of the current scheduling block. The data may be binned in time to save storage space. This database will be emptied at the end of each scheduling block.
  4. Science data (images and spectra) are written into a temporary archive storage area in the form of plots and tables. The images will be marked as quick look data to distinguish them from final images.
  5. Data reduction scripts and logs are written into a temporary archive storage area.
  6. Parameters may be fed back to the Scheduler (on-line execution). This can be e.g. quality (dynamic range, SNR) evaluation on a test source.
  7. The Observer is notified (by email) that dirty maps and spectra are available.

Issues to be Determined or Resolved:  

  1. How does the Scheduler retrieve pipeline results?
    LD: There are no plans to do this directly as far as I know. This is the role of TELCAL. The quick-look results can be used by the operator to make decisions about what to do.
    CW: This requirement comes from Memo 11 and we need to decide if it is needed.
  2. We will need separate reduction scripts/heuristics for the Quick Look Pipeline and the main Science Pipeline. For example, the Quick Look Pipeline needs to know what to do if if bandpass calibration is NOT available YET, keeping in mind that the Quick Look Pipeline has to keep up with the data. The quick look heuristics should be a very simple version of the science heurisitcs with very few decision points and options.
  3. Do we want to send plots to the archives and email the observer each time this pipeline is run? If "every time the pipeline is run" means every time an image is produced probably not, e.g. think of snapshot images ... One possibility is to provide an initial URL which the user can monitor at their convenience, with an official notification at the end of the session, or at a user defined point.
  4. Being able to plot intensity (amplitude and/or phase) as a function of baseline and time (for a selected frequency), or as a function of time and frequency (for a selected baseline) is potentially problematic from the point of view of the Quick Look pipeline having to hold lots of data. If the operator or staff astronomer is allowed to select any frequency in the band, then all the calibrated data since the beginning of the session must be saved somewhere. It may not be as big as the raw data (i.e. if we've averaged up over 10 s) but it still would be a substantial chunk of stuff over a few hours observing. That's why I have limited this mode to having a maximum time extent of one scheduling block. I need to know if this is OK. If this requirement is not needed, that would be even better ...]
  5. Do we need an off-line execution mode for the Quick Look Data Pipeline? We probably need a testing / debugging mode for the quick look procedure but re-running failed quick-look procedures should probably not be part of the operations plan.
  6. Most of the time, Quick Look Data Processing will work on data from a single scheduling block. Operations which can easily accumulate data over time (such as an integrated spectrum) should be possible to accomodate. We should also be able to make images using all the uv data obtained in the current session, not just the most recent scheduling block. Is that possible?

    7. LD: I would like to minimize processing dependencies between scheduling blocks. Operations which can easily accumlate data over time from sceduling blocks would be fine as that data would still be useful if the program could not be finished.
  7. The calibration procedure needs to be outlined in more detail so it is a bit easier to relate the procedure to the TELCAL calibration results particularly with respect to temperature / flux calibration. The current ICD looks like it is based on IRAM procedures which are a bit different from current VLA procedures. A pointer to the appropriate documentation would be useful.

Notes:  

  1. This Use Case was created by C. Wilson to help define Quick Look requirements. Relevant SSR Use Case from SSR Memo 11 is 4.5.2 (Process Quick Look Data) (parts of it) by R. Lucas.
  2. If the Quick Look Data Processing of a previous Session is still active, the Quick Look for the current Observing Session has priority for allocation of computing resources. Quick Look heuristics should be designed to keep up and minimize end of session processing.
  3. An Observer must be able to look at the Pipeline results of recently observed Programmes without downgrading the Quick Look peformance on the currently observed Programme. This will probably be handled by the archive and so should not affect current quick look performance.

    Last modified: 12aug03