Use Case: Science Pipeline: Process Single Field Interferometric Data (without single dish data)

Interferometric Use Case for ALMA (based on ALMA SW Memo 11, Science Requirements and Use Cases, modified to include detailed pipeline processing requirements). The science pipeline reduction is made within 12 hours after SB completion (or breakpoint), not in quasi real time as the quick-look pipeline. This Use Case does not include polarization measurement or self-calibration.

Goal:   Process data from a detection experiment or high-fidelity imaging experiment of a single ALMA field through the Science Pipeline. Only data processing steps are identified here.

Contact Authors:   J. Pety, F. Gueth, D. Shepherd

Role(s)/Actor(s):
Primary: Pipeline Subsystem.
Secondaries:

• Archive - Holds raw data & processed results.
  
• Scheduling Subsystem - Activates pipeline processing.
  
• Telcal Subsystem - Provides on-line calibration to the pipeline (stored in the archive).
  
• Pipeline Subsystem - Generates data products during and at the end of the project. Uses Offline software functionality to reduce & image data at intermediate steps. Results and data reduction/imaging script are stored in the archive.
  
• Offline Subsystem - Uses pipeline data reduction/imaging script as a starting point for offline processing.
  

Priority:   Critical

Performance:   Need to feedback data and results in a 'timely fashion.' Exact timing requirements TBD (for science processing, depends on hardware speed, number of parallel processing). On average, the science pipeline must keep up with the data stream.

Frequency of Use:   Perform this Science Pipeline Use Case for each single field imaging experiment run by ALMA. The Science Pipeline may be run more than once per project.

Preconditions:

1. Valid project, SBs, and data exist in the archive (including on-line calibration values and WVR corrections).
2. Scheduling subsystem activates pipeline processing.

Basic Course:

NOTE: All steps in the Basic Course should be saved to a master script.

1. Provide analysis of the following values recorded by the on-line system:
   • Weather data (e.g. wind, temperature, pressure, humidity)
     
   • Relative phase (to detect phase jumps).
     
   • Shadowing.
     
   • Tracking errors.
     
   • Pointing and focus corrections (provided by TelCal).
     
   • Total power measurements.
     
   • Tsys (to verify receiver tuning, e.g. DSB vs SSB).
     
   • Water column (WVR measurements).
     
   • FTS measurements.
     
2. Output:
   • Quality flags vs. time.
     
   • Warning when something wrong has been detected.
     
   • Report.
     

1. Check validity of WVR phase corrections.
   
2. Perform RF bandpass calibration. Determine deg polynomials vs bandwidth.
   Detectable problems: Calibrator too weak; Delays are wrong; Absorption line in quasar.
3. Perform temporal phase fluctuation calibration. Determine interpolation interval.
   Detectable problems: Baseline problems; phase jumps.
4. Perform absolute flux calibration. Determine time interval and antennas to be used in the average (depends on total number of antennas)..
   Detectable problems: No meaningful solutions; absolute calibrator measurement is not useable; bad antenna efficiencies.
5. Perform temporal gain fluctuation calibration. Determine interpolation interval.
   Detectable problems: Amplitude jumps; poor flux calibration.
6. Output:
   • Updated quality flags vs. time.
     
   • Calibrated visibilities & uv weights.
     
   • Warning when something wrong has been detected.
     
   • Report.
     

1. Resample to velocity resolution requested by PI if needed (e.g. if the correlator resolution is smaller than the required spectral resolution).
   
2. Change phase center of the image if desired.
   
3. Perform continuum subtraction.
   
4. Output:
   • uv visibilities and weights.
     
   • Notification of what was done to the data.
     

1. Fourier transform uv data to the image plane. Determine dirty beam characteristics for spectral channels, weighting scheme (robust?), taper if desired to obtain specific clean beam shape.
   
2. Deconvolve the image:
   
   • Use source model given by PI to select best deconvolution method (e.g. CLEAN, multi-scale CLEAN, MEM) or use accumulated experience from similar project already observed (if we try to compare different methods directly on true data, results can be biased: e.g. some methods can give less noise but systematic biases that are only detectable with models.
     
   • Estimate noise on dirty image.
     
   • Compare clean beam with dirty beam. Verify whether it is consistent with synthesized beam as deduced from uv_max.
     
   • Setup and run CLEAN deconvolution:
     • Define 3 dimensional support (deconvolution regions, spatial and spectral).
       
     • Determine CLEAN stopping criteria (fraction of noise and/or fraction of peak intensity and/or maximum number of components).
       
     • Begin CLEAN iterations.
       
     • Analyze residual map and cumulative flux curves.
       
     • Analyze history of clean components.
       
   • Alternate Course: Set up and run MEM deconvolution:
     
     • Define 3 dimensional support (deconvolution regions, spatial and spectral).
       
     • Determine MEM stopping criteria (RMS factor to clean down to).
       
     • Determine best variant of this plan for a particular case. May want to use an image at another frequency as an initial guess if available.
       
   • Compare theoretical dirty beam to a map of a point source calibrator to estimate dirty beam uncertainty.
     
3. Alternate Course: uv analysis, provide simple uv fits to the data.
   
4. Output:
   • Report of major decisions.
     
   • Clean images and clean beams (with frequency dependency).
     
   • Specification of support (deconvolution regions) for cleaning.
     
   • Empirical noise map.
     

1. Collect all quality assessment information generated during pipeline processing.
   
2. Format into a single report.
   
3. Output:
   • Summary report giving Quality flags vs. time, all warnings generated by the pipeline, quality metrics, fidelity measure of the resulting image.
     

Postconditions:

1. Pipeline calibrated data are sent to archive.
2. Pipeline processing script is sent to the archive.
3. Standard, processed images are sent to archive.
4. Quality assessment metrics and fidelity measure of data products sent to archive.
5. Pipeline sends notification to Scheduling subsystem stating that data storage is complete and it is ready for the next project?

Issues to be Determined or Resolved:  

1. Observing Conditions Analysis:
   • Is there a particular order that is best for determining how different parameters affect data quality?
2. Calibration:
   • The calibration scheme above assumes the bandpass is constant over time. Is this a valid assumption given the precision level requirements for ALMA?
3. Imaging:
• Should we question the use of FFT vs DFT?
• Is robust weighting a good thing to do as this will change the dirty beam shape that has been "optimized" in the choice of antenna layout?
• Do we make a primary beam correction? If yes, is the primary beam well enough known?

Notes:  
This Use Case was created by J. Pety to help define Pipeline heuristics. Relevant SSR Use Cases from SSR Memo 11 are: 4.5.1 (Process Calibrations); 4.5.2 (Process QuickLook Data); 4.5.3 (Process Science Data); 4.7.1 (Reduce Single Field [in pipeline mode]).

Notes from SSR Memo 11: Use Case 4.7.1 (Reduce Single Field)

1. Processing of the calibrations (e.g. pointing, focus, phase, flux) and processing of science data are performed by distinct Pipelines.
2. Data is automatically flagged for various detected situations (e.g. large point errors)
3. The WVR-based phase corrected or uncorrected data are used according to the options in the setup.

Last modified: 03jul03