Use Case: Science Pipeline: Process Wide-Field Mapping Interferometric Data (without single dish data)

Interferometric Use Case for ALMA (based on ALMA SW Memo 11, Science Requirements and Use Cases, modified to include detailed pipeline processing requirements). The science pipeline reduction is made within 12 hours after SB completion (or breakpoint), not in quasi real time as the quick-look pipeline. This Use Case does not include polarization measurement or self-calibration.

Goal:   Process data from a detection experiment or high-fidelity imaging experiment of an ALMA mosaic through the Science Pipeline. Only data processing steps are identified here.

Contact Authors:   J. Pety, F. Gueth, D. Shepherd, C. Wilson

Role(s)/Actor(s):
Primary: Pipeline Subsystem.
Secondaries:

• Archive - Holds raw data & processed results.
  
• Scheduling Subsystem - Activates pipeline processing.
  
• Telcal Subsystem - Provides on-line calibration to the pipeline (stored in the archive).
  
• Pipeline Subsystem - Generates data products during and at the end of the project. Uses Offline software functionality to reduce & image data at intermediate steps. Results and data reduction/imaging script are stored in the archive.
  
• Offline Subsystem - Uses pipeline data reduction/imaging script as a starting point for offline processing.
  

Priority:   Critical

Performance:   Need to feedback data and results in a 'timely fashion.' Exact timing requirements TBD (for science processing, depends on hardware speed, number of parallel processors). On average, the science pipeline must keep up with the data stream.

Frequency of Use:   Perform this Science Pipeline Use Case for each mosaic imaging experiment run by ALMA. The Science Pipeline may be run more than once per project.

Preconditions:

1. Valid project, SBs, and data exist in the archive (including on-line calibration values and WVR corrections).
2. Scheduling subsystem activates pipeline processing.

Basic Course:

NOTE: All steps in the Basic Course should be saved to a master script.

1. Provide analysis of the following values recorded by the on-line system:
   • Weather data (e.g. wind, temperature, pressure, humidity)
     
   • Relative phase (to detect phase jumps).
     
   • Shadowing.
     
   • Tracking errors.
     
   • Pointing and focus corrections (provided by TelCal).
     
   • Total power measurements.
     
   • Tsys (to verify receiver tuning, e.g. DSB vs SSB).
     
   • Water column (WVR measurements).
     
   • FTS measurements.
     
2. Output:
   • Quality flags vs. time.
     
   • Warning when something wrong has been detected.
     
   • Report.
     

1. Check validity of WVR phase corrections.
   
2. Perform RF bandpass calibration. Determine deg polynomials vs bandwidth.
   Detectable problems: Calibrator too weak; Delays are wrong; Absorption line in quasar.
3. Perform temporal phase fluctuation calibration. Determine interpolation interval.
   Detectable problems: Baseline problems; phase jumps.
4. Perform absolute flux calibration. Determine time interval and antennas to be used in the average (depends on total number of antennas)..
   Detectable problems: No meaningful solutions; absolute calibrator measurement is not useable; bad antenna efficiencies.
5. Perform temporal gain fluctuation calibration. Determine interpolation interval.
   Detectable problems: Amplitude jumps; poor flux calibration.
6. Output:
   • Updated quality flags vs. time.
     
   • Calibrated visibilities & uv weights.
     
   • Warning when something wrong has been detected.
     
   • Report.
     

1. Resample to velocity resolution requested by PI if needed (e.g. if the correlator resolution is smaller than the required spectral resolution).
   
2. Perform continuum subtraction.
   
3. Output:
   • uv visibilities and weights.
     
   • Notification of what was done to the data.
     

1. For each field, fourier transform uv data to the image plane. Determine dirty beam characteristics for spectral channels, weighting scheme (robust?), taper if desired to obtain specific clean beam shape.
   
2. The individual dirty images (F_i) are then combined into a single image (J) in a way that maximizes the SNR in the least-square sense. In the same operation, the division by the primary beam is made. This division implies amplification of noise at the border of each field. To limit this effect, the primary beam is truncated at a level which can be changed by the end-user. This results in a noise pattern almost constant near the mosaic phase center but which increases a lot near the mosaic edges. Formula: J = {sum_i F_i.(B_i/sigma_i^2)} / {sum_i(B_i^2/sigma_i^2)}
3. Deconvolve the image:
   
   • Use source model given by PI to select best deconvolution method (e.g. CLEAN, multi-scale CLEAN, MEM) or use accumulated experience from similar project already observed (if we try to compare different methods directly on true data, results can be biased: e.g. some methods can give less noise but systematic biases that are only detectable with models.
     
   • Estimate noise on dirty image.
     
   • Compare clean beam with dirty beam. Verify whether it is consistent with synthesized beam as deduced from uv_max.
     
   • Setup and run CLEAN deconvolution:
     • Define 3 dimensional support (deconvolution regions, spatial and spectral).
       
     • Determine CLEAN stopping criteria (fraction of noise and/or fraction of peak intensity and/or maximum number of components). Will need to use a noise map to account for increased noise at the mosaic edges
       
     • Begin CLEAN iterations.
       
     • Analyze residual map and cumulative flux curves.
       
     • Analyze history of clean components.
       
   • Alternate Course: Set up and run MEM deconvolution:
     
     • Define 3 dimensional support (deconvolution regions, spatial and spectral).
       
     • Determine MEM stopping criteria (RMS factor to clean down to).
       
     • Determine best variant of this plan for a particular case. May want to use an image at another frequency as an initial guess if available.
       
   • Compare theoretical dirty beam to a map of a point source calibrator to estimate dirty beam uncertainty.
     
4. Output:
   • Report of major decisions.
     
   • Clean images and clean beams (with frequency dependency).
     
   • Specification of support (deconvolution regions) for cleaning.
     
   • Empirical noise map.
     

1. Collect all quality assessment information generated during pipeline processing.
   
2. Format into a single report.
   
3. Output:
   • Summary report giving Quality flags vs. time, all warnings generated by the pipeline, quality metrics, fidelity measure of the resulting image.
     

Postconditions:

1. Pipeline calibrated data are sent to archive.
2. Pipeline processing script is sent to the archive.
3. Standard, processed images are sent to archive.
4. Quality assessment metrics and fidelity measure of data products sent to archive.
5. Pipeline sends notification to Scheduling subsystem stating that data storage is complete and it is ready for the next project.

Issues to be Determined or Resolved:  

1. Observing Conditions Analysis:
   • Is there a particular order that is best for determining how different parameters affect data quality? Can some of the observing condition analysis can be done by examining the auxiliary (meta) data and / or the calibration data as opposed to the extracting the entire data set? It would be nice to be able to make a decision about whether (as opposed to how) to to process the data before extracting it.
2. Calibration:
   • The calibration scheme above assumes the bandpass is constant over time. Is this a valid assumption given the precision level requirements for ALMA?
   • Will the calibrations always be done in the order indicated above ?
3. Imaging:
• Should we question the use of FFT vs DFT?
• Is robust weighting a good thing to do as this will change the dirty beam shape that has been "optimized" in the choice of antenna layout?
• We will have to include correction for the primary beam. Is the primary beam well enough known?
4. Is there any way to automatically set the truncation level to something clever when the fields are merged into a mosaic?
5. What if the current mosaic must be merged with another mosaic whose uv coverage (i.e. the resolution) is quite different from the current mosaic?

Notes:  
This Use Case was created by J. Pety to help define Pipeline heuristics. Relevant SSR Use Cases from SSR Memo 11 are: 4.5.1 (Process Calibrations); 4.5.2 (Process QuickLook Data); 4.5.3 (Process Science Data); 4.7.2 (Reduce Multi Field [in pipeline mode]).

Notes from SSR Memo 11: Use Case 4.7.1 (Reduce Single Field)

1. Processing of the calibrations (e.g. pointing, focus, phase, flux) is done by the TelCal subsystem, which is distinct from the processing of science data performed by the Science Pipeline.
2. Data is automatically flagged for various detected situations (e.g. large point errors)
3. The WVR-based phase corrected or uncorrected data are used according to the options in the setup.

Last modified: 13aug03