STATUS REPORT:

OFF-LINE DATA PROCESSING RE-USE ANALYSIS DEMO

A. Kemball (NRAO) and R. Lucas (IRAM)
March, 2002

1. Overview

This update summarizes the current status of the ALMA-AIPS++ re-use
analysis test, currently underway through the end of April 2002.

2. Test objectives

The objectives posed for the test, excerpted from the original
memorandum, are as follows:

"...
   i) How AIPS++ can be adapted to reduce data of an instrument
      for which it was not originally designed ?

  ii) How long is the learning curve for developers who have
      sufficient experience in the processing of millimeter
      data, but no experience at all of the AIPS++ programming
      environment ?

 iii) Can we perform an end-to-end experiment on actual, real-life
      millimeter-wave spectroscopic data ?
..."

This is a collaborative effort between AIPS++ and ALMA computing,
represented by participating staff from IRAM, NRAO/AIPS++ and
ESO/ALMA.

The test runs through the end of April 2002, in two phases. Phase I is
concerned with reduction of a specific spectral-line IRAM Plateau de
Bure interferometer dataset in AIPS++, and Phase II involves the
reduction of other similar datasets by scientists familiar with IRAM
reduction using the AIPS++ data reduction tools resulting from the
Phase I work. A potential Phase III, concerning AIPS++ performance
evaluation for large arrays, is under negotiation to start after Phase
II.

3. Current status

The following items have been completed in the test:

i) We agreed upon an export FITS format for un-calibrated uv-data from the IRAM
   interferometer at Plateau de Bure, namely the ALMA test-interferometer
   data format (Lucas and Glendenning 2001; ALMA Computing Memo. 15),
   which was reviewed and accepted.

ii) A representative IRAM interferometer dataset was selected for Phase I
    of the test. These data include HCO+ (J=1-0) and 13CO J=2-1
    spectroscopic observations at 89 GHz and 210 GHz towards GG Tauri. The
    principal investigator on these observations was Stephane Guilloteau
    (IRAM project G067). The data were taken on several different dates in
    1997.

iii) The representative dataset was reduced in CLIC/GILDAS by IRAM, and
     intermediate and final calibration and imaging results were made
     available to AIPS++.

iv) AIPS++ installed a build of the CLIC/GILDAS software system at NRAO,
    with assistance from IRAM, who also provided guidance in
    reducing IRAM interferometer data in CLIC.

v) AIPS++ examined the algorithms in CLIC/GILDAS relevant to the
   test. There were extensive, helpful discussions between AIPS++ and
   IRAM regarding the details of how these algorithms are implemented in
   CLIC/GILDAS, and how they are routinely used in Plateau de Bure data
   reduction. The primary new algorithms which were required in AIPS++
   are primarily concerned with the selection of radiometrically
   corrected or uncorrected data, and polynomial solutions for both the
   bandpass response and amplitude and phase solutions over time (as
   opposed to traditional self-calibration in separate frequency or time
   bins).

vi) An important part of this test has been personal interaction on the
    technical details involved in the test. NRAO and IRAM have hosted
    reciprocal visits during the course of the test thus far, as follows:
    a) Dominique Broguiere (IRAM) visited NRAO in Socorro from September
    9-22, 2001; b) Kumar Golap (NRAO/AIPS++) visited IRAM from November
    27th to December 8th, 2001, and spent one day (December 10th) at
    Observatoire de Paris; c) Athol Kemball (NRAO/AIPS++) and Kumar Golap
    (NRAO/AIPS++) visited IRAM from February 6th to March 5th, 2002, and
    Observatoire de Paris on March 6th.

vii) The partners in the test have also worked together on sharing
     information on development within AIPS++. This has included assistance
     with configuring an AIPS++ build at IRAM, informal instruction on
     development using the AIPS++ library and the contents of the library,
     and short programming tasks related to the test. AIPS++ code resulting
     from the test have been distributed to the partners and this effort
     will continue with more specific programming descriptions of the
     various changes in AIPS++ specific to the IRAM reduction.

viii) As an example of code re-use, the atmosphere model of
      J. Cernicharo et al., as implemented as ATM in CLIC/GILDAS, was
      integrated into AIPS++ as a tool available at the command line.

ix) A filler was developed by AIPS++ to read and process the ALMA-TI FITS
    export data format and convert it into the native AIPS++ uv-data
    format. No major data format changes in AIPS++ were required, but the
    on-site radiometrically corrected data were added as an
    instrument-specific column in the AIPS++ dataset. This is within the
    guidelines for the AIPS++ data format, which accommodates many diverse
    telescopes in a common core format, with instrument-specific
    extensions as required.

x) The CLIC task PHCOR 0 was implemented in AIPS++ as a Glish
   script. This selects between radiometrically corrected and uncorrected
   data based on the CLIC quality metric.

xi) The Chebyshev and spline polynomial fitting applications for bandpass
   (RF), and phase (PHAS) and amplitude (AMP) solution versus time were
   implemented within AIPS++ in C++, re-using the FORTRAN fitting kernel
   from CLIC/GILDAS.

xii) The flux-density scale computation performed by CLIC task FLUX was
     implemented in AIPS++ as a Glish script.

xiii) No major changes in the AIPS++ imaging software have proved necessary
      for the test thus far, although some small defects were corrected in
      channel selection, made visible by the IRAM data. The Phase I dataset
      has been filled into AIPS++ from ALMA-TI, a bandpass has been solved
      and applied using the RF algorithm, and a phase solution determined
      and applied using the PHAS algorithm. A resulting AIPS++ image is very
      close to that produced by CLIC, but quantitative comparisons are
      underway at present, both of the final image and intermediate
      calibration solutions. The calibration solutions are very close by
      visual inspection.

xiv) A full report on Phase I and Phase II is under joint preparation by
     Athol Kemball (NRAO) and Robert Lucas (IRAM) within an agreed outline
     which will address all questions posed for the test, as listed above.

4. Current work

The current focus is on completing the packaging and user interface to
the Phase I tools in AIPS++, as a tool called "clic". The user
interface and users guide for this tool will need to be complete
before Phase II.  Final test of AMP and PHCOR non-zero as well as
phase-transfer from 3mm to 1mm are being concluded at present with the
code that has already been developed for this purpose. Further
reciprocal visits are planned in April to continue Phase II, and to
continue the AIPS++ development training and related discussions,
as well as complete the final report due at the end of April.