VLBA data calibration pipeline processing (using AIPS)

This web document is intended to provide information on the VLBA data calibration pipeline. That is, we are currently running data of selected VLBA projects correlated at the VLBA correlator in Socorro through an automated process using AIPS to provide a best effort a-priori calibration of the data.

General information on the VLBA, how to propose for observations with the VLBA, how to schedule VLBA observations, and how to reduce VLBA data can be obtained through the VLBA and AIPS homepages.

Topics in this document include:

Because of continuous updating of the scripts,
this page is also continuously being updated!

As the pipeline scripts have not finalized, and we're constantly looking for improvements and new features to make calibration of VLBA data easier, more robust, and include additions for more general application, please bear with us and check this page once in a while for changes and updates (see below). The date quoted directly above this paragraph does not necessarily reflect the latest update.

The VLBA data calibration pipeline is a 'runfile' in AIPS which is primarily based on the AIPS VLBAUTIL procedures. The VLBAUTIL procedures are described in Appendix C of the AIPS Cookbook. The VLBA data calibration pipeline performs calibration tasks and makes lots of plots and some simple images to find out whether the calibration has worked - the VLBA data calibration pipeline is not intended to produce high fidelity images. The VLBA data calibration pipeline will be available to AIPS users as part of the AIPS distribution and the Midnight Job.

At the moment the VLBA data calibration pipeline works reasonably well for simple VLBA-only experiments correlated in Socorro, at frequencies between about 1 and 15 GHz, both for continuum and spectral line experiments. It has been successfully run on higher and lower frequencies, but we're not advertising that yet for general use - in particular the spectral RFI flagging must be improved. We plan to extend and improve the VLBA data calibration pipeline gradually to calibrate these other frequencies, as well as preform polarization calibration, apply more robust data flagging and include non-VLBA antennas (in particular the single-dish VLA, GBT, and EB). Keep watching this VLBA data calibration web page..

At the moment the VLBA data calibration pipeline works according to the (bit outdated) flow chart as shown here in ps or in gif, but note that improvements and possible bug fixes are continuously implemented as experience and feedback from the users become available.

We're starting to run the VLBA data calibration pipeline on every suitable experiment from July 2002 to investigate its performance and explore the distribution of the calibrated UVFITS files to the user. Images and calibration plots are not distributed, they are only produced to check the calibration before we distribute the data to the PI. As we distribute single-source UVFITS files, the user could import the files into his/her favorite imaging package (e.g. aips++, difmap, AIPS).

The VLBA data calibration pipeline is an effort to streamline and simplify customary calibration of data from the Very Long Baseline Array (VLBA). The VLBA is operated by the National Radio Astronomy Observatory (NRAO) as a facility of the National Science Foundation (NSF) operated under cooperative agreement by Associated Universities, Inc. (AUI).

The Astronomical Imaging Processing System (AIPS) is NRAO's software package for calibration, data analysis, image display, plotting, and a variety of ancillary tasks on (mostly interferometric, radio-) astronomical data. The VLBA data calibration pipeline is an NRAO product (like AIPS itself), comes as an AIPS 'runfile' (with an associated 'helpfile'), and therefore assumes the terms and conditions applicable to using AIPS; this is mostly the GNU General Public License (GNU GPL). AIPS is copyright (c) 1995 through 2009 by AUI.

To make use of the VLBA data calibration pipeline, you must have a relatively current installation of AIPS on your computer (AIPS version 31DEC02 or later), preferably run the AIPS 'Midnight Job' (MNJ), and follow the instructions as provided in the VLBA data calibration pipeline 'helpfile'. If you do not run the MNJ, you may be restricted to using an older version. Please report any bugs you may find after making sure you are using the latest versions (the bug might already have been fixed).

Note that the VLBA data calibration pipeline comes as is, without warranty or guarantee that it works (although we try to make it work)!

[That is, we do not assume any responsibility for whatever happens (probably nothing malignant) in case you decide to use the scripts.]

When you have calibrated your data using the VLBA data calibration pipeline, it would be appreciated if you would mention the VLBA pipeline next to an acknowledgment to NRAO in the publication of your scientific results; e.g. "The VLBA data was calibrated using NRAO's "VLBA data calibration pipeline" in AIPS (Sjouwerman, Mioduszewski & Greisen, 2003, in preparation).

The VLBA data calibration pipeline is not supposed to handle, or cannot do:

Correct for bad observing weather/phase-coherence/RFI/tactic/techniques! Data from other correlators than the VLBA correlator (ie. in Socorro NM, USA) Note: data from the JIVE correlator can be processed with an EVN pipeline. High-fidelity flagging of visibilities in single (and multi-) source files Advanced imaging (multi-field & -resolution, deep cleaning, MEM, etc) Advanced hybrid-mapping (high-fidelity self-cal iterations, tricks & kicks) Apply doppler-tracking for individual spectral line sources (aka CVEL)

Updates and older versions of the VLBA data calibration pipeline scripts

We intend to archive older versions of the VLBA data calibration pipeline, along with older versions of the VLBAUTIL scripts. This should accommodate sites not running the AIPS 'Midnight Job' (MNJ) and make it possible to recreate 'features' (that is, check the results in one version with a different version if one does not trust the results). We will provide the collected older versions and keep track of new updates and their revision history at this spot in the near future. We anticipate however, that the sites running the MNJ get updates and new releases through the MNJ so that they always have the hottest versions (and their bugs? ;-) without any special intervention.

We are currently working out the details; watch this space for later updates. For now, please contact Loránt Sjouwerman if you have data in the correlator queue (i.e. your data has been observed recently) and you would like to obtain your pipeline calibrated data. Highest priority will be given to projects of novice users for which the VLBA data calibration pipeline is known to work; though the latter may change on a regular basis! Lowest priority will be projects out of range of the current capabilities, projects of experienced users, and projects observed in the more distant past.

You can try to pipeline your own data with these instructions

If you are running the MNJ, you are almost set to go. If not, you have to get the two VLBAPIPE files in the table below and put them in the appropriate places. We highly recommend that you use a local environment to place your aips pipeline files. However, if you don't want to keep the files in your local environment (see section 3.10 (page 3-16) of the AIPS Cookbook) and make the VLBA data calibration pipeline script available to other users at your institute, it is easiest to source the AIPS LOGIN.[C]SH file, and then type $CDTST to find the variable $RUNFIL and $HLPFIL (usually, if your AIPS installation is, for example, in /home/AIPS, then $RUNFIL is usually /home/AIPS/RUN, and $HLPFIL would be /home/AIPS/31DEC02/HELP). HOWEVER, the latter is highly discouraged in case you want to edit the procedure to your own likings and/or run the MNJ - always ONLY do that in your local environment! (You also wisely may not have the privileges to do that). Alternatively, pre-defined variables such as $HOME, $FITS, $DA01 (your first AIPS data disk), etc, can be used as a local environment (but a separate AIPS-runfile directory is preferred). After placing the files in one of these directories, define the non-predefined directory as a variable (e.g. in csh: setenv MYAREA `pwd` before starting AIPS!) - skip this step for HOME, FITS, etc, and if you have the MNJ enabled. Start up AIPS as usual, and if you do not use the default $RUNFIL area, but e.g. your $MYAREA area to place the pipeline scripts, remember to initialize the 'local version'. That is, within AIPS you make the files in the local directory available to AIPS by typing (e.g.) version 'myarea' or version 'home'. Then type explain vlbapipe within AIPS to read the help available.

The AIPS 'helpfile' contains information on how to use the 'runfile' and is maintained properly. In case it bombs on you, it may be necessary to also fetch the 'frozen utilities' - a version of the VLBAUTIL procedures that are known to work with the VLBAPIPE script. If it still fails, it may be that the individual underlying AIPS tasks have changed. Either update your AIPS installation, or send us a bug report (depending on your AIPS version and latest overhaul of the AIPS MNJ version) - if you use an AIPS version older than 31DEC02, don't expect the pipeline 'runfile' to work.

For both the MNJ and newly fetched files, you have to type run vlbautil and run vlbapipe (each on a separate command line) in your AIPS window to make sure you are using the latest 'runfiles'. If you forget this, either you might end up using an older version, or it will complain about not finding or recognizing the command. The same will happen if you forget to place the 'helpfile' or 'runfiles' in the appropriate directories (or misplace them).

After defining the procedure in AIPS, read the 'helpfile' for its use (type explain vlbapipe) and type inputs vlbapipe to review its current parameter settings. Invoke the VLBA data calibration pipeline by simply typing vlbapipe; it will check the inputs for possible problems, and if none are found, tell it that the tape is ready to be read (by typing 0 when the tape drive has settled down) and/or sit back and let the VLBA data calibration pipeline do it's work. Do not forget to check the plots and images it has made, as they may point to specific problems with your data set (dead IF channels, RFI, failing fringe-fitting, etc). If something is definitely wrong, usually these problems are solved by flagging the bad data (in flag/FG table one!) and rerunning the VLBA data calibration pipeline. Other common problems are that you selected a time range for your bandpass calibrator in which it has too few visibilities (e.g. when it is late on source), in which the source is below the horizon for one or more antennas, or the timerange does not correspond to the scan on your bandpass calibrator - change either or both. For other problems contact Loránt Sjouwerman

Please consult one of the following if you have any questions regarding parts of the VLBA data calibration pipeline. Most likely you want to contact Loránt Sjouwerman, but if your question obviously regards one of the other items listed in the table below, the relevant addresses are given as well.

Again, when you have calibrated your data using the VLBA data calibration pipeline, it would be appreciated if you would mention the VLBA pipeline next to an acknowledgment to NRAO in the publication of your scientific results; e.g. "The VLBA data was calibrated using NRAO's "VLBA data calibration pipeline" in AIPS (Sjouwerman, Mioduszewski & Greisen, 2003, in preparation). Thanks!

We on our side acknowledge the efforts of the Cast of AIPS, in particular Eric Greisen, for making and maintaining the AIPS-tasks and environment needed to reduce VLBI data in AIPS over the years. Chris Flatters started collecting the tasks into the VLBAUTIL procedures, which was continued by Amy Mioduszewski; the VLBAPIPE procedures heavily depend on these VLBAUTIL procedures. Finally, Greg Taylor allowed me to work on this pipeline.

Pipeline ideology / Copyrights, etc. / Current status / Scripts & releases / Pipelined projects / Points of contact

VLBA Home Page

The VLBA data calibration pipeline processing webpage originated June 5th, 2002.
Maintained by Loránt Sjouwerman: lsjouwerman@nrao.edu.

Copyright © 2009 Associated Universities, Inc.
The National Radio Astronomy Observatory is a facility of the National Science Foundation operated under cooperative agreement by Associated Universities, Inc.