9.4 Calibration strategy

If you have multiple frequency IDs in your data, you may want to separate the data for different FREQID before performing any calibration. Use UVCOP to do this and take advantage of the opportunity to delete data flagged by the correlator with FLAGVER=1. Beginning with 31DEC07, you no longer need to re-run INDXR on the output files. Again there is a procedure to do this for you:

VLBAFQS will normally be run after searching for subarrays (VLBASUBS §9.2.1.5) and before fixing polarization labels (VLBAFPOL §9.2.1.8).

We can now begin the process of calibrating VLBI data. As the calibration process proceeds, both amplitude and phase corrections are incorporated into the CL tables. VLBI correlator output is in terms of dimensionless “correlation coefficients.” To convert to Janskys, large amplitude correction factors have to be entered into the CL tables. In addition, phase correction factors must be entered into the CL table to correct for phase offsets and ramps as functions of frequency and time. These corrections must be made so that the data can be averaged over frequency and time without loss of coherence. The process of determining the phase corrections is known as “fringe-fitting” in VLBI; see §9.4.8.

Unlike VLA users, VLBI users normally do not attempt to calibrate the absolute phase of the data using external calibrator sources. VLBI users just calibrate out the phase derivatives with respect to time and frequency. The absolute phases are normally left uncalibrated and “self-calibration” methods are then used to generate images (see §9.6). The alternative of absolute-phase calibration using an external calibrator, known as “phase-referencing” in VLBI circles, is a little more difficult in VLBI than for the VLA. In general terms, “phase-referencing” VLBI data in  is accomplished by similar methods as used for VLA data in ; be sure to read §9.4.1.2 and §9.4.8.4 for details on how to calibrate phase-referenced observations.

For astrometric data reduction methods in the reader is referred to the guide to  astometric data reduction available from within  by typing HELP ASTROMET.

Optimum fringe-fitting results are obtained if amplitudes are calibrated first, since, in this case, the data will be weighted appropriately (the  task FIXWT may be used to adjust the weights in the data to reflect the scatter of the actual data). We therefore describe the process of amplitude calibration first in §9.4.4.2. Then, in §9.4.8, we describe the calibration of residual phase using “fringe-fitting” techniques. Note however, that for observations of strong sources or observations using only VLBA antennas (where, particularly at centimeter wavelengths, the sensitivities on all baselines are roughly the same), the order of the two calibration steps may be reversed.

9.4.1 Incremental calibration philosophy

The general strategy adopted by  for calibration is, starting with the lowest version of the CL table, to incorporate step-by-step amplitude and phase corrections for a number of different effects. At each stage either an existing CL table is modified or a new version is created from a lower version by a task which applies a certain type of calibration. Note that the actual visibilities are not changed until you are satisfied that you have the best possible calibration file; at this point the task SPLIT can be used to apply the calibration information of the best CL table to the data. However at each point along the way the effect of a particular CL table on the data can be viewed using POSSM or VPLOT by setting DOCAL=1 and GAINUSE equal to the chosen CL table. Since many CL tables may be produced in the course of calibrating a VLBI data set it is important to keep a note of which effects are included in each one. Ideally one should delete CL tables which are judged incorrect and ensure that the accumulated corrections lie in the highest numbered CL table. It is suggested that version 1 of the CL table, as produced by FITLD, be copied to CL version 2 using TACOP before any calibration is begun, and that CL version 2 be used as the starting point in the calibration sequence. An effort is made within  to insure that CL version 1 is not deleted inadvertently. If this does occur however it can be re-generated using task INDXR (described in §9.2.1.6). (Note that INDXR cannot re-generate some types of information, e.g., the phase-cals inserted by the MKIII correlator so that it is important to try to preserve the first CL table.) The task TASAV can be used to back up your tables by copying all of them to a dummy file containing no data. This can be used to save a “snapshot” of the tables at various points in your data processing for insurance purposes. The tables can be copied back into your data file if necessary using TACOP.

You can also use the verb HINOTE to add comments into the history file. This can be very useful for checkpointing progress during calibration.

When you are ready to apply the calibrations, you run either SPLIT or SPLAT. Both tasks can average data in the spectral domain if appropriate. SPLAT can also time-average the data and produces multi-source data sets on output if requested.

9.4.1.1 Smoothing and applying corrections in SN and CL tables

The various stages of calibration described below produce SN tables which are then used to create CL tables using CLCAL. The ancillary tasks SNCOR, SNSMO, SNEDT, and CLCOR can be used to modify the SN and CL tables directly. It is important to choose the proper methods of interpolation in these tasks.

SN tables can be smoothed using tasks SNEDT and SNSMO before being used to update CL tables using CLCAL. SNSMO uses superior smoothing methods to those available in CLCAL and should always be used to do any smoothing of VLBI data, i.e., data with non-zero delays and rates. The adverb DOBLANK now controls which data are actually altered by the smoothing; use it carefully.

Typical inputs for SNSMO would be:

Typical inputs for CLCAL would be:

Note well that SNVER=0 means here to combine the solutions from all SN tables, GAINVER=0 means to apply the solutions to the highest numbered CL table and GAINUSE=0 means to write a new CL table. If two SN tables contain two similar attempts at finding corrections and SNVER=0 then, effectively, CLCAL will be inconsistent in applying the solutions from these two tables. Basically, CLCAL simply concatenates all SN tables and then, in 31DEC05, merges apparently identical records (same time and antenna) to eliminate blanked solutions and to complain about otherwise non-identical solutions.

The parameters INTERPOL and SAMPTYPE allow the user to choose between several different methods of smoothing the SN files followed by interpolation to the times in the CL table. Use EXPLAIN CLCAL  C R to view all the options. The default interpolation option is INTERPOL = ’2PT’, in which the SN table is linearly interpolated betwen the measurements in the SN table. Using SAMPTYPE = ’BOX’ causes the SN table to be smoothed with a boxcar function before being interpolated onto the CL table. The smoothing times for delay, rate etc. are specified in parameter BPARM. DOBLANK controls how both failed and good solutions are handled when smoothing. DOBLANK ≥ 0 replaces failed solutions with smoothed ones, while DOBLANK ≤ 0 replaces good solutions with smoothed ones. However it is recommended that SN smoothing be done prior to CLCAL using task SNSMO.

With good quality data, the INTERPOL = ’AMBG’ option should work well. Note, however, that this option uses the SN solutions immediately before and after a CL entry to make the interpolation and it uses any SN solution found for any source specified in CALSOUR. Therefore, if CALSOUR is left blank (allowing all sources) and delay and rate solutions were significantly different for different sources, then inappropriate solutions may be applied for a few minutes before or after a source change.

One way of avoiding this problem is to run CLCAL with INTERPOL = ’AMBG’ several times, once for each source, setting both SOURCE and CALSOUR to the name of the desired source with all other inputs remaining unchanged. Another way of avoiding the problem is to use INTERPOL = ’SELF’. In this option, only solutions found on a given source are used to calibrate that source and the SN table entries closest in time for that source are used with interpolation. This is not as good as doing multiple runs with the INTERPOL = ’AMBG’ option because there can be jumps in phase at points equidistant from two SN table entries.

If there are bad SN solutions, INTERPOL = ’POLY’ is used to fit a polynomial to the rate solutions and then integrate this polynomial to determine the phase corrections to be entered into the CL table.

A final note on CLCAL. It is sometimes the case that a priori information is not available for all antennas in a single format. For example, you may have system temperature information for VLBA antennas in your SN version 2 table and for non-VLBA antennas in SN version 3. You can merge this information by running CLCAL twice with the same GAINVER and GAINUSE; each time you should explicitly set the SN version number and list the antennas to be processed using the ANTENNAS adverb. If you leave the ANTENNAS adverb blank, the final CL table will contain information only for antennas present in the last SN table processed.

9.4.1.2 Running CLCAL for phase referencing observations

In particular, this example illustrates how to set the inputs for CLCAL for the specific case when phase corrections determined for the cal source ’J1636-16’ are to be transferred to the target source ’P1643-12’ in a phase referencing experiment:

It’s a good idea to always apply the calibration information to both the cal and target sources when running CLCAL for phase-referencing observations. This allows you to monitor the cal source data to check the progress of the phase calibration procedure.

9.4.2 Processing observing log and calibration information

As of 1 April 1999, the VLBA correlator provides calibration transfer information, as described in §9.2.1.2 for VLBA antennas. Experiments correlated after November 2003 also have full calibration transfer for the VLA, the GBT, Arecibo and the Bonn 100m. Consequently, you can skip §9.4.2 entirely unless you have data from non-VLBA telescopes (e.g., the VLA before November 2003, Space, other European telescope) or other correlators or you wish to process the log files manually for other reasons.

This section describes the processing of external calibration information, as supplied in ASCII log files. The information that may be used by  includes T_sys or related total power measurements, edit flags as written by the tracking stations or on-line monitor control system, weather information, and pulse-calibration data. These external data can be read into  by tasks ANTAB, UVFLG, APCAL, and PCLOD respectively, as described in §9.4.2.5, §9.4.3, §9.4.4.2 and §9.4.8.5.

You should have received information about where to obtain your calibration data. VLBA calibration log files may be obtained by ftp as described below. Similar calibration files for other participating antennas and VSOP should be obtained from the appropriate sites.

9.4.2.1 Automatic formatting of VLBA and VLBA-like log files

For VLBA antennas, the external calibration file for a given experiment can be downloaded from

The external calibration file, if suitably close to the standard VLBA format, concatenated with the gain curve file, can be automatically subdivided and re-formatted to comply with  requirements using task VLOG. Typical inputs would be:

A sequence of output text files will be created in the specified ($FITS here) directory named BZ199.* with suffixes:

1. .TSYS: T_sys calibration data, including gain curves, suitable for direct use by ANTAB. An INDEX record is constructed for each frequency ID if possible. If no match can be made to the FQ data, a warning message is printed and the INDEX keyword is omitted; the INDEX keyword must then be inserted by hand (see the HELP file for ANTAB). Gain curve entries for the frequency and time range in the uv data are copied to this output file. It is recommended to insert T_sys values at the end of scans immediately before source changes to avoid interpolation problems for sources of greatly differing flux density or use INTERPOL = ’SELF’ in CLCAL. Occasionally spurious data from previous observing runs or system startup files will end up in the .TSYS file and must be edited out by hand.
2. .FLAG: Flag data, suitable for direct use by UVFLG. In the older format “antennas=vlba_xx” the appropriate antenna and day numbers are inserted.
3. .WX: Weather data, as used by APCAL if performing an opacity solution. This file is altered to conform with APCAL requirements (e.g., WEATHER keyword) and lines with bad entries (*) are commented out.
4. .PCAL: Pulse-calibration data, for input to PCLOD. No editing is performed.

Files with suffixes .SCAN and .MKIII contain scan summaries and MkIII information and are for information purposes only.

9.4.2.2 VLA and EVN log files

VLA calibration files, named xxxxxcal.y.gz (stored in gzipped format) can be obtained from the same server and disk directory as for VLBA files. The VLA file starts with an explanatory preamble, including minor editing instructions. See §C.8 for more detailed instructions.

The EVN (European VLBI Network) prepares calibration tables in ANTAB format. Follow the links to “EVN Data Calibration in ANTAB Format” from the http://www.nfra.nl:80/jive/evn/evn.html page. There are files called xxxxx.newantab in directories named by the month and year. These files are supposed to include parameters of the gain curves as well.

9.4.2.3 SVLBI log files

HALCA calibration files may be obtained by pointing a web browser to http://www.vsop.isas.ac.jp and following the link to “HALCA Calibration”. You will need to get the .TSM files which are in a VLBA-like log format and the halca_gains.key file. Note that the .TSM files are not always kept up to date.

9.4.2.4 Manual formatting of log files

Partitioning of the calibration file can and must be done by hand if the calibration file format is sufficiently distinct from the standard VLBA format or, possibly, if it contains multiple frequency bands. If your observation used non-VLBA antennas, you will need to edit the calibration text file manually to add any log information supplied for these antennas. The necessary steps are as follows:

Extract the flagging and T_sys information from the calibration text file. You will also need to prepend gain curves to the T_sys file. Try EXPLAIN ANTAB to see an example file in the proper format. The parameter TIMEOFF should be set to zero for each station since both flag information and data are stored with UTC times. The keyword DTIMRANG is also supported which pads each flagged time interval to insure that even very short flag intervals are applied.

While older VLBA format calibration files were supplied with the ANTENNA keyword, newer VLBA format calibration files are supplied with the ANT_NAME keyword. In the former case, the file should be edited to insert the antenna numbers as listed in the AN table (use PRTAN on your  file to find these) and the absolute day numbers must be replaced by relative day numbers with respect to the  reference date. In the latter case, no adjustments to either day numbers or antenna numbers are necessary.

9.4.2.5 Loading calibration log information

The calibration information in the external text files such as T_sys and gain curve measurements are read into TY and GC tables using ANTAB. These tables are then used by APCAL to generate an amplitude solution (SN) table, allowing an optional solution for atmospheric opacity. The user is advised to read the ANTAB help file closely and check the syntax of the text file carefully.

The INDEX keyword is used to assign the tabulated T_sys data to individual  IF channels and polarizations. Up-to-date information on the usage of the INDEX keyword may be found by typing EXPLAIN ANTAB. Be careful to match up the proper polarization labels for the tabulated T_sys information. The frequency and polarization association for each IF channel in the  file can be compared (use LISTR with OPTYPE = ’SCAN’) with that at the head of the calibration text file.

The CONTROL group at the head of the calibration file is used only to specify a default index mapping. If the IF channel orders in the calibration file and the uv file are identical it is not required.

Source flux densities are not specified in the ANTAB input file. If source flux densities are required by APCAL, the source (SU) table will be searched. Use SETJY to insert flux densities if necessary.

The parameter TIMEOFF in the input file adds a time offset to the all entries. Non-VLBA stations sometimes measure the system temperature between, rather than during, scans causing ANTAB to be unable to match the measurements with the source and frequency ID. The ANTAB input parameter OFFSET serves the same purpose, but is more successful since the scan times are expanded at both ends.

ANTAB permits specification of IF-dependent and tabulated gains; the format description may be found by typing EXPLAIN ANTAB.

ANTAB can be run multiple times to append to the same TY and GC tables. Also, calibration files from separate antennas (e.g., VLA) which have T_sys data tabulated in a different format can be concatenated and processed in one run. In this case the INDEX keyword must be specified for each antenna to fix the data format.

Note that ANTAB will (usually) ignore calibration data for which there are no corresponding uv data (see the help file for ANTAB). There is one exception however: calibration data for an antenna that does not appear in the AN table will cause ANTAB to fail. If ANTAB quits under such circumstances, you have two choices. You can edit the calibration text file, removing all reference to the missing antennas; or you can use the input adverb SPARM to specify explicitly the names of antennas for which there are calibration data, but which do not appear in the AN table.

*** The use of SPARM is no substitute for careful inspection of the calibration text files. ***

Having created the input text file, typical inputs for ANTAB would be:

PRTLEV = 2 will echo the calibration file as it is processed, which can be useful in locating format errors.

9.4.3 Data editing

Before proceeding to calibrate, you should first flag any obviously bad data. In summary, initial editing is based on the flagging information supplied by the on-line antenna monitor systems, which is applied using UVFLG. This information may be extracted to a .FLAG file as outlined in the previous section or subsequent editing based on the station report logs, or elevation limits can also be performed using UVFLG. Finally, graphical editing tasks such as EDITR and IBLED may be used for interactive baseline-based editing. Until the data are converted into single-source data format, flagging information is stored in the FG table instead of being used to discard data directly. Flag tables may also be used with single-source files (at least with all tasks offering the FLAGVER adverb). One may also undo a flag operation using OPCODE=’UFLG’ in UVFLG. Note that this operation works only when the operation being undone is in the current flag table or, when there is no flag table, the data to be unflagged is actually present in the data set. (Note that many tasks delete fully flagged data when copying a data set.)

To edit uv data by reading a text file listing periods of known errors (e.g., the .FLAG text file created by VLOG) run UVFLG with the following inputs:

This will generate a FG table with entries read from $MYVLB:BC25CAL.FLAG.

To edit uv data based on elevation limits, UVFLG can be used with input parameters: