9.2 Loading and inspecting data

In theory,  can process data from multiple frequency bands (FQ numbers in  parlance) coexisting within the same data set. In practice, many observers prefer to separate the data for different frequency bands as soon as possible after loading the data and process each FQ number separately. If you wish to do this, you should do it immediately after performing the relevant steps in §9.2, using the task UVCOP or procedure VLBAFQS.

9.2.1 Loading data from the VLBA correlator

9.2.1.1 Running FITLD

Data generated by the VLBA correlator are loaded from DAT (or Exabyte) tape (or from disk files) into  using FITLD. First, physically load your tape and MOUNT it (§3.9), then run FITLD. Often the data on your tape will be divided into a number of separate files (corresponding to separate “correlator jobs”). In this case, run FITLD with NCOUNT set equal to the number of files on the tape (or a suitably large number), as listed on the paper index which comes with the tape. Also set DOCONCAT = 1  C R to ensure that all tape files with the same structure are concatenated into a single  file. Note that standard tape handling tasks (e.g., PRTTP and TPHEAD) can be used to inspect the tape contents.

Note that antennas, sources, frequency IDs, and other things may be numbered differently in different correlator jobs. FITLD fixes all this for you, but only if you set DOCONCAT = 1 and, better still, load as many files as possible in each execution of FITLD. To help with this, beginning with the 31DEC03 release, FITLD can load VLBA correlator data from multiple disk files so long as they have the same name plus a consecutive post-pended number beginning with 1. If you forget to put all the related data together with FITLD you can use MATCH to align the antenna numbers followed by DBCON later.

Typical inputs to FITLD would be;

This may seem a bit formidable. For straightforward VLBA observations, there is a collection of procedures to simplify matters including the loading of data. Enter

Because the data files tend to be very large, you will usually write compressed data (DOUVCOMP=1). These files take about 1/3 of the space of ‘uncompressed’ data sets, but cause information about the weights of individual polarizations, spectral channels, and IFs to be lost. There is some loss in dynamic range and sensitivity when the weight information is (partially) compromised. (See Appendix F for an expanded discussion of when to and when not to write ‘compressed’ data sets.) If your observation has more than one DAT or Exabyte tape, simply run FITLD for each tape. Setting DOCONCAT 1 and setting the output file name completely will ensure that the data from separate tapes with compatible observing band/data structure will be appended to existing  files. Generally, after loading all of your data, you will have one file for each such observing band and/or observing mode. However, observations which require multiple passes through the correlator (including MkIII Modes A, B, and C observations) will have one file per observing mode per correlation pass. Data from separate correlator passes can be concatenated using task VBGLU and/or merged with task VBMRG.

Adverb CLINT, which specifies the CL table time sampling interval, must be short compared to the anticipated coherence time. CLINT should be set such that the shortest anticipated fringe-fit interval is spanned by a few CL entries. Time sampling in the CL table that is too coarse can lead to calibration interpolation errors when applying the fringe-fit solutions at later stages of the data reduction. If the interval is made unnecessarily short the CL table may become unmanageably large.

It is recommended that corrections for digital representation of the correlated signals be performed in FITLD under control of adverb DIGICOR, but only for data from the VLBA correlator. DIGICOR should be set to one for all continuum and nearly all spectral line experiments. However, in the special case of spectra with very strong narrow features, the absence of correlator zero-padding may limit the accuracy of the quantization corrections. See the FITLD help file for further information. The details of digital correction for FX correlators can be found in Radio Science 33, 5, 1289–1296, “Correction functions for digital correlators with two and four quantization levels”, by L. Kogan.

Adverb DELCORR enables amplitude corrections for known delay decorrelation losses in the VLBA correlator, as described in  Memo 90 (1995, “Delay decorrelation corrections for VLBA data within ” by A. J. Kemball). Setting DELCORR=1 will create a correlator parameter frequency (CQ) table for each file written by FITLD. Do this for the VLBA correlator only. The presence of this table enables the delay decorrelation correction once the residual delays have been determined in fringe-fitting. These corrections will not be applied if the data were not correlated at the VLBA correlator or if the CQ table is missing. For older FITLD files the CQ table can be generated using task FXVLB and this must be done before any changes in the frequency structure of the file are made. The CQ table is used for rate and delay amplitude decorrelation corrections after residual delay and rate errors have been determined by fringe-fitting, and are being applied to the data. The CQ table has no immediate effect on the data written by FITLD but is essential for later processing.

The WTTHRESH adverb can be applied to drop incoming data with playback weights less than the specified limit. Note that data flagged in this way are unrecoverable except by re-running FITLD. The data weights are normalized to unity so good data usually have weights close to 1.0. You should examine your data carefully if you use WTTHRESH to make sure that you have not discarded too much data at this stage. Typically 0.8 or higher is good for the VLBA, but for non-VLBA stations a lower value such as 0.6 or 0.7 may be appropriate.

Calibration data have been transferred from the correlator with your data if your data include VLBA antennas and were correlated after 1 April 1999 and your IMHEADER listing shows the presence of GC, TY, WX, PC and FG tables, as in the example below. If you loaded more than one tape file, you must merge the calibration tables. Beginning with the 31DEC02 version, VLBALOAD does the merging for you. See §9.2.1.2 for additional details. Note that, as this example shows, it is possible your data have calibration transfer tables even though they were correlated before 1 April 1999. If your IMHEADER does not show GC and TY tables, you do not have calibration transfer and must manually load calibration information in from text files. Also, even if you have calibration transfer, you may still have to manually load calibration information for some non-VLBA antennas (see http://www.vlba.nrao.edu/astro/obscor/cal-transfer/ for some information in this regard).

The output files produced by FITLD are in standard multi-source format (as described in §4.1) and contain data from all the target and calibrator observations in your observation. FITLD also writes a large number of extension tables including an index (NX) table, and many tables containing calibration information. A description of the VLBA correlator table types is given in §9.7. If you are missing the CORR-ID random axis, your  release is stale (pre-15APR97) and you are strongly encouraged to upgrade to the latest release; much of the information presented in this chapter will not be usable with pre15APR97 releases of . Your catalog header should be similar to the one, obtained using verb IMHEADER, given below. If you have GC, TY, FG, WX, and PC tables as in this example data header, your data were processed with calibration transfer - see §9.2.1.2 for more details.

Note that the sort order of the output data set is listed as ** rather than TB and that there are no attached CL and NX tables. This happens when FITLD detects a what might be a sub-array condition (two frequency IDs or two sources observed at the same time) on reading the data. In clear cases, the actual simultaneous freqency IDs and sources will be reported. In this case, FITLD detected the use of multiple integration times on different baselines in the data set; this is common for SVLBI data. The message reported by FITLD in this case takes the form:

Unless any of the following criteria are met, the data written by FITLD are immediately ready for further processing.

•  If the sort code has been blanked as in this example, you must sort the data (use UVSRT or MSORT).
•  If the source subarray condition is encountered, you may need to run USUBA.
•  If the frequency ID subarray condition is encountered, you must separate the frequency IDs into separate data sets; procedure VLBAFQS will do this for you.
•  If FITLD does not leave behind CL and NX tables, you must run INDXR to create them. Procedure VLBAFIX will do all of the above for you.
•  If you wish to join together data processed in multiple correlator passes, you must run VBGLU and/or VBMRG.

FITLD can also be used to load archived  data previously written to tape using either FITTP or FITAB, as described in §5.1.2. In this case the VLBA correlator-specific adverbs, such as those enabling digital and delay corrections, are not active.

9.2.1.2 Calibration transfer

Beginning on 1 April 1999, the VLBA correlator attaches calibration information for VLBA and some non-VLBA antennas directly to the output FITS files. If your IMHEADER listing shows GC, TY, WX, FG, and PC tables, then the correlator has provided calibration information; this service is called calibration transfer. Note that projects correlated at slightly earlier dates may also have calibration transfer information. You must have 15APR99 or later version of  to take advantage of calibration transfer. Not all antennas provide all the information needed for calibration transfer to the VLBA correlator, see

The information processed by the correlator is somewhat redundant so that the calibration tables, the GC table in particular, must be merged using TAMRG, a very general and hence complicated task. Beginning with 31DEC00, there is a procedure to do this for you in the VLBAUTIL package:

You should use VLBAMCAL after you have finished loading the data from tape, but before you either change the polarization structure of the data with FXPOL (or VLBAFIX), load any calibration data for non-VLBA telescopes, or apply the calibration data. Note that VLBALOAD runs VLBAMCAL automatically when needed beginning in the 31DEC02 release.

A procedure named MERGECAL also has been provided for the same purpose with 15APR99 and later versions of . You must first compile MERGECAL, before you can examine its inputs and run it:

Note that the first step, RUN MERGECAL, does not run MERGECAL; it only loads the procedure and defines necessary adverbs. The inputs above will process GC, TY, and PC tables version 1 into GC, TY, and PC tables version 2. We strongly recommend that you preserve the original versions loaded by FITLD since if they are corrupted, they can only be recovered by re-running FITLD. You may also find it productive to examine/edit the entries in the MERGECAL’d tables using PRTAB, SNPLT, and TABED.

It is also recommended that FG version 1 be copied over to FG version 2 using TACOP and that all further flagging operations modify FLAGVER 2 (or higher). Don’t forget to set FLAGVER (in later tasks) to take advantage of the pre-loaded and user-set flagging information.

9.2.1.3 Repairing VLBA data after FITLD

As listed above, there are a variety of reasons why VLBA data may need some repair after FITLD has been run. They may need to be sorted into strict time order, to have the subarray nomenclature corrected, to be split into different frequencies, to have the polarization structure fixed, and/or to have the original index (NX) table and calibration ((CL) recreated. These repairs can all be done by the procedure VLBAFIX, which will examine the data and perform any of the necessary fixes. If the data contain subarrays then the procedure must be told to split the data into multiple subarrays (SUBARRAY=2), otherwise it will assume no subarrays and force all the data into one subarray.VLBAFIX is intended to replace VLBASUBS, VLBAFQS and VLBAFPOL, all of which can be run individually instead.

Remember that all of the VLBAUTIL procedures have HELP files with good discussions about when to use the simple procedures and when to use the tasks directly.

9.2.1.4 Sorting and indexing VLBA correlator data

If multiple integration times are used on different baselines, the VLBA correlator will write data that are not in strict time-baseline (TB) sort order. In general, task UVSRT can be used to sort randomly ordered uv data files in , but has significant disk space requirements through the use of intermediate scratch files. A special task, MSORT, has been written which uses a direct memory sort with sufficiently large buffers to accommodate the scale over which the data deviate from true time-baseline sort order. No intermediate scratch files are used and it can be significantly faster than UVSRT for this special case. MSORT competes with UVSRT in performance even in other cases, particularly when the individual visibility records are large due to many spectral channels and/or IFs. The inputs to MSORT are similar to those required by UVSRT and take the form:

Note that if the input and output file names are identical, the input file is sorted in place. In-place sorting is dangerous, but may be necessary if there is insufficient disk space for a second copy of the data set or for the intermediate scratch files required by UVSRT. Never abort an in-place sort in progress because you will destroy the integrity of your data set. VLBAFIX will perform this operation if needed (§9.2.1.3).

9.2.1.5 Subarraying VLBA correlator data

If the project was observed without using subarrays (defined as times at which separate antennas are simultaneously observing different sources or at different frequencies), this step involving USUBA is not necessary and should be skipped.

If the observations have been scheduled in separate subarrays, defined either by source or frequency selection, the subarrays should be labeled in  before proceeding any further. The VLBA correlator does not conserve subarray information, which in any event often has no unique characterization. This is specified in  using task USUBA which allows subarrays to be defined through either the input adverbs, an external KEYIN text file, or through the use of an automatic algorithm to identify and label subarrays found in the data. The automatic algorithm is recommended, but its results should be checked closely.

If you have subarrays, they need to be sorted, have the subarray nomenclature corrected, and/or have the index (NX) table and calibration (CL) version 1 table rebuilt. In this case, there is a simplified procedure to combine the three repair operation, VLBASUBS. Only use this procedure if you know you have subarrays.

The only user-controllable input is the CL table interval; see discussion above. VLBAFIX will perform this operation if requested (§9.2.1.3).

For automatic subarray labeling by USUBA, representative input parameters would be:

Sometimes FITLD erroneously identifies a subarray condition, usually because of spurious total-power data points. In such cases, you can set OPCODE = ’ ’ ; SUBARRAY = 1 to force all data into the first subarray.

9.2.1.6 Indexing VLBA correlator data

If FITLD had not written NX or CL tables or it was necessary to sort the data as described in §9.2.1.4, you must run task INDXR. INDXR will generate an NX table and, if need be, a CL table. Typical parameters for INDXR are:

Note that CPARM(4) can be set to zero unless the correlator model is required in later reduction (e.g., in astrometry or geodesy observations) The CL table sampling interval △t should be chosen subject to the same considerations given regarding adverb CLINT in the discussion of FITLD in §9.2.1.1. VLBAFIX will perform this operation if needed (§9.2.1.3).

9.2.1.7 Concatenating VLBA correlator data

Sometimes an observation is correlated using multiple passes through the VLBA correlator. In this context, multiple pass means different IFs/pass; this is due to data rate limitations in the correlator. Be careful to have FITLD load each pass into a separate disk file; otherwise a very confused data set will be produced. If it is desired to join together the IFs correlated on each pass, the task VBGLU should be used. VBGLU can only join data sets which are identical except in the frequencies covered. Task MATCH may be used to make the antenna, source, and frequency ID numbers in one data set the same as those in another data set so that they may be used as inputs to VBGLU.

The inputs to VBGLU are rather simple. Each of the input files to be glued together is specified via INNAME–IN4NAME, and an output file is specified via OUTNAME. The choice of input file 1 is no longer (in 31DEC06) important. No data are lost in the revised version of this task.