Data Analysis

Two types of data analysis were required for mapping: (1) off-line data analysis (before and after importing the data into AIPS) for further, more careful identification of bad data, and (2) data analysis involving 3D inversion, deconvolution and phase calibration for the purpose of imaging. This section describes the procedure used for the data analysis required for data editing and calibration. Section 4.5 describes the procedure used for mapping.

Data Editing

In addition to the data flagging information derived from on-line system and data monitoring, the data was also examined off-line using the programs xtract, rantsol and badbase (see Chapter 3). In long observing sessions, there were invariably antennas which did not produce usable signals for some fraction of time. Often, this was either due to th malfunctioning of a subsystem or the antenna being taken for some kind of maintenance work over time scale much longer than the threshold time intervals set in the on-line monitoring programs (see Section 4.3.1). Such situations were easily identified from the plots of amplitude from all baselines (or for all baselines with a single antenna) as a function of time for the calibrator scans. The calibbp program was also used for bandpass calibration of the data, using the phase calibrator scans. The antenna-based average bandpass solutions and the baseline based band-passes were examined to identify frequency channels affected by intermittent narrow band RFI or by any other source of data corruption.

The program badbase uses the output of rantsol to generate a summary of the antenna based solutions (see section 3.5.2, Chapter 3). This was found to be very useful in identifying intermittent baseline based problems in the data. The AIPS tasks used for calibration are rather sensitive to the presence of bad baselines in the data. Flagging data corrupted by baseline based errors was therefore essential. The algorithm for the computation of antenna based complex gains implemented in rantsol is robust in the presence of such data and was hence crucial for the identification and flagging of such data.

The above mentioned procedure was followed for data from both the polarization channels separately. All the flagging information generated from on-line monitoring and off-line data analysis, was converted to flagging tables to be used later for flagging data in AIPS.

Data recorded in the native LTA format was imported to AIPS via the FITS format. The program gl2fit was used to convert the LTA database into FITS format. It was noticed that data for some of the baselines was in the illegal number representation identified as NaN (not-a-number) or Inf (infinite) numbers. Most of the data analysis programs (including the robust algorithm of rantsol) behave erratically in the presence of such numbers. However, computer representation of such numbers is well documented in the IEEE number representation format and can be easily and reliably identified in the software. This form of bad data was therefore removed using an on-the-fly filter used in all off-line data analysis programs, including gl2fit. Another form of bad data manifests itself in the form of the normalized visibility amplitude being greater than unity. This is again easily identified in the software, and the gl2fit program flags such data while converting it to the FITS format.

Due to synchronization problems between the various network programs of the GMRT data acquisition software and the online array control software, the time stamp of the data records is sometimes corrupted and the value of time stamp of the successive data records does not increase monotonically. Data with such time stamp corruption is unusable in AIPS. Also, a few data records at the beginning of each scan were found to be regularly bad. All such data records were filtered out using the program tmac. tmac was used as a data filter in a data pipe-line, before the program gl2fit. If the observation was split into a number of LTA files, the program ltacat was also used before tmac in the data pipe-line to concatenate the various LTA files into one file. LTA files for some observations also required changing the values of some of the keywords in the Global an Scan headers of the LTA data base. This was done using the program fixit.

A typical data pipe-line set up to convert the and LTA file to FITS format was therefore, [ltacat] $\Rightarrow$ [fixit] $\Rightarrow$ tmac $\Rightarrow$ gl2fit, where ' $\Rightarrow$' indicates the flow of data and the parenthesis are used to indicate that the program were used only if necessary. Use of UNIX pipes eliminated the necessity of saving the intermediate LTA files, which would otherwise have required several giga bytes of disk space.

Further data editing was done after importing the data into AIPS. The bad data/ antennas/ baselines identified earlier were translated to AIPS readable flagging files and applied inside AIPS. Various steps used for further data editing and calibration inside AIPS are described below.

Data Editing and Calibration in AIPS

The first step in the sequence of data analysis is the amplitude and phase calibration of the visibilities. This section describes, in a stepwise fashion, the sequence of various AIPS tasks that were used for data calibration. A typical AIPS task depends on a large number of parameters, which control the behavior of the tasks. The settings of the relevant parameters of these tasks are also discussed below.

1. The FITS file generated using the procedure described above, was imported into AIPS using the task FITLD. This results into a multi-source visibility database written in AIPS, by default in the compressed format.

2. Normally, multi-source file contains a number of tables, including the CL table, version 1 (CL1)

   At present, the CL1 table is not generated when the GMRT data is converted into FITS format. This table is also used by AIPS calibration programs as a template to determine the time resolution for the subsequent CL and antenna gain tables (the SN tables). Hence, the task INDXR was always run to generate the table CL1. This also generates the NX table, which is used by AIPS to navigate in the database. This, by default, produces a CL table with a time resolution of 5 min. Since the phase calibrators used for all the observations were strong enough to provide sufficient signal to noise ratio, the antenna based gain solutions could be computed for every integration cycle of $\sim 17$ sec. The CL and SN tables were later used to not only calibrate the data but also identify bad data. The minimum time resolution for gain solutions was therefore set to $\sim 20$ sec by setting APARM=0.33,0 before running INDXR.

3. The flux density for the flux calibrators must be accurately set before proceeding for calibration. The flux density scale of the standard VLA flux calibrators 3C48 and 3C286 is encoded in the task SETJY. This task was run on the multi-source visibility database to set the flux densities for these flux density calibrators using the following settings: OPCODE='calc'; SOURCE='3C286','3C48',''.

4. The task UVFLG reads the data flagging information from a disk file supplied via the INFILE keyword. This task was used to apply all the flagging information generated from the on-line monitoring and off-line data analysis described earlier.

5. The visibility amplitude of an unresolved source remains constant as a function of baseline length. Plot of visibility amplitudes as a function of $\sqrt{u^2 + v^2}$ for the flux density calibrator scans was examined using the task UVPLT. The SOURCE keyword was appropriately set for the purpose. Obviously discrepant data were identified, after allowing for some spread due to differences in antenna sensitivities. Such data, which either had very low or very high amplitude can have severe repercussions for calibration.

   UVFLG was used to flag these discrepant points with INFILE=''. The keywords ANTENNA, BASELINE, and TIMERANGE were used to specify the offending antennas/baselines and any time range for which the data was required to be flagged.

   The primary task for calibration in AIPS is CALIB. This task is, however, rather sensitive to the presence of bad/dead antennas. (e.g., in one case (G356.3-1.5) the presence of about 10 bad baselines out of a total of about $\sim$360 good baselines (for antennas C06 and C05) gave severely under estimated antenna gains). The initial effort outside AIPS to identify and flag bad data/baselines paid good dividends at this stage of processing.

6. In some cases, the graphical flagging task TVFLG was also used to interactively flag bad data.

7. CALIB was then used to compute the antenna based complex gains for the flux density calibrator scans with SNVER=0, GAINUSE=0, SOLINT=0.33. This generates the first antenna gains table (SN1) containing complex gain solutions with a time resolution of $\sim 20$ sec.

8. To further identify bad data, the data was examined using UVPLT and VPLOT with DOCALIB=1 and SNVER=1. With these settings, the SN1 table is applied to the data on-the-fly. This essentially takes care of any short term variations in the antenna gains as well as variations in the sensitivity of the antennas.

   This procedure helps in identifying mildly discrepant data, which cannot be corrected by any antenna based correction. As before, UVFLG was used to flag such data. SN1 was then deleted using the task EXTDEST with INEXT='SN' and INVERS=0. This deletes the latest SN table generated in any previous run of CALIB. Steps 5 to 8 were repeated till a satisfactory SN1 table was generated for the flux density calibrator.

9. The task CLCAL was then used with INTERPOL = 'MWF'; INTPARM = 0.5,0.5; CALSOURCE = '3C286','3C48',''; SOURCE = '' to generate the second version of CL table (CL2) containing the median window filtered solutions derived from SN1. The resulting table CL2 contains flux calibration for all sources in the database and constitutes the basic flux calibration table.

10. Steps 5 to 9 were repeated for each phase calibrator to identify and flag bad data. This resulted into the third CL table (CL3) containing the median window filter solutions for the antenna gains, derived from the phase calibrator scans; this was used later to correct for slow variations in the antenna based complex gains.

    A fourth CL table (CL4) was also generated containing the the gain solutions for the phase calibrator with a time resolution of $\sim 20$ sec (INERPOL='2PT'). This was later used for band pass calibration.

11. GETJY was then used with CALSOURCE='3C286','3C48' and SOURCE='' to derive the flux density of the phase calibrator. If all was well in the above procedure, the derived flux density should be close to the value listed in the VLA calibrator manual and the error bars of less than 10% (after multiplication by the ratio of the system temperatures for the flux and phase calibrator fields, since the GMRT visibility database as yet does not contain the $T^{sys}$ table). This corrects for the difference in the system temperatures between the flux and phase calibrators.

12. In all the above, a single, clean frequency channel, free from RFI, was used. All the calibrators used were strong enough to warrant the use of a single frequency channel. However, to use other channels for the purpose of imaging, a bandpass calibration also needs to be applied to calibrate any channel dependent gain variations.

    In the calibration procedure adopted here, it is assumed that the time calibration (determined in the above procedure) and bandpass calibration can be separated and determined independently. The time calibration table (CL4) was therefore applied to all the channels before deriving the band pass calibration table, the BP table. AIPS offers interpolation of the BP table in time to apply band pass calibration to data of the target source. Hence, the time calibrated data within the calibrator scans needs to be averaged, to generate a scan averaged band pass solution. The resulting band pass solutions (one per calibrator scan) can then be interpolated in time to take care of any slow variations in the band shape.

    However, data affected by intermittent RFI needs to be flagged before the data is averaged in time. RFI on calibrator scans was identified using the task FLGIT on these calibrator scans. This task examines the data after subtracting a linear fit to the band shapes from individual baselines. A user specified set of channels is used to determine the linear fit. All data with residuals outside the user specified limits are then flagged. All channels were flagged for a given integration time containing bad data. This was achieved with the following settings for FLGIT: BCHAN=C0; ECHAN=C1; DOCALIB=1; GAINUSE=4 where, C0 and C1 are the first and the last frequency channels to be used. Several sets of (C0, C1) for range of clean frequency channels can be specified, which alone will be used for the linear fits, via the NBOXES and BOX keywords. The flagging criterion can be specified via the APARM keyword.

    FLGIT was also used later on the calibrated data on the target sources. However, since the signal to noise ratio on individual baselines for extended sources can vary a lot, such automated procedures are of limited use. Identification of bad data on the target source was therefore usually done manually using tasks like UVPLT, UVFLG, TVFLG and SPFLG.

13. The graphical data editing task SPFLG was sometimes used at this stage to identify and flag bad data. This task displays the data in the time-frequency plane from one baseline at a time and provides the same interface as that of TVFLG to graphically flag data.

14. Next, the BP table was derived by time averaging the data within the phase calibrator scans (after application of the time calibration). This was done using the task BPASS with DOCALIB=1; GAINUSE=4; BPASSPRM(5)=1; BCHAN=C0; ECHAN=C1 where C0 and C1 refer to the first and the last frequency channel to be used.

15. The BP table and band pass corrected band shapes were examined using the task POSSM. Large oscillations across the band were sometimes found in a few antennas. These antennas were usually flagged from the entire data base.

16. Finally, the task SPLIT was used to apply the time and band pass calibration (the CL and BP tables respectively) to the data on the phase calibrators as well as the data on the target source and single source multi channel calibrated databases generated. This was done using the following settings: DOCALIB=1; GAINUSE=3; DOBAND=3; BPVER=1.

Flux density calibration

Flux density calibration was done using observations of one of the two VLA flux density calibrators, 3C286 or 3C48. Time variability of these sources has been found to be small from the VLA monitoring of the flux densities of these sources. The absolute flux densities of these sources was derived by careful observations by Perley & Crane (1986) using the VLA in D-array configuration and they found that the Baars scale (Baars et al.1977) was slightly in error. They adjusted the flux density of 3C295 to that of Baars value and derived corrections for the flux densities of 3C286 and 3C48. These corrected flux densities are encoded in the AIPS task SETJY which was used to set the flux densities for these sources used for flux density calibration. The adopted flux densities of 3C286 and 3C48 were $28$ and $42.7$ Jy respectively (at 325 MHz). Observations of 3C48 with the VLA has shown that the flux density derived using SETJY gives the 325-MHz flux density with an accuracy of $\sim2$%.

The flux density calibrators were typically observed at the beginning and at the end of each observation. The phase calibrators used for these observations are also listed as good secondary VLA calibrators. The flux density calibrator scans were used to derive the flux densities of these secondary calibrators as a consistency check. The phase calibrators were also used as secondary calibrators to correct for any slow variations in the antenna gains. All fields observed for this dissertation also had many other sources in the field. For some of these sources, the 325-MHz flux densities were available from other independent observations as well (VLA calibrators, targeted VLA observations or the Texas survey (Douglas et al.1996) which gives the spectral index and point source flux densities at 365 MHz). These flux densities were also used for a consistency check on the flux calibration and to eliminate the possibility of any systematic flux calibration error.

The background temperature in the Galactic plane can change quite substantially for separate pointings. For accurate flux density calibration, one must measure the system temperature for the flux density calibrator field as well as for the target field. To also account for small time dependent variations in the system temperature, it should be monitored regularly during the length of the observations. The planned periodic injection of calibrated noise at the front-end of each antenna to measure the system temperature has not yet been implemented at the GMRT. In its absence, the system temperature was measured at a few positions around the target source in the Galactic plane and the measured system temperature used to correct for the differences in the background temperature between the field of interest and the flux density calibrator. The background temperature from the 408-MHz all sky survey (Haslam et al.1995) was also estimated as a consistency check. With this scheme, we estimate that the 325-MHz flux densities from GMRT are accurate to $\sim 15\%$.

Phase calibration

Slow variations of the antenna based complex gains occur on time scales of a few tens of minutes. The relative phase variations between the antennas due to this needs to be corrected so as to phase the array over several hours. These slow variations are measured using periodic observations of a phase calibrator. Since the system temperature at 325 MHz in the Galactic plane is a factor of 3-5 higher than away from the plane, the phase calibrators must also be strong (typically $>10$ Jy) to provide enough signal to noise ratio for the computation of antenna based complex gains. Temporal as well spatial variations in the ionospheric total electron content at 325 MHz is expected to be the major source of phase corruption. This can produce phase variations over the scale of the array (and sometimes even across the primary beam of each antenna). It is therefore not advisable to use a phase calibrator too far from the target field since the antenna based complex gains obtained from the phase calibrator may not reflect the phase variations in the direction of the target field.

Using the antenna based phase variation derived from continuous observations of the phase calibrators for several hours (Figs. 2.11 and 2.12), it was estimated that phase variations over a time scale of about half an hour could be approximated well by linear interpolation. This is thus the time scale at which one needs to observe the phase calibrator ($\sim30$ minutes). The three VLA 327 MHz calibrators 1709-299, 1830-36, and 1822-096 with 327-MHz flux densities of 6, 28, and 13 Jy respectively, were used as phase calibrators. The angular separation in the sky between the phase calibrators and the target field was typically $10-15{^\circ}$. The maximum error in phase due to errors in the antenna co-ordinates, when the phases from the phase calibrator are transferred to the target source was estimated to be a few degrees (see Section 2.6.1). Tests done by phasing the data from one calibrator using periodic observations of another calibrator show that the array is phased over time scales of $\sim$half an hour using this procedure.

Bandpass calibration

The antenna based complex gains vary across the passband, primarily due to the antenna based band shape and residual fixed delay errors. These variations in the complex gains must be corrected before the visibilities from individual frequency channels are averaged.

As mentioned above, the phase calibrators were strong enough to provide enough signal to noise ratio for the computation of channel dependent antenna based complex gains. The antenna band shape corrections were therefore derived using the phase calibrators. An average gain was computed for each of the phase calibrator scans per channel and the linearly interpolated values applied to the target source data to correct for the channel dependent complex gains.