From efomalon@nrao.edu Wed Jun 22 12:45:08 2005 Date: Mon, 20 Jun 2005 21:47:20 -0600 From: Ed Fomalont To: sbhatnag@aoc.nrao.edu, kgolap@aoc.nrao.edu, Joe McMullin , Steven T. Myers Subject: AIPS++ Wide-field Testing COMMENTS ON AIPS++ REDUCTION OF AXAF, Chandra Deep Field: Ed Fomalont June 20, 2005 This report is based on reductions at the AOC between June 14 and June 20, 2005. The goals were to test the wproject imaging in AIPS++ for accuracy and speed, and to compare with AIPS wide-field imaging. Since the data were edited and self-calibrated, additional comments on the type of support for EVLA needs for wide-field imaging will also be suggested. 1. MAIN CONCLUSION: The wproject imaging in AIPS++ is easy to set-up and pretty fast. I would use this for wide-field imaging in the future. Some of the support (editing, selfcalibration, data averaging) could be improved, and these will be discussed below. 2. THE DATA: All of the uv-data are in imager-a:/home/imager-a2/efomalon AXAF_ALL.ms The concatenated data set for five day times six hours of AXAF data, 9.7 million rows of data, each with RR, LL, each with 7 channels. Flagging of outliers also was done. There is some remaining low level interference in IF2. The data were obtained in CnB configuration with a resolution of 11". DATA column is that calibrated with respect to a nearby source. This calibration and editing was done in AIPS, although it could have been done in AIPS++. CORRECTED_DATA column after two rounds of self cal. AXAF_ALL_IF1.ms Data for IF1 only, 1.356 GHz AXAF_ALL_IF2.ms Data for IF2 only, 1.426 GHz AIPS_IF1.ms Data for IF1 only, as self-calibrated in AIPS AIPS_AVG_IF1.ms Data for IF1 only, as self-calibrated in AIPS, and averaged to reduce data volume to 22% of original (done in AIPS). This was the data set used for the imaging comparison in AIPS and AIPS++ below. It was faster. 3. THE IMAGING/CLEANING For the test below, we used the self-calibrated data set that was averaged in AIPS (see below). Only IF1 was imaged. More information about imaging of both IF's is given below. First, a low resolution image was made to see all of the bright sources. The image is AXAF_LOW.icln which was a 2000x2000 image with 8"grid , a uvtaper of 30", and uvrange of 10,000 wavelengths was added to the script image_aipspp.g AIPS++ used a 3000x3000 grid with 2.5" pixel separation, which covers the main primary and first side-lobe and includes all of the sources seen in the low resolution image. AIPS used 61 512x512 facets with 2.5" pixel separation to cover about the same area. EXECUTION TIMES TO REACH 0.07 Jy threshold in cleaning was tabulated. Because the algorithms are different, it is difficult to make clear comparisons between the AIPS and AIPS++, but cleaning to the same threshold seemed a reasonable criterion. AIPS++ was processed on imager-a, AIPS processing was on imager-b, both with 2 Gb of memory and similar in cpu power, and both relatively empty during the tests. AIPS++ 4,200 sec 20,000 iterations AIPS 7,000 sec 20,000 iterations The scripts are given in image_aipspp.g for aips++ image_aips.01G for aips The file needed to specify the faceting in aips is in BOX61, which is generated by the AIPS task SETFC. The resulting images are in AIPSPP_HIGH_IF1.icln AIPS++ image AIPS_HIGH_IF1.icln AIPS image Making and Clean Boxes: The images shown above were made with no clean boxes (no masking). Because of the relatively high sidelobes for this southern source, masking the cleaning areas will produce better results. This was done in both AIPS++ (using interactivemask.g) and AIPS, although the results are not shown here. In AIPS++ a mask can be made using interactivemask.g after a light clean in order to mask the bright sources. After cleaning ~1000 iterations, the mask can be changes to include more sources or large regions. Restarting clean with revised masks in AIPS++ is very easy. In AIPS, the clean boxes can be revised as execution proceeds. Cleaning (in IMAGR) can be restarted as well. But, in general, the setting of clean boxes is more convenient in AIPS++ 4. AIPS++/AIPS COMPARISON: The aips++ algorithm is about 1.8 times faster. When using the unaveraged data set, which is 4.3 times larger, AIPS++ executed about 2.5 times faster than AIPS. The images are in excellent agreement. Try blinking the two images with a pixrange of -0.00003, 0.0004. The AIPS++ image rms is 19.0 microJy; the AIPS image rms is 20.0 microJy. However, there is a slight ripple which is present only on the AIPS++ image, although it doesn't affect the rms value. The two brightest sources have a peak of 40 uJy and the entire field contains about 380 uJy; hence, the dynamic range (peak/rms) is about 2,000:1. 5. SELF-CALIBRATION AND EDITING: Both data sets were self-calibrated and edited in AIPS and AIPS++ independently. The determination of the gain solutions and the application of them to the data base were faster in AIPS++. However, the inspection of the phase and gain calibration solutions were more convenient in AIPS (SNPLT) than in AIPS++ (calibrater.plotcal). Plotcal needs more flexibility (plotting RR and LL together, taking differences of phases) in order to look for bad data. An editing and smoothing calibration program in AIPS (SNSMO) was convenient for flagging periods of bad weather and when there were gain drop-outs. Some sort of calibration editing via plotcal in AIPS++ would be very convenient. Editing was limited to clipping data which was greater than 0.7 Jy, and was done conveniently in both systems. Deeper clipping could be done by averaging the RR and LL and all seven channels. 6. DATA AVERAGING: After obtaining the best self-calibration data base, the data volume can be decreased by over a factor of four by averaging data from the same baseline on the different days. Since the imaging/cleaning various linearly with the data volume, a factor of four speed-up is essential when various cleaning and masking options are generally needed to obtain the best images. The method used in AIPS to average the day follows. This should be implemented in AIPS++ with medium priority. a. Change all time tags to hour-angle. This is done by the AIPS task TI2HA. b. Re-sort the data from time/baseline order to baseline/time order. This is done by the AIPS task MSORT, sort='BT'. c. Average the data over each baseline over intervals of hour angle. The length of the averaging time depends on baseline length (Actually the rate of change of u-v over time) and depends on the desired undistorted field of view. This task will average data on the different days at nearly the same hour angle, and also average over periods longer than the sampling time for the shorter baselines. The data weight is appropriately adjusted. This is done by the AIPS task UBAVG. d. Re-sort the data from baseline/time order to time/baseline order using MSORT. Only step c. requires some sophistication; the other steps are simply replacing the time stamp and reordering the data. With five days of data, the data volume was decreased to 1/4.5 of the original size. 6. IMAGING THE IF'S SEPARATELY: The following AIPS++ images shows that imaging with separation IF's and summing the two IF's, even with the relatively close frequencies in this experiment, will produce a better image than imaging with both IF's together. The AIPS images showed very similar differences. The various files are: AXAF_HIGH_SC3_F1.icln IF1 Clean image after several phase selfcals AXAF_HIGH_SC3_F2.icln IF2 Clean image after several phase selfcals AXAF_HIGH_SC3_FSUM.icln 0.5*(IF1+IF2) AXAF_HIGH_SC3_FDIF.icln 0.5*(IF1-IF2) AXAF_HIGH_SC3_FALL.icln Best clean using both IF's in one deconvolution. rms of fields F1 20 uJy Image IF1 alone F2 23 uJy Image IF2 alone F1-F2 16 uJy Image of 0.5*(IF1-IF2) very interesting. Noise is down, but residual of brightest source and those beyond the half power remain. PB Sidelobe sources are well-subtract. Hence, most of difference is in area <50% of main beam. Could be spectral properties of sources also. F1+F2 16 uJy Sum of images IF1+IF2 FALL 22 uJy Imaged both IF's together. Not as good as the IF summed image. These results suggest that when imaging wide-bandwidth data, the flux density changes of the sources with frequency in the field (from spectral index variation to primary beam variations) are a problem. However, imaging relatively narrow frequency ranges separately and then adding the resultant clean images will give reasonable results. Perhaps, this should be a first step in 'true' wide-band imaging. Determine the flux density versus frequency variations for the stronger sources using narrow-band imaging, and then run a true wide-band imaging/cleaning using the measured flux density versus frequency for the brighter sources. It is only these that will affect the ultimate dynamic range limit. 7. MIS-POINTING AND SQUINT-AFFECTS: The imaging here ignored any pointing changes and the VLA squint-effect. Artifacts associated with the brightest two sources probably show the effect, which become significant above 1000:1 dynamic range. This field has sufficient signal to noise so that residual pointing and the known affects of beam-squint can be incorporated, but will require sophisticated algorithms. This calibration, the flux density variation with time, may be more important than that associated with the flux density changes with frequency. 8. OUTLIER FIELDS IN GENERAL: Because these observations were at C-configuration, one large image 3000x3000 with 2.5" pixels was sufficient to cover the primary beam and the first primary beam sidelobe. However, in A-configuration a mapsize of 25000x25000 will be necessary to cover an area out beyond the first primary beam sidelobe, and there is always a chance of a strong outlier source every further out, especially at frequencies less than 1.4 GHz. Hence, the goal for the VLA and E-VLA is to make an ~8000x8000 image to cover the primary beam, with small outlier imaging fields for the more distant outliers, perhaps as many as 50. I hope that the wproject imaging scheme can be run efficiently on 8000x8000 field with about 50 outliers. SUMMARY: 1. The incorporation of the wproject algorithm for AIPS++ wide-field imaging has improved the efficiency of AIPS++ for wide-field imaging. It executes about 1.8-2.5 times faster than AIPS. This number should be regarded as approximate since it is difficult to compare the two algorithms, and it will vary under different circumstances. For example, it is unknown if this execution ratio will increase or decrease with larger-sized images, with many more facets. The other data set that needs a larger image size should be so investigated. The wproject algorithm should be implemented in AIPS. 2. The images are virtually identical and the quality from both systems are very similar. They dynamic range is about 2000:1 and is probably limited roughly equally by systematic errors than receiver noise. 3. The mechanics of setting up the cleaning, masking (clean boxes) and restarting clean are easier in AIPS++. One drawback is that the AIPS++ wide-field imaging needs at least 1 Gb of memory, whereas AIPS does not. Two mask images that were used for some of the imaging are: MANY_SOURCES.mask Mask to cover all outliers and bright sources in field BRIGHT_SOURCES.mask Mask to cover all outliers and the entire primary beam. 4. The main supporting task of self-calibration is faster using AIPS++. However, inspection of the gain results and additional flagging during regions of poor stability are easier to do in AIPS. Averaging data over many days is not yet supported in AIPS++. 5. In AIPS++, the addition of outlier fields that can be included in the wproject algorithm, should be considered with relatively high priority. For VLA and E-VLA work, images of 8000x8000 pixels with many outlier fields will cover most anticipated projects. The AIPS++ system (as well as AIPS) is not far from this goal. However, relatively slow execution make experimentation difficult. 6. To go beyond the 2000:1 dynamic range, pointing and beam-squint problems should be tackled. These are likely to be more important than wide-frequency problems which can be handled to first order by splitting up the imaging into several narrow band frequency ranges.