Testing - NGC4826c regression script (synthesis+SD combo) DSS - 26 & 27feb07 ----------------------------------------------------------------------- Summary of issues: --------------------- - problems with printing/saving an image.ps file from the viewer: I specified the output file name, clicked on Print buttong. I just want to save to a file and ghostview, not actually print. The Viewer Printer Control GUI makes the printer name red and won't send this to ghostview. If I just want to ghostview and save to disk, I shouldn't need a printer name. The save button doesn't seem to work. The popup help says: "press this button to save an X11 pixmap to disk" but then it gives me a choice of saving ps, eps, ... The popup help seems to be wrong (or I don't understand it). When I select 'ps' it resets my outfile name back to "unset" -BUMMER. and then does nothing. The only way I can get an image saved to disk is to use the print button (which also forces me to send the image to a printer). Can this be fixed? People will want to make print outs of their hard earned images. E.g. we need to provide some form of gratification besides coffee and chocolate. - Note: there is no task to do joint deconvolution - must use the tool kit (fairly complicated). Will the tool-kit cookbook (https://wikio.nrao.edu/bin/view/Software/Cookbook-Tool-based) be available and updated for the test? - 4 warning messages are generated during the script. 2 are messages that should not be there in setjy and regrid (e.g. a warning is issued when there is nothing wrong or out of place). 2 are real and valid messages that tell the user they must do something. My standard request: WARN messages given when nothing is wrong should be deleted: For setjy, if the spwid is not selected, don't mention it at all. For regrid, if the axis is of unit length, give a Normal-level message. For flagdata, if the "chunk" is not selected, don't mention it. Note, I would like to see the "chunk" terminology disappear completely. Give spwid, field, pol as the identifier for what is being processed and then show the little percent numbers count up, giving a warm, fuzzy feeling that the correct data is being selected and flagging is proceeding as planned. - Note that the regression message at the end of the run says: ----FAILED Regression test for NGC4826 <------------- #### This is ODD, it passed all the individual sections but the final combined grade it says it FAILED. Maybe I don't understand python syntax but it looks to me like this should have passed. Please look at this to see what goes. - imhead('image.im') # produces both a nice logger output and an extraction of the code, in the command line window, e.g.: {'header': {'axisnames': ['Right Ascension', 'Declination', 'Stokes', 'Frequency'], 'axisunits': ['rad', 'rad', '', 'Hz'], etc... Please suppress the command line output (or format it nicely like in the logger so it is not confusing. - viewer table.lock issue - a few more details: viewer ia.open('mosaic.image') ia.moments(outfile='mosaic.mom0', moments=0, axis=3, mask='indexin(4,[4:25])', includepix=[0.09,10.0]) ia.close() # I closed out the viewer with the 'x's in upper right corner. Then # restarted it: viewer tb.clearlocks() # table lock problem again: # Process 4257: waiting for read-lock on file /home/sola/dss/casa.testing/casa/regression/ngc4826c/mosaic.mom0/table.lock # what is odd is that I asked to start the viewer but did not specify a dataset or image. And I definitely closed out the ia function. - Problem with CASA mosaic (will be solved soon): AIPS++ mosaic is larger: mosaic commands were: im:=imager('n4826_both.ms'); im.setdata(fieldid=[1:7],spwid=[1:4]); im.setimage(nx=256, ny=256, cellx='1arcsec', celly='1arcsec', stokes='I', mode='channel', nchan=30, start=47, step=4, fieldid=1, spwid=[1:4]); im.setmfcontrol(scaletype="NONE" , minpb=0.1); im.setvp(dovp=T); im.setoptions(ftmachine='mosaic'); CASA mosaic command is: mosaic('n4826_both.ms', 'mosaic', nchan=30, start=46, width=4, spwid=range(3), fieldid=range(7), cell=[1.,1.], imsize=[256,256], stokes='I', mode='channel', mfalg='mfclark', niter=500) mosaic help says: scaletype -- Image plane flux scale type default='NONE'; example: scaletype='SAULT' minpb -- Minimum PB level to use default=0.1; example: minpb=0.15 This is striking - the data and image setup are exactly the same and yet the CASA mosaic is about 1/2 the size. Further, the noise level is much higher on the CASA mosaic edges than in the aips++ image. But they both use the same scale type and same minpb. It looks like the CASA mosaic is not being properly weighted by the primary beam (is dovp=false?) or the scaletype is wrong or something else has gone bonkers. created 2 ps files for comparison: bima.mosaic.mom0.app.eps - aips++ image made with above parameters bima.mosaic.mom0.eps - casa image made here. This was the problem I had with the NGC7538 mosaic I tried to make over the weekend - I needed to make a larger mosaic but there was no way to control it. #### Whew - solved. Kumar had disabled the minpb parameter and then Joe had requested that the ftmosaic gridding performed Sault weighting to bring the noise level to uniform within the mosaic field. Kumar will enable the minpb parameter. Joe will add a toggle to the mosaic parameter fluxscale=true/false (default = true), make sure the fluxscale image is still generated and then document this all of this. ----------------------------------------------------------------------- ----------------------------------------------------------------------- Actual detailed test notes: alias casadaily 'source /home/ballista/casa/daily/casainit.csh; casapy' casadaily Regression scripts: https://wikio.nrao.edu/bin/view/Software/CASA /home/casa/code/xmlcasa/scripts/ NOTE: refant = NAME now, not ID # ==> previous scripts will get the wrong antenna because they specified ID #. Useful commands to work around problems: ------------------------------------------ tb.clearlocks() mp.clearplot restore() # source /home/ballista/casa/daily/casainit.csh; casapy ----------------------------------------------------------------------- make sure I have the latest version of the script: cp /home/casa/code/xmlcasa/scripts/g192_regression.py . casadaily execfile 'ngc4826c_regression.py' datapath Out[7]: '/home/ballista/casa/daily/data/regression/ATST3/NGC4826/' listobs('/home/ballista/casa/daily/data/regression/ATST3/NGC4826/n4826_mom0.im', verbose=true) # Interesting, listobs only works for ms's imhead('/home/ballista/casa/daily/data/regression/ATST3/NGC4826/n4826_mom0.im') # can't this be in a better format? This is just an extraction of the code, e.g.: {'header': {'axisnames': ['Right Ascension', 'Declination', 'Stokes', 'Frequency'], 'axisunits': ['rad', 'rad', '', 'Hz'], etc... Ah, check it out. this code extraction was in the command line window. The logger has a nice formatted summary Tue Feb 27 20:33:59 2007 NORMAL image::summary: Direction reference : J2000 Spectral reference : LSRK Velocity type : RADIO Rest frequency : 1.42041e+09 Hz Pointing center : 12:56:44.240000 +21.41.05.100000 Telescope : BIMA Observer : Date observation : 1998/04/16/03:33:47 Axis Coord Type Name Proj Shape Tile Coord value at pixel Coord incr Units ------------------------------------------------------------------------------------------------ 1 1 Direction Right Ascension SIN 256 256 12:56:44.240 129.00 -1.000000e+00 arcsec 2 1 Direction Declination SIN 256 256 +21.41.05.100 129.00 1.000000e+00 arcsec 3 2 Stokes Stokes 1 1 I 4 3 Spectral Frequency 1 1 1.415e+09 1.00 1.000000e+03 Hz Velocity 1140.94 1.00 -2.110611e-01 km/s - suppress the command line window output please. ## Interesting use of range: fieldid=range(7) fieldid Out[8]: [0, 1, 2, 3, 4, 5, 6] WARN issue: --Feather-- WARN ImageRegrid::regrid(...) (file /home/ballista/casa/daily/code/include/images/Images/ImageRegrid.cc, line 1982): Cannot regrid axis 4 because it is of unit length - removing from list # Joe said that this is benign (but disconcerting to the user) ## WARN messages during regridding o the 12m channel image: ia.open('n4826_t12mchan.im') ia.regrid(outfile='n4826_t12motf.chregrid.im', shape=[256,256,1,30],csys=csys.torecord()) ia.close() WARN ImageRegrid::regrid(...) (file /home/ballista/casa/daily/code/include/images/Images/ImageRegrid.cc, line 1982): The Stokes axis cannot be regridded - removing from list # benign. WARN Deconvolver::open() (file /home/ballista/casa/daily/code/synthesis/implement/MeasurementEquations/ Deconvolver.cc, line 205): No psf given; please define one before deconvolving WARN Deconvolver::open() (file /home/ballista/casa/daily/code/synthesis/implement/MeasurementEquations/ Deconvolver.cc, line 205): Use the function open with the psf # these last 2 warning messages are apparently real warnings to the user that they have to make and insert a Gaussian psf into the regridded image before it can be used for deconvolution. - Note: there is no task to do joint deconvolution - must use the tool kit (fairly complicated). Will the tool-kit cookbook (https://wikio.nrao.edu/bin/view/Software/Cookbook-Tool-based) be available and updated for the test? viewer - mosaic.image - not a great image, some stripes but also no deconvolution region was set. channel with no emission: n Std Dev RMS Mean Variance Sum 7656 0.09080 0.09103 -0.006633 0.008244 -50.78 Flux Med |Dev| IntQtlRng Median Min Max -0.9638 0.05789 0.1155 -0.007171 -0.3566 0.3553 Humm, looks like I can't get stats on the whole image cube. channel with brightest emission: n Std Dev RMS Mean Variance Sum 5900 0.2612 0.2672 0.05645 0.06822 333.0 Flux Med |Dev| IntQtlRng Median Min Max 6.320 0.07684 0.1534 0.002829 -0.3828 1.846 comparison with earlier aips++ processing: # bima-only stats (using clean deconvolution): # Peak = 841 mJy/bm # RMS = 97.5 - good comparison. # use ngc1333 regression script to figure out how to make a moment map. ia.open('mosaic.image') ia.moments(outfile='mosaic.mom0', moments=0, axis=3, mask='indexin(4,[4:25])', includepix=[0.09,10.0]) ia.close() # worked like a charm. I closed out the viewer with the 'x's in upper right corner. Then restarted it: viewer tb.clearlocks() # table lock problem again: # Process 4257: waiting for read-lock on file /home/sola/dss/casa.testing/casa/regression/ngc4826c/mosaic.mom0/table.lock # what is odd is that I asked to start the viewer but did not specify a dataset or image. And I definitely closed out the ia function. - mosaic.mom0 - on source peak: n Std Dev RMS Mean Variance Sum 5358 21.91 31.27 22.31 479.9 1.195e+05 Flux Med |Dev| IntQtlRng Median Min Max (no beam) 7.899 18.71 15.28 1.476 161.4 - Area with little emission: n Std Dev RMS Mean Variance Sum 3238 6.359 13.60 12.02 40.44 3.891e+04 Flux Med |Dev| IntQtlRng Median Min Max (no beam) 4.381 8.802 11.33 1.465 37.31 - when I processed this mosaic in aips++ I got: bima mom0 Peak = 162.7 Jy/bm-km-s RMS = 12.2 Jy/bm-km-s - good comparison. Trying to print bima.mosaic.mom0.eps I specified the output file name, clicked on Print buttong. I just want to save to a file and ghostview, not actually print. The Viewer Printer Control GUI makes the printer name red and won't send this to ghostview. If I just want to ghostview and save to disk, I shouldn't need a printer name. The save button doesn't seem to work. The popup help says: "press this button to save an X11 pixmap to disk" but then it gives me a choice of saving ps, eps, ... When I select 'ps' it resets my outfile name back to "unset" -BUMMER. and then does nothing. The only way I can get an image saved to disk is to use the print button (which also forces me to send the image to a printer). Dach. Now, I copied over offline/tst3.may05/bima.song.ngc4826/n4826_mom0.im to here and looked at the mosaic shape: closed and opened the viewer (no table lock problem) viewer # there are significant differences in the low level emission but the peak is the same. AIPS++ mosaic is larger: mosaic commands were: im:=imager('n4826_both.ms'); im.setdata(fieldid=[1:7],spwid=[1:4]); im.setimage(nx=256, ny=256, cellx='1arcsec', celly='1arcsec', stokes='I', mode='channel', nchan=30, start=47, step=4, fieldid=1, spwid=[1:4]); im.setmfcontrol(scaletype="NONE" , minpb=0.1); im.setvp(dovp=T); im.setoptions(ftmachine='mosaic'); CASA mosaic command is: mosaic('n4826_both.ms', 'mosaic', nchan=30, start=46, width=4, spwid=range(3), fieldid=range(7), cell=[1.,1.], imsize=[256,256], stokes='I', mode='channel', mfalg='mfclark', niter=500) mosaic help says: scaletype -- Image plane flux scale type default='NONE'; example: scaletype='SAULT' minpb -- Minimum PB level to use default=0.1; example: minpb=0.15 This is striking - the data and image setup are exactly the same and yet the CASA mosaic is about 1/2 the size. Further, the noise level is much higher on the CASA mosaic edges than in the aips++ image. But they both use the same scale type and same minpb. It looks like the CASA mosaic is not being properly weighted by the primary beam (is dovp=false?) or the scaletype is wrong or something else has gone bonkers. created 2 ps files for comparison: bima.mosaic.mom0.app.eps - aips++ image made with above parameters bima.mosaic.mom0.eps - casa image made here. This was the problem I had with the NGC7538 mosaic - I needed to make a larger mosaic but there was no way to control it. It seems like the CASA mosaic algorithm #### Whew - solved. Kumar had disabled the minpb parameter and then Joe had requested that the ftmosaic gridding performed Sault weighting to bring the noise level to uniform within the mosaic field. Kumar will enable the minpb parameter. Joe will add a toggle to the mosaic parameter fluxscale=true/false (default = true), make sure the fluxscale image is still generated and then document this all of this. ---------------------------------- Regression stats printed to log file: ********** Data Summary ********* * Observer: Project: t108c115.n48 *Observation: BIMA(10 antennas) * Telescope Observation Date Observer Project * BIMA [ 4.39941e+09, 4.39942e+09] t108c115.n48 *Data records: 109260 Total integration time = 628492 seconds * Observed from 04:04:31 to 10:39:23 *Fields: 7 * ID Name Right Ascension Declination Epoch * 0 NGC4826-F0 12:56:44.24 +21.41.05.10 J2000 * 1 NGC4826-F1 12:56:41.08 +21.41.05.10 J2000 * 2 NGC4826-F2 12:56:42.66 +21.41.43.20 J2000 * 3 NGC4826-F3 12:56:45.82 +21.41.43.20 J2000 * 4 NGC4826-F4 12:56:47.40 +21.41.05.10 J2000 * 5 NGC4826-F5 12:56:45.82 +21.40.27.00 J2000 * 6 NGC4826-F6 12:56:42.66 +21.40.27.00 J2000 *Spectral Windows: (4 unique spectral windows and 1 unique polarization setups) * SpwID #Chans Frame Ch1(MHz) Resoln(kHz) TotBW(kHz) Ref(MHz) Corrs * 0 64 LSRD 114950.387 1562.5 100000 115271.2 YY * 1 64 LSRD 115040.402 1562.5 100000 115271.2 YY * 2 64 LSRD 115130.143 1562.5 100000 115271.2 YY * 3 64 LSRD 115220.157 1562.5 100000 115271.2 YY *Antennas: 10 * ID= 1-3: ANT1=UNKNOWN, ANT2=UNKNOWN, ANT3=UNKNOWN, ANT4=UNKNOWN, * ID= 5-7: ANT5=UNKNOWN, ANT6=UNKNOWN, ANT7=UNKNOWN, ANT8=UNKNOWN, * ID= 9-9: ANT9=UNKNOWN, ANT10=UNKNOWN ********************************* ********** Regression *********** * * * Passed Feather 1 image max test *-- Feather 1: Image max153.349822998,153.3498 * Passed Feather 2 image max test *-- Feather 2: Image max 1.88162565231,1.88415 * Passed Feather 3 image max test *-- Feather 3: Image max1.92072474957,1.93 * Passed Joint Deconvolution 1 image max test *-- Joint Decon1: Image max1.84647893906,1.846 * Passed Joint Deconvolution 2 image max test *-- Joint Decon2: Image max1.78493392467,1.784 * Passed Feather 1 flux test *-- Feather 1: Flux 1523.51520532,1523.515 * Passed Feather 2 flux test *-- Feather 2: Flux 105.462814824,105.4628 *-- Feather 3: Flux 216.057406679,189.259 * Passed Joint Deconvolution flux test *-- Joint Decon1: Flux 73.3946071367,73.969 * Passed Joint Deconvolution flux test *-- Joint Decon2: Flux 198.311397143,198.31 ----FAILED Regression test for NGC4826 <------------- ######### ********************************* ********* Benchmarking ***************** * * Total wall clock time was: 608.099266052 Total CPU time was: 565.15 Processing rate MB/s was: 1.24979595015 * Breakdown: * * feather time was: 1.80597400665 * feathercube time was: 9.3050031662 * feathersynth time was: 181.157290936 * sdmodel time was: 176.527826071 * joint decon time was: 239.30316186 ***************************************** This is ODD, it passed all the individual sections but the final combined grade it says it FAILED. viewer ---------------------- end of testing for today.