Date - 2001.03.20 Tester - S.T. Myers (AOC) Platform - Linux (kernow) Note: bugs are denoted by lines with prefix: >>>BUG: and queries are denoted by lines with prefix: >>>QUERY: while comments are prefixed by: >>>COMMENT: Some enhancement requests are given after >>>REQUEST: ------------------------------------------------------------------------------- GOAL: Sort out polarization calibration problems in K and Q band data. 1. Date - 2001.03.20 Version - Weekly (1.5 build 270) Data - /home/sandrock/smyers/VLA/Pol/20010204A/20010204AK.UVF Script - /home/sandrock/smyers/Testing/Mar01/20Mar01/2001-03-20-myers.g - include '2001-03-20-myers.g' - makemsf(fitsf='20010204AK.UVF',msf='20010204AK.ms') Fields: 8 Name: 1159+292 RA: 11:59:31.83 Dec: +29.14.43.83 (J2000 ) Name: 0713+438 RA: 07:13:38.16 Dec: +43.49.17.20 (J2000 ) Name: 0927+390 RA: 09:27:03.01 Dec: +39.02.20.85 (J2000 ) Name: 0854+201 RA: 08:54:48.87 Dec: +20.06.30.64 (J2000 ) Name: 1310+323 RA: 13:10:28.66 Dec: +32.20.43.78 (J2000 ) Name: 1331+305 RA: 13:31:08.29 Dec: +30.30.32.96 (J2000 ) Name: 1229+020 RA: 12:29:06.70 Dec: +02.03.08.60 (J2000 ) Name: 1256-057 RA: 12:56:11.17 Dec: -05.47.21.52 (J2000 ) Data records: 160234 Total integration time = 8020 seconds Observed from 04-Feb-2001/08:27:30 to 04-Feb-2001/10:41:10 Data descriptions: 2 (2 spectral windows and 1 polarization setups) ID Ref.Freq #Chans Resolution TotalBW Correlations 1 22485.1MHz 1 50000 kHz 50000 kHz RR RL LR LL 2 22435.1MHz 1 50000 kHz 50000 kHz RR RL LR LL Antennas: 29 ID= 1-6: 1=VLA:N72, 2=VLA:N16, 3=VLA:N24, 4=VLA:N56, 5=VLA:OUT, 6=VLA:E8, ID= 7-11: 7=VLA:E16, 8=VLA:N8, 9=VLA:E28, 10=VLA:W24, 11=VLA:W32, ID= 12-16: 12=VLA:E20, 13=VLA:W36, 14=VLA:E4, 15=VLA:E24, 16=VLA:W12, ID= 17-21: 17=VLA:E36, 18=VLA:N40, 19=VLA:N48, 20=VLA:E32, 21=VLA:W4, ID= 22-26: 22=VLA:W8, 23=VLA:W16, 24=VLA:W20, 25=VLA:N32, 26=VLA:W28, ID= 27-29: 27=VLA:N64, 28=VLA:E12, 29=VLA:VPT Create msplot tool using GUI Plot RR LL vs. time X-Y plot. Did some flagging in boxes, then hit REVERT, and the thing dumped a bunch of crap to the glish window like warning, .flag is not a field in [save=, print=, busy=T, select=[nrecords=4, records=[*594=[region=[blc=[30076.7988 -0.0330464132] , trc=[38824.707 0.207459614] ], valid=T], *600=[region=[blc=[30076.7988 -0.0330464132] , trc=[32018.7227 2.38403916] ], valid=T], *606=[region=[blc=[30076.7988 -0.0330464132] , trc=[34238.0664 1.90302718] ], valid=T], *612=[region=[blc=F, trc=F], valid=F]], addregion=, cancel=, cancelall=, list=, start=, clear=, stop=, editing=F, done=T, blc=[30076.7988 -0.0330464132] , trc=[34238.0664 1.90302718] ], callbacks=[button1=, buttonup=], getworldcoords=, worldcoords=, initialize=, finalize=, standardscaling=, standardlabel=, title=, topframe=[*agent*=], holderframe=[*agent*=], topmodeframe=[*agent*=], label=[*agent*=], modeframe=[*agent*=], policyframe=[*agent*=], mode=[policylabel=[*agent*=], policy=[*agent*=, setcontexts=, setcontext=, getcontexts=, hascontext=, getcontext=, setwidth=, addtowrench=, insertandemit=, insert=, get=, clear=, disable=, done=, setstatus=], list=[*agent*=, shorthelp=List all editing definitions, hlptxt=List all editing definitions, hdtxt=List all editing definitions, hlpwd=60, popwhere=0, widgetset=[frame=, button=, scale=, text=, scrollbar=, label=, entry=, message=, listbox=, canvas=, have_gui=, tk_hold=, tk_release=, tk_iconpath=, tk_checkcolor=, well, you get the idea. No error messages were in the log. Look at Field 1 (1159+292) It is too difficult to figure out what point is what antenna etc in the XY plot (that should appear in the banner instead of the useless world coordinates). Until that is available this is painful to use. Also, it should be clearly shown what the mouse buttons do (set boxes for editing only). It would be good to have cursor/keyboard commands like in difmap to identify points, switch ants and ifs etc. I guess will have to set up iteration over antennas etc. Iteration over antennas shows really bad antenna #13 I think (the summary thumbnail plot at end is nice, too bad you cant use it to go back and select an individual panel or something). >>>BUG: Tried to get it to stop between iterations, but "Pause between plots" and "Continue between plots" do the same thing - with only a short pause. Broken in daily also. Submitted as defect AOCso02416. Use graphical selection in Data Selection for Interferometer Number to select baselines vs. AN13. But gives error General selection string is [uvdist=: non-numeric type in array constructor [0.001, private.ranges[uvdist][2]] File: msplot.g, Line 1954 Stack: .() .(), ifr_number=[1014 2014 3014 4014 6014 7014 8014 9014 10014 11014 12014 13014 14014 16014 17014 18014 19014 20014 21014 22014 23014 24014 25014 26014 27014 28014] , field_id=1, data_desc_id=[1 2] ] Caught an exception! Event type=run exception=::glish_value_type(const Value *value) - unknown Glish type I dont know why this isnt working. Seems to give this error no matter what I do now. Start over. >>>REQUEST: That incomprehensible query on keeping flags needs to be something sensible, like: .--------------. .-----------------. .-------------. | Save & Apply | | Save & NotApply | | Do Not Save | | Flags | | Flags | | Flags | `--------------' `-----------------' `-------------' where it does applys current flags and saves in a new flag table does not apply new flags but does save in a new flag table for later use throws away current flags (does not make flag table) Are the flag tables cumulative? Try again. Creat msplot tool. Plots up default data vs time OK. Try to select baselines with Baseline chooser. First time it dumps warning, .data_desc_id is not a field in [=] error, boolean array index has 1 elements, array has 2 error, boolean array index has 1 elements, array has 2 error, boolean array index has 1 elements, array has 8 warning, .time is not a field in [=] warning, .time is not a field in [=] error, types are incompatible warning, .uvdist is not a field in [time=1858/11/17/00:00:00.000] warning, .uvdist is not a field in [time=1858/11/17/00:00:00.000] warning, .uvdist is not a field in [time=1858/11/17/00:00:00.000] warning, .uvdist is not a field in [time=1858/11/17/00:00:00.000] warning, .uvdist is not a field in [time=1858/11/17/00:00:00.000] and makes blank chooser. Second time it works. Did it plot what I wanted? I think so. Flag all this data. Deselect antennas and plot. I cant see a difference. Iterate over antennas. It didnt work, and it looks different (worse). Start over. 2. Try the display editor. AN13 clearly bad. Hard to get colormap that is sensible given data range (3C273/279 are very bright). Would be nice to be able to pick more transfer functions (like in xv?) like logaritmic. Histogram eq seems to help. Try to blast away AN13. I dont think it did anything. Looks like I will have to have help to do this. Leave msplot. I thought I told it to flag stuff but it clearly doesnt know this. Sigh. It is pretty clear that there is a long way to go to make msplot generally useable. 3. Use flagger from GUI. Create tool. Try to select antenna from MS - I expected this to bring up the Chooser like in msplot, but it brought up the Catalog. >>>BUG: In flagger.setantennas, select from MS should bring up the Chooser as in msplot. Submitted as defect AOCso02418. Enter [13] manually. Do all the other choices default sensibly to everything? It is not clear from the User Reference. I will assume so. It would be nice if .timerange defaulted to everything (or there was a way of setting this from the spanner). I just want to wipe AN13 from all data. Try .query(query='FIELD_ID >= 0') but from GUI. This barfed at Table query 'FIELD_ID >= 0' failed : Table query 'FIELD_ID >= 0' failed : command: Error in select expression: mismatch in operand data types Scanned so far: select from $1 where 'FIELD_ID >= 0'&&((ANTENNA1==12)||(ANTENNA2==12)) OK, lose the quotes. That seemed to work. Done. Did this actually apply flags? Display in msplot - indeed, AN13 is gone and the data looks much better! Could automate this as killant := function(msf,ants=[]) { # # Assume ms has been created - check with default os tool # if( ants == [] ) { print 'Blank antenna list - no flagging' return } if( dos.fileexists(file=msf) ) { print 'Found MS file ',msf; } else { fail "MS file does not exist"; } msfilename := msf; kfg := flagger(msfile=msf); kfg.setflagmode('flag'); kfg.setantennas(ants); kfg.query(query='FIELD_ID >= 0'); kfg.done(); print 'Flagged MS file ',msf; } Should probably include more fail checking using is_fail. Put this in 2001-03-20-myers.g Still some low values. Lets go and quack it using GUI. Build function also for 2001-03-20-myers.g : quackall := function(msf) { # # Quack first 10s from each scan # # Assume ms has been created - check with default os tool if( dos.fileexists(file=msf) ) { print 'Found MS file ',msf; } else { fail "MS file does not exist"; } msfilename := msf; kfg := flagger(msfile=msf); kfg.setflagmode('flag'); kfg.quack(); kfg.done(); print 'Quacked MS file ',msf; } Look at it in msplot. Still some dropouts, but a little cleaner. Will need to go through and flag in X-Y mode, but need to be able to step through antennas and have the scope (eg. all Poln, SpWid) be easily toggled as in the display editor. >>>REQUEST: The DONE button should be on Tool window, not in Menu only. I think this was discussed in the msplot demo session. 4. Lets go ahead and try the automapper again. - automap(msf='20010204AK.ms',prefix='test2cm.20mar01.'); Results: Flux density for 0713+438 (spw=1) is: 0.528 +/- 0.001 Jy Flux density for 0713+438 (spw=2) is: 0.530 +/- 0.001 Jy Flux density for 0854+201 (spw=1) is: 2.744 +/- 0.003 Jy Flux density for 0854+201 (spw=2) is: 2.747 +/- 0.003 Jy Flux density for 0927+390 (spw=1) is: 8.176 +/- 0.012 Jy Flux density for 0927+390 (spw=2) is: 8.200 +/- 0.012 Jy Flux density for 1159+292 (spw=1) is: 1.358 +/- 0.001 Jy Flux density for 1159+292 (spw=2) is: 1.359 +/- 0.001 Jy Flux density for 1229+020 (spw=1) is: 14.668 +/- 0.019 Jy Flux density for 1229+020 (spw=2) is: 14.655 +/- 0.019 Jy Flux density for 1256-057 (spw=1) is: 22.657 +/- 0.030 Jy Flux density for 1256-057 (spw=2) is: 22.601 +/- 0.030 Jy Flux density for 1310+323 (spw=1) is: 1.208 +/- 0.001 Jy Flux density for 1310+323 (spw=2) is: 1.209 +/- 0.001 Jy 0713+438 IF1 I= 0.527 P= 0.010 chi= 55.40 0713+438 IF2 I= 0.524 P= 0.011 chi= 2.55 0854+201 IF1 I= 2.737 P= 0.117 chi= -90.52 0854+201 IF2 I= 2.737 P= 0.117 chi= -91.85 0927+390 IF1 I= 8.131 P= 0.127 chi= 132.66 0927+390 IF2 I= 8.149 P= 0.130 chi= 132.96 1159+292 IF1 I= 1.357 P= 0.019 chi= -69.48 1159+292 IF2 I= 1.359 P= 0.019 chi= -88.12 1229+020 IF1 I= 14.636 P= 0.991 chi= -88.93 1229+020 IF2 I= 14.616 P= 0.977 chi= -85.01 1256-057 IF1 I= 22.563 P= 2.474 chi= 101.97 1256-057 IF2 I= 22.511 P= 2.470 chi= 101.77 1310+323 IF1 I= 1.206 P= 0.029 chi= -48.24 1310+323 IF2 I= 1.205 P= 0.029 chi= -53.45 1331+305 IF1 I= 2.504 P= 0.319 chi= 66.00 1331+305 IF2 I= 2.510 P= 0.319 chi= 66.00 Versus AIPS results: I Peak mJy P Peak mJy Pol % RLPD deg -------------- -------------- ----------- -------------- 0713+438 511.29 ± 0.90 1.41 ± 0.23 0.28 ± 0.04 31.76 ± 7.78 0854+201 2668.92 ± 0.65 115.23 ± 1.90 4.32 ± 0.07 -88.22 ± 0.92 0927+390 8181.94 ± 2.52 142.57 ± 7.00 1.74 ± 0.08 124.92 ± 1.45 1159+292 1377.02 ± 1.44 15.70 ± 0.64 1.14 ± 0.05 -100.76 ± 1.93 1229+020 14218.67 ± 12.01 858.19 ± 12.05 6.04 ± 0.08 -86.88 ± 1.81 1256-057 21928.05 ± 34.34 2544.29 ± 1.50 11.60 ± 0.01 97.89 ± 0.21 1310+323 1186.02 ± 0.02 23.72 ± 0.42 2.00 ± 0.04 -46.76 ± 0.94 1331+305 2408.60 ± 0.45 324.17 ± 1.42 13.46 ± 0.06 66.00 ± 0.00 Much better agreement than before (see 2001-03-10-myers.txt), especially in the flux denisty of 0927+390 which was coming out 15Jy before. There are still some worrying differences in angle. Most are a couple of degrees, though 1159+292 is 20-30 degrees off and the two IFs do not agree in the aips++ results. The 2 IFs should agree better in more cases (eg. 1310+323). 5. Try msplot again. Edit each SpWid, all antennas on same plot RR LL. Remove low points. Done. >>>REQUEST: I note that most selections in msplot default to what was selected in previous sessions, except the Xaxis,Yaxis go back to uvdist,data respectively. Might consider having these retain last settings. Maybe the buttons should show the current selected axes also. What the heck are all these files: 20010204AK.ms.flags.1 20010204AK.ms.flags.2 20010204AK.ms.flags.3 20010204AK.ms.msplot.1 20010204AK.ms.msplot.2 20010204AK.ms.msplot.3 20010204AK.ms.msplot.4 20010204AK.ms.msplot.5 20010204AK.ms.msplot.6 20010204AK.ms.msplot.7 20010204AK.ms.msplot.8 20010204AK.ms.msplot.all There is no explanation in the User Reference to these. Are you supposed to use them for anything? If they are scratch files, why doesn't aips++ clean up after itself? I see in the log stuff like Saving current flag columns to flag column table /export/home/sandrock/smyers/Testing/Mar01/20Mar01/20010204AK.ms.flags.3 successful creation of permanent-locked table /export/home/sandrock/smyers/Testing/Mar01/20Mar01/20010204AK.ms.flags.3: 2 columns, 160234 rows created table /export/home/sandrock/smyers/Testing/Mar01/20Mar01/20010204AK.ms.flags.3 with 160234 rows Saving flag column FLAG_ROW to flag column table Saving flag column FLAG to flag column table ... flag column table /export/home/sandrock/smyers/Testing/Mar01/20Mar01/20010204AK.ms.flags.3 now will contain original flag columns. I guess that means that this is now the "backup" for the new flags that were applied to the current ms. But what are the .msplot.# files? >>>REQUEST: There should be some explanation in the User Reference (and eventually the cookbook) about the files dropped down by msplot. 6. Try automap again and see if this changes anything. The data has been calibrated once, so what do I need to do to rerun this? Is there a "reset" procedure? I think that my script will run OK from here (it should reset the MODEL_DATA and CORRECTED_DATA columns as needed). >>>COMMENT: It occurs to me that we (actually aips++) should be maintaining the FAQ for stuff like this. The official FAQ from the aips++ page is terribly out of date. - automap(msf='20010204AK.ms',prefix='test2cm.20mar01.'); 0713+438 IF1 I= 0.527 P= 0.010 chi= 55.39 0713+438 IF2 I= 0.525 P= 0.011 chi= 2.51 0854+201 IF1 I= 2.737 P= 0.117 chi= -90.45 0854+201 IF2 I= 2.737 P= 0.117 chi= -91.86 0927+390 IF1 I= 8.131 P= 0.127 chi= 132.66 0927+390 IF2 I= 8.150 P= 0.129 chi= 132.98 1159+292 IF1 I= 1.357 P= 0.019 chi= -69.46 1159+292 IF2 I= 1.359 P= 0.019 chi= -88.00 1229+020 IF1 I= 14.636 P= 0.991 chi= -88.94 1229+020 IF2 I= 14.616 P= 0.976 chi= -84.99 1256-057 IF1 I= 22.563 P= 2.474 chi= 101.96 1256-057 IF2 I= 22.511 P= 2.470 chi= 101.78 1310+323 IF1 I= 1.206 P= 0.029 chi= -47.80 1310+323 IF2 I= 1.207 P= 0.029 chi= -53.30 1331+305 IF1 I= 2.504 P= 0.319 chi= 66.00 1331+305 IF2 I= 2.510 P= 0.319 chi= 66.00 : parse error File: glish eval, Line 1 Stack: eval() []() .() Where did this error come from??? Nothing in the log file. These results are essentially the same as before (not a big surprise). Do I need to include an extra round of self-cal/clean? Probably wouldn't hurt. 7. Look at solutions with calibrater.plotcal. Choice of tables to browse: test2cm.20mar01.dcal1 - Jones polarization solutions test2cm.20mar01.gcal1 - Jones gain solutions test2cm.20mar01.gcalout1 - after flux calibration Plotted up gcalout1 amps. Dismissed the Plot Window. Tried to select SpWid=1 only, and couldn't get the plots to show. Had to restart calibrater. Works fine as long as you don't dismiss the PGPlotter. >>>BUG: Should be able to remake PGPlotter window when dismissed. If not, then there should not be DISMISS and DONE buttons on the plotter! Submitted as defect AOCso02417. AMPS,PHASE,RLPHASE looks fine. Nothing obvious (though AN18 has amplitudes that are about 1/4 nominal, but seem to be stable). I generally delete AN18 from data. Lets try that here. 8. Try again - killant(msf='20010204AK.ms',ants=[18]); successful read/write open of default-locked table 20010204AK.ms: : shape: Table::lock cannot sync; another process changed the number of columns File: tableserver.g, Line 71 Stack: .() .() .() spaste() tablenote() ... columns, 160234 rows Table query FIELD_ID >= 0 failed : command: Table::lock cannot sync; another process changed the number of columns Table query FIELD_ID >= 0 failed : Table query FIELD_ID >= 0 failed : command: Table::lock cannot sync; another process changed the number of columns I see no reason for this to have failed. Try from the GUI. I get the same errors. Something is screwed up. Exit from aips++. Restart. - include '2001-03-20-myers.g'; - killant(msf='20010204AK.ms',ants=[18]); This worked now. - automap(msf='20010204AK.ms',prefix='test2cm.20mar01.'); 0713+438 IF1 I= 0.529 P= 0.010 chi= 55.54 0713+438 IF2 I= 0.526 P= 0.011 chi= -79.01 0854+201 IF1 I= 2.740 P= 0.117 chi= -90.69 0854+201 IF2 I= 2.738 P= 0.118 chi= -93.25 0927+390 IF1 I= 8.113 P= 0.125 chi= 132.72 0927+390 IF2 I= 8.130 P= 0.126 chi= 133.88 1159+292 IF1 I= 1.356 P= 0.020 chi= -88.49 1159+292 IF2 I= 1.358 P= 0.020 chi= -84.58 1229+020 IF1 I= 14.616 P= 1.004 chi= -89.62 1229+020 IF2 I= 14.628 P= 0.992 chi= -85.23 1256-057 IF1 I= 22.535 P= 2.429 chi= 102.05 1256-057 IF2 I= 22.535 P= 2.431 chi= 102.32 1310+323 IF1 I= 1.208 P= 0.030 chi= -47.87 1310+323 IF2 I= 1.209 P= 0.029 chi= -26.25 1331+305 IF1 I= 2.503 P= 0.318 chi= 66.00 1331+305 IF2 I= 2.509 P= 0.316 chi= 66.00 This made some significant changes. The two IFs on 1159+292 are in better agreement now, though 1310+323 still has problems. Next step, add another iteration in the selfcal stage. 9. Found serious problem in script - I was still using the cellsize from the C-Band data which is horribly inadequate! Turn this into another input variable. Note that this data was from BnA, and thus the beam is highly elongated, for 1159+292 we have Beam fit: 0.287769 by 0.135543 (arcsec) at pa 86.7516 (deg) though for 1331+305 we get Beam fit: 0.267496 by 0.231027 (arcsec) at pa 91.0232 (deg) for natural weighting. Use 0.04arcsec cells. - automap(msf='20010204AK.ms',prefix='test2cm.20mar01.',clcell='0.04arcsec'); 0713+438 IF1 I= 0.530 P= 0.005 chi= -119.11 0713+438 IF2 I= 0.531 P= 0.005 chi= 0.05 0854+201 IF1 I= 2.745 P= 0.123 chi= -90.63 0854+201 IF2 I= 2.746 P= 0.121 chi= -93.61 0927+390 IF1 I= 8.144 P= 0.122 chi= 131.99 0927+390 IF2 I= 8.163 P= 0.124 chi= 135.62 1159+292 IF1 I= 1.357 P= 0.020 chi= -83.62 1159+292 IF2 I= 1.358 P= 0.020 chi= -83.22 1229+020 IF1 I= 14.626 P= 0.993 chi= -90.40 1229+020 IF2 I= 14.641 P= 0.983 chi= -86.97 1256-057 IF1 I= 22.568 P= 2.455 chi= 102.35 1256-057 IF2 I= 22.557 P= 2.441 chi= 102.27 1310+323 IF1 I= 1.209 P= 0.030 chi= -45.44 1310+323 IF2 I= 1.211 P= 0.029 chi= -52.24 1331+305 IF1 I= 2.504 P= 0.320 chi= 66.00 1331+305 IF2 I= 2.508 P= 0.319 chi= 66.00 Versus AIPS results: I Peak mJy P Peak mJy Pol % RLPD deg -------------- -------------- ----------- -------------- 0713+438 511.29 ± 0.90 1.41 ± 0.23 0.28 ± 0.04 31.76 ± 7.78 0854+201 2668.92 ± 0.65 115.23 ± 1.90 4.32 ± 0.07 -88.22 ± 0.92 0927+390 8181.94 ± 2.52 142.57 ± 7.00 1.74 ± 0.08 124.92 ± 1.45 1159+292 1377.02 ± 1.44 15.70 ± 0.64 1.14 ± 0.05 -100.76 ± 1.93 1229+020 14218.67 ± 12.01 858.19 ± 12.05 6.04 ± 0.08 -86.88 ± 1.81 1256-057 21928.05 ± 34.34 2544.29 ± 1.50 11.60 ± 0.01 97.89 ± 0.21 1310+323 1186.02 ± 0.02 23.72 ± 0.42 2.00 ± 0.04 -46.76 ± 0.94 1331+305 2408.60 ± 0.45 324.17 ± 1.42 13.46 ± 0.06 66.00 ± 0.00 Better still. 1310+323 is still the problem source, with a 7deg diff between IF 1 and 2. In general, I am getting lower polarized fluxes despite slightly higher total intensities. ------------------------------------------------------------------------------ This file can be found as 2001-03-20-myers.txt. The scripts can be found as 2001-03-20-myers.g