Date - 2001.12.18 Tester - S.T. Myers (AOC) Platform - Linux (sandrock) Note: bugs are denoted by lines with prefix: >>>BUG: and queries are denoted by lines with prefix: >>>QUERY: while comments are prefixed by: >>>COMMENT: Some enhancement requests are given after >>>REQUEST: Notes added later for clarification are indicated by >>>NOTE: ------------------------------------------------------------------------------- GOAL: test script for gain curve and opacity correction at Q band 1. Date - 2001.12.18 Version - Weekly (1.7 build 061) Directory - /home/sandrock/smyers/Testing/2001-12-18/ Data - 20010204AQ.UVF Script - 2001-12-18-myers.g Note - use optical depth at Qband of 0.066 from tip curve that day First, run without any editing or gain curve as a baseline: - include '2001-12-18-myers.g' - makemsf(par) - automap(par) 1331+305 spwid= 1 [I=1.456, Q=0, U=0, V=0] Jy, (Perley-Taylor 99) 1331+305 spwid= 2 [I=1.455, Q=0, U=0, V=0] Jy, (Perley-Taylor 99) Flux density for 0713+438 (spw=1) is: 1.491 +/- 0.092 Jy Flux density for 0713+438 (spw=2) is: 1.642 +/- 0.101 Jy Flux density for 0854+201 (spw=1) is: 295.851 +/- 24.435 Jy Flux density for 0854+201 (spw=2) is: 364.576 +/- 29.755 Jy Flux density for 0927+390 (spw=1) is: 648.923 +/- 53.669 Jy Flux density for 0927+390 (spw=2) is: 763.242 +/- 63.953 Jy Flux density for 1159+292 (spw=1) is: 3.094 +/- 0.164 Jy Flux density for 1159+292 (spw=2) is: 3.655 +/- 0.213 Jy Flux density for 1229+020 (spw=1) is: 13.523 +/- 0.103 Jy Flux density for 1229+020 (spw=2) is: 13.464 +/- 0.102 Jy Flux density for 1256-057 (spw=1) is: 21.898 +/- 0.115 Jy Flux density for 1256-057 (spw=2) is: 21.763 +/- 0.106 Jy Flux density for 1310+323 (spw=1) is: 1.038 +/- 0.003 Jy Flux density for 1310+323 (spw=2) is: 1.024 +/- 0.002 Jy 0713+438 IF1 I= 1.478 P= 0.172 chi= -23.34 0713+438 IF2 I= 1.613 P= 0.140 chi= -117.40 0854+201 IF1 I= 292.259 P= 15.623 chi= -85.06 0854+201 IF2 I= 359.033 P= 20.589 chi= -53.69 0927+390 IF1 I= 641.828 P= 19.401 chi= 148.83 0927+390 IF2 I= 752.233 P= 25.789 chi= 158.05 1159+292 IF1 I= 3.069 P= 0.111 chi= 156.87 1159+292 IF2 I= 3.635 P= 0.105 chi= -113.75 1229+020 IF1 I= 13.429 P= 0.807 chi= -111.87 1229+020 IF2 I= 13.452 P= 0.743 chi= -90.95 1256-057 IF1 I= 21.913 P= 2.670 chi= 80.74 1256-057 IF2 I= 21.896 P= 3.042 chi= 99.67 1310+323 IF1 I= 1.020 P= 0.237 chi= -123.35 1310+323 IF2 I= 1.004 P= 0.386 chi= -111.05 1331+305 IF1 I= 1.414 P= 0.194 chi= 66.00 1331+305 IF2 I= 1.478 P= 0.226 chi= 66.00 Versus AIPS results: I Peak mJy P Peak mJy Pol % RLPD deg -------------- -------------- ----------- -------------- 0713+438 266.94 ± 0.99 0.43 ± 0.63 0.16 ± 0.24 -123.72 ± 37.49 0854+201 2641.82 ± 1.66 110.76 ± 2.44 4.19 ± 0.09 -67.15 ± 2.37 0927+390 5636.36 ± 4.86 91.46 ± 3.10 1.62 ± 0.05 134.59 ± 0.74 1159+292 1189.96 ± 2.27 27.79 ± 0.31 2.34 ± 0.02 -99.14 ± 0.04 1229+020 11706.89 ± 7.95 369.76 ± 7.44 3.16 ± 0.06 -100.59 ± 0.56 1256-057 19792.85 ± 3.55 1788.23 ± 7.06 9.03 ± 0.04 99.77 ± 0.54 1310+323 965.03 ± 2.07 10.25 ± 1.02 1.06 ± 0.10 -69.85 ± 3.14 1331+305 1401.21 ± 1.06 188.42 ± 2.65 13.45 ± 0.20 66.00 ± 0.00 There are some severe problems (eg. 0854+201, 0927+390, 1256-057). Get a load of the 300 and 600 Jy fluxes on 0854+201, 0927+390! Probably due to a couple bad antennas in fluxscale() as we found at Kband in 2001-12-17-myers.txt log. Fire up msplot. Indeed, 0854 and 0927 have been amplified in flux, and have some antennas missing that are there in other sources. 2. Delete results and start over. - makemsf(par) Try msplot to identify bogus data. Use "display-as-image" mode. AN4 RR - intermittent (delete all) AN13 low gain all but scans on 0854,0927 (probably the culprit, delete) AN22 RR IF2 intermittent (or missing) scan 0854+201 -> ok? Note that the lack of any rms or difference plots means only the most apparent errors can be dealt with in this mode. Can probably get by with using killant(par.msf,ants=[4,13]), though do it interactively this time around by setting all Ants,Times,Corrs,SpWins and using the bombsight to nuke AN4 and AN13. >>>BUG: Got the Zoom problem again after Flag, had to unzoom to get back. It seems to occur after you actually flag data at least with the bombsight. It comes back in some arbitrary high zoom when right after you FLAG. Keep an eye on this to see if I can repeat it later. - automap(par) 0713+438 IF1 I= 1.487 P= 0.068 chi= -4.88 0713+438 IF2 I= 1.630 P= 0.081 chi= -21.10 0854+201 IF1 I= 295.132 P= 13.001 chi= -65.75 0854+201 IF2 I= 363.845 P= 15.385 chi= -64.54 0927+390 IF1 I= 647.062 P= 11.397 chi= 145.08 0927+390 IF2 I= 761.151 P= 13.579 chi= 140.38 1159+292 IF1 I= 3.091 P= 0.077 chi= -98.48 1159+292 IF2 I= 3.655 P= 0.084 chi= -100.42 1229+020 IF1 I= 13.504 P= 0.447 chi= -99.81 1229+020 IF2 I= 13.444 P= 0.453 chi= -101.24 1256-057 IF1 I= 21.862 P= 2.003 chi= 99.42 1256-057 IF2 I= 21.738 P= 1.973 chi= 97.92 1310+323 IF1 I= 1.035 P= 0.016 chi= -83.05 1310+323 IF2 I= 1.023 P= 0.013 chi= -101.42 1331+305 IF1 I= 1.456 P= 0.200 chi= 66.00 1331+305 IF2 I= 1.457 P= 0.197 chi= 66.00 Still bad. Use calibrater.plotcal to look at gcal1 table. Looks like AN13 has some problems, but I thought I deleted it! >>>REQUEST: plotcal needs to have some way to identify points (antennas, sources) other than just antennas by color and by scan number (like the info in the msplot display window). (Submitted as enhancement AOCso03277) Note - I listed this as "non-serious" even though this really does restrict the utility of plotcal. Look in msplot again. >>>BUG: Indeed, it came back in zoomed mode. Its happened enough to bug(). (Submitted as AOCso03278) >>>BUG: Ive come across a minor bug in the msplot display in that sometimes when I move the datamax slider and stop at some value, it starts repeating the value in the window to the side e.g. 846846846 and doesn't display. You have to jog the slider a little to the side to stop this. (Submitted also as part of AOCso03278) It does think that AN13 and AN4 are deleted, and I dont see many really low antennas that are missing from 0854 and 0927. Note that the usual Qband culprits show up: AN18 low 25-50% AN1 slightly low AN21 slightly low View the 20010204.qband.gcal1 table in the catalog tool. See if AN4 and AN13 still have solutions. >>>BUG: The table display is really crappy (slow) when scrolling rows. This makes it very frustrating to use. It should be able to do this better, but I guess thats the price of doing this in glish. AN4,13 solutions are flagged (weight 0, flag F F) but do have gain values. Is fluxscale possibly using these anyway? Go back to plotcal - now AN4 and 13 are definitely not there. Lets try doing this over again. - makemsf(par) Then edit in msplot GUI again. Still having zoom problem. -automap(par) 0713+438 IF1 I= 0.283 P= 0.013 chi= -4.88 0713+438 IF2 I= 0.287 P= 0.014 chi= -21.10 0854+201 IF1 I= 2.775 P= 0.121 chi= -64.76 0854+201 IF2 I= 2.773 P= 0.117 chi= -64.53 0927+390 IF1 I= 5.398 P= 0.095 chi= 145.09 0927+390 IF2 I= 5.392 P= 0.096 chi= 140.35 1159+292 IF1 I= 1.126 P= 0.028 chi= -98.44 1159+292 IF2 I= 1.123 P= 0.026 chi= -100.41 1229+020 IF1 I= 12.392 P= 0.410 chi= -99.72 1229+020 IF2 I= 12.379 P= 0.417 chi= -101.29 1256-057 IF1 I= 20.874 P= 1.912 chi= 99.42 1256-057 IF2 I= 20.840 P= 1.892 chi= 97.94 1310+323 IF1 I= 1.015 P= 0.016 chi= -83.14 1310+323 IF2 I= 1.013 P= 0.013 chi= -101.41 1331+305 IF1 I= 1.456 P= 0.200 chi= 66.00 1331+305 IF2 I= 1.457 P= 0.197 chi= 66.00 Well, now its fine. The angle on 1310+323 is still off. This is very strange. Its like the flags didnt actually take first time around. Ive had this happen before, I think. Repeat a third time to be sure - worked OK. 3. Opacity correction re-correction opac := function( tau0, el ) { # # Do atmospheric absorption correction on vector of el # tau(el) = tau0/sin(el) -> oc = 1/[ 1 - tau(el) ] # maximum tau = 1e3 tau := tau0/sin( el ); # Cap on high optical depths tau[ tau > 0.999 ] := 0.999; oc := 1.0/( 1.0 - tau ); return oc; } I was right before the change I made yesterday 2001-12-17. Grrr. Now redo with gain correction. - makemsf(par) - vlagaincor(par) - killant(par.msf,ants=[4,13]) - automap(par) 0713+438 IF1 I= 0.283 P= 0.013 chi= -4.88 0713+438 IF2 I= 0.287 P= 0.014 chi= -21.14 0854+201 IF1 I= 2.773 P= 0.122 chi= -65.80 0854+201 IF2 I= 2.773 P= 0.117 chi= -64.54 0927+390 IF1 I= 5.398 P= 0.095 chi= 145.14 0927+390 IF2 I= 5.392 P= 0.096 chi= 140.63 1159+292 IF1 I= 1.125 P= 0.028 chi= -98.52 1159+292 IF2 I= 1.123 P= 0.026 chi= -100.42 1229+020 IF1 I= 12.392 P= 0.410 chi= -99.71 1229+020 IF2 I= 12.380 P= 0.417 chi= -101.31 1256-057 IF1 I= 20.873 P= 1.912 chi= 99.37 1256-057 IF2 I= 20.841 P= 1.891 chi= 97.87 1310+323 IF1 I= 1.015 P= 0.016 chi= -83.13 1310+323 IF2 I= 1.013 P= 0.013 chi= -101.25 1331+305 IF1 I= 1.456 P= 0.200 chi= 66.00 1331+305 IF2 I= 1.457 P= 0.197 chi= 66.00 This didnt make much difference, my guess is that something is wrong in the gain correction (e.g. its not being applied to all the visibilities. 4. Do a check of before/after - makemsf(par) Now extract the first timeslot msf := par.msf; mst := table(msf, readonly=F); ti := tableiterator(mst, "TIME"); ti.next(); st := ti.table(); nvis := st.nrows(); print 'Nvis = ',nvis; vis1 := st.getcol("DATA"); ti.done(); mst.close(); Note, when I run this I get - msf := par.msf; - mst := table(msf, readonly=F); - ti := tableiterator(mst, "TIME"); : make_iterator: Table::lock cannot sync; another process changed the number of columns File: tableserver.g, Line 71 Stack: .(), tableserver.g line 1351 .(), table.g line 1052 .(), table.g line 1093 is_fail(), table.g line 1093 tableiterator() - ti.next(); warning, operand to .next is not a record error, F is not a function value This thing has happened before, and I dont know whats going on. Usually when I wait for a while after making the ms file. Works OK after mst.done(). Note, looks like I need an st.done() also: msf := par.msf; mst := table(msf, readonly=F); ti := tableiterator(mst, "TIME"); ti.next(); st := ti.table(); nvis := st.nrows(); print 'Nvis = ',nvis; vis1 := st.getcol("DATA"); Everything works ok but - vis1 := st.getcol("DATA"); : get_column: Table::lock cannot sync; another process changed the number of columns File: tableserver.g, Line 71 Stack: .(), tableserver.g line 739 .(), table.g line 762 .() Something is screwed up, because if I repeat it it is OK. vis1 := st.getcol("DATA"); st.done(); ti.done(); mst.done(); - vlagaincor(par); Filled gain coefficients for VLA at Qband Found 29 antennas in table Data is from telescope VLA Found 8 fields Reference frame J2000 warning, operand to .next is not a record error, F is not a function value warning, operand to .done is not a record error, F is not a function value Completed gain and opacity correction F - include '2001-12-18-myers.g' T - vlagaincor(par); Filled gain coefficients for VLA at Qband Found 29 antennas in table Data is from telescope VLA Found 8 fields Reference frame J2000 Processing timestamp 2001/02/04/08:24:40.001 Mean correction opac 1.07744 , gain 1.03818 Corrected 1159+292 with 210 rows at EL = 66.67 Processing timestamp 2001/02/04/08:24:43.333 Mean correction opac 1.07744 , gain 1.03775 Corrected 1159+292 with 240 rows at EL = 66.68 ... Processing timestamp 2001/02/04/10:39:49.999 Mean correction opac 1.07133 , gain 1.10109 Corrected 1159+292 with 506 rows at EL = 82.45 Completed gain and opacity correction Worked this time - something is terribly wrong in the interface. >>>BUG: I dont understand this behavior, some table locking problem or syncing seems to be screwed up. Now get the "after"... msf := par.msf; mst := table(msf, readonly=F); ti := tableiterator(mst, "TIME"); ti.next(); st := ti.table(); nvis2 := st.nrows(); print 'Nvis = ',nvis2; vis2 := st.getcol("DATA"); st.done(); ti.done(); mst.done(); Again, strange behavior, with lots of ": get_column: Table::lock cannot sync;" errors but just repeating the command works. OK - now compare - vis1:: [shape=[4 1 210] ] - vis2:: [shape=[4 1 210] ] - rr1 := vis1[1,,] - rr2 := vis2[1,,] - rr12 := rr1/rr2 - rr12 [1+0i inf+infi 1+0i inf+infi 1+0i inf+infi 1+0i inf+infi 1+0i inf+infi ... inf+infi inf+infi inf+infi inf+infi inf+infi inf+infi inf+infi inf+infi] Lots of zeroes, need to filter those out. - rr1[1:10] [-0.10357666-0.075378418i 0+0i -0.0524597168-0.0603637695i 0+0i 0.0423278809+0.0657348633i 0+0i -0.0172576904+0.00387763977i 0+0i 0.127563477-0.040435791i 0+0i] - rr2[1:10] [-0.10357666-0.075378418i 0+0i -0.0524597168-0.0603637695i 0+0i 0.0423278809+0.0657348633i 0+0i -0.0172576904+0.00387763977i 0+0i 0.127563477-0.040435791i 0+0i] OK, they are identical. Modify the script to add: # # put back in ms # st.putcol("DATA",vis); st.done(); printf(' Corrected %s with %d rows at EL = %.2f\n', fields[ifld], nvis, dq.convert(el,'deg').value ); } and rerun vlagaincor(par). again had to repeat as first time bombed. Ran OK. Fumble through extraction again - msf := par.msf; - mst := table(msf, readonly=F); - ti := tableiterator(mst, "TIME"); - ti.next(); T - st := ti.table(); - nvis3 := st.nrows(); print 'Nvis = ',nvis3; Nvis = 210 - vis3 := st.getcol("DATA"); : get_column: Table::lock cannot sync; another process changed the number of columns File: tableserver.g, Line 71 Stack: .(), tableserver.g line 739 .(), table.g line 762 .() - vis3 := st.getcol("DATA"); - st.done(); T - ti.done(); T - mst.done(); T - rr3 := vis3[1,,] - rr3[1:10] [-0.10357666-0.075378418i 0+0i -0.0524597168-0.0603637695i 0+0i 0.0423278809+0.0657348633i 0+0i -0.0172576904+0.00387763977i 0+0i 0.127563477-0.040435791i 0+0i] - rr2[1:10] [-0.10357666-0.075378418i 0+0i -0.0524597168-0.0603637695i 0+0i 0.0423278809+0.0657348633i 0+0i -0.0172576904+0.00387763977i 0+0i 0.127563477-0.040435791i 0+0i] It still doesnt seem to do anything, even though if I manually perform the operations in the script, e.g. vcv := array(1,4,nvis); for ( i in 1:nvis ) { vcv[1:4,i] := array( vc[i], 4 ); } vis *:= vcv; It clearly does the multiply correctly. Something must not be working in the putcol() st.putcol("DATA",vis); st.done(); even after adding the st.done(). Aha, from the documentation on table, looks like I need a st.flush after putcol. Try this...though I would have thought the .done() would also flush. >>>BUG: The documentation says table.done() is a synonym for table.close() and that close() will flush, yet it did not seem to actually do this. I would need to do more testing to verify this but it seemed that this did not work in the case above... NOTE - I realize in the above tests I have been leaving tables open as I have not been doing ti.terminate() to prematurely end the iteration. Sheesh. Leave and reenter aips++ in case there is hanging tables and crap. It sure made alot of dish chatter when I quit! - include '2001-12-18-myers.g' - makemsf(par) Then, msf := par.msf; mst := table(msf, readonly=F); ti := tableiterator(mst, "TIME"); ti.next(); st := ti.table(); nvis1 := st.nrows(); print 'Nvis = ',nvis1; vis1 := st.getcol("DATA"); st.done(); ti.terminate(); ti.done(); mst.done(); Worked OK. - vlagaincor(par) Again, msf := par.msf; mst := table(msf, readonly=F); ti := tableiterator(mst, "TIME"); ti.next(); st := ti.table(); nvis2 := st.nrows(); print 'Nvis = ',nvis2; vis2 := st.getcol("DATA"); st.done(); ti.terminate(); ti.done(); mst.done(); - rr1 := vis1[1,,] - rr2 := vis2[1,,] - rr21 := rr2/rr1 - rr21 - rr21[1:10] [1.13384354+1.70148908e-08i inf+infi 1.15058517-2.54541082e-08i inf+infi 1.09909308+3.62296824e-08i inf+infi 1.1041584+3.30443828e-09i inf+infi 1.11294556-2.8314588e-08i inf+infi] Bueno! That does the trick. Test more tomorrow. 5. Date - 2001.12.19 Version - Weekly (1.7 build 061) Directory - /home/sandrock/smyers/Testing/2001-12-18/ Data - 20010204AQ.UVF Script - 2001-12-18-myers.g Continue... - killant(par.msf,ants=[4,13]) - automap(par) 0713+438 IF1 I= 0.283 P= 0.013 chi= -4.88 0713+438 IF2 I= 0.287 P= 0.014 chi= -21.13 0854+201 IF1 I= 2.765 P= 0.122 chi= -65.80 0854+201 IF2 I= 2.770 P= 0.117 chi= -64.54 0927+390 IF1 I= 5.520 P= 0.097 chi= 145.14 0927+390 IF2 I= 5.514 P= 0.099 chi= 140.63 1159+292 IF1 I= 1.171 P= 0.029 chi= -98.52 1159+292 IF2 I= 1.169 P= 0.027 chi= -100.42 1229+020 IF1 I= 12.387 P= 0.409 chi= -99.71 1229+020 IF2 I= 12.374 P= 0.417 chi= -101.31 1256-057 IF1 I= 21.128 P= 1.935 chi= 99.37 1256-057 IF2 I= 21.096 P= 1.914 chi= 97.87 1310+323 IF1 I= 1.018 P= 0.016 chi= -83.11 1310+323 IF2 I= 1.016 P= 0.013 chi= -101.25 1331+305 IF1 I= 1.456 P= 0.200 chi= 66.00 1331+305 IF2 I= 1.457 P= 0.197 chi= 66.00 Now there are some changes (0927+390 5.39->5.52, 1256-057 20.9->21.1, 1159+292 1.12->1.17) but they seem smaller than I might expect, though it is true that the only real effect should be relative to the ZA at which the flux calibrator 1331+305 was observed. And the source that passed closest to zenith (1159+292) is averaged over scans at a albeit small range of elevations. Note that the fluxes are in line with what one expects taking into account the flux scale change versus when I did the AIPS (95->99), but the uncorrected fluxes are a little closer than the corrected ones. Probably need more tests, wait to see what Claire and George turn up in their detailed check versus AIPS. NOTE - the scripts automap.g, vlagaincor.g in the scripts area, and 2001-12-17-myers.g in the testlogs area have been updated to reflect the corrections given above. The new versions supersede previous ones. There still are likely bugs in the scripts, so if you are using these look for updates in the scripts area (check the revision date on the file) before using. ------------------------------------------------------------------------------ Testing logs archived at http://www.aoc.nrao.edu/~smyers/aips++/testlogs/ This file can be found as 2001-12-18-myers.txt The script can be found as 2001-12-18-myers.g