Date - 2000.09.27 Tester - S.T. Myers (AOC) Platform - Linux (kernow) Version - Stable (1.4 build 295) Note: bugs are denoted by lines with prefix: >>>BUG: and queries are denoted by lines with prefix: >>>QUERY: while comments are prefixed by: >>>COMMENT: Some enhancement requests are given after >>>REQUEST: ------------------------------------------------------------------------------- GOAL: Try some GUI testing for 3C286 polarization calibration Dataset: /home/kernow/smyers/Testing/Feb2000/CALS00FEB.CEDT 1. Load data into ms m := fitstoms < msfile = cband.ms < fitsfile = CALS00FEB.CEDT > CREATE note: got sent back to tool manager even though was still in ms:m go back to ms:m window > summary GO > close > DONE Do first point source calibration on 1331+305 imgr := imager < filename = cband.ms CREATE note: again, got sent back to tool manager when should be imager >>>BUG: This should not happen. Note that Debra has bugged this already, so I will leave it be. Not sure when and why this changed. go back to imager:imgr had to set function group from basic to setup: should default to setup group > setdata < msselect = "FIELD_ID==2 && SPECTRAL_WINDOW_ID==1" < spwid = 1 < fieldid = 2 note: I know I dont need msselect here as it duplicates the spwid and fieldid >>>COMMENT: having spwid & fieldid + msselect is confusing, should be like setdata in calibrater (see below) GO > setjy < fieldid = 2 < spwid = 1 < fluxdensity = [7.486 0 0 0] GO DONE cl := calibrater < filename = cband.ms CREATE > setdata < msselect = "FIELD_ID==2 && SPECTRAL_WINDOW_ID==1" GO >>>COMMENT: It would help greatly if there was a graphical menu of select items that could be input here to construct the msselect string (more like a selection object) > setsolve < type = "T" < t = 10.0 < phaseonly = True < table = "cband.run1.ph" GO >>>BUG: Note I had actually gone back to do the setjy after starting the calibrater, and tried to get rid of cl and remake it, but it kept trying to make me rename it cl1 and cl2 even though I had closed and done the older versions. This is stupid. Eventually it crashed with - warning, event glishtk.fail () dropped LocalExec::SetStatus: abnormal child termination for /home/sneffels3/aips++/stable/linux_egcs/bin/glishtk pure virtual method called Restart aips++, and hope the setjy is still in the MODEL_DATA column etc. cl := calibrater < filename = cband.ms CREATE > setdata < msselect = "FIELD_ID==2 && SPECTRAL_WINDOW_ID==1" GO > setsolve < type = "T" < t = 10.0 < phaseonly = True < table = "cband.run1.ph" GO note: I see no indication that it has actually set these things. Shouldnt something go to the logfile like "setsolve complete"? > solve Output: Starting calibrater::solve Initializing solvable atmospheric gain/transmission (T-matrix) For interval of 10 seconds, found 28 slots Solving for T T Jones Slot=1, 1331+305, spw=1: 24-Feb-2000/09:58:00 to 24-Feb-2000/09:58:09 T Jones Initial fit per unit weight = 7.31975 Jy, sum of weights = 2400 T Jones Trouble converging: will not keep these solutions T Jones Final fit per unit weight = 19.893 Jy, after 11 iterations . . . T Jones Slot=28, 1331+305, spw=1: 24-Feb-2000/12:23:50 to 24-Feb-2000/12:23:59 T Jones Initial fit per unit weight = 7.3363 Jy, sum of weights = 3900 T Jones Trouble converging: will not keep these solutions T Jones Final fit per unit weight = 21.5602 Jy, after 11 iterations T Jones Did not converge for 28 time intervals Storing T matrix in table cband.run1.ph Finished calibrater::solve 26.31 real 18.99 user 1.15 system > close DONE note: the cl tool doesnt actually seem to disappear from the tools in use even if I refresh Hmmm, did I need to select polarizations [RR,LL]? Does this mean it didnt get any solutions? Browse table t := table < tablename = cband.run1.ph > browse No row in table - looks like calibrater failed. Try going back and editing in visplot > close note: it doesnt seem to want to let me close this! > done nada DONE note: t does seem to have disappeared from tools in use vp := visplot < msfile = cband.ms < set fields = 2, spectral window =1, polarizations = RR,LL < try X = time, Y = visibility Nothing seems to plot, get error - %PGPLOT, invalid y limits in PGSWIN: Y1 = Y2. Wants to plot corrected visibilities, use VISY to choose observe vis This works. There is clearly an antenna with low gain. Lets try to find this with the VISZ = observed amplitude, then Display Z: It is clearly AN 20. But I cant seem to make it edit this antenna out. Try editing it in the YvsX window (I know it wont get the crosspol data though). FLAG (several boxes worth) DISMISS note: visplot produced a bunch of errors to the glish window. visplotapplyflags < flagfile = cband.ms.flags.1 (presumably this is what visplot made) GO DISMISS This time in calibrater, specify polarizations. But I cant seem to find the ms.summary output in the bloody log window. It seems to have lost the stuff from when glish crashed. Go back, t := table < tablename = cband.ms CREATE > summary Output: Table summary: cband.ms Shape: 25 columns by 138390 rows Info: [type=Measurement Set, subType=, readme=This is a MeasurementSet Table holding measurements from a Telescope ] Table keywords: [ANTENNA=Table: cband.ms/ANTENNA, ARRAY=Table: cband.ms/ARRAY, FEED=Table: cband.ms/FEED, FIELD=Table: cband.ms/FIELD, OBSERVATION=Table: cband.ms/OBSERVATION, OBS_LOG=Table: cband.ms/OBS_LOG, SOURCE=Table: cband.ms/SOURCE, SPECTRAL_WINDOW=Table: cband.ms/SPECTRAL_WINDOW, SYSCAL=Table: cband.ms/SYSCAL, WEATHER=Table: cband.ms/WEATHER, SORTED_TABLE=Table: cband.ms/SORTED_TABLE, SORT_COLUMNS=ARRAY_ID FIELD_ID SPECTRAL_WINDOW_ID TIME] Columns: UVW ANTENNA1 ANTENNA2 ARRAY_ID CORRELATOR_ID EXPOSURE FEED1 FEED2 FIELD_ID FLAG_ROW INTERVAL OBSERVATION_ID PULSAR_BIN PULSAR_ID SCAN_NUMBER SPECTRAL_WINDOW_ID TIME WEIGHT DATA SIGMA WEIGHT_SPECTRUM FLAG MODEL_DATA CORRECTED_DATA IMAGING_WEIGHT UVW keywords: [UNIT=m, MEASURE_TYPE=UVW, MEASURE_REFERENCE=] ANTENNA1 keywords: [UNIT=, MEASURE_TYPE=] ANTENNA2 keywords: [UNIT=, MEASURE_TYPE=] ARRAY_ID keywords: [UNIT=, MEASURE_TYPE=] CORRELATOR_ID keywords: [UNIT=, MEASURE_TYPE=] EXPOSURE keywords: [UNIT=s, MEASURE_TYPE=] FEED1 keywords: [UNIT=, MEASURE_TYPE=] FEED2 keywords: [UNIT=, MEASURE_TYPE=] FIELD_ID keywords: [UNIT=, MEASURE_TYPE=] FLAG_ROW keywords: [UNIT=, MEASURE_TYPE=] INTERVAL keywords: [UNIT=s, MEASURE_TYPE=] OBSERVATION_ID keywords: [UNIT=, MEASURE_TYPE=] PULSAR_BIN keywords: [UNIT=, MEASURE_TYPE=] PULSAR_ID keywords: [UNIT=, MEASURE_TYPE=] SCAN_NUMBER keywords: [UNIT=, MEASURE_TYPE=] SPECTRAL_WINDOW_ID keywords: [UNIT=, MEASURE_TYPE=] TIME keywords: [UNIT=s, MEASURE_TYPE=EPOCH, MEASURE_REFERENCE=TAI] WEIGHT keywords: [UNIT=, MEASURE_TYPE=] DATA keywords: [UNIT=, MEASURE_TYPE=] SIGMA keywords: [UNIT=, MEASURE_TYPE=] WEIGHT_SPECTRUM keywords: [UNIT=, MEASURE_TYPE=] FLAG keywords: [UNIT=, MEASURE_TYPE=] MODEL_DATA keywords: [CHANNEL_SELECTION=[[1:2,] 0 0 1 1]] > browse note: it looks like SPECTRAL_WINDOW_ID is 0,1 so have I been selecting IF=2 instead of 1 all along? Are the SPECTRAL_WINDOW_ID numbers different than spwid? note: looks like the polarizations are just encoded in the four elements of the DATA block, eg. [1.242@45.048, 0.009@174.503, 0.019@43.925, 1.265@60.606]. How do we specify only RR,LL in calibrater? DISMISS done DONE >>>COMMENT: It would be really great to be able to copy and paste from the browser windows but that doesnt seem to be able to work. cl1 := calibrater < filename = cband.ms CREATE > setdata < msselect = "FIELD_ID==2 && SPECTRAL_WINDOW_ID==1" GO > setsolve < type = "T" < t = 10.0 < phaseonly = True < table = "cband.run2.ph" GO > solve note: same crap happens as before. Does selection of type="T" mean that RR and LL phases are solved for or does it include RL,LR? This is not clear from the URG either. I have a feeling this convergence crap is what we have been finding before. I will probably have to corner George to sort this out further. > close DONE note: again I cant get rid of old cl and now cl1. Can I do it from glish? Try cl.done() cl1.done() > cl.done() Successfully closed empty server: calibrater > cl1.done() Successfully closed empty server: calibrater#2 but they still appear on the Tools in Use! They just wont die! >>>BUG: This seems to be a problem. I will go ahead and bug() it... ---> Submitted as AOCso01835 exit note: it put up the stop window to ask if I wanted to apply the visplot flags - I already ran visplotapplyflags! What is up there? Did that not work? 2. After that dismal experience, lets see if the map tool is any more useable myms := fitstoms < fitsfile = CALS00FEB.CEDT < msfile = cband.ms CREATE go back to ms:myms tool > close GO DONE warning, event glishtk.fail () dropped LocalExec::SetStatus: abnormal child termination for /home/sneffels3/aips++/stable/linux_egcs/bin/glishtk Try again, restart aips++ Hey, all the windows are gone but Im stuck in the glish window. Cannot use cntl-Z or cntl-C. Try kill -9 the following processes: ps -elf | grep aips++ 000 S smyers 1519 1495 0 60 0 - 520 do_sel 12:49 pts/16 00:00:00 /home/sneffels3/aips++/stable/linux_egcs/bin/timer -id -pipes 18 17 -in 000 S smyers 1526 1495 0 60 0 - 519 do_sel 12:49 pts/16 00:00:00 /home/sneffels3/aips++/stable/linux_egcs/bin/shell_client -id -pipes 20 000 S smyers 1529 1495 0 60 0 - 519 do_sel 12:49 pts/16 00:00:00 /home/sneffels3/aips++/stable/linux_egcs/bin/shell_client -id -pipes 22 000 S smyers 1532 1495 0 60 0 - 519 do_sel 12:49 pts/16 00:00:00 /home/sneffels3/aips++/stable/linux_egcs/bin/shell_client -id -pipes 2 000 S smyers 1536 1495 0 60 0 - 519 do_sel 12:49 pts/16 00:00:00 /home/sneffels3/aips++/stable/linux_egcs/bin/shell_client -id -pipes 2 000 S smyers 1537 1495 0 60 0 - 520 do_sel 12:49 pts/16 00:00:00 /home/sneffels3/aips++/stable/linux_egcs/bin/timer -id -pipes 13 12 -i 000 S smyers 1541 1495 0 60 0 - 1741 do_sel 12:49 pts/16 00:00:00 /home/sneffels3/aips++/stable/linux_egcs/bin/gtable -id -pipes 29 28 - 000 S smyers 1563 1495 12 60 0 - 4136 do_sel 12:53 pts/16 00:00:29 /home/sneffels3/aips++/stable/linux_egcs/bin/ms -id -pipes 7 5 -interp 000 S smyers 1564 1495 0 60 0 - 520 do_sel 12:53 pts/16 00:00:00 /home/sneffels3/aips++/stable/linux_egcs/bin/timer -id -pipes 10 7 -in Now cntl-Z works. kill -9 that job now. Now restart aips++ myms := fitstoms < fitsfile = CALS00FEB.CEDT < msfile = cband.ms CREATE go back to ms:myms tool > close GO DONE note: no crash this time, but again myms will not disappear from tools in use, and DONE button doesnt seem to have worked. Do myms.done() from glish. Now myms is gone from list. >>>BUG: Seems to me that the DONE buttons should do the equivalent of the done method. If not, then done should appear under the functions of that tool. mymap := map < msfile = cband.ms CREATE get back the map:mymap window > summary GO > solvecal > sourcemodels > spanner-CREATE note: at this point it put up a window that was registered with the input box as Athol showed last week, but it turns out it is always at the front and cannot be moved! Its covering other windows I need, and even worse the bottom of it is off the bottom of my screen. I have to move the parent window - hey, that doesnt actually move it either! Its stuck there. I cant get to the DISMISS button. This sucks. Have to use CREATE&SEND even though I havent created anything. Move the Tool Manager window up to top and spanner-CREATE again. Now I see the bottom, but I have a hard time placing this emacs window so I can type in this report. And it is stuck in front when I change virtual desktops. And I cannot change focus back to type text into the source1 box! DISMISS >>> BUG: This clearly doesnt work. Instead of bug()ing it I will talk to Athol. > solvecal > sourcemodels > spanner-CREATE < source1 = ??? note: Does it want source names or field-ids? I cant tell, because no matter what I do I cannot enter anything in there. Wait, it seems I have to select the parent window to bring focus back to the sub-window. > solvecal > sourcemodels > spanner-CREATE < source1 = "1331+305" < model1 > spanner-CREATE < catalog SEND&DISMISS get a bunch of errors WheneverStmt ref count error, its.wrenchmenu[poption] is not an agent has this actually filled in anything? All I see is there. and now sourcemodels is filled with . Is there anyway to view these?? when I go back to the spanner-CREATE is has a black X box next to it, so I assume that the catalog didnt take go back and insert flux manually > solvecal > sourcemodels > spanner-CREATE < source1 = "1331+305" < model1 > spanner-CREATE < fluxdensity iquv = [7.486 0 0 0] SEND&DISMISS CREATE&SEND > selection > spanner-CREATE < fieldnames = "1331+305" < spwids = [1] CREATE&SEND > solvers > spanner-CREATE < type1 = "T" < solver1 > spanner-CREATE < t = 10.0 < phaseonly = True < table = cband.run1.ph CREAT&SEND CREATE&SEND GO this then went through all sources and spwids even though I thought that I selected only 1331+305, spwid=1 at every possible place. Try again > selection > spanner-CREATE < fieldnames = "1331+305" < spwids = [1] < msselect = "FIELD_ID==2 && SPECTRAL_WINDOW_ID==1" CREATE&SEND GO No, this is doing the same thing. Looks like the solver settings didnt take, > solvers > spanner-CREATE < type1 = "T" < solver1 > spanner-CREATE < t = 10.0 < phaseonly = True < table = cband.run1.ph CREAT&SEND note: this made a blank glish/tk window - someone had better get this glish/tk crap fixed and pronto CREATE&SEND had to go back and reset the model also. It seems when I go back to the spanner-CREATE from sourcemodels it has unset the model, even though if I do spanner-CREATE there and then reslect fluxdensity instead of catalog it has remembered the fluxdensity. GO frack! still screwed up. It is clearly having problems in the setjy stage not doing the selection correctly. >>>COMMENT: The selection operations in map and in aips++ should be better, perhaps though the ability to use a global selection that supersedes all the individual selections Note that when it gets to the solve stage it has the same convergence problems that I had above using calibrater directly. I see no progress possible here - strike 2 today. Hey, when I came back to the aips++ desktop after doing something else, it had crashed with another glish/tk bomb - warning, event glishtk.fail () dropped LocalExec::SetStatus: abnormal child termination for /home/sneffels3/aips++/stable/linux_egcs/bin/glishtk Again I had to cntl-Z and kill -9 from the hung glish This is getting ridiculous. 3. Is there any hope trying imagerwizard? imagerwizard > GO 1. FITS file CALS00FEB.CEDT NEXT 2. Spectral window 1 NEXT 3. Field 2 1331+305 NEXT 4. Cellsize 0.25arcsec The parameters used are: Antenna diameter = 25meters Wavelength = 0.0614112297 meters Maximum baseline = 6944.46388 meters Cellsize = 0.608013117 arcsec Pixels = 576 How come it didnt use my request of 0.25 arcsec? Is it overriding to make a low-res version first? 5. There seems to be something in the map at the center, with max value 0.179 with rms in outer parts of 0.009 note: clean does not report a total cleaned flux, which would be useful info. That should be put into its log output along with the reports on residuals. 6. Unfortunately, the last region I selected was the off-source one to get the rms. Alas. Seems to have more or less worked. Would have been nice to incorporate setjy and selfcal - perhaps in calibraterwizard? 4. On Athol's suggestion, play more with the subwindow problem in map. mymap := map < msfile = cband.ms CREATE get back the map:mymap window GO > solvecal > sourcemodels > spanner-CREATE Here as usual I get the child window registered from the parent, which goes off the bottom of the screen. It is supposed to follow the parent window but does not. The best I can do is hit the Create&Send. Note that however I dismiss this subwindow I get glish errors error, its.wrenchmenu[poption] is not an agent WheneverStmt ref count a whole bunch of them. Note that I am using click-to-focus in my window manager. In general to focus into the child window I must click on the parent (eg. to enter text into a entry) if I have clicked anywhere else in the screen. 8. Back to the imager/calibrater testing. I should be able to make it work somehow. imgr := imager < filename = cband.ms CREATE > setdata < spwid = 1 fieldid = 2 msselect = "FIELD_ID==2" GO By selection 138390 rows are reduced to 48558 > setjy < fieldid = 2 spwid = 1 fluxdensity = [7.486 0 0 0] By selection 138390 rows are reduced to 24279 should I have selected SPECTRAL_WINDOW_ID==1 in msselect? I wanted to avoid the possible difference between spwid=1 and that. t := table('cband.ms') t.browse() Lets see what setjy did to the ms. It is clear that it put the fluxdensity in the MODEL_DATA column for entries with FIELD_ID = 1 and SPECTRAL_WINDOW_ID = 0 so aips++ must translate into numbers starting with 1 instead of 0 thats in the ms itself. This is kind of confusing, but probably means I can assume it will work OK. I do not see any close() or done() functions for imager! The DONE button doesnt do cleanup, so do from glish imgr.done() cl := calibrater < filename = cband.ms CREATE > setdata < msselect = "FIELD_ID==2 && SPECTRAL_WINDOW_ID==1" GO By selection 138390 rows are reduced to 24279 > setsolve < type = "T" t = 10.0 preavg = 0.0 phaseonly = True table = cband.run1.ph appen = True GO > setsolve < type = "G" t = 60.0 preavg = 0.0 phaseonly = False table = cband.run1.gn appen = True GO > solve GO Caught an exception! Event type=run exception=(/aips++/stable/code/trial/implement/MeasurementComponents/DOcalibrater.cc : 365) Failed AlwaysAssertExit retval Caught exception: (/aips++/stable/code/trial/implement/MeasurementEquations/VisEquation.cc : 321) Failed AlwaysAssertExit correctedvb.nRow()==corruptedvb.nRow() Ok its clearly still screwed up. Lets try deleting AN 20. fl := flagger < msfile = cband.ms CREATE > setantennas < ants = [20] > setpol < pol = [1,2,3,4] > setids < fieldid = [2] < spectralwindowid = [1] > timerange < starttime = '2000/02/24/09:43:23' < endtime = '2000/02/24/12:56:27' NOTE: THE TIMERANGE FOR FLAGGING SHOULD DEFAULT TO THE WHOLE DATASET! < trial = False GO Table query ((TIME/(24*3600)>=MJD(2000/02/24/09:43:23))&&(TIME/(24*3600)<=MJD(2000/02/24/12:56:27))) failed : command: Error in select expression: Keyword SPECTRALWINDOW_ID does not exist Scanned so far: select from $1 where ((TIME/(24*3600)>=MJD(2000/02/24/09:43:23))&&(TIME/(24*3600)<=MJD(2000/02/24/12:56:27)))&&((ANTENNA1==19)||(ANTENNA2==19))&&((SPECTRALWINDOW_ID== Table query ((TIME/(24*3600)>=MJD(2000/02/24/09:43:23))&&(TIME/(24*3600)<=MJD(2000/02/24/12:56:27))) failed : Table query ((TIME/(24*3600)>=MJD(2000/02/24/09:43:23))&&(TIME/(24*3600)<=MJD(2000/02/24/12:56:27))) failed : command: Error in select expression: Keyword SPECTRALWINDOW_ID does not exist Scanned so far: select from $1 where ((TIME/(24*3600)>=MJD(2000/02/24/09:43:23))&&(TIME/(24*3600)<=MJD(2000/02/24/12:56:27)))&&((ANTENNA1==19)||(ANTENNA2==19))&&((SPECTRALWINDOW_ID== It is clear here that internally flagger is using the wrong name for SPECTRALWINDOW_ID as it should be SPECTRAL_WINDOW_ID. >>>BUG: SPECTRALWINDOW_ID should be SPECTRAL_WINDOW_ID in flagger ---> Submitted as defect AOCso01837 Hey, there is no close or done function, do it manually fl.done() This did not seem to work. DONE does not work either. >>>COMMENT: there should uniformly be close(), and done() functions in the GUI, or the DONE button should actually invoke done() Try it in visplot again vp := visplot < cband.ms Use the Display Z ( visibility amplitude ) as the only way I know to knock out all polns I try to select a box on the AN 20 part with the rectange drawer but I cant make it actually select that region (the guide says to double click - that doenst seem to work right, I can double click all I want and nothing happens like the double-clicking time is set wrong. This is verified by LIST which says that there are no records to list. Got George to look at it - he agrees the rectangle selection is just not working so bug() it. Im not sure the crosshair works right either but at least it will generate flag records. >>>BUG: Rectangle selection in visplot display does not work ---> submitted as defect AOCso01839 This is just getting plain silly - nothing I try actually works in the way advertised in the URG. Grrr. As there seems to be no way to blast AN 20 all polns, give up for now. >>>BUG: There seems to be no way to close() the visplot tool from the GUI, and there is not even a DONE button (though there is a Done item under the File menu). This actually didnt work and left the window hanging! I had to cntl-Z out of glish and kill -9 to get out of this. 7. Synopsis: I am clearly driving calibrater incorrectly. I will try to hook up with George to see what I doing wrong. I find the selection process obtuse and unnecssarily Byzantine and this would be a good area to improve in the near term. I also see little gain in using the map function. It seems to offer little over using the individual imager and calibrater bits at this point, and its not clear to me that this is a good direction to be heading. IMPO the "wizards" are much more promising. Of course, the priority must be to make sure the core calibrater and imager routines are robust and working, and it is not clear that they are. The seeming inability to be able to close certain tools, and the fact that only some tool actually had close and done functions in the GUI is puzzling. The shutdown of tools should be made more uniform and robust. Frustration Meter: I wanna go home...