Date - 2002.06.06 Tester - J.E. Hibbard (CV) Platform: Merger Version: Weekly AIPS++ Version 1.7 Build 402 Running through "Getting Results, Vol 3: Telescope Specific Processing: 1. VLA reduction" (http://aips2.nrao.edu/stable/docs/gettingresults/grvol3/node2.html) This is in preparation for the summer school. Therefore, I will primarily use the GUI. Reading... S1.1, S1.2, S1.3, S1.3.1: Given the detail of the discussion here, I think it would be worth repeating eqn 1.7 from http://aips2.nrao.edu/stable/docs/gettingresults/grvol2/node2.html here, rather than having the student wade through grvol2 to get the basic gist of the ME. S1.3.2: This is detailed information that would do better to appear in S1.8.2, where the user actually invokes these functions, rather than here where they have nothing to really attach to. 1st paragraph: "..allows the user to set which data will be treated in any subsequent executions of the operational functions." => "..allows the user to set which data will be treated in subsequent executions of the operational functions, and remains in effect until another setdata or the reset function is called." (or something like that) "Both setapply and setsolve can be executed more than once..." => "Both setapply and setsolve can be executed more than once, with each execution adding to those that have been run previously" (or something like that). Does "reset" clear the setapply, setsolve state? If so, mention this. 2nd paragraph: "The solve function will apply (on-the-fly, and..." What does "on-the-fly" mean? I usually take this to mean "as the data is coming in", but I don't think that is what is meant. 3rd paragraph: I find the first sentence confusing, and think it could be removed without loss of content. If desired, you could add a parathetical sentence at the end which says "(Since solve only acts on the DATA column of the MS, running correct has no effect on subsequent solve executions). 4th paragraph: I simply can't wrap my brain around this, especially the first sentence. I hope no students ask me to interpret this! I end this section as I began - with the suggestion that this discussion be moved into the calibration section right before setapply and setsolve are run. Perhaps even pointing to the specific examples being demonstrated. S1.3.3, S1.4, S1.5, skipping S1.5.1, S1.5.2: Should mention that this is gotten to from the gui via general.ms.ms. general.ms.ms. fitstoms msfile: ngc5921.ms fitsfile: /aips++/data/demo/NGC5921.fits Should mention that single quotes are necessary for command line operations, but should be left out of gui entry. Should mention that parameters not specified in command line description by visible in the gui should be left in their default state. summary verbose: T Should say something useful about the summary, a la Steves write-up. S1.6: as Steve already noted, msplot is obtained via general.ms.msplot. S1.7: ok, why did we go into msplot in S1.6, just to leave it for S1.7.0.1, S1.7.0.2? The discussion of msplot in S1.6 should go into S1.7.1. S1.7.0.1: These examples should include an example of flagging a baseline. Last I checked, this required using 0-based counting, and NOT using the numbers given in the summary. All the more reason for including an example. From one of my working scripts: ############################################################ ## For these data, we need to flag baseline between ant 8 & 23. ## You would niavely think that the following would work: ## fg.query('ANTENNA1==8 and ANTENNA2==23'); ## But it doesn't; that would flag ant 9 w/24. ## NOTE!!!! THIS MAY CHANGE IN FURTHER RELEASES OF AIPS++!!! ## To get the desired result, do it as follows fg.query('ANTENNA1==7 and ANTENNA2==22'); ############################################################ It is NOT true that the full range of command-based flagging operations are described *fully* in the flagger. "Briefly" would be more appropriate. OK, since I am being told about flagger, I will go ahead and try flaggin autocorrelations in the spectral line data. synthesis.flagger.flagger. flagger toolname: myflagger msfile: ngc5921.ms auto trial: F LOGGER> Flagging 1619 rows OK, we are being told about the autoflag tool, so I'll go ahead and try that too. wait; I didnt know where autoflag was, so I recreated a flagger tool to check its function list - not there. Now, how do I done that tool? I know I done'd it before, but there is no done button now. So I try running auto() again, and it tells me that it is flagging 1619 rows again - this reminds me that this behavior might be confusing to first time users - the flag operations do not take into consideration previous flagging operations. So this should be mentioned. Anyways, I guess I have to done the flagger from the Tools in Use. AHA - when I went to tools in use, I realized the gui was not pulled out to its full size. So I went back to myflagger and made the gui bigger, and there aer the buttons on the bottom. Lets see if I can reproduce this: nope, came up the right size. I must have resized without realizing it. S1.7.0.2: Anyways, I now see autoflag: synthesis.autoflag.autoflag. toolname: myautoflag msfile: ngc5921.ms Hmm, more functions that I expected, so press . This sounds pretty interesting, especially the frequency medianing function. Wouldnt it be cool to flag on CORR-MODEL? Submitted as an enhancment. But I should probably look at the pre-flagged data in msplot first, so lets do that and come back to this. I'll come back to it, so lets just Back to the tutorial: S1.7.1: Man, is THAT skimpy! We should at least include a link to the Jan 2002 Newsletter article about msplot. general.ms.msplot. toolname: msplot msfile: ngc5921.ms edit=F This has been more fully commented on by others. I would just add comments that under spectral selection, look at a range of channels in the middle of the band, eg msplot. num chan: 1 start chan: 10 chan increment: 1 width of chan:40 and to have the "Flag all: channels" select box checked for plotting X vs Y, and the "Extend selection over all: channels" in the raster display. >>>BUG: documentation on spectral selection is out of date. They use "step" instead of "channel increment". Submitted. >>>BUG: I do not get the option to flag all channels when I do the above. Apparently this option is not available yet. We were told that it would be available. What up with that?? Submitted. (this is for x-y plots. It is apparently enabled in the raster display). >>>BUG: I zoomed into an msplot, but can't seem to zoom out again. There is no unzoom button, and cntr-middle button doesnt work. Submitted. OK, enough of this. Nothing obvious to flag. I doubt autoflag is worth it, but what the hell. I'll run it with the defaults. Settimemed examines points for each correlator, at each time, for each channel, and marks it for flagging based on a local time median (ie, time median for that correlator for that channel centered at that time) synthesis.autoflag.autoflag. toolname: myautoflag msfile: ngc5921.ms settimemed thr: 5.0 hw: 10 rowthr: 5 rowhw: 10 col: DATA expr ABS I Note: this just sets up the scope of the flags; it doesnt actually flag anything BUG>>> according to the documentation, hw is the half-width of the median filter in time, but its an integer. So what are its units? The number of timescans? Setfreqmed examines points for each correlator, at each time, for each channel, and marks it for flagging based on a local channel median (ie, median for channels around a given channel, for that correlator at each time) setfreqmed thr: 5.0 hw: 10 rowthr: 10.0 col: DATA expr ABS I Hmm. I wonder what that will do for narrow spectral lines, and at regins near the bandedge? setsprej flags points after fitting a spectral baseline to each uv datapoint. This is like the flagging that UVLIN can do. Since I did setfreqmed, I'm not going to do this. setuvbin flags outliers in the uvdist vs plane. setuvbin thr: 0.001 nbins 50 col: DATA expr ABS I OK, apply all of the critieria and flag: run trial: T Hmm - timemed generated errors b/c of observations with too few timeslots. I'll runs settimemed with hw=6 and thr=6 Ah - still got the same errors. Running settimemed twice does NOT reset the previous run. Have to reset it first: reset methods:timemed settimemed thr: 6.0 hw: 6 rowthr: 5 col: DATA expr ABS I run trial: T Oooops: LocalExec::SetStatus: abnormal child termination for /aips++/weekly/linux/bin/autoflag Server 'autoflag' has failed unexpectedly! You will need to create the relevant tool again. If that causes unexpected behavior, please restart AIPS++ Please submit a bug-report using bug() if you can reproduce the problem. Added method 5: timemed (TimeMedian) *thr = 6 *hw = 6 *rowthr = 5 *rowhw = 10 norow = F column = DATA *expr = ABS I debug = F fignore = F Tried to do a running abort didnt work. Tried to kill and delete in the Tools in Use, but didnt work - had to cntr-C out and back in. Damn. Not clear if I need to run the flagger.auto() command again. Where are the flags? Did they get into the data when I done'd the flagger, or are they sitting in a table somewhere? I'll assume that they got entered when I left the tool. So back to autoflag: synthesis.autoflag.autoflag. toolname: myautoflag msfile: ngc5921.ms setfreqmed thr: 5.0 hw: 10 rowthr: 10.0 col: DATA expr ABS I setuvbin thr: 0.001 nbins 50 col: DATA expr ABS I settimemed thr: 6.0 hw: 6 rowthr: 5 rowhw: 10 col: DATA expr ABS I run trial: T Still complains about too few timeslots for chunks wiht 12, 14, 34 timeslots. But otherwise ran. Now lets do it for real: run trial: F Getting an error: WARNING: progress meter trying to update beyond range WARNING: progress meter trying to update beyond range WARNING: progress meter trying to update beyond range Doesnt tell me how many points it flagged. Does this mean none? Well, the data did look pretty clean. Back to the tutorial: S1.8.1, S1.8.1.1: synthesis.calibrator.calibrator. toolname: mycalibrator filename: ngc5921.ms compress: T (well, thats the default, so lets hope it works!) (why is it called filename instead of msfile, as in all the other tools?) LocalExec::SetStatus: abnormal child termination for /aips++/weekly/linux/bin/calibrater Tried again, same error. Had to exit and get back in. At least didnt have to cntr-C synthesis.calibrator.calibrator. toolname: mycalibrator filename: ngc5921.ms compress: T LocalExec::SetStatus: abnormal child termination for /aips++/weekly/linux/bin/calibrater Well, that blows. Did autoflagging screw up or what? Out, back in. This time try it with compress=F. Nope, still blows up. Lets get rid of the flag file. Nope, same error. Lets blow it all away and start again. I hope this doesn't happen at the tutorial! Maybe we should discourage people from deviating from the script? general.ms.ms. fitstoms msfile: ngc5921.ms fitsfile: /aips++/data/demo/NGC5921.fits synthesis.flagger.flagger. flagger toolname: myflagger msfile: ngc5921.ms auto trial: F (AHA! I didnt do anything to the windows - the flagger window wasnt made large enough to see the done button!) synthesis.autoflag.autoflag. toolname: myautoflag msfile: ngc5921.ms setfreqmed thr: 5.0 hw: 10 rowthr: 10.0 col: DATA expr ABS I setuvbin thr: 0.001 nbins 50 col: DATA expr ABS I settimemed thr: 6.0 hw: 6 rowthr: 5 rowhw: 10 col: DATA expr ABS I run trial: F (same error on progree meter) synthesis.calibrator.calibrator. toolname: mycalibrator filename: ngc5921.ms compress: T same error. OK, this is screwed. One more time, from the top, this time w/o autoflag. rm last ms. general.ms.ms. fitstoms msfile: ngc5921.ms fitsfile: /aips++/data/demo/NGC5921.fits synthesis.flagger.flagger. flagger toolname: myflagger msfile: ngc5921.ms auto trial: F synthesis.calibrator.calibrator. toolname: mycalibrator filename: ngc5921.ms compress: T takes some time - waiting for read-lock on ngc5921.ms/table.lock... Ok, this works - I'll attribute the above to autoflag. Maybe I'm not supposed to apply 3 methods at once? >>>BUG: autoflag failed on test spectral line dataset Back to S1.8.1.2: Oops, I wasnt even supposed to start a calibrator - I need to do an imager first. ME be damned, this is confusing! synthesis.imager.imager. imager toolname: myimager filename: ngc5921.ms compress: F (hmm, why is compress F here but it was T in calibrator?) ->all setjy fieldid: 1 fluxdensity: [14.8009,0,0,0] standard: Perley-Taylor 99 (setting fluxdensity was supposed to be changed) Go back to the calibrator I started (I did not done this - will this be a problem??), using Tools in Use & show. S1.8.2.1,1.8.2.2 setdata msselect:'FIELD_ID<=2' (leave rest as defaults) NOTE!!! Here, the single quote ARE required!! Worth a mention. setsolve type: G t: 300 refant:15 table: ngc5921.gcal (single quote are not required, although it will work with them) (there should be some mention of what preaverage is. Hellifino) solve How about a discription of what is being written to the logger. Final Fit per unit weight?? WAITAMINUIT!!!! This is an OLD version of the script!!! George has one that uses a subset of the data. ??? DAMMIT - I've been working off the stable version of the documentation!! Why do we have 3 versions of the documentation? Why would anyone want outdated documentation, even if they are using the outdated code? Whatever - printed out grvol3 from daily - nope, its exactly the same. I am clearly out of the loop here. What document am I supposed to be testing on?? OK, I'll continue to work though grvol3/node2, but I'll substitute the glish from Georges most recent spectral line script (email to NAUG on 5/13/02) instead of using the stuff that is written. rm -r the gcal table, reset setdata mode: channel nchan: 40 start: 11 step: 1 msselect:'FIELD_ID<=2' setsolve type: G t: 300 refant:15 table: ngc5921.gcal solve plotcal plottype: AMP tablename: ngc5921.gcal redid this with mutliplot=T, and decided I didnt want to sit and hit that stop sign button over and over, but there was not stop option, so I hit the done button on the pgplotter and got the following error message about a hundred times: "calutil.g", line 797: error, : "calutil.g", line 797: operand to .text is not a record File: pgplotter.g, Line 74 Stack: .(), calibrater.g line 403 .(), glish eval line 1 eval(), toolmanagersupport.g line 362 is not a function value I want to get through this before tomorrow, so I'm not going to bother to bug this. S1.8.3: Probably worth mentioning that at present you can only apply one bp, and that in future releases different interpolations will be available. setdata mode: channel nchan: 63 start: 1 step: 1 msselect:'FIELD_ID==1' NOTE!! You have to specifically reset these fields to calculate the bpass for ALL channels. The gui comes up with the previously entered values set. This is not apparent from the script version and should be mentioned. reset setapply type: G t: 0 table: ngc5921.gcal select: setsolve type: B t: 300 refant:15 table: ngc5921.bcal solve (again, some discussion of what is being reported) plotcal plottype: AMP tablename: ngc5921.bcal multiplot: F Well, this is a lousy way to look at your bandpass. plotcal plottype: AMP tablename: ngc5921.bcal multiplot: T Thats not much better. But I guess its ok for students. I'll send along my specline_plotbp.g script for anyone who wants it. S1.8.4.2: fluxscale tablein: ngc5921.gcal tableout: ngc5921.fluxcal reference: 1331+30500002 transfer: 1445+09900002 (ok, it took me about 4 times to get this right - the entries want NO single quotes, and no brackets on 1445+09900002. These types of differences between the gui and script approach need to be properly addressed.) plotcal plottype: AMP tablename: ngc5921.fluxcal multiplot: F WARNING! Last time I ran this, multiplot was T - this gets you into the stop-sign problem mentioned above. We really need a way to get out of that stop sign. S1.8.5,S1.8.5.2: setdata mode: channel nchan: 63 start: 1 step: 1 msselect:'FIELD_ID IN [1:3]' reset setapply type: G t: 0 table: ngc5921.fluxcal select: 'FIELD_ID==2' setapply type: B t: 0 table: ngc5921.bcal select: '' (mention that select needs to be set back to blank - not clear from script) correct Skipping S1.8.6, S1.9 onto S1.10: imaging. Personally, I don't think students should be taught to clean a datacube prior to continuum subtraction, since multiplicative errors scale with the peak flux remaining in the field, and for the majority of observations, you will be spending >90% of your clean cycles on the same flux in each channel, and may not properly clean the line. But since cleaning in the map plane doesnt work and we dont have a script for continuum subtraction in the UV plane, I guess we'll have to do it this way. S1.10.0.2: click on imager:myimager at the head of the tool manager. This is left over from setjy. setdata mode: channel nchan: 55 start: 3 step: 1 (note: this is different from GM's script) fieldid: 3 setimage nx: 256 ny: 256 cellx: 10 [arcsec] celly: 10 [arcsec] stokes: I mode: channel nchan: 55 start: 3 step: 1 (note: this is different from GM's script; why not show a cube of all the channels, and mention the averaging as an alternative?) fieldid: 3 (when would this ever not be the same as in setdata?) weight type: briggs rmode: norm robust: 1 clean type: clark niter: 3000 gain: 0.2 threshold: 0 displayprogress: T model: ngc5921.model mask: image: ngc5921.image residual: ngc5921.residual interactive: F (if I wasnt in a rush, I would try that interactive as true) OK, it ends here. As a minimum, I would do continuum subtraction and moment analysis, and show them imageprofilefitter. In the next email, I will send scripts that do this. The moment analysis is very crude - next week I want to work on a script that does something like what MOMNT in AIPS does. -john ------------------------------------------------------------------------------- Date: Mon, 10 Jun 2002 12:18:30 -0600 (MDT) From: George Moellenbrock Reply-To: aips2-naug@donar.cv.nrao.edu To: aips2-naug@donar.cv.nrao.edu Subject: Re: [Aips2-naug] documentation comments John, Thanks for your hard work on the script/doc. The grvol3 doc is getting updates this week by Debra and I; sorry it was a bit out-of-date for you. Many of your doc comments are already in the direction we are heading. The scripts I circulated are indeed more up-to-date at the moment. Here are replies to a few particulars: [Calibrater Tool Operation section] > 4th paragraph: I simply can't wrap my brain around this, especially > the first sentence. I hope no students ask me to interpret this! John, this is the most important point. Take a deep breath and think about it. The set* functions set things up and the solve and correct function actually do the work. The point is that you can set things up differently depending upon what calibration tables you have available at any given point in the process. Thus, the calibrater tool has "state". >>>>BUG: I do not get the option to flag all channels when I > do the above. Apparently this option is not available yet. > We were told that it would be available. What up with that?? > Submitted. (this is for x-y plots. It is apparently enabled > in the raster display). I said I would try. I have an idea for this, but it is very unlikely that it will be ready for next week. >>>BUG: I zoomed into an msplot, but can't seem to zoom out again. > There is no unzoom button, and cntr-middle button doesnt work. > Submitted. Yes, I have had trouble zooming when in edit=T mode. > (why is it called filename instead of msfile, as in all the > other tools?) Reconciling parameter names across tools is on our list. > Oops, I wasnt even supposed to start a calibrator - I need to > do an imager first. ME be damned, this is confusing! The ME is worth it. Not to wax too philosophical, but the ME enables a very useful way of thinking about calibration; in particular, about how the different calibration components (e.g, B, G, D) can be examined on the same footing (despite how they differ algebraicly). There really is nothing particularly new; it is not so much a different way about looking at calibration as an appropriately more unified way of looking at it. For those wishing to have a black box to reduce their data, the ME (and for that matter, the particular commands and mechanics you use to operate it) is unimportant, and granted, the cookbook should eventually accomodate this view at some reasonable level. But to those wishing to understand what they're doing, and optimize the calibration of their data (I presume the SS students are largely in this category, not to mention NRAO staff members!), I personally think it is a very effective way of looking at the calibration problem. I will be giving the calibration lecture at the SS, and I will be using the ME to anchor the discussion. Hopefully, I'll be able to get this point across. Taken together, a basic understanding of the ME, and comfort with the statedness of the calibrater tool, makes for a very handy way to think about and perform calibration. It is really cool, I think. > ->all > setjy > fieldid: 1 > fluxdensity: [14.8009,0,0,0] > standard: Perley-Taylor 99 > > > (setting fluxdensity was supposed to be changed) I think there are more important things to deal with, but I'll see if I can quickly tidy this. > (there should be some mention of what preaverage is. Hellifino) The preaverage parameter is discussed in the polarization cal section where it is an important concept. I probably could improve the description. > How about a discription of what is being written to the logger. > Final Fit per unit weight?? Good idea. > WAITAMINUIT!!!! This is an OLD version of the script!!! > George has one that uses a subset of the data. ??? > DAMMIT - I've been working off the stable version of the > documentation!! Why do we have 3 versions of the > documentation? Why would anyone want outdated documentation, > even if they are using the outdated code? Whatever - > printed out grvol3 from daily - nope, its exactly the same. > I am clearly out of the loop here. What document am I > supposed to be testing on?? It's ok John! The doc isn't quite up-to-date with the script, but I really don't think we are very sensitive to the difference. By the end of this week, there will be lots of improvements in the doc. In the end, I don't think the doc should be too tightly coupled to any particular dataset. Most important is completeness, and most of your comments have been in this area. > S1.8.3: Probably worth mentioning that at present you can only > apply one bp, and that in future releases different interpolations > will be available. Actually, you can get a time series of bandpass solutions, just like G. The ME buys you this. I am currently working with Dana Balser on doing just this so that he can track the VLA's wave-guide BP ripple. Interpolation is important, of course. At the moment, the nearest neighbor will be applied to the data. The SS dataset isn't very interesting for this point because the observation is so short. > setdata > mode: channel > nchan: 63 > start: 1 > step: 1 > msselect:'FIELD_ID==1' > > > NOTE!! You have to specifically reset these fields to calculate > the bpass for ALL channels. The gui comes up with the previously > entered values set. This is not apparent from the script version > and should be mentioned. I thought persistence of parameter settings was an important and useful feature of the gui. In the gui, all params are remembered. You can see them, so you can change them if you want. The ball is in the user's court. A useful point: At the CLI, they are NOT remembered, and defaults will be used for unspecified parameters. Even thought the behavior is different, I think this makes sense: if you can see them, they have a memory, if you can't see them, defaults are used. [imager.setdata] > (note: this is different from GM's script; why not > show a cube of all the channels, and mention the > averaging as an alternative?) I had the averaging in there because I got it working about the same time I was developing this script. I left it in because I thought it was a more interesting example. [In setimage] > fieldid: 3 > (when would this ever not be the same as in setdata?) In mosaicing, where you might select many fieldids in setdata, but want an image centered on one of them. I've been wondering if when only one fieldid is specified in setdata, that value could be the default in setimage. One problem with this idea is that you setimage may be run before setdata. Need to talk to Kumar. Thanks again, John, for your many useful comments. Cheers, George ------------------------------------------------------------------------------- Date: Mon, 10 Jun 2002 15:41:52 -0400 From: John Hibbard Reply-To: aips2-naug@donar.cv.nrao.edu To: aips2-naug@donar.cv.nrao.edu Subject: Re: [Aips2-naug] documentation comments George, thanks for the feedback. Its nice to spend time working on something and to see that it really is read, even if I may regret what I wrote later! Just some quick replies: George Moellenbrock wrote: > > > 4th paragraph: I simply can't wrap my brain around this, especially > > the first sentence. I hope no students ask me to interpret this! > > John, this is the most important point. Take a deep breath > and think about it. The set* functions set things up and the > solve and correct function actually do the work. The point is > that you can set things up differently depending upon what > calibration tables you have available at any given point in > the process. Thus, the calibrater tool has "state". I do get what is going on. My complaint is about the way it is written. Thats quite a first sentence. And I do think the verbiage "set the calibrator tool to a particular state which persists, ... (which themselves operate on the basis of the state of the tool)" is jargon-y. I can't think of a better way to say it, but would like for someone to try. If this section goes into the calibration section as I suggest, then perhaps it would be easier to talk to the actual code that is being run, and why the resets appear where they do. > > Oops, I wasnt even supposed to start a calibrator - I need to > > do an imager first. ME be damned, this is confusing! > > The ME is worth it.... No, no, don't get me wrong - I *like* the ME. I don't like having to start an imager tool just to do setjy, then go to a calibrator tool. Its easy enough with a script, but a little confusing from the GUI. I think setjy should be taken into the calibrator tool, even if it does act on the wrong side of the ME. Its not 100% in keeping with the aips++ philosophy, but won't the user ALWAYS want to do this step as part of calibration? No need to reply - I didnt really think this would lead to any changes. > The preaverage parameter is discussed in the polarization cal section > where it is an important concept. I probably could improve the > description. That is section 1.9, and in a section spectral line people may not read, yet preaverage is first run accross in section 1.8.2, so just bring the discussion up to when the students will first see the parameter. > > NOTE!! You have to specifically reset these fields to calculate > > the bpass for ALL channels. The gui comes up with the previously > > entered values set. This is not apparent from the script version > > and should be mentioned. > > I thought persistence of parameter settings was an important and useful > feature of the gui. In the gui, all params are remembered. You can see > them, so you can change them if you want. The ball is in the user's > court. A useful point: At the CLI, they are NOT remembered, and defaults > will be used for unspecified parameters. Even thought the behavior is > different, I think this makes sense: if you can see them, they have > a memory, if you can't see them, defaults are used. Thats fine; it should just be mentioned in the documentation as a difference between the command line vs GUI operation. Same with the use of single quotes, which is different between command line and GUI. > I had the averaging in there because I got it working about the same time > I was developing this script. I left it in because I thought it was a > more interesting example. Yes, its more interesting, but the final cube is less interesting for viewing and taking spectra. Cheers, john -------------------------------------------------------------------------------