From efomalon@cv3.cv.nrao.edu Fri Oct 26 15:47:54 2001 Date: Fri, 26 Oct 2001 15:16:43 -0600 From: Ed Fomalont To: aips2-naug@zia.aoc.NRAO.EDU Subject: msplot using X,Y plots Hello all, This memo (also sent in as a long-winded defect) is a complement to the Oct 17 report on msplot by Greg. He used the raster (ala TVFLAG) method for editing the data. I used the X-Y plots of amplitude versus u-v spacing to edit the data. Both these reports are meant to inundate Ralph as he works on msplot. There are lots of little bugs and improvements that we both are suggesting, but the two major bugs I think are: 1. Msplot in edit=T mode takes way too long. Ralph is aware of this and has already shortened the plot access by 50%. However, with the 2 million rows in my test data set, 15 minutes is still too long. If the access time could be dropped to less than 4 minutes, about twice that needed for edit=F, that would be acceptable. For some reason the copying of the original flag tables to a flag file takes much too long. Putting this copying in C++ code is needed. 2. I tried editing outliers from the data base which has seven sources, each with 2 IF's and 4 Stokes. The flagging went well during the session and I terminated in the usual way (hitting the done button and keeping the flag). However, when I replotted the flagged data, it was clear that some of the flagging boxes (about 40 were probably made to flag all of the sources and stokes) were ignored. I checked on this several times. I don't know if Greg had similar problems, although the two flagging methods are probably handled differently in the code. Okay, here is my defect to aips++ and to Ralph. I also suggested changes to the documentation. ---------------------------------------------------------------- Hi Ralph, Here are some suggestions when you get around to revising the 'Editing vis X-Y Plots' documentation. My comments on the documentation are associated with the '**' symbols. There are also defects intermixed with the suggestions, and these can be found by the '??' symbols. Below is an ascii rendition of 'Editing via X,Y plots' in the daily release (October 26, 2001) Editing via X,Y plots msplot will plot any of a range of quantities in X,Y scatter plots. This may then be edited by drawing a box on the display using the mouse. ** (please add) You must specify edit=T in order to have editing capability in msplot. Use the X axis, and Y axis buttons to control what is plotted along the axes. Optionally use the Polarization Selection button to select the polarizations to be plotted e.g. RR, RL, LR, or LL. You can usually leave this as the default, and all full polarizations will be displayed (as different colors). ** I'm not convinced this is a good default. Why not just say that all four polarizations are picked by default. ** It is often messy to edit data with both RR LL and RL LR together. Optionally use the Spectral selection button to select the Spectral Windows and the range of channels. Optionally use the Fields button to further select the data. To start flagging, move to the plot window. You will select regions by pressing the mouse button 1 (usually the left button), and then dragging to the top right corner. The selected region will be drawn in a green cross-hatched pattern, and the coordinates of the region will be printed out. You may make as many of these regions as you like. ** You only flag the data that have been plotted, with two exceptions: To control whether the edits are applied to all channels and all correlations uniformly, you may select the corresponding button on the edit window. You can flag only those spectral windows that are displayed. If you plot only every nth data point, then the flag boxes will only apply to those points plotted. To see the currently defined edits, press the List button. ?? I tried this after several flags, but got 'no valid records to list' If you make a mistake, press the Cancel button to forget the current edit or Clear to forget all the stored edits. Note that the cross-hatched region will remain but will turn light grey. ?? Cancel did not turn a region to light grey and did not cancel the flagged region, However, the Clear button did what was advertised. Once you are finished defining edits, they may be applied and the display reloaded using the Apply button. You may now continue defining more edits. ** I believe this is now the Flag button. Also, as indicated below, if the X and Y ranges are unset or defaulted, I believe the new appropriate scaling of the plot should be made. If you really mess up, then you may revert to the original flagging status on startup using the Revert button. ** Yes, this really works The same set of steps may be used with the Locate button instead of the Flag button. This lists the selected points to the logger. No flagging is done. This is useful for finding outlier points. ** I don't think the above needs its own paragraph. Why not make it another bullet item, such as The Locate button will list all of points which are contained in all of displayed flagging boxes. The points are given in the logger. After listing, the boxes become grey and are inactive. To continue flagging or locating, draw more boxes (including those over the locate boxes) and press the appropriate button. ?? The above all works EXCEPT: When you make more than one box and press the locate button, the points in all of the boxes are printer in the logger, but only the first box turns grey. However, the other boxes are inactive. This last paragraph needs rewriting. ORIGINAL: The flags are applied with each press of the Flag button. On exit from msplot, you will be asked whether to keep these flags permanently or to restore the old flag columns. If you choose to keep these flags, then the flag column table will contain the flag columns as they were prior to this run of msplot. If you choose to restore the old flagg columns then the flag column table will contain the flag columns after applying these edits. SUGGESTION: (It is not clear how to exit from msplot. The are two present methods. One is to hit the file 'box' on the msplot windown, and go down and hit 'done'. The second is to hit the red done buttom in the bottem right of the plot. The latter is the preferred way, I think. The little yellow box under the button should say 'finished editing or viewing'.) NEW PARAGRAPH: The flags are applied with each press of the Flag button. To exit from msplot, press the red done button near the bottom right of the msplot window. A box will come up and you will have thirty seconds to choose: yes (default): The flags from the present session will be added to the pre-existing flag columns, and the flag table .flags.n file (where n will be incremented as needed) will contain the pre-existing flag columns. no : The flags from the present session will NOT be applied, but placed in the flag table .flags.n. The pre-existing flag columns in the measurement set will be returned. ---------------------------------------------------------------------------------------------- Other defects and improvements: 1. For a data set with 2 million rows (4 stokes, 1 channel), getting msplot up with edit=F takes 2 minutes. With Edit=T takes 15 minutes. Copying the original flag table should not take this long and I hope it can be sped up. Otherwise, users with more than a few hours of VLA data will find this waiting time excessive. There is no reason why edit=T should take more than twice as long as edit=F. 2. Self-scaling suggestion for the plot. I think the plot should self-scale to the entire X and Y range if X and Y are set to 'unset' or default. Otherwise, use the indicated values. Thus, if high points are flagged, the plot after hitting the flag buttom should self-scale to the new range. If you unflag, than the range might increase again. 3. A somewhat related problem: Suppose you have a measurement set and a flagtable. How do you apply that flag table to the measurement set; ie. replace the existing flags? I didn't see anything in flagger. It is not unusual to have a set of flag tables which are useful for different types of subsequent flagging or imaging. 4. For large data bases with several sources, I believe that msplot in the X,Y edit mode has some problems. Even if the plot after flagging looks good, after 'doneing' msplot and the replotting the data (with the new flags persumably in the measurement set, it is clear that some of the flagwindows from the previous session have been ignored. This only seems to happen on largish data sets, perhaps only with multiple sources. I'll send you more specific information on this defect. THIS IS A SERIOUS ERROR.