Date: Thu, 30 Mar 2006 15:17:14 -0700 From: David Whysong Subject: Re: Summary so far. > - The only param whose choice changes the further params shown is > the first one (alg) for simplicity. One could think of making > this possible for others (e.g. weight.type='briggs') I was thinking of something a bit more complex, and I thought I wasn't alone... something like this: [CLEAN] _____________________________________________________________________ Task to deconvolve (multiple) ms using CLEAN. _____________________________________________________________________ imagenames :: Names of input/output images model = | model image (I/O) complist = | component list (I/O) mask = | mask image (I/O) image = | restored image (O) residual = | residual image (O) psf = | psf (beam) image (O) _____________________________________________________________________ setimage :: Set position and stokes shape of output images nx = ny = cellx = celly = stokes = doshift = phasecenter = shiftx = shifty = _____________________________________________________________________ setchannels :: Set frequency axis shape of output images mode = | Chan mode (mfs,channel,velocity) nspw = | Number of spectral windows nchan = | Number of channels for each window chwid = | Channel width (GHz,kms) per window _____________________________________________________________________ weighting :: Set controls for visibility gridding weights type = | Weight mode (briggs,uniform... rmode = | Mode for Briggs weighting ... uvmin = ! Min uvradius for nonzero weight uvmax = | Max uvradius for nonzero weight taper = | Taper type (none,Gaussian) tapbmaj = | Gaussian taper bmaj ... _____________________________________________________________________ selection [on/all data] :: Set controls for selecting data bchan = | Beginning channel echan = | End channel nchav = | Number of channels to average ants = | List of antennas to select ... _____________________________________________________________________ clean :: Set common controls for cleaning algorithm = | 'clark'|'hogbom' niter = | Max number of iterations ... _____________________________________________________________________ wf [on/off] :: Set controls for wide-file imaging wplanes = 1 | if > 1 then wprojection nfacets = 1 | if > 1 then uv faceting _____________________________________________________________________ mosaic [on/off] :: Set controls for mosaicing gridtype = | 'image','uv' ... _____________________________________________________________________ sd [on/off] :: Set controls for single-dish data inclusion useac = F ! Use interferometer autocorrelations ... _____________________________________________________________________ interaction :: Set controls for interaction interactive = F | Interactive clean? async = F ! Run asynchronously? _____________________________________________________________________ commands: GO, SAVE, GET, ESC, EXIT, SCRIPT, TOOLSCRIPT, HELP, EXPLAIN Here are the changes from your version: > NOTE: input ms(s) must have been selected using SELECT task. Is it absolutely necessary to run SELECT even if I want to operate on all my data? There should be a sensible default to select all data in the ms. > alg = 'wf','mosaic','sd' | Choice of algorithm(s) I removed this; the selection of each of these is moved into the areas where you define the parameters for each. In the spirit of grouping similar things together, I'd just put an on/off switch in the wf, mosaic, and sd sections. If it's turned off, then just print "sd: off" and hide the sd options (but note, sd can of course be turned on again from within this epar-interface). I also think I wasn't alone in calling for the option to SELECT from within the task (as well as before the task); I envisioned that this would operate similarly as I described the algorithm selection. Have a SELECT section with an [on/off] switch; if set to 'on' then you can enter the selection parameters just as if you were running the SELECT task in stand-alone mode. Finally, I strongly believe that any option that doesn't make sense to the current configuration of the task should not be displayed. For example, most of the weighting parameters aren't necessary if one is doing simple uniform or natural weighting. This is pretty simple to implement if you organize the parameters into a tree-like structure of dependencies. A few other comments: >4. Since we envision that tasks have the same interface as tool > methods, tasks can also be built from tasks as well as the toolkit > (mix and match). For example, multiple CALIB calls can be > assembled into a guided full calibration task. maybe AUTOCAL. But > this gets dangerously close to a pipeline and maybe is best left to > the pipeline or its emulator in the offline package. Why is it dangerous to have pipeline tasks be tasks in casapy? I'd think that this would be an advantage! A data-reduction pipeline is an automated way of doing exactly the same things that are done during interactive data reduction. Having the online pipeline be separate and substantially different in function from the offline package seems redundant. >5. Though not discussed, I would personally consider > moving autoflag up to the task level as AUTOFLAG and > keep flag or flagger (including auto options) at the toolkit > level. Just MHO. That seems reasonable. At the risk of incurring your wrath, I want to point out that from a user's perspective, there isn't much (if any) difference between a tool method and a task, since both have the same interface. So the 'task definition' discussion seems beyond the scope of the UI determination. We don't care if something is a tool method or a task, so long as it follows the interface and it gets the job done. >6. While talking about calibration, it was apparent that AUTOFLAG and > AUTOCAL plus voluminous plot outputs are insufficient and there > needs to be some sort of SNIFFER to summarize relevant info for > diagnostics. I suggested expanding this to some sort of VERIFY task, that attempts to check the calibration using some basic heuristic and notify the user if there are obvious trouble areas. This could be combined (in a procedure) with a separate calibration-based flagging task. -- David Whysong =============================================================================== Date: Thu, 30 Mar 2006 16:46:35 -0700 (MST) From: Steven T. Myers Subject: Re: Summary so far. I guess I'd better open this up... On Thu, 30 Mar 2006, David Whysong wrote: > > - The only param whose choice changes the further params shown is > > the first one (alg) for simplicity. One could think of making > > this possible for others (e.g. weight.type='briggs') > > I was thinking of something a bit more complex, and I thought I wasn't > alone... something like this: > > > [CLEAN] > _____________________________________________________________________ > Task to deconvolve (multiple) ms using CLEAN. > _____________________________________________________________________ > imagenames :: Names of input/output images > model = | model image (I/O) > complist = | component list (I/O) > mask = | mask image (I/O) > image = | restored image (O) > residual = | residual image (O) > psf = | psf (beam) image (O) > _____________________________________________________________________ > setimage :: Set position and stokes shape of output images > nx = > ny = > cellx = > celly = > stokes = > doshift = > phasecenter = > shiftx = > shifty = > _____________________________________________________________________ > setchannels :: Set frequency axis shape of output images > mode = | Chan mode > (mfs,channel,velocity) > nspw = | Number of spectral windows > nchan = | Number of channels for each window > chwid = | Channel width (GHz,kms) per window > _____________________________________________________________________ > weighting :: Set controls for visibility gridding weights > type = | Weight mode > (briggs,uniform... > rmode = | Mode for Briggs weighting > ... > uvmin = ! Min uvradius for nonzero weight > uvmax = | Max uvradius for nonzero weight > taper = | Taper type (none,Gaussian) > tapbmaj = | Gaussian taper bmaj > ... > _____________________________________________________________________ > selection [on/all data] :: Set controls for selecting data > bchan = | Beginning channel > echan = | End channel > nchav = | Number of channels to average > ants = | List of antennas to select > ... > _____________________________________________________________________ > clean :: Set common controls for cleaning > algorithm = | 'clark'|'hogbom' > niter = | Max number of iterations > ... > _____________________________________________________________________ > wf [on/off] :: Set controls for wide-file imaging > wplanes = 1 | if > 1 then wprojection > nfacets = 1 | if > 1 then uv faceting > _____________________________________________________________________ > mosaic [on/off] :: Set controls for mosaicing > gridtype = | 'image','uv' > ... > _____________________________________________________________________ > sd [on/off] :: Set controls for single-dish data inclusion > useac = F ! Use interferometer > autocorrelations > ... > _____________________________________________________________________ > interaction :: Set controls for interaction > interactive = F | Interactive clean? > async = F ! Run asynchronously? > _____________________________________________________________________ > commands: GO, SAVE, GET, ESC, EXIT, SCRIPT, TOOLSCRIPT, HELP, EXPLAIN > > > > Here are the changes from your version: > > > NOTE: input ms(s) must have been selected using SELECT task. > Is it absolutely necessary to run SELECT even if I want to operate on > all my data? There should be a sensible default to select all data in > the ms. SELECT also opens the ms in the tools necessary and its default is to just do the open and select all the data. Another model would be to specify the msnames inside CLEAN and put some selection there. I had thought that the majority didnt want this. Your mileage may vary. > > alg = 'wf','mosaic','sd' | Choice of algorithm(s) > I removed this; the selection of each of these is moved into the areas > where you define the parameters for each. In the spirit of grouping > similar things together, I'd just put an on/off switch in the wf, > mosaic, and sd sections. If it's turned off, then just print "sd: off" > and hide the sd options (but note, sd can of course be turned on > again from within this epar-interface). The whole point was to hide those whole blocks of parameters if not asked for. If you didnt ask for mosaic, you didnt see the mosiac ones. I guess it wasnt clear from the example that if you just asked for 'wf' you only saw the 'wf' ones. I'll add that to the example, sorry. Note if you dont ask for anything it assumes you are doing single-field (no wf or mosaic or sd). You have to have the alg= to tell it what to include later. > I also think I wasn't alone in calling for the option to SELECT from > within the task (as well as before the task); I envisioned that this > would operate similarly as I described the algorithm selection. Have a > SELECT section with an [on/off] switch; if set to 'on' then you can > enter the selection parameters just as if you were running the SELECT > task in stand-alone mode. You may not have been alone but my impression (or prejudice) was to separate SELECT (for reasons given above and in the summary). I'm not sure how multiple-ms selection can be simply done inside but I can see how outside. But there may be reasons to include some 'cross-ms' selection inside but you have to be careful (bchan and echan dont make sense). > Finally, I strongly believe that any option that doesn't make sense to > the current configuration of the task should not be displayed. For > example, most of the weighting parameters aren't necessary if one is > doing simple uniform or natural weighting. This is pretty simple to > implement if you organize the parameters into a tree-like structure of > dependencies. Yes, thats why alg= was there. I noted that one could go further and make for each section some key mode params (maybe marked with *) that changing will also cause other changes. I thought the preference was for a single branch level in the hierarchy tree - one could go deeper, but at the cost of some complexity in explaining it. I have no deep preference here, other than keying off the single alg= just seems simpler... Note statements like "pretty simple to do" mean different things to us and the programmers. > > > 4. Since we envision that tasks have the same interface as tool > > methods, tasks can also be built from tasks as well as the toolkit > > (mix and match). For example, multiple CALIB calls can be > > assembled into a guided full calibration task. maybe AUTOCAL. But > > this gets dangerously close to a pipeline and maybe is best left to > > the pipeline or its emulator in the offline package. > > Why is it dangerous to have pipeline tasks be tasks in casapy? I'd > think that this would be an advantage! A data-reduction pipeline is an > automated way of doing exactly the same things that are done during > interactive data reduction. Having the online pipeline be separate and > substantially different in function from the offline package seems > redundant. ' I didn't imply it was dangerous, only that since the pipeline was being developed (and paid for) it seems inefficient to recreate it in our tasks. The finite programmer time might be best spent on the levels distictly between pipeline and toolkit (not too close to either). > > 5. Though not discussed, I would personally consider > > moving autoflag up to the task level as AUTOFLAG and > > keep flag or flagger (including auto options) at the toolkit > > level. Just MHO. > > That seems reasonable. At the risk of incurring your wrath, I want to > point out that from a user's perspective, there isn't much (if any) > difference between a tool method and a task, since both have the same > interface. So the 'task definition' discussion seems beyond the scope > of the UI determination. We don't care if something is a tool method > or a task, so long as it follows the interface and it gets the job > done. True. This comment was to Joe and the team pointing out that there were some things (some of which have lots of glish) that could be moved from toolkit to (Python) tasks. There will be no Python in the toolkit BTW. > > 6. While talking about calibration, it was apparent that AUTOFLAG and > > AUTOCAL plus voluminous plot outputs are insufficient and there > > needs to be some sort of SNIFFER to summarize relevant info for > > diagnostics. > > I suggested expanding this to some sort of VERIFY task, that attempts > to check the calibration using some basic heuristic and notify the > user if there are obvious trouble areas. This could be combined (in a > procedure) with a separate calibration-based flagging task. I missed that. Noted. Though I think thats what we call the SNIFFER on the VLBA (I just borrowed the name). -s =============================================================================== Date: Thu, 30 Mar 2006 18:18:34 -0700 From: David Whysong Subject: Re: Summary so far. On 3/30/06, Steven T. Myers wrote: > > >> NOTE: input ms(s) must have been selected using SELECT task. > > Is it absolutely necessary to run SELECT even if I want to operate on > > all my data? There should be a sensible default to select all data in > > the ms. > > SELECT also opens the ms in the tools necessary and its default is to > just do the open and select all the data. Another model would be > to specify the msnames inside CLEAN and put some selection there. > I had thought that the majority didnt want this. Your mileage may > vary. Okay, I see that the ms.open call is necessary, so one must call SELECT before doing anything else. What I'm suggesting is a combination approach, where you can run SELECT before CLEAN; if not, or if you specify new selectrion parameters for CLEAN, then CLEAN will run it for you. (See below.) > >> alg = 'wf','mosaic','sd' | Choice of algorithm(s) > > I removed this; the selection of each of these is moved into the areas > > where you define the parameters for each. In the spirit of grouping > > similar things together, I'd just put an on/off switch in the wf, > > mosaic, and sd sections. If it's turned off, then just print "sd: off" > > and hide the sd options (but note, sd can of course be turned on > > again from within this epar-interface). > > The whole point was to hide those whole blocks of parameters if not asked > for. If you didnt ask for mosaic, you didnt see the mosiac ones. I guess > it wasnt clear from the example that if you just asked for 'wf' you only > saw the 'wf' ones. I'll add that to the example, sorry. Note if you dont > ask for anything it assumes you are doing single-field (no wf or mosaic or > sd). You have to have the alg= to tell it what to include later. Hm, I'm not sure we understand each other. I had the impression that if you do something like this: clean [alg = 'wf','mosaic'] ...then you would get the menu for clean, which would NOT include the options for 'sd' but there would be a toggle which would allow you to set 'sd' from within the menu. Is that correct, or are the hidden parameters completely unavailable without exiting and re-starting the menu? What I suggested is the following (for the 'clean' example): allow the user to specify parameters when entering the epar-style menu from the command line, exactly like "clean [alg='wf','mosaic', weight='uniform'"]. The epar menu comes up with those choices already filled in, which would then cause the menu to display the relevant parameters for those algorithms. Completely separately from that is the question of what parameters are displayed by the epar menu. I strongly believe that it should allow all possible and sensible choices that are available for the task, but not show parameters which would have no impact on the task. So in this model, if you run "clean [alg='wf'] then you won't see all the 'mosaic' or 'sd' options - but you DO see a single option for 'mosaic' and 'sd' which lets you select those options. If you toggle one of those options, the menu would then display the new parameters that can be set for those options. I think this is a cleaner interface, because you don't specify options in two places (when you enter the epar-menu from the command lnie, and within the epar-menu). > > I also think I wasn't alone in calling for the option to SELECT from > > within the task (as well as before the task); I envisioned that this > > would operate similarly as I described the algorithm selection. Have a > > SELECT section with an [on/off] switch; if set to 'on' then you can > > enter the selection parameters just as if you were running the SELECT > > task in stand-alone mode. > > You may not have been alone but my impression (or prejudice) was to > separate SELECT (for reasons given above and in the summary). I'm not > sure how multiple-ms selection can be simply done inside but I can see > how outside. But there may be reasons to include some 'cross-ms' > selection inside but you have to be careful (bchan and echan dont make > sense). I agree that SELECT should be separate; I'm just suggesting that the various tasks (such as CLEAN or CALIB or whatever) have the ability to call SELECT *if necessary* before they continue about their business. If you want to do multiple-ms selection, then have select parameters for each input ms, like this: ms = ____ select [on/all data] [add another ms] ...or this: ms = data1.ms, data2.ms, data3.ms select for data1.ms [all data] select for data2.ms [all data] select for data3.ms bchan=5 echan=10 ... This requires a small amount of additional logic; the epar-style front end must parse the ms line in order to know how many sets of SELECT parameters to collect. The task would then iterate over each ms and call SELECT with the specified parameters. (If no parameters were specified, then the call to SELECT is not strictly necessary, though if it were called it should be harmless.) This is a minor point, and I believe it would be trivial to implement (assuming SELECT will have some intelligence about when selection needs to be done, which I think was generally agreed upon by everone). Here is why I think this is a good idea: 1. Current AIPS and Miriad users are accustomed to specifying selections when they run their tasks, and not as a separate step. 2. I believe this requires little more than a function call at the beginning of the task, along with the extra XML to get the SELECT keywords. 3. For simple cases, it reduces the number of preliminary steps that one would have to do before imaging. > > Finally, I strongly believe that any option that doesn't make sense to > > the current configuration of the task should not be displayed. For > > example, most of the weighting parameters aren't necessary if one is > > doing simple uniform or natural weighting. This is pretty simple to > > implement if you organize the parameters into a tree-like structure of > > dependencies. > > Yes, thats why alg= was there. I noted that one could go further > and make for each section some key mode params (maybe marked with *) > that changing will also cause other changes. I thought the preference > was for a single branch level in the hierarchy tree - one could go > deeper, but at the cost of some complexity in explaining it. I have > no deep preference here, other than keying off the single alg= just > seems simpler... That is a bit simpler from a programming perspective but the interface is not as clean when you have to specify some parameters on the command line and some in the menu. I think that novice users (which may be most users, once the software is released outside this building) would greatly benefit from having all options available in the menu. I can easily imagine someone going into the clean menu and wondering where the mosaicing parameters are... Novices will also be happier if they aren't faced with extra parameters that are irrelevant to the choices they have made. It will reduce confusion and make the package easier to learn. Walter suggested a hierarchy of commands based on a user's experience; here, I'm suggesting that we let the user keep things simple on a more fine-grained scale. If the user doesn't want to deal with the more intricate weighting schemes, he can just set 'natural' or 'uniform' and be done with it, without seeing another half-dozen blank parameters in the menu. The resulting hierarchy could be multi-layer, but I don't immediately see areas where it would be, and I'm sure it would never be a deep hierarchy. I'd just like parameters to disappear when they aren't needed, from all parts of the menu - including selection, weighting, and the clean algorithm choices (I don't think Hogbom clean has major/minor cycles, for example)... not just in a couple places. > Note statements like "pretty simple to do" mean different > things to us and the programmers. I know very well how this looks from a programmer's perspective... I *think* it's pretty straightforward, because there aren't any inter-dependencies between the parameter lists for various options... and by that, I mean that selecting 'mosaic' only alters the mosaicing options, but it doesn't alter the parameters required for other things like 'sd' or weighting or imagenames, etc. -- David Whysong =============================================================================== Date: Thu, 30 Mar 2006 22:07:34 -0700 (MST) From: Steven T. Myers Subject: Re: Summary so far. As far as I can tell, we agree. The alg parameter controls what the user sees (you see the special params for options specified in the alg param), and that can be set on the command line or at any later time, since its just a normal parameter. It may be we want to have other paramters that cause branches, but we will have to do more work to tell whether this is necessary or desirable (but it is clearly possible). Another bit that I forgot to point out is that you get the parameter (in Joe's current version) interface by doing inp clean or inp clean 'ms' or inp clean ['wf','mosaic','sd'] I'm not sure what will bring up the gui form (epar?). The bit on including select in the task is possible, I'll leave for others to comment. -s