NRAO AIPS++ Users Group Meeting - DRAFT MINUTES Date: 2004-8-11 (Wednesday) Time: 1300 MDT Video Hub: SOC-conf (CV should call in to Socorro) Rooms: SOC317/CV311 AIPS++ Threat Level is: Beige = Sleepy [Too tired, need vacation] Draft Minutes: 1. NAUG News o stable "SS7.5" - tagged 8/17; a patched version will hopefully be available Aug 23 or shortly thereafter. o CDR2 (see below under ALMA) o TST2 (see below under ALMA) o RECURRING - The items that need NAUG review/testing can be found at: http://projectoffice.aips2.nrao.edu/testing.html (also accessed from the Module Testing link on front page). o RECURRING - Testing reports (general discussion) *Ongoing - any testers with stuff to report? 2. AIPS++/ISD Status Report o Framework - progress report (Joe) Near term focus in the SSG is on preparing the code base and libraries for inclusion into a new framework, and setting requirements for ACS prototyping and the full-up new framework within e2e. Contacts have been made with ACS with plans to have a "lite" version of ACS suitable for offline processing use. The schedule is to have a limited number of tasks runable in March 2005. Doug described larger plans for a more general framework, in the context of NVO and OPTICON (an EU based project, Preben Grosbol is the ESO contact). Bill Cotton has bound AIPS into Python, purportedly allowing calling of AIPS tasks directly from Python. It would be worthwhile to give this PyAIPS a try. It would be advantageous to be able to do similar things with aips++ (call directly from Python without passing through another framework) for certain applications. o Tasking - outline of goals, scope, & plans (general discussion) - CLI parameter interface to function(s) = "Task" What is needed is the ability to set up parameters (with type and range checking) and say "GO" to a function or set of aips++ functions. Similar things have been tried (e.g. PyRAF for IRAF in Python) and should be investigated. - bundling capabilities For example, AIPS CALIB and IMAGR. Currently, in calibrater, you run a series of set functions and solve functions and apply functions. A calibtask would bundle this with parameters telling what terms to calibratate (and possibly in what order). There was some worry about how many parameters is too many. *Action item: Myers proposes scoping out a calibtask as an example. Another example would be a combined calibrater and imager task, e.g. for self-cal. o Other development: Tim Cornwell has been working on simulator and making good progress. Hopefully we can get rid of the deprecated "newsimulator" soon! 3. ALMA o CDR2 report (Joe) Steve's summary: The ALMA CDR2 was held in Boulder July 9-10. The offline subsystem design, progress, and plans were presented (including data capture and the export data format in addition to aips++; pipeline subsystem was also there). We have yet to see any comments from the panel, though the exit summary was positive. We do not foresee substantial changes to our plans. Hopefully, the ALMA JAO and NRAO executive will make a postive statement regarding the results of the CDR for aips++ in particular, and will help to move beyond the seige atmosphere that the project has labored under these past years. There were some concerns expressed at the CDR by non-panel attendees that the focus on particular test cases in the development so far has biased the reported benchmarks and progress. In particular, Ed's testing of AW602 was construed to be at odds with the reported performance. This was not the case, as documented in further correspondence. For details, see Ed's test log and compiled replies at: http://www.aoc.nrao.edu/~smyers/aips++/testlogs/2004-07-06-fomalont.txt and his scripts for aips++ http://www.aoc.nrao.edu/~smyers/aips++/testlogs/2004-07-06-AW602.g and AIPS http://www.aoc.nrao.edu/~smyers/aips++/testlogs/2004-07-06-AW602.001 The upshot is that the problems seen were expected (and in line with what previous tests and benchmarks found) though the impact for this particular dataset was somewhat worse due to the balance of operations. Basically, these tests inherited alot of improvements from our development and upcoming development already underway (and planned before Ed's tests) should address the others. There were also minor issues of use of glish-based operations (e.g. using glish-based flagger instead of c++-based autoflag, see George's reply in the testlog), plus the usual sub-optimal documentation to confuse things. In any event, this dataset will be folded into the benchmarking so we should be able to track progress. Ed's tests were extremely helpful, and I think the subsequent correspondence addressed the issues and acknowleged the problems. It is unfortunate that it has been taken somewhat out of context. We can only hope that this sort of misunderstanding goes away as we move forward with the ALMA and EVLA development :) o TST2 plans (Joe) The focus in on (spectroscopic) mosaicing. There is an ALMA TST2 page: http://projectoffice.aips2.nrao.edu/almatst2/ALMA_TST2.html In particular, see: Goals/Scope - http://aips2.nrao.edu/projectoffice/almatst2/TST2_goals.html Internal Testing Plan - http://aips2.nrao.edu/projectoffice/almatst2/TST2_plan.html Results/Comparison - http://aips2.nrao.edu/projectoffice/almatst2/TST2_results.html - datasets are NGC1333 (VLA SiO 1-0) and NGC4826 (BIMA CO 1-0) - scripts (will be available after pre-testing) - general cookbook structure (see Main Event below) Internal testing has uncovered bugs and issues, e.g. the vlafiller does not handle the tolerances for spectral windows in the presence of the doppler tracking scheme used, so it was proposed to fill in AIPS (which presents its own issues for field IDs for the different freq IDs). Debra asked what it would take to make the vlafiller do what is needed, and Joe stated that it would not be possible given current manpower and targets to do this without delaying the test. He argued that the test focus was on mosaicing not filling, and since we were already doing the BIMA line-length correction outside aips++ it was in line with the test goals to fill outside in AIPS anyway. NAUG testing should commence pending availability of a stable (see above). NAUG testers are Myers, Brisken, and possibly Hibbard. *Action Item: Allocate datasets to NAUG testers (Myers) o Single dish ALMA development (general discussion) There was a meeting called by the CV folks to discuss single dish issues for GBT and ALMA, but since those with single dish experience were absent (e.g. Jeff Mangum, Al Wootten, Tom Wilson) the meeting was aborted. Brian Glendenning, Debra Shepherd and Joe McMullin were in attendance from here and John Hibbard was only NAUG member there. Given that the SSG (and the ALMA Offline group in particular) will have to implement this, I made the point that in the future someone (e.g. Joe) will need to have at least a heads-up on any relevant deliberations if not actual participation. There is some worry that current GBT software development was being done in such a way that it would be of no benefit to ALMA (or make any use of ALMA development). Presumably this will be addressed in these deliberations and in the upcoming GBT software review. *Action Item: Keep the NAUG and SSG in the loop (Hibbard) 4. EVLA o Preparation for EVLA development No discussion, deferred until Bryan returns 5. Main Event - Presentations and Discussion Items o Cookbook For previous tests, we have produced a special cookbook geared toward the given test. In the future, it is clear that we need a more general guide/cookbook combination (e.g. for maintainability). Joe proposes the following outline for NAUG comment: --- Introduction - Getting Started Basic initialization Tools in AIPS++ Troubleshooting Measurement Sets Summary of Data Reduction steps Data Examination and Editing Assessing data quality Visibility displays msplot viewer Data Editing Interactive msplot viewer Autoflag clip quack misc Calibrating Interferometer Data Calibration Strategy - Combining different calibrations "Gain" (time-dependent) calibration (G) Flux density scale calibration Bandpass calibration (B) Instrumental Polarization calibration (P, D) Ionospheric calibration (F) Application of calibration to data Splitting the data Imaging Interferometer Data Basic imaging data weights cell and image size, shifting multiple fields multiple spectral windows Deconvolution basic clean multiple fields clean boxes interactive clean other algorithms Mosaicing Single Dish Imaging Wide Field Imaging (non-coplanar effects) Manipulating Clean components Self-calibration Displaying Images viewer contour raster vector interactive displays load manipulate display intensity and color transfer labelling annotation slicing blinking printer displays of data image misc Image Analysis combining images subtract polarization, position angle misc statistics moments image manipulation blanking image fitting 2d gaussian gaussian fits to slices source finding image analysis building/selection on data cubes modifying image headers Brief Command Summaries autoflag calibrater componentmodels flagger imager viewer msplot image Step-by-step guide to continuum data analysis -- example of 0957+561 Editing and Calibration Fill data Inspect and Edit Calibrate Making and Cleaning images Data Analysis Step-by-step guide to spectral line analysis - NGC 5921 Editing and Calibration Fill data Inspect and Edit Calibrate Making and Cleaning images Data Analysis Moment analysis Annotated example scripts NGC 5921::VLA::red-shifted HI emission U Cam::IRAM::1/3 mm line and continuum emission GG Tau::IRAM::1/3 mm line and continuum emission G 192::VLA::NH3 line emission NGC 7538::VLA::NH3 line emission Gravitation Lens 0957+561::VLA::Polarized continuum H121::IRAM::CO J=1-0::3 mm line emission Instrument Specific AIPS++ Guidelines VLA IRAM BIMA ALMA --- The was discussion about the split into general interferometry and aips++ "user guide" and instrument-specific "cookbook". It was noted that our current such split into "Getting Results" Volume 1 - Basic Tools Volume 2 - Generic Processing Volume 3 - Telescope Specific Processing has not been deemed entirely successful. In particular, the need for a single "one-stop" specific document e.g. for VLA implies that at least some of the generic stuff needs to be in the specific cookbook also. Myers proposed that if the documentation were built in such a way that "custom cookbooks" could be built automatically, say by combining the relevant Volume 1-3 text using an outline like that above, then all parties could be satisfied (the split docs are still available, as are the combined custom guides). *Action Item: review outline and comment (NAUG) 6. AIPS++ Developments - see latest targets and info at Project Office: http://projectoffice.aips2.nrao.edu/ 7. Upcoming meetings and deadlines: o 2004 Aug 23 NAUG TST2 pre-testing o 2004 Sep 15 ALMA TST2 begins o 2004 Sep 27-28 ALMA ASAC meeting (CV)? o 2004 Oct 1 ALMA R2 The agendas for past NAUG meetings are archived at: http://www.aoc.nrao.edu/~smyers/aips++/agenda/ The minutes for past NAUG meetings are archived at: http://www.aoc.nrao.edu/~smyers/aips++/minutes/