NRAO AIPS++ Users Group Meeting - Minutes Date: 2003-8-13 (Wednesday) Time: 1300 MDT Video Hub: CV-conf Rooms: SOC317/CV311/GB241/TUCN505 AIPS++ Threat Level is: Yellow = Elevated [We have some breathing space, but the pressure is on.] 1. AIPS++/ISD Status Report (Joe, Steve) o Reorganization report - The AIPS++ IPT is now under Socorro Ops with Jim U. and Brian G. at the helm. This buys some much-needed stability for the developers, and also gives us strong advocates (who also want results!) up above. o Since software development at NRAO is now "project driven", the activities of the NAUG now need to be coordinated with the relevant testing groups of the ALMA and EVLA projects, for example. There is still an important role for the NAUG in pre-testing functionality before the projects (particularly ALMA) do their more official testing, to help with documentation, and to guide development in a broader sense. *Comment - The format of the NAUG meetings will be about 50% testing information and reports, and 50% discussion of development issues and priorities. o Green Bank - as part of the reorganization, Green Bank has been at least temporarily been seconded Jim Braatz and Bob Garwood to work under the Software Development division on whatever. For time being, the NAUG is out of the loop for GBT stuff. We will still need to deal with single dish issues for (E)VLA+GBT and ALMA however. At some point down the road (6 months? 1 year?) a reunification will probably be mandated by the powers that be, and divergence should be minimized in the meantime. *Action Item - Joe & Steve are keeping tabs (as best as we can!) on GB developments, such as the information posted at: http://wiki.gb.nrao.edu/ (e.g. under GB.Planning) o Part of the motivation for moving the AIPS++ IPT from DM to SOC-ops was to put it under the same line as AIPS. We are optimistic that this will enable better cooperation between the two Science Software projects. A particular area of overlap will be in EVLA development, where it makes sense to develop and prototype the hard stuff in whichever package is appropriate and port to the other. How this coordination is carried out is TBD. One possibility is that the NAUG meetings be integrated, as this is a good point of contact between scientists and developers in a (relatively) non-political forum. *Action Item - Jim, Greg, Joe & Steve will investigate whether the AIPS group would like to participate in NAUG meetings. o Project Office - updated and revamped: http://projectoffice.aips2.nrao.edu/ Please take a look at this, particularly the Release Status and Development Status items that appear on the front page, and the relevant ALMA and EVLA Development Plans that are linked in. *Action Item - Joe will add a column to the tables indicating items that are ready for NAUG testing. o Benchmarking - the performance target for ALMA is to be within a factor of 2 of equivalent packages (AIPS & GILDAS) by R1.1. Sanjay, Kumar and George have been making good progress, and it looks like nearly there in imaging and calibration for moderately sized problems. Stay tuned. *FYI - Interested NAUG members should check out Sanjay's documents on the performance, which are linked under the first two items on the Current Release Status table on the Project Office page: Imager: http://aips2.nrao.edu/projectoffice/imager.ps Calibrater: http://aips2.nrao.edu/projectoffice/calibrater.pdf Many thanks to Sanjay, George, and Kumar for working hard on these improvements! o The new stable snapshot SS1 = v1.9 Build 19.047 is out this week, see http://aips2.nrao.edu/docs/reference/updates.html *Comment - The AIPS++ "Version" number has been up-reved from 1.8 to 1.9, and the AIPS++ Build number now continues as 19.xxx. We are numbering the Stable Snapshots starting with SS1, which are to be issued every 2 months, giving 3 for each 6-month ALMA release (only 2 for R1). The ALMA releases R1, R1.1, etc. are for ALMA Software and are nominally independent of the AIPS++ build numbering. Sorry, this is sort of confusing! 2. NAUG Plan 2003 (Steve) o NAUG activities in the coming year will be focused on pre-testing and input on ALMA development, and monitoring of progress of the performance and robustness improvements. Also on the horizon is EVLA support, particularly in thinking about the hard stuff like wide-field, wide-band, high dynamic range imaging, and RFI. Eventually, we should think about look-and-feel improvements to the user interface, etc., as well as VLBA support (particularly in the context of EVLA Phase II). o I am in the process of revising the 2003 NAUG Plan, which is similar to the previous version, though with GBT stuff removed and synthesis stuff expanded. This will appear under: http://www.aoc.nrao.edu/~smyers/aips++/plan/ as NAUGplan2003v3* files. This plan is written as if the whole of 2003 is scheduled, but only the last 4 months are now available (e.g. 33% of the hours scheduled). *Action Item - Steve will complete this new version of the plan. o The new plan has everyone "assigned" to a requirements task area (corresponding to the sections in the ALMA and EVLA offline req docs). Proposed leads and seconds in each area are: - 1.0 General [Myers] - 2.0 Interface & Documentation [Van Moorsel?, Shepherd] - 3.0 Data Handling [Myers, EVLAx] - 4.0 Calibration & Editing [EVLAx, Hibbard?] - 5.0 Imaging [Fomalont] - 6.0 Analysis [Hibbard?] - 7.0 Visualization [Brisken] - 8.0 Simulation & Special Modes [various] *Action Item - We will try to set up a system where the NAUG members are notified of stuff in each area to be tested (sort of like defect assignment within AIPS++, but more informal). o The biggest need is for pre-testing of ALMA deliverables for R1 (01Oct03) and R1.1 (01Apr04). Testing periods should be Aug-Sep 2003 and Nov-Dec 2003 respectively. *Action Item - Joe, Steve, & Kumar should get the first items out to testers asap! 3. ALMA (Debra, Kumar) o Debra Shepherd is now Brian G.'s deputy and is the Offline Subsystem scientist in charge of Offline requirements and testing. She has drawn up an ALMA Offline Testing Plan covering 2003-2006. This will rely upon NAUG pre-testing of things, and thus coordination is needed between our testing and the ALMA testing. *FYI - ALMA Offline Testing Plan, draft 2003-Jun-19 http://www.aoc.nrao.edu/~dshepher/alma/ALMA.sftw.tstplan.19jun03.pdf These are based on the requirements audit. *FYI - ALMA Offline Audit: http://aips2.nrao.edu/projectoffice/ALMAoffline_audit.pdf o Debra has assembled test datasets for the ALMA testing which could also form the basis for NAUG testing (it would good to pre-test using the ALMA datasets!). From Debra: *NGC7538 - a project that Miller has allowed me to work with. There are many fields, not overlapping, looking at different sources. Src D has the good continuum emission and expected weak NH3 line emission. I've done a real rough data reduction on the source D field but no continuum subtraction, careful flux cal, or analysis. I think I'll choose this data as an ALMA test but it is also applicable for VLA/EVLA testing. Data, preliminary script available in /home/sola3/dss3/alma/tsts/offline/tst1.jan04/NGC7538 *G192 - science observations of NH3, continuum subtraction required. My summer student and I have been working on this. It is almost done so it might not be appropriate for new testing although if someone else wants to reduce it along a different path, I'm happy to provide the data locally. I can also provide the script. Data and complete script available: /home/sola2/dss2/g192/vla.nh3/reduce *G192 - quick snapshot of water maser observations. Self-calibration needed (no flux cal or bandpass obs available). I've done a quick reduction in AIPS++, no self-cal done yet. The data and rough script are available at /home/sola/dss/current.projects/g192/vla.water.maser Debra also has other VLA and ALMA datasets being used for testing *FYI - see the benchmarking datasets at: http://www.aoc.nrao.edu/~dshepher/alma/benchmarks/ *Note - IRAM datasets will be necessary to test some developments, such as multi-band capability. o Debra is also assembling ALMA Use Cases that will guide Test Cases for the various subsystems (including Offline and Pipeline). *FYI - Check out the draft use cases at: http://www.aoc.nrao.edu/~dshepher/alma/usecases/ o Early focus of ALMA development is on performance, robustness, and core classes for pipeline use. Much of the interface (e.g. GUI) work is deferred until later cycles. o ALMA CDR1 review was conducted in June. *FYI - Minutes of CDR1 are at: http://aips2.nrao.edu/projectoffice/alma/cdr1_minutes.pdf o ALMA design is proceeding toward CDR2 in June 2004. *FYI - Kumar & Joe have drawn up an Offline Software Development Plan: http://aips2.nrao.edu/projectoffice/alma/offline_plan.pdf Lindsey is working on pipeline use cases as well as drawing up the Pipeline Plan. *Action Item - Joe will link in relevant Pipeline docs to the Project Office 4. EVLA (George, Steve, Joe) o EVLA Software Project Scientist job has been advertised internally, and some interviews have occurred. We hope that whoever takes this job will have substantial NAUG involvement. *Note - when the EVLAx person is on board, more detailed plans will be made for EVLA testing and what will be needed from the NAUG in this area. o Use Cases - I have some prototype use cases, which I will try to massage into shape in the next month. These consist of: 1.0 VLA Single-Pointing Narrow-Field Continuum 1.1 Intensity only 1.2 Polarimetry (circular feeds) 2.0 VLA Single-Pointing Narrow-Field Spectral Line 1.1 Intensity only Are there others that would be ready to go? *FYI - draft use cases can be found at: http://www.aoc.nrao.edu/~smyers/aips++/vla/usecase/VLAUseCases.pdf o Post-processing requirements - the draft requirements were completed and submitted for comment in July 2003. *FYI - The EVLA Post-processing Requirements (2003-Jul-03) are at: http://www.aoc.nrao.edu/~smyers/evla/ssr/evla_offline.pdf Comments and replies are compiled at: http://www.aoc.nrao.edu/~smyers/evla/ssr/evla_offline_comments.txt 5. AIPS++ Developments (Joe, Kumar, George, David, Sanjay) o Targets completed, see the Project Office front page. Highlights: *Viewer - Dave King has added more MS editing functionality, including prototype blinking, zoom fixes, and statistical editing (RMS, deviation vs. mean). See the viewer docs for viewerdisplaypanel option records: http://aips2.nrao.edu/docs/user/Display/node182.html e.g. under "MS and Visibility selection" for RMS and running mean difference. Blinking demo was completed, and a usable version is forthcoming shortly. *uv-plane continuum subtraction (ms.uvlsf) - George has checked in this new function. See the daily URM entry for ms.uvlsf, currently http://aips2.nrao.edu/daily/docs/user/General/node355.html *image-plane continuum subtraction (image.continuumsub) - George has checked this in, see the daily URM entry for image.continuumsub, currently http://aips2.nrao.edu/daily/docs/user/General/node44.html *calibrater scan-based time gridding (calibrater.setsolve) - George has added this capability to calibrater, along with some critical performance improvements. From the URM: "The solution interval t, if > 0.0, specifies the duration of data used for each calibration solution. In general, the solution intervals are measured from the beginning of data segments for each field and spectral window. If t is large enough, a single solution may encompass data from more than one scan (as long as the field and spectral window are the same). The solution interval represents a coherence time, not an integration time w.r.t. gaps in the time series; in effect, such gaps are ignored, and the lastest time in the solution is never more than t seconds after the earliest time. If t = 0.0, one solution per scan will be performed and delivered, regardless of the (variable) scan durations." See the daily URM entry for calibrater.setsolve, currently http://aips2.nrao.edu/daily/docs/user/SynthesisRef/node24.html o "SPLIT" capability - Kumar briefly described the proposed "SPLIT" functionality (the ability to write out subsets of data into a new ms, as in AIPS SPLIT or SPLAT). The split MS facilities in order priority: i) Selection of source and spectral window and outputting DATA or MODEL or CORRECTED_DATA or linear combinations of. Selection of the CORRECTED_DATA means freezing a calibration stage. ii) select channels and/or Average (or smoothing) iii) select time and/or average a) Correct for frame changes specially for topocentric line data. b) optimal time averaging [lower priority...e.g short baselines averaged over longer times] iv) Reject flagged rows v) change phase center of data (hence redo u,v,w) vi) Concatenate SPWs or fields into one... vii) Change ephemerides or correction of online system assumptions 6. Upcoming meetings and deadlines: o 2003 Aug 10-17 Single Dish Summer School (GB) o 2003 Aug 25 ANASAC (ALMA) face-to-face meeting (O'Hare) o 2003 Sep 5-6 ASAC (ALMA) Meeting (Hamilton, Ontario, Canada) o 2003 Sep 8-9 EVLA Advisory Committee Meeting (SOC) o 2003 Oct 1 ALMA R1 o 2004 Apr 1 ALMA R1.1 o 2004 Jun 1 ALMA CDR2 = Judgement Day! The agendas for past NAUG meetings are archived at: http://www.aoc.nrao.edu/~smyers/aips++/agenda/ The minutes for past NAUG meetings are archived at: http://www.aoc.nrao.edu/~smyers/aips++/minutes/