EVLA Archive 'To Do' Items for the WIDAR Correlator Arrival and Beyond - JMB - 2008-01-09 - JMB - 2008-03-31 - JMB - 2008-04-22 At VLA Site ------------------------------------------------------------------- - NGAS Module(s) - one or more will acquire the metadata files (sdm.xml) from the SDM staging host and the BDFs from the CBE. This NGAS module serves as an independent backup. We may choose to dispense with this backup eventually, but in the early phases of evla operation it's prudent to have an on-site backup. Requirements : NGAS modules in the std. NRAO archive configuration, network mounted (http), Oracle access (Oracle server already exists at VLA) - 2008-04-22 : James and I have decided to use our first ngas host newtech-1 as the VLA site backup. This is the machine with 12 HD's and is not really useful at the aoc. Will install NGAS v3.2 and move newtech-1 to the site. - The NGAS Module must function independently of any off-site (VLA) systems, notably the Oracle server in the AOC. We will need to set up a mirror Oracle NGAS DB on the Oracle server at the VLA. Need to think about this a bit more.. - SDM Staging Host - we can use the current staging host (igloo) for this. The mirroring software that keeps the AOC staging host (eskimo) up-to-date should probably be tuned up, or possibly replaced. It works fairly reliably, but not 100% reliably. - Binary Data Format CBE interface - this is probably not a module or process that (we) need to create, rather it is a port on the CBE to which the archive sends data file requests, and receives data files. - CBE/MCAF file ready signal - the CME and MCAF systems need to generate a signal or set a state whereby the archive system knows when a set of files are complete and available for archiving. We need to decide whether we want a signal or a state set. I prefer a state variable, it's safer. Or we may rename the BDF files when they close, and the archiving process is then able to recognize a closed (completed) file. Andreas says ESO does somthing like this. I think the Unix mv command waits for a file to close. At AOC Site ------------------------------------------------------------------- - NGAS Modules - These would be the 'permanent' archive storage modules. We will probably go to 8 TB modules with the next purchase. Need to do : - finalize the NGAS configuration : - RAID groups : 8 HD's (7 + 1 hot spare) - RAID group/partition identifiers for the NGAS DB (talk to Andreas) - NGAS program startup/shutdown options * Essentially finished. Host newtech-4 has NGAS v3.2 with generic online plugin. Works on mount points, etc. 'NRAO Configuration' is now under test. - NGAS program or Red Hat Linux - power management (James, Andreas)? Andreas info about BIOS - Linux restarts.. - NGAS module physical configuration options : - larger number of HD's, outside of ngas host box? - make cheaper TB/$$ - talk to James * Getting ready to buy a couple of big NGAS hosts from NewTech. Will migrate archive data from older disk arrays.. * 2008-04-22 : purchased newtech machine having 18 TBs, second purchase of another identical machine in a month or so. - NGAS installation script - write a new ngas installation script for the RAID configuration make ngas instation simpler for rookies. - SDM Staging Host (mirror) - we can use the current VLA mirror host eskimo with minimal changes. - SDM Metadata Cataloger - reads the sdm.xml file(s) on the AOC staging mirror and retrieves the desired metadata (not all), and loads it into the archive db tables. Need to do : - get copies of the Java SDM classes, write up an SDM.xml writer which uses the filled ASDM tables I have stashed in Oracle. I want to be able to write a few sdm.xml's and look at them. Then write the sdm.xml parser, and db loader. (Will work with Rich Moeser).. - with a little luck this will be pretty straightforward. My greatest concern is using the ALMA generated classes. Remember they defined some of their own data-types in the ASDM. Rich Moeser and I will be the first real users of the ASDM... - currently (2008-01-07) ALMA in Europe is going to build SDM classes for us. This may happen in 2008-02. They currently have a set of ASDM classes where all variables are strings. Not very useful, probably not worth me experimenting with those classes.. * SDM .xsd's acquired from ALMA. Rich built a maven project area, and created a set of java classes (jaxb). * 2008-04-22 : Rich M has written a prototype MCAF and also written a wrapper for the SDM jaxb classes. I am beginning to write the SDM Cataloger. - Archive Loading Controller - this process determines when a set of sdm.xml + BDF's are complete and the files are closed and ready to copy into the archive. The 'ALC' will either recognize a state posted at the AOC or receive a signal. The ALC will activate the SDM Cataloger and cause the NGAS system to retrieve the BDF's. Need to do : - design a state or signal that indicates which sdm.xml and BDF's are available. Work with Martin and Rich. - Test the BDF download request that would be sent to the CBE. The NGAS system is designed to pull files into itself. Should be simple to modify it to directly request sdm.xml's and BDF's. - Invent a file naming conventions for sdm.xml and BDF's. All files that are described by a sdm.xml should have some sort of commonality in their names. (Martin, Rich, John) - The ALC could be a fairly simple script, needs to communicate with NGAS thru http. (Python would be nice). - Archive Queries, data retrieval - The current archive query tool and db will retrieve data from NGAS modules. I have quite a bit of VLBA data stored in an NGAS module with metadata in the archive db. It can be browsed and data files retreived, the user doesn't know the data is stored in NGAS. The archive db table schema was originally designed to support widar data configurations. So hopefully, modifications to the archive db schema (metadata0 will be minimal. Need to do : - small items concerning user downloading sdm.xml + BDF's from archive. I can only think of fairly minor changes.. - the archive db schema may require some minor tweaks, so I want to get some representative sdm.xml files early. I guess this will be part of the WIDAR prototype test phase. Ordered to do list ------------------------------------------------------------------------ High Priority : NGAS : finalize NRAO NGAS configuration (JB + Andreas + James) NGAS : modify ngams to support NRAO configuration - RAID friendly (JB) SDM : build sdm.xml writer and reader - maybe use ALMA classes with string variables, use ASDM tables from VLBA data (JB) SDM : build sdm -> db loader (JB + Rich) Medium Priority : NGAS : make new ngams installation script (JB) ALC : design files ready state or signals (Martin + Rich + JB) ref Andreas W for how ESO does this. Lower Priority : NGAS : investigate power down (sleep) modes for NGAS modules (James + JB)