JMB  to do list : updated 2011-06-17

1st priority ------------------------

- automated archive content checkers.

     Check that SDM sets from 'today' are cataloged okay.
     /home/aocngas-master/e2e/code/shell/evla/checkSDMcataloged

     Check that the expected bdfs are actually in the archive
     /home/pecos3/jbenson/code/shell/runArchiveFileValidityChecker

     Recognizes severe errors.

  (in operation, needs to be checked, upgraded)

     Modify AAT server to high-light file sets in query response page.
     Modify checker processes to clear error conditions when a lost file 
     is archived.

  (awaiting next AAT servlet release)


- modify NGAS to deliver files directly to destination direectory. 

  Currently the delivered files are streamed back to aocngas-master via http, then 
  aocngas-master writes the stream to the destination file.
  Can do this with sdm/bin/bdf processing plugins for NGAS.
  modified ngamsCClient to accept processingPars,
  built new plugin to physically do the copy : ngamsExportEVLABDFDppi.py

  (needs thorough testing by JMB).

- get a profiler running on archive.war/tomcat. find the memory leaks.

  (no useful progress made)


- Postgres switch over

    Start experimenting with Postgres for AAT and NGAS
    Installed psycopg2-2.4 for python postgre connections.

    (haven't done much)



- NGAS clone hosts in CV

    Set up subscription to update the clone hosts in CV

    Modify AAT to deliver data from CV NGAS hosts when requested

  (waiting on internet connection speed)


- Check the file checksums in the NGAS system.

    Wrote python/archiveOracle/directedNGASchecksums.py. Recalc's checksums in 
    specified slot_id..

- NGAS - ngas won't reliably advance to next available storage partition when the current
  partition fills.. NGAS checks the free space on a disk partition AFTER it has
  written a file. If space < cutoff, it sets complete=1 in db.. No checking before writing.
  So I reset the FreeSpaceDiskChangeMb="2000" and MinFreeSpaceWarningMb="4000"

  (need to contact Andreas)


- document data ingest, delivery
  (in progress)

2nd priority ------------------------

- web page gui that displays ngas server status, data ingest processes and queues
  maybe buttons to shutdown and restart ngas programs

  (thinking about this item, really would like to have it)

- VLA and VLBA access keys for the same project may be different. 
  (have to think about this)

- Add GBT observing metadata to archive db. 
     Will allow aat queries for GBT. GBT data files will remain in CV.
     GBT (person) will supply meta-data in format friendly to archive db tables.

  (waiting on GBT for meta-data files, may be a long wait)

- design archive db table schema for calibrated/image file sets (for VO and EVLA)
    - define file set naming scheme
    - test whether can use ngas_file_sets for calibrated files sets.. need to preserve 
      the original file names, rename files according to ALMA-EVLA standards..
      evla_cal_X1234567890.log, etc.
    - design new meta-data table for data products set description
    - will use meta data definitions from VO - supplied use case file set to Doug. 
      Should have meta data definitions bu May 01.

  (working on this now and then)

In User Testing ---------------------

- build tool for shipping HD archive files

  user required to be logged in (my.nrao acct)
  user has unlocked proprietory files

  (in user testing.)

- AAT to support delivering a set of scans as specified by the user.

    - have modified getSDMFileSet.py, sdmToMSScript.py in archiveDelivery
      waiting on importevla2 in casapy-test to be part of upgrades 
      to casapy (importevla)
    - am running e2eftp.cgi.2.6.6 which places scan number list in getSDMFileSet
      args list.
    - am running getSDMFileSet.py.1.5.9 which recognizes scans in arch list and
      if list not equal to ALL, calls $pecos4/sdmToMSSript.py, else older version
  
  (in user testing)

- change proprietary unlocking, exposed in html, vunerable to editing..
  (user testing)

- getSDMFileSet.py, sdmtoMSScript.py, sdmToFitsScript.py .... read sw ids from ms.summary
  (testing)

- update archive servlet software :
    prune out useless code in authentication (tickets?)
   (in next update)

- build/test new server, ArchiveDeliver. 

  Will support download requests from automated processes such as VO or automated
  pipelines (VLBAPipeline). Supports public domain data files only.

  (completed, in operation)