VLBA FITS-IDI Files to Archive DB and NGAS file system ---  JMB 2008 Aug 5


   1. Data files are copied from tapes to a staging area.
      
      Using /users/e2emgr/vlbaload, current staging at /home/aocngas15/VLBAtemp (newtech-5).

      /home/aocngas15/VLBAtemp/tapes/VSN001234/file_1..

      At completion of each file copy, vlbaload renames the file
      mv file_1 VLBA_VSN001234_file1.uvfits

      The vlbaload also writes the first 2880 bytes of the file into a separate
      additional file, hdr_1, etc. These are not archived..


   2. Data files validity check /home/pecos3/jbenson/code/shell/validityCheckVLBAFITS

      - the vlba fits-idi files contain a non-fits block (28800) at the beginning
        of each file. An sql insert statement..

        check if that block is present, if so repair the file.

      - check if the first block is a std fit header.. if not mark the file,
        not repairable..   filename -> filename.bad

      - valid files renamed filename -> filename.chkd

   3. move data files to NGAMS_ARCHIVE_CLIENT/queue /home/pecos3/jbenson/code/shell/moveVLBAFITStoQueue

      - Only files that have passed the validityCheck and are so marked by '.chkd'
        appended to the filename are considered for copying to the 'queue' directory.

      - Files that are successfully copied to the 'queue' directory are remarked
        replacing the '.chkd' suffix with '.loaded'

      - File sizes are compared between the copied and staged files. Not the same throws
        and error message and halts the process.

      - The file copies may be replaced by symbolic links in the future. The current problem
        is that NGAS uses the filesize of the link instead of the actual file.. to be solved.

      - The loaded files may be held in the staging area indefinitely..

      - Files marked .uvfits.loaded may be deleted by running :
        /home/pecos3/jbenson/code/shell/cleanUpVLBAFITStapes

        The script queries the archive db to see if each file has been loaded in the
        archive db before allowing deletion. Generally a file must also be over one
        day old before being deleted.


   4. Load VLBA FITS meta data into db archive, /home/pecos3/jbenson/code/shell/updateCatalogVLBAFITStoNGAS 

      - loads Archive metadata for data files found in the NGAMS_ARCHIVE_CLIENT/archived area.
        File having filename___STATUS.xml companion files that have size greater than  0, and do not
        have any entries already in the archive db, will be processed.. 

      - SQL scripts are generated in $sqlscripts : /home/archive/e2e/archive/catalogs/NRAO/sqlscripts.
        The sql scripts are transmitted to Oracle every 20 minutes. 

      - The datafiles in the NGAMS_ARCHIVE_CLIENT/archived area may be deleted by : 
        /home/pecos3/jbenson/code/shell/cleanUpVLBAFITSarchived. This process runs through the 
        .uvfits files and checks that they have proper entries in the archive db. Their presence
        in the /archived directory accompanied by non-zero filename___STATUS.xml files indicates
        that they have been archived successfully by NGAS. The cleanUpVLBAFITSarchived script
        will delete eligible files. 
 

   5. ngamsArchiveClient removes files in NGAMS_ARCHIVE_CLIENT/archived area
      after n days if told to do so. I currently have that feature disabled. Before files can
      be deleted their metadata must be loaded in the archive db. I use cleanUpVLBAFITSarchived
      to delete NGAMS_ARCHIVE_CLIENT/archived files..

   6. Currenty (2008-Aug-5) I am using an older version of ngamsArchiveClient supplied by Jens.
      The latest version is a multi-threaded process that has a few problems. The old version
      works reliably for file sizes < 2.2GB.

   Steps 1, 3 and 5 write into a log file.........