Data path and Processes for VLBA Archive Tapes Data Ingest into NGAS

JMB 2010-Mar-01

At aocngas-master.aoc.nrao.edu -------------------------------------------------------------------------------------

Data analysts copy archive dats onto :

/home/ngas_staging/tapes/VSNname/files

VSN002113/
VLBA_VSN002113_file1.uvfits



run : /home/aocngas-master/e2e/code/shell/validityCheckVLBAFITS

      tests : is first record a FITS record

                  if not, it fixes the file, removed the sql insert statement stuck on the front of the file

               is first record a standard FITS format header

                  if not: rename this file VLBA_VSN002113_file1.uvfits.bad
                  else  : rename this file VLBA_VSN002113_file1.uvfits.chkd

run : /home/aocngas-master/e2e/code/shell/moveVLBAFITStoQueue

      looking for .chkd files in this a directory

      copy files to /home/ngas_staging/NGAMS_ARCHIVE_CLIENT/queue
               VLBA_VSN002113_file1.uvfits.chkd -> VLBA_VSN002113_file1.uvfits   

      rename staged file to .loaded
               VLBA_VSN002113_file1.uvfits.chkd -> VLBA_VSN002113_file1.uvfits.loaded


run : /home/aocngas-master/e2e/code/shell/moveVLBAFITStoQueue

      copies .chkd files to ../NGAMS_ARCHIVE_CLIENT/queue (without the .chkd)

      rename files in /home/ngas_staging/tapes/VSNname/file -> .loaded


run : /home/aocngas-master/e2e/code/shell/updateCatalogVLBAFITStoNGAS

      cd to /home/ngas_staging/NGAMS_ARCHIVE_CLIENT/archived

      select files like :  *___VLBA_VSN00*file*.uvfits

      test : is this file in the Oracle nraongas db table? 

             if no, skip to next file

      test : is this file in the Oracle archive db?              
           
             if no, run /home/aocngas-master/e2e/code/cprogs/catalogVLBAFITS
     
             does NOT mark files '.cata'


run :  /home/aocngas-master/e2e/code/shell/cleanUpVLBAFITStapes

       in /home/ngas_staging/tapes : look for VLBA_VSN00*file*.uvfits.loaded

       in /home/ngas_staging/NGAMS_ARCHIVE_CLIENT/archived for matching *$file"*.xml
       if found, grep for 'SUCCESS' the xml file

          if successful, delete /home/ngas_staging/tapes/VSN*/file


run :  /home/aocngas-master/e2e/code/shell/cleanUpVLBAFITSarchived

       in /home/ngas_staging/NGAMS_ARCHIVE_CLIENT/archived
          find files older than 2 days like : *___VLBA_VSN00*file*.uvfits

       is the file loaded in the Oracle Archive db?
          File already cataloged in ARCHIVE db table delete it, delete .xml file too