Data path and Processes for VLBA-DIFX Data Ingest into NGAS

JMB 2009-Sept-13
JMB 2010-Mar-03

On host: aocngas-master -------------------------------------------------------------------------------------

Cronjob polls at xxx hour intervals 

   /home/pecos3/jbenson/code/shell/validityCheckVLBADIFX

   checks and repairs  VLBA DIFX FITS-IDI files before they are archived.

       Loop through all difx volume directories

       if not .chkd, skip that file set directory

       currently there is very little checking, set directory .chkd


   /home/pecos3/jbenson/code/shell/moveVLBADIFXtoQueue /home/ngas_staging ALL

   finds new DIFX file sets in staging area :

       cp  files to :
       /home/ngas_staging/NGAMS_ARCHIVE_CLIENT

       Make entityUID in NGAS_FILE_SETS table for each file .

           echo "insert into NGAS_FILE_SETS "  >> $lockfilename;
           echo "(file_id, file_version, file_set_id, entity_type_name, entity_id) " >> $lockfilename;
           echo "values " >> $lockfilename;
           echo "('"$difxfile"', '1','"$difx_fileset"','"$entityType"','"$entityUid"'); "  >> $lockfilename;

       change set directory to .loaded

   /home/pecos3/jbenson/code/shell/updateCatalogVLBADIFXtoNGAS
   reads, catalogs files in /home/ngas_staging/NGAMS_ARCHIVE_CLIENT/archived

   /home/aocngas-master/e2e/code/shell/verifyVLBADIFXCataloged ALL