Data path and Processes for EVLA-WIDAR Data Ingest into NGAS

JMB 2009-Sept-01
JMB 2010-Feb-25

At evlangas-1.vla.nrao.edu -------------------------------------------------------------------------------------

cronjob at 10 minute intervals polls the SDM file set staging area:

mcaf_staging = /home/mc2host/evla/sdm  

   /home/evlangas-2/ngas_staging/code/shell/autoingestSDMBDFintoArchive

      found new SDM file set(s) in the mcaf staging area :

      sdmdif in mcaf_staging IS named .bad or .chkd or .loaded or .cata : 
           skip to next sdm set

      Run the transferSDMBDFtoStage process (copys SDM and BDF files to /home/evlangas-2/ngas_staging/evla)

       /home/evlangas-2/ngas_staging/code/shell/transferSDMBDFtoStage $mcafsdmdir

           sdmdif in mcaf_staging IS named .bad or .chkd or .loaded or .cata : 
               skip to next sdm set (which is the end of transferSDMBDFtoStage)
               doesn't commincate the no go to outside world?
               already ran this test in autoingestSDMBDFintoArchive --

            /home/pecos3/jbenson/code/shell/copySDMtoStage $ngas_staging $mcafsdmdir

               modify file set name 

               cp -fr $mcaf_staging"/"$mcafsdmdir $sdmstaging"/"$testfiledir
               mkdir $sdmstaging"/"$testfiledir"/ASDMBinary";

            if error return from copySDMtoStage
               rename mcaf and ngas_staging sdm set .bad.
          
            /home/pecos3/jbenson/code/shell/copyBDFtoStage $ngas_staging $mcafsdmdir;

               read the Main.xml table, reconstruct the bdf file name
               if bdffilename NOT in the cbe bdf staging area :
                   Warning message
                   continue
               else
                   touch $bdffullfilename;
 
            if no bdfs found, 
                mv $mcafsdmdir $mcafsdmdir".bad";
                mv $sdmstaging"/"$mcafsdmdir $sdmstaging"/"$mcafsdmdir".bad";
                continue

            validityCheckSDMBDF $ngas_staging $mcafsdmdir

            if all is okay : 
               mv $mcafsdmdir $mcafsdmdir".chkd";
            if bad :
               mv $mcafsdmdir $mcafsdmdir".bad"



      Run the moveSDMBDFtoQueue process (copies the SDM and BDF's to the NGAMS_ARCHIIVE_CLIENT queue dir)

       /home/evlangas-2/ngas_staging/code/shell/moveSDMBDFtoQueue $ngas_staging $sdmdir 

            rename the sdm.xml, bdf  files to the EVLA-NGAS convention.
            cp sdm files to NGAMS_ARCHIIVE_CLIENT/queue
            insert into NGAS_FILE_SETS archive db table 

            cp bdf files (sym links) to NGAMS_ARCHIIVE_CLIENT/queue
            insert into NGAS_FILE_SETS archive db table 

            makes separate sql file for NGAS_FILE_SETS inserts

            mv $sdmdir $sdmdir.loaded

      Run the updateArchiveCatalogSDMBDF process (runs sdmcataloger, archive db metadata cataloging)

       /home/evlangas-2/ngas_staging/code/shell/updateArchiveCatalogSDMBDF $ngas_staging $sdmdir 

            run SDMCataloger (read SDM.xml files only)
            /home/pecos4/jbenson/code/E2E/trunk/sdmcataloger/javacmd $sdmdir $sqlscriptdir
            insert into ARCHFILELOCATION archive db table

            mv $sdmdir $sdmdir.cata


Clean up staged files once per day :

run : /home/evlangas-2/ngas_staging/code/bin/cleanUpSDMStaging 

      Deletes SDMfile sets on EVLA-MCAF staging area.
      Files of type .bad or .cata are deleted after 5 days..
      No other checking

run : /home/evlangas-2/ngas_staging/code/bin/cleanUpSDMBDFfilesets /home/evlangas-2/ngas_staging ALL

      Deletes SDMfile sets on EVLA ngas_staging staging area.
      Files of type .bad or .cata are deleted after 5 days..
      No other checking   


run : /home/evlangas-2/ngas_staging/code/bin/cleanUpSDMBDFarchived /home/evlangas-2/ngas_staging ALL

      NOT YET : Deletes SDM and BDF files that have been successfully archived by NGAS.
      Files found in /home/evlangas-2/ngas_staging/NGAMS_ARCHIVE_CLIENT/archived over 3 days old


NGAS Server on evlangas-1.vla.nrao.edu -------------------------------------------------------------------------

     NGAS Server is now turned OFF................