EVLA SDM-BDF File Sets to Archive DB and NGAS file system ---  JMB 2008 Sept 16


   1. Data files are copied from mcaf and cbe staging areas to an ngas staging area.
      
      ngas_staging staging areas : 

           host : evlangas-1 : /home/evlangas2/ngas_staging/evla.
           host : aocngas-1  : /home/aocngas15/ngas_staging/evla

      logon : e2emgr


      MCAF : delivers completed SDM file sets to /home/mctest/sdm_staging/complete

      CBE  : delivers completed BDF file sets to /......??

      Script copySDMtoStage : copies completed SDM files set to staging area.
                              renames file sets 
                                - change any "_" to "-"
                                - truncate fract of day to 7 digits
                                - delete leading 3 digits in file set name 
                              if ASDMBinary directory doesn't exist, mkdir it
                              script does not delete SDMfiles on MCAF staging area.
                      
                              copy excludes file with names including :
                              'sys', 'delay', 'shift', 'FLUX', 'band'`

                              >copySDMtoStage ngas_staging_directory SDMfile or ALL

      Script copyBDFtoStage : copy BDF's from CBE to staging area 
                              searchs all SDM file sets in ngas_staging/evla for BDFs
                              read BDF file UIDs from SDM Main Table rows
                              copy BDF's to SDM file set ASDMBinary directory
                              script does not delete BDFs on CBE staging area.

                              the copyBDFtoStage script may be called to execution by
                              copySDMtoStage, and/or run independently. It just looks 
                              missing BDF's.

   2. Data files validity check the SDM-BDF files sets in the ngas staging area

      ngas_staging staging areas : 

           host : evlangas-1 : /home/evlangas2/ngas_staging/evla.
           host : aocngas-1  : /home/aocngas15/ngas_staging/evla

      logon : e2emgr

      Script validityCheckSDMBDF : checks the SDM-BDF files sets 
                                   - are there any BDF's in the ASDMBinary directory
                                   - are there matching BDF's for each uid in the Main.xml table 
                                   - does the uid in the BDF header match with the file name
                                   - are there BDF's that don't match any Main.xml rows..

                                   - need to do some sort of SDM content checking..

                                   - valid files renamed filesetname -> filesetname.chkd

   3. Move data files to NGAS input queue directory : 
        
      ngas_staging staging areas : 

           host : evlangas-1 : /home/evlangas2/ngas_staging/evla.
           host : aocngas-1  : /home/aocngas15/ngas_staging/evla

      logon : e2emgr

      NGAMS_ARCHIVE_CLIENT areas :

           host : evlangas-1 : /home/evlangas2/ngas_staging/NGAMS_ARCHIVE_CLIENT
           host : aocngas-1  : /home/aocngas15/ngas_staging/NGAMS_ARCHIVE_CLIENT

        Script moveSDMBDFtoQueue : 

                                  - Copy or move files to the NGAS queue directory :
                                    NGAMS_ARCHIVE_CLIENT/queue

                                  - Only files that have passed the validityCheck and are so marked 
                                    by '.chkd' appended to the filename are considered for copying 
                                    to the 'queue' directory.

                                  - Files that are successfully copied to the 'queue' directory are 
                                    remarked replacing the '.chkd' suffix with '.loaded'

                                  - File sizes are compared between the copied and staged files. Not 
                                    the same throws an error message and halts the process.

                                  - Rename files according to telescope_fileset_filename.xml
                                    for NGAS.

        Remember to use NGAS version with this plugin : ngamsASDMPlugIN.py 

        The file copies may be replaced by symbolic links in the future. The current problem
        is that NGAS uses the filesize of the link instead of the actual file.. to be solved.

        The loaded files may be held in the staging area indefinitely..

   4. Load SDM BDF meta data into db archive

       Script updateCatalogSDMBDFtoNGAS 

      - loads Archive metadata for data files found in the NGAMS_ARCHIVE_CLIENT/archived area.
        File having filename___STATUS.xml companion files that have size greater than  0, and do not
        have any entries already in the archive db, will be processed.. 

      - SQL scripts are generated in $sqlscripts : /home/archive/e2e/archive/catalogs/NRAO/sqlscripts.
        The sql scripts are transmitted to Oracle every 20 minutes. 

      - The datafiles in the NGAMS_ARCHIVE_CLIENT/archived area may be deleted by : 
        /home/pecos3/jbenson/code/shell/cleanUpVLBAFITSarchived. This process runs through the 
        .uvfits files and checks that they have proper entries in the archive db. Their presence
        in the /archived directory accompanied by non-zero filename___STATUS.xml files indicates
        that they have been archived successfully by NGAS. The cleanUpVLBAFITSarchived script
        will delete eligible files. 
 

   5. ngamsArchiveClient removes files in NGAMS_ARCHIVE_CLIENT/archived area
      after n days if told to do so. I currently have that feature disabled. Before files can
      be deleted their metadata must be loaded in the archive db. I use cleanUpVLBAFITSarchived
      to delete NGAMS_ARCHIVE_CLIENT/archived files..

   6. File set clean up.

      - Files marked .loadedxx may be deleted by running :
        /home/pecos3/jbenson/code/shell/cleanUpSDMBFDstage

        The script queries the archive db to see if each file has been loaded in the
        archive db before allowing deletion. Generally a file must also be over one
        day old before being deleted.


   6. Currenty (2008-Aug-5) I am using an older version of ngamsArchiveClient supplied by Jens.
      The latest version is a multi-threaded process that has a few problems. The old version
      works reliably for file sizes < 2.2GB.

   Steps 1, 3 and 5 write into a log file.........