2011-Mar-28 - JMB 

db tables supporting file sets that hold higher level data products..

file types to archive :
   VLBA-PIPE --
   .jpg,.ps.fits,.uvtab,.pickle,.log,.html,.xml,.py


table : ngas_file_sets

        file_set_id       varchar2(128 byte)
        file_id           varchar2(128 byte)
        file_version      number(12,0)
        entity_id         varchar2(48 byte)   like arch_file_id
        entity_type_name  varchar2(32 byte)   defined keywords (new) --
                                              'aips.fittp.uvfits',
                                              'aips.fittp.clfits',
                                              'aips.fittp.imfits',
                                              'aips.run','aips.log',
                                              'fill.log','fill.run',
                                              'move.log','move.run',
                                              'data.log',
                                              'ri.gif','vis.gif',
                                              'vis.gif','uv.gif'
                                              'package.task.file-type'

        format            varchar2(48 byte)   mime-type recognized by NGAS
        file_id_alias     varchar2(128 bytes) user supplied
        native_ext        varchar2(24)        like xml, uvfits, idifits, others

table : file_set_properties
        file_set_id       varchar2(128 byte)
        file_set_id_alias varchar2(128 bytes)
        file_set_catagory varchar2(32 bytes)  'project' or 'position' or 'other'
        catagory_id       varchar2(48 bytes)  '10B-205' or 'J015950.2+002344' or 'something else'
        collection        varchar2(48)        Collection membership ('VLA-PIPE')
        calib_level       number              Calibration level (0,1,2,3)
        starttime         number              mjd.fract
        stoptime          number              mjd.fracr
        telescope
        telescope_config(s)
        obs_bands
        processing_history



example : /home/archive_surveys3/pipeline/position/J015833.5-245932

Lorant uses two file set catagories :

project name - can hold many observations at different positions (sources)

               AB0604 

position     - can hold many observations at the same position different projects.

               J012434.8+034726


File types (Lorant's pipeline) :

   .jpg, .JPG, .imfits, .gif, .uvfits, .vis.gif, .ri.gif, .uv.gif, .log, .run
   .clfits

prototype file_set_id :

   VLA_10B-123_100914T034555            
   VLA_J012434.8+034726_110104T123422

   (second term : project_code or sky_position)
   (which date to use : checkin date/time -- difx style)


Archive Data Files Table -- 'format' column contains a link to a display of 
                            file set files..



--------------------------------------------------------------------------------------------------------------

What needs to be built :

  db tables : 

     file_sets, file_set_properties
      
        status : ready for testing

     ingest set tool -
       
        copies and renames files, submits to NGAS,
        makes entries in file_set, file_set_properties tables

        parses meta-data from uvfits files (aips)
   
           write data miner for aips.uvfits
               
           catalogVLBAAIPS.c 

        parses meta-data from aips.imfits files

           parser exists. catalogImageFITS.c 


  trial run ----

  given a file set like Lorants pipeline data --

     some process copies file sets into /home/ngas_staging/filesets
       
        separate subdirectories per file set. 
        for now my file_set name convention

        test file set - /home/archive_surveys3/pipeline/position/J022716.5+333446

        rename - /home/ngas_staging/filesets/VLA_J022716.5+333446_110121T180633

        subdir name is file_set_id..

        delete file that will not be archived : .shtml, .pos, index*, removed*

        files present .uvfits, .imfits, .ri.gif, .uv.gif, .vis.gif, .jpg, .JPG

        load entries into ngas_file_sets