Notes to me concerning the E2E Archive Tables - JMB 
 
    last update 14 Oct 2001


    What values will we need to place in meta-data catalog
    tables for the e2e archive? We need to describe the archive
    files sufficiently well enough to satisfy any possible 
    reasonable query..

    There are several different catagories of archived data files :

    1). raw telescope data, observed data in native archive formats
        - VLA Export tapes,
        - VLBA FITS-IDI,
        - GBT FITS,
        - EVLA-WIDAR (not yet determined)
        - AIPS FITS files (maybe)

    2). calibrated (and flagged) telescope data, data has been through
        AIPS++ pipeline processing, format : AIPS++ MS

    3). images produced by AIPS++ pipleline, AIPS++ Image files

    4). monitor and ancillary data files

    5). scripts and logs


    What meta-data tables would we need for 1)., raw telescope 
    data. In particular, VLA archive data..

    - An Archive Files Table :

        project_code
        observer name
        observer id number
        file  start/stop date_times (internal DB date format)
        telescope name
        telescope configuration
        telescope data format (VLA Export, FITS-IDI, FITS-GBT, EVLA, MS_V2)
        data type (raw, calibrated, etc..)
        arch_file_id
        archive file directory path
        archive file name
        archive file entry date
         

    - An Observations Table :

        project_code
        observation_type (pointing, tipping, mosaic, tracking, ...)
        observation  start/stop date_times (IAT, internal DB date format)
        source_name (user specified name)
        source ra and dec (J2000) (decimal degrees?)
        source_type (star, planet, satellite, asteroid, comet..)
        reference frame (ICRF, ..)
        calib_type (ALMA-like: flux, phase, bandpass, polarization, pointing..)
        arch_file_id
        data_desc_id
        subarray_id
        total exposure (secs)
        integration time interval (secs) - here for now
        uv_range  (uv min, max meters)
        qualifier (stealth column never used except by me)

    - An Antennas Table :

        project_code
        arch_file_id
        antenna_id
        antenna name
        station name
        antenna mount type
        dish diameter
        antenna type 
        axis offset
        coordinate reference frame

    - A Data Description Table :

        project_code
        data_desc_id
        arch_file_id
        row entry date
        baseband channel id  (IF band)
        receiver id (GBT only I think) 
        baseband (IF) reference frequency
        baseband-correlator output polarization (4 values)
        subband channel_id number  
        subband reference freq     
        subband bandwidth          
        subband net sideband
        subband num_lags           
        subband integration intrvl (goes here eventually)
        .
        .


    - An Image Table :

        project_code
        field name (?)
        image_file_id
        arch_file_id 
        data_desc_id
        telescope
        observer
        image creation date      
        observing date
        image type

        pointing image size (pixels)
        pointing center frame
        pointing center values (ra, dec) 
        pointing center pixels
        pointing pixel separations  (delt_ra, delt_dec)
        pointing pixel units 

        freq axis pixels
        freq value
        freq reference pixel
        freq pixel separation

        polarization axis pixels
        polarization axis description

        restoring beam - major axis (value, units)
        restoring beam - minor axis (value, units)
        restoring beam - pos angle  (value, units)

        max flux (Jy/beam)
        min flux (Jy/beam)

         

----------------------------------------------------------------------------

   Questions :

   - E2E archive supports EVLA, VLA, VLBA, GBT... Do we have a
     separate set of catalog tables for each telescope or are they
     all somehow joined together?

   - If the catalog tables can be of identical format for the
     different telescopes, then they can be joined or separate.
     We should strive  to define table formats that all telescopes can use.

   - I think we can find a balance between an extremely detailed
     description of of the archive files (too large and cumbersome),
     and a smaller description which may return a larger set of archive
     files. We can filter the larger set of archive files on the way
     to the output portal, whatever it might be. The catalog tables
     can be designed such that there is no filtration necessary for 
     most queries. Only the more extreme (weirdoo) queries will 
     require filtration..

   - Very soon, I need to identify what meta-data that we will need 
     that is not in an Aips++ MS. Athol will freeze the next release
     7 Nov. Need to get a list to him pronto..

       not AIPS++  array_config   OBS  char[6]   - on VLA filler tape
       not AIPS++  observerid     OBS  int       - on VLA filler tape

       not AIPS++  source_type    char  - 
       not AIPS++  calib_type     char  - 'code' in  SOURCE table

       in AIPS++   observer       in OBS but not filled in by vlafiller  ?


   Design 'Rules' :

   - It should never be necessary to read through all files in the
     archive in order to answer a query. The catalog tables must be
     at least complete enough to identify a located fraction of the 
     archive within which the query can be satisfied. This is the
     scheme of meta-data catalog tables sometimes followed by archive
     data filtering.

     A lot of the archive data formats (AIPS++ MS, VLA Export,
     FITS-IDI) have 'meta-data' tables located at the beginning of 
     the data file. For example, searching for a particular VLBA
     station by reading the actual FITS-IDI files would require 
     only reading through the 'Antenna' tables. I think generally a
     search filter would not have to read through the visibilities.



----------------------------------------------------------------------------