JMB updated 2011-10-24 to do : ------------------------------------------------------------------------------------------ 1st priority ------------------------ - upgrade AAT query page. user friendly dates, ra, dec - Done, not yet deployed - new authentication scheme in new version. Done, not yet deployed. - Add MS + UVFITS option in the Archive Files page. not done. - HD Shipping Tool. Add separate billing address. not done. - Organize OBSSET filler from VOTable.xml not done - NGAS hosts to PostgreSQL build new nraongas db in postgres. - upgrade /home/aocngas-master/e2e/code/shell/evla/submitHDConversion Needs to download sdmbdfs only, no conversion. - upgrade ArchiveDeliver allow proprietary delivers, authenticat by token, checks against project list (in analysts file somewhere) - new NGAS versions from Andreas Test on newtech-4 retrieved tar file : ngas_buildout_with_python2.7.tar.gz unpack on /users/ngasmgr oh nuts, this looks like a development version, python buildout not what we really want. Check the ngamsPClient and ngamsCClient. I could use those independantly. Need updated version ngamsCClient to support direct file copies.. - automated archive contents checkers. (under test). Check that SDM sets from today are cataloged okay. /home/aocngas-master/e2e/code/shell/evla/checkSDMcataloged Check that the expected bdfs are actually in the archive /home/pecos3/jbenson/code/shell/runArchiveFileValidityChecker Recognizes severe errors. (in operation, needs to be checked) Modify AAT server to high-light file sets in query response page. (in operation) - automated process that crawls through the AAT download log files... Looks for thrown errors and download processes that did not finish,, Sends email. (operating) - web page gui that displays ngas server status, data ingest processes and queues maybe buttons to shutdown and restart ngas programs Rich will build something in the future whose classes amy be used for this purpose.. (thinking about it) - modify NGAS to deliver files directly to destination directory. Currently the delivered files are streamed back to aocngas-master via http, then aocngas-master writes the stream to the destination file. Can do this with sdm/bin/bdf processing plugins for NGAS. modified ngamsCClient to accept processingPars, built new plugin to physically do the copy : ngamsExportEVLABDFDppi.py testing using newtech-4 : direct copy and ngas http copy run at about the same time... $peocs3/code/shell/ngasretrievecmd testing using 9.7 GB file on aocngas-7, ngas transfer 66 MB/sec, cp -f 75 MB/sec the retrieves are ngamsCClient operations from getSDMFiles.py on nm-post-1 A new version of ngas will be available from Andreas. Python 2.7, ugraded clients. Need this to make progress. - get a profiler running on archive.war/tomcat. find the memory leaks. (no useful progress) - build tool for shipping HD archive files user required to be logged in (my.nrao acct) user has unlocked proprietory files ** need to add emails to jbenson/analysts.. currently sends to user only. ** need to send shipping order info to helpdesk process ** need to allow public EVLA data (converted to MS,UVFITS) access to hd ** need to allow VLBA data access to hd . modified shippingCart.jsp : delete button, prettify modified submitHDConversion, picks up errors from CASA, resets existing MS (in production) - build/test new server, ArchiveDeliver. Will support download requests from automated processes such as VO or automated pipelines (VLBAPipeline) Fixed problem with file_set requests, blocks prorpietary data (completed, in operation) - Postgres switch over Start experimenting with Postgres for AAT and NGAS PostgreSQL archivepg, ngaspg installed on c3po. archivepg inserts working except for iarray_type in obseravtion. create table scripts in $pecos3/code/postgreSQL problem ingesting 128 items in iarray_type data type. postgreSQL limit of 100? reset the limit?? postgreSQL inconsistent with Oracle.. Oracle supports iarray_type[128] type, postgreSQL no. PostgreSQL requires defined array in column'sub_desc_id'. Get started building postgreSQl nraongas db Written ngamsPostgreSQL.py -- Need a 32 bit psycopg2 module for NGAS, runs on old 32 bit python NGAS uses python 2.3.2 Daniel Lyons looked into building a 32-bit psycopg module using the 2.3.2.python. Too many obsolete items. We need to swx NGAS over to python 2.4..Many items is site-packages dir. NGAS runs under aoc python 2.4 when I swx in the proper import modules. (2011-08-03) Will get new NGAS version from Andreas, python 2.7... -- can run old NGAS under comp. div. python, connects to postgreSQL okay. -- startup NGAS /users/ngasmgr/bin/ngamsStartup64 - AAT to support delivering a set of scans as specified by the user. - have modified getSDMFileSet.py, sdmToMSScript.py in archiveDelivery waiting on importevla2 in casapy-test to be part of upgrades to casapy (importevla) - am running e2eftp.cgi.2.6.6 which places scan number list in getSDMFileSet args list. - am running getSDMFileSet.py.1.5.9 which recognizes scans in arch list and if list not equal to ALL, calls $pecos4/sdmToMSSript.py, else older version - version that uses CASA importevla to select scans (in production) - design archive db table schema for calibrated/image file sets (for VO and EVLA) - define file set naming scheme - test whether can use ngas_file_sets for calibrated files sets.. need to preserve the original file names, rename files according to ALMA-EVLA standards.. evla_cal_X1234567890.log, etc. - design new meta-data table for data products set description - will use meta data definitions from VO - supplied use case file set to Doug. Should have meta data definitions bu May 01. - Iterating VO Table items with Jarad. Built VOCataloger that create .sql scripts for archive db. (working on this now and then) - Add GBT observing metadata to archive db. Will allow aat queries for GBT. GBT data files will remain in CV. GBT (person) will supply meta-data in format friendly to archive db tables. (waiting on GBT for meta-data files, may be a long wait) - NGAS clone hosts in CV Set up subscription to update the clone hosts in CV Modify AAT to deliver data from CV NGAS hosts when requested (waiting on internet connection speed) - getSDMFileSet.py Is not recognizing ERROR condition during download stage. Need to alert user in email. (done) - NGAS - ngas won't reliably advance to next available storage partition when the current partition fills.. NGAS checks the free space on a disk partition AFTER it has written a file. If space < cutoff, it sets complete=1 in db.. No checking before writing. So I reset the FreeSpaceDiskChangeMb="2000" and MinFreeSpaceWarningMb="4000" (need to contact Andreas) - change proprietary unlocking, exposed in html, vunerable to editing.. updated archive servlet 5.8.9 to use token to look up file lock/unlock/public status (webtest testing) - getSDMFileSet.py, sdmtoMSScript.py, sdmToFitsScript.py .... read sw ids from ms.summary (testing) - VLA and VLBA access keys for the same project may be different. (have to think about this) - update archive servlet software : prune out useless code in authentication (tickets?) (in next update) - document data ingest, delivery (in progress) - upgrade evla log file accessability (not sure what I mean) 2nd priority ------------------------ - check error code matching for VLA - Lorant - setting telescope = VLA, Frazer (????) - fix project tbl query, time ranges - ingest aips fits tables into ngas, ngas_file_set table - getSDMFileSet.py, update with four threads for retrieving sdm/bdfs from four ngas hosts - TAP DB: add more TAP tables in archive db to VO - DQ : find a reasonable set of data quality keywords for archive db - get WIDAR notes (MR) into system as logs (for now). - add get this file button to obs scripts / logs display - fix e2eftp.cgi for old VLBA raw data files (VSN003616/file_3). in AIPS friendly : /home/e2earchive/BD069_xVSN003616F003.fits (and .uvfits) should be : /home/e2earchive/BD069_xVSN003616F003_1 instead are : /home/e2earchive/BD069_xVSN003616F003.fits /home/e2earchive/VLBA_VSN004847_file77.uvfits - write a python script that will retrieve an SDM file set that is a subset of a larger SDM file set. start with a python script that a user can run.. needs to rewrite main, scan, subscan tables . modified SDMQuery to return a list of desired bdf uids.. SDMQuery : javacmd -sdmfile /home/pecos4/jbenson/sdmfiles/WSCloop3_010.55104.963627453704 -sources 0217+738,0157+7443 -timerang 2009-Sep-30T23:10:0/2009-Sep-30T23:59:00 working on producing a new, editted sdm file set.. - look at VO data access protocols for use in casa - archive interface - VLA Config :write py to troll thru VLA archive db and determine aray config, correct exisiting entries.. - VALIDATION: think about evla sdmbdf data set validation prior to archive ingest... (Doug) under test : ------------------------------------------------------------------------------------------ - modify SDMCataloger to write scans, subscans to proper columns (observation: scans, subscan) (done) - modify archive db table schema a bit... add project_id (pds), and sb_id, eb_id added project_id (from pds) to project table. (done) - debug advquery.jsp returns users project list as links to archive file query. (??) - removed project segment query (done) - DB DOC: redo Archive DB Tables doc; use evla sdm/bdf in examples (done) - build gizmo that displays python /home/pecos3/jbenson/code/python/showGetSDMFilesProcs.py to users; from archive query pages.. (??) - start moving cronjob, etc. processes from evlangas-1 to AOC hosts. moved ingestSDMBDFintoNGAS from evlangas-1 to aocngas-4 others on a single host ? (done, cronjobs run as e2emgr@aocngas-master) - SDMCataloger : needs to create merge project table rows sql for this project. look at code/oracle/cprogs10g/compressProjectTbl.pc (wrote compressProjectTblAll.py) - expose casapy logs to users (casapy.log?) : currently being written into /home/e2earhciev/CASAlogs (done, attached to email) - update archive email to users... . different messages for different telescopes . error/failure reports.. . local start/stop date/times (done) - add scan, subscan columns to observation db table - 2011-02-03 done copy elev_min, elev_max to scan, subscan where bdf_file like 'uid%' and starttime > 55500.0 - 2011-02-03 done note : scan/subscan entries start 01 Oct 2010 (done) - upgrade email to users with evla download/convert report. (done) - design/build a new version of 'autoingestBDFintoArchive' that round-robins bdf files into four NGAMS_ARCHIVE_CLIENT queues : 1/NGAMS_ARCHIVE_CLIENT ... 4/NGAMS_ARCHIVE_CLIENT/queue $pecos3/code/shell/autoingestBDFintoArchive modified for 4 ngas hosts. $pecos3/code/shell/autoingestSDMBDFintoArchive modified for 4 ngas hosts. (done) - image tool : hi, i noticed that the tool returns zero images if the requested position is more than the stated search radius from any image center even if the position falls inside a FIRST image which are almost 1 degree across. should the search bring up any image that the requested position falls within? --- Right, need to expand the query for tiled surveys, as opposed to pointed observation images. Used Minimum Bounding Arrays algorithm - D.W. Randolph (done) - getSDMFileSet.py : add capability to spectral average data. That is, include in the this pipeline after an ms is created, use casa to spectral average, then create uvfits file... Use casa split... Not fast. Probably waiting for evla to swx to sdm1.0, and casa to importasdm (1.0).. under test 2009-11-17 on pecos. - move the casa-py importASDM, etc to another computer. Brings gila nearly belly-up Under test 2009-11-17, using dubhe.aoc.nrao.edu Will move to ftp server after memory upgrade - 2010-01-21 - evlangas-1 : modify the sdm/bdf ingestion to independently retrieve the bdfs from Martin's cbe area. create a new directory, /home/evlangas-2/ngas_staging/evla/bdfs. Create sym links to files in /home/cbe-control/wcbe/data, mark files in ./bdfs as loaded when loaded.. Create new set of bdf sym links from py scripts which looks up bdf file names in nraongas ngas_files db... Built createBDFsymlink sh that cross compares bdf's in /home/evlangas-2/ngams_staging/evla/bdfs with those in /home/cbe/etc... and checks in ngags_files table in nraongas db.. - moveSDMBDFtoQueue call parseSDMTableHeader : SAX version of table header parser Have built autoingestBDFintoArchive, testing, it works. Have modified copyBDFtoQueue, copyBDFtoStage Have modified copyBDFtoStage to create a sym link instead of copying the bdf's. - add new column to OBSSUMMARY tbl that contains the string rep. of polarizarion states (RR,LL,etc..) POLAR column exists, fill it.. filled it with sql update in 'testquerys.sql' done : ------------------------------------------------------------------------------------------ - finish copyNGAStoNGAS.py runs cloning aocngas-x to cv-evla-ngas-x - ngamsArchiveClient :continue to fiddle around with NGAMS ngamsArchiveClient, old version 3.1.1 with 64 bit file_bits correction... not working right... received and installed new ngamsCClient from Andreas (2009-09-25), installed on newtech-4, aocngas-2, evlangas-1. Appears to work, ingests > 2.2GB files.. - VLAArchiveSQL.c : Added sql update to enter the VLA array config in the archive table row. Look's it up in the archive:config table (scheduled moves). - SDMCataloger : rewrite SDMCataloger (SDM1.0) to individually unmarshal selected sdm tables. Avoid unmarshalling very big tables for no reason... Done, it works, very fast.. SDMCataloger version 1.21 (fixed average interval calculation) - ingestSDMBDFintoNGAS : written, replaces ngasArchiveClient V4.0/Beta3 which has problems. - getSDMFileSet_sdm1.0.py, sdmToMSScript.py : Uses importasdm in casa-test instead of asdm2MS. - MUNIN : moving munin code/shell and code/cprogs to aocngas-master. Get all checkins current.. - CRONJOBS: clean up cronjobs, especially on aocngas-master - NEWTECH-5: plan for /home/ngas_staging move from newtech-5 to aocngas-1 partition.. - DIFX :ingest DIFX files; > 2GB business, other files type, .txt and .log files.. Installed new ngamsArchiveClient that appears to handle the > 2GB files okay. Testing phase. Upgraded ingestVLBAFITStoNGAS, moveVLBADIFXtoQueue runs on e2emgr@aocngas-master - getSDMFileSet.py : added chmod -R a+rwx for recursive priviledge update.. - loadOBSSUMMARYTbl : added if_chans 'AB','AC','AD','BD' - install NGAS on cv-ngas-evla-1 .. -3, start clone copies.. experiment with CLONE disk to disk use copyNGAStoNGAS.py and CLONE copying aocngas-3 to cv-evla-ngas-3 - retrieve SDMBDF file count in ngas and predicted file count in ngas_file_sets, display in archive file page send link to archiveRouter file_set_info=WSCloop3_010.55104.963627453704 new querytype='FILESETINFO', make query, make results table - columns ngas_file_sets: - finish archive copies to NGAS : archive_VLA munin:/export/home/munin14 - checking archive_VLA2 munin:/export/home/munin3 - checking archive_VLA4 munin:/export/home/munin15 - checking archive_VLA5 munin:/export/home/munin16 - checking archive_VLBA11 munin:/export/home/munin17 - done archive_VLBA12 fitful:/export/home/fitful2 - done archive_VLBA13 fitful:/export/home/fitful3 archive_VLBA14 fitful:/export/home/fitful4 - done archive_VLBA15 fitful:/export/home/fitful5 - done archive_VLBA16 fitful:/export/home/fitful6 archive_VLBA17 fitful:/export/home/fitful7 archive_VLBA18 fitful:/export/home/fitful8 archive_VLBA19 fitful:/export/home/fitful9 archive_GBT fitful:/export/home/fitful10 - delete it, no longer supported