2011-Jun-16 : JMB - Restarting Archive 


1. Restart the NGAS programs :

   on hosts aocngas-1 - acongas-8
            cv-evla-ngas-1 - cv-evla-ngas-8
            aocngas-master

   login in to each host : ngas@aocngas-1

   aocngas-1.ngas$ /opsw/packages/bin/ngamsServer -cfg /etc/ngamsServer.conf -autoOnline -force & 

   aocngas-1.ngas$ tail -f /NGAS/log/LogFile.nglog


2. Restart the ingest processes :

   login ngas@aocngas-master

   aocngas-master.ngas$ /home/aocngas-master/e2e/code/shell/ingestVLBAFITSintoNGAS /home/ngas_staging aocngas-4 &
   aocngas-master.ngas$ /home/aocngas-master/e2e/code/shell/ingestSDMBDFintoNGAS /home/ngas_staging/1 aocngas-5 &
   aocngas-master.ngas$ /home/aocngas-master/e2e/code/shell/ingestSDMBDFintoNGAS /home/ngas_staging/2 aocngas-6 &
   aocngas-master.ngas$ /home/aocngas-master/e2e/code/shell/ingestSDMBDFintoNGAS /home/ngas_staging/3 aocngas-7 &
   aocngas-master.ngas$ /home/aocngas-master/e2e/code/shell/ingestSDMBDFintoNGAS /home/ngas_staging/4 aocngas-8 &


3. Restart the socket servers :

   login ngas@aocngas-master

   aocngas-master.ngas$ python /home/archive.nrao.edu/bin/ngams_socketserver.py &

   login archmgr@nm-post-1

   nm-post-1.archmgr$ python /home/archive.nrao.edu/bin/archive_socketserver.py &


4. Check lock semaphores hung since power failure :
   
   at /home/archive/e2e/archive/catalogs/NRAO/sqlscripts/autosqlloadcatalog.lock 
   from chewbacca.e2emgr$ /home/archive.nrao.edu/bin/autosqlloadcatalog

   at /home/ngas_staging/1,2,3,4/NGAMS_ARCHIVE_CLIENT/queue/ingestSDMBDFintoNGAS.lock

   at /home/ngas_staging/NGAMS_ARCHIVE_CLIENT/queue/ingestVLBAFITSintoNGAS.lock


5. Check for files that bounced off the ingest processes :

   /home/ngas_staging/1,2,3,4/NGAMS_ARCHIVE_CLIENT/bad

   mv file ../queue