Start up and run several ngamsServer daemoons on a single host

2009-07-24  JMB

-- what do I know 

   ngamsServer has :
   -multipleSrvs:           If specified, it will be possible to run multiple
                           servers on one node. In this mode, the Host ID in
                           the NGAS Hosts DB table is "<Host Name>:<Port No>"
                           rather than just the hostname.
    ngamsServer.conf :
    <Server ArchiveName="NRAO-ARCHIVE"
          BlockSize="65536"
          MaxSimReqs="30"
          RootDirectory="/NGAS"
          VolumeDirectory="volumes"
          PortNo="7777"  ---------------  Port Number
          ProxyMode="1"
          Simulation="0"
          SwVersion=""/>


-- Try something, try to run one ngas server using a different port number .....

2009-07-28   JMB

setup NGAS on new host, aocngas-1, use this machine for multiple daemons

- create nagmsServer7777.conf, ngamsServer7778.conf

  <Server ArchiveName="NRAO-ARCHIVE"
          BlockSize="65536"
          MaxSimReqs="30"
          RootDirectory="/NGAS"
          VolumeDirectory="volumes"
          PortNo="7777"
          ProxyMode="1"
          Simulation="0"
          SwVersion=""/>

- modify ngas_hosts db :

update ngas_hosts set srv_port = '7778' where host_id = 'aocngas-1';
update ngas_hosts set ip_address = '10.64.1.201' where host_id = 'aocngas-1';
update ngas_hosts set host_id = 'aocngas-1:7778' where ip_address = '10.64.1.201';
update ngas_hosts set mac_address = '00:30:48:D2:1C:10', cluster_name='aocngas-master', ngas_type='NAU', n_slots=9 where host_id = 'aocngas-1:7777';
commit;


  <Log LocalLogFile="/NGAS/log/7777/LogFile.nglog"
       LocalLogLevel="4"
       LogBufferSize="0"
 
- start up ngasServer(s)

/opsw/packages/bin/ngamsServer -cfg /opsw/packages/cfg/ngamsServer7778.conf -autoOnline -force -multipleSrvs &
/opsw/packages/bin/ngamsServer -cfg /opsw/packages/cfg/ngamsServer7777.conf -autoOnline -force -multipleSrvs &
 
- This seems to work. Status checks return SUCCESS, ONLINE. Two separate logfiles created.
http://aocngas-1.aoc.nrao.edu:7778/STATUS
http://aocngas-1.aoc.nrao.edu:7777/STATUS

-----------------------------------------------------------------------------------------------------
2009-07-28 JMB

Test this with archiving two different files...

VLBA_VSN009999_file2_test7777.uvfits
VLBA_VSN009999_file2_test7778.uvfits

aocngas-1.ngas$ ngamsCClient -host aocngas-1 -port 7777 -cmd ARCHIVE -fileUri /home/ngas_staging/tests/VLBA_VSN009999_file2_test7777.uvfits

Status of request:

Host:           aocngas-1:7777
Port:           7777
Command:        ARCHIVE

Date:           2009-07-28T12:56:24.662
Error Code:     0
Host ID:        aocngas-1:7777
Message:        Successfully handled Archive Push Request for data file with URI: VLBA_VSN009999_file2_test7777.uvfits
Status:         SUCCESS
State:          ONLINE
Sub-State:      IDLE
NG/AMS Version: v3.2-ALMA/2007-02-26T17:00:00

ngamsCClient -host aocngas-1 -port 7778 -cmd ARCHIVE -fileUri /home/ngas_staging/tests/VLBA_VSN009999_file2_test7778.uvfits

Status of request:

Host:           aocngas-1:7778
Port:           7778
Command:        ARCHIVE

Date:           2009-07-28T13:08:02.639
Error Code:     0
Host ID:        aocngas-1:7778
Message:        Successfully handled Archive Push Request for data file with URI: VLBA_VSN009999_file2_test7778.uvfits
Status:         SUCCESS
State:          ONLINE
Sub-State:      IDLE
NG/AMS Version: v3.2-ALMA/2007-02-26T17:00:00


** Appears to work...

Setup :aocngas-1:7777 -> Volume01 - Volume05
       aocngas-1:7778 -> Volume06 - Volume10

test files :

/home/ngas_staging/tests/VLBA_VSN009999_file1_test7777a.uvfits
/home/ngas_staging/tests/VLBA_VSN009999_file1_test7778a.uvfits
572MB

aocngas-1.ngas$ date;ngamsCClient -host aocngas-1 -port 7777 -cmd ARCHIVE -fileUri /home/ngas_staging/tests/VLBA_VSN009999_file1_test7777a.uvfits;date
Wed Jul 29 12:56:33 MDT 2009

Status of request:

Host:           aocngas-1:7777
Port:           7777
Command:        ARCHIVE

Date:           2009-07-29T12:56:44.968
Error Code:     0
Host ID:        aocngas-1:7777
Message:        Successfully handled Archive Push Request for data file with URI: VLBA_VSN009999_file1_test7777a.uvfits
Status:         SUCCESS
State:          ONLINE
Sub-State:      IDLE
NG/AMS Version: v3.2-ALMA/2007-02-26T17:00:00

Wed Jul 29 12:56:45 MDT 2009

************ 12 secs 47.5 MB/sec

aocngas-1.ngas$ date;ngamsCClient -host aocngas-1 -port 7778 -cmd ARCHIVE -fileUri /home/ngas_staging/tests/VLBA_VSN009999_file1_test7778a.uvfits;date
Wed Jul 29 12:57:27 MDT 2009

Status of request:

Host:           aocngas-1:7778
Port:           7778
Command:        ARCHIVE

Date:           2009-07-29T12:57:40.020
Error Code:     0
Host ID:        aocngas-1:7778
Message:        Successfully handled Archive Push Request for data file with URI: VLBA_VSN009999_file1_test7778a.uvfits
Status:         SUCCESS
State:          ONLINE
Sub-State:      IDLE
NG/AMS Version: v3.2-ALMA/2007-02-26T17:00:00

Wed Jul 29 12:57:40 MDT 2009

************ 13 secs 43.8 MB/sec

Approximately simultaneous ingests 

aocngas-1:7777 : Wed Jul 29 13:01:10 MDT 2009 - Wed Jul 29 13:01:29 MDT 2009 (19 sec)
aocngas-1:7778 : Wed Jul 29 13:01:14 MDT 2009 - Wed Jul 29 13:01:28 MDT 2009 (14 sec)

2 * 570 MB / 19sec + 60 MB/sec

-----------------------------------------------------------------------------------------------
New speed test files :
/home/ngas_staging/tests/speedtest_7777a.uvfits
/home/ngas_staging/tests/speedtest_7778a.uvfits

raw cp time 45 secs; file size 2.07 GB

aocngas-1:7777 : Wed Jul 29 13:18:10 MDT 2009 - Wed Jul 29 13:19:25 MDT 2009 (75 sec) 
aocngas-1:7778 : Wed Jul 29 13:18:12 MDT 2009 - Wed Jul 29 13:19:25 MDT 2009 (73 sec) 57 MB/sec

solo copy : 
aocngas-1:7777 : Wed Jul 29 13:24:48 MDT 2009 - Wed Jul 29 13:25:35 MDT 2009 (47 sec) 44 MB/sec

-----------------------------------------------------------------------------------------------
Write two files to the same ngas daemon

/home/ngas_staging/tests/speedtest_7777x1.uvfits
/home/ngas_staging/tests/speedtest_7777x2.uvfits

aocngas-1:7777 (1): Wed Jul 29 13:41:21 MDT 2009 - Wed Jul 29 13:43:33 MDT 2009 (132 sec)
aocngas-1:7777 (2): Wed Jul 29 13:41:22 MDT 2009 - Wed Jul 29 13:43:34 MDT 2009 (132 sec)

-----------------------------------------------------------------------------------------------
Rerun this test with new LogFile.nglog's for comparison :
/home/ngas_staging/tests/speedtest_7777b.uvfits
/home/ngas_staging/tests/speedtest_7778b.uvfits

aocngas-1:7777 : Wed Jul 29 13:56:20 MDT 2009 - Wed Jul 29 13:57:36 MDT 2009 (76 sec)
aocngas-1:7778 : Wed Jul 29 13:56:21 MDT 2009 - Wed Jul 29 13:57:36 MDT 2009 (75 sec)

2009-07-29T13:56:21.556 [INFO] Handling HTTP request: client_address=('10.64.1.201', 41260) - method=POST - path=|ARCHIVE| - content-length=20679321
60 - content-type=ngas/archive-request - content-disposition=attachment; filename="/home/ngas_staging/tests/speedtest_7778b.uvfits"; no_versioning="
0"; wait="1"; time_out="-1" - user-agent=NG/AMS C-API [ngamsServer:handleHttpRequest:1266:26575:Thread-5]

2009-07-29T13:56:21.638 [INFO] Archive Push Request - Data size: 2067932160 [ngamsHighLevelLib.py:saveFromHttpToFile:463:26575:Thread-5]

copy file to staging area ::
2009-07-29T13:57:16.181 [INFO] Saved data in file: /NGAS/volumes/Volume06/staging/NGAMS_TMP_FILE___2009-07-29T13:56:21.635-18-speedtest_7778b.uvfits
. Bytes received: 2067932160. Time: 54.545 s. Rate: 37912735.76 Bytes/s [ngamsHighLevelLib.py:saveFromHttpToFile:488:26575:Thread-5] 

38 MB/sec 

create checksum ::

2009-07-29T13:57:16.957 [INFO] Invoking Checksum Plug-In: ngamsGenCrc32 to handle file: /NGAS/volumes/Volume06/data/2009-07-29/1/speedtest_7778b.uvf
its [ngamsArchiveUtils.py:postFileRecepHandling:347:26575:Thread-5]
2009-07-29T13:57:34.944 [INFO] Invoked Checksum Plug-In: ngamsGenCrc32 to handle file: /NGAS/volumes/Volume06/data/2009-07-29/1/speedtest_7778b.uvfits. Result: -1673317199 [ngamsArchiveUtils.py:postFileRecepHandling:352:26575:Thread-5]

checksum adds 16 secs -> 71 secs

finish up -> 75 secs