Date - 2001.12.14 Tester - S.T. Myers (AOC) Platform - Linux (sandrock) Note: bugs are denoted by lines with prefix: >>>BUG: and queries are denoted by lines with prefix: >>>QUERY: while comments are prefixed by: >>>COMMENT: Some enhancement requests are given after >>>REQUEST: Notes added later for clarification are indicated by >>>NOTE: ------------------------------------------------------------------------------- GOAL: continue development of script for gain curve and opacity correction 1. Date - 2001.12.14 Version - Weekly (1.7 build 047) Directory - /home/sandrock/smyers/Testing/2001-12-12/ Data - 20010204AK.UVF Script - 2001-12-14-myers.g Bits to look up the VLA antenna number (name) # # Set the antenna number lookup # anam := spaste(msf, '/ANTENNA'); atab := table(anam); nant := atab.nrows(); anam := atab.getcol("NAME"); printf(' Found %d antennas in table\n',nant); atab.close(); Then, when antennas are pulled out, do the lookup (should do something about invalid antennas). # # Get AN number vectors # an1v := st.getcol("ANTENNA1")+1; an2v := st.getcol("ANTENNA2")+1; an1 := anam[ an1v ]; an2 := anam[ an2v ]; Note that these columns seem to be of type string. Will we have a problem using an1 and an2? Now fix gain curve code to use this gaincv := function( an1, an2, el, c) { # # Input vectors (same length) an1,an2,el # Assumes gain curve information is stored in array c[3,nant] # Elevation el in radians # Antenna number an1,an2 elements must correspond to columns in c # Example: Kband AN2 IF1 L (Oct00) # c[1:3,2] := [1.0041e+00, -6.0815e-04, 2.2647e-05 ]; # Need to pull out appropriate coefficients array pick g1 := c[1,an1] + c[2,an1]*el + c[3,an1]*el*el; g2 := c[1,an2] + c[2,an2]*el + c[3,an2]*el*el; g := sqrt( g1*g2 ); # Need to sort out abnormally low gains g[ g < 1.e-4 ] := 1.e-4; return g; } NOTE - The statement g[ g < 1.e-4 ] := 1.e-4; is fixed from the 2001-12-12 version. Now need to fill c array. Use the Oct00 gain curves that are found in $AIPS_ROOT/TEXT/IONS/VLA.GAINS the relevant bits of which are ;;----------------------------------------------------------------------- ; gain curve file for the VLA. bjb. 2001jul27. ; the format of the lines looks like: ; bb aa yyy1m1d1 yyy2m2d2 c0 c1 c2 c3 ; where: ; b = band designation (currently one of: 4, P, L, C, X, U, K, Q) ; aa = antenna number. if aa=0, this is an average over all antennas. ; yyyy1 m1 d1 = year, month, day before which this gain curve entry is ; not valid ; yyyy2 m2 d2 = year, month, day after which this gain curve entry is ; not valid ; c0, c1, c2, c3 = gain curve coefficients ; ; the subroutine that reads the lines in handles free format (i.e., ; any number of spaces > 0 between the entries). The date strings ; are 8 consecutive characters with no blanks ; ; lines starting with ';' are comment lines ; ;----------------------------------------------------------------------- U 0 19750101 21000101 9.6350E-01 1.3945E-03 -1.5014E-05 0.0000E+00 U 1 19750101 21000101 1.0213E+00 -1.6223E-04 -1.5142E-05 0.0000E+00 U 2 19750101 21000101 9.9380E-01 5.2939E-04 -1.1844E-05 0.0000E+00 U 3 19750101 21000101 9.5077E-01 1.7396E-03 -1.5404E-05 0.0000E+00 U 4 19750101 21000101 9.3945E-01 1.6803E-03 -9.2158E-06 0.0000E+00 U 5 19750101 21000101 9.6259E-01 1.5072E-03 -1.5226E-05 0.0000E+00 U 6 19750101 21000101 9.7797E-01 9.2114E-04 -9.6542E-06 0.0000E+00 U 7 19750101 21000101 9.5072E-01 1.6461E-03 -1.3873E-05 0.0000E+00 U 8 19750101 21000101 9.6315E-01 1.3102E-03 -1.1699E-05 0.0000E+00 U 9 19750101 21000101 9.8908E-01 7.4323E-04 -1.2785E-05 0.0000E+00 U 10 19750101 21000101 9.4594E-01 1.9071E-03 -1.6902E-05 0.0000E+00 U 11 19750101 21000101 9.7378E-01 1.2212E-03 -1.4299E-05 0.0000E+00 U 12 19750101 21000101 9.2886E-01 2.0099E-03 -1.4326E-05 0.0000E+00 U 13 19750101 21000101 9.7188E-01 1.2060E-03 -1.2958E-05 0.0000E+00 U 14 19750101 21000101 9.5259E-01 1.5981E-03 -1.3701E-05 0.0000E+00 U 15 19750101 21000101 9.9007E-01 7.6713E-04 -1.5014E-05 0.0000E+00 U 16 19750101 21000101 9.3265E-01 2.0155E-03 -1.5228E-05 0.0000E+00 U 17 19750101 21000101 9.4203E-01 2.3229E-03 -2.3322E-05 0.0000E+00 U 18 19750101 21000101 9.8450E-01 1.3243E-03 -2.9131E-05 0.0000E+00 U 19 19750101 21000101 9.4425E-01 2.1415E-03 -2.0617E-05 0.0000E+00 U 20 19750101 21000101 9.3657E-01 2.9488E-03 -3.4398E-05 0.0000E+00 U 21 19750101 21000101 9.9317E-01 5.8221E-04 -1.4247E-05 0.0000E+00 U 22 19750101 21000101 9.4996E-01 1.7295E-03 -1.5152E-05 0.0000E+00 U 23 19750101 21000101 9.7844E-01 9.0703E-04 -9.8916E-06 0.0000E+00 U 24 19750101 21000101 9.7704E-01 8.9316E-04 -9.0036E-06 0.0000E+00 U 25 19750101 21000101 9.1715E-01 2.1045E-03 -1.3433E-05 0.0000E+00 U 26 19750101 21000101 9.9063E-01 4.9646E-04 -6.6641E-06 0.0000E+00 U 27 19750101 21000101 9.5552E-01 1.6396E-03 -1.5122E-05 0.0000E+00 U 28 19750101 21000101 9.8287E-01 8.9881E-04 -1.1829E-05 0.0000E+00 U 29 19750101 21000101 1.0000E+00 6.9500E-04 0.0000E+00 0.0000E+00 ;----------------------------------------------------------------------- K 0 19750101 21000101 9.4020E-01 2.4481E-03 -2.9080E-05 0.0000E+00 K 1 19750101 21000101 1.0645E+00 -2.1800E-03 1.9091E-06 0.0000E+00 K 2 19750101 21000101 9.9400E-01 7.1267E-04 -2.1991E-05 0.0000E+00 K 3 19750101 21000101 9.0775E-01 3.4519E-03 -3.2439E-05 0.0000E+00 K 4 19750101 21000101 8.7106E-01 4.4819E-03 -3.9123E-05 0.0000E+00 K 5 19750101 21000101 9.5978E-01 2.0537E-03 -2.6473E-05 0.0000E+00 K 6 19750101 21000101 9.6549E-01 1.7126E-03 -2.1309E-05 0.0000E+00 K 7 19750101 21000101 9.0808E-01 3.2659E-03 -2.9129E-05 0.0000E+00 K 8 19750101 21000101 9.2997E-01 2.7050E-03 -2.6238E-05 0.0000E+00 K 9 19750101 21000101 9.4561E-01 2.4109E-03 -2.7114E-05 0.0000E+00 K 10 19750101 21000101 9.3171E-01 2.4068E-03 -2.1307E-05 0.0000E+00 K 11 19750101 21000101 9.9932E-01 3.5149E-04 -2.0688E-05 0.0000E+00 K 12 19750101 21000101 8.4878E-01 4.8514E-03 -3.9183E-05 0.0000E+00 K 13 19750101 21000101 9.1357E-01 2.8711E-03 -2.4172E-05 0.0000E+00 K 14 19750101 21000101 8.9678E-01 4.0089E-03 -3.9249E-05 0.0000E+00 K 15 19750101 21000101 9.8260E-01 1.4266E-03 -3.0393E-05 0.0000E+00 K 16 19750101 21000101 8.7698E-01 4.1427E-03 -3.5156E-05 0.0000E+00 K 17 19750101 21000101 8.5035E-01 6.7215E-03 -7.5902E-05 0.0000E+00 K 18 19750101 21000101 9.9495E-01 6.8603E-04 -2.4564E-05 0.0000E+00 K 19 19750101 21000101 9.7246E-01 1.7074E-03 -2.6990E-05 0.0000E+00 K 20 19750101 21000101 9.3173E-01 3.3235E-03 -4.0633E-05 0.0000E+00 K 21 19750101 21000101 1.0011E+00 2.3808E-04 -2.2524E-05 0.0000E+00 K 22 19750101 21000101 9.1979E-01 3.0976E-03 -3.0121E-05 0.0000E+00 K 23 19750101 21000101 9.8703E-01 9.1300E-04 -1.6280E-05 0.0000E+00 K 24 19750101 21000101 9.9937E-01 2.3530E-04 -1.4748E-05 0.0000E+00 K 25 19750101 21000101 8.5658E-01 4.8396E-03 -4.1113E-05 0.0000E+00 K 26 19750101 21000101 9.7852E-01 8.5349E-04 -8.7236E-06 0.0000E+00 K 27 19750101 21000101 9.2672E-01 3.0128E-03 -3.1142E-05 0.0000E+00 K 28 19750101 21000101 9.4566E-01 2.9426E-03 -4.0201E-05 0.0000E+00 K 29 19750101 21000101 1.0000E+00 1.3900E-03 0.0000E+00 0.0000E+00 ;----------------------------------------------------------------------- ; need the new gain curves for Q-band since the subreflector ; rotation has been turned on (around june 1, 2001?) Q 0 19750101 21000101 8.5390E-01 6.9920E-03 -9.7635E-05 0.0000E+00 Q 3 19750101 21000101 8.2699E-01 8.6734E-03 -1.1053E-04 0.0000E+00 Q 4 19750101 21000101 8.5282E-01 6.9641E-03 -8.3056E-05 0.0000E+00 Q 5 19750101 21000101 7.8219E-01 6.6175E-03 -4.7822E-05 0.0000E+00 Q 6 19750101 21000101 8.6940E-01 7.1348E-03 -1.0007E-04 0.0000E+00 Q 8 19750101 21000101 7.9055E-01 1.0010E-02 -1.2145E-04 0.0000E+00 Q 10 19750101 21000101 7.9209E-01 7.7903E-03 -7.4262E-05 0.0000E+00 Q 11 19750101 21000101 8.6475E-01 3.7786E-03 -6.3711E-05 0.0000E+00 Q 12 19750101 21000101 7.6261E-01 1.1277E-02 -1.3603E-04 0.0000E+00 Q 13 19750101 21000101 9.5367E-01 3.4600E-03 -6.8573E-05 0.0000E+00 Q 14 19750101 21000101 8.2851E-01 9.2871E-03 -1.3003E-04 0.0000E+00 Q 16 19750101 21000101 7.8560E-01 1.0037E-02 -1.1899E-04 0.0000E+00 Q 18 19750101 21000101 8.7915E-01 8.0377E-03 -1.7129E-04 0.0000E+00 Q 19 19750101 21000101 8.7289E-01 7.1793E-03 -1.0484E-04 0.0000E+00 Q 20 19750101 21000101 7.8001E-01 9.2160E-03 -9.7229E-05 0.0000E+00 Q 21 19750101 21000101 1.2366E+00 -6.5267E-03 -2.5926E-05 0.0000E+00 Q 22 19750101 21000101 7.5337E-01 1.1281E-02 -1.3046E-04 0.0000E+00 Q 24 19750101 21000101 1.0165E+00 -7.3935E-04 -3.9395E-05 0.0000E+00 Q 25 19750101 21000101 6.7155E-01 1.3864E-02 -1.4794E-04 0.0000E+00 Q 26 19750101 21000101 1.0983E+00 -3.1293E-03 -3.4942E-05 0.0000E+00 Q 27 19750101 21000101 7.9194E-01 9.6426E-03 -1.1291E-04 0.0000E+00 Q 28 19750101 21000101 9.1300E-01 5.5659E-03 -9.4216E-05 0.0000E+00 Q 29 19961218 21000101 5.8500E-01 1.1650E-02 6.4210E-05 -2.5900E-06 Make this into a fill function vlagainfill := function( band ) { # fill array as vector if ( band == 'K' | band == 'k' ) { c := [ 1.0645E+00, -2.1800E-03, 1.9091E-06, # AN 1 9.9400E-01, 7.1267E-04, -2.1991E-05, # AN 2 9.0775E-01, 3.4519E-03, -3.2439E-05, # AN 3 8.7106E-01, 4.4819E-03, -3.9123E-05, # AN 4 9.5978E-01, 2.0537E-03, -2.6473E-05, # AN 5 9.6549E-01, 1.7126E-03, -2.1309E-05, # AN 6 9.0808E-01, 3.2659E-03, -2.9129E-05, # AN 7 9.2997E-01, 2.7050E-03, -2.6238E-05, # AN 8 9.4561E-01, 2.4109E-03, -2.7114E-05, # AN 9 9.3171E-01, 2.4068E-03, -2.1307E-05, # AN 10 9.9932E-01, 3.5149E-04, -2.0688E-05, # AN 11 8.4878E-01, 4.8514E-03, -3.9183E-05, # AN 12 9.1357E-01, 2.8711E-03, -2.4172E-05, # AN 13 8.9678E-01, 4.0089E-03, -3.9249E-05, # AN 14 9.8260E-01, 1.4266E-03, -3.0393E-05, # AN 15 8.7698E-01, 4.1427E-03, -3.5156E-05, # AN 16 8.5035E-01, 6.7215E-03, -7.5902E-05, # AN 17 9.9495E-01, 6.8603E-04, -2.4564E-05, # AN 18 9.7246E-01, 1.7074E-03, -2.6990E-05, # AN 19 9.3173E-01, 3.3235E-03, -4.0633E-05, # AN 20 1.0011E+00, 2.3808E-04, -2.2524E-05, # AN 21 9.1979E-01, 3.0976E-03, -3.0121E-05, # AN 22 9.8703E-01, 9.1300E-04, -1.6280E-05, # AN 23 9.9937E-01, 2.3530E-04, -1.4748E-05, # AN 24 8.5658E-01, 4.8396E-03, -4.1113E-05, # AN 25 9.7852E-01, 8.5349E-04, -8.7236E-06, # AN 26 9.2672E-01, 3.0128E-03, -3.1142E-05, # AN 27 9.4566E-01, 2.9426E-03, -4.0201E-05, # AN 28 1.0000E+00, 1.3900E-03, 0.0000E+00, # AN 29 9.4020E-01, 2.4481E-03, -2.9080E-05 ] # AN 30 = avg else { if ( band == 'Q' | band == 'q' ) { c := [ 8.5390E-01, 6.9920E-03, -9.7635E-05, # AN 1 na = avg 8.5390E-01, 6.9920E-03, -9.7635E-05, # AN 2 na = avg 8.2699E-01, 8.6734E-03, -1.1053E-04, # AN 3 8.5282E-01, 6.9641E-03, -8.3056E-05, # AN 4 7.8219E-01, 6.6175E-03, -4.7822E-05, # AN 5 8.6940E-01, 7.1348E-03, -1.0007E-04, # AN 6 1.0000E+00, 0.0000E+00, 0.0000E+00, # AN 7 no rx 7.9055E-01, 1.0010E-02, -1.2145E-04, # AN 8 1.0000E+00, 0.0000E+00, 0.0000E+00, # AN 9 no rx 7.9209E-01, 7.7903E-03, -7.4262E-05, # AN 10 8.6475E-01, 3.7786E-03, -6.3711E-05, # AN 11 7.6261E-01, 1.1277E-02, -1.3603E-04, # AN 12 9.5367E-01, 3.4600E-03, -6.8573E-05, # AN 13 8.2851E-01, 9.2871E-03, -1.3003E-04, # AN 14 1.0000E+00, 0.0000E+00, 0.0000E+00, # AN 15 no rx 7.8560E-01, 1.0037E-02, -1.1899E-04, # AN 16 8.5390E-01, 6.9920E-03, -9.7635E-05, # AN 17 na = avg 8.7915E-01, 8.0377E-03, -1.7129E-04, # AN 18 8.7289E-01, 7.1793E-03, -1.0484E-04, # AN 19 7.8001E-01, 9.2160E-03, -9.7229E-05, # AN 20 1.2366E+00, -6.5267E-03, -2.5926E-05, # AN 21 7.5337E-01, 1.1281E-02, -1.3046E-04, # AN 22 8.5390E-01, 6.9920E-03, -9.7635E-05, # AN 23 na = avg 1.0165E+00, -7.3935E-04, -3.9395E-05, # AN 24 6.7155E-01, 1.3864E-02, -1.4794E-04, # AN 25 1.0983E+00, -3.1293E-03, -3.4942E-05, # AN 26 7.9194E-01, 9.6426E-03, -1.1291E-04, # AN 27 9.1300E-01, 5.5659E-03, -9.4216E-05, # AN 28 5.8500E-01, 1.1650E-02, 6.4210E-05, # AN 29 8.5390E-01, 6.9920E-03, -9.7635E-05 ] # AN 30 = avg } else { if ( band == 'U' | band == 'u' ) { c := [ 1.0213E+00, -1.6223E-04, -1.5142E-05, # AN 1 9.9380E-01, 5.2939E-04, -1.1844E-05, # AN 2 9.5077E-01, 1.7396E-03, -1.5404E-05, # AN 3 9.3945E-01, 1.6803E-03, -9.2158E-06, # AN 4 9.6259E-01, 1.5072E-03, -1.5226E-05, # AN 5 9.7797E-01, 9.2114E-04, -9.6542E-06, # AN 6 9.5072E-01, 1.6461E-03, -1.3873E-05, # AN 7 9.6315E-01, 1.3102E-03, -1.1699E-05, # AN 8 9.8908E-01, 7.4323E-04, -1.2785E-05, # AN 9 9.4594E-01, 1.9071E-03, -1.6902E-05, # AN 10 9.7378E-01, 1.2212E-03, -1.4299E-05, # AN 11 9.2886E-01, 2.0099E-03, -1.4326E-05, # AN 12 9.7188E-01, 1.2060E-03, -1.2958E-05, # AN 13 9.5259E-01, 1.5981E-03, -1.3701E-05, # AN 14 9.9007E-01, 7.6713E-04, -1.5014E-05, # AN 15 9.3265E-01, 2.0155E-03, -1.5228E-05, # AN 16 9.4203E-01, 2.3229E-03, -2.3322E-05, # AN 17 9.8450E-01, 1.3243E-03, -2.9131E-05, # AN 18 9.4425E-01, 2.1415E-03, -2.0617E-05, # AN 19 9.3657E-01, 2.9488E-03, -3.4398E-05, # AN 20 9.9317E-01, 5.8221E-04, -1.4247E-05, # AN 21 9.4996E-01, 1.7295E-03, -1.5152E-05, # AN 22 9.7844E-01, 9.0703E-04, -9.8916E-06, # AN 23 9.7704E-01, 8.9316E-04, -9.0036E-06, # AN 24 9.1715E-01, 2.1045E-03, -1.3433E-05, # AN 25 9.9063E-01, 4.9646E-04, -6.6641E-06, # AN 26 9.5552E-01, 1.6396E-03, -1.5122E-05, # AN 27 9.8287E-01, 8.9881E-04, -1.1829E-05, # AN 28 1.0000E+00, 6.9500E-04, 0.0000E+00, # AN 29 9.6350E-01, 1.3945E-03, -1.5014E-05 ] # AN 30 = avg } else { c := array( [1,0,0],3,30 ); } } } } # reshape as array c::shape := [3,30]; return c; } Just insert par.band := 'k'; band := par.band; c := vlagainfill( band ); into script at an appropriate point. Finally for today, work out how to get the reference frame for the direction from the FIELDS table. Try - fnam := spaste(msf, '/FIELD'); - ftab := table(fnam); - ftab.colkeywordnames("REFERENCE_DIR") QuantumUnits MEASINFO - ftab.getcolkeyword("REFERENCE_DIR","QuantumUnits") rad rad - ftab.getcolkeyword("REFERENCE_DIR","MEASINFO") [type=direction, Ref=J2000] Looks like we want something like - fref := ftab.getcolkeyword("REFERENCE_DIR","MEASINFO"); - printf(' Reference frame %s\n',fref.Ref); Reference frame J2000 and use fref.Ref in script fdirs := ftab.getcol("REFERENCE_DIR"); fields := ftab.getcol("NAME"); ftab.close(); # # Process directions - store in record of directions # dirq := [=]; for ( i in 1:nfld ) { raq := dq.quantity(fdirs[1,,i],'rad'); deq := dq.quantity(fdirs[2,,i],'rad'); dirq[i] := dm.direction(fref.Ref,raq,deq); } >>>QUERY: Can you have a measurement set with different epochs for the directions (each row)? >>>NOTE: Walter mentioned that the direction could be a polynomial, and I see in http://aips2.nrao.edu/stable/docs/notes/229/229.html for the FIELD table that the dimension of REFERENCE_DIR columns is (2, NUM_POLY+1) to represent higher order terms. But the Reference value in the column header keyword would seem to have to be still one of 'J2000' etc. What happens on DBCON? >>>NOTE: Ralph says: The answer is no and, when I asked this question a few years ago was told it was for performance reasons. This general philosphophy is used throughout the MS with one significant exception. All the spectral windows do not have to be in the same frame so that different spectral windows can be Barycentric, Topocentric, Local Standard of Rest etc. >>>NOTE: I just asked Athol and he said basically the same thing, that currently everything is in the same epoch (as listed in the column keyword), but that this was just a convention that might be liberalized like in the case for the spectral_windows with a REFERENCE_FRAME_ID column added (or something like this). Personally, the current scheme makes life much easier... Test it now, - include '2001-12-14-myers.g' - vlagaincor(par) Got a huge number of errors of the type error, numeric and non-numeric types mixed error, non-compatible types for assignment and the machine was locked (with lots of disk thrashing and swap) - had to kill aips++.g process from outside of aips++ (again there needs to be much better error handling). Note that I have had some instances of aips++ lockup (though not the disk thrashing) after this, it may be that the aips++ guys are doing something with the server writing the release (Wes?). Recovered and continue... Note: had to handle ANTENNA1 etc column output as an1 := as_integer( anam[ an1v ]); an2 := as_integer( anam[ an2v ]); instead of an1 := anam[ an1v ]; an2 := anam[ an2v ]; as suspected earlier. Also had typo in gaincv. Fixed. - include '2001-12-14-myers.g' - vlagaincor(par) Filled gain coefficients for VLA at Kband Found 29 antennas in table Data is from telescope VLA Found 8 fields Reference frame J2000 Processing timestamp 2001/02/04/08:27:30.000 Corrected 1159+292 with 72 rows at EL = 67.26 Processing timestamp 2001/02/04/08:27:33.334 Corrected 1159+292 with 156 rows at EL = 67.27 Processing timestamp 2001/02/04/08:27:36.666 Corrected 1159+292 with 380 rows at EL = 67.28 Processing timestamp 2001/02/04/08:27:40.001 ... Corrected 1159+292 with 650 rows at EL = 82.24 Processing timestamp 2001/02/04/10:41:10.000 Corrected 1159+292 with 650 rows at EL = 82.23 Completed gain and opacity correction Seems to have worked. Do still need to verify that the corrections applied were sensible. Job for next week. This procedure will be put in the scriptorum as vlagaincor.g along with automap.g ( http://www.aoc.nrao.edu/~smyers/aips++/scripts/ ). ------------------------------------------------------------------------------ Testing logs archived at http://www.aoc.nrao.edu/~smyers/aips++/testlogs/ This file can be found as 2001-12-14-myers.txt The script can be found as 2001-12-14-myers.g