; UVFLG ;--------------------------------------------------------------- ;! Flags UV-data ;# TASK UV CALIBRATION EDITING ;----------------------------------------------------------------------- ;; Copyright (C) 1995-1996, 1998, 2000-2001, 2003-2004 ;; Associated Universities, Inc. Washington DC, USA. ;; ;; This program is free software; you can redistribute it and/or ;; modify it under the terms of the GNU General Public License as ;; published by the Free Software Foundation; either version 2 of ;; the License, or (at your option) any later version. ;; ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; ;; You should have received a copy of the GNU General Public ;; License along with this program; if not, write to the Free ;; Software Foundation, Inc., 675 Massachusetts Ave, Cambridge, ;; MA 02139, USA. ;; ;; Correspondence concerning AIPS should be addressed as follows: ;; Internet email: aipsmail@nrao.edu. ;; Postal address: AIPS Project Office ;; National Radio Astronomy Observatory ;; 520 Edgemont Road ;; Charlottesville, VA 22903-2475 USA ;----------------------------------------------------------------------- UVFLG LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC UVFLG: Task to Flag selected UV data on disk. INNAME UV data (name). INCLASS UV data (class). INSEQ 0.0 9999.0 UV data (seq. #). 0 => high INDISK 0.0 9.0 Disk unit #. 0 => any INFILE Text file name. SOURCES Sources selected/deselected SUBARRAY Subarray number 0=>all. SELBAND Bandwidth to flag (kHz) SELFREQ Frequency to flag (MHz) FREQID Freq. ID to flag. -1=>all. 0=> first Freq. ID. TIMERANG Time: start day,hr,min,sec stop day,hr,min,sec. BCHAN 0.0 9999.0 First frequency channel ECHAN 0.0 9999.0 Last channel 0=>all BIF 0.0 9999.0 First IF 0=>1 EIF 0.0 9999.0 Last IF 0=>all ANTENNAS Antennas BASELINE Baselines with ANTENNAS STOKES Stokes' parameters FLAGVER Flag table version number Used w single-source too APARM Clip on ampl. and weight (single source data only) 1:lower limit of amplitude 2:upper limit of amplitude 0 => use defaults sign is significant! 3:limit of weight The data with ampl. outside of a given range or weight less the given value are flagged (unflagged) Elevation flagging 4:elevation lower cutoff deg 5:elevation upper cutoff deg 6:Factor of suppression of spurious fringes caused by pulse cal when fringe rate close to zero. It must be >1 0 => do not flag due to this reason 7:>0 => Print source, time, baseline, U,V of expected zero fringe rate (U=0) 0 => do not print APARM=0 => flagging(unflag.) does not depend on APARM OPCODE 'FLAG','UFLG' or 'REAS' REASON Reason (24 char.) BDROP -1.0 First vis. no. to flag (not with flag tables) EDROP -1.0 Last vis. no. to flag (not with flag tables) DOHIST -1.0 1.0 > 0 -> add to history ---------------------------------------------------------------- UVFLG Type: Task Use: Data can be flagged or unflagged. If the data file is a multi-source (i.e. has an SU table) data file the flagging entries are made in the flag (FG) table. For single-source data files, flagging entries may also be made in an FG table or the data may be changed by setting the weights for the selected data negative. NOTE: not all adverbs are used for both types of files. If many flagging criteria are to be supplied at once, they can be entered in a text file which can be read by UVFLG. Adverbs: USERID.....User number. 0 => current, 32000 => any user. INNAME.....UV file name (name). Standard defaults. INCLASS....UV file name (class). Standard defaults. INSEQ......UV file name (seq. #). 0 => highest. INDISK.....Disk unit #. 0 => any. INFILE.....Name of the text file containing flagging instructions. Adverb values are entered in a text file with descriptions of data to be flagged separated by a '/'. There canbe up to 40000 selection criteria in the edit file. See EXPLAIN UVFLG. If INFILE is used, then adverbs such as TIMERANG, BIF, EIF.. are ignored. Only flagging by elevation and pulse cal works together with INFILE. SOURCES....List of sources in a raw data file to be selected. If any names begins with a '-' all sources listed are deselected (i.e. all others are). SUBARRAY...The subarray desired for the operation. 0=> all. SELBAND....Bandwidth of data to be flagged. If more than one IF is present SELBAND is the width of the first IF required. Units = kHz, 0=> all SELFREQ....Frequency of data to be flagged. If more than one IF is present SELFREQ is the frequency of the first IF required. Units = MHz, 0=> all FREQID.....Frequency identifier to flag (you may determine which is applicable from the OPTYPE='SCAN' listing produced by LISTR. If either SELBAND or SELFREQ are set their values overide that of FREQID, however setting SELBAND and SELFREQ may occasionally result in an ambiguity, in which case the task will request that you use FREQID. 0=> program will select the first in the FQ table, -1=> flag all FREQID values. Note this useage is slightly different from that in other tasks. TIMERANG...The specified time range. 1 = Start IAT day (day 0 = first day in data base) 2 = Start IAT hour 3 = Start IAT minute 4 = Start IAT second 5 = Stop IAT day (day 0 = first day in data base) 6 = Stop IAT hour 7 = Stop IAT minute 8 = Stop IAT second Stop IAT = 0 => Stop IAT = Start IAT Both = 0 => all times stop IAT = start IAT not 0, flag +/- 0.5 sec of the stated time Times are NO LONGER altered from the user input except as stated above. Remember that some displays of time are integerized, i.e., a displayed 17 could be anywhere from 16.5 to 17.5. You must now supply any desired rounding down (Start iAT) and up (Stop IAT) if you enter both times. BCHAN......The first frequency channel selected. ECHAN......The last frequency channel selected. BIF........The first IF selected. EIF........The last IF selected. ANTENNAS...A list of the antennas to be flagged. If any number is negative then all antennas listed are NOT to be flagged and all others are. BASELINE...If any elements of BASELINE are non zero then all baselines between antennas named in ANTENNAS and those named in BASELINE are specified. If ANTENNAS contains a negative value then all baselines NOT specified are selected. See EXPLAIN UVFLG for examples. STOKES.....Specifies the Stokes types to be flagged Recognized values are 'I', 'Q', 'U', 'V', 'IQU', 'IQUV', 'IV', 'RR', 'LL', 'LR', 'RL', 'HALF'(=RR,LL), 'FULL' (=RR,LL,RL,LR) 'CROS'(=RL,LR), ' '=>all stokes' types. Alternatively, the "mask" to be applied can be recognised e.g. "1001" for flag RR and LR. FLAGVER....Specifies the version of the flagging table into which the flags are written (0 => highest, or 1 if no FG table exists) for all multi-source data sets and for all single-source data sets which already have a flag table or when FLAGVER is not 0. If FLAGVER is 0 and the single-source file does not already have a FG table, then the flags are actually applied to the data. APARM......The data with amplitude less than APARM(1) or more than APARM(2), or with weight less than APARM(3) are flagged (unflagged), if the data satisfy the other conditions of flagging (unflagging). For autocorrelation data (such as single-dish data) the data can have either sign so the signs of APARM(1) and APARM(2) are significant. For cross-correlation data, the amplitude is always positive. These parameters work only in the case of a single source file, when the FG table is absent. If APARM(2)=0, no flagging is done based on amplitude. If APARM(3)=0, no flagging is done based on weight. Single and multisource flagging (unflagging) on elevation is possible using APARM(4) and APARM(5). These specify the range in elevation in degrees over which data are to be flagged (unflagged). APARM(4) >= APARM(5) => no flagging on elevation. If APARM(4) = 0 and APARM(5) > 0, APARM(4) is set to -90. Spurious fringes appear due to pulse cal injection, if fringe rate close to zero. These spurious fringes are self averaged at fringe rate stopping procedure if fringe rate is not close to zero. Time interval required to suppress the spurious fringes is calculated by Kogan at VLBA Test Memo No 58, 1998. This time interval depends on suppression factor. The greater the suppression factor the longer the flagging time interval will be. The desired suppression factor is determined by APARM(6)>1. The value of the suppression factor should be selected experimentally considering pcal tones output less than rms of noise. APARM(7)>0 allows to print source, time, baseline, and U,V of expected zero fringe rate (U=0) So the user can pay a special attention to the PCAL tones at the vicinity of the points. All other flag selections are applied simultaneously. For example, setting a specific time range will cause only data below the elevation cutoff in that time range to be flagged (or unflagged). Elevation flagging can be combined to similar effect with antenna, baseline, Stokes, IF and channel selection. If all APARM=0 the flagging(unflagging) does not depend on amplitude, weight or elevation, i.e. all data satisfied the other conditions are flagged or unflagged. OPCODE.....Tells whether to flag or unflag the data. 'FLAG' => flag the data, 'UFLG' => unflag, 'REAS' => unflag all with the given REASON. ' '=>FLAG NOTE: Compressed single source data which are flagged directly (without writing to a flag table), cannot be recovered with OPCODE = 'UFLG'. REASON.....Reason for flagging the data, up to 24 characters. Raw data files only. If APARM(4) and APARM(5) are set and REASON is left blank then 'ELEVATION RANGE EXCLUDED' is used as a REASON If APARM(6) and/or APARM(7) are set and REASON is left blank then 'small fringe rate' is used as a REASON BDROP......First visibility number to flag. 0=>first. NOTE: Only data specified by BOTH BDROP-EDROP and the other adverbs are affected. EDROP......Last visibility number to flag. 0=>last BDROP and EDROP apply only when applying the flags to the data immediately, i.e., when there is no flag table. DOHIST.....True means add the individual flags in the history file, else just record the use of UVFLG. (True is > 0) ---------------------------------------------------------------- UVFLG: Task to Flag selected UV data on disk. Documentor: E.B.Fomalont Related Programs: CLIP, PRTUV, UVPLT, UVCOP, PRTAB PURPOSE UVFLG is a general-purpose editing task. The user specifies a time range, an antenna or correlator and a polarization channel etc. UVFLG works in one of two modes depending on whether the data file is a single source, calibrated data file or a multi source, raw data file. The presence of a source ('SU') extention table is used to distinguish which. In the former case, the visibility data are flagged; in the latter, entries are made in the flagging ('FG') table. The contents of the FG file can be examined with AIPS task PRTAB. As of AIPS version 15JUL95 UVFLG will accept two different formats of the ascii files. The new format described below was introduced to allow the ascii flagging files generated by the VLBA monitor system to be read into AIPS FG tables with no user editing being required. Previously users had to modify the day numbers to be 0-relative and change the antenna nasmes to their equivalent numbers read from the AN table. UVFLG can read both the new and the old format files in a manner transparent to the users. The visibility record is flagged bad by setting its weight (number of 10-second integrations in the record) negative. Data can also be unflagged in which case the visibility record weight is made positive. Make a copy of a single source u-v data base before extensive flagging is begun since the UV data file is overwritten and it is sometimes difficult correcting for flagging errors. The time syntax in AIPS is different than that in the DEC-10. Time zero is defined as midnight of the first day in the data base and four numbers are needed for the start time and end time of any edit. These times are inserted in TIMERANG. If both the Start and Stop IAT times are zero, then all data in the file is flagged. All data within 0.5 seconds of the time range specified will be flagged if the two times are equal (or the stop time given is zero while the start time is not 0). TIMES ARE NO LONGER ADJUSTED BY UVFLG. YOU MUST PROVIDE ANY ROUNDING DOWN AND UP TO ACCOUNT FOR THE ACCURACY OF THE DISPLAYS FROM WHICH YOU DETERMINED THE TIMES TO BE FLAGGED. Remember that the numbers entered into AIPS are floating point, so you can enter, particularly seconds, in decimal fraction form. After execution, UVFLG will list the number of visibility records which have been selected and the number of correlators flagged or unflagged. These numbers are the counts of record numbers (a time stamp and antenna pair) which have been modified by UVFLG. This report is only given for single source data files as no visibility records are modified in raw data files. INFILE If more than one set of data are to be flagged a text file can be read. If INFILE is used, then adverbs such as TIMERANG, BIF, EIF.. are ignored. Only flagging by elevation and pulse cal works together with INFILE. Examples: INFILE='disk$res:[username]flag.dat' (VMS) INFILE='myarea:flag.dat' (Unix) where MYAREA is an environment variable set before starting AIPS: %setenv MYAREA /mnt/username This table will be read using the Caltech KEYIN routine and is free format; a "!" denotes that the rest of the line is a comment, a " /" denotes the end of the entry. Adverbs are entered with the same name as in AIPS but with no minimum match. The '=' sign is optional and all expressions must be enclosed in parentheses '(' and ')'. Elements of an array MUST be separated by commas. Only one entry should be made on a given line of the editing table although a given entry may span several lines. Adverb values are initialized to their null values before each entry in the table is read. Up to 40000 editing entries may be made for single source files; there is no limit for multi-source files. Note however, that if multiple baselines are specified, each will count as one of the 40000 entries. The adverbs which may be entered in this table are: SOURCES, SUBARRAY, TIMERANG, BCHAN, ECHAN, BIF, EIF, ANTENNAS, BASELINE, STOKES, OPCODE, REASON, BDROP, EDROP and FREQID. In addition, a parameter TIMEOFF can be given which is the number of seconds to add to the timerange to convert it to IAT times. The current value to correct UTC to IAT is about +24.0 sec. In addition, a parameter DTIMRANG can be given which is the number of seconds to increase TIMERANG from each side i.e to subtract DTIMRANG from the left border of TIMERANG and add DTIMRANG to the right border of TIMERANG. NEW FORMAT: UVFLG will now also accept adverbs ANT_NAME and BAS_NAME in place of ANTENNAS and BASELINE. The original adverbs are still valid though. In the following example entry in a RUN table all data on day 1 between 01:00:00 and 01:05:00 IAT involving antenna 7 and FREQID 3 is to be flagged: ! Flag Antenna 7 1/1:0:0 to 1/1:5:0; all IFs and frequencies, ! any vis. number in a single source data file: TIMERANG=1,1,0,0, 1,1,5,0 ANTENNAS=7 FREQID=3 / NEW FORMAT: in the new format this entry might be typed as: TIMERANG=242,1,0,0, 242,1,5,0 ANT_NAME='KP' FREQID=3 / where KP is antenna 7 in the data, 242 is the day number of the observations. In the following example, for a multi-source data file, Antenna 10, left circular polarization, is flagged for all times on sources 3CXYZ and 0000-900 because the antenna is the wrong color: ANTENNAS=10 SOURCES='3CXYZ','0000-900' STOKES 'LL' REASON='ANTENNA WRONG COLOR' / Note: commas are not allowed between ADVERBS and there should also be at least one blank before each "/" used to delimit an entry. SOURCES For multi source data sets a list of sources may be specified. If any of the names listed begin with a "-" then all sources listed are deselected, i.e. the described operation will be performed on all OTHER sources. The "-" sign will be ignored in the consideration of the source name. ANTENNAS, BASELINE Antennas to be flagged are specified in array ANTENNAS. If any antenna number is negative the absolute value will be used and all antennas listed will be deselected, i.e. the descriped operation will be done on all OTHER antennas. NEW FORMAT: The adverb ANT_NAME can replace ANTENNAS in the new format. Following ANT_NAME will be the left-justified name (<= 8 characters) and placed in quotes. If using ANT_NAME there can be no deselection as described above. Only one ANT_NAME can be specified per entry. Baselines are specified with the BASELINE adverb. If BASELINE contains non zero elements then all baselines between all antennas specified in ANTENNAS and those specified in BASELINE are selected. If ANTENNAS contains a negative entry then all the specified operation will be carried out on all baselines OTHER than those explicitly named. NEW FORMAT: The adverb BAS_NAME can replace BASELINE in the new format. Following BAS_NAME will be the left-justified name (<= 8 characters) and placed in quotes. Examples: If ANTENNAS = 1,2,3 and BASELINE = 4,5 then baselines 1-4, 2-4, 3-4, 1-5, 2-5 and 3-5 are selected. If ANTENNAS = -1,2,3 and BASELINE = 4,5 then ALL baselines EXCEPT 1-4, 2-4, 3-4, 1-5, 2-5 and 3-5 are selected. STOKES The Stokes type of the data to be flagged can be specified as on of the following: 'I', 'Q', 'U', 'V', 'IQU', 'IQUV', 'IV', 'RR', 'LL', 'LR', 'RL', 'HALF'(=RR,LL) and 'FULL' (=RR,LL,RL,LR), ' '=>all stokes' types. A given polarization for antenna flagging can be specified by 'RR' or 'LL'. Note that specification of 'RL' or 'LR' will flag 'RL' AND 'RR' or 'LR' AND 'LL' respectively. To flag a particular fringe e.g.'RL', specify this using the appropriate mask - in this case '0010'. The order of the mask bits is RR, LL, RL, LR. OPCODE If OPCODE='FLAG' the specified data is to be flagged either by entries in the flagging table or in the visibility data as appropriate. If OPCODE='UFLG' then the data is unflagged in the visibility records or flagging table entries are set to be deselected if they match the specified entry. NOTE: if there is ANY overlap in the specifications of data to be unflagged and an entry in the flagging table then the entire table entry is flagged (deselected, turned off). Thus, if some range of the specifications in the table entry are to remain flagged, these portions should be reflagged. This does not apply to flagging done directly to the data. OPCODE='REAS' deselect all FG table entries with the specified REASON. NOTE: all other criteria are ignored. REASON Entries made in a flagging table can be labeled with a reason that the data was flagged. OPCODE can then be used to deselect entries in the flagging table with this reason thereby effectively unflagging the data. The entry may be up to 24 characters long. REASON can also be used with OPCODE='UFLG' to selectively unflag data. If the first 4 characters of REASON are blank and unflagging is specified then REASON is ignored. Thus OPCODE='REAS'; REASON='' will cause all entries in a flagging table to be turned off; that is they will not be applied to the data but will not be physically removed. BDROP, EDROP If flagging by visibility number is desired BDROP and EDROP can be used. The visibility numbers can be obtained from PRTUV, UVFND etc. Visibility points between BDROP and EDROP will be flagged (or unflagged) if and only if they were specified by APARM. In the following example visibility points 1000 through 1005 are flagged for R polarization on any baseline to antenna 7: ANTENNA=7,0; BASELINE=0; STOKES='R'; BDROP=1000; EDROP=1005 Note: if the flagging by visibility number is not desired then use BDROP=0; EDROP=0. BDROP AND EDROP APPLY TO SINGLE SOURCE FILES ONLY AND THEN ONLY WHEN NOT USING A FLAG TABLE.