2.2 List of flag commands (flagdata, flagcmd)

There are two flagging tasks in version 4.0 of CASA. The main task flagdata is a replacement for the old flagdata/flagdata2/testautoflag/flagautocorr. This task can take parameters input from the interface as well as input from an external text file or a list of strings. This is the so-called list mode. All the flagging commands that apply to an MS can be combined into one single text file and read in for processing. Similarly, the flagging commands can be written in a Python list of strings and passed in the command line to the task. Only the summary mode is not allowed in list mode to avoid confusion when several summaries are present in the list. The user can run flagdata separated to get the summary of the list.

The other task, flagcmd is a replacement for the previous version of flagcmd. It has a re-factored interface and it can do much more than the previous version. All the flagging modes available in flagdata are also allowed in flagcmd (except again for the summary mode). Several input modes are available: table, list and xml. Actions to perform on the input are: apply, unapply, list, clear, plot and extract.

2.2.1 Apply/ un-apply

It is possible to truly unapply flag commands in flagcmd. In order to guarantee that only the data selected in the commands is unapplied, the framework will first unapply the selected rows and then re-apply the overlapping data that got unapplied in the first pass. This is a true unapply action, but it will take longer to process because it will re-apply all the remaining commands that have the column APPLIED set to True inside the FLAG_CMD sub-table of the MS.

2.2.2 Using flag-commands in the flagcmd task

Flag commands can be given as input to the flagcmd task in several ways. The default is the 'table' which will have the flag commands inside the FLAG_CMD sub-table of the MS in a column called COMMAND.

The other input, as already detailed above is the 'list' mode, which will have the flag commands in a text file or from a list of strings.

The 'xml' input is meant to be used for the online flags from the Flag.xml file in the MS. It also assumes that the Antenna.xml file is present.

For all the input modes, the action='list' can be used to list the flags commands on the screen and optionally to save them to an external output file when the parameter savepars is set to True.

When the action='apply' is used in the inpmode table, the flags will be applied to the MS and the APPLIED column will be updated to True. On the other hand, when action='unapply' the APPLIED column will be set to False for all selected table rows.

2.2.2.1 Examples

1. Input commands from the FLAG_CMD sub-table.

   Figure 8: Screenshot of the FLAG_CMD sub-table of a MS with flag commands for the online flags.

   

   # Apply only the flag commands that contain reason='CORRELATOR_DATA_INVALID'.
   The default action for flagcmd is to apply the flag commands from a table input,
   therefore it is not necessary to specify inpmode='table' and action='apply'.
   
    flagcmd(vis='uid.ms', reason='CORRELATOR_DATA_INVALID')
   

2. Input commands from an ASCII file.

   # Create an input text file with all the flag commands to apply to the MS.
   > cat flags.txt
    mode='manual' autocorr=True
    mode='shadow'
    mode='clip' clipminmax=[0,2] clipzeros=True correlation='ABS_ALL'
    mode='tfcrop' spw='9' correlation='ABS_YY' ntime=51.0
    mode='extend' extendpols=True
   
   # Pass the input to the task
     flagcmd(vis='uid.ms', inpmode='list', inpfile='flags.txt', action='apply')
   

3. Input commands via a list of strings.

   # Give the flagging commands in the command line as a list of strings.
     flagcmd(vis='uid.ms', inpmode='list',
             inpfile=["mode='manual' autocorr=True",
                      "mode='shadow'",
                      "mode='clip' clipminmax=[0,2] clipzeros=True correlation='ABS_ALL'", 
                      "mode='tfcrop' spw='9' correlation='ABS_YY' ntime=51.0",
                      "mode='extend' extendpols=True"], action='apply')
   

2.2.3 Using flag-commands in the flagdata task

2.2.3.1 Examples

1. Run flagdata in list mode, using a flag-command file.

   # Create an input text file with all the flag commands to apply to the MS.
   > cat flags.txt
    mode='manual' autocorr=True
    scan='1~3' mode='manual'
    mode='clip' clipminmax=[0,2] clipzeros=True correlation='ABS_ALL'
    mode='tfcrop' spw='9' correlation='ABS_YY' ntime=51.0
    mode='extend' extendpols=True
   
   # Pass the input to the task
     flagdata(vis='uid.ms', mode='list', inpfile='flags.txt')
    
   # Get a summary
     flagdata(vis='uid.ms', mode='summary')
   

2. Run flagdata in list mode, using a Python list of strings.

   # Create an input list with all the flag commands to apply to the MS.
   
    cmds = ["mode='manual' autocorr=True",
            "scan='1~3' mode='manual'",
            "mode='clip' clipminmax=[0,2] clipzeros=True correlation='ABS_ALL'",
            "mode='tfcrop' spw='9' correlation='ABS_YY' ntime=51.0",
            "mode='extend' extendpols=True"]
   
   # Pass the input to the task
     flagdata(vis='uid.ms', mode='list', inpfile=cmds)
    
   # Get a summary
     flagdata(vis='uid.ms', mode='summary')
   

3. Select by reason in list mode.

   # Apply only the flag commands that contain reason='BAD_ANT'. Let's say the
   input file contains the following flag commands.
   
   > cat flags.txt
     antenna='DV06' reason='BAD_ANT'
     mode='clip' clipzeros=True reason='CLIPZEROS'
     mode='shadow' reason='SHADOW'
     antenna='DV01' timerange='2012/02/01:09:14:0~09:54:0' reason='BAD_ANT'
     antenna='DV10;DV23' timerange='2012/02/01:09:15:0~09:54:0' reason='BAD_ANT'
     
    # We want to apply the above flag commands in three different MSs, but in the
    first step we only want to flag the bad antennas, so we will select the cmds
    by the reason given in the file.
    
      flagdata(vis='uid1.ms', inpmode='list', inpfile='flags.txt',
                reason='BAD_ANT')
   
      flagdata(vis='uid2.ms', inpmode='list', inpfile='flags.txt',
                reason='BAD_ANT')
   
      flagdata(vis='uid3.ms', inpmode='list', inpfile='flags.txt',
                reason='BAD_ANT')
   

4. Saving the parameters as flag commands, to a file.

   # Sometimes we want to flag using the interface but would like to save those
   parameters to a file. It may also be useful to write along a reason for those
   flags.
   
     flagdata(vis='uid1.ms', mode='clip', clipzeros=True, savepars=True,
               outfile='myflagcmd.txt', cmdreason='CLIPZEROS')
     
   # This will flag all the zeros in the MS and save the parameters to
   'myflagcmd.txt' like below:
   
   > cat myflagcmd.txt
   mode='clip' correlation='ABS_ALL' clipoutside=True datacolumn='DATA'\
   channelavg=False clipzeros=True reason='CLIPZEROS'
   
   # NOTE: if the outfile is left empty (default case), the task will save the
   parameters to the FLAG_CMD sub-table.
   

2.2.3.2 Using (and mis-using) the FLAG_CMD subtable

The FLAG_CMD sub-table is meant to store only meta-data selections such as online flags. Using it to save other parameters (from modes such as clip, quack, shadow, etc.) is possible but carries a risk that in future releases these these parameters or their default values are renamed or changed. Use it at your own risk! There will be no automatic way to rename any parameter that changes in the future.

Keeping the state of the flags stored in the FLAG_CMD sub-table is also not guaranteed. The state can get out-of-sync very easily with the interchangeable use of all flagging tasks such as flagdata, flagcmd, and interactive flagging using plotms. Note that plotms does not write to the FLAG_CMD sub-table.