2.4 Flagging Calibration-Tables

Next: 3 Frequently Asked Questions Up: 2 Running the flagger Previous: 2.3 Summaries, displays, reports Contents

Subsections

2.4 Flagging Calibration-Tables

2.4.1 Ability to flag the new Calibration Table format

The new Calibration Table format can be handled by the tasks flagdata, flagcmd and the AgentFlagger tool in a transparent way, just like a Measurement Set Table. If the calibration table was created before CASA 4.1, the cal table handler will create a dummy OBSERVATION column and OBSERVATION sub-table in the input calibration table to adapt it to the new cal table format. There is limited support for older cal tables, therefore we recommend the user to re-create his/her calibrations using CASA 4.1 before running flagdata.

There is an open-time check that inspects the table type and uses the appropriate FlagDataHandler implementation for all the I/O operations:

CASA <2>: flagdata('X7ef.tsys', mode='unflag')

CASA <2>: af.open('X7ef.tsys')
INFO	AgentFlagger::open	Table type is Calibration
  Out[2]: True

Support for the new Calibration Table format has been introduced transparently, so it is possible to flag them using any of the already available modes for Measurement Sets. However, due to the nature of the data in calibration tables, it only makes sense to use the following flag agent groups:

Auto-flagging (outliers detection) agents: clip, tfcrop, rflag:

# Using the flagdata task
flagdata('X7ef.tsys',mode='clip',clipzeros=True,clipminmax=[0.,600.],datacolumn='FPARAM' )

# Using the agentflagger tool
af.open('X7ef.tsys')
af.selectdata()
agentClip={'mode':'clip','clipzeros':True,'clipminmax':[0.,600.],'datacolumn':'FPARAM'}
af.parseagentparameters(agentClip)
af.init()
af.run()
af.done()

Manual flagging (meta-data selections like antenna/time-range):

# using the flagdata task
flagdata('X7ef.tsys', antenna='DV09')

# Using the agentflagger tool
af.open('X7ef.tsys')
af.selectdata()
agentManual={'mode':'manual','antenna':'DV09'}
af.parseagentparameters(agentManual)
af.init()
af.run()
af.done()

Auxiliary modes: unflag, summary, display

# using the flagdata task
flagdata('X7ef.tsys', mode='summary', spwchan=True)

# using the agentflagger tool
af.open('X7ef.tsys')
af.selectdata()
agentSummary={'mode':'summary', 'spwchan':True}
af.parseagentparameters(agentSummary)
af.init()
summary = af.run()
af.done()

Notice that the parameters names and types are the same as described in the flagdata command-line help.

2.4.2 Supported columns and expressions

For the autoflagging agents (clip, tfcrop, rflag) it is possible to choose the column to operate with, using the 'datacolumn' parameter, being 'FPARAM', 'CPARAM' and 'SNR' the currently supported columns. The given column will be validated and the correct one will be chosen based on the type of input:

# giving the correct datacolumn in the task
flagdata(vis='cal.fewscans.bpass', mode='clip', datacolumn='CPARAM', clipminmax=[0.,6.],clipzeros=True)

# using the default datacolumn in the agentflagger tool
af.open('cal.fewscans.bpass')
af.selectdata()
af.parseclipparameters(clipminmax=[0.,6.],datacolumn='CPARAM', clipzeros=True)
INFO    AgentFlagger::parseAgentParameters  Validating data column CPARAM based on input type
INFO    AgentFlagger::parseAgentParameters  Will use data column CPARAM
af.init()
af.run()
af.done()

There is no default data column for cal tables. The task has DATA as the default for MSs. Once the framework has detected the input to be a cal table, it will try to use the given data column. If that does not exist (such as DATA), it will try to use FPARAM, then CPARAM. If none of the them exist, it will exit with an error.

# using the flagdata task
flagdata(vis='cal.fewscans.bpass', mode='clip', clipminmax=[0.,6.])

# using the default datacolumn in the agentflagger tool
...
af.parseclipparameters(clipminmax=[0.,6.])
INFO    AgentFlagger::parseAgentParameters  Validating data column DATA based on input type
WARN    AgentFlagger::validateDataColumn    Only FPARAM, CPARAM and SNR are supported for cal tables
INFO    AgentFlagger::parseAgentParameters  Will use data column CPARAM
...

Only for the autoflagging modes, it is also possible to choose the element of interest to inspect for flagging, through the correlation parameter, specifying a list of comma-separated elements or '' to select all the elements (which is also the default). If the CPARAM datacolumn is used, the default correlation will be ABS_ALL because this column contains complex data. For the other two supported columns (FPARAM,SNR), the default will fall back to REAL_ALL.
When flagging using mode RFlag, it is important to notice that RFlag mixes the data from all the solutions (similar to the parallel and cross-hand correlation products in an MS) to determine the flagging thresholds. If some of the solutions have higher ranges than others, they will dominate the algorithm, thus showing results only for the higher ones. We recommend that the usual way to proceed is to run flagdata using only the parallel-hand correlation products (or analogous for cal tables) and then extend it to the cross-hand correlation products to obtain the correct results.
```
agentClip={'mode':'clip',clipminmax':[0.,600.],'datacolumn':'FPARAM','correlation':'Sol1'}
af.parseagentparameters(agentClip)
af.init()
...
INFO	Clip::setAgentParameters	Visibility expression is REAL SOL1

agentClip={'mode':'clip','clipminmax':[0.,600.],'datacolumn':'CPARAM','correlation':''}
af.parseagentparameters(agentClip)
af.init()
...
INFO	Clip::setAgentParameters	Visibility expression is ABS SOL1,SOL2
```
```
agentClip={'mode':'clip','clipminmax':[0.,6.]}
af.parseagentparameters(agentClip)
AgentFlagger::parseAgentParameters  Validating data column DATA based on input type
AgentFlagger::validateDataColumn    Only FPARAM, CPARAM and SNR are supported for cal tables
AgentFlagger::parseAgentParameters  Will use data column CPARAM
af.init()
...
INFO    Clip::setAgentParameters     Visibility expression is ABS SOL1,SOL2,SOL3,SOL4
```

2.4.3 Meta-Data selection

As far as meta-data selection is concerned, we are bounded to what MSSelection interface for Calibration Tables offers (currently field, scan, time, spw, antenna and observation selections). There are two ways to apply meta-data selections that can be used independently or combined together:

FlagDataHandler selection (global): Create a sub-table with the selection to be used by all the agents. The data selection parameters have to be parsed through the selectdata() method of the AgentFlagger tool:

# using the flagdata task
flagdata(vis='X7ef.tsys', field='TW Hya')

# using the agentflagger tool
af.open('X7ef.tsys')
af.selectdata(field='TW Hya')
agentManual={'mode':'manual'}
af.parseagentparameters(agentManual)
af.init()
af.run()
af.done()

Agent selection (specific to each agent): To restrict the rows that each agent inspect for flagging. The data selection parameters can be parsed as normal agent parameters:

# using the flagdata task
flagdata(vis='X7ef.tsys', antenna='DV09')

# using the agentflagger tool
af.open('X7ef.tsys')
af.selectdata()
agentManual={'mode':'manual','antenna':'DV09'}
af.parseagentparameters(agentManual)
af.init()
af.run()
af.done()

Combined selection : To iterate only through a global selection, and at the same time have each agent inspecting a particular group of rows for flagging:

# using the flagdata task
flagdata(vis='X7ef.tsys', field='TW Hya', antenna='DV09')

# using the agentflagger tool
af.open('X7ef.tsys')
af.selectdata(field='TW Hya')
agentManual={'mode':'manual','antenna':'DV09'}
af.parseagentparameters(agentManual)
af.init()
af.run()
af.done()

2.4.4 Combining several modes in a single run

As with MeasurementSet tables, the re-factored flagger framework allows running several flagging agents at the same time, to reduce I/O overhead, for instance:

Combination of tfcrop and rflag:

# using the flagdata task
flagdata(vis='cal.fewscans.bpass', mode='list', 
         inpfile=["mode='rflag' datacolumn='CPARAM' correlation='Sol1'",
                   mode='tfcrop' datacolumn='CPARAM' correlation='Sol1'"])

# using the agentflagger tool
af.open('cal.fewscans.bpass')
af.selectdata()
agentRflag={'mode':'rflag','datacolumn':'CPARAM','correlation':'Sol1'}
agentTfcrop={'mode':'tfcrop','datacolumn':'CPARAM','correlation':'Sol1'}
af.parseagentparameters(agentRflag)
af.parseagentparameters(agentTfcrop)
af.init()
af.run(writeflags=True)
af.done()

**Figure 10:** Rflag and TFCrop agents combined in a single run applied on a EVLA band-pass calibration table

Unflag + Clip + Display

# using the flagdata task
flagdata(vis='X7ef.tsys', mode='unflag')
flagdata(vis='X7ef.tsys', mode='clip',clipminmax=[0.,600.],datacolumn='FPARAM',correlation='Sol1',
         display='data')

# using the agentflagger tool
af.open('X7ef.tsys')
af.selectdata()
agentUnflag={'mode':'unflag'}
agentClip={'mode':'clip','clipminmax':[0.,600.],'datacolumn':'FPARAM','correlation':'Sol1'}
agentDisplay={'mode':'display','datadisplay':True}
af.parseagentparameters(agentUnflag)
af.parseagentparameters(agentClip)
af.parseagentparameters(agentDisplay)
af.init()
af.run(writeflags=True)
af.done()

**Figure 11:** Clipping agent operating on ALMA TSys CalTable (TW Hya Science Verification data)

Unflag + Manual Selection (channel edges) + Summary

# using the flagdata task
flagdata(vis='X7ef.tsys', mode='unflag')
flagdata(vis='X7ef.tsys', spw='*:0~9;118~127', display='data')

# using the agentflagger tool
af.open('X7ef.tsys')
af.selectdata()
agentUnflag={'mode':'unflag'}
agentManual={'mode':'manual','spw':'*:0~9;118~127'}
agentDisplay={'mode':'display','datadisplay':True}
af.parseagentparameters(agentUnflag)
af.parseagentparameters(agentManual)
af.parseagentparameters(agentDisplay)
af.init()
af.run(writeflags=True)
af.done()

**Figure 12:** Clipping agent operating on ALMA TSys CalTable (TW Hya Science Verification data)

Unflag + Manual Selection (antenna) + Summary

# using the flagdata task
flagdata(vis='X7ef.tsys', mode='unflag')
flagdata(vis='X7ef.tsys', antenna='DV09')

# next command will return the summary with baseline counting
flagdata(vis='X7ef.tsys', mode='summary', basecnt=True)

# using the agentflagger tool
af.open('X7ef.tsys')
af.selectdata()
agentUnflag={'mode':'unflag'}
agentManual={'mode':'manual','antenna':'DV09'}
agentSummary={'mode':'summary'}
af.parseagentparameters(agentUnflag)
af.parseagentparameters(agentManual)
af.parseagentparameters(agentSummary)
af.init()
summary = af.run(writeflags=True)
af.done() 

CASA <12>: summary
  Out[12]: 
{'nreport': 2,
 'report0': {'antenna': {'DV04': {'flagged': 0.0, 'total': 14336.0},
                         'DV06': {'flagged': 0.0, 'total': 14336.0},
                         'DV07': {'flagged': 0.0, 'total': 14336.0},
                         'DV08': {'flagged': 0.0, 'total': 14336.0},
                         'DV09': {'flagged': 14336.0, 'total': 14336.0},
                         'DV10': {'flagged': 0.0, 'total': 14336.0},
                         'PM01': {'flagged': 0.0, 'total': 14336.0},
                         'PM02': {'flagged': 0.0, 'total': 14336.0},
                         'PM03': {'flagged': 0.0, 'total': 14336.0}},

Next: 3 Frequently Asked Questions Up: 2 Running the flagger Previous: 2.3 Summaries, displays, reports Contents

R. V. Urvashi 2013-09-11