2.4 Flagging Calibration-Tables

Next: 3 Frequently Asked Questions Up: 2 Running the flagger Previous: 2.3 Summaries, displays, reports Contents

Subsections

2.4 Flagging Calibration-Tables

2.4.1 Ability to easily flag the new Calibration Table format

The new Calibration Table format can be handled by the AgentFlagger tool in a transparent way, just like a Measurement Set Table. There is an open-time check that inspects the table type and uses the appropriate FlagDataHandler implementation for all the I/O operations:

CASA <2>: af.open('X7ef.tsys')
INFO	AgentFlagger::open	Table type is Calibration
  Out[2]: True

In principle support for the new Calibration Table format has been introduced transparently, so it is possible to flag them using any of the already available modes for Measurement Sets. However, due to the nature of the data in calibration tables, it only makes sense to use the following flag agent groups:

Auto-flagging (outliers detection) agents: clip, tfcrop, rflag:

af.open('X7ef.tsys')
af.selectdata()
agentClip={'mode':'clip','clipzeros':True,'clipminmax':[0.,600.],'datacolumn':'FPARAM'}
af.parseagentparameters(agentClip)
af.init()
af.run()
af.done()

Manual flagging (meta-data selections like antenna/time-range):

af.open('X7ef.tsys')
af.selectdata()
agentManual={'mode':'manual','antenna':'DV09'}
af.parseagentparameters(agentManual)
af.init()
af.run()
af.done()

Auxiliary modes: unflag, summary, display

af.open('X7ef.tsys')
af.selectdata()
agentSummary={'mode':'summary'}
af.parseagentparameters(agentSummary)
af.init()
summary = af.run()
af.done()

Notice that the parameters names and types are the same as described in the flagdata command-line help.

2.4.2 Supported columns and expressions

For the autoflagging agents (clip, tfcrop, rflag) it is possible to choose the column to operate with, using the 'datacolumn' parameter, being 'FPARAM', 'PARAMERR' and 'SNR' the currently supported columns:

agentClip={'mode':'clip','clipminmax':[0.,6.],'datacolumn':'PARAMERR'}
af.parseagentparameters(agentClip)
af.init()
...
INFO	Clip::setAgentParameters	data column is PARAMERR

But notice that the default datacolumn is FPARAM:

agentClip={'mode':'clip','clipminmax':[0.,6.]}
af.parseagentparameters(agentClip)
af.init()
...
INFO	Clip::setAgentParameters	data column is FPARAM

And it is also possible to choose the element of interest to inspect for flagging, trough the correlation parameter, specifying a list of comma-separated elements or 'ALL' to select all the elements (which is also the default):

agentClip={'mode':'clip',clipminmax':[0.,600.],'datacolumn':'FPARAM','correlation':'Sol1'}
af.parseagentparameters(agentClip)
af.init()
...
INFO	Clip::setAgentParameters	Visibility expression is REAL SOL1

agentClip={'mode':'clip','clipminmax':[0.,600.],'datacolumn':'FPARAM','correlation':'ALL'}
af.parseagentparameters(agentClip)
af.init()
...
INFO	Clip::setAgentParameters	Visibility expression is REAL SOL1,SOL2

Also notice that the default expression is 'ALL':

agentClip={'mode':'clip','clipminmax':[0.,6.]}
af.parseagentparameters(agentClip)
af.init()
...
INFO	Clip::setAgentParameters	Visibility expression is REAL SOL1,SOL2

2.4.3 Meta-Data selection

As far as meta-data selection is concerned, we are bounded to what MSSelection interface for Calibration Tables offers (currently only field, spw and antenna selections). There are two way to apply meta-data selections, with can be used independently or combined together:

FlagDataHandler selection (global): Create a sub-table with the selection to be used by all the agents. The data selection parameters have to be parsed trough the selectdata() method of the AgentFlagger tool:
```
af.open('X7ef.tsys')
af.selectdata(field='TW Hya')
agentManual={'mode':'manual'}
af.parseagentparameters(agentManual)
af.init()
af.run()
af.done()
```
Agent selection (specific to each agent): To restrict the rows that each agent inspect for flagging. The data selection parameters can be parsed as normal agent parameters:
```
af.open('X7ef.tsys')
af.selectdata()
agentManual={'mode':'manual','antenna':'DV09'}
af.parseagentparameters(agentManual)
af.init()
af.run()
af.done()
```

Combined selection : To iterate only trough a global selection, and at the same time have each agent inspecting a particular group of rows for flagging:

af.open('X7ef.tsys')
af.selectdata(field='TW Hya')
agentManual={'mode':'manual','antenna':'DV09'}
af.parseagentparameters(agentManual)
af.init()
af.run()
af.done()

2.4.4 Combining several modes in a single run

As with MeasurementSet tables, the re-factored flagger framework allows running several flagging agents at the same time, to reduce I/O overhead, for instance:

Combination of tfcrop and rflag:

af.open('cal.fewscans.bpass')
af.selectdata()
agentRflag={'mode':'rflag','datacolumn':'PARAMERR','correlation':'Sol1'}
agentTfcrop={'mode':'tfcrop','datacolumn':'PARAMERR','correlation':'Sol1'}
af.parseagentparameters(agentRflag)
af.parseagentparameters(agentTfcrop)
af.init()
af.run(writeflags=True)
af.done()

**Figure 10:** Rflag and TFCrop agents combined in a single run applied on a EVLA band-pass calibration table

Unflag + Clip + Display

  
af.open('X7ef.tsys')
af.selectdata()
agentUnflag={'mode':'unflag'}
agentClip={'mode':'clip','clipminmax':[0.,600.],'datacolumn':'FPARAM','correlation':'Sol1'}
agentDisplay={'mode':'display','datadisplay':True}
af.parseagentparameters(agentUnflag)
af.parseagentparameters(agentClip)
af.parseagentparameters(agentDisplay)
af.init()
af.run(writeflags=True)
af.done()

**Figure 11:** Clipping agent operating on ALMA TSys CalTable (TW Hya Science Verification data)

Unflag + Manual Selection (channel edges) + Summary

af.open('X7ef.tsys')
af.selectdata()
agentUnflag={'mode':'unflag'}
agentManual={'mode':'manual','spw':'*:0~9;118~127'}
agentDisplay={'mode':'display','datadisplay':True}
af.parseagentparameters(agentUnflag)
af.parseagentparameters(agentManual)
af.parseagentparameters(agentDisplay)
af.init()
af.run(writeflags=True)
af.done()

**Figure 12:** Clipping agent operating on ALMA TSys CalTable (TW Hya Science Verification data)

Unflag + Manual Selection (antenna) + Summary

af.open('X7ef.tsys')
af.selectdata()
agentUnflag={'mode':'unflag'}
agentManual={'mode':'manual','antenna':'DV09'}
agentSummary={'mode':'summary'}
af.parseagentparameters(agentUnflag)
af.parseagentparameters(agentManual)
af.parseagentparameters(agentSummary)
af.init()
summary = af.run(writeflags=True)
af.done() 

CASA <12>: summary
  Out[12]: 
{'nreport': 2,
 'report0': {'antenna': {'DV04': {'flagged': 0.0, 'total': 14336.0},
                         'DV06': {'flagged': 0.0, 'total': 14336.0},
                         'DV07': {'flagged': 0.0, 'total': 14336.0},
                         'DV08': {'flagged': 0.0, 'total': 14336.0},
                         'DV09': {'flagged': 14336.0, 'total': 14336.0},
                         'DV10': {'flagged': 0.0, 'total': 14336.0},
                         'PM01': {'flagged': 0.0, 'total': 14336.0},
                         'PM02': {'flagged': 0.0, 'total': 14336.0},
                         'PM03': {'flagged': 0.0, 'total': 14336.0}},

Next: 3 Frequently Asked Questions Up: 2 Running the flagger Previous: 2.3 Summaries, displays, reports Contents

R. V. Urvashi 2012-11-01