Instructions for running DiFX

DiFX can be run in two ways - from the GUI or from the command line. I'll cover the GUI first, then the command line. You may wish to download the example ascii and data tarballs if you don't have some data ready to process yourself. If you wish to use the example .input file, you'll need to edit it and change the paths to the other configuration and data files. If you go through the procedure from scratch starting with the .skd file, this won't be necessary.

Running via the GUI

Firstly, fire up DiFXGUI with java -jar $CORR_ROOT/DiFXGUI/DiFXGUI.jar. If you get any error messages, make sure you have set up $CORR_ROOT/DiFXGUI/COMMANDS.CONFIG correctly.

Generally you'll want to generate a template correlator configuration using a vex file. Its important that you generate the model information first before generating the input file, because some of the operations used by the GUI depend on knowledge of what source is being observed at what time, and it extracts this information from the model files. So, the general procedure is:

  1. Select Model->Generate a model using CALC from a vex file, and select your file
  2. Wait!!!! This has to run in the background, and can take a while for long experiments. Check with ps -aux to see when it finished running, or just wait for the GUI to come back to life.
  3. Select Main->Create new configuration file from vex file, and select your file.
  4. After a second or two, the basic info should pop up. Some users may experience the scrollbars not working properly - sorry about that, I'm trying to fix it.
  5. Browse through the tabs and check the information to make sure its accurate and change anything that needs changing.
  6. Go to the Datastream tab, enter clock information in the telescope table, and add the data files for each datastream.
  7. Go to the Node tab, and select the nodes you want to use in this correlation
  8. Save the current configuration, and then save it to disk using Main->Save configuration file as.
  9. Launch! Thats the button in the top right hand corner.

That's the basics of how things work. You can then open that same input file later, edit it, launch another correlation etc. Play around with the different pages if you like, mostly the stuff that is automatically generated will be correct and you'll only need to enter clocks and data files, and probably change some of the Common settings based on the experiment. The Config settings tab allows you to set different configurations for different sources. This is mainly used when pulsar binning. The creation of pulsar binning setup files is not yet integrated into the GUI, so email me if you would like to do some pulsar binning. Its basically another ascii file - an example of one is in the examples.tar file you can grab below.

Running via the command line

This is trickier as you have to generate all the text files that control the correlation yourself. You can still use the same utility programs that the GUI invokes if you want. Otherwise, its probably handy to have a look at the description of the ascii file formats to see how things are organised. In short, all the control files are ascii, and all bar the machinefile (which tells MPI which nodes to run one) are keyword/value pairs, one per line. See these examples for additional guidance, and get in touch with me if you have questions. The machinefile is just one node name per line, very simple. So, assuming you can get all those set up, and you've decided which nodes you will run on and made the machinefile accordingly, its actually pretty simple to launch a correlation:

mpirun -np N -machinefile machines.blah $CORR_ROOT/mpifxcorr/src/mpifxcorr experiment.input

where N is the number of nodes, machines.blah is your machinefile and experiment.input is your correlator input file. You can optionally add -nolocal if you don't want to run on the machine you are launching on. The first node (the machine you are launching on, unless you specify -nolocal in which case the first name in the machinefile) will be the master, the next Ntelescopes will be datastreams, and the other N - (Ntelescopes + 1) will be processing cores.

Tips and tricks

There are one or two peculiarities that are worth noting at this point, so you don't get thoroughly confused if they do pop up. One of these is that the correlator uses negative values for geometric delay to indicate bad data that can be ignored. This is fine for ground-based telescopes, but a consequence of this is that if the delay file contains negative values (which one could conveivably get from a space-based telescope) the correlator would be unable to distinguish this from the bad data flag. If we ever get to the point of working with such situations, it could be dealt with in several ways, but for now the problem that can occur with global VLBI is having an antenna which is participating in an experiment which is under the horizon for some of the time - CALC gives negative delay values in this case. Since that telescope obviously won't be recording good data at that time anyway and thus the delays are meaningless, its fine to just zero the delays, but I didn't want to hardwire this into gencalc_delays. So if the correlator dies, complaining "Error - cannot handle negative delays!!! Need to unimplement the datastream check for negative delays to indicate bad data. Aborting now - contact the developer!", this will be the reason. I have a version of gencalc_delays which zeros all negative delays - contact me if you want to get it, otherwise its a trivial change to make anyway.

Also, especially if you edit your control files manually, be sure to give full paths to files!! Otherwise, when you start MPI the slave machines will almost certainly start in your home directory, which might not be the one you started MPI from. So full paths are the way to go.

Back to the main page