A fact of life with VLBI has been that the data are recorded on tapes and tapes have various limitations. Tapes are not infinitely long and they cannot be stopped and started instantaneously. Whenever they are stopped, they must be resynchronized at the correlator, which takes a few to a few tens of seconds. Also tapes have finite bandwidth and capacity. The instantaneous bandwidth cannot be exceeded. This section provides some detail on various aspects of tape management.
Many of the concerns described here do not apply to the disk based recording systems (mainly Mark5) that began to be deployed in 2003. The disk systems can start and stop very quickly and take less time to be synchronized on the correlators. There still are limitations on the total bit rate and recording volume. But disk management is be much less visible to the user than tape management. For the rest of this section, we are talking about tapes. Hopefully by late 2005, this whole section can be relegated to a historical appendix to this manual.
The scheduler always has to worry about not exceeding the amount of tape that has been allocated for the project. For projects only including stations that use automatic tape allocation (eg VLBA, VLA, and GBT), that should be the only concern other than providing occasional gaps for readback tests, as described later, and not stopping the tape too often. Automatic tape handling is described in more detail below in the section on automatic tape allocation and in the description of the AUTOTAPE input parameter. If there are any stations at which automatic tape handling is not being used, the scheduler has a lot more to worry about. Tape reversals, without automatic tape handling, can only occur at scan boundaries. For efficient tape usage, the schedule must have scan breaks at appropriate intervals given the tape type and recording speed. More is said about this below, especially in the section on tape lengths.
The schedules sent to the antennas will also have to specify such details as track assignments, head positions etc unless automatic tape allocation is used. The user, however, should not need to worry about these details since SCHED has defaults that work fine in all but odd test observations. For the masochistic, the details of the SCHED defaults are given in Appendix B.5.
The maximum bit rate that can be recorded on a VLBA tape is 256 Mbps. The systems at the VLBA antennas and at the VLA and Green Bank have two drives and can use both simultaneously to achieve 512 Mbps. Mark IV systems can have 2 head stacks on a single drive to achieve 512 Mbps. In addition, they can record at 16 Mbps per track, twice the normal maximum rate. With both heads recording at 16 Mbps per track, the Mark IV systems can reach 1024 Mbps.
When part of an project uses the two drive (VLBA) or two head (Mark IV) mode, all of that project will be recorded in that mode. If some scans use narrower bandwidth, the tape drives will be slowed. This avoids lots of confusion in keeping track of tape usage.
There are a number of concerns related to tape handling at scan boundaries. SCHED provides a variety of ways to specify what is to be done. Between scans, tapes can be kept running or can be stopped. If they are stopped, they can be started at the nominal start time, or at some other time, usually earlier. The actions taken can strongly influence how well correlation proceeds. Whenever a tape is stopped, there is some time lost to resynchronize on the correlator. Also, with short segments of tape motions, it is possible for the VLBA correlator to wind up sufficiently far out of sync that it does not recover until either the end of the pass or the end of the correlator job. Basically, if the schedule consists of long scans, it doesn’t really matter much what is done. But if it has many short scans, such as when phase referencing, it can be highly advantageous to keep the tape moving between scans.
Obviously, tapes must be stopped at the ends of passes so that their direction of recording can change. They also must stop when they are changed. If there are long gaps (many minutes) between scans, tape use efficiency is enhanced if the tapes are stopped. If automatic tape handling is being used (normally true at the VLBA and VLA, but not elsewhere), tape stoppage at pass and tape ends will happen automatically and the user need not worry about them. The user should, however, provide occasional tape stoppages of at least 2 minutes (4 min in VLBA 512 Mbps mode) to allow for readback tests. These need not be at the ends of passes.
As the gap between scans gets shorter, or the scans become more frequent, it becomes advantageous to keep the tapes rolling between scans, assuming that the correlator does not require that they stop (the older Mark III system had such a requirement). One way to do this is to use DURation rather than DWELL to specify scan times and not give any command such as GAP that will cause an interval to be scheduled between scans. This simply butts the scans together. If there are gaps between the scans, such as when DWELL or GAP are used, the default behavior of SCHED will automatically keep the tape running if the gap is less than 8 seconds times the speed up factor (usually 2). That interval can be adjusted with the parameter MINPAUSE.
One can request, preferably with parameter PRESTART, that recordings be started before the nominal start of a scan. This can cause the recording medium to keep moving if the gap is short. For longer gaps, it can help optimize the amount of correlator output by providing time for the correlator to synchronize.
Tapes for the Mark III and VLBA systems come in two nominal lengths, 9600 ft and 18000 ft. Actually, nearly all of the shorter tapes have now been removed from the system so most users will not encounter them. The lengths vary plus the usable length is shortened by the amount of leader needed at each reel. Experience has shown that it is best not to count on more than 17600 feet for the long “thin” tapes. Most short “thick” tapes are over 9000 feet, but there are a significant number as short as 8500. It is probably reasonable to assume that the short tapes will be 8800 ft, half the length of the long tapes, for scheduling purposes.
There are three basic recording speeds to consider, at least when using the VLBA correlator. The “normal” speed it that at which 4 million data bits per second are recorded on each track. Mark III and Mark IV format data at low density are recorded at 135 ips (inches per second). VLBA format data at low density are recorded at 133.33 ips. All formats at high density are recorded at 80 ips, which is now the standard for most observations. Recording rates of half and twice these values are used when the number of bits per second per track is half or twice the 4 Mbps. High density recordings can only be made on “thin” tapes. The VLBA correlator plays back all observations at a track bit rate of 8 Mbps, using 160 ips for high density recordings. Thus there is a speed up factor for recordings made at lower speeds — a factor of 2 for 4 Mbps tracks and a factor of 4 for 2 Mbps tracks.
The following table gives the recording times per pass to expect with these tape speeds and recording rates:
When full automatic tape handling is not being used, scans should be scheduled so that they are in blocks that constitute a pass. This is not absolutely necessary, because, regardless of the scan times, SCHED will make sure that the scan will fit on the tape and, if not, will reverse the tape. If a scan cannot fit, SCHED will complain and die. Because of these actions, no data will be lost if the schedule is not in blocks of a tape pass, but some amount of tape will be wasted — it is better to be aware of the pass lengths. In fact, it is not uncommon for a user to find that, when he/she deletes a scan, the amount of tape used increases. This happens when a schedule that went to the end of tape on each pass gets out-of-sync and has to reverse the tape before the end. Once this starts to happen, it often continues for the rest of the experiment. Note that it is typical to round down the pass times to, for example, 13 minutes for those cases where the full number is 13:02 or 13:13.
Scheduling in blocks of tape passes is more complicated if there is a mixture of thin, high density tape at some sites and thick, low density tapes at other sites. For Mark III, the longer tapes will hold 3.37 times as much data per pass as the shorter tapes. Probably the best way to deal with this is to schedule in blocks such that the 2 blocks fit per pass on the thick tapes and 7 blocks fit on the high density, thin tapes. The appropriate time for this for Mark III or VLBA formats is 6:17. Be careful with scans less than 30 seconds in such a schedule because the passes could get out of sync. Mixed thick and thin tape observations should be rare, especially since the VLBA correlator refuses to deal with thick tapes.
Note that, if there is a problem at a station and a tape does not start on time, or a scan is missed, the tape position will be behind that expected by SCHED. At the end of the pass, the tape will reverse when the schedule tells it to, which in this case will be some distance from the end. For this reason, the next pass is likely to run out of tape before it is finished and some data will be lost. The pass after that will begin at the expected place and there will be no further problems.
Users of PCFS controled stations (eg EVN) should be aware that during long scans no Tsys information is acquired. It is necessary to insert gaps to make sure Tsys measurements are made.
The amount of data that can fit on a tape depends on the length of the tape, the tape speed, and on the number of passes. The Mark III and VLBA systems make longitudinal recordings using many heads, all mounted on a head stack. The width of each head is 38 microns and the “head pitch”, or spacing between heads, is 698.5 microns. This difference allows many passes to be recorded with each head simply by shifting the whole head stack over by something more than 38 microns between passes (48 microns is used on the VLBA). The Mark III systems typically use 12 passes while the VLBA and Mark IV use 14.
Many recording modes do not use all of the heads at once. Mark III systems use 28 heads while VLBA and Mark IV systems use 32 (they all actually use the same 36 head headstack so the head pitch etc is the same). Mark III Mode A uses all heads in a pass. Modes B and C use half the heads while Mode E uses a quarter. There are many VLBA modes, but they typically use a quarter, half, or all heads, although lesser options are available. If not all the heads are used in a pass, more passes can be made at the same head position, increasing the total time over which data can be recorded on tape. The setup file parameter TPMODE gives the number of passes that can be recorded at each head position. SCHED will figure it out if it is not given and will report it in the summary.
If multiple setups are used in a project, SCHED will figure out which uses the minimum number passes per head position (most tracks per pass) and will use that number of passes per head position for all setups. This wastes tape because some tracks never get recorded, but the alternative is a bookkeeping mess. Therefore, it is strongly recommended that all setups used in a project use the same number of heads per pass (have the same TPMODE).
It is perfectly possible to use setups that record at different speeds in the same schedule. As long as they all use the same number of heads, the tape will be used efficiently. However, a change of recording speed causes the VLBA correlator to need a new job. This is not a fundamental problem, but there is overhead for each job which causes work for the correlator staff. Current guidelines on how often changes of this sort can be made can be found in the guidelines for scheduling document found by going to the VLBA home page from the NRAO home page on the WWW. As of late 1996, changes more often than every 2 hours are discouraged. Note that using different tape speeds in different subarrays at the same time will preclude simultaneous correlation and will present the current job script making program with bookkeeping problems it is not equipped to handle. Don’t do it!
The total time that can be recorded on a tape is the time per pass times the number of passes per head position times the number of head positions. The following table gives the total time per tape, and also the total bit rate, for various combinations of these parameters.
For mostly historical interest, below is the old version of the table which includes the tape times for the low density and/or thick tapes. The 512 and 1024 Mbps modes were not available at the time.
If the tape allocated is insufficient to record for the whole allocated time at the desired bit rate, there are several options. The most common is to use a duty factor of less than one — to stop the tape for some of the time. Otherwise, it is necessary to cut down the bit rate or to request more tapes (good luck). Some periods of stopped tapes are desirable for readback tests.
One way in which the performance of the recording system is monitored is to read back some data from the tapes and check the playback quality. On the VLBA and Mark IV systems, this is done any time there is a gap of longer than about 2 minutes per tape (actually somewhat less). If a really bad head is found, it is possible to make a copy of the data from that head on one of the system tracks (recall that the headstacks have 36 heads). Some day, the correlator will be able to use this data, although not yet. On the VLBA, where there are two tape drives available, it may be possible to switch to the other drive to avoid the problem. It is in the interests of both the individual project and of overall system maintenance for occasional gaps to be inserted in all projects for readback tests to be done. These gaps are often inserted at the ends of passes when automatic tape allocation is not being used.
Readback tests on the VLBA systems cannot be done on both drives at once so, when using the 2 tape, 512 Mbps mode, a gap of a bit less than 4 minutes will be required to complete a set of tests.
SCHED will look for gaps long enough for readback tests. It will report the number for each station in the summary file. If there are less than one every 2 hours, it will complain, but go ahead and make schedules anyway.
If a project is being scheduled that uses stations that have only one tape drive, it is necessary to schedule in a gap of about 15 minutes for that tape change to be accomplished. Some stations can do it faster, but it is best not to count on it. The VLBA stations and the VLA all have 2 tape drives each and the project will switch from one to the other when the first runs out. Then the operators can change the full tape at their leisure. For these stations, it is not necessary to schedule tape change times. SCHED will complain, but not die, if a single tape station is requested to change tapes in less than 15 minutes. The SCHED input parameter TAPE is used to force tape changes. It can take station names as arguments if you only want to change tapes at a subset of stations.
An additional concern is that postpasses are needed for thin tapes. This can consume up to 22 minutes. See the beginning of this section for more details.
Sometimes it is a good idea to synchronize tape changes at at least some of the sites. For example, at least in late 1996, the VLBA correlator will not process the time between tape changes at different sites if they are less than something like 3 minutes apart. For optimized schedules, this can present a bit of a problem, so the input parameter TAPESYNC is provided to try to cause all stations that are near the end of their tape to change when the first reaches the end. With automatic tape allocation, the user does not have control of this factor and should not worry about it, although it might cause some loss of data.
It is now considered necessary to postpass VLBA/MarkIV thin tapes after recording. This means winding the tape all the way to the end and back to the beginning without stopping. This avoids irregularities in the tape pack that could otherwise lead to tape damage in handling and shipping. These irregularities are caused by starts and stops. Since tapes cost over $1000 each, this is a serious issue. SCHED will give the necessary commands to request postpasses be done at stations that use VLBA control files.
Unfortunately, a postpass takes between 11 and 22 minutes depending on how far the the tape has to be fast-forwarded to reach the end. This is down from a maximum of 33 minutes prior to late 1997 when it was determined that rewind speed could be used. For two tape stations, this is not a problem. For single tape stations, it is can be serious since it adds this much to the tape change time. Also, if there is an inadequate gap in the schedule, the next tape can be out of the expected position on the first pass which can cause additional losses when the first reverse pass runs out of tape early. SCHED provides a way to avoid postpasses by keeping track of tape motions. If the last pass went from the far end of the tape to the beginning without stopping, a postpass is not required and is not requested by SCHED. Therefore, users with single tape stations (all except the VLBA and VLA) who are using more than 1 thin tape per station are advised to not stop the tape on the last pass before each tape change.
Automatic tape allocation is provided on the VLBA. In fact, its use is required for most observations. With automatic tape allocation, the only thing the user needs or can worry about is the total tape consumption. It much be kept below that allocated for the experiment. There are two reasons for the use of automatic tape allocation. The first is to relieve the user of all the worry about tape handling. It is no longer necessary to schedule everything in units of passes. The second is to decouple the tape change times from the times allocated to each project. This allows the tape changes at the stations to be done on a fixed schedule that is the same every week and which fits well with the work schedule for the site techs. This does cause tapes to contain data from more than one project, but the bookkeeping is in place to insure that they don’t get erased until all the projects are correlated. Recall that the VLBA stations are unmanned most of the time. The second reason is why the use of automatic allocation is required.
Automatic tape allocation is not available for all stations, including all that use VEX files. SCHED knows which stations can use it and schedules accordingly. For global observations to be correlated on the VLBA correlator, the scheduler needs to worry about tape handling at the stations without auto allocation, but can ignore it for the ones that have it. For any observations to be correlated on other correlators (including JIVE), automatic tape allocation cannot be used and SCHED will not allow it.
Two levels of automatic tape handling are available on the VLBA. They are invoked using the parameter AUTOTAPE. The lower level allows the on-line system to decide where on the tape to start the next scan. However if the scan runs out of tape going in the current direction, the recorder will stop and the rest of the scan will be lost. After the first forward pass, any scan starting less than 450 feet from the end of the tape will be started in the opposite direction. The first forward pass always runs to the low tape sense to determine the tape length. It is the author’s opinion that, because of the variable length of tapes, projects scheduled with this scheme are very likely to exhibit unexpected behavior with possible considerable loss of data. It should work ok for long scans that intentionally try to drive well beyond the end of the tape, but this is almost worse than worrying about specifying the tape behavior at schedule time. This level of automation is not recommended.
The higher level of automation is the method of choice for tape scheduling. It would be even better if the speed of synchronization on the correlator were improved, but that project doesn’t seem to be getting any priority. With this mode, whenever the tape gets to an end, it stops and reverses. The minimum time lost is the time required to reverse the tape at the correlator, which is about 5 seconds times the speed up factor. However some time to resync after the reversal is required and the actual time is more like 10 seconds times the speed up factor and somewhat variable. For most experiments the data loss should not be a significant problem. Scans of arbitrary length can be scheduled. The user only need to worry about not exceeding his/her total tape allotment. The total amount of tape used in a schedule is reported in the summary file. Note that the same behavior, where scans starting within 450 feet of the expected end of tape after the first pass will start in the opposite direction, also applies with this level of automation. 450 feet is 67.5 seconds at 80ips — the speed that gives a speed up factor of 2 on the VLBA and is used by a large fraction of observations.
When automatic tape handling is requested, as is required for most VLBA projects, it is futile to worry about coordinating scans with tape passes. Don’t bother to try. There are two reasons for this. First, the tape position at the start of the project will depend on where it was left by the previous project. It will not necessarily be at an end. Second, the autoallocation software uses the real length of the tape as measured on the first pass. Tapes vary in length by several hundred feet depending on their history of abuse. So you cannot know ahead of time where a scan will be on the tape so there is no point in trying to schedule passes unless some antennas in the array are not using autoallocation.
If a station that is using automatic tape allocation is scheduled in a scan on a source that is below the antenna hardware limits (not just the horizon), that antenna will be removed from the scan by SCHED. This will not happen when or where automatic allocation is not in use because the tape management could be messed up. This action is provided mainly to help the Mauna Kea site technicians who have a very long drive to change tapes and whose antenna is often scheduled in scans for which the source has not yet risen. Mark5 stations that are not using tape will also be scheduled in this manner. This behavior can be overridden using DODOWN