The BDF specification document in its present form attempts to provide both documentation of the format being written currently by the ALMA-BL and ACA correlators, as well as provide the basis of the format that will be used by the EVLA WIDAR correlator and other applications (e.g., the ALMA total power detectors). Unfortunately, in its present state, it is a hybrid: adding some extensions not used by ALMA's correlators, while also not being an efficient or completely adequate definition for future applications. The priority in the current revision has been to document the format currently in use by ALMA. Please note that, as a consequence, the documented version does not conform exactly to either of Francois Viallefond's format versions 096 or 097. It is expected that after this version of the format has been approved, it will be revised soon thereafter to address the deficiencies. These revisions are expected to have minimal impact on the output produced by the current applications while improving certain features of the format for efficiency, and to improve support for other applications and/or use cases. Highlights and unresolved issues: 1) The description of the size of the "POL" axis will be extensively revised. 2) The description of the "dimensionality" and "numTimes" elements will be extensively revised. 3) Zero lag components, elements and attributes will be made fully optional to support the ACA correlator. 4) Rationalization of data types will be deferred to a later BDF version. 5) A glossary will be provided. 6) Baseband names, spectral window names, sideband and sideband image ids: should they occur in the BDF? 7) Is an "UNKNOWN" value required for "CorrelatorName" enumeration? 8) Sideband (pair) identification: should it be in the BDF? 9) Are elements of ZERO_LAGS always real-valued, always complex-valued, or can the choice be BDF-instance dependent? 10) Should baseline ordering (BAL axis) be changed? 11) Should ACTUAL_TIMES be stored as a time difference relative to time stored in the SDM to save space? 12) Should execBlock element be optional (rather than mandatory)? Received comments are detailed below with identifying numbers. Responses to comments below are introduced by a line of three asterisks, and terminated by a line of dashes. =========================================================================== EVLA Comments ------------- 1. II. General comments, concerns, and questions A. Terminology * Should use sub-array rather than "set of antennas" (e.g., p. 3, 3rd para. -- some confusion as to whether "set of antennas" here meant "subarray" or "subset of antennas within a given subarray") * Need a glossary, defining some of the terms: - sub-integration, integration, sub-scan, scan - URI *** A glossary will be created specifically for this document. Translations of the terms as used in the BDF specification document to those normally used by each project will be provided in the glossary as well (where needed, e.g., "subarray"). [JP] [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 2. * For ALMA, the initial correlator is variously referred to as the ALMA-B, ALMA-BL, and baseline correlator. Pick one term and stick with it! B. Some of the writing is unnecessarily opaque, confusing, and hard to read. * p. 3, 2nd para., 2nd sentence *** The second paragraph has just one sentence; perhaps the comment refers to a sentence in the third paragraph. Nevertheless, we agree that some of the writing is opaque, and we will try to improve it. [JP] ALMA-B replaces ALMA-BL. ALMA_BASELINE_XXX is used is some enumerations which will remain. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 3. * p. 14, POL sizes *** This section will be clarified and improved. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 4. * p. 19, dimensionality or numTimes *** This section will be improved. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 5. * p. 22 footnote -- what the heck does this mean? *** This will replaced with some further explanatory text in section 2. Related to EVLA's comment E, below. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 6. C. Does the SDM know the URI only of the main data header (subscan)? or are there direct references to the (sub-)integration headers etc.? *** This issue will not be addressed by this document. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 7. D. Various items seem out-of-place in the BDF, and would seem more logically placed in the SDM - NetSideband (p. 9) *** This table summarizes the enumerations used in this document which define their basic meaning. Their contextual meaning can be found in the appropriate sections in the document which use the specific enumerations. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 8. E. Should add an introductory comment on multiple occurrences of the same string as a binary component, a header element (shape of the binary component), and a URI (e.g., FLAGS, flags, and flags). This can be quite confusing! *** Will provide explantory text in section 2. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 9. F. The sometimes-use of CROSS_DATA to include auto- as well as cross-correlation data was not very clear. - Should explicitly state that you can store auto-corr'ns EITHER separately (in AUTO_DATA) or together with the cross-corr'ns (CROSS_DATA); make it clear why you would do one or the other; and explicitly discuss how you (a reader) could tell which had been done. *** Accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 10. - How do you know whether this is being done or not? (Seems to be the existence of a "ANT, BAL" axis) *** Will add explanation to section 5.4. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 11. - When both are stored as CROSS_DATA, do you store (e.g.) 5R x 5L as well as 5L x 5R? or what? - p. 13, Sizes: note that ANT,BAL gives (presumably) N*(N+1)/2 values, rather than N*(N-1)/2 *** Accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 12. G. It would be helpful to give some idea of WHY various obscure choices were made. - How do WEIGHTS differ from ACTUAL_DURATIONS? *** Will add text in sections 5.1 and 5.4. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 13. H. Should explicitly state, somewhere high up, whether times are UTC or IAT or local or what. I. actualTimes are not needed if you use a reasonably short dump time (which can be quite long for small arrays). - The EVLA will likely not use actualTimes, esp. in the first few years. *** The time basis is defined in the SDM, and all times recorded in the BDF will be expressed accordingly. The ACTUAL_TIMES binary is an optional component in all cases, so no changes are required. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 14. J. It might be worthwhile to declare the length of each binary element in the header, rather than inferring this by tracing the structure. This would let a partially implemented or an obsolete reader find what it wants without having to understand all possible structures. *** MP The size information for each binary component is already included in an attribute of the associated element in the main data header. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 15. III. Page-by-page comments pp. 8-10, 3.1 Enumerations * AxisName - Why is TIM here? How is TIM to be used in this context? *** TIM is used in the 'axes' attribute of the 'dimensionality' element. No explanation for the use of any of the AxisName enumerations is given in the table, so TIM is not exceptional in this regard. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 16. - Holography does not seem to be fully defined: : What lies along the HOL axis? Positions, phases, ???? ...Why is a separate HOL axis required? Can't holography data simply be stored as visibilities? : Why is HOL the fastest-changing axis in the list, rather than one of the slowest? *** Holography is not fully implemented in this version of the specification; it will be addressed in the next revision. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 17. * BasebandName - Seems odd to have what appears to be a hard limit of 8 *** BasebandName is an enumeration, not a simple integer; some limit must occur. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 18. - Why not start at BB_0? *** BB_1 is no worse than BB_0! In ALMA, we agreed to use 1-based numbers and to hide 0-based indexes from end users. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 19. * CorrelatorName - Add GBT_SPECTROMETER, GBT_DBE - Add DIFX (VLBI software correlator) *** Rejected. Other values can be added in future revisions. [MP][JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 20. - Add UNKNOWN or OTHER, as a catch-all for correlators we've not thought of yet *** These values are only required if the format is forever fixed, which is not the intention. At this time, these values are not needed. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 21. * NetSideband - Should be in the SDM, not the BDF - What is NOSB? *** Will add description of NOSB use in dicussion of 'sideband' attribute of 'spectralWindow' element. A note indicating that NOSB is equivalent to "not specified" will be added to the table. NetSideband assists in understanding the frequency order of the data and is very ALMA-specific. For EVLA NOSB can be used. [MP][JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 22. * SpectralResolutionType - FULL_RESOLUTION is also used when frequency averaging is done in the correlator, which is confusing - Glad to see this will soon vanish! p. 12, Types * WEIGHTS - What exactly are these WEIGHTS? Is this just a form of data valid count, i.e., keeping track of the number of bits correlated? If so, how does this differ from ACTUAL_DURATIONS? *** Will add descriptive text to sections 5.1 and 5.4. FULL_RESOLUTION & CHANNEL_AVERAGE will be removed in next revision. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 23. * ZERO_LAGS - Why do we care about these for the EVLA? *** Will add an EVLA-specific note about ZERO_LAGS. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 24. pp. 13-14, Sizes * ANT,BAL gives (presumably) N*(N+1)/2 values, rather than N*(N-1)/2. Do you store both (e.g.) 3R x 3L and 3L x 3R, even though they're redundant? * POL: contrast of "polarization" and "polarization product" flagging is confusing at best. - Do you mean antenna- vs. baseline-based flagging? - Presumably, for pol'n (R or L)-based flagging, the POL axis should be 2 if the PolProduct lists have more than ONE item (i.e., RR, LL). - Should spell out when the POL axis size is 3 (presumably auto-corr'ns, if stored separately (?)) *** POL axis size description will be rewritten. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 25. pp. 14-15, Element Ordering * ANT - EVLA also refers to antennas as strings. But why is this relevant? - The antenna names/orders (also baselines!) are really an agreement between the BDF and the SDM. The correlator has no idea what the numbers mean, and in the EVLA case at least the strings are not passed to the CBE (which writes the binary data). If all you get are integer identifiers, as implied when you say "integer order", it's someone else's job to ensure that that order matches up with the string names or whatever. *** The reviewers' comment was presumably inspired by the ALMA-specific note in the text. Nothing in the common parts of the document need to be changed. [MP] The ALMA-specific note will be removed. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 26. * BAL - The rationale given seems insufficient to justify such an odd order for the baselines. Sure, this makes it slightly easier to add antennas, but why is that worth worrying about? It would seem more sensible to stick with the usual (1,2), (1,4), (1,7), (2,4), (2,7), (4,7). - The discussion of "BAL, ANT" was confusing. pp. 15-17, Components * Should explicitly state (if it's true!) that an element in the main header gives the shape a binary component will have, if that element subsequently appears (it need not). - If that shape is empty, no such binary component can appear in this sub-scan. - If the shape is given in the main header, the corresponding binary component *might* appear. *** Accepted. Will add explanatory text. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 27. * FLAGS - Which value means "flagged" -- one (true) or zero (false)? *** Flags are bit-wise values with a '1' in a specific bit position representing the source or cause of the flag (true). A zero means not flagged (false) [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 28. - Should explicitly point out that the different bits in the 32-bit integer correspond to different conditions (reasons for flagging). *** Accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 29. - Outside this review, but of interest: Are these just the flags from the correlator, or do they include flags from monitor-and-control as well (e.g., LO not locked)? *** No action. A flag inserted in this attachment is relayed to the correlator if it affects the quality of the correlation data. The exact items to be flagged are still TBD. There will be a code development effort in the coming months to specify these flags with an update to the BDF document as appropriate. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 30. * ACTUAL_TIMES - Why not write difference between this and the time stored in the SDM instead, to save space? - Why are ACTUAL_TIMES required at all? The differences in imaging are negligible, if you dump with any reasonable time resolution. *** The ACTUAL_TIMES binary is never required. In normal operations it is expected that there will be no blanking & thus this attachment won't be present. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 31. * ACTUAL_DURATIONS - Note that the use of integer nanoseconds introduces problems with rounding, since the "tick" in WIDAR at least is not an integral number of nanoseconds. The Correlator Backend software will have to be careful these errors do not accumulate. *** Noted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 32. * ZERO_LAGS - These must be complex rather than real for cross-correlations. *** Open issue. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 33. * AUTO_DATA - Need to specify the order in which you store the parallel and cross-hands, and which of the two possible cross-hands are stored. *** Accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 34. - Should be more explicity about when you use real vs. complex values -- e.g., what happens if you're writing all three of RR, LL, and RL? Intrinsically RL is complex while RR and LL are real, but it's not clear how those three should then be stored. *** Accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 35. pp. 18-22, Main data header * Embedding the archive name in the BLOB means that you have to modify every BLOB if you move the binary data from one place to another. This seems worrisome, and prone to error. * byteOrder - Should list all possibilities, not just the one ALMA likes. *** "byteOrder" comment: accepted. The Archive subsystem should provide tools to move data sets from one location to another and change the archive name. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 36. - Why IEEE_Low_Endian? What does Low_Endian have to do with IEEE? - Little_Endian and Big_Endian seem more standard choices. *** Will change enumeration values to "Little_Endian" and "Big_Endian". [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 37. * dimensionality or numTimes - This is incompletely specified at best. Either leave it out entirely (preferably) or state explicitly that this is reallY for future development. *** It is not entirely for future development, as the numTimes element is currently used for ALMA's total power data. This feature may be under-developed or confusing, but it is required. This section will be revised. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 38. * execBlock - Again, if you move the SDM you'll have to change this URI, which seems painful. - Maybe make this optional? It's not clear that it's needed. * numAntenna - "Typically, this is the number of antennas in the (sub-)array." When would it *not* be? *** Will delete "Typically, ..." sentence. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 39. * correlationMode - How about a new correlationMode, CROSS_WITH_AUTO, meaning that the cross- and auto-corr'ns are stored together in CROSS_DATA? while CROSS_AND_AUTO would imply that the two are stored separately, i.e., we get both CROSS_DATA and AUTO_DATA. * dataStruct - Please say directly that this defines the shapes of the binary components! *** "correlationMode" suggestion is rejected. "dataStruct" suggestion is accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 40. * ALMA note at top of p. 21 just trails off -- clearly some missing words there. * spectralWindow - sd/crossPolProducts -- why is there an ordered list, when the order was already defined in the earlier table? *** Comment about "sd/crossPolProducts" will be addressed in extensive revision of text regarding polarization products. [MP] [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 41. - Many of these attributes seem more logically put in the SDM (order of pol'n products; id; image; sideband) since they're only important for the scientific interpretation of the BDF, in the same way as the channel<->freq. conversion given in the SDM. * p.22 footnote: very hard to understand! *** Footnote will be deleted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 42. * weights - Should probably pad each binary component out to a multiple of 8 or 16 bits -- since weights are pointers into a lookup table, the weights component might otherwise muck up byte/word boundaries. *** This is the already the case, but a clarification will be added. The MIME format provides for the use of the "padding" parameter in case pad bits are needed at the end of a binary MIME part. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 43. pp. 23-25, Data subset header * schedulePeriodTime - Shouldn't this be in the SDM, rather than here? * abortObservation - There was some disagreement over whether this was needed, but it doesn't seem like a terrible idea anyhow. * crossData - Should add an EVLA note that we'll be using FLOAT32_TYPE to begin with. *** Accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 44. p. 28: Note "Error! Bookmark not defined.FLAGS" *** Noted. [MP - This is a PDF generation error] ============================================================================== Lindsey Davis Questions / Clarifications -------------------------- 45. Introduction, page 3, paragraph 4 (Not ALMA related) Are the project models for ALMA and EVLA aligned ? Specifically is an ALMA subscan the same thing as an EVLA subscan conceptually? Likewise for a scan but the idea of a subscan is directly related to the ALMA and EVLA BLOBs. *** These questions will hopefully be sufficiently addressed by the inclusion of a document glossary. [MP]{JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 46. 1.1. Context, page 3, 4 (Not ALMA related) For the EVLA do the binary blobs contain all the information required by Telcal? This is not the case for ALMA where Telcal uses both the meta data from data capture and the binary data. *** No response. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 47. Highlevel Organization, page 6 and 7, example page 30 and 31 I find the use of desc.xml in the subset MIME headers a little confusing but this may be due to my MIME / XML standards ignorance. Why is this not something specific like sdmSubsetHeader.xml as is the case for the main header ? *** Rejected. May be reconsidered in future revision. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 48. Enumerations, page 8 When I look at the enumeration as a whole I am struck by the fact that the AxisName enumerations are all limited to 3 characters instead being defined for clarity, e.g TIM instead of TIME, BAB instead of BASEBAND. Is there a historical reason or existing standard for this? I am NOT suggesting that this be changed, but I am curious as most of the other quantities don't have this limitation. The polarizations are all 2 characters but that seems logical and is based on FITS standards. *** No action. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 49. 6.1 Main Data Header, page 18 A number of quantities are defined in terms of "TBD" which depends on which archive the data is stored in. Will this be silently changed as data gets replicated to other archives, e.g the ALMA regional archives ? If so is this the only thing that is changed ? *** The specific archive is independent of this document. TBD is just a holding place for the actual archive name and other specific information. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 50. 6.1. Main Data Header, page 18 Will the byte order be fixed for all time or will it change if the correlator machines change in order to say avoid byte swapping. *** If byte ordering is swapped to big-endian, then it will be noted here. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Detailed -------- 51. 1.1 Context, page 3, 4 The EVLA note should probably be grayed out in the same way the ALMA note is. *** The reviewer probably printed the document in B&W rather than color. The EVLA notes do have a colored background. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 52. 3. Enumerations, page 9 INT64_TYPE and LONGLONG_TYPE seem to be duplicate definitions of the same thing. Is this correct ? *** Yes, both are required for compatibility with ALMA and EVLA. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 53. INT_TYPE. The Java type is missing. IDL primitive integer types. I am little confused by the idl integer mappings. I don't think there is an IDL int only short, long, and long long (and their unsigned equivalents) which map to short, int and long in Java? Is this correct ? *** This XML + binary will never be translated to IDL. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 54. 4. Data stream types, page 11 I don't understand the meaning of the footnote at the end of the page especially as it is referenced in the description of the sample example element. *** The note on zeroLags will be removed. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 55. 5.3.2, page 13 In the ALMA note should "one integration" be "one integration or sub integration" or have I misunderstood. *** Yes, this should include one sub-integration interval. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 56. 5.3.2 Sizes, page 14 I find the description of how the size of the polarization axis is computed for the FLAGS binary data confusing. The text says the size depends on how the flagging is done by polarziation or by polarization product. How is this determined from the data ? The text says that the polarization axis may take the size 1,2,3,4 for the WEIGHTS, ACTUAL_TIMES, and ACTUAL_durations. How is this determined from the data? Note: Some of this is explained in the appendix containing the equations but I still think the description is a bit confusing. *** The description of the size of the POL axis will be improved. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 57. 5.4 Components, page 16 and 17 How do you know from the data that the AUTODATA are real or complex, e.g. contain 1 or 2 FLOAT_32 type numbers. Later on I found something about this in the appendix but it might be useful to clarify this here. *** Will add short reference to 'sdPolProducts' to explain how to determine whether real or complex numbers are stored (in whatever combination) in AUTO_DATA. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 58. How do you know from the data what N is for the WEIGHTS ? *** Will add reference to 'weights' element to explain how N is determined. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 59. 6.1 Main Header, datastruct, page 21 The ALMA note is missing information for the ACA correlator. --- This is a typo that will be removed. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 59a. 6.1 Main Header, datastruct, page 21 Must the order of the child elements flags, actualTimes, ..., weights correspond to the order of the associated binary components in the data and in the subset header? Is the order of the associated binary components fixed in the BLOB? *** Neither these elements nor the binary components have a fixed order, nor are their orders correlated. Will add a brief explanation of this to section 6.1. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 60. 6.1 Main Header, spectralWindow, page 21 The spectral window id is defined to be spw_N, where N is an integer. In the example on page 31 the spectral windows listed are not in sequence of N, 2 comes before 1. Is this ok ? Is the scaling factor normally different for each spectral window? *** The scaling factor can be different for each spectral window because the scaling factor is a function of the spectral window bandwidth. spectral window ids can appear in any order, and N can take any value. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 61. 6.1.1 Data Stream Dependencies, page 23 Why are flags optional for BASEBAND_WIDE autoData ? I am little confused about the relationship between mandatory and optional main header / subset header / binary components. Take the example of flags. The flags element is mandatory in the main header. I think this means that it must be present there even if there are no flags binary components in the data. Is this correct ? Within each subset container header a flags binary component may exist. However if one does not exist does this mean that the subset header flags element does not exist or that it is undefined, e.g. = ""? *** Will provide further explanation of relation btw. binary components and two header elements in section 2. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 62. 7.4.1.1 Binary component sizes, page 28 and 29 The definition of standard and non-standard mode is unclear. There is still an Error!Bookmark not defined.FLAGS in the document. The flags formula looks incorrect? See equivalent channel averages formula. It would be very useful to summarize the limits used to compute the minimum and maximum quantities in the size table. I tried to do this and sometimes I could approximate the numbers and sometimes not, probably because my input limits were incorrect. The size table needs a caption. *** For ALMA , CROSS_ONLY is a non-standard mode. I will update accordingly. I can add values used for the tables. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 63. 7.4.1.2 Binary component sizes, page 32 and 33 The ACTUAL_TIMES and ACTUAL_DURATIONS still contain Error!Reference not found statements. For the channel average data will the number of channels per spectral window default to 1 in the CROSS_DATA and AUTO_DATA formulae for ALMA data or will they sometimes be other than 1. The example XML files has numSpectralPoint values of 7 and 8, but I think most of the data I have seen so far at least in simulation has numSpectralPoint equal to 1? See previous comment on defining limits for computing minimum and maximum sizes. The size table also needs a caption. *** There can be up to 10 channel average regions per spectral window with each region providing a spectral point. In our tests SDMs, we have defined only one channel average region. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 64. 7.4.2.3 Data rate This table has a caption but it is out of order in the document because other tables are missing captions. It might be useful to make the presentation of this section consistent with 7.4.1.3. *** Captions will be fixed. [JP] ============================================================================== Brian Glendenning 65. p.x [ALMA] I assume the contents of this document describe the correlator software which will be merged from the ACA branch, and is expected to define the 5.1 release. Please confirm. *** Yes. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 66. p.x It would be useful to list the "overhead" the format takes over a pure binary representation without overhead. I'm guessing it's very small, but it would be nice to see it listed. *** I can add size information for the XML headers along with the binary size, but it will be insignificant. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 67. p.3 s1 baseband-wide detectors -> baseband-wide continuum detectors. *** Accepted. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 68. p.3 s1 You should describe the hierarchy dump -> scan, one bullet for each level. *** This will be resolved by the glossary (for the most part). [MP][JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 69. p.3 s1 Here or somewhere you should describe the relationship between the document version and the format version. *** Accepted. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 70. p.5 [ALMA] Future revision should hold examples of all ALMA uses of the BDF (in an appendix). Example headers are the best feature of FITS convention documents. *** Specific header examples are provided at the end of the document. Each version of document should go with a specific version of software with software release notes specifying the BDF version. [JP]. An overview example is also provided at the beginning of the document to assist in its readability. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 71. p.6 Does the fact that MIME boundary strings can occur in the octet stream mean that formally our MIME messages do not comply with the standard? Is there anything to be done about it, e.g. escape sequences? *** Formally, the MIME messages cannot be guaranteed to comply with the standard unless the binary components are scanned for unintentional boundary strings. In practice, if the boundary strings are long, the chances of a string appearing in a binary component will be small. I intended for this to be clearly stated; I will add some further text. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 72. p.6 last bullet: I'd really like an example in here. *** Will add an example. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 73. p.9 primitive data types. Why do we have (e.g.) both INT64, LONGLONG. Seems like unnecessary duplication. Similarly, why do we have undefined lengths (I assume INT_TYPE could be 32 or 64). I would only have the INT16, 32, 64 an unsigned variants. *** Redundancy is needed for compatibility with current ALMA and future EVLA needs. These will be rationalized in an upcoming version. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 74. p.16 ACTUAL_TIMES and other places. Where is the time system recorded (TAI vs. UTC vs. TT vs. TDB vs. ALMA Array Time...). *** The time base is defined in an SDM table external to these binary blobs. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 75. p.18 By inference is EVLA Big-Endian? As an aside, IEEE_Low_Endian seems like a bad name because, e.g., integers are not IEEE values! Maybe MSB_FIRST or LSB_FIRST would be clearer? *** Will use "Little_Endian" and "Big_Endian" [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 76. p.19 An example would have helped me here - I'll have to take your word for it :-) *** The "dimensionality or numTimes" section will be rewritten. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 77. p.20 Is execBlock also an EVLA construct? *** Unresolved. [JP] execBlock will be optional. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 77a. p.26 Will repository access instructions go here before v.1 is finalized? *** Yes. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 78. p.28 Missing bookmark *** Noted. [MP - PDF generation error] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 79. p.32 7.4.1.3 The SSR data rate should not constrain the data format; the data rate will be turned up over time. *** The data format is not constrained by the data rate. I can provide a maximum data rate which may never be allowed. [JP] ============================================================================== Kawase (Fujitsu) 80. 5.3.2 Sizes (page 13) In the blue hatched box, the ALMA note on the size of the TIM axis should be applied not only to ALMA-B correlator but also to ACA correlator. So, please make mention of ACA correlator. CROSS_DATA (page 16) In the blue hatched box, Please add a statement that "scaled 16-bit signed integer" is INT16_TYPE and "scaled 32-bit signed integer" is INT32_TYPE for clarity. *** Accepted. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 81. crossData (page 25) In the blue hatched ALMA note, it says "Currently, ALMA only supports SHORT_TYPE and LONG_TYPE." but I can't find LONG_TYPE in the "Primitive DataType" in page 9. I assume it must be "Currently, ALMA only supports INT16_TYPE and INT32_TYPE." *** Accepted; will be corrected. [JP] ============================================================================== Manabu WATANABE 82. 4 Data stream type (page 11) | Any given data processor will likely produce data in only a small | subset of the nine possible types of data streams. -------- Where does the "nine" come from? *** Will be corrected. Twelve is the correct number. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 83. 5.1 Types (page 12) | ACTUAL_DURATIONS | total amount of time used in deriving the data Please consider to put an additional remark as "(exposure)" to the description to make it clearer (for me). *** Accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 84. 5.3 Axes (pp.12-13) | For example, if the Nth level of a tree is indexed by the SPW axis, | the N+1st level may be indexed by any one of the BIN, APC, SPP, or | POL axes, but not the TIM, BAL, ANT or BAB axes. --------- Is it needed to add "HOL" axis after "POL" axis? *** Noted for future revision. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 85. 5.3.2 Sizes (page 14) | - POL: For AUTO_DATA and CROSS_DATA binary components, the axis size (snip) | In the case of polarization-based flagging, this means that if the | sdPolProduct or crossPolProduct lists have more than two items, the | POL axis size is two, otherwise, it is one. Please consider to add the case of polarization products-based flagging. *** This section will be improved and clarified. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 86. 5.3.4 Element ordering (page 15) | BAL | In a matrix formed by labelling rows and columns with antenna (snip) | Finally, note that the order of the correlation products recorded | in the binary must agree with the order of antennas in each baseline. | For example if the baseline is (1, 2), the product must be 1 * 2, | not 2 * 1. Does the asterisk in the "1 * 2" and "2 * 1" mean a complex conjugate of the value? If so, I believe the complex conjugacy of the visibility might depend on the sidebandness. *** No, 1 * 2 simply means the correlation of antennas 1 and 2, in that order. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 87. | SPW | Order of spectralWindow elements within a baseband element in the | main data header Please add the following note for ACA correlator: ALMA note The ACA correlator has following constraints on the order of SPW: 1. The spectral windows should be arranged in ascending order of the first channel of the spectral window in terms of 3.8..HKz channnels. 2. Two spectral windows should be defined contiguously if these spectral windows are in the image sideband to each other. LSB first and USB follows in that case. *** Accepted. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 88. Page 21. In a blue hatched box, there is an unnecessary fragment of "The ACA correlator" at the end of the note. *** This is a typo & will be removed [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 89. Page 23. In the table 5, zeroLags in the AUTO_ONLY and CROSS_AND_AUTO for FULL_RESOLUTION should be italicized because ACA correlator does not produce any zero lags. *** Accepted. The BDF should be sufficient for the ACA correlator. Annotations should be made when the BL correlator deviates from the ACA correlator. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 90. Page 31. In the "7.4.1.2 Example of Full Resolution Data", there are four spectral windows defined in a baseband. Each of the spectral window has 7680 spectral points so that the sum of the number of spectral points for all spectral windows in a baseband excesses the maximum number of the spectral points for a baseband, i.e., 8192. *** This will be fixed. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 91. Page 31. In the "7.4.1.2 Example of Full Resolution Data", I don't understand sideband="NOSB" in the case of ALMA. *** I believe that Band 1 and/or 2 is a no-sideband receiver. [JP] ============================================================================== Francois Viallefond Partr I: Some detailed comments 92. Section 1.1 ALMA note: --------------------- Add something like: Several other data products are produced along the signal path, total powers from square law detectors, 1/ 8 GHz wide total powers at the levl of the first downconverters, 2/ 2GHz wide total powers at the level of the baseband processors, 3/ total powers sub-band wide at the level of the Tunable Filter Bank (TFB, only with the ALMA_B correlator), 4/ measurements from Water Vapor Radiometers (WVR) which are resampled by the Correlator Data Processors to use them for the atmospheric phase corrections, 5/ holography data. Although the format described in this document may host all these data, the current plan is to store only the items 2, 4 and 5. *** Agreed. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 93. section 2: --------- "but no such applications are known to exist"; for this reason this document does not describe the specification to support this feature. Third part, "For ALMA and EVLA ...: Remove "spectral" in front of data! this qualifier is not useful here. Removing that allows to cover the baseband-wide and holo data. *** For correlators, "spectral" does seem appropriate. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 94. section 3: --------- You must introduce the section by saying that you highlight here only a subset of the enumerators, those presently used in the context of the BDF. (==> drop all the unsigned in PrimitiveDataType) *** Accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 95. - PrimitiveDataType not concise; e.g. what is the difference between INT16-type and SHORT_TYPE? - Add HologragraphyChannelType enum *** Rejected for current version. Will be improved in following version. A consolidation of data type labels should be made. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Section 5.3: ----------- "This means that the when" ==> This means that when *** Accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 100. It may help the reader to say that this "important exception" is relevant only for the meta-data binary components. *** Accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 101. Section 5.3.2 in ALMA note: -------------------------- add the use case WVR/BBWTP/Holo The size of the TIM axis is the number of integration (multiplied by the number of sub-integrations per integration if necessary) for the WVR, baseband_wide and holograpy total power data. Add an item for HOL HOL: Size is equal to the number of enumerators appearing in the list of values of the holographyChannel attribute of the spectral window element. *** Deferred [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 102. Section 5.3.4: ------------- ANT The format constrains nothing. The order is set by the list of antennas in the SDM metadata. Note that it is defined as a list of tokens (not of integers). 1,2,5,10,12 will be interpreted as a list of antenna names. This comment is actualy unrelated to the SDM-BDF. *** Remove ALMA comment and add comment regarding antenna tokens & lists are external to the BDF in the ConfigDescritpion table in the SDM. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 103. End of the section 5.3.4: ------------------------ Index values are abstract values. They reflect something like 0,1,2,..,N For all radiotelescopes this abstract index is associated to an antenna name and this mapping is defined in the ConfigDescription SDM table where is given the list of antenna names. Indeed 1,2,5,10 may be names for the EVLA but they will be decoded as a sequence of words, each composed of one or two characters with this example *** See reply to previous comment. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 104. Antenna names are unknown in the BDF. So for BAL, you should give the axample based on a sequence of indices , eg {0,1,2,3} *** Accepted. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 105. There is nothing specific for the EVLA vs ALMA concerning the order of the spectral points. In both case the information about the actual values is in the SDM metadata. *** The SPP axis can be used by EVLA to record either frequency or lag spectra. The note is present to indicate this fact. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 106. ALMA notes: if there is one for SPP there should be also one for SPW saying that when there are two sidebands separated by the correlator (90 deg switch), they are contiguous. In both cases SPW and SPP, it is a convention between CONTROL and CORRELATOR *** Remove ALMA note regarding SPP. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 107. Section 5.4: ----------- ACTUAL_DURATIONS "Proportional to the number of valid samples". I do not understand the meaning of this! Note that, whatever the number of valid data, the size of ACTUAL_DURATIONs will always be the same! If there are blank data the actual value could of the actual durations could be set to a value of 0 but this is not required up to now. Knowing if data are unvalid relies on what is in FLAGS. Should we want to flag on the basis of ACTUAL_DURATION=0, this will need to be explicitly said. For aLMA we will revisit this topic in the context of a FBT about flags. *** It is the *values* of the durations that are proportional to the number of valid samples. Not the number of elements in the attachment. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 108. CROSS_DATA: ---------- the primitive data type names must be set in the context of the list of enumerators refactored to be concise. *** Will be fully addressed in a following revision. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 109. WEIGHTS: ------- Cannot be implemented with the current specification. I believe (hope) that it is agreed that it must be an integer number of octets. If this is acceptable then one need to add something like BYTE_2_TYPE, BYTE_3_TYPE, BYTE_4_TYPE etc. in the PrimitiveDataType enumeration to have a uniform way of knowing the size of a binary block. *** Rejected. The current specification is complete, even if a particular implementation is not. Text will be added in this section to address padding in the binary component. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 110. Section 6.1: ----------- xmlns: This must be 'universal' for the BDF, just like with the w3c namespaces. It cannot be specific to a given archive! Obviously we cannot force to have the EVLA data and the ALMA data in the same archive! Hence the namespace must be above the telescope dedicated archives. The archives will require a mapping or their own rules to find the schemas in their repository. The schema will need to be available and accessible (usualy via http) for any user at the URL corresponding to the namespace. *** Accepted. Will remove reference to archive repository. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 111. "0/sdmDataObject.xsd" This is inadequate for this document! 0 is the major schema version number. So according to Brian G. this document being associated to version 1 this leads to 1/sdmDataObject.xsd. This document should explain briefly how versioning works or at least refers to the document explaining the strategy adopted for versioning. *** A reference to the versioning document will be added, as will a short description of version number. All examples in the BDF document will conform to the version described by the BDF document. Replace '0/' with the BDF document version supported by a given XML document instance. [MP] [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 112. byteOrder: --------- As soon as a data provider would like to use something else, not IEEE_Low_Endian, it will be necessary to set this value un-fixed in the schema. Currently it is fixed telling that the implementations have to support only this. *** Rejected. There will be (at least) two choices in the schema. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 113. In this context the "ALMA note" is not necessary because it is not specific to ALMA. *** Rejected. Any restriction is a limitation in the implementation, not the specification. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 114. startTime: --------- By experience (question that I had several times) it seems useful to be more specific here! It is the midpoint-interval/2 of the first dump. *** Accepted. Will add a description. [JP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 115. dimensionality or numTimes: should be ... with one chunk per BLOB, ... I would help to say: "Note that the attribute numTimes cannot be used when there is more that one subset per BLOB" *** This section will be revised extensively. [MP] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -