Class to combine multiple files in a single HDF5 file. More...
#include <MultiHDF5.h>
Public Member Functions | |
MultiHDF5 (const String &name, ByteIO::OpenOption, Int blockSize=0) | |
Open or create a MultiHDF5 with the given name. | |
virtual | ~MultiHDF5 () |
The destructor flushes and closes the file. | |
virtual void | reopenRW () |
Reopen the underlying file for read/write access. | |
virtual void | fsync () |
Fsync the file (i.e., force the data to be physically written). | |
Private Member Functions | |
virtual void | doAddFile (MultiFileInfo &) |
Do the class-specific actions on adding a file. | |
virtual void | doDeleteFile (MultiFileInfo &) |
Do the class-specific actions on deleting a file. | |
virtual void | flushFile () |
Flush the file itself. | |
virtual void | close () |
Flush and close the file. | |
virtual void | writeHeader () |
Write the header info. | |
virtual void | readHeader (Bool always=True) |
Read the header info. | |
virtual void | extend (MultiFileInfo &info, Int64 lastblk) |
Extend the virtual file to fit lastblk. | |
virtual void | readBlock (MultiFileInfo &info, Int64 blknr, void *buffer) |
Read a data block. | |
virtual void | writeBlock (MultiFileInfo &info, Int64 blknr, const void *buffer) |
Write a data block. | |
Private Attributes | |
HDF5File | itsFile |
Class to combine multiple files in a single HDF5 file.
Public interface
This class is a container file holding multiple virtual files. It is primarily meant as a container file for the storage manager files of a table to reduce the number of files used (especially for Lustre) and to reduce the number of open files (especially when concatenating tables).
A secondary goal is offering the ability to use an IO buffer size that matches the file system well (large buffer size for e.g. ZFS).
The SetupNewTable constructor has a StorageOption argument to define if a MultiFile has to be used and if so, the buffer size to use. It is also possible to specify that through aipsrc variables.
A virtual file is spread over multiple (fixed size) data blocks in the MultiFile. A data block is never shared by multiple files. For each virtual file MultiFile keeps a MultiFileInfo object telling the file size and the blocks numbers used for the file. When flushing the MultiFile, this meta info is written into a header block and, if needed, continuation blocks. On open and resync, it is read back.
A virtual file is represented by an MFFileIO object, which is derived from ByteIO and as such part of the casacore IO framework. It makes it possible for applications to access a virtual file in the same way as a regular file.
It is possible to delete a virtual file. Its blocks will be added to the free block list (which is also stored in the meta info).
In principle it is possible to use the MultiFile functions directly. However, in general it is much easier to use an MFFileIO object per virtual file as shown below.
// Create a new MultiFile using a block size of 1 MB. MultiFile mfile("file.mf', ByteIO::New, 1048576); // Create a virtual file in it. MFFileIO mf1(mfile, "mf1", ByteIO::New); // Use it (for example) as the sink of AipsIO. AipsIO stream (&mf1); // Write values. stream << (Int)10; stream << True; // Seek to beginning of file and read data in. stream.setpos (0); Int vali; Bool valb; stream >> vali >> valb;
Definition at line 105 of file MultiHDF5.h.
casacore::MultiHDF5::MultiHDF5 | ( | const String & | name, | |
ByteIO::OpenOption | , | |||
Int | blockSize = 0 | |||
) |
virtual casacore::MultiHDF5::~MultiHDF5 | ( | ) | [virtual] |
The destructor flushes and closes the file.
virtual void casacore::MultiHDF5::close | ( | ) | [private, virtual] |
Flush and close the file.
Implements casacore::MultiFileBase.
virtual void casacore::MultiHDF5::doAddFile | ( | MultiFileInfo & | ) | [private, virtual] |
Do the class-specific actions on adding a file.
Implements casacore::MultiFileBase.
virtual void casacore::MultiHDF5::doDeleteFile | ( | MultiFileInfo & | ) | [private, virtual] |
Do the class-specific actions on deleting a file.
Implements casacore::MultiFileBase.
virtual void casacore::MultiHDF5::extend | ( | MultiFileInfo & | info, | |
Int64 | lastblk | |||
) | [private, virtual] |
Extend the virtual file to fit lastblk.
Implements casacore::MultiFileBase.
virtual void casacore::MultiHDF5::flushFile | ( | ) | [private, virtual] |
Flush the file itself.
Implements casacore::MultiFileBase.
virtual void casacore::MultiHDF5::fsync | ( | ) | [virtual] |
Fsync the file (i.e., force the data to be physically written).
Implements casacore::MultiFileBase.
virtual void casacore::MultiHDF5::readBlock | ( | MultiFileInfo & | info, | |
Int64 | blknr, | |||
void * | buffer | |||
) | [private, virtual] |
Read a data block.
Implements casacore::MultiFileBase.
Read the header info.
If always==False, the info is only read if the header counter has changed.
Implements casacore::MultiFileBase.
virtual void casacore::MultiHDF5::reopenRW | ( | ) | [virtual] |
Reopen the underlying file for read/write access.
Nothing will be done if the file is writable already. Otherwise it will be reopened and an exception will be thrown if it is not possible to reopen it for read/write access.
Implements casacore::MultiFileBase.
virtual void casacore::MultiHDF5::writeBlock | ( | MultiFileInfo & | info, | |
Int64 | blknr, | |||
const void * | buffer | |||
) | [private, virtual] |
Write a data block.
Implements casacore::MultiFileBase.
virtual void casacore::MultiHDF5::writeHeader | ( | ) | [private, virtual] |
Write the header info.
Implements casacore::MultiFileBase.
HDF5File casacore::MultiHDF5::itsFile [private] |
Definition at line 149 of file MultiHDF5.h.