Abstract base class to combine multiple files in a single one. More...
#include <MultiFileBase.h>
Public Member Functions | |
MultiFileBase (const String &name, Int blockSize=0) | |
Open or create a MultiFileBase with the given name. | |
virtual | ~MultiFileBase () |
The destructor flushes and closes the file. | |
Int | fileId (const String &name, Bool throwExcp=True) const |
Return the file id of a file in the MultiFileBase object. | |
Int | addFile (const String &name) |
Add a file to the MultiFileBase object. | |
void | deleteFile (Int fileId) |
Delete a file. | |
Int64 | read (Int fileId, void *buffer, Int64 size, Int64 offset) |
Read a block at the given offset. | |
Int64 | write (Int fileId, const void *buffer, Int64 size, Int64 offset) |
Write a block at the given offset. | |
void | flush () |
Flush the file by writing all dirty data and all header info. | |
void | resync () |
Resync with another process by clearing the buffers and rereading the header. | |
virtual void | reopenRW ()=0 |
Reopen the underlying file for read/write access. | |
virtual void | fsync ()=0 |
Fsync the file (i.e., force the data to be physically written). | |
String | fileName () const |
Get the file name of the MultiFileBase. | |
Bool | isWritable () const |
Is the file writable? | |
Int64 | blockSize () const |
Get the block size used. | |
uInt | nfile () const |
Get the nr of virtual files. | |
Int64 | size () const |
Get the total nr of data blocks used. | |
const vector< MultiFileInfo > & | info () const |
Get the info object (for test purposes mainly). | |
const vector< Int64 > & | freeBlocks () const |
Get the free blocks (for test purposes mainly). | |
Protected Member Functions | |
void | setNewFile () |
Set the flags and blockSize for a new MultiFile/HDF5. | |
Protected Attributes | |
String | itsName |
Int64 | itsBlockSize |
Int64 | itsNrBlock |
Int64 | itsHdrCounter |
vector< MultiFileInfo > | itsInfo |
Bool | itsWritable |
Bool | itsChanged |
vector< Int64 > | itsFreeBlocks |
Private Member Functions | |
void | writeDirty (MultiFileInfo &info) |
virtual void | doAddFile (MultiFileInfo &)=0 |
Do the class-specific actions on adding a file. | |
virtual void | doDeleteFile (MultiFileInfo &)=0 |
Do the class-specific actions on deleting a file. | |
virtual void | flushFile ()=0 |
Flush the file itself. | |
virtual void | close ()=0 |
Flush and close the file. | |
virtual void | writeHeader ()=0 |
Write the header info. | |
virtual void | readHeader (Bool always=True)=0 |
Read the header info. | |
virtual void | extend (MultiFileInfo &info, Int64 lastblk)=0 |
Extend the virtual file to fit lastblk. | |
virtual void | writeBlock (MultiFileInfo &info, Int64 blknr, const void *buffer)=0 |
Write a data block. | |
virtual void | readBlock (MultiFileInfo &info, Int64 blknr, void *buffer)=0 |
Read a data block. |
Abstract base class to combine multiple files in a single one.
Public interface
This class is a container file holding multiple virtual files. It is primarily meant as a container file for the storage manager files of a table to reduce the number of files used (especially for Lustre) and to reduce the number of open files (especially when concatenating tables).
A secondary goal is offering the ability to use an IO buffer size that matches the file system well (large buffer size for e.g. ZFS).
The SetupNewTable constructor has a StorageOption argument to define if a MultiFile has to be used and if so, the buffer size to use. It is also possible to specify that through aipsrc variables.
A virtual file is spread over multiple (fixed size) data blocks in the MultiFile. A data block is never shared by multiple files. For each virtual file MultiFile keeps a MultiFileInfo object telling the file size and the blocks numbers used for the file. When flushing the MultiFile, this meta info is written into a header block and, if needed, continuation blocks. On open and resync, it is read back.
A virtual file is represented by an MFFileIO object, which is derived from ByteIO and as such part of the casacore IO framework. It makes it possible for applications to access a virtual file in the same way as a regular file.
It is possible to delete a virtual file. Its blocks will be added to the free block list (which is also stored in the meta info).
In principle it is possible to use the MultiFile functions directly. However, in general it is much easier to use an MFFileIO object per virtual file as shown below.
// Create a new MultiFile using a block size of 1 MB. MultiFile mfile("file.mf', ByteIO::New, 1048576); // Create a virtual file in it. MFFileIO mf1(mfile, "mf1", ByteIO::New); // Use it (for example) as the sink of AipsIO. AipsIO stream (&mf1); // Write values. stream << (Int)10; stream << True; // Seek to beginning of file and read data in. stream.setpos (0); Int vali; Bool valb; stream >> vali >> valb;
Definition at line 134 of file MultiFileBase.h.
Open or create a MultiFileBase with the given name.
Upon creation the block size can be given. If 0, it uses the block size of the file system the file is on.
virtual casacore::MultiFileBase::~MultiFileBase | ( | ) | [virtual] |
The destructor flushes and closes the file.
Add a file to the MultiFileBase object.
It returns the file id. Only the base name of the given file name is used. In this way the MultiFileBase container file can be moved.
Int64 casacore::MultiFileBase::blockSize | ( | ) | const [inline] |
virtual void casacore::MultiFileBase::close | ( | ) | [private, pure virtual] |
Flush and close the file.
Implemented in casacore::MultiFile, and casacore::MultiHDF5.
void casacore::MultiFileBase::deleteFile | ( | Int | fileId | ) |
Delete a file.
It adds its blocks to the free block list.
virtual void casacore::MultiFileBase::doAddFile | ( | MultiFileInfo & | ) | [private, pure virtual] |
Do the class-specific actions on adding a file.
Implemented in casacore::MultiFile, and casacore::MultiHDF5.
virtual void casacore::MultiFileBase::doDeleteFile | ( | MultiFileInfo & | ) | [private, pure virtual] |
Do the class-specific actions on deleting a file.
Implemented in casacore::MultiFile, and casacore::MultiHDF5.
virtual void casacore::MultiFileBase::extend | ( | MultiFileInfo & | info, | |
Int64 | lastblk | |||
) | [private, pure virtual] |
Extend the virtual file to fit lastblk.
Implemented in casacore::MultiFile, and casacore::MultiHDF5.
Return the file id of a file in the MultiFileBase object.
If the name is unknown, an exception is thrown if throwExcp is set. Otherwise it returns -1.
String casacore::MultiFileBase::fileName | ( | ) | const [inline] |
Get the file name of the MultiFileBase.
Definition at line 181 of file MultiFileBase.h.
References itsName.
void casacore::MultiFileBase::flush | ( | ) |
Flush the file by writing all dirty data and all header info.
virtual void casacore::MultiFileBase::flushFile | ( | ) | [private, pure virtual] |
Flush the file itself.
Implemented in casacore::MultiFile, and casacore::MultiHDF5.
const vector<Int64>& casacore::MultiFileBase::freeBlocks | ( | ) | const [inline] |
Get the free blocks (for test purposes mainly).
Definition at line 204 of file MultiFileBase.h.
References itsFreeBlocks.
virtual void casacore::MultiFileBase::fsync | ( | ) | [pure virtual] |
Fsync the file (i.e., force the data to be physically written).
Implemented in casacore::MultiFile, and casacore::MultiHDF5.
const vector<MultiFileInfo>& casacore::MultiFileBase::info | ( | ) | const [inline] |
Get the info object (for test purposes mainly).
Definition at line 200 of file MultiFileBase.h.
References itsInfo.
Bool casacore::MultiFileBase::isWritable | ( | ) | const [inline] |
uInt casacore::MultiFileBase::nfile | ( | ) | const |
Get the nr of virtual files.
Read a block at the given offset.
It returns the actual size read.
virtual void casacore::MultiFileBase::readBlock | ( | MultiFileInfo & | info, | |
Int64 | blknr, | |||
void * | buffer | |||
) | [private, pure virtual] |
Read a data block.
Implemented in casacore::MultiFile, and casacore::MultiHDF5.
Read the header info.
If always==False, the info is only read if the header counter has changed.
Implemented in casacore::MultiFile, and casacore::MultiHDF5.
virtual void casacore::MultiFileBase::reopenRW | ( | ) | [pure virtual] |
Reopen the underlying file for read/write access.
Nothing will be done if the file is writable already. Otherwise it will be reopened and an exception will be thrown if it is not possible to reopen it for read/write access.
Implemented in casacore::MultiFile, and casacore::MultiHDF5.
void casacore::MultiFileBase::resync | ( | ) |
Resync with another process by clearing the buffers and rereading the header.
The header is only read if its counter has changed.
void casacore::MultiFileBase::setNewFile | ( | ) | [protected] |
Set the flags and blockSize for a new MultiFile/HDF5.
Int64 casacore::MultiFileBase::size | ( | ) | const [inline] |
Get the total nr of data blocks used.
Definition at line 196 of file MultiFileBase.h.
References itsNrBlock.
Write a block at the given offset.
It returns the actual size written.
virtual void casacore::MultiFileBase::writeBlock | ( | MultiFileInfo & | info, | |
Int64 | blknr, | |||
const void * | buffer | |||
) | [private, pure virtual] |
Write a data block.
Implemented in casacore::MultiFile, and casacore::MultiHDF5.
Referenced by writeDirty().
void casacore::MultiFileBase::writeDirty | ( | MultiFileInfo & | info | ) | [inline, private] |
Definition at line 208 of file MultiFileBase.h.
References casacore::MultiFileInfo::buffer, casacore::MultiFileInfo::curBlock, casacore::MultiFileInfo::dirty, casacore::False, and writeBlock().
virtual void casacore::MultiFileBase::writeHeader | ( | ) | [private, pure virtual] |
Write the header info.
Implemented in casacore::MultiFile, and casacore::MultiHDF5.
Int64 casacore::MultiFileBase::itsBlockSize [protected] |
Definition at line 242 of file MultiFileBase.h.
Referenced by blockSize().
Bool casacore::MultiFileBase::itsChanged [protected] |
Definition at line 247 of file MultiFileBase.h.
vector<Int64> casacore::MultiFileBase::itsFreeBlocks [protected] |
Definition at line 248 of file MultiFileBase.h.
Referenced by freeBlocks().
Int64 casacore::MultiFileBase::itsHdrCounter [protected] |
Definition at line 244 of file MultiFileBase.h.
vector<MultiFileInfo> casacore::MultiFileBase::itsInfo [protected] |
Definition at line 245 of file MultiFileBase.h.
Referenced by info().
String casacore::MultiFileBase::itsName [protected] |
Definition at line 241 of file MultiFileBase.h.
Referenced by fileName().
Int64 casacore::MultiFileBase::itsNrBlock [protected] |
Definition at line 243 of file MultiFileBase.h.
Referenced by size().
Bool casacore::MultiFileBase::itsWritable [protected] |
Definition at line 246 of file MultiFileBase.h.
Referenced by isWritable().