openPMD-api
|
Classes | |
struct | ADIOS2FlushParams |
Public Types | |
enum | StreamStatus { StreamStatus::DuringStep, StreamStatus::OutsideOfStep, StreamStatus::StreamOver, StreamStatus::NoStream, StreamStatus::Parsing, StreamStatus::Undecided } |
using | FlushTarget = ADIOS2IOHandlerImpl::FlushTarget |
using | ParsePreference = Parameter< Operation::OPEN_FILE >::ParsePreference |
using | AttributeMap_t = std::map< std::string, adios2::Params > |
Public Member Functions | |
BufferedActions (BufferedActions const &)=delete | |
BufferedActions (ADIOS2IOHandlerImpl &impl, InvalidatableFile file) | |
void | finalize () |
Implementation of destructor, will only run once. | |
adios2::Engine & | getEngine () |
adios2::Engine & | requireActiveStep () |
template<typename BA > | |
void | enqueue (BA &&ba) |
template<typename BA > | |
void | enqueue (BA &&ba, decltype(m_buffer) &) |
template<typename... Args> | |
void | flush (Args &&...args) |
template<typename F > | |
void | flush_impl (ADIOS2FlushParams flushParams, F &&performPutsGets, bool writeLatePuts, bool flushUnconditionally) |
Flush deferred IO actions. More... | |
void | flush_impl (ADIOS2FlushParams, bool writeLatePuts=false) |
Overload of flush() that uses adios2::Engine::Perform(Puts|Gets) and does not flush unconditionally. | |
AdvanceStatus | advance (AdvanceMode mode, bool calledExplicitly) |
Begin or end an ADIOS step. More... | |
void | drop () |
AttributeMap_t const & | availableAttributes () |
std::vector< std::string > | availableAttributesPrefixed (std::string const &prefix) |
void | invalidateAttributesMap () |
AttributeMap_t const & | availableVariables () |
std::vector< std::string > | availableVariablesPrefixed (std::string const &prefix) |
void | invalidateVariablesMap () |
Public Attributes | |
std::string | m_file |
The full path to the file created on disk, including the containing directory and the file extension, as determined by ADIOS2IOHandlerImpl::fileSuffix(). More... | |
std::string | m_IOName |
ADIOS requires giving names to instances of adios2::IO. More... | |
adios2::ADIOS & | m_ADIOS |
adios2::IO | m_IO |
std::vector< std::unique_ptr< BufferedAction > > | m_buffer |
The default queue for deferred actions. More... | |
std::map< std::string, BufferedAttributeWrite > | m_attributeWrites |
Buffer for attributes to be written in the new (variable-based) attribute layout. More... | |
std::vector< BufferedAttributeRead > | m_attributeReads |
std::vector< BufferedUniquePtrPut > | m_uniquePtrPuts |
When receiving a unique_ptr, we know that the buffer is ours and ours alone. More... | |
std::vector< std::unique_ptr< BufferedAction > > | m_alreadyEnqueued |
This contains deferred actions that have already been enqueued into ADIOS2, but not yet performed in ADIOS2. More... | |
adios2::Mode | m_mode |
std::map< unsigned, std::unique_ptr< I_UpdateSpan > > | m_updateSpans |
The base pointer of an ADIOS2 span might change after reallocations. More... | |
PreloadAdiosAttributes | preloadAttributes |
std::set< std::string > | uncommittedAttributes |
bool | optimizeAttributesStreaming = false |
ParsePreference | parsePreference = ParsePreference::UpFront |
StreamStatus | streamStatus = StreamStatus::OutsideOfStep |
Friends | |
struct | BufferedGet |
struct | BufferedPut |
struct | RunUniquePtrPut |
struct | WriteDataset |
|
strong |
Enumerator | |
---|---|
DuringStep | A step is currently active. |
OutsideOfStep | A stream is active, but no step. |
StreamOver | Stream has ended. |
NoStream | File is not written is streaming fashion. Begin/EndStep will be replaced by simple flushes. Used for: 1) Writing BP4 files without steps despite using the Streaming API. This is due to the fact that ADIOS2.6.0 requires using steps to read BP4 files written with steps, so using steps is opt-in for now. Notice that while the openPMD API requires ADIOS >= 2.7.0, the resulting files need to be readable from ADIOS 2.6.0 as well. This workaround is hence staying until switching to a new ADIOS schema. 2) Reading with the Streaming API any file that has been written without steps. This is not a workaround since not using steps, while inefficient in ADIOS2, is something that we support. |
Parsing | Rationale behind this state: When user code opens a Series, series.iterations should contain all available iterations. If accessing a file without opening a step, ADIOS2 will grant access to variables and attributes from all steps, allowing us to parse the complete dump. This state indicates that no step should be opened for parsing purposes (which is necessary in streaming engines, hence they are initialized with the OutsideOfStep state). A step should only be opened if an explicit ADVANCE task arrives at the backend.
|
Undecided | The stream status of a file-based engine will be decided upon opening the engine if in read mode. Up until then, this right here is the status. |
AdvanceStatus openPMD::detail::BufferedActions::advance | ( | AdvanceMode | mode, |
bool | calledExplicitly | ||
) |
Begin or end an ADIOS step.
mode | Whether to begin or end a step. |
calledExplicitly | True if called due to a public API call. False if called from requireActiveStep. Some engines (BP5) require that every interaction happens within an active step, meaning that we need to call advance() implicitly at times. When doing that, do not tag the dataset with __openPMD_internal/useSteps (yet). |
void openPMD::detail::BufferedActions::flush_impl | ( | ADIOS2FlushParams | flushParams, |
F && | performPutsGets, | ||
bool | writeLatePuts, | ||
bool | flushUnconditionally | ||
) |
Flush deferred IO actions.
flushParams | Flush level and target. |
performPutsGets | A functor that takes as parameters (1) *this and (2) the ADIOS2 engine. Its task is to ensure that ADIOS2 performs Put/Get operations. Several options for this:
|
writeLatePuts | Some things are deferred until right before Engine::EndStep() or Engine::Close(): 1) Writing attributes in new ADIOS2 schema. 2) Running unique_ptr Put()s. |
flushUnconditionally | Whether to run the functor even if no deferred IO tasks had been queued. |
std::vector<std::unique_ptr<BufferedAction> > openPMD::detail::BufferedActions::m_alreadyEnqueued |
This contains deferred actions that have already been enqueued into ADIOS2, but not yet performed in ADIOS2.
We must store them somewhere until the next PerformPuts/Gets, EndStep or Close in ADIOS2 to avoid use after free conditions.
std::vector<BufferedAttributeRead> openPMD::detail::BufferedActions::m_attributeReads |
std::map<std::string, BufferedAttributeWrite> openPMD::detail::BufferedActions::m_attributeWrites |
Buffer for attributes to be written in the new (variable-based) attribute layout.
Reason: If writing one variable twice within the same ADIOS step, it is undefined which value ADIOS2 will store. We want the last write operation to succeed, so this map stores attribute writes by attribute name, allowing us to override older write commands. The queue is drained only when closing a step / the engine.
std::vector<std::unique_ptr<BufferedAction> > openPMD::detail::BufferedActions::m_buffer |
The default queue for deferred actions.
Drained upon BufferedActions::flush().
std::string openPMD::detail::BufferedActions::m_file |
The full path to the file created on disk, including the containing directory and the file extension, as determined by ADIOS2IOHandlerImpl::fileSuffix().
(Meaning, in case of the SST engine, no file suffix since the SST engine automatically adds its suffix unconditionally)
std::string openPMD::detail::BufferedActions::m_IOName |
ADIOS requires giving names to instances of adios2::IO.
We make them different from the actual file name, because of the possible following workflow:
So, we just give out names based on a counter for IO objects. Hence, next to the actual file name, also store the name for the IO.
std::vector<BufferedUniquePtrPut> openPMD::detail::BufferedActions::m_uniquePtrPuts |
When receiving a unique_ptr, we know that the buffer is ours and ours alone.
So, for performance reasons, show the buffer to ADIOS2 as late as possible and avoid unnecessary data copies in BP5 triggered by PerformDataWrites().
std::map<unsigned, std::unique_ptr<I_UpdateSpan> > openPMD::detail::BufferedActions::m_updateSpans |
The base pointer of an ADIOS2 span might change after reallocations.
The frontend will ask the backend for those updated base pointers. Spans given out by the ADIOS2 backend to the frontend are hence identified by an unsigned integer and stored in this member for later retrieval of the updated base pointer. This map is cleared upon flush points.