openPMD-api
Classes | Public Types | Public Member Functions | Public Attributes | Friends | List of all members
openPMD::detail::BufferedActions Struct Reference
Collaboration diagram for openPMD::detail::BufferedActions:
Collaboration graph
[legend]

Classes

struct  ADIOS2FlushParams
 

Public Types

enum  StreamStatus {
  StreamStatus::DuringStep, StreamStatus::OutsideOfStep, StreamStatus::StreamOver, StreamStatus::NoStream,
  StreamStatus::Parsing, StreamStatus::Undecided
}
 
using FlushTarget = ADIOS2IOHandlerImpl::FlushTarget
 
using ParsePreference = Parameter< Operation::OPEN_FILE >::ParsePreference
 
using AttributeMap_t = std::map< std::string, adios2::Params >
 

Public Member Functions

 BufferedActions (BufferedActions const &)=delete
 
 BufferedActions (ADIOS2IOHandlerImpl &impl, InvalidatableFile file)
 
void finalize ()
 Implementation of destructor, will only run once.
 
adios2::Engine & getEngine ()
 
adios2::Engine & requireActiveStep ()
 
template<typename BA >
void enqueue (BA &&ba)
 
template<typename BA >
void enqueue (BA &&ba, decltype(m_buffer) &)
 
template<typename... Args>
void flush (Args &&...args)
 
template<typename F >
void flush_impl (ADIOS2FlushParams flushParams, F &&performPutsGets, bool writeLatePuts, bool flushUnconditionally)
 Flush deferred IO actions. More...
 
void flush_impl (ADIOS2FlushParams, bool writeLatePuts=false)
 Overload of flush() that uses adios2::Engine::Perform(Puts|Gets) and does not flush unconditionally.
 
AdvanceStatus advance (AdvanceMode mode, bool calledExplicitly)
 Begin or end an ADIOS step. More...
 
void drop ()
 
AttributeMap_t const & availableAttributes ()
 
std::vector< std::string > availableAttributesPrefixed (std::string const &prefix)
 
void invalidateAttributesMap ()
 
AttributeMap_t const & availableVariables ()
 
std::vector< std::string > availableVariablesPrefixed (std::string const &prefix)
 
void invalidateVariablesMap ()
 

Public Attributes

std::string m_file
 The full path to the file created on disk, including the containing directory and the file extension, as determined by ADIOS2IOHandlerImpl::fileSuffix(). More...
 
std::string m_IOName
 ADIOS requires giving names to instances of adios2::IO. More...
 
adios2::ADIOS & m_ADIOS
 
adios2::IO m_IO
 
std::vector< std::unique_ptr< BufferedAction > > m_buffer
 The default queue for deferred actions. More...
 
std::map< std::string, BufferedAttributeWritem_attributeWrites
 Buffer for attributes to be written in the new (variable-based) attribute layout. More...
 
std::vector< BufferedAttributeReadm_attributeReads
 
std::vector< BufferedUniquePtrPutm_uniquePtrPuts
 When receiving a unique_ptr, we know that the buffer is ours and ours alone. More...
 
std::vector< std::unique_ptr< BufferedAction > > m_alreadyEnqueued
 This contains deferred actions that have already been enqueued into ADIOS2, but not yet performed in ADIOS2. More...
 
adios2::Mode m_mode
 
std::map< unsigned, std::unique_ptr< I_UpdateSpan > > m_updateSpans
 The base pointer of an ADIOS2 span might change after reallocations. More...
 
PreloadAdiosAttributes preloadAttributes
 
std::set< std::string > uncommittedAttributes
 
bool optimizeAttributesStreaming = false
 
ParsePreference parsePreference = ParsePreference::UpFront
 
StreamStatus streamStatus = StreamStatus::OutsideOfStep
 

Friends

struct BufferedGet
 
struct BufferedPut
 
struct RunUniquePtrPut
 
struct WriteDataset
 

Member Enumeration Documentation

◆ StreamStatus

Enumerator
DuringStep 

A step is currently active.

OutsideOfStep 

A stream is active, but no step.

StreamOver 

Stream has ended.

NoStream 

File is not written is streaming fashion.

Begin/EndStep will be replaced by simple flushes. Used for: 1) Writing BP4 files without steps despite using the Streaming API. This is due to the fact that ADIOS2.6.0 requires using steps to read BP4 files written with steps, so using steps is opt-in for now. Notice that while the openPMD API requires ADIOS >= 2.7.0, the resulting files need to be readable from ADIOS 2.6.0 as well. This workaround is hence staying until switching to a new ADIOS schema. 2) Reading with the Streaming API any file that has been written without steps. This is not a workaround since not using steps, while inefficient in ADIOS2, is something that we support.

Parsing 

Rationale behind this state: When user code opens a Series, series.iterations should contain all available iterations.

If accessing a file without opening a step, ADIOS2 will grant access to variables and attributes from all steps, allowing us to parse the complete dump. This state indicates that no step should be opened for parsing purposes (which is necessary in streaming engines, hence they are initialized with the OutsideOfStep state). A step should only be opened if an explicit ADVANCE task arrives at the backend.

Todo:
If the streaming API is used on files, parsing the whole Series up front is unnecessary work. Our frontend does not yet allow to distinguish whether parsing the whole series will be necessary since parsing happens upon construction time of Series, but the classical and the streaming API are both activated afterwards from the created Series object. Hence, improving this requires refactoring in our user-facing API. Ideas: (1) Delayed lazy parsing of iterations upon accessing (would bring other benefits also). (2) Introduce a restricted class StreamingSeries.
Undecided 

The stream status of a file-based engine will be decided upon opening the engine if in read mode.

Up until then, this right here is the status.

Member Function Documentation

◆ advance()

AdvanceStatus openPMD::detail::BufferedActions::advance ( AdvanceMode  mode,
bool  calledExplicitly 
)

Begin or end an ADIOS step.

Parameters
modeWhether to begin or end a step.
calledExplicitlyTrue if called due to a public API call. False if called from requireActiveStep. Some engines (BP5) require that every interaction happens within an active step, meaning that we need to call advance() implicitly at times. When doing that, do not tag the dataset with __openPMD_internal/useSteps (yet).
Returns
AdvanceStatus

◆ flush_impl()

template<typename F >
void openPMD::detail::BufferedActions::flush_impl ( ADIOS2FlushParams  flushParams,
F &&  performPutsGets,
bool  writeLatePuts,
bool  flushUnconditionally 
)

Flush deferred IO actions.

Parameters
flushParamsFlush level and target.
performPutsGetsA functor that takes as parameters (1) *this and (2) the ADIOS2 engine. Its task is to ensure that ADIOS2 performs Put/Get operations. Several options for this:
  • adios2::Engine::EndStep
  • adios2::Engine::Perform(Puts|Gets)
  • adios2::Engine::Close
writeLatePutsSome things are deferred until right before Engine::EndStep() or Engine::Close(): 1) Writing attributes in new ADIOS2 schema. 2) Running unique_ptr Put()s.
flushUnconditionallyWhether to run the functor even if no deferred IO tasks had been queued.

Member Data Documentation

◆ m_alreadyEnqueued

std::vector<std::unique_ptr<BufferedAction> > openPMD::detail::BufferedActions::m_alreadyEnqueued

This contains deferred actions that have already been enqueued into ADIOS2, but not yet performed in ADIOS2.

We must store them somewhere until the next PerformPuts/Gets, EndStep or Close in ADIOS2 to avoid use after free conditions.

◆ m_attributeReads

std::vector<BufferedAttributeRead> openPMD::detail::BufferedActions::m_attributeReads
Todo:
This one is unnecessary, in the new schema, attribute reads do not need to be deferred, but can happen instantly without performance penalty, once preloadAttributes has been filled.

◆ m_attributeWrites

std::map<std::string, BufferedAttributeWrite> openPMD::detail::BufferedActions::m_attributeWrites

Buffer for attributes to be written in the new (variable-based) attribute layout.

Reason: If writing one variable twice within the same ADIOS step, it is undefined which value ADIOS2 will store. We want the last write operation to succeed, so this map stores attribute writes by attribute name, allowing us to override older write commands. The queue is drained only when closing a step / the engine.

◆ m_buffer

std::vector<std::unique_ptr<BufferedAction> > openPMD::detail::BufferedActions::m_buffer

The default queue for deferred actions.

Drained upon BufferedActions::flush().

◆ m_file

std::string openPMD::detail::BufferedActions::m_file

The full path to the file created on disk, including the containing directory and the file extension, as determined by ADIOS2IOHandlerImpl::fileSuffix().

(Meaning, in case of the SST engine, no file suffix since the SST engine automatically adds its suffix unconditionally)

◆ m_IOName

std::string openPMD::detail::BufferedActions::m_IOName

ADIOS requires giving names to instances of adios2::IO.

We make them different from the actual file name, because of the possible following workflow:

  1. create file foo.bp -> would create IO object named foo.bp
  2. delete that file (let's ignore that we don't support deletion yet and call it preplanning)
  3. create file foo.bp a second time -> would create another IO object named foo.bp -> craash

So, we just give out names based on a counter for IO objects. Hence, next to the actual file name, also store the name for the IO.

◆ m_uniquePtrPuts

std::vector<BufferedUniquePtrPut> openPMD::detail::BufferedActions::m_uniquePtrPuts

When receiving a unique_ptr, we know that the buffer is ours and ours alone.

So, for performance reasons, show the buffer to ADIOS2 as late as possible and avoid unnecessary data copies in BP5 triggered by PerformDataWrites().

◆ m_updateSpans

std::map<unsigned, std::unique_ptr<I_UpdateSpan> > openPMD::detail::BufferedActions::m_updateSpans

The base pointer of an ADIOS2 span might change after reallocations.

The frontend will ask the backend for those updated base pointers. Spans given out by the ADIOS2 backend to the frontend are hence identified by an unsigned integer and stored in this member for later retrieval of the updated base pointer. This map is cleared upon flush points.


The documentation for this struct was generated from the following files: