Backend-Specific Configuration

While the openPMD API intends to be a backend-independent implementation of the openPMD standard, it is sometimes useful to pass configuration parameters to the specific backend in use. For each backend, configuration options can be passed via a JSON-formatted string or via environment variables. A JSON option always takes precedence over an environment variable.

The fundamental structure of this JSON configuration string is given as follows:

{
    "adios": "put ADIOS config here",
    "adios2": "put ADIOS2 config here",
    "hdf5": "put HDF5 config here",
    "json": "put JSON config here"
}

This structure allows keeping one configuration string for several backends at once, with the concrete backend configuration being chosen upon choosing the backend itself.

The configuration is read in a case-sensitive manner. Generally, keys of the configuration are lower case. Parameters that are directly passed through to an external library and not interpreted within openPMD API (e.g. adios2.engine.parameters) are unaffected by this and follow the respective library’s conventions.

The configuration string may refer to the complete openPMD::Series or may additionally be specified per openPMD::Dataset, passed in the respective constructors. This reflects the fact that certain backend-specific parameters may refer to the whole Series (such as storage engines and their parameters) and others refer to actual datasets (such as compression).

For a consistent user interface, backends shall follow the following rules:

  • The configuration structures for the Series and for each dataset should be defined equivalently.

  • Any setting referring to single datasets should also be applicable globally, affecting all datasets.

  • If a setting is defined globally, but also for a concrete dataset, the dataset-specific setting should override the global one.

  • If a setting is passed to a dataset that only makes sense globally (such as the storage engine), the setting should be ignored except for printing a warning. Backends should define clearly which keys are applicable to datasets and which are not.

Configuration Structure per Backend

ADIOS2

A full configuration of the ADIOS2 backend:

{
  "adios2": {
    "engine": {
      "type": "sst",
      "parameters": {
        "BufferGrowthFactor": "2.0",
        "QueueLimit": "2"
      }
    },
    "dataset": {
      "operators": [
        {
          "type": "blosc",
          "parameters": {
              "clevel": "1",
              "doshuffle": "BLOSC_BITSHUFFLE"
          }
        }
      ]
    }
  }
}

All keys found under adios2.dataset are applicable globally as well as per dataset, keys found under adios2.engine only globally. Explanation of the single keys:

  • adios2.engine.type: A string that is passed directly to adios2::IO:::SetEngine for choosing the ADIOS2 engine to be used. Please refer to the official ADIOS2 documentation for a list of available engines.

  • adios2.engine.parameters: An associative array of string-formatted engine parameters, passed directly through to adios2::IO::SetParameters. Please refer to the official ADIOS2 documentation for the allowable engine parameters.

  • adios2.engine.usesteps: Described more closely in the documentation for the ADIOS2 backend.

  • adios2.dataset.operators: This key contains a list of ADIOS2 operators, used to enable compression or dataset transformations. Each object in the list has two keys:

    • type supported ADIOS operator type, e.g. zfp, sz

    • parameters is an associative map of string parameters for the operator (e.g. compression levels)

Any setting specified under adios2.dataset is applicable globally as well as on a per-dataset level. Any setting under adios2.engine is applicable globally only.

Other backends

Do currently not read the configuration string. Please refer to the respective backends’ documentations for further information on their configuration.