openPMD supports writing to and reading from HDF5
For this, the installed copy of openPMD must have been built with support for the HDF5 backend.
To build openPMD with support for HDF5, use the CMake option
For further information, check out the installation guide,
build dependencies and the build options.
HDF5 internally either writes serially, via
POSIX on Unix systems, or parallel to a single logical file via MPI-I/O.
The following environment variables control HDF5 I/O behavior at runtime.
||Sets the MPI-parallel transfer mode to collective or independent.|
OPENPMD_HDF5_INDEPENDENT: by default, the HDF5’s MPI-parallel write/read operations are MPI-collective.
Since we call those underneath each
loadChunk call, these methods become MPI-collective as well.
Please refer to the HDF5 manual, function H5Pset_dxpl_mpio for more details.
In case you decide to use independent I/O with parallel HDF5, be advised that we did see performance penalties as well problems with some MPI-I/O implementations in the past.
For independent parallel I/O, potentially prefer using a modern version of the MPICH implementation (especially, use ROMIO instead of OpenMPI’s ompio implementation).
- GitHub issue #554
- Axel Huebl, Rene Widera, Felix Schmitt, Alexander Matthes, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, and Michael Bussmann. On the Scalability of Data Reduction Techniques in Current and Upcoming HPC Systems from an Application Perspective, ISC High Performance 2017: High Performance Computing, pp. 15-29, 2017. arXiv:1706.00522, DOI:10.1007/978-3-319-67630-2_2