Upcoming New HDF5 Features: Multi-threading, sparse data storage, and encryption in HDF5

HDF and HDF-EOS Workshop XXVII (2024)

1. Upcoming New HDF5 Features Multi-threading, sparse data storage, and encryption in HDF5 Elena Pourmal [email protected] John Mainzer [email protected]

2. Outline Intro to Lifeboat, LLC Multi-threaded access to data in HDF5 Sparse data in HDF5 Integrity of data in HDF5 (HDF5 encryption) ESIP, July 25, 2024

3. Lifeboat, LLC Goal: Sustain and enhance open source HDF5 Founded in August 2021; located in Champaign, IL and Laramie, WY www.lifeboat.llc [email protected] Funded by DOE SBIR/STTR Program Phase II: “ Toward multi-threaded concurrency in HDF5 ” (started in April 2023) Phase II: “ Supporting sparse data in HDF5 ” (started in April 2024) Phase I: “Protecting the confidentiality and integrity of data stored in HDF5” (aka“HDF5 Encryption”) (started in February 2024) We don’t make HDF5… we make HDF5 better

4. Lifeboat, LLC (cont’d) H5+ product In near term - Collection of pluggable connectors and tools to enhance functionality of open-source HDF5 and tools Encryption, multi-threaded access to data, full implementation of single writer/multiple reader access mode, data recovery tool In long term - Better engineered multi-threaded HDF5 library with full set of features along with a rich collection of connectors to enhance functionality and to access data on all kinds of storage systems (e.g., traditional FS, Cloud, Object Store) We don’t make HDF5… we make HDF5 better

5. HDF5 terminology for this talk Virtual File Driver (VFD) Maps HDF5 logical file to storage Virtual Object Layer Connector (VOL connector) Maps HDF5 objects to storage ESIP, July 25, 2024

6. HDF5 Architecture (1.12.0 and later releases) HDF5 C API VOL Layer VFD Layer HDF5 Library proper (Native VOL) POSIX MPI I/O File System Storage HDF5 Application … POSIX API ROS3 S3 Cloud Terminal VOL Connector (e.g., DAOS) Pass-through VOL connector GPU Connector specific API Filter plugins (e.g., compression) Filters S3 API Plugins Filter, VFD, VOL new Library component Library component Legend … … S# HSDS REST VOL Connector h5pyd S3 Cloud Subfiling ESIP, July 25, 2024

7. Multi-threaded access to data in HDF5 Enabling multi-threaded VOL connectors with open-source HDF5

8. Parallel programming paradigms Message passing (MPI and MPI I/O); application Runs on one or more computer nodes Divides a computation between multiple processes Uses message passing (MPI) over the network to share and exchange data Threading (OpenMP, Pthreads , Windows threads); application Runs on one node Uses multiple CPUs on the node to execute parts of the application Example: One can divide a loop into four smaller loops and run them simultaneously on separate CPUs (OpenMP) Each CPU processes a thread Applications can use both OpenMP and MPI (hybrid model) ESIP, July 25, 2024

9. Parallelism in HDF5 HDF5 supports message passing model (MPI and MPI I/O) Most benefits are achieved on parallel file systems Used by Climate Modeling applications No support for threading: HDF5 is thread-safe but not multi-threaded meaning that multiple threads operate in the HDF5 library without corrupting data in memory and/or in the HDF5 file We are working on enabling multi-threaded access to HDF5 data via multi-threaded VOL connectors OpenMP is much easier to use Supported by C, C++ and Fortran GNU, Intel, PGI, NAG, Clang, Absoft , IBM XL compilers, HPE Cray compilers, MSVC, etc. Python Numpy , SciPy and Pandas packages ESIP, July 25, 2024

10. Bypass VOL Connector VFD Layer Updated Native VOL File System HDF5 Application new MT Bypass VOL connector HDF5 C API VOL Layer Bypass VOL Concept Initially will support I/O for contiguous and chunked datasets; no data filtering Checks if I/O is supported; routs to native VOL or to connector proper (hits mutex) Queries HDF5 library for the location of raw data (hits mutex) Executes raw data I/O in parallel in multiple threads Has thread-pool to perform I/O and can be used with a single-threaded application Functionality will be extended as parts of the HDF5 library (e.g., metadata cache) are converted See documentation https://github.com/LifeboatLLC/MT-HDF5 Check HUG23 “Toward Multi-Threaded Concurrency in HDF5” talk by John Mainzer First version of the connector is available H5I, H5E , H5P, H5CX, H5VL, H5S, H5FD Multi-threaded HDF5 modules required by VOL connectors Converted Low priority for the first release ESIP, July 25, 2024 In progress File System

11. Proof of concept: Digitally Reconstructed Neurons Blue Brain Project https://www.epfl.ch/research/domains/bluebrain/ { "0000": { "points”: np.empty ((9610, 3), np.float32), “offsets": np.empty (21, np.uint64) }, "0001": { "points": np.empty ((14983, 3), np.float32), "offsets": np.empty (48, np.uint64) }, ... } Synthetic Data Presented: Datasets: 500 000 Total size: 640 GB File Space Strategy: Paged allocation Page size: 64 kB (not a default value!) 1k - 100M neurons Slide courtesy of Luc Grosheintz, Blue Brain Project, EPFL Blue Brain Project System: Cray EX (Perlmutter @NERSC) 512 GB memory per node 2 x AMD EPYC 7763 CPUs per node 64 cores per CPU

12. Benchmark results Presented experimental setup: 8 Threads 3 measurements for each run HDF5 1.10.1 HDF5 : Plain HDF5 with 512 MB page buffer, 75% reserved for raw data; paged allocation 64KB pages Direct / Page-Aware : The two variants of the prototype: Left: Read metadata for the file using HDF5 Right: Read metadata from pre-computed JSON file Page-Aware : Pages of HDF5 file are brought into memory and threads read raw data from memory We see ~5x speedup when using 8 threads; 2x speedup when using 2 threads ESIP, July 25, 2024

13. Using bypass VOL connector with single-threaded application (preliminary results) Benchmark: Read contiguous or chunked dataset by 4 hyperslabs with Thread-safe HDF5 library Thread-safe HDF5 library with bypass VOL connector and thread pool ; each thread reads 1MB of data C program using Pthreads ; each thread reads 1MB of data Compare performance Systems Linux box Cray EX node (Perlmutter @NERSC) macOS 1.6x to 3x scaling is achieved on Linux and Cray systems Note: no scaling on macOS due to global lock on FS; need to investigate and document We are looking for applications to test the connector! ESIP, July 25, 2024

14. Linux box results (64GB dataset) ESIP, July 25, 2024

15. Perlmutter results (64GB dataset) ESIP, July 25, 2024

16. Perlmutter results (640GB dataset) ESIP, July 25, 2024

17. Sparse Data Storage in HDF5 New storage paradigm for sparse and variable-length data

18. New Storage Paradigm: Structured Chunk Chunked dataset Chunked storage : all chunk elements are stored 0 0 0 0 0 0 0 0 0 0 0 0 66 69 72 0 0 96 99 96 102 0 may represent a value that is not-defined Structured Chunk storage for sparse data : Locations and values of defined elements (specified by the “ hyperslab ” selection) are stored in different sections of the chunk Section 0 Section 1 Encoded selection 66 69 72 96 99 96 102 If we write a shown sub-array using hyperslab selection how the chunk will be stored in the file? ESIP, July 25, 2024 Structured chunk may have more than 2 sections New way of storing variable-length data in HDF5 3 sections when storing sparse variable-length data Each section can be compressed with its own compression (or filter pipeline) No changes to the programming model A few new APIs including H5Pset_filter to solve argument passing issue for the compression filters

19. Sparse Storage Implementation Status Design documents can be found in Lifeboat GitHub repo https://github.com/LifeboatLLC/SparseHDF5 Programming model and APIs File Format extensions Shared chunk cache Better performing chunk cache including multi-threaded implementation Improved I/O pipeline in HDF5 library Current status File Format is finalized (subject to change until implementation is completed) Started implementation of APIs and command-line tools to support sparse storage White paper with benchmark results that motivate structured chunk compression https://github.com/LifeboatLLC/SparseHDF5/blob/main/benchmarks/Sparse-VL-Benchmarks-2024-01-16.pdf Designs for shared chunk cache and I/O pipeline are under development Prototype should be available in Spring 2025 ESIP, July 25, 2024

20. Integrity of Data in HDF5 HDF5 Encryption

21. Why introduce encryption to HDF5? Data integrity is currently not guaranteed by the HDF5 library! HDF5 Encryption: One of the long-standing feature requests for HDF5 Preserves data integrity Encryption protects data from being stolen, changed or compromised Adds extra layer of protection for sensitive HDF5 data; it complements solutions provided by the organizations to assure data confidentiality and integrity during data transfer, storage and computation Works with the standard open source HDF5 library Delivered as a VFD plugin Allows random access to encrypted data (i.e., no need to decrypt the whole HDF5 file) Supports any encryption method that allows computation of the maximum ciphertext page size from the plaintext page size Phase I prototype will be delivered with encryption built into the latest HDF5 release 1.14.4 and will support AES and/or Twofish symmetrical-key encryption (available in GNU Open Source gcrypt library) ESIP, July 25, 2024

22. Acknowledgement This work is supported by the U.S. Department of Energy, Office of Science under award number: DE-SC0023583 for Phase II SBIR project "Toward multi-threaded concurrency in HDF5” DE-SC0023583 for Phase II SBIR project “Supporting Sparse Data in HDF5” DE-SC00 24823 for Phase I SBIR project “ Protecting the confidentiality and integrity of data stored in HDF5" ESIP, July 25, 2024

23. Thank you! Questions? ESIP, July 25, 2024