SWMR (Single Writer, Multiple Readers)¶
SWMR lets a single process append to an unlimited dataset in place while other processes read it concurrently, and it interoperates with the reference HDF5 C library and h5py in both directions. This page covers how to lay out a SWMR-capable dataset, append to it durably, follow it from a reader, and recover a file left flagged by a writer that exited uncleanly.
Runnable example
A complete single-process demonstration lives in examples/swmr.rs. Run it with:
It writes and then reads in one process to show the mechanics; in practice the writer and reader are separate processes.
How it works¶
The writer appends chunks and flushes them in dependency order, child structures before the parent metadata that references them, with a durability barrier after each step. The dataset's authoritative size (its dataspace dimension) is published last, as the single commit point: before it a reader sees the old length, after it the new one, and never a torn view. A reader therefore only ever observes a consistent prefix of the data. To pick up newly appended data, a reader re-reads with refresh().
Because the on-disk format is standard HDF5, a reader opened by this crate can follow a file being written by the reference C library or h5py in SWMR mode, and vice versa.
Laying out the dataset¶
A SWMR-capable dataset must have one unlimited dimension and be chunked. The latest format indexes such a dataset with an Extensible Array, which is selected automatically. Create it with the usual writing builder: set the initial extent with with_shape, mark the dimension unlimited with with_maxshape(&[u64::MAX]), and pick a chunk shape with with_chunks.
use hdf5_pure::FileBuilder;
let mut builder = FileBuilder::new();
builder
.create_dataset("log")
.with_i32_data(&[0, 1, 2]) // initial rows
.with_shape(&[3])
.with_maxshape(&[u64::MAX]) // one unlimited dimension
.with_chunks(&[1]);
builder.write("stream.h5").unwrap();
Appending in place¶
Open the existing file with SwmrWriter::open and append with one of the typed helpers. Each append call flushes durably, leaving the file valid for any concurrent reader throughout.
use hdf5_pure::SwmrWriter;
let mut writer = SwmrWriter::open("stream.h5").unwrap();
writer.append_i32("log", &[3, 4, 5]).unwrap();
writer.append_i32("log", &[6, 7]).unwrap();
writer.close().unwrap(); // clears the SWMR flag; or just drop the writer
close() clears the file's SWMR-write flag and flushes, marking the file cleanly closed. Prefer calling it over relying on Drop, so the rare flush error surfaces; dropping the writer also clears the flag.
The append helpers are:
| Method | Appends |
|---|---|
append_i32(dataset, &[i32]) |
i32 values |
append_f64(dataset, &[f64]) |
f64 values |
append_raw(dataset, &[u8]) |
raw little-endian element bytes |
The typed helpers encode their values as little-endian bytes and forward to append_raw. With any of them, the appended length must be a whole number of chunks and the dataset's current length must already be chunk-aligned.
Appends must be chunk-aligned
Both the dataset's current length and each appended length must be multiples of the chunk length. An append that is not chunk-aligned, or whose byte length is not a whole number of elements, returns an error and publishes nothing, so a reader still sees the prior consistent prefix. After such an error the writer should be dropped rather than reused, because its in-memory mirror may have advanced past what reached disk.
Following a growing file¶
Open the file for reading with File::open_swmr, which retains a live filesystem handle so the reader can re-read appended data. The initial view is a consistent snapshot; call refresh() to advance to a newer one after the writer appends.
use hdf5_pure::File;
let mut file = File::open_swmr("stream.h5").unwrap();
let n = file.dataset("log").unwrap().shape().unwrap()[0];
// ... later, after the writer appends ...
file.refresh().unwrap(); // re-read appended data
let ds = file.dataset("log").unwrap();
println!("now {} rows", ds.shape().unwrap()[0]);
refresh() is the SWMR reader's refresh primitive, analogous to the C library's H5Drefresh and h5py's Dataset.refresh(). After it returns, newly fetched Dataset and Group handles observe the appended chunks and extended dimensions. Existing handles borrow &self, so they must be dropped before calling refresh(); re-fetch them afterward, as shown above. See reading for the dataset access APIs used here.
Refresh cost
Each refresh() re-reads the entire file from disk (O(file size)) and re-validates the superblock checksum; a transient parse failure from catching a writer mid-flush is retried a bounded number of times. When following a large, steadily growing log, budget refresh frequency accordingly. refresh() returns an error if the file was not opened with File::open_swmr (the in-memory File::from_bytes path cannot refresh).
Recovering a flagged file¶
While a SwmrWriter is open, the file's superblock carries an active-SWMR-writer flag (matching the reference C library and h5py) so concurrent readers may open it accordingly. close() or dropping the writer clears it. If a writer process exits without a clean close, the file is left flagged. Recover it with the h5clear equivalent:
clear_swmr_flag is safe to call on a file whose flag is already clear.
Supported subset and requirements¶
SWMR append supports the following subset, distinct from the general editing and writing paths:
| Requirement | Detail |
|---|---|
| Dimensionality | exactly one unlimited dimension |
| Storage | chunked (Extensible Array index, latest format) |
| Filters | unfiltered (no compression on the appended dataset) |
| Append granularity | chunk-aligned appends |
| File layout | no userblock (zero base address); latest-format v2/v3 superblock |
| Growth | unbounded |
| Build | requires std (the default); the in-memory/WASM path cannot refresh |
SwmrWriter::open rejects files that fall outside this subset, returning Error::SwmrAppendUnsupported for a non-latest-format superblock or a userblock file before performing any mutating write.