Vitastor 3.2.0 released

2026-08-31

This is a bugfix release, and most of the fixes came from a new approach to testing: a chaos test for the block store, which now runs in CI on every change.

Also thanks to Yury Luneff from MIND Software for another part of the fixes.

New features

Besides the chaos test and the stability fixes, the release also includes several new features:

  • Online resize for ublk, NBD, QEMU and VDUSE, plus CSI online node expansion for all volume types. That is, the size of attached devices now changes automatically after vitastor-cli modify --resize.
  • VitastorFS mount helpers mount.vitastorfs and mount.vitastorblk - the FS can now be mounted from /etc/fstab.
  • NFS --readahead can now be configured in the mount command: increasing it speeds up reading 2-3 times.
  • Using the client without io_uring is allowed again (the use_sync_send_recv option) - with RDMA enabled it’s just as fast.
  • Added a /healthcheck handler to the monitor.
  • NBD, ublk and NFS servers now handle SIGINT and SIGTERM and stop gracefully.
  • CSI now uses, and the VitastorFS documentation now suggests, the safe soft,sync mount options.
  • This mostly concerns the old store, but still: write operations can no longer hang forever when the journal runs out of space. The store now returns EAGAIN in such cases, OSDs handle that error, and retrying the request becomes the client’s duty. The fix changes error handling on the client side, so updating clients is recommended when using the old store.

Chaos testing

The block store is now covered by a DST-style (deterministic simulation testing) test driven by a seed. The store runs in a simulated environment: io_uring delivers completions out of order and with a random delay, the volatile write cache of the disk loses a random subset of itself on a simulated power outage, and writes that were in flight at that moment land fully, partially (torn at a sector boundary) or not at all. The store itself is the real thing - journal, metadata, flusher, compaction, checksums. A model of the written data is kept alongside, and after every power outage the recovered state is checked against it, including object bitmaps.

The test runs both store implementations against the very same workload in 24 configurations: old and new store, server and desktop drives (with and without capacitors), two-phase (EC) and instant (replicated) writes, checksums disabled / enabled with a 4 KB / 16 KB block. CI runs 200 seeds on every build, one process per seed, and prints the list of failing seeds - after which any failure is reproduced exactly by test_blockstore_chaos --seed N.

To put a number on it: run against the store from 3.1.0, the current chaos test fails on 83 of 200 seeds without counting memory leaks and the strict ban on writing the same sector twice in parallel, and on 97 of 200 seeds counting them.

  #  configuration                           failed/total
  8  v2 fsync=0 csum=16384 instant          4/9
  9  v2 fsync=1 csum=none  instant          1/8
 10  v2 fsync=1 csum=4096  instant          1/8
 12  v1 fsync=0 csum=none  two-phase        8/8
 13  v1 fsync=0 csum=4096  two-phase        8/8
 14  v1 fsync=0 csum=16384 two-phase        8/8
 15  v1 fsync=1 csum=none  two-phase        8/8
 16  v1 fsync=1 csum=4096  two-phase        8/8
 17  v1 fsync=1 csum=16384 two-phase        8/8
 18  v1 fsync=0 csum=none  instant          8/8
 19  v1 fsync=0 csum=4096  instant          8/8
 20  v1 fsync=0 csum=16384 instant          8/8
 21  v1 fsync=1 csum=none  instant          1/8
 22  v1 fsync=1 csum=4096  instant          3/8
 23  v1 fsync=1 csum=16384 instant          8/8

The dominant problems:

  • Old store: the OSD aborts with BUG: Unexpected dirty_entry ... unstable state during flush - on every seed of all six two-phase (EC) configurations - or refuses to start with Fatal error (bug): big_write journal_entry was allocated over another object.
  • New store: after a power outage an object becomes unreadable with a checksum error (retval=-33), and in one case comes back holding the data of another object.

All of these problems are now fixed.

Should you upgrade?

  • Old store (meta_format 2, OSDs created before 3.0) - yes, regardless of your hardware. It failed the chaos test in every configuration, including the recommended ones.
  • Desktop drives without capacitors (immediate_commit=none) - yes. Significant SYNC-related bugs are fixed in both store versions, such as the loss of acknowledged data on a power failure.
  • Checksums with a block larger than 4 KB (csum_block_size) - yes, and this affects both stores. Reads of partially written blocks could fail with a spurious checksum error on perfectly good data, and in the new store an object could become unreadable after a power outage even on server SSDs.
  • New store on server SSDs with capacitors, checksums off or 4 KB - upgrading is recommended but not urgent. The chaos test did not catch the new store failing in that configuration on 3.1.0, and the fixes below for it cover rarer situations.

Fixes for the default configuration (server SSDs, checksums off or 4 KB)

Old store (meta_format 2):

  • On a restart after the journal was cut short at an entry located at the beginning of a block, journal writing continued incorrectly, so that new entries weren’t taken into account on another quick OSD restart (before a full cycle of writing the whole journal).
  • The order of several concurrent journal writes was tracked incorrectly, which means a write could be acknowledged before it was successfully journaled.
  • Two writes of the same journal sector could be in flight at once. In theory the device could execute them in an arbitrary order and corrupt the journal.
  • Object and OSD space counters could underflow after a restart.
  • On OSD shutdown, dirty entry metadata was released through free() without being a pointer, corrupting the heap. Several memory leaks are fixed as well; the tests now run with leak checking enabled (it used to be disabled in CI).
  • Reads didn’t work at all, crashing the OSD with a division by zero instead, when checksums were disabled and inmemory_journal was disabled.
  • With inmemory_journal disabled, an unmodified journal sector - or even the journal superblock - could be overwritten with incorrect data, after which the OSD would refuse to start.

New store (meta_format 3):

  • Objects with a big_intent base entry could disappear on a power outage during compaction, if their data was already mutated and their metadata wasn’t updated yet. This situation is rare for server SSDs, but possible in theory.

Fixes for drives without capacitors (immediate_commit=none)

Old store:

  • Several hangs and crashes around deletions, rollbacks and syncs are fixed - a rollback of a version with a newer unsynced write above it hung forever; a deletion could trip an assertion in the flusher or during journal writing; a deletion concurrent with a sync corrupted internal state.
  • Journal replay could resurrect stale writes of a deleted object, or refuse to start with “big_write journal_entry was allocated over another object”.

New store:

  • SYNC could return success without flushing anything (#77, thanks to Changwei Ye). If two clients synced at the same time, the second one consumed the first one’s counter and returned immediately, so data acknowledged as synced could be lost on a power failure.
  • A write could be lost while a newer write of the same object survived, and the object stayed at a version with no predecessor, that is, held a mix of two versions. Now all entries of one object which aren’t committed to the disk yet are written into the same metadata block and can’t disappear independently.
  • An object could disappear completely after a restart if the power went away in the middle of compacting it.
  • New write requests could simply hang after a restart.

Fixes for a checksum block larger than 4 KB

Old store only:

  • Reads of partially written big writes verified the checksum against the data on the disk instead of the explicit zero padding the checksum was computed with - so a read could fail with a spurious checksum error on perfectly good data. Without this fix, every chaos test seed of the affected configurations fails.
  • Two crashes during compaction in fill_partial_checksum_blocks().
  • A crash in read_bitmap() when reading a deleted version of an object.
  • Crashes with assert(fulfilled == read_op->len) in dequeue_read().

Other fixes

  • PGs are now repeered when pg_minsize changes.
  • Fixed the make-etcd script with IPv4 and pool auto-detection in snap-create (both broken in 3.1.0).
  • vitastor-dd now detects seekable inputs and outputs, including stdin/stdout.
  • In VitastorFS, the READDIR cookie offset and a stack overflow on a long READDIR from the cache are fixed, and the iterator cache size is now limited.
  • Fixed a potential crash on an incomplete socket receive, possible in theory in rare cases even with the MSG_WAITALL flag in use.
  • Fixed a crash on RDMA-CM init when RDMA-CM is unavailable.
  • Fixed a client crash on a missing pool or PG configuration during request execution.
  • Configurations with a metadata entry larger than meta_block_size are now rejected at OSD initialisation instead of leading to a division by zero in vitastor-disk.
  • Inverted ranges in DESCRIBE are rejected; the inode lookup during chained reads is guarded.
  • WRITE_STABLE over an unstable WRITE is also rejected in the old store (unused by the OSD and with undefined behaviour).
  • Fixed several UBSan and sanitizer complaints: negative disk offsets, NULL pointer arithmetic, zero-length memset on NULL, uninitialised memory.
  • Fixed a number of messenger corner cases: client IDs, outbox_push for a client that no longer exists, an incorrect initial EPOLLIN in the HTTP client.

The full commit log is available at the release link below.