Deterministic I/O and simulation testing for Zig.
Long term, Marionette is aiming to be the deterministic std.Io for Zig:
production libraries accept std.Io, and tests swap in Marionette's
deterministic implementation. Today, Marionette ships the simulator, trace,
fault, disk, and network primitives that make that direction concrete.
Marionette already runs real, unmodified Zig code under deterministic
simulation: a storage engine (xit-vcs/xitdb) and a cooperative-concurrency
library (g41797/mailbox), both with seed-reproducible replay, and has
surfaced reproducible recovery and correctness counterexamples in the process.
Today the demonstrated tiers are:
- deterministic
std.Io.Filestorage simulation with crash/torn-write faults; - cooperative
std.Iotask scheduling forMutex/Conditioncode; - scheduler-backed
std.Io.netstreams with deterministic latency, partitions, timeouts, healing, and retry; - scheduler-aware disk operations that park tasks behind earlier deadlines;
- experimental typed message modeling with deterministic loss, latency, and partitions.
Write production-shaped code against std.Io wherever possible, and add small
Marionette handles only when the application actually needs them. In tests,
drive control to inject faults. Production.env() supplies host I/O and the
rooted real disk; application-owned protocol seams receive Marionette endpoints
only in simulation.
const mar = @import("marionette");
fn writeAndRecover(io: std.Io, root: std.Io.Dir) !KVStore {
var store = try KVStore.init(io, root);
try store.put(1, 41, .sync);
try store.put(2, 99, .no_sync);
try store.recover(.strict);
return store;
}
// In simulation: deterministic, fault-injectable, replayable from a seed.
var world = try mar.World.init(std.testing.allocator, .{ .seed = 0xC0FFEE });
defer world.deinit();
const sim = try world.simulate(.{ .disk = .{ .sector_size = 16 } });
var sim_store = try writeAndRecover(sim.env.io(), std.Io.Dir.cwd());
// In production: real disk, same code path.
var tmp = std.testing.tmpDir(.{});
defer tmp.cleanup();
var production = try mar.Production.init(.{
.allocator = std.testing.allocator,
.root_dir = tmp.dir,
.io = std.testing.io,
});
defer production.deinit();
const prod_env = production.env();
var prod_store = try writeAndRecover(prod_env.io(), tmp.dir);
For file-backed code like this, that parity is the point. You don't write a "simulator version" of your code. You write your code behind Marionette-owned authorities, and Marionette gives you a deterministic environment to run it in.
Distributed and storage systems fail in ways that are hard to reproduce: a torn write under crash, a network partition during quorum, a race between two timers. By the time you have a stack trace, the conditions that caused the bug are gone.
Deterministic simulation testing turns those bugs into seeds. Every run is reproducible. Every failure is replayable. You compress weeks of fuzz-testing into seconds, and when something breaks in CI, the seed alone is enough to debug it.
Marionette brings that approach to Zig. It's inspired by the techniques behind FoundationDB, TigerBeetle, and Antithesis, but designed to be a drop-in library, not a framework you build your system around.
Here's a WAL recovery test that crashes the disk mid-write, corrupts a sector, and asserts that committed records survive while unsynced ones don't.
const Case = mar.SimCase(WalStore);
pub fn scenario(case: *Case) !void {
const store = &case.app;
const disk = case.control().disk;
try store.put(committed_key, committed_value, .sync);
try disk.setFaults(.{ .crash_lost_write_rate = .always() });
try store.put(volatile_key, volatile_value, .no_sync);
try disk.crash();
try disk.restart();
try disk.corruptSector(wal_path, record_size);
try store.reopen();
try store.recover(.strict);
}
pub const checks = [_]mar.StateCheck(Case){
.{ .name = "synced records recover, unsynced records are rejected", .check = recoveredStateIsSafe },
};
test "wal recovery" {
try mar.expectSimPass(.{
.allocator = std.testing.allocator,
.seed = 0xC0FFEE,
.simulate = .{},
.init = WalStore.init,
.scenario = scenario,
.checks = &checks,
});
}
test "wal recovery fuzz" {
try mar.expectSimFuzz(.{
.allocator = std.testing.allocator,
.seed = 0xC0FFEE,
.seeds = 16,
.simulate = .{},
.init = WalStore.init,
.scenario = scenario,
.checks = &checks,
});
}
mar.SimCase(App) is the standard wrapper for simulation tests: init
receives mar.Sim, scenario receives *mar.SimCase(App), and app state
lives at case.app. Harnesses that need lower-level World access or
unusual ownership drive mar.World directly.
Three pieces, either way:
init-mar.Sim.scenario-case.appand simulator authorities such as-case.control().checks-*const mar.SimCase(App).
expectSimPass runs twice with one fixed seed and compares the traces.
expectSimFuzz requires a nonzero seeds count and performs the same replay
check for each derived seed. expectSimFailure asserts that a deliberately
buggy scenario reaches a replayable failure, useful for proving your checker
actually works. Custom harnesses that outgrow these helpers drive mar.World
directly.
Every Marionette test has two halves.
** io** is what production-shaped storage code should usually see. In
simulation,
sim.env.io() returns Marionette's deterministic std.Io backend.
In production, production.env().io() returns the host std.Io supplied at
setup. Application code that wants trace events should accept a narrow
mar.Recorder, not all of mar.Env.var store = try KVStore.init(io, root, recorder);
try store.put(1, 41, .sync);
** control** is what your test code uses to inject faults. It's only available in simulation. It mirrors
env's structure: every resource has a control surface.try control.disk.crash();
try control.disk.corruptSector(path, offset);
try control.network.partition(&side_a, &side_b);
try control.network.setLossiness(.{ .drop_rate = .percent(20) });
try control.network.heal();
Env is still the harness-owned bundle that supplies io(), recorder(),
clock/random helpers, and remaining Marionette capabilities. Code that only
needs file I/O should prefer std.Io so it stays ordinary Zig code.
Network simulation works the same way. Here's a partition test against a toy replicated register:
fn partitionScenario(case: *Case) !void {
const isolated = [_]mar.NodeId{0};
const majority = [_]mar.NodeId{ 1, 2, client_node_id };
try case.control().network.partition(&isolated, &majority);
try case.app.write(.{ .version = 1, .value = 41, .retry_limit = 2 });
try case.control().network.heal();
try case.app.write(.{ .version = 1, .value = 41, .retry_limit = 1 });
try checkReplicaCommitted(&case.app, 0, 1, 41);
}
Messages have configurable loss, latency, clogs, and partition dynamics through focused control.network calls such as setLossiness(...), setLatency(...), and setPartitionDynamics(...). Application code sends through a node-scoped endpoint with endpoint.send(to, message) and receives with while (try endpoint.receive()) |envelope|.
The external-style std_io_net_kv example
imports only std and implements a fixed-frame PUT/GET service over
std.Io.net. In simulation, node-scoped std.Io handles come from
sim.envForNode(node).io(), so reconnecting sockets keep the same process
identity instead of consuming topology. Its Marionette harness partitions the
client from a queued response, surfaces the dropped delivery as error.Timeout,
heals the link, and retries the request. A correct server deduplicates the
retry; a planted buggy mode applies it twice and violates an exact revision
oracle.
Simulation processes are first-class runtime owners. sim.killProcess(node)
cancels that node's scheduler-backed Io.async/Io.concurrent work, closes
process-local files/listeners/connections, and wakes surviving TCP peers with
reset errors.
sim.registerProcess(node, lifecycle) plus sim.restartProcess(node) reruns a
registered initializer with that node's Env after volatile state has been
discarded by the lifecycle's on_kill callback. Explicit process kill leaves
the shared disk model intact; disk crash is the operation that first applies
pending-write crash faults, then uses the same process-kill path.
zig build validate-std-io-net-kv
zig build run-example -- std-io-net-kv --seed 12648430 --trace
zig build run-example -- std-io-net-kv-bug \
--seed 12648430 --trace --expect-failure
See Testing std.Io.net Code Deterministically for the trace and exact supported boundary. This is an external-style capability demonstration, not a third-party SUT finding.
Marionette has scheduler-backed cooperative std.Io tasks and futex waits for
Mutex / Condition code. Io.async and Io.concurrent run as deterministic
simulator tasks, and awaiting from either a task or the scenario drives the
same scheduler. The pinned g41797/mailbox validation target runs unmodified
and exercises timed receive, send/wake, same-deadline timeout ordering, and
byte-identical same-seed replay. The internal
validate-bounded-queue target adds a canonical FIFO oracle and a planted
lost-wakeup deadlock to demonstrate concurrency bug detection without counting
it as an external SUT finding. This is cooperative std.Io concurrency, not
preemptive OS thread or memory-model testing; see
Std.Io Direction for the exact boundary.
The optional validate-ochi target composes the pinned, unmodified
ochi-team/ochi storage implementation
with mar.SimCase. It starts the store, ingests a line, flushes index and data
tables, queries the inserted data, exercises atomic catalog replacement,
crashes the simulated disk, reopens Ochi, and queries the data again through
Marionette's deterministic std.Io. Ochi is a lazy dependency and this target
is not part of the default test step. The validation runs one simulation per
process because Ochi's private temporary-file counter intentionally persists
across in-process runs.
zig build validate-ochiThe optional validate-dusty target runs the pinned, unmodified
lalinsky/dusty HTTP client/server
library through Marionette's deterministic std.Io. dusty's real
Server.listen accept loop runs as a simulated task: its router, llhttp
parser, and connection pool serve a keep-alive GET/POST pair plus a second
connection over simulated std.Io.net streams with injected latency, and the
harness then shuts the server down through cooperative cancellation. Two
shutdown shapes are validated: a clean shutdown where error.Canceled lands
in dusty's accept park and the drain finds no active connections, and a
hung-connection shutdown where a keep-alive handler is still parked in a
read, dusty's drain times out, and its deferred Group.cancel sweeps the
parked handler on the way out. Fault scenarios partition the link before and
during a response, pin dusty's observed error.Timeout contract, heal and
retry against an exact body oracle, and sweep every chunk cut point while
rejecting short-success partial responses. Everything replays
byte-identically from the same seed. dusty is a lazy dependency and this
target is not part of the default test step.
zig build validate-dustyThis is an external-style capability demonstration of the std.Io.net
boundary, not a third-party SUT finding. The 0.6 slice chain is complete:
pooled keep-alive reuse and large byte-exact transfers landed with the
pool and transfer scenarios, which also found two confirmed dusty bugs
(see FOUND_BUGS.md), and opt-in task start jitter lets seed sweeps
explore start orderings such as connect-before-listen races.
The optional validate-beanstalkz target runs the pinned, unmodified
g41797/beanstalkz beanstalkd
work-queue client against a harness-owned in-memory beanstalkd that speaks
the text protocol over simulated std.Io.net streams. It covers a
produce/consume round trip with bury/kick state transitions, sequential
connection churn (fresh connect, put, quit, shutdown(.both), close cycles
drained in FIFO order), a blocking reserve-with-timeout parked across a
five-second virtual publish delay, and a server-process crash under a
parked reserve: the surviving client wakes with a reset that surfaces as
the library's CommunicationFailure contract, then a registered process
restart recovers on a fresh incarnation. Everything replays byte-identically
from the same seed. beanstalkz is a lazy dependency and this target is not
part of the default test step.
zig build validate-beanstalkzEvery run produces a structured trace. When a check fails, you get the full sequence of events that led to the violation, plus the seed to reproduce it.
register.write.start version=1 value=41 retry_limit=8
register.message kind=propose to=0 version=1 value=41
replica.accept replica=0 version=1 value=41 accepted=true
register.message kind=propose to=1 version=1 value=41
replica.accept replica=1 version=1 value=41 accepted=true
register.write.quorum version=1 value=41 acks=2
register.invariant_violation kind=committed_divergence replica=1 ...
You write trace records with mar.Recorder or, inside harness-shaped code,
env.record(...). Application code, scenario code, and checks can all record.
Failed runs print the trace automatically. Passing runs hand it back to you so
you can persist it, diff it, or feed it to whatever observability you already
have.
- Overview
- Architecture
- Trace Format
- Run
- Findings
- API Target Spec
- BUGGIFY
- Network Model
- Network API Direction
- std.Io.net Client/Server Example
- Disk Fault Model
- API
- Determinism
- Examples
- Roadmap
- Prior art
- TigerBeetle Lessons
- Releasing
- Blog
Marionette is early. This is a 0.x release: there is no API stability
guarantee before 1.0. The intended-stable surface today is World, Env,
Control, SimCase, runSimCase / expectSim*,
Disk, SimDisk, RealDisk, Production, and Recorder. The public
Endpoint(Message) message-modeling surface remains experimental while its
ownership and transport contract are validated against a real SUT.
The simulator currently models clock, deterministic randomness, disk, a
directory-aware std.Io.File/Dir subset, experimental typed message modeling, a narrow scheduler-backed
std.Io.net stream subset with accept/read suspension plus latency and
send-time loss, delivery-time partitions, deterministic healing, and
literal-only host lookup (an unmodified std.http.Client runs against
simulated servers; real DNS stays unsupported), and
cooperative std.Io tasks, groups, and futex waits for Mutex / Condition code,
validated against the pinned g41797/mailbox and g41797/beanstalkz
targets and the internal bounded-queue capability demo, plus the pinned
Ochi storage target. It does not model
arbitrary OS thread scheduling or memory-level concurrency; code that depends on
those needs separate testing. The simulator also models
deterministic allocation faults through Env.allocator() and cooperative
cancellation: Future.cancel and Group.cancel deliver error.Canceled at
futex, sleep, and net suspension points following std.Io's one-shot
protocol. A one-shot sim.transitionToLiveness(core) ends the fault regime
so bounded runs can prove the core makes progress once faults stop.
Networking has two sibling testing altitudes. Node-scoped std.Io.net is the
canonical literal same-code path for codecs, framing, partial I/O, and stream
lifecycle. Experimental Endpoint(Message) explores protocol/state-machine
behavior above the wire; production uses an application-owned transport seam,
and Marionette does not claim that the real transport runs through the endpoint
model. The former Marionette-owned production adapters were removed in 0.6.
Queue suspension and broader scheduler parity are planned.
Scheduler-backed fibers are tested on Linux and macOS. The x86_64 Windows
fiber path is deliberately disabled until its Win64 entry ABI has execution
coverage. RealDisk.syncDir returns error.DirectorySyncUnsupported because
Zig 0.16 does not expose a portable directory-sync operation; it never reports
durability that it did not perform.
If you're building something where determinism matters and you want to try it, the examples/ directory is the best place to start. Open issues and PRs welcome.
zig fetch --save https://github.com/sb2bg/marionette/archive/refs/tags/v0.6.1.tar.gzThen wire the module into your test build in build.zig and import it:
const marionette = b.dependency("marionette", .{
.target = target,
.optimize = optimize,
});
tests_module.addImport("marionette", marionette.module("marionette"));
const mar = @import("marionette");Requires Zig 0.16.x.
Marionette stands on the shoulders of FoundationDB's simulation testing, TigerBeetle's VOPR, and the broader DST tradition. The bugs they catch are bugs everyone has; this library tries to make catching them easy in Zig.
MIT