During what should have been a routine deployment, our pipeline briefly ran two copies of itself at the same time. Nothing failed. No alerts fired. For that window, though, the system was producing output that could not be trusted.

Some context. We had recently migrated a critical data pipeline off a managed Python ETL service and onto long-running Rust containers on a standard orchestration platform. The performance improvement was significant: a run that took 160 seconds in Python finished in 20 seconds in Rust, on a pipeline that processes millions of records per execution and fires every two and a half minutes. But the migration story that stayed with me was not about performance. It was about a bug we almost did not find.

The Setup

The pipeline generates periodic file datasets consumed by a downstream control system. Every two and a half minutes it produces a batch of files that the consumer reads as a single coherent unit, then makes automated decisions based on what it finds. It learns that a new dataset is ready through an event notification.

The consumer has no way to distinguish partial correctness from full correctness at ingestion time. It assumes each output is internally consistent and complete. Nothing in the ingestion path verifies that. If the files it reads were produced by two different processes in two different states, every decision made downstream is built on data that should never have been combined.

"Mostly correct" is not an acceptable standard for a system making automated decisions in real time.

The Bug

When we moved to containers, we used the orchestrator's built-in rolling deployment. It starts the new task, waits for a health check heartbeat, then shuts down the old one.

If you have ever assumed that a rolling deployment guarantees a clean handoff between tasks, this is where that assumption breaks.

There is a window, bounded by the health check interval, during which both the old and the new task are alive and running. We expected a handoff. We got parallel execution.

We caught the symptom during beta: multiple job runs completing for the same time window. The old task and the new one were both generating files concurrently and writing to the same storage paths, with no coordination between them.

I traced through the worst case sequence.

Task A completes a full run. All files written. Dataset internally consistent.

Task B starts its own run, writes the first half of the files, then receives a termination signal and stops.

The first half of the dataset now comes from Task B and the second half from Task A.

At this point you might assume the output is still valid. Same input data, same code, so the records should be identical no matter which task produced them.

But the Rust dataframe library we used sorts with an unstable algorithm by default. Identical inputs can produce the same records in different orders across separate runs. That behavior is harmless when each run is consumed independently. Here it was what corrupted the data.

Task B's first half and Task A's second half carried the same records arranged in incompatible sequences, split across a file boundary the consumer would never know existed. The notification fired. The consumer ingested the dataset. The ordering assumptions it relied on no longer held.

No alerts. No monitoring surface pointed here. The only signal was overlapping timestamps that should never have existed, and two engineers staring at a log wondering why one time window had two completed runs.

Two Ways Out

Once we understood the failure, we mapped out two fixes. The question we argued about was where to put the failure boundary: in coordination, or in isolation.

Option 1: A distributed lock

Before writing any files, a task acquires a lock in a key-value store that supports conditional writes. If another task already holds it, the new one backs off and waits. Locks carry a lease so a stuck lock does not block the pipeline indefinitely. During the deployment window at most two processes contend for it, the expiring old task and the starting new one, and only one proceeds.

This was my preferred approach. The failure boundary is explicit, and you can inspect lock state directly when something goes wrong. The tradeoff is a hard dependency on an external system: if the lock store degrades during a deployment, the pipeline stalls until the lease expires or the dependency recovers.

Option 2: Run-ID isolation

Each task generates a unique identifier at startup and writes its output into a directory named after that identifier. No two tasks share a directory. Overlapping runs produce output in completely separate locations and never touch the same paths.

My team chose this one. The boundary here is isolation instead of coordination. No shared state, no conflict. There is no external dependency to fail. The tradeoff is storage: orphaned directories from incomplete runs accumulate and need periodic cleanup, which we had not fully accounted for at the time. Both complete and incomplete datasets were retained.

Both options are correct. We were picking which failure mode we wanted to own, and the team's reasoning was sound. Adding a coordination dependency to fix a correctness problem trades a consistency risk for an availability risk. Option 2 keeps the failure surface entirely inside the application.

I still think Option 1 is the cleaner design. But I understand why we did not go that way.

What Changed

After the fix shipped, overlapping runs and mismatched dataset timestamps mostly stopped showing up. Deployments that previously required careful timing now behave predictably regardless of when they land relative to a pipeline execution window.

The Actual Lesson

Orchestration systems guarantee availability, not correctness. A passing health check means the new task is alive. It says nothing about whether the old task has stopped writing, or whether the data your downstream consumer is about to ingest came from one process or two.

If two instances of a stateful workload can run at once, even for a few seconds inside a deployment your orchestrator treats as routine, correctness is your application's job. You have to build that guarantee yourself.

Orchestration systems guarantee availability. Everything else is your problem.