Architecture · · By Keegan Ott · 3 min read
Durable workflows outside Azure
The first tempting replacement for Durable Functions is a queue and a background worker. I tried to avoid that shortcut: it keeps the work moving, but quietly drops most of what made the workflow durable.
A durable orchestration records progress so it can survive process death, machine replacement and long waits. Its apparent control flow is reconstructed through replay. Activities perform side effects; the orchestrator decides what happens next.
This was the part of the port that made me slow down. Workflow code looked ordinary enough to invite a rewrite, while hiding years of failure handling in the runtime underneath it. The closer I looked, the less comfortable I was calling a queue consumer equivalent.
The awkward inventory
- persisted execution history and recoverable state;
- deterministic replay;
- durable timers and external events;
- fan-out, fan-in and sub-orchestration;
- activity retry policies and terminal failure;
- versioning for workflows already in flight.
Writing this list down was useful because it stopped “we can build that” from passing as a design. A job runner is fine for many jobs. It just is not an orchestration engine.
Replay changes how code is written
Orchestrator code may execute repeatedly. Reading current time, generating randomness, performing network I/O or mutating external state inside it produces different results on replay. Those operations belong in activities or deterministic framework APIs.
Persistence is a protocol
My first concern with SQL Server was the transaction boundary, not the schema. History appends, work-item locks, completion and new scheduling must have coherent transactional boundaries. Competing workers need leases or equivalent concurrency control, with recovery when an owner disappears.
At-least-once activity execution also makes idempotency mandatory. Stable operation identifiers, uniqueness constraints and inbox/outbox patterns prevent a replayed activity from charging, publishing or updating twice.
Version running instances
Deployments occur while workflows are waiting. Changing branch logic or activity names can make recorded history incompatible with new code. Safer strategies include versioned orchestrators, additive contracts, explicit migration points and allowing old definitions to drain.
I prefer the boring option here: keep an old definition alive long enough to drain rather than perform clever surgery on persisted history. It costs some deployment tidiness, but it is easier to explain at 2am when one long-running instance refuses to move.
The tests I care about
Before cutover, restart workers and the database during each workflow stage; inject duplicate completions; delay activities beyond leases; test poison inputs; restore a backup containing unfinished instances; and confirm operators can inspect, terminate and replay work safely.
I would be suspicious of any migration signed off only with a happy path. The useful evidence is an unfinished workflow resuming after its worker and database have both had a bad day.