Architecture · 17 July 2026 · By Keegan Ott · 6 min read

Rebuilding an Azure event-driven system for air-gapped Kubernetes

The application already worked in Azure. My job was to make it work without Azure, without flattening its durable workflows into a pile of background jobs.

I worked on this problem while engineering a self-hosted edition of an Azure-first .NET platform. The original system used event-driven Azure Functions and durable orchestrations. The target had no access to Azure and needed to run on Kubernetes with locally operated messaging, persistence and scaling.

At first it sounded like a translation exercise: managed service on the left, self-hosted equivalent on the right. That framing lasted until the first serious inventory. A product name was often covering three or four behaviours, and the replacement matched only the visible one. The work became less about choosing technology and more about finding the promises the old platform had been keeping for us.

I can’t publish the client system or its operational details, so this is the part worth discussing in public: which behaviours had to survive, where the obvious replacements were misleading, and what I would try to break before calling the port finished.

The first spreadsheet was more useful than the first prototype

A dependency inventory that says “replace Azure Functions” is too shallow. Functions may be providing several distinct behaviours:

• message-triggered execution and retry;
• durable orchestration state and replay;
• timers, fan-out and fan-in;
• horizontal scaling based on queue depth;
• dead-letter handling and operational visibility.

I wrote those down as requirements. Otherwise it was too easy to point at a running container and declare the Function replaced, while retries or dead-letter handling had quietly vanished.

The target architecture

The self-hosted design split the workload into containerised .NET services deployed to Kubernetes. RabbitMQ supplied asynchronous messaging, SQL Server held application and workflow state, and KEDA connected queue activity to Kubernetes scaling.

The rough mapping ended up like this:

• Azure Functions became independently deployable .NET workers.
• Azure Service Bus responsibilities moved to RabbitMQ exchanges and queues.
• Durable orchestration state moved to SQL Server.
• Azure’s consumption scaling became KEDA-driven replica scaling, including scale-to-zero where safe.
• Cloud configuration and secrets became deployment-owned Kubernetes integrations.

Port the orchestration semantics carefully

Durable workflows are where superficial migrations tend to fail. An orchestration is not merely a sequence of method calls. It has persisted state, deterministic replay, activity retries, timers and rules about what may happen inside the orchestrator.

We preserved that programming model by moving durable state to SQL Server and maintaining the necessary provider integration. That allowed existing workflow intent to survive without pretending that ordinary background jobs were equivalent.

Determinism still applies

Replay-based orchestration code must remain deterministic. Current time, random values and external I/O belong behind activities or framework-provided deterministic APIs. A platform change is a good time to add replay tests around any workflow that has accumulated incidental side effects.

Idempotency belongs in the application

At-least-once delivery means duplicate execution is normal, not exceptional. Consumers need stable operation identifiers, transactional state transitions and safe handling when the same message arrives again. Broker acknowledgements alone cannot provide end-to-end exactly-once behaviour.

RabbitMQ needed decisions, not defaults

A queue per worker was a starting point, not a topology. I still had to decide who owned each queue, what could retry, and which failures needed isolating:

• durable exchanges and queues as deployment configuration;
• versioned contracts that can coexist during rolling upgrades;
• bounded retry paths rather than instant requeue loops;
• dead-letter queues with enough context to diagnose and replay safely;
• prefetch limits matched to workload cost and concurrency.

Wolverine provided useful .NET messaging and handler conventions, but framework convenience does not remove the need to decide delivery semantics explicitly.

Scale on work, not CPU

CPU is a poor leading indicator for asynchronous consumers. Queue depth and message age describe demand more directly. KEDA can expose those signals to Kubernetes and adjust replicas accordingly.

Scale-to-zero is valuable in resource-constrained environments, but only for workers whose startup latency and connection behaviour tolerate it. Minimum replicas, polling intervals, activation thresholds and graceful shutdown all need workload-specific values. A pod terminated while holding an unacknowledged message should return that work cleanly to the queue.

Disconnected operations change the definition of done

An application is not air-gap-capable merely because its runtime images are local. Every transitive dependency must be accounted for:

• container images and base images;
• NuGet packages and build tooling;
• Kubernetes charts and operators;
• licence checks, certificate chains and time sources;
• telemetry exporters, identity providers and update mechanisms.

Build and deployment must succeed against an internal registry and package source with external network access removed. Testing that constraint early is much cheaper than discovering a hidden download during installation.

What I would validate before cutover

This is where I stop trusting architecture diagrams. A diagram cannot show whether a half-finished activity is safe to repeat, or whether the installer has one forgotten internet dependency. I want the ugly tests:

• Kill consumers during processing and verify redelivery is safe.
• Restart the broker and database during active orchestrations.
• Run old and new message-contract versions together.
• Fill queues faster than consumers can drain them and observe scaling limits.
• Prove dead-letter inspection and controlled replay.
• Install from an empty, disconnected environment using only approved artefacts.
• Restore state from backup and resume incomplete workflows.

What I took away from it

I came away less interested in whether an architecture is called cloud-native. The portable part was much less glamorous: explicit messages, durable state, failures an operator can see, independent scaling and a release process that works without a helpful internet connection.

Azure had been doing more work for us than the architecture diagram admitted. Outside it, every retry, lock, scale rule and recovery path had an owner. I considered the port credible when duplicate messages and dead nodes became routine tests rather than scary edge cases.

Keegan Ott
Cloud-native data platform developer in Perth, Australia. Online as dreeko.