Delivery · · By Keegan Ott · 3 min read
Building a genuinely offline Kubernetes release
Getting the main image into a private registry is the easy part. The interesting failures come later, when an installer reaches for one forgotten chart, certificate endpoint or utility image.
I start with the network cable test
On a clean cluster, I enumerate everything installation or operation can fetch: application and init images, operator images, charts, CRDs, NuGet or npm packages used during build, database tooling, licence services, identity metadata, certificate chains, telemetry endpoints and update checks.
Pin artefacts by version and preferably digest. Mutable tags turn a reviewed bundle into a moving target and make later reconstruction difficult.
The tedious entries are usually the valuable ones. The main services are obvious; the tiny migration container or chart dependency everybody forgot is what stops an installation in a room where nobody can simply pull it from the internet.
What actually goes on the transfer media
- OCI images with a machine-readable manifest;
- Helm charts and all chart dependencies;
- configuration schemas and environment-specific overlays;
- database migrations and rollback guidance;
- SBOMs, checksums, signatures and licence material;
- installation, verification, backup and recovery tooling.
The bundle should be transferable, inspectable and importable without running an internet-connected package manager.
Separate product from environment
Keep immutable product artefacts separate from site configuration. Operators should supply registry locations, storage classes, ingress, certificates, identity endpoints and resource policy without rebuilding application images.
Plan trust and time
Disconnected environments still need certificate validation and reliable time. Decide how internal certificate authorities enter trust stores, how certificates rotate, how signatures are verified offline and what happens when clocks drift. Public OCSP, cloud key vaults and externally hosted identity discovery documents are common hidden dependencies.
This work changed how I read “no external dependencies” in a design document. I now treat it as a claim to test, not a property inherited from running on-premises. DNS, clocks and certificate validation have a habit of exposing the difference.
Upgrades are where the fiction breaks
Every release should state supported upgrade paths, required capacity, schema compatibility and rollback limits. Test upgrade with realistic retained data, then restore from backup. Kubernetes rollback does not reverse a destructive database migration.
The release is finished when this works
Provision an empty representative environment, deny all egress, import only the approved bundle, install using the operator runbook, exercise core workflows, restart dependencies, rotate a certificate, collect diagnostics and restore state. Record every undeclared prerequisite as a release defect.
This is slower than calling a connected-cluster deployment “offline ready”. It is also the difference between a release and a box of hopeful artefacts.