Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Air-gapped install: mirroring BNK into a private registry

Some ROKS environments forbid pulling install resources from external repositories at deploy time. For these, roksbnkctl can replicate every artifact BNK needs — the F5 charts and images, plus the non-F5 dependencies — from the F5 Artifact Repository (FAR, repo.f5.com) into a private registry you control, and then install BNK entirely from there. With the mirror in place, the cluster needs no egress to repo.f5.com, quay.io, docker.io, or charts.jetstack.io to bring up BNK.

The mirror target is a registry you own — IBM Container Registry (ICR), the default, or a generic OCI-compliant registry (Artifactory, Harbor, Quay, registry:2). Choosing and configuring the target is its own chapter, Registry targets: ICR and Artifactory; this chapter is the air-gap flow itself.

The registry surface is CRUD-shaped, modeled on the IBM COS client:

CommandWhat it does
roksbnkctl registry targetShow or set the mirror target — icr / generic (see Registry targets)
roksbnkctl registry bomPrint the bill-of-materials — every chart + image BNK needs
roksbnkctl registry replicateMirror the BOM into the target registry
roksbnkctl registry adoptRecord a mirror that already exists, without re-copying (no source access)
roksbnkctl registry listWhat’s present in the target
roksbnkctl registry diffBOM vs. target — what’s missing or stale
roksbnkctl registry verifyEvery BOM artifact present + digest-matched
roksbnkctl registry pruneRemove mirrored artifacts no longer in the BOM
roksbnkctl registry deleteDelete all replicated artifacts from the target + clear the record

The bill-of-materials

The BOM comes from the f5-bigip-k8s-manifest that FAR publishes per BNK release — the same manifest roksbnkctl reads for version discovery. It enumerates every F5 chart and image (for BNK 2.3, that’s 25 charts + 56 images), tag-pinned. roksbnkctl unions in the two dependencies the F5 manifest does not cover — cert-manager (the Jetstack chart + its quay.io images) and the registry.k8s.io/kubectl node-labeler image — plus the manifest chart itself (release/f5-bigip-k8s-manifest), because the install needs to read it and a mirror that lacked it would send every install back to repo.f5.com. So the mirror is complete:

$ roksbnkctl registry bom
charts  repo.f5.com/charts/f5-lifecycle-operator   v2.21.13-0.0.28
charts  repo.f5.com/charts/f5-tmm                  15.430.5-0.2.157
...
images  repo.f5.com/images/tmm-img                 v10.159.3-0.1.5
images  quay.io/jetstack/cert-manager-controller   v1.17.3
images  registry.k8s.io/kubectl                   v1.36.0
...
81 F5 artifacts + 6 dependency artifacts

The walk-through

$ roksbnkctl cluster up               # the durable cluster
$ roksbnkctl registry target icr      # choose the target (icr is the default)
$ roksbnkctl registry replicate       # mirror the BOM (one-time per BNK version)
$ roksbnkctl registry verify          # confirm completeness
$ roksbnkctl bnk up                   # installs from the mirror

The walk-through runs registry replicate after the Cluster phase for convenience, but replicate needs no cluster. It is a purely host-side, registry-to-registry copy: it pulls each artifact by digest from the FAR source (repo.f5.com) and pushes to your target registry via go-containerregistry, from wherever roksbnkctl runs. So you can pre-seed the mirror as a standalone supply-chain step, before any cluster exists — all it requires is a configured registry: block, the FAR source credential, and network reachability to both repo.f5.com and your registry:

$ roksbnkctl -w mirror init --config-file mirror.yaml   # registry: block + bnk.far_auth_local_file
$ roksbnkctl -w mirror registry target generic          # or icr
$ roksbnkctl -w mirror registry replicate               # copies FAR → your registry, no cluster
$ roksbnkctl -w mirror registry verify

registry replicate is a deliberate, occasional step — it is not part of the composite roksbnkctl up (like gateway, you run it explicitly). With no --target, it uses the workspace’s registry.target (default icr). It copies each artifact by digest, idempotently — re-running only moves what changed (after a BNK version bump, registry diff shows exactly the changed artifacts).

How the install reads the mirror

registry replicate records what it copied — and where — in registry-mirror.json. The BNK phase reads that record and redirects the install to the target: every chart and image resolves from your private registry instead of FAR, so BNK comes up with no external pulls. The air-gap acceptance is exactly this, with egress to the external registries blocked.

The record is only used when it describes the mirror the workspace is currently configured for — same target kind, repository and host. A record outlives the mirror it describes: repoint registry.generic_repo_prefix at another repository, or switch registry.target, and the record on disk still names the old one. Nothing re-probes on read, so an unchecked record would redirect the whole install onto a registry nobody asked for, and terraform would apply it without complaint. bnk up refuses instead, naming both repositories:

the recorded registry mirror does not describe the configured mirror: it was
written for repository "bnk-mirror", the configured repository is "docker-local".
  Acting on it would use a mirror this workspace is not configured for.
  Re-record it:   roksbnkctl -w prod registry replicate   (needs the FAR source)
  Already filled: roksbnkctl -w prod registry adopt       (no source access needed)

The same check gates the node CA-trust install, which would otherwise push the old mirror’s CA onto every worker. When the configured mirror cannot be resolved from config at all — an unset generic_host, an ICR region with no known registry host — the record is left alone: knowing less than the record is not grounds for discarding it.

The target’s own pull credential is wired in for you:

  • ICR — the cluster authenticates to *.icr.io with iamapikey + the workspace IBM Cloud API key, for both the FLO chart pull and the image pulls.

  • Generic OCI — chart and image pulls authenticate with the same basic-auth credential replication used (registry target generic_username / generic_password), so a private registry needs no anonymous/public project. Concretely: chart pulls helm registry login with it, and the pods get a mirror-secret dockerconfig built from it, created in every namespace that pulls images — cert-manager, the FLO/BNK namespaces, and kube-system for the node-labeler — and referenced from the CNEInstance. You do not create any of it.

    This is the one place the two mirror kinds differ. An in-cluster/ICR mirror authorizes by RBAC and gets no pull secret; an external one (Harbor, Artifactory) gets mirror-secret. Dropping the pull secret for every mirror would force people to make their Harbor project world-readable — for a registry holding F5’s proprietary images, not an acceptable requirement.

Doing it all in CI, with nothing installed

Every command in this chapter runs unchanged inside the tools-runner container, so a pipeline can mirror the registry and install from it with no roksbnkctl, terraform, helm, kubectl or ibmcloud on the runner. The registry target itself comes from the environment — ROKSBNKCTL_REGISTRY_TARGET, ROKSBNKCTL_GENERIC_HOST, ROKSBNKCTL_GENERIC_REPO_PREFIX, ROKSBNKCTL_GENERIC_USERNAME, ROKSBNKCTL_GENERIC_PASSWORD — so nothing has to template a config.yaml:

docker run --rm -v "$PWD/state:/work" \
  -e IBMCLOUD_API_KEY -e ROKSBNKCTL_REGION=eu-gb \
  -e ROKSBNKCTL_PREFIX=ci -e ROKSBNKCTL_CLUSTER_NAME=my-cluster -e ROKSBNKCTL_CLUSTER_CREATE=false \
  -e ROKSBNKCTL_REGISTRY_TARGET=generic \
  -e ROKSBNKCTL_GENERIC_HOST=harbor.example.com \
  -e ROKSBNKCTL_GENERIC_REPO_PREFIX=bnk-mirror \
  -e ROKSBNKCTL_GENERIC_USERNAME=admin -e ROKSBNKCTL_GENERIC_PASSWORD \
  ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.19.0 -w ci \
  init --non-interactive --override-from-env

For the full disconnected picture — a private registry and licensing through a proxy in another cluster, as a two-job pipeline — see Flow C in CI.

Why the install never phones home

Two things have to be true for a mirrored install to be genuinely disconnected, and roksbnkctl handles both:

  1. The manifest is mirrored. f5-bigip-k8s-manifest is the BOM’s own source, so it is easy to overlook — but the install reads it to derive the FLO and CIS chart versions. It is a normal OCI chart, so it is replicated like any other artifact and pulled from the mirror.
  2. FLO reads the manifest from the cluster, not a registry. The F5 Lifecycle Operator resolves the BNK manifest by listing cluster-scoped CNEManifest CRs and matching spec.version; only when none matches does it fall back to pulling the manifest chart from the CNEInstance’s spec.registry.uri. roksbnkctl converts the manifest into a CNEManifest CR and applies it before the CNEInstance, so FLO finds it in the cluster and never fetches a manifest at all. (Without the CR, that fallback is what breaks a mirrored install: FLO reports “No CNEManifest exists which contains expected manifestVersion” and the CNEInstance never reconciles.)

One mirror, many clusters

Nothing ties a mirror to a single cluster. registry replicate is a supply-chain step, not a cluster step — the record it writes (registry-mirror.json) just tells the install where to pull from. So several workspaces can point at the same registry: replicate once, then every cluster installs from it.

The second workspace still needs a record of its own, though, because bnk up refuses to render against a mirror it has no record of. Use registry adopt:

$ roksbnkctl -w cluster-b registry adopt
  ✓ 10.241.0.4 holds 89 repositories under "bnk-mirror"
  ✓ recorded the mirror CA from 10.241.0.4 (nodes install it before pulling)
✓ adopted the mirror at 10.241.0.4/bnk-mirror — `bnk up` will render against it

adopt derives the record from the workspace’s own registry config and asks the mirror what it holds — it needs no access to repo.f5.com at all, which matters because an air-gapped operator usually has none. Re-running replicate would also work when the source is reachable, and copies nothing (every artifact is present and digest-matched), but it cannot run without the source. Add --verify-contents to adopt when you do have the source and want the mirror digest-checked before it is recorded.

That pairs naturally with a shared licensing cluster: one cluster holds the F5 License Proxy and reaches F5, one registry holds the artifacts, and any number of air-gapped clusters install from the registry and license through the proxy — reaching neither repo.f5.com nor F5’s licensing service themselves.

The per-target host, namespace/repository, and credential specifics — including the ICR namespace and a full Artifactory walkthrough — are in Registry targets.

TLS trust. Charts and images are pulled over the target’s HTTPS endpoint. ICR and a public Artifactory carry publicly-trusted certificates. For a custom or self-signed cert, give roksbnkctl the CA itself — registry target generic_ca <file> records it in the workspace (registry.generic_ca_b64), and bnk up installs it on every node before pulling, so you do not have to touch node trust by hand. roksbnkctl will not adopt a self-signed CA it merely discovered over the wire; see Configuration reference for the pin (generic_ca_sha256).

A truly disconnected cluster (no worker egress)

Mirroring removes external pulls at install time, but by default the ROKS cluster is still built with a public gateway on each subnet — its workers keep Internet egress. For a genuinely disconnected cluster with no egress at all, set:

cluster:
  create: true
  name: dc-roks
  public_gateway: false     # no public gateways → no worker Internet egress

init also prompts for this (“Attach public gateways for worker Internet egress?”); answer No for a private cluster. It renders the cluster_public_gateway = false terraform variable, so no ibm_is_public_gateway is created and no subnet attaches one. An omitted or true value is the current behavior (egress on).

This is an expert topology. A cluster with no egress can only pull images and reach IBM Cloud services over private paths that roksbnkctl does not build for you. Before setting public_gateway: false you must provide:

  • Private reachability to your mirror registry — e.g. the registry in a VPC the cluster reaches over a Transit Gateway, or a Virtual Private Endpoint (VPE). All BNK images/charts then come from the mirror (this chapter).
  • VPEs / private service endpoints for the IBM Cloud services the cluster needs (Container Registry if you mirror to ICR, COS for remote state, etc.).
  • Licensing without F5 egress — license through an in-cluster or VSI F5 License Proxy (bnk.license_mode: f5licenseproxy) that itself has the one controlled egress path to F5, or a disconnected license mode, so the cluster never calls F5 directly.

The public_gateway toggle only removes the egress path; wiring the private paths above is the operator’s responsibility. Note the cluster master keeps its public service endpoint regardless — this toggle governs worker/subnet egress, not the API endpoint.

bnk up checks the private paths before it installs

Because those private paths are yours to wire, bnk up verifies them rather than assuming them. Before it plans anything it probes the mirror — and the F5 License Proxy, when one is configured — from every node, in every availability zone, and fails if the mirror is unreachable from any of them.

The check has to run on a node. The host running roksbnkctl typically sits on the services VPC with egress, while the workers are air-gapped behind the gateway; a probe from there returns a confident green for a mirror the cluster cannot route to. This was not hypothetical during validation — registry adopt timed out reaching Harbor from the operator host while all three nodes reached it fine.

Without the gate, an unroutable mirror surfaced as ImagePullBackOff and then a helm context deadline exceeded about ten minutes later, naming neither the registry nor the node. A security group or subnet fixed in two zones out of three is a common shape, and one pod per node is what makes it visible.

A registry with no CA to install — already in the node trust bundle, or publicly signed — is still probed. Being trusted and being reachable are different things, and only the second fails silently.

Both of the gate’s timers are tunable under bnk.preflight; raise the retry budget when the Transit Gateway is attached in the same run as the install. The full output, rationale, and per-topology walkthroughs are in Appendix A.