Preface
Foreword
Standing up F5 BIG-IP Next for Kubernetes (BNK) on IBM Cloud Red Hat OpenShift (ROKS) used to be a multi-step deployment that hit a different surface at every step. A terraform init/plan/apply against an HCL tree somebody handed you. A manual ibmcloud ks cluster config to pull a kubeconfig. A separate IBM Cloud CLI install with its own apt-source dance. A manual oc adm policy add-scc-to-user privileged to let iperf3 actually run. SSH, environment variables, kubeconfig — each a small thing, and together half a day before BNK was even on the cluster.
roksbnkctl collapses that into a single static binary plus four interchangeable execution backends (local, docker, k8s, ssh:<target>) plus an opt-in in-cluster ops pod. One command brings a workspace up; one command tears it down; the connectivity, DNS, and throughput tests run from whichever network vantage the question actually requires. The tool exists because the manual path has too many moving parts for somebody who just wants to evaluate BNK or run a customer demo.
This book is the user-facing documentation for roksbnkctl, and ships with the binary.
Who this book is for
Four audiences:
- BNK evaluators kicking the tires on F5 BIG-IP Next for Kubernetes who want a low-friction path to a working trial deployment.
- F5 sales engineers (SEs) who need a repeatable demo and proof-of-concept toolchain for customer engagements.
- Customer engineers standing up BNK in their own IBM Cloud account, either for evaluation or as the foundation of a production rollout.
- Contributors who want to extend
roksbnkctl— add a backend, add a test suite, ship a new chapter. See Part IX — Contributing.
How to read this book
The book is organised so it can be read either way.
- Linear: Parts I-VII walk from concepts through your first deployment, day-2 operations, and the built-in test suite. New users should read in order.
- Reference: Part VIII (Command reference, Configuration reference, Terraform variable reference, Glossary) is exhaustive and indexed for lookups during day-to-day work. Search at the top of every page also reaches into reference material.
If you have 30 minutes and an IBM Cloud account, skip straight to Chapter 7 — Quick start. It’s the canonical “first cluster up” walkthrough and the rest of the book makes more sense after you’ve seen the happy path end-to-end.
Part IX is for contributors who want to build roksbnkctl from source or extend it.
Prerequisites
This book assumes:
- Basic familiarity with IBM Cloud — you have an account, you know what an API key is, and you’ve used the IBM Cloud console at least once.
- Basic familiarity with Kubernetes — you know what a pod, service, and namespace are; you’ve run
kubectl(oroc) before. - A working terminal on Linux or macOS. Windows is supported for
roksbnkctlitself, with documented limitations around interactive SSH (see Chapter 16).
You do not need prior experience with:
- Terraform —
roksbnkctlembeds a vetted HCL tree and drivesterraformfor you. You can ignore the underlying HCL until you want to customize it (Chapter 13). - OpenShift specifics — the tool treats ROKS as Kubernetes with a thin SCC + project overlay; the few OpenShift-specific gotchas are called out in Chapter 22 and Chapter 26.
- F5 BIG-IP Next — BNK is the thing the book deploys; you don’t need to be a Big-IP engineer to evaluate it. Chapter 1 is the 5-minute “what is this product” primer.
Book conventions
- Code blocks: shell commands use
bashsyntax highlighting; YAML snippets useyaml; HCL fragments usehcl; sample command output is shown in plaintextblocks to distinguish output from input. - Cross-references: every chapter ends with a “Cross-references” section linking related chapters. Inline links use the form
[Chapter 7 — Quick start](./07-quick-start.md)— a chapter number, an em-dash, the chapter title, and the relative path to the chapter source. - PRD links: design documents under
docs/prd/are linked as full GitHub URLs (e.g.https://github.com/jgruberf5/roksbnkctl/blob/main/docs/prd/03-EXECUTION-BACKENDS.md) so they resolve from the published book at GitHub Pages. The PRDs are the design surface; the book is the user surface — read PRDs only if you’re contributing or want the why behind a design call. - Forward references to post-v1.0 work: where a feature is explicitly queued for a v1.x release (e.g.
terraform --backend k8s, multi-hop SSHProxyJump), the prose flags it in future tense and points atdocs/PLAN.md§“What’s deliberately deferred to post-v1.0” for the roadmap.
Welcome.
What is BIG-IP Next for Kubernetes (BNK)
F5 BIG-IP Next for Kubernetes (BNK) is F5’s containerised, Kubernetes-native re-imagining of the BIG-IP data plane. It runs the BIG-IP Traffic Management Microkernel (TMM) as pods inside a cluster, and exposes its configuration surface through Custom Resources rather than the classic TMSH / iControl REST APIs. The point of BNK is to give Kubernetes workloads the L4 and L7 traffic management features F5 customers already rely on — advanced load balancing, TLS termination, WAF policy, GSLB — without bolting an external appliance onto the cluster’s edge.
This chapter sets the context for the rest of the book. If you already deploy and operate BNK day-to-day you can skim it; if you arrived here knowing generic Kubernetes but new to F5’s product family, read it first.
Where BNK fits in F5’s product family
F5 has historically delivered traffic management as the BIG-IP appliance: a hardened Linux box (physical or virtual) running TMOS, with TMM as the data-plane kernel module. BIG-IP works extremely well at the cluster edge — north-south traffic — but it sits outside the cluster and is configured through its own control surface.
The next-generation lineage is BIG-IP Next: the same TMM data plane refactored to run as a regular Linux process, configurable through declarative APIs instead of imperative TMSH. BIG-IP Next ships in three deployment shapes:
- BIG-IP Next for VMs / Bare Metal — same form factor as classic BIG-IP, modernised control plane.
- BIG-IP Next Service Proxy for Kubernetes (SPK) — telco-focused, for 5G core workloads.
- BIG-IP Next for Kubernetes (BNK) — general-purpose, runs inside any conformant Kubernetes cluster.
BNK is the focus of this book. It is the option you pick when your workloads already live in Kubernetes and you want F5 traffic management without standing up a separate appliance fleet.
What problems BNK solves
A standard Kubernetes cluster ships with a basic Service / Ingress story: kube-proxy iptables rules, a community ingress controller, maybe an external load balancer in front. That covers the common case but falls short when you need:
- Real L7 traffic management for north-south traffic — fine-grained routing, header manipulation, TLS termination with custom cipher suites, mTLS enforcement, advanced HTTP/2 + HTTP/3 handling, WAF policy enforcement at the edge.
- East-west service mesh-style features without a sidecar — connection pooling, circuit breaking, retries, observability for pod-to-pod traffic, applied at a per-namespace or per-workload granularity.
- GSLB-style global traffic management — health-checked DNS responses that send a client to the nearest healthy cluster, integrated with the cluster’s own service health.
- Compliance and regulated workloads — DDoS mitigation, behavioural anomaly detection, audit logging that an enterprise security team will accept.
BNK delivers all of the above as cluster-native primitives. You install it once, and from then on you express traffic management intent through CRDs (F5BigIpCtx, F5IngressTls, F5GslbPool, etc.) committed alongside your application manifests.
The components
BNK isn’t a single binary; it’s a set of cooperating components installed into a cluster. The pieces you’ll see most often:
- TMM (Traffic Management Microkernel) — the data plane. Runs as DaemonSet pods on dedicated worker nodes. Every packet handled by BNK passes through TMM.
- FLO (F5 Lifecycle Operator) — the control-plane operator. Watches BNK CRDs and reconciles them into TMM data-plane configuration. Owns the lifecycle of the TMM pods themselves: image pulls, version upgrades, rolling restarts.
- CIS (Container Ingress Services) — Kubernetes-native ingress controller piece. Watches
Ingressand BNK ingress CRDs, programmes TMM to terminate the corresponding traffic. - CNE Instance — Cloud-Native Edge configuration, the umbrella resource that ties a BNK install to its tenant context.
- Cert-Manager — not strictly an F5 component, but a hard dependency. BNK uses cert-manager to mint and rotate the certificates TMM presents to clients.
Deeper chapters reference these names; you don’t need to memorise them now. The thing to take away is that BNK is an operator + DaemonSet pattern: the operator (FLO) reconciles your declarative intent into running data-plane pods (TMM).
Where BNK runs
BNK runs on a conformant Kubernetes cluster. F5 publishes a support matrix — read it for definitive answers — but in practice you’ll see BNK deployed on:
- Managed Kubernetes: ROKS (IBM Cloud’s managed OpenShift), OpenShift Dedicated, EKS, AKS, GKE.
- Self-managed OpenShift on bare metal or VMs.
- Upstream Kubernetes in private clouds, with an LB provider that BNK can integrate with.
This book targets ROKS specifically. The next chapter explains why. The decisions and patterns documented here will translate to other Kubernetes flavours, but the bundled Terraform that roksbnkctl ships only knows how to provision ROKS.
North-south and east-west, in one install
It’s worth calling out explicitly: BNK is not “just an ingress controller” and it’s not “just a service mesh data plane”. It’s both, in the same install:
- North-south (client outside the cluster talking to a workload inside) — BNK fronts a
LoadBalancer-typed service, terminates TLS, applies WAF policy, routes to backend pods. Replaces the role a hardware BIG-IP or community ingress controller would play. - East-west (pod-to-pod or namespace-to-namespace inside the cluster) — BNK can be inserted into the path with no application sidecar, providing per-workload connection pooling, retries, and observability.
A single BNK install can handle both at once. Customer architectures often start with the north-south story (the obvious replacement for an existing BIG-IP appliance), then expand into east-west as the team gets comfortable with the operator-driven configuration model.
Pointer to F5’s official docs
Everything in this chapter is intentionally a sketch — enough to make the rest of this book legible. For definitive and up-to-date product information, including the full CRD reference, version compatibility matrix, sizing guidance, and license model, see F5’s official BNK documentation: https://clouddocs.f5.com/bigip-next/latest/.
The rest of this book focuses on deploying BNK with roksbnkctl and validating that the deployment works end-to-end. It does not duplicate F5’s product documentation; it complements it.
For an at-a-glance view of how roksbnkctl’s components fit together — the four execution backends, the cluster, the jumphost, the IBM Cloud control plane — see the architecture diagram at the top of Chapter 17 — Execution backends. For the happy-path lifecycle from one command to the next, see Chapter 7 — Quick start.
Why ROKS (Red Hat OpenShift on IBM Cloud)
This book and the roksbnkctl tool target ROKS — IBM Cloud’s managed Red Hat OpenShift offering — specifically. Other Kubernetes flavours can run BNK, and most of the patterns you’ll learn here translate, but the bundled Terraform that roksbnkctl ships only knows how to provision a ROKS cluster.
This chapter explains the rationale behind that choice. If you’re already using ROKS, you can skim this. If you’re evaluating whether ROKS is the right substrate for your BNK trial, read in full.
What ROKS is
ROKS is short for Red Hat OpenShift on IBM Cloud. It’s IBM Cloud’s managed-OpenShift service: you ask IBM for a cluster, IBM provisions the masters, etcd, the OpenShift control plane, and a pool of worker nodes; you get a kubeconfig and start deploying.
ROKS clusters are real OpenShift. They run the same Operator Lifecycle Manager (OLM), the same oc CLI, the same SecurityContextConstraints (SCC) model, the same routes-and-services machinery you’d find on any OpenShift install. The only thing IBM has done differently is take responsibility for keeping the control plane and the underlying infrastructure healthy.
What IBM manages, what you manage
The boundary between “IBM’s responsibility” and “your responsibility” is the principal value proposition of any managed Kubernetes service. For ROKS the line falls roughly here:
| Concern | Owner |
|---|---|
| Master nodes (API server, scheduler, controllers) | IBM |
| etcd (persistence + backups) | IBM |
| OpenShift control plane (OLM, ingress operator, image registry) | IBM |
| OpenShift version upgrades for the control plane | IBM (you opt in to a major-version bump) |
| Worker node provisioning (VPC VSIs, subnets, security groups) | IBM, on your behalf via the cluster API |
| Worker node OS patching and CVE remediation | IBM |
Worker pool sizing and lifecycle (workers create/delete) | You |
| Pod workloads running on the cluster | You |
| Application-level RBAC, network policy, TLS, service accounts | You |
| BNK install + configuration | You — this is what roksbnkctl automates |
The thing to internalise: with ROKS you do not rack hardware, install RHEL, run openshift-install, manage etcd backups, or chase CVE patches across a worker fleet. IBM does all of that. You start at “I have an OpenShift cluster” and go from there.
Why managed-OpenShift over self-managed for BNK evaluation
If you want to evaluate BNK quickly, the calculus is straightforward. Self-managed OpenShift is a multi-week lift before you have a cluster:
- Provision the underlying VMs (OpenStack / vSphere / bare metal).
- Run
openshift-installand debug whatever doesn’t go right. - Configure DNS, load balancers, container registry mirrors.
- Stand up monitoring + logging + cert-manager.
- Now you can start thinking about BNK.
ROKS compresses that to one Terraform apply of ~50 minutes. You get back a kubeconfig that authenticates against a real OpenShift cluster, with cert-manager already installable via OLM, and a worker pool of the size and zone topology you specified. From there, the BNK install is the same set of CRDs and Helm charts it would be on any OpenShift cluster.
For a sales-engineering demo or a customer proof-of-concept, “I have a cluster in 50 minutes” beats “I have a cluster in 2 weeks” every time. That trade-off is the reason this book exists in this shape.
Why OpenShift (not just any Kubernetes) for BNK
BNK runs on conformant Kubernetes generally, but it integrates more cleanly with OpenShift specifically because:
- Operator-driven install — BNK is shipped as a set of operators. OpenShift has Operator Lifecycle Manager (OLM) as a first-class citizen, so the install pattern is familiar to OpenShift admins.
- SecurityContextConstraints (SCC) — TMM pods need elevated capabilities (notably
NET_ADMIN, raw socket access, hugepages). OpenShift’s SCC model formalises that grant; on upstream Kubernetes you’d be configuring PodSecurityAdmission policies by hand. - Routes — OpenShift’s
RouteCRD predates and is more capable thanIngress. BNK can act as an alternate Route implementation, slotting into existing OpenShift application architectures without forcing teams to migrate. - Image streams + the internal registry — useful for the BNK supply chain (FAR images, license bundles) which can be mirrored once and consumed by many installs.
If you’re already an OpenShift shop, BNK fits naturally. If you’re not, BNK still works but you’ll need to translate this book’s OpenShift-specific examples (SCCs, oc adm policy, Route) to your platform’s equivalents.
What’s out of scope for this book
A short list of Kubernetes flavours this book does not cover:
- EKS / AKS / GKE — BNK runs on these, but
roksbnkctl upwon’t provision them. You’d use cloud-specific tooling, then deploy BNK on top with the standard Helm charts F5 publishes. - Self-managed OpenShift on bare metal or VMs — same: no
roksbnkctl up. You’d useopenshift-install, then deploy BNK. - K3s, RKE2, microk8s — BNK’s not formally supported on these for production; useful for local dev work but outside this book’s scope.
The patterns from later chapters — workspaces, the --on flag, the connectivity / DNS / throughput tests — would still be useful on any of these, but the lifecycle commands (init, up, down, cluster register) assume ROKS.
What you need before continuing
To follow this book end-to-end you need:
- An IBM Cloud account with billing enabled. The free tier won’t provision a worker pool; you’ll need a Pay-As-You-Go or Subscription account.
- An IBM Cloud API key with permission to create ROKS clusters in the target account.
- A resource group to scope cluster resources to. The default
Defaultgroup works fine for a single-user evaluation; production deployments tend to use a dedicated group per environment.
The next chapters walk through installation and the quick-start path. By the end of Chapter 7 you’ll have a deployed BNK trial on a fresh ROKS cluster.
What roksbnkctl does (and doesn’t do)
roksbnkctl is a single-binary CLI for deploying and validating F5 BIG-IP Next for Kubernetes (BNK) onto IBM Cloud ROKS. It exists to compress a multi-step deployment — clone the right Terraform, configure it, run terraform, fetch a kubeconfig, install BNK, run smoke tests — into a four-command lifecycle.
This chapter is about scope. What roksbnkctl owns, what it deliberately does not own, and what’s coming in future releases. Read it before you reach for the tool to do something it isn’t trying to do.
The 4-command lifecycle
The everyday user-facing flow is four commands:
roksbnkctl init # answer a few prompts about region, RG, cluster name
roksbnkctl up # terraform plan + apply (~50 min for fresh ROKS + BNK)
roksbnkctl test # connectivity + DNS + throughput against the deployment
roksbnkctl down # tear it all back down when you're done evaluating
That’s it. From “I have an IBM Cloud API key” to “deployed BNK with a passing throughput test” with no manual terraform apply, no hand-editing kubeconfig paths, no chasing down BNK Helm charts — then a clean tear-down when you’re done so you stop paying for the cluster.
Chapter 7 walks through this end-to-end with sample output.
What roksbnkctl owns
roksbnkctl’s scope is everything between “you have an IBM Cloud API key” and “you have a working BNK install you can run tests against”. Concretely:
- Workspace state — kubectl-style per-environment isolation under
~/.roksbnkctl/<workspace>/. Each workspace has its own config, terraform state, kubeconfig, scratch artefacts. Switch withroksbnkctl ws use <name>or override per-command with-w <name>. - Terraform-exec orchestration — wraps HashiCorp’s
terraform-execlibrary to driveterraform init/plan/apply/destroywith the right state file, the rightTF_DATA_DIR, the right tfvars layering. You don’t runterraformdirectly;roksbnkctl updoes. - Kubeconfig fetch — after a successful
up, fetches the admin kubeconfig from IBM Cloud’s container service API and writes it to~/.kube/configat mode 0600. Retries on the 404s that happen during cluster propagation lag. - COS supply chain — the BNK install needs FAR images and JWT licenses staged in IBM Cloud Object Storage.
roksbnkctl cos instance/bucket/objecthandles instance creation, bucket lifecycle, and streaming object I/O (multipart for large files) without making youpip installthe IBM COS SDK separately. - Post-deploy validation —
roksbnkctl testruns three suites: HTTP/HTTPS connectivity (built-innet/http, no externalcurl), DNS resolution (built-innet.Resolver, no externaldig), and iperf3 throughput (deploys aniperf3 -spod into the cluster, runs the client, parses JSON output, tears down). - Credentials handling — IBM Cloud API key resolution chain: env vars (
IBMCLOUD_API_KEYetc.), OS keychain (macOS Keychain / libsecret / Windows Credential Manager viazalando/go-keyring), opt-in base64 in workspace config, interactive prompt as last resort. Plaintext keys inconfig.yamlare rejected.
If any of those words don’t make sense yet, don’t worry — later chapters cover each in depth.
What roksbnkctl does not try to do
Equally important: the explicit non-goals. roksbnkctl deliberately stays out of these spaces because well-established tools already cover them:
- Not a generic IBM Cloud CLI. That’s
ibmcloud. If you want to manage VPCs, IAM policies, classic infrastructure, Watson, or any of the hundred-plus other IBM Cloud services, useibmcloud.roksbnkctl ibmcloud <args...>exists as a convenience passthrough that loads workspace credentials, but it doesn’t try to replaceibmcloud’s surface. - Not a generic Kubernetes CLI. That’s
kubectl.roksbnkctl kubectl <args...>is again a passthrough that loads the workspace’s kubeconfig; it does not try to be a kubectl re-implementation. (Phase 2 internalises a small subset —roksbnkctl k get/apply/logs/exec/port-forward— so the happy path doesn’t require a hostkubectlbinary, but that’s targeted convenience, not replacement.) - Not an OpenShift admin tool. That’s
oc. Same story:roksbnkctl oc <args...>passthrough, no attempt to re-implement. - Not a BNK runtime UI. Once BNK is deployed, you configure it through its CRDs (
F5BigIpCtx,F5IngressTls, etc.).roksbnkctldoesn’t ship a TUI / web UI for editing those — it gets you to a deployed BNK and steps out of the way. - Not a Terraform authoring tool. The HCL lives in this repo’s
terraform/directory and is embedded into the binary at build time.roksbnkctlruns that HCL; it doesn’t help you write more of it. If you fork the HCL, pointroksbnkctlat your fork viatf_source: githubortf_source: local. - Not an arbitrary workload deployer. BNK is the workload. The iperf3 / nginx fixtures used by
roksbnkctl testexist only to validate BNK; they’re not a general-purpose deployment surface.
The principle is “do one thing well”. roksbnkctl does BNK-on-ROKS lifecycle and validation. Every other concern is delegated to the right purpose-built tool.
The relationship to bundled HCL
A core design decision worth surfacing: the Terraform that drives the deployment lives in this repo under terraform/, and is embedded into the roksbnkctl binary at build time via Go’s embed package.
This means:
-
One install gets you the CLI + a matched HCL pair. No “clone the right tag of the terraform repo separately” step.
-
Versioning is unified. A
roksbnkctl v1.0release ships with a specific snapshot of the HCL. Upgrading the binary upgrades the HCL atomically. There’s no skew between “binary version” and “Terraform version”. -
Power users can override. The workspace config has a
tf_source:block:tf_source: type: embedded # default; uses HCL bundled into the binary # type: local # path: /path/to/your/terraform # type: github # repo: yourfork/roksbnkctl-terraform # ref: my-branchtf_source: localis the right setting if you’re iterating on the HCL itself.tf_source: githublets you point at a fork of the terraform repo if you’ve published one separately. The default —embedded— covers the everyday case.
Chapter 13 covers the tfvars layering rules; this is just the elevator pitch for “the HCL ships with the binary”.
What v1.0 ships and what’s queued for v1.x
This book ships with v1.0. The surface it documents:
- kubectl internalisation —
roksbnkctl k get/apply/logs/exec/port-forwardis a first-class verb talking to the cluster directly viaclient-go. Hostkubectlis informational only; the only required prereq on PATH isterraform. - Four execution backends — every external tool (
ibmcloud,iperf3,terraform) selectable acrosslocal | docker | k8s | sshvia--backend. iperf3 runs entirely in-cluster by default; ibmcloud runs in a pinned-version Docker container if you don’t want to install it; any tool proxies through a jumphost via--backend ssh:<target>. Chapter 17 is the user-facing surface; PRD 03 is the design rationale. - GSLB-aware DNS testing — the DNS probe is miekg/dns-based with multi-vantage support, so you can verify that BNK’s GSLB is returning different answers from different network locations. Chapter 21 covers it.
- Polished book — all 32 chapters, every code example verified, four Mermaid diagrams (architecture / lifecycle / GSLB cross-vantage / backend matrix), per-Part worked examples.
A handful of items are explicitly deferred to v1.x:
terraformover--backend k8sand--backend ssh(state-file portability design needed).- Multi-hop SSH
ProxyJumpfor the--onandssh:<target>paths. - Windows full TTY (interactive
shellon Windows ships as line-buffered; full PTY is a v1.x item). - Typed OpenShift CRDs (today’s unstructured printer works; richer per-type output is queued).
- Cross-driver cluster-sharing for
e2e-test-full.sh(each driver brings up its own cluster today).
See docs/PLAN.md §“What’s deliberately deferred to post-v1.0” for the full roadmap.
Pointers to the next chapters
- Chapter 4 — Installation gets the binary on your machine.
- Chapter 7 — Quick start walks the 4-command lifecycle with sample output.
- Chapter 16 — The –on flag and SSH jumphosts covers running passthrough commands over SSH against an auto-discovered jumphost — useful in customer-firewalled and air-gapped scenarios.
Installation
This chapter gets roksbnkctl onto your machine and verifies it works. Three install paths are covered: build-from-source (native Go), build-with-Docker (no host Go required), and the all-in-one container image (no install at all — run roksbnkctl and every tool it needs from one container).
Pre-built binaries are attached to every GitHub Release (Linux, macOS, Windows × amd64, arm64). The book also ships as an offline PDF (roksbnkctl-book-<tag>.pdf) on the same release page. A Homebrew tap is on the v1.x roadmap; until then macOS users grab the binary from the release page or build from source.
Quick install (one line)
The fastest path — downloads the latest release for your OS/arch and puts it on PATH:
Linux / macOS:
curl -fsSL https://raw.githubusercontent.com/jgruberf5/roksbnkctl/main/install.sh | sh
Windows (PowerShell):
irm https://raw.githubusercontent.com/jgruberf5/roksbnkctl/main/install.ps1 | iex
Each script downloads the release archive, verifies its checksum, extracts the binary, runs roksbnkctl install to copy it into ~/.local/bin (Linux/macOS) or a PATH directory (Windows) — replacing any existing copy — then deletes the archive, leaving only the installed binary. Options:
- Pin a version: set
VERSION=v1.23.1before the command (e.g.curl -fsSL … | VERSION=v1.23.1 sh, or$env:VERSION='v1.23.1'on Windows). - Change the install directory / pass install flags: set
ROKSBNKCTL_INSTALL_ARGS="--dir $HOME/bin".
Manage the installed binary with the built-in verbs:
roksbnkctl version— prints the release (e.g.v1.23.1) for a release binary, or a git-derived build string for a local build.roksbnkctl upgrade— on a terminal, lists the newer releases on GitHub and lets you pick one;--version vX.Y.Zpins a specific release;--yestakes the latest non-interactively.roksbnkctl uninstall— removes the installed binary (the opposite ofinstall).
The manual methods below (build-from-source, Docker build, runner container) remain available.
Choosing an install method
| Method | You get | Host needs | Best for |
|---|---|---|---|
| A — native build / release binary | a roksbnkctl binary | Go 1.26+ (build) or nothing (release binary); the runtime tools you use (above) | a developer laptop |
| B — Docker build | a roksbnkctl binary | Docker only (for the build) | a host without Go; CI that wants a binary artefact |
| C — runner container | a container with roksbnkctl + every tool | Docker/Podman + a volume for state | CI runners, fleet provisioning, air-gapped sites |
Methods A and B put a binary on PATH and assume the runtime tools are installed too. Method C (Path C below) is self-contained: nothing on the host but a container runtime and a place to keep state.
Prerequisites
- Linux, macOS, or native Windows. The full deploy path — cluster, BNK, and testing phases — runs on native Windows without WSL: roksbnkctl drives Terraform’s provisioning glue itself (no
curl/tar/grep/bashon the host), and the wholeup→ shared-Transit-Gateway →downflow is validated on Windows PowerShell. A few interactive conveniences (the TTY-bound SSH shell, ssh-agent integration) are still not first-class on Windows; use--on/backends or a Linux/macOS host for those. - Git to clone the repository (only if building from source — not needed if you grab a pre-built binary).
- Go 1.26 or newer if you want a native build. If you don’t have Go (or have an older version), use the Docker-based build or a pre-built release binary.
- Terraform >= 1.5 on PATH at runtime — required for
roksbnkctl up/plan/apply/down. - Helm 3 on PATH at runtime —
doctorflags it as required duringroksbnkctl up. The charts install via thehelm_releaseresource (in-process via thehashicorp/helmprovider — see Chapter 10 §“The terraform-native deployment model”), but FAR chart-version discovery still shells out to hosthelm: thedata.external.versionslookup runshelm registry login+helm pullto read the FLO/CIS chart versions. Sohelmis genuinely required. Keephelminstalled to staydoctor-green.
The remaining tools (ibmcloud, kubectl, oc, iperf3, docker) are optional and only needed for the corresponding passthrough or backend.
You do not need Docker installed to use roksbnkctl with the default local backend. Docker is required only if you opt in to --backend docker for terraform / ibmcloud. The k8s and ssh backends are alternatives that need neither host Docker nor host Go.
Installing prerequisites
Install paths per platform. terraform and helm are flagged required by doctor (helm is shelled out for FAR chart-version discovery during roksbnkctl up — keep it installed to stay green); the rest are optional, install only what you need.
macOS — Homebrew
brew install terraform # required
brew install helm # flagged required by doctor — FAR chart-version discovery shells out to `helm`
brew install --cask ibmcloud-cli # optional — only for `roksbnkctl ibmcloud …` passthrough
brew install kubectl # optional — only for `roksbnkctl kubectl …` passthrough (`roksbnkctl k *` is internalised)
brew install iperf3 # optional — only for `--backend local`/`--backend ssh:<t>` throughput tests
# oc (Red Hat OpenShift CLI) — optional, only for `roksbnkctl oc …` passthrough.
# No brew formula; install via the Red Hat mirror tarball:
curl -sSL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-mac.tar.gz \
| sudo tar -xz -C /usr/local/bin oc
If you installed ibmcloud-cli, add the plugins roksbnkctl uses:
ibmcloud plugin install kubernetes-service -f
ibmcloud plugin install cloud-object-storage -f
Linux — Ubuntu / Debian
# terraform — required
wget -qO- https://apt.releases.hashicorp.com/gpg \
| sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
| sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y terraform
# helm 3 — flagged required by doctor (FAR chart-version discovery shells out to `helm` to read the FLO/CIS chart versions)
curl https://baltocdn.com/helm/signing.asc \
| sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] \
https://baltocdn.com/helm/stable/debian/ all main" \
| sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update && sudo apt-get install -y helm
# ibmcloud CLI + plugins — optional, for `roksbnkctl ibmcloud …` passthrough with --backend local
curl -fsSL https://clis.cloud.ibm.com/install/linux | sudo sh
ibmcloud plugin install kubernetes-service -f
ibmcloud plugin install cloud-object-storage -f
# kubectl — optional, only for `roksbnkctl kubectl <args>` passthrough (`roksbnkctl k *` is internalised and needs no host install)
sudo snap install kubectl --classic
# or via direct download:
# curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
# chmod +x kubectl && sudo mv kubectl /usr/local/bin/
# oc (Red Hat OpenShift CLI) — optional, only for `roksbnkctl oc <args>` passthrough.
# No apt package; install via the Red Hat mirror tarball:
curl -sSL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz \
| sudo tar -xz -C /usr/local/bin oc
# iperf3 — optional, only for `--backend local` / `--backend ssh:<t>` throughput tests
sudo apt-get install -y iperf3
Instructions above target Ubuntu and Debian. For other Linux distributions (RHEL, Fedora, Arch, openSUSE, Alpine, …), a quick online search for “install terraform on <your distro>” — and the same pattern for ibmcloud, kubectl, and iperf3 — yields the equivalent commands. HashiCorp ships an RPM repo at https://rpm.releases.hashicorp.com covering RHEL/Fedora, and most distributions package kubectl and iperf3 in their official repos; the IBM Cloud CLI installer at https://clis.cloud.ibm.com/install/linux is a single curl-pipe-sh that works across distros.
Windows — Chocolatey
choco install terraform
choco install kubernetes-helm # required — terraform local-exec provisioner shells out to `helm`
choco install ibmcloud-cli # optional
choco install kubernetes-cli # optional, provides kubectl
choco install openshift-cli # optional, provides oc (Red Hat OpenShift CLI)
choco install iperf3 # optional
Or via Scoop:
scoop install terraform helm ibmcloud-cli kubernetes-cli openshift-cli iperf3
If choco/scoop don’t carry openshift-cli for your version, grab the Windows tarball from the Red Hat mirror directly:
Invoke-WebRequest -Uri https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-windows.zip -OutFile oc.zip
Expand-Archive oc.zip -DestinationPath "$env:USERPROFILE\bin\"
# then add %USERPROFILE%\bin to your PATH
After installing ibmcloud-cli, add the plugins:
ibmcloud plugin install kubernetes-service -f
ibmcloud plugin install cloud-object-storage -f
Windows TTY-bound SSH features (the roksbnkctl shell --on <target> interactive path) have known limitations on Windows; file-based SSH keys + non-interactive commands work, but ssh-agent named-pipe integration is a v1.x item. See docs/PLAN.md §“What’s deliberately deferred to post-v1.0”.
Path A — native build (requires Go 1.26+)
If go version reports 1.26 or newer, this is the simplest path:
git clone https://github.com/jgruberf5/roksbnkctl.git
cd roksbnkctl
go mod tidy # first time only — populates go.sum
make build # → bin/roksbnkctl
# Install via roksbnkctl itself (recommended — copies into ~/.local/bin):
./bin/roksbnkctl install
That’s the whole thing. The install subcommand is idempotent and copies the running binary into a directory on your PATH. Default destination is ~/.local/bin/roksbnkctl.
Make targets you’ll use most often:
make build # go build -ldflags ... -o bin/roksbnkctl ./cmd/roksbnkctl
make test # go test ./...
make vet # go vet ./...
make tidy # go mod tidy
make clean # rm -rf bin/
If make build fails, the most likely cause is Go too old. The module declares go 1.26.0 in go.mod (forced by transitive deps from the SSH/integration test layers); older versions error out with go: module requires Go 1.26. Either upgrade Go or fall back to the Docker path below.
Path B — Docker-based build (no host Go required)
This path is ideal for sealed CI workstations, custom VM images, or anywhere installing Go on the host is awkward. The official golang:1.26-alpine image has everything needed (the minimum tracks whatever the transitive dependencies demand — currently the k8s.io/* v0.36 modules, which declare go 1.26); the build artefact lands in ./bin/ owned by your host user.
git clone https://github.com/jgruberf5/roksbnkctl.git
cd roksbnkctl
docker run --rm -v "$PWD:/work" -w /work \
--user "$(id -u):$(id -g)" -e HOME=/tmp \
golang:1.26-alpine sh -c 'go mod tidy && go build -o bin/roksbnkctl ./cmd/roksbnkctl'
./bin/roksbnkctl install
Anatomy of the docker invocation:
| Flag | Why |
|---|---|
-v "$PWD:/work" | Bind-mount the repo into the container at /work. |
-w /work | Container working directory matches the mount. |
--user "$(id -u):$(id -g)" | Output binary is owned by your host user, not root. |
-e HOME=/tmp | Go writes its module cache under $HOME; /tmp is writable by any user. Without this, go mod tidy fails on a writable-/root permission error. |
golang:1.26-alpine | Pinned major version; matches go.mod’s minimum. |
Cross-compile via Docker
Set GOOS / GOARCH env vars in the same docker run to produce binaries for other platforms:
# macOS arm64 (Apple Silicon)
docker run --rm -v "$PWD:/work" -w /work \
--user "$(id -u):$(id -g)" -e HOME=/tmp \
-e GOOS=darwin -e GOARCH=arm64 \
golang:1.26-alpine sh -c 'go mod tidy && go build -o bin/roksbnkctl-darwin-arm64 ./cmd/roksbnkctl'
# Windows amd64 (compile-only; not tested at runtime)
docker run --rm -v "$PWD:/work" -w /work \
--user "$(id -u):$(id -g)" -e HOME=/tmp \
-e GOOS=windows -e GOARCH=amd64 \
golang:1.26-alpine sh -c 'go mod tidy && go build -o bin/roksbnkctl.exe ./cmd/roksbnkctl'
Each binary is statically linked (Alpine + CGO_ENABLED=0 is the cross-compile default) so the produced file has no runtime library dependencies.
Path C — run from the all-in-one container image (no install)
If you’d rather not put a binary and its tools on the host at all — CI runners, fleet provisioning, air-gapped sites — use the runner image. It carries roksbnkctl plus every tool it dispatches (terraform, helm, kubectl, oc, ibmcloud, iperf3, h2load), so one docker run is a complete roksbnkctl with nothing on the host:
docker run --rm \
-v bnk-state:/work \
-e IBMCLOUD_API_KEY="$IBMCLOUD_API_KEY" \
ghcr.io/jgruberf5/roksbnkctl-tools-runner:latest version
podman run works identically. The image’s ENTRYPOINT is roksbnkctl, so everything after the image name is roksbnkctl’s own arguments (version, up, test matrix --dry-run, …). Tags: :latest and a per-release :vX.Y.Z are published to GHCR on each release; :dev tracks main.
State lives in a volume, not the image
roksbnkctl keeps its workspaces — config.yaml, terraform state, kubeconfigs, keys — under ROKSBNKCTL_HOME, which the image sets to /work/.roksbnkctl. The image itself is stateless; mount durable storage at /work or every run starts empty:
- A named volume (recommended) —
-v bnk-state:/work. Docker seeds a fresh named volume from the image’s/work, which is owned by the image’suid 1000, so it’s writable out of the box. - A host bind-mount —
-v "$PWD/bnk-state:/work"— carries the host directory’s ownership, which may not be writable byuid 1000. Pre-create it writable (mkdir -p bnk-state && chmod 0777 bnk-state, orchownit to1000), or add--user "$(id -u):$(id -g)". This is the usual docker bind-mount caveat, not a roksbnkctl quirk.
For a truly stateless runner — no volume at all — a COS/S3 remote state backend is the planned answer (PRD 16); until it lands, mount a volume.
Use --backend local inside the runner
Every tool roksbnkctl dispatches is already on PATH in the image, so the docker/k8s execution backends (whose whole job is to fetch a tool from elsewhere) aren’t needed here — runs are local, with no docker-in-docker. Chapter 18 — Choosing a backend covers when the other backends matter (the binary-on-host case).
What’s not in it
The runner is a use image, not a docs/dev-build image — the book toolchain (mdbook, pandoc, texlive) is deliberately excluded to keep it lean. Build the book with the separate roksbnkctl-tools-mdbook image (see Chapter 31 — Building from source).
Because it’s one self-contained artefact, the runner is also the natural thing to mirror behind a firewall for an air-gapped install — pair it with the registry mirror (Chapter 10a — Air-gapped install).
The install subcommand
roksbnkctl install [--dir PATH] [--force]
install copies the running binary into a directory on PATH. Defaults:
- Destination:
~/.local/bin/roksbnkctl— this directory is on the defaultPATHfor most modern Linux and macOS user environments and does not require sudo. - Mode:
0755. - Idempotent: if the running binary is already at the destination, no-op (no error).
Override the destination with --dir:
./bin/roksbnkctl install --dir ~/bin
sudo ./bin/roksbnkctl install --dir /usr/local/bin
--force overwrites an existing file at the destination. Without it, install refuses if the destination is a different binary.
If ~/.local/bin is not on your PATH, add it. On bash:
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
exec $SHELL -l
On zsh, swap ~/.bashrc for ~/.zshrc.
Verifying the install
Two quick checks: version (proves the binary runs) and doctor (proves the runtime environment is set up for actual work).
roksbnkctl version
roksbnkctl version
Sample output:
roksbnkctl v1.0.0 (commit abc1234, built 2026-05-10T14:22:08Z)
Docs: https://jgruberf5.github.io/roksbnkctl/book/
The version string is populated via -ldflags at build time; make build VERSION=v1.0.0 injects an explicit tag. A bare make build produces something like dev (commit abc1234, built ...). The Docs: URL is a compile-time constant (internal/cli/meta.go::DocsURL) — every binary built from this tree points at the same book URL.
roksbnkctl doctor
roksbnkctl doctor
doctor runs the prereq + credentials report. Sample output on a healthy machine looks like this (yours will differ depending on which optional binaries you have installed and whether you’ve initialised a workspace):
✓ terraform /usr/bin/terraform (Terraform v1.15.2) (required for `roksbnkctl up`)
✓ helm /usr/local/bin/helm (v3.20.2) (required for `roksbnkctl up`; terraform `local-exec` provisioners shell out to helm)
⚠ iperf3 not on PATH (needed for `roksbnkctl test throughput`)
✓ kubectl /usr/local/bin/kubectl (clientVersion:) (optional; `roksbnkctl kubectl` passthrough)
✓ oc /usr/local/bin/oc (Client Version: 4.21.10) (optional; `roksbnkctl oc` passthrough)
✓ ibmcloud /usr/local/bin/ibmcloud (ibmcloud 2.43.0 ...) (optional; `roksbnkctl ibmcloud` passthrough)
✓ kubeconfig /home/jgruber/.kube/config (needed for cluster-side ops)
✓ workspace default (per-environment config + state)
✓ ibmcloud api key resolved via OS keychain (auth for terraform + IBM SDK calls)
✓ ibm cloud auth OK (account: Main F5 Account) (verifies API key works against IBM IAM)
Each row is <status> <name> <detail> <why we care>. Failures are red ✗ and exit non-zero; warnings are yellow ⚠ and don’t fail the run. terraform and helm are the hard-required checks at v1.0 — the rest are either optional passthroughs or specific to test suites. Chapter 5 walks through what each check is verifying and how to fix common failures.
OS support matrix
| OS | Native build | Docker build | Cross-compile target | Runtime status |
|---|---|---|---|---|
| Linux (amd64, arm64) | yes | yes | yes | first-class |
| macOS (amd64, arm64) | yes | yes | yes | first-class |
| Windows (amd64, arm64) | yes | yes | yes | compile-only; roksbnkctl shell --on and roksbnkctl exec --on jumphost PTY behaviour limited |
“First-class” means the v1.0 acceptance criteria are validated on those platforms; “compile-only” means the binary builds and runs but interactive features (notably TTY-bound SSH) have known limitations and are not part of the v1.0 release gate.
The Windows limitations are tracked in PRD 01 (the SSH client design) and largely come down to golang.org/x/crypto/ssh’s incomplete PTY handling on Windows and the absence of an SSH agent named-pipe protocol. File-based SSH keys work; full PTY and ssh-agent integration on Windows are on the v1.x roadmap (see docs/PLAN.md §“What’s deliberately deferred to post-v1.0”).
Required prerequisites — terraform and helm at v1.0
The cluster lifecycle needs two binaries on PATH:
terraform(>= 1.5) — hard-required for any cluster lifecycle command (up,down,plan,apply).helm(3.x) — flagged required bydoctorduringroksbnkctl up. The charts install via thehelm_releaseresource — thehashicorp/helmprovider speaks the Helm 3 protocol via an embedded Go runtime, so no hosthelmis shelled out for the chart installs (see Chapter 10). FAR chart-version discovery does shell out to hosthelm, though: thedata.external.versionslookup runshelm registry login+helm pullto read the FLO/CIS chart versions, sohelmis genuinely required. Keepinghelminstalled keepsdoctorgreen.
Optional binaries — only needed for the corresponding passthrough or fallback path:
iperf3— only needed for--backend localand--backend ssh:<target>throughput modes. The default--backend k8sruns iperf3 entirely in cluster (no host binary needed).kubectl/oc— only needed for theroksbnkctl kubectl <args...>/roksbnkctl oc <args...>passthroughs. The everyday verbs (get,apply,describe,delete,logs,exec,port-forward) are internalised underroksbnkctl kand need no host binary.ibmcloud— only needed for theroksbnkctl ibmcloud <args...>passthrough on--backend local. The cluster-lifecycle path uses IBM Go SDKs internally and does not shell out toibmcloud. Thedocker,k8s, andsshbackends ship their own ibmcloud — no host install needed.docker— only needed for--backend docker. Optional; thek8sandsshbackends are alternatives if docker isn’t available.
Run roksbnkctl doctor to see exactly what your environment is missing for the workflow you intend to run.
Updating
git pull && make build is the source-build update mechanism (or re-run the Docker build for the containerised path).
roksbnkctl self update upgrades from a tagged GitHub release. Use it once you’ve installed an initial v1.0 binary:
roksbnkctl self update
# Checks https://github.com/jgruberf5/roksbnkctl/releases/latest, downloads
# the matching asset for your OS+arch, verifies the checksum, swaps the
# binary atomically.
Next
With a working binary on PATH, Chapter 5 — Doctor explains what every doctor check is looking at, Chapter 6 — Workspaces explains the ~/.roksbnkctl/<workspace>/ layout, and Chapter 7 — Quick start walks the 4-command lifecycle end-to-end.
Doctor: checking your environment
roksbnkctl doctor is the prereq + credentials report. It runs in under five seconds, exits non-zero on any hard error, and prints a tabular report that maps one-to-one to the runtime dependencies the rest of the tool reaches for.
This chapter walks every check, explains what each row’s “why we care” blurb means, covers why kubectl and oc are informational rather than required, and describes the --target SSH probe.
What doctor checks
A bare roksbnkctl doctor runs the general checks: tooling on PATH, kubeconfig location, the resolved workspace, and the IBM Cloud authentication chain. Sample output on a healthy machine looks like this:
roksbnkctl doctor
✓ terraform /usr/bin/terraform (Terraform v1.15.2) (required for `roksbnkctl up`)
✓ helm /usr/local/bin/helm (v3.20.2) (required for `roksbnkctl up`; terraform `local-exec` shells out to helm)
⚠ iperf3 not on PATH (needed for `roksbnkctl test throughput`)
✓ kubectl /usr/local/bin/kubectl (clientVersion:) (internalised in roksbnkctl k *; passthrough still works if installed)
✓ oc /usr/local/bin/oc (Client Version: 4.21.10) (internalised in roksbnkctl k *; passthrough still works if installed)
✓ ibmcloud /usr/local/bin/ibmcloud (ibmcloud 2.43.0 ...) (optional; `roksbnkctl ibmcloud` passthrough)
✓ kubeconfig /home/jgruber/.kube/config (needed for cluster-side ops)
✓ workspace default (per-environment config + state)
✓ workspace permissions owner-only (credentials in the workspace are protected by file mode alone)
✓ ibmcloud api key resolved (auth for terraform + IBM SDK calls)
✓ ibm cloud auth OK (account: 1a2b3c..., user: you@example.com) (verifies API key works against IBM IAM)
Each row has the same shape:
<status> <name> <detail> <why we care>
- status is one of
✓(green / OK),⚠(yellow / warning), or✗(red / error).Skippedchecks render as⚠. - name is the dependency or capability being checked.
- detail is the resolved value — usually a path, a version line, or an error message.
- why we care is a parenthetical clause naming the
roksbnkctlfeature that depends on this row.
The BackendName column on the underlying Check struct (internal/doctor/check.go) is reserved for the per-backend probes that land in Sprint 4 (PRD 03). Until then it stays empty for every general check.
Each check explained
terraform — required
One of two hard-required binaries for the roksbnkctl up happy path. roksbnkctl shells out to terraform via terraform-exec for plan/apply/destroy; without it nothing in the cluster lifecycle works.
Pass condition: a binary on PATH, version 1.5 or newer.
Failure mode: not on PATH. Fix: install Terraform from terraform.io, or your distro’s package manager, then re-run doctor.
helm — required
The second hard-required binary (doctor flags it required). It is needed because FAR chart-version discovery — the data.external.versions lookup — runs helm registry login + helm pull to read the FLO/CIS chart versions during roksbnkctl up. Without helm on PATH, the apply fails partway through the cluster lifecycle with:
Error: local-exec provisioner error
Error running command 'helm upgrade --install cert-manager ...':
exit status 127. Output: /bin/sh: 1: helm: not found
Pass condition: a helm (v3.x) binary on PATH. Doctor parses helm version --short for the version detail.
Failure mode: not on PATH. Fix: install Helm 3 from helm.sh/docs/intro/install/, or via your distro’s package manager:
# Linux (Ubuntu/Debian — official Helm apt repo):
curl https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update && sudo apt-get install -y helm
# macOS:
brew install helm
# Windows:
choco install kubernetes-helm
The BNK charts (cert_manager / flo / cne_instance) install via the helm_release terraform resource, which uses the hashicorp/helm provider’s embedded Helm 3 runtime — no host helm is shelled out for the chart installs themselves. Doctor still flags helm required because FAR chart-version discovery shells out to host helm (helm registry login + helm pull) to read the FLO/CIS versions. See Chapter 10 §“The terraform-native deployment model”.
iperf3 — informational
Used only by roksbnkctl test throughput in its host-iperf3 modes. After Sprint 4 lands the k8s execution backend (PRD 03), iperf3 moves entirely in-cluster and this row goes away for the everyday user.
Failure mode: not on PATH. Fix: install iperf3 if you plan to use the throughput test today; otherwise ignore.
kubectl — informational
kubectl is informational. The everyday verbs (get, apply, describe, delete, logs, exec, port-forward) are native Go via client-go and live under roksbnkctl k. Missing host kubectl does not disable the happy path; it only disables the roksbnkctl kubectl <args...> passthrough.
If kubectl is on PATH, the row is still ✓ and shows the version line. If it’s missing, the row is informational, not a warning, and the detail explains where the equivalent functionality lives.
oc — informational
Same story as kubectl — the OpenShift-relevant verbs are internalised (Project, Route, and ImageStream are available via roksbnkctl k get). Host oc is preserved as an escape hatch; a missing oc is informational, not a warning.
ibmcloud — optional
Required only for the roksbnkctl ibmcloud <args...> passthrough. The cluster-lifecycle path uses IBM Go SDKs internally — roksbnkctl up does not shell out to ibmcloud — so you can skip this binary if you don’t need the passthrough.
kubeconfig
Resolves the kubeconfig path via $KUBECONFIG first, then ~/.kube/config. Cluster-side commands (status, logs, every k <verb>) need it.
roksbnkctl up writes the admin kubeconfig at ~/.kube/config (mode 0600) on a fresh apply.
If you already have a multi-cluster ~/.kube/config, do not point $KUBECONFIG at
~/.roksbnkctl/<workspace>/state/kubeconfig. That is a directory of per-phase
kubeconfigs Terraform writes as a side effect of config_dir, each carrying a
roughly one-hour IAM token that nothing refreshes. An hour after an apply they
authenticate as nobody, and the only symptom is:
error: You must be logged in to the server (the server has asked for the client to provide credentials)
Use the workspace-scoped verbs instead — they select the kubeconfig that addresses this workspace’s cluster, and skip any whose credential has expired:
roksbnkctl -w <workspace> k get pods -A
roksbnkctl -w <workspace> kubectl get nodes
To refresh the admin kubeconfig at ~/.kube/config, run roksbnkctl kubeconfig --download.
roksbnkctl doctor reports expired phase kubeconfigs under workspace phase kubeconfigs.
Failure mode: $KUBECONFIG and ~/.kube/config both missing. Fix: run roksbnkctl kubeconfig --download to fetch the admin kubeconfig from IBM Cloud.
workspace
Reports the resolved workspace name and whether its config.yaml exists.
✓ default— the current workspace pointer at~/.roksbnkctl/config.yamlresolves and the named workspace has a populatedconfig.yaml.⚠ "default" not initialised— the directory may exist (created byroksbnkctl ws new) butconfig.yamlis empty. Runroksbnkctl initto populate.✗ no config context— the global config can’t be loaded at all.
The one-off -w / --workspace flag overrides which workspace doctor reports against. See Chapter 6 — Workspaces.
workspace permissions
Reports whether the workspace tree is owner-only, and tightens it if not.
config.yaml can hold ibmcloud.api_key_b64 and registry.generic_password_b64; state/terraform.tfstate holds whatever those resolve to. All of it is base64 at most, which is obfuscation and not encryption — the file mode is the only protection, so config.yaml is 0600 and the directories are 0700.
The check is read-only. Loading a workspace already tightens what it can, so anything still loose by the time doctor looks is something the repair could not fix — which is exactly what is worth reporting.
✓ owner-only— nothing to do.✗ <path> readable by other users and could not be tightened (N path(s))— the only case that needs your hands. A read-only mount, or a filesystem with no POSIX modes (a WSL DrvFs mount without metadata, an SMB share). Move the workspace onto a filesystem that can hold0600, or use the keychain or env-var credential paths so nothing sensitive lands in the file at all.
The row is absent on Windows: Go’s Chmod there only toggles the read-only bit, so there is no owner-only mode to assert and a permanent unactionable warning would be worse than no row.
The scope is the workspace directory and its immediate children: config.yaml, registry-mirror.json, cluster-outputs.json, ssh/, and every state-* tree. Children are enumerated rather than matched against a list of known names, so a state tree added by a future phase is covered the day it appears. Owner-only on a state directory makes everything inside it unreachable regardless of those files’ own modes, which is why the check does not walk deeper.
The repair happens on load, not here, and announces itself once:
⚠ workspace "prod" was readable by other users on this host; tightened 3 path(s) to owner-only.
config.yaml can hold your IBM Cloud API key and registry password. If this host is shared, rotate them.
Workspaces written by a release before v1.50.0 are 0644/0755 on disk and will produce that line the first time they are read. The exposure that already happened is not undone — on a shared host, rotate the API key.
ibmcloud api key
Resolves the API key via the chain documented in Chapter 14 — Credentials: env var → OS keychain → workspace config (base64) → TTY prompt.
Pass condition: the chain produces a non-empty key. The key value is never printed — only the source (“resolved”).
Failure mode: IBMCLOUD_API_KEY unset and no keychain entry for workspace "<name>". Fix: either export IBMCLOUD_API_KEY=... for the session, or re-run roksbnkctl init and accept the keychain-save prompt.
ibm cloud auth
Round-trips the resolved key against IBM IAM via the SDK (Verify() call). Confirms the key is not just present but actually authenticates.
Pass condition: IAM accepts the key; the row reports the resolved account and user identity.
Failure modes:
BXNIM0415E: Provided API key could not be found— the key is malformed or has been deleted in IBM Cloud.network is unreachable/i/o timeout— your workstation can’t reachiam.cloud.ibm.com. Common in customer-firewall scenarios; route through a jumphost (Chapter 16) to confirm the key works from inside the customer network.
Common failures and how to fix them
The chapter readers most often land on. Each row maps a real-world symptom to its fix:
| Symptom | Likely cause | Fix |
|---|---|---|
terraform not on PATH | not installed | install Terraform >= 1.5; re-run doctor |
kubeconfig: $KUBECONFIG and ~/.kube/config both missing | never ran up against this workspace | roksbnkctl kubeconfig --download or run roksbnkctl up |
ibmcloud api key: ... no keychain entry | new shell, key not exported | export IBMCLOUD_API_KEY=... or re-run roksbnkctl init |
ibm cloud auth: BXNIM0415E | bad / rotated key | regenerate the key in the IBM Cloud console; update the keychain via roksbnkctl init |
ibm cloud auth: i/o timeout | corp-firewalled workstation | use --on jumphost to test from inside the customer network |
workspace "foo" not initialised | ws new was run but init was not | run roksbnkctl init -w foo |
workspace: no config context | ~/.roksbnkctl/config.yaml corrupt | inspect the file; worst case delete it and re-run init |
If a fix isn’t here, Chapter 26 — Troubleshooting covers the longer tail.
The --target <name> SSH check
The --on jumphost flag adds an optional second mode for doctor: probe an SSH target before you try to use it.
roksbnkctl doctor --target jumphost
This adds one row per resolved target:
✓ ssh:jumphost ubuntu@169.45.91.177:22 (TOFU recorded) (verifies the target is reachable)
The probe:
- Resolves the target’s
host,user,port, and key source from~/.roksbnkctl/<workspace>/config.yaml. - Connects via the
internal/remoteSSH client. - Validates the host key against
~/.roksbnkctl/known_hosts(TOFU prompt on first contact, unless--insecure-host-key). - Runs a no-op command (
true) to confirm the channel works end-to-end.
Failure modes specific to the SSH probe:
host key mismatch— the target was rebuilt; edit~/.roksbnkctl/known_hoststo clear the entry, then re-probe.unable to authenticate— the key source resolved but the remote rejected it. Checkkey_path/key_sourcein workspace config; ifkey_source: agent, verifyssh-add -lshows the right key.dial tcp: i/o timeout— thehost:portis unreachable. Verify withnc -vz <host> 22from a known-good network.
Pass --target all to probe every target listed in the workspace’s targets: block. Useful in CI when you want a single command that asserts every entry is reachable.
Reading the exit code
doctor exits with:
0— all checks are green or warnings only. Warnings do not faildoctor. The everyday workflow can proceed.- non-zero — at least one row produced an
✗error. The first error string is also written to stderr so wrapper scripts can grep it.
This is the contract scripts/e2e-test.sh and the Makefile rely on: a script that runs roksbnkctl doctor && roksbnkctl up --auto will only proceed past doctor if the environment is genuinely ready.
The “warnings don’t fail” rule is deliberate. An iperf3 not on PATH warning is informational — the everyday up / test connectivity flow doesn’t need it. Forcing exit-1 on every warning would be too aggressive for the common case.
If you want to gate scripts strictly (e.g. CI workflows that must have iperf3 installed because they run the throughput suite), parse the output rather than relying on the exit code:
if ! roksbnkctl doctor | grep -q '^✓ iperf3'; then
echo "iperf3 missing — install it before running test throughput" >&2
exit 1
fi
What doctor is not
A few deliberate non-features worth naming:
- Not a fix-it tool.
doctorreports; it never installs, never modifies workspace config, never calls IBM Cloud APIs that mutate state. The IAM verify call is read-only. Ifdoctorcould break things, users couldn’t run it freely — and “rundoctor” needs to be a safe first move. - Not a backend probe. Per-backend availability checks (docker daemon reachable, k8s ops pod healthy, ssh target reachable) ship as separate
BackendName-tagged rows viadoctor --backend <name>(PRD 03). The--targetprobe was the early prototype of that pattern. - Not concurrent-safe. The CLI invokes
doctoronce per command; the side-channel for “why we care” blurbs ininternal/doctor/doctor.godoesn’t synchronise. Don’t run twodoctors against the same process.
Cross-references
- Chapter 4 — Installation introduces
doctoras the post-install verification step. - Chapter 6 — Workspaces explains the
workspacerow and the-woverride. - Chapter 14 — Credentials is the deep dive on the
ibmcloud api keyresolution chain. - Chapter 16 — The
--onflag covers the--targetprobe’s underlying SSH client. - Chapter 24 — Day-2 ops is the canonical reference for the internalised
k <verb>commands that makekubectl/ocinformational.
Workspaces
A workspace is a per-environment bundle of config + state. The shape is modelled on kubectl contexts: you can have many of them, exactly one is “current” at a time, and a -w flag lets you address a specific one for a single command without flipping the pointer.
This chapter covers the on-disk layout, the everyday init / use / list flow, the full roksbnkctl workspaces command tree, the -w / --workspace override, and how creating or deleting a workspace moves the “current” pointer for you.
Workspace selection follows you automatically. Creating a workspace (
initorws new) makes it current; deleting the current workspace moves the pointer to another existing one, or clears it when none remain. There is no phantomdefaultfallback — with nothing selected, commands say so instead of guessing.
The on-disk layout
Every workspace lives under ~/.roksbnkctl/<name>/:
~/.roksbnkctl/
config.yaml # global; current_workspace pointer
known_hosts # SSH host keys (shared across workspaces)
default/ # workspace "default" (mode 0700)
config.yaml # this workspace's inputs (mode 0600)
cluster-outputs.json # post-apply cluster identity (when present)
state/ # BNK trial state (mode 0700)
terraform.tfstate
terraform.tfvars
kubeconfig # admin kubeconfig (mode 0600)
tf-source/ # bundled HCL extracted to disk
scratch/ # docker bind-mounts, helm caches
state-cluster/ # cluster-phase state (separate tree)
terraform.tfstate
cluster-phase-override.tfvars
prod/ # workspace "prod"
config.yaml
state/
...
Three things are worth calling out:
~/.roksbnkctl/config.yamlis global — non-secret user-wide preferences plus thecurrent_workspacepointer. It is not a workspace config; the per-workspace files live one level deeper.state/andstate-cluster/are intentionally separate soroksbnkctl cluster upandroksbnkctl updon’t tangle their Terraform state. Most users won’t touch either directly.cluster-outputs.jsonis the persisted identity of the workspace’s ROKS cluster — written bycluster uporcluster register, read byroksbnkctl upso BNK trials don’t have to re-state cluster identity in every tfvars. It also records the create-time settings the cluster was built with, and for those it outranksconfig.yaml— the record says what the cluster is, the config says what is wanted. See Chapter 8a §Which side wins.
Override the base directory with the ROKSBNKCTL_HOME env var. Test fixtures use this; everyday users shouldn’t need it.
terraform.applied.tfvars — what’s deployed right now
roksbnkctl keeps a per-phase snapshot of the effective Terraform var-file inputs that produced the workspace’s current state. After every successful terraform apply — roksbnkctl cluster up or roksbnkctl bnk up — roksbnkctl writes a canonical-HCL summary of “what var-files said” to the phase’s state directory. Re-create / audit / handoff workflows get a file-on-disk record of the inputs rather than reconstructing them from config.yaml (or memory).
Where it lives
| Workspace shape | Phase | Path |
|---|---|---|
ShapeSplit / ShapeClusterOnly | Cluster phase | ~/.roksbnkctl/<workspace>/state-cluster/terraform.applied.tfvars |
ShapeSplit | Trial phase | ~/.roksbnkctl/<workspace>/state/terraform.applied.tfvars |
ShapeLegacySingle | both phases (collapsed) | ~/.roksbnkctl/<workspace>/state/terraform.applied.tfvars |
On ShapeLegacySingle, the file is a union of all sources (since the legacy shape doesn’t separate cluster and trial state) and the header comment records phase=legacy-single so the reader doesn’t mistake it for either a cluster-only or trial-only snapshot. See PRD 07 §“Design” for the format spec.
What it captures
A canonical HCL var-file: one assignment per line, variables sorted alphabetically within each source section. Each section is preceded by a comment line documenting which source contributed the values:
# === from config.yaml ===— vars derived from the workspace’sconfig.yaml(written toterraform.tfvarson disk).# === from terraform.tfvars.user ===— the workspace-local user override file. If the file doesn’t exist, the section header is# === from terraform.tfvars.user (missing) ===and the body is empty.# === from cluster-phase override ===—state-cluster/cluster-phase-override.tfvars(cluster-phase snapshots only).
Source-attribution comments matter because the same variable can appear in multiple sources; the “winner” — the value Terraform actually used — is the last section to mention it. The comments let the reader trace why a particular value ended up live.
Lifecycle
- Written after every successful
terraform apply. Plan flows don’t write the snapshot — the nameterraform.applied.tfvarswould mislead if a plan-time write existed. - Overwritten each apply. If you want history, copy the file aside before re-running
upor wirerestic/ a git commit hook against~/.roksbnkctl/<workspace>/. - Untouched by destroy.
cluster down/bnk downleave the priorup’s snapshot in place; that’s what was last deployed. The file’s mtime + the absence of Terraform state is the “torn down on<date>” signal. - Never read by
roksbnkctlitself. The snapshot is an output for the user — never an input the tool depends on. Making it an input would create a feedback loop where redacted values get written back as the literal string<redacted>.
Redaction
Every variable carrying a credential is redacted: ibmcloud_api_key, bigip_password, and registry_mirror_password. ibmcloud_api_key is the one whose value comes from the cred resolver rather than being authored by the user; the other two are authored in config.yaml (base64-obfuscated there) and would otherwise appear in this file in the clear. Since this page tells you the snapshot is suitable for committing, a rendered credential missing from that list is a credential in a repository — so adding one to the render without adding it to redactedVarNames now fails a test. See PRD 04 §“Cred tmpfile-bind-mount pattern” for why the API key isn’t in tfvars in the first place. The redacted line carries an inline comment:
ibmcloud_api_key = "<redacted>" # source: cred resolver, not persisted
For team-handoff scenarios (a teammate receives this file out-of-band and wants to re-create the workspace): replace the <redacted> value with the teammate’s own API key, or simply remove the ibmcloud_api_key line so the cred resolver supplies it from the teammate’s own environment (keychain, shell env, ~/.bluemix/api_key, etc.) at apply time. Every other line round-trips verbatim.
The file mode is 0600 regardless. The non-redacted contents (workspace identifiers, region, resource group, cluster name, tunable values) aren’t credential-grade secrets, but aren’t world-readable-grade either. Tight permissions are the cheap default.
What it’s not
- Not an input to subsequent applies. The
-var-filechain on the next apply is unchanged:config.yaml-derived →terraform.tfvars.user→ phase overrides. - Not a record of Terraform defaults. If
variable "foo" { default = "bar" }and the user never setfoo, the snapshot omitsfooentirely. Capturing defaults would require runningterraform outputagainst the variables block — separate concern. - Not a state-derived value capture. Computed expressions, resource references, locals, and data-source values aren’t var-file inputs and don’t appear.
terraform consoleagainst the live state dir is the right tool for those. - Not a
TF_VAR_*env capture.roksbnkctldoesn’t setTF_VAR_*today — everything goes via-var-file— so the snapshot covers the complete input surface. A future cycle that starts usingTF_VAR_*will need to extend this file.
Safe-to-commit guidance
The file is suitable for git commit alongside config.yaml after the user verifies the redaction matches their threat model. The standard reminder applies: the workspace dir may contain other semi-sensitive material — cluster-outputs.json records the cluster’s crn and admin identity hints; the state/ and state-cluster/ trees include terraform.tfstate (which contains resource IDs, IAM bindings, and any value Terraform’s provider exposed); the kubeconfig files are mode 0600 for a reason. Review the whole workspace dir with the same lens before committing.
roksbnkctl does not touch .gitignore. If you commit the workspace, you commit the workspace; if you don’t, you don’t. The tool stays out of that decision.
Worked example
For a ShapeSplit cluster phase apply, ~/.roksbnkctl/canada-roks/state-cluster/terraform.applied.tfvars looks like:
# Generated by roksbnkctl v1.4.0 at 2026-05-14T10:23:17Z after terraform apply on phase=cluster.
# Re-generated each apply. Do not edit by hand — your changes will be overwritten.
# === from config.yaml ===
cluster_name = "canada-roks"
ibmcloud_api_key = "<redacted>" # source: cred resolver, not persisted
region = "ca-tor"
resource_group_name = "default"
# === from terraform.tfvars.user ===
worker_count = 4
# === from cluster-phase override ===
deploy_bnk = false
Re-applying from this snapshot alone reconstructs the inputs the user wrote; embedded Terraform module defaults are not captured (see §“What it’s not” above for the full list of what’s out of scope).
The header records the binary version and apply timestamp so the reader can correlate the snapshot to a specific roksbnkctl invocation. Alphabetic ordering within each section means re-running apply with identical inputs produces a byte-identical file (idempotency — handy for diffing snapshots across applies).
The everyday workspace routine
The minimum daily routine:
# Initialise (creates ~/.roksbnkctl/<name>/config.yaml and makes it current).
# With no -w and no current workspace, init asks for a name (defaults to "default").
roksbnkctl init
# Switch which workspace is "current"
roksbnkctl ws use prod
# See all workspaces and which one is current
roksbnkctl ws list
roksbnkctl init -w <name> is the one-shot path that creates the directory and populates config.yaml interactively — and selects it. Everything else (ws new, ws use, ws delete) is the deconstructed form for users who want finer-grained control. The init interview itself (which now lists your account’s regions and existing clusters) is covered in Chapter 7 — Quick start.
Skip the interview: init --config-file
config.yaml is the single declarative input, so if you already have one — from a prior workspace, a colleague’s hand-off, or roksbnkctl init example > config.yaml followed by an edit — seed the workspace from it and skip the prompts entirely:
roksbnkctl init example > config.yaml # annotated template, every axis documented
$EDITOR config.yaml
roksbnkctl init -w myws --config-file ./config.yaml
This writes ~/.roksbnkctl/myws/config.yaml directly — unknown fields are rejected, not silently dropped — and is fully non-interactive when the file is complete (ibmcloud.region, ibmcloud.resource_group, prefix, tf_source.type). The API key is not required in the file; it resolves from the environment / keychain at run time (see Credentials). A local path or an http(s) URL is accepted, and --config-file composes with --override-from-env to inject secrets from the environment — the full unattended / CI patterns (including --non-interactive, which builds config.yaml from ROKSBNKCTL_* env alone) are in Chapter 7a — Unattended setup.
Raw terraform-variable overrides
For the rare case where you need to override a raw terraform variable the config.yaml schema doesn’t surface, drop a terraform.tfvars.user at the workspace root (~/.roksbnkctl/<ws>/terraform.tfvars.user, mode 0600) — the lifecycle auto-layers it on every up / plan / apply / down for both phases — or pass --var-file <path> on a phase command (cluster up, bnk up, …) for a one-shot. The precedence chain is: rendered terraform.tfvars → terraform.tfvars.user → a phase command’s --var-file flags → phase overrides. See Chapter 13 — Terraform variables. (init seeds the workspace from config.yaml — that’s the seed surface.)
The full command tree
roksbnkctl workspaces ... # canonical name
roksbnkctl ws ... # alias
ws new <name> — empty skeleton
Creates ~/.roksbnkctl/<name>/ with no config.yaml. Useful when you want the directory to exist (so ws use works) before you run init.
roksbnkctl ws new staging
# ✓ Created workspace "staging" and made it current (run `roksbnkctl init -w staging` to configure)
Creating the skeleton selects it (sets current_workspace), so the next bare command runs against it. Most users skip ws new and use roksbnkctl init -w staging directly, which creates, configures, and selects in one go.
ws use <name> — switch current
Sets the current_workspace pointer in ~/.roksbnkctl/config.yaml:
roksbnkctl ws use prod
# ✓ Current workspace: prod
roksbnkctl ws current
# prod
Refuses to point at a non-existent workspace. The pointer is the only thing that changes — workspace state stays put.
ws current — print the pointer
roksbnkctl ws current
# default
Prints the current workspace name on stdout. If no pointer is set, prints a hint like “no current workspace; run roksbnkctl ws use <name> or roksbnkctl init” to stderr and exits 0 with empty stdout — so WS=$(roksbnkctl ws current) produces an empty string in scripts rather than spurious output.
ws list — table view
roksbnkctl ws list
NAME CURRENT REGION CLUSTER TF SOURCE
default * us-south bnk-quickstart embedded@v1.0.0
prod eu-de bnk-prod embedded@v1.0.0
staging us-south bnk-staging local:./terraform
The * marker on CURRENT highlights the active workspace. Other columns reflect each workspace’s config.yaml. Rows where config.yaml is missing or unparseable still show the name, with the other columns blank — the list never errors out because of one corrupt workspace.
ws delete <name> [--force]
Removes the workspace directory and the OS-keychain entry for its API key.
You can delete the current workspace directly — there’s no “switch first” dance. If the workspace you delete was current, the pointer moves to another existing workspace (the alphabetically-first remaining one); if it was the last workspace, the pointer is cleared (there’s no fallback default — see §“The current-workspace pointer”).
One safety rail remains: delete refuses if Terraform state lists provisioned resources (unless --force). That catches the foot-gun where you forget to run roksbnkctl down first.
roksbnkctl ws delete staging
# Delete workspace "staging"? [y/N]: y
# ✓ Deleted workspace "staging"
# ✓ Current workspace is now "default" # only printed when "staging" was current
# Refused — state still has resources
roksbnkctl ws delete prod
# Error: workspace "prod" has terraform-managed resources; pass --force to delete anyway
# I really mean it
roksbnkctl ws delete prod --force
# ✓ Deleted workspace "prod"
--force skips both the prompt and the state-non-empty check. Use it sparingly — there’s no “undo” for rm -rf ~/.roksbnkctl/<name>/.
If a
downerrored partway and left cloud resources stranded, the state guard will (correctly) blockws delete. Sweep the orphans withroksbnkctl cleanupfirst, then delete.
The current-workspace pointer
The pointer lives at ~/.roksbnkctl/config.yaml:
current_workspace: prod
Every command that doesn’t pass -w reads this pointer. The workspace verbs keep it in step with reality, so you rarely set it by hand:
initandws newselect the workspace they create — afterwards it’s current.ws userewrites it.ws deletemoves it off a deleted current workspace (to another existing one), or clears it when the last workspace is removed.
There is no fallback default. When the pointer is empty — you’ve deleted every workspace, or never created one — a command run without -w reports no workspace selected; create one with roksbnkctl init or pick one with roksbnkctl ws use <name> rather than silently operating on a phantom default. (roksbnkctl init with no -w and no current pointer asks for a workspace name, defaulting to default on the first run — so the bootstrap experience is unchanged.)
If the pointer references a workspace whose directory is gone (e.g. someone rm -rf’d it by hand), commands report workspace "prod" is not initialised; run roksbnkctl init first — repoint with ws use <other> or recreate with init.
-w / --workspace for one-off overrides
Every command accepts -w <name> to override the current pointer for a single invocation:
# Doctor against "prod" without flipping the global pointer
roksbnkctl -w prod doctor
# Run init for a new workspace called "staging"
roksbnkctl init -w staging
# Get pods from the "default" cluster while currently on "prod"
roksbnkctl -w default k get pods -A
Use this when:
- You’re scripting against multiple workspaces in a single run (CI runner that exercises
default+e2e-cleanupback-to-back). - You want to run a one-off command against a different environment without losing your current context.
- You’re testing a fresh workspace before promoting it to current.
The flag only affects the running command — the pointer in ~/.roksbnkctl/config.yaml is unchanged. After the command exits, the next bare roksbnkctl reads the original pointer.
Deleting the workspace you’re in
ws delete removes the current workspace directly — there’s no “switch first” step. It moves the pointer to another existing workspace, or clears it when none remain.
# End-to-end test cleanup — just destroy and delete, in any order
roksbnkctl down --auto
roksbnkctl ws delete default --force
# ✓ Deleted workspace "default"
# ✓ Current workspace is now "e2e-cleanup" # whatever remained, alphabetically first
# Delete everything — the last delete clears the pointer
roksbnkctl ws delete e2e-cleanup --force
# ✓ Deleted workspace "e2e-cleanup"
# ✓ No workspaces remain; current workspace cleared
With no workspaces left, the pointer is empty and the next bare command reports no workspace selected (no phantom default); roksbnkctl init starts fresh.
Using a workspace’s environment in your shell
roksbnkctl shell drops you into a subshell with KUBECONFIG, IBMCLOUD_API_KEY, IC_API_KEY, and IBMCLOUD_REGION pre-loaded from the current workspace. KUBECONFIG resolves the way kubectl
resolves it — the first existing entry in an inherited $KUBECONFIG, else
~/.kube/config — not the workspace state directory:
roksbnkctl shell
# (now in a subshell)
echo $KUBECONFIG
# /home/you/.kube/config
exit
# (back to the parent shell)
Same for -w:
roksbnkctl -w prod shell
Useful when you want to run host kubectl / host oc / arbitrary tools with the workspace context loaded. The internalised verbs (roksbnkctl k get, etc.) read the same context automatically — you don’t need to be in a subshell to use them.
Common workspace patterns
A handful of patterns that come up in practice:
| Use case | Pattern |
|---|---|
| Different IBM Cloud accounts | default for personal, acct-foo for an account-specific key |
| Different regions | us-south, eu-de workspaces with distinct cluster.name values |
| Throwaway short-lived clusters | bnk-trial-N workspaces; delete with --force after down (delete the current one directly — it auto-switches) |
| CI vs local dev | dev and ci workspaces; ci uses IBMCLOUD_API_KEY from env, dev uses keychain |
| Recover a half-torn-down env | roksbnkctl cleanup sweeps stranded <prefix>-* cloud resources, then ws delete |
Workspaces are cheap. If a flow benefits from isolation, make a new one rather than fighting with --var-file overrides on the existing one.
Forward-link to Chapter 12
This chapter covers the workspace-as-a-unit: how to create, switch, list, delete. The schema of the per-workspace config.yaml itself — every field, default, valid range — is Chapter 12 — Workspace config.
Quick start: from API key to deployed BNK
This chapter walks the 4-command lifecycle (init → up → test → down) end-to-end. By the time you reach the bottom you’ll have a deployed BNK trial on a fresh ROKS cluster, a passing connectivity test, and a clean tear-down command ready when you’re done.
The lifecycle, at a glance:
sequenceDiagram
autonumber
actor User
participant CLI as roksbnkctl
participant TF as terraform-exec (embedded HCL)
participant IBM as IBM Cloud API
participant K8s as ROKS cluster + BNK
User->>CLI: roksbnkctl init
CLI->>CLI: write workspace config.yaml
CLI-->>User: workspace ready
User->>CLI: roksbnkctl up --auto
CLI->>TF: terraform plan + apply
TF->>IBM: provision cluster, VPC, jumphost
IBM-->>TF: 77 resources created
TF->>K8s: helm install cert-manager + flo + BNK
K8s-->>CLI: cluster + BNK up
CLI-->>User: kubeconfig saved, jumphost target auto-populated
User->>CLI: roksbnkctl test
CLI->>K8s: connectivity + DNS + (optional) throughput
K8s-->>CLI: pass/fail per check
CLI-->>User: green
User->>CLI: roksbnkctl down --auto
CLI->>TF: terraform destroy
TF->>IBM: 77 resources destroyed
CLI-->>User: workspace state retained
The walkthrough assumes:
- You have a
roksbnkctlbinary onPATH(Chapter 4). - You have an IBM Cloud API key for an account with permission to create ROKS clusters.
terraform >= 1.10is onPATHandroksbnkctl doctorlooks healthy (Chapter 5). The floor is 1.10, not 1.5: roksbnkctl enforces it (requireTerraformVersion) because remote state locking uses terraform’s native S3 lockfile (Chapter 12a).
If roksbnkctl doctor is not green for terraform and IBMCLOUD_API_KEY resolves, fix those first — nothing below will work otherwise.
Note. The output blocks below are illustrative — version strings, cluster IDs, IPs, and timing all vary between runs. The shape of each step is what to look for.
Step 1 — set the API key
The cleanest way to make roksbnkctl see your API key is the IBMCLOUD_API_KEY environment variable. roksbnkctl init will offer to save it to your OS keychain afterwards, so you only paste it once.
export IBMCLOUD_API_KEY="ibmcloud-api-key-value-here"
If you’d rather not export it in your shell, roksbnkctl init will prompt for it on a TTY and offer the same keychain-save afterwards. See Chapter 14 for the full resolution chain.
Step 2 — roksbnkctl init
Initialises a workspace under ~/.roksbnkctl/default/ (or under <name>/ if you pass -w <name>). Verifies the API key against IBM IAM, resolves the resource group, and writes config.yaml.
Flag-value syntax. Short flags accept their value as
-w value(one space) or-w=value(immediate equals); the stuck-together form-wvalueis rejected at parse time before any workspace dir is created. The same rule applies to every short flag (-w,-f,-n,-c,-l,-o). Long flags follow the same two shapes:--workspace valueor--workspace=value. So if you typo the long-flag name as a short — e.g.roksbnkctl init -ws canada-roks --config-file ./config.yaml— the binary rejects it immediately with:roksbnkctl: "-ws" is not a recognised flag (looks like a stuck-together short-flag-value, which this binary does not accept). Use one of these forms instead: -w s (short, space) -w=s (short, equals) --workspace s (long, space) --workspace=s (long, equals) Did you mean '--workspace s'?Recovery is what the error names: re-run with
--workspace canada-roks(or-w canada-roks).
roksbnkctl init
init interviews your account: after it verifies the API key it asks whether to create a new cluster, then lists the regions (or the existing clusters) your credentials can reach — you pick from a menu instead of typing free-text.
Sample interactive session (creating a new cluster):
roksbnkctl init
Workspace name [default]:
→ Verifying IBM Cloud credentials ... ✓ you@example.com (account 1a2b3c...)
Create a new ROKS cluster? [Y/n]: y
Region for the new cluster:
1) au-syd
2) br-sao
* 3) ca-tor
...
10) us-south
Choice [1-14] [3]: 10
Workspace prefix [bnk-quickstart]:
Workers per zone (min 1; cluster spans all 3 AZs) [1]: 2
OpenShift version [4.20]:
Create registry COS instance? [Y/n]: y
Resource group [default]:
→ ✓ Resource group "default" (id ...)
Create Transit Gateway? [Y/n]: y
Install cert-manager? [Y/n]: y
Deploy BIG-IP Next (BNK)? [Y/n]: y
Add a testing client? [y/N]: y
Region to install the test client in:
...
Choice [1-14] [10]: 3
Create a new client VPC for it? [Y/n]: y
Create per-zone cluster jumphosts? [y/N]: n
Save IBMCLOUD_API_KEY to OS keychain for this workspace? (y/N): y
✓ Current workspace: default
✓ Wrote ~/.roksbnkctl/default/config.yaml
What just happened:
- A workspace called
defaultnow exists at~/.roksbnkctl/default/and is current — the next bare command runs against it (no separatews useneeded). - You picked the region from the account’s available regions; the cluster spans all three availability zones with
workers-per-zone × 3workers. One per zone (3 total) is the minimum — a ROKS cluster is always multi-AZ, soinitfloors the count there. - The optional testing client (TGW jumphost + client VPC) can live in a different region from the cluster — you’re prompted for its region separately.
config.yamlrecords the region, resource group, prefix, worker sizing, the per-resourceresources:toggles, and BNK component defaults. The API key is saved to your OS keychain (macOS Keychain, libsecret on Linux, or Windows Credential Manager) under serviceroksbnkctl; subsequent runs resolve it from there without prompting.
You can re-run roksbnkctl init to update workspace settings; existing values become the prompt defaults.
Reusing an existing cluster
Answer n to Create a new ROKS cluster? and init lists the account’s running OpenShift clusters to pick one:
Create a new ROKS cluster? [Y/n]: n
→ Listing running OpenShift clusters...
Choose an existing cluster:
* 1) bnk-demo (us-south, 4.20.x_openshift)
2) roks-prod-ca (ca-tor, 4.19.x_openshift)
Choice [1-2] [1]: 1
The chosen cluster’s region and name are recorded, and init populates cluster-outputs.json — the same identity file cluster register writes — so a subsequent roksbnkctl up deploys BNK straight onto it, with no cluster up and no separate register step.
Step 3 — roksbnkctl up --auto
The deployment. Runs terraform plan, runs terraform apply, fetches the admin kubeconfig from IBM Cloud, writes it to ~/.kube/config at mode 0600. The --auto flag skips the plan-and-confirm gate; without it up shows the plan and asks “apply? [y/N]” before continuing.
roksbnkctl up --auto
Sample output (heavily abridged — a real run is ~50 minutes and prints terraform’s full plan + apply log):
roksbnkctl up --auto
→ Resolving terraform source ... embedded (v1.0.0)
→ Extracting bundled HCL to ~/.roksbnkctl/default/state/tf-source/embedded-terraform/
→ Pre-creating kubeconfig + scratch directories
→ Rendering auto-tfvars from config.yaml ... ok
→ terraform init -reconfigure
Initializing provider plugins... done.
→ terraform apply (auto-approved)
module.roks_cluster.ibm_container_vpc_cluster.cluster: Creating...
module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [10m elapsed]
module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [20m elapsed]
module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [30m elapsed]
module.roks_cluster.ibm_container_vpc_cluster.cluster: Creation complete after 38m12s
module.cert_manager.helm_release.cert_manager: Creation complete after 2m11s
module.flo.helm_release.flo: Creation complete after 4m02s
module.cne_instance.kubernetes_manifest.cne_instance: Creation complete after 1m42s
module.license.helm_release.license: Creation complete after 2m18s
module.testing.tls_private_key.jumphost_shared_key: Creation complete after 0s
module.testing.ibm_is_instance.tgw_jumphost: Creation complete after 1m48s
Apply complete! Resources: 77 added, 0 changed, 0 destroyed.
→ Fetching admin kubeconfig for cluster "<cluster-id>"
✓ Wrote /home/you/.kube/config (chmod 0600)
✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`
What just happened:
- 77 resources were created across ROKS, cert-manager, FLO, CNE Instance, BNK license, and a small testing footprint (the TGW jumphost).
- An admin kubeconfig was fetched directly from IBM Cloud’s container service API (no
ibmcloud ks cluster configshell-out) and written at mode 0600. - A
jumphosttarget was auto-populated in your workspace config from terraform outputs. This makes Chapter 16’s--on jumphostflag work without any further configuration.
The actual elapsed time on a fresh run is dominated by ROKS cluster creation (~30-40 min) and cert-manager + FLO Helm install (~10 min). Re-runs are dramatically faster because terraform’s idempotence skips already-created resources.
Step 4 — roksbnkctl status
Quick sanity check: workspace pointer is right, cluster is reachable, BNK pods are healthy.
roksbnkctl status
Sample output:
Workspace: default
Region: us-south
RG: Default (id: ...)
Cluster: bnk-quickstart (id: <cluster-id>) — Ready
TF source: embedded (v1.0.0)
Last apply: 2026-05-08T14:22:08Z
Nodes: 2/2 Ready
BNK pods: flo (3/3), cis (1/1), cert-manager (3/3), cne-instance (1/1)
If anything is not green here, jump to Chapter 26 — Troubleshooting.
Step 5 — roksbnkctl test
Run the built-in validation suite. Bare test runs the connectivity + DNS checks (the throughput test takes a few minutes and is opt-in).
roksbnkctl test
Sample output:
roksbnkctl test
→ Suite: connectivity
✓ https://www.f5.com (200, 312ms)
✓ https://api.openshift.com (200, 88ms)
✓ https://us-south.containers.cloud.ibm.com (200, 142ms)
→ Suite: dns
✓ www.f5.com → 23.50.149.94 (A, 12ms)
✓ api.openshift.com → 35.190.27.231 (A, 18ms)
3 connectivity checks passed; 2 DNS checks passed; 0 failed.
For the throughput suite specifically:
roksbnkctl test throughput --mode east-west
Sample output:
→ Deploying iperf3 server pod into namespace "roksbnkctl-test"
✓ Pod ready (iperf3-server-...)
→ Exposing via ClusterIP service
✓ Service ready (cluster-ip: 172.21.45.108:5201)
→ Running iperf3 -c against the service from local
✓ throughput: 9.41 Gbits/sec (mean over 10s)
→ Tearing down iperf3 fixture
✓ pod and service deleted
The --mode east-west flag uses a ClusterIP service and runs the host iperf3 client through oc port-forward for in-cluster traffic; --mode north-south uses a LoadBalancer for outside-the-cluster traffic. See Chapter 22 for the full design.
The connectivity test uses Go’s built-in net/http — no external curl is shelled out — and similarly DNS uses Go’s net.Resolver. The --insecure flag on test connectivity skips TLS validation if you need to test against self-signed endpoints.
Step 6 — explore (optional)
A few useful follow-ups now that the cluster’s up:
# tail the F5 Lifecycle Operator logs
roksbnkctl logs flo -f
# drop into a shell with the workspace's KUBECONFIG + IBMCLOUD_API_KEY exported
roksbnkctl shell
# run a one-shot kubectl with the workspace context loaded
roksbnkctl kubectl get pods -A
# run ibmcloud through the auto-discovered jumphost
roksbnkctl ibmcloud --on jumphost ks cluster ls
The --on jumphost flag is covered in detail in Chapter 16. It lets you run any of the passthrough commands (exec, shell, kubectl, oc, ibmcloud) from inside the cluster’s network — useful when your workstation is behind a corporate firewall that can’t reach IBM Cloud directly.
Step 7 — roksbnkctl down --auto
Tear it all back down when you’re finished. The teardown is terraform destroy under the hood, with the same resilience to transient IBM API errors as up.
roksbnkctl down --auto
Sample output:
roksbnkctl down --auto
→ terraform destroy (auto-approved)
module.testing.ibm_is_instance.tgw_jumphost: Destroying...
...
module.roks_cluster.ibm_container_vpc_cluster.cluster: Destroying...
module.roks_cluster.ibm_container_vpc_cluster.cluster: Still destroying... [5m elapsed]
module.roks_cluster.ibm_container_vpc_cluster.cluster: Destruction complete after 8m16s
Destroy complete! Resources: 77 destroyed.
✓ Workspace "default" state retained at ~/.roksbnkctl/default/
(run `roksbnkctl ws delete default` to remove the workspace dir)
down retains the workspace dir and config so you can up again with the same settings. To remove the workspace entirely:
roksbnkctl ws delete default
This refuses if terraform state still lists resources (use --force to override) and cleans up the keychain entry.
What you just did
In effectively three commands you:
- Provisioned a fresh ROKS cluster on IBM Cloud.
- Installed cert-manager, F5 Lifecycle Operator, and a complete BNK trial on top of it.
- Validated the deployment with HTTP connectivity + DNS resolution + (optionally) throughput tests.
- Got an auto-discovered jumphost target ready for any
--on jumphostfollow-ups.
The same flow runs against multiple workspaces, multiple regions, and multiple resource groups — see Chapter 6 for the multi-environment patterns. From here, Chapter 16 covers the --on flag, Chapter 24 covers day-2 operations, and Chapter 26 covers what to do when one of the above steps doesn’t go right.
Unattended setup: seeding a workspace from a file, URL, or environment
The quick start drives init interactively. A CI pipeline
or a fleet operator wants the opposite: hand init a finished config, fetch it
from a URL, and inject secrets from the environment so a single committed
template stands up many workspaces without ever baking an API key into version
control.
Three init options make that work:
| Option | What it does |
|---|---|
--config-file <path|url> | Seed the workspace config.yaml directly (no interview) |
--override-from-env | Overlay specific config.yaml fields from environment variables |
--non-interactive | Build config.yaml from environment variables alone — no file, no prompt (argv+env runners) |
Getting a starting point: roksbnkctl config
Both forms of the input can be printed, so neither has to be written from memory. With no flags it prints an annotated template, ready to pipe:
roksbnkctl config yaml > config.yaml # every setting, with the comment explaining it
roksbnkctl config env > .env # the same settings as ROKSBNKCTL_* overrides
The .env template has every variable commented out. That is deliberate: an
empty assignment is not the same as leaving a setting alone. ROKSBNKCTL_X=
sets X to empty, which overrides whatever default would otherwise apply —
uncomment only what you mean to set.
Converting between the two
The same settings expressed either way, which is what you want when a pipeline needs as environment variables what a colleague handed you as a file:
roksbnkctl config env --from-yaml config.yaml > .env
roksbnkctl config yaml --from-env .env > config.yaml
The converted output carries no comments — it is a populated file, not a template — and includes only what the input actually set, so it stays a statement of intent rather than a snapshot of every default. The handful of required fields are emitted even when empty, so a missing one is visible instead of silently absent.
Round-tripping is the quickest way to check a .env says what you think:
roksbnkctl config yaml --from-env .env # read it back as config.yaml
The conversion runs the same override machinery --override-from-env uses,
so a value lands where it would in a real run. A variable that this prints under
bnk.tmm_replicas is the variable that sets bnk.tmm_replicas; there is no
second parser that could disagree.
A worked example — the Appendix C Small cluster as environment variables:
cat > small.yaml <<'YAML'
ibmcloud: {region: us-east, resource_group: default}
prefix: small
cluster: {create: true, name: small, workers_per_zone: 2, worker_flavor: bx2.8x32}
bnk: {manifest_version: 2.4.0-EA, cneinstance_size: Tiny, tmm_replicas: 3}
YAML
roksbnkctl config env --from-yaml small.yaml
# ROKSBNKCTL_CLUSTER_CREATE=true
# ROKSBNKCTL_CLUSTER_NAME=small
# ROKSBNKCTL_CNEINSTANCE_SIZE=Tiny
# ROKSBNKCTL_MANIFEST_VERSION=2.4.0-EA
# ROKSBNKCTL_PREFIX=small
# ROKSBNKCTL_REGION=us-east
# ROKSBNKCTL_RESOURCE_GROUP=default
# ROKSBNKCTL_TMM_REPLICAS=3
# ROKSBNKCTL_WORKERS_PER_ZONE=2
# ROKSBNKCTL_WORKER_FLAVOR=bx2.8x32
The pattern
flowchart LR
T["config.yaml.tmpl<br/>(committed, no secrets)"] -->|--config-file| I["roksbnkctl init"]
E["$IBMCLOUD_API_KEY<br/>$ROKSBNKCTL_*"] -->|--override-from-env| I
I --> W["~/.roksbnkctl/<ws>/config.yaml<br/>(complete, secrets filled)"]
Commit a template with the secret fields blank, and let the pipeline supply the real values from its environment:
# config.yaml.tmpl — committed to git, no secrets
ibmcloud:
region: "" # ← ROKSBNKCTL_REGION
resource_group: "" # ← ROKSBNKCTL_RESOURCE_GROUP
api_key_b64: "" # ← IBMCLOUD_API_KEY (raw, base64-encoded)
prefix: "" # ← ROKSBNKCTL_PREFIX
cluster: { create: true, name: "" }
tf_source: { type: embedded }
resources:
bnk: { create: true }
tgw_jumphost: { create: true }
client_vpc: { create: true }
export IBMCLOUD_API_KEY="$(vault read -field=key secret/ibmcloud)"
export ROKSBNKCTL_REGION=eu-de
export ROKSBNKCTL_RESOURCE_GROUP=default
export ROKSBNKCTL_PREFIX=ci-run-42
roksbnkctl init -w ci-run-42 \
--config-file https://raw.githubusercontent.com/acme/infra/main/config.yaml.tmpl \
--override-from-env
No secret ever lands in git; the same template seeds every account.
Exit codes
Unattended callers branch on $?, so the codes are an interface:
| code | meaning |
|---|---|
0 | The command did what was asked. |
1 | It went wrong. The default for anything without a more specific code. |
2 | The invocation was rejected — a malformed flag, an argument the command cannot accept. Nothing was attempted. |
126 | Permission denied: SSH authentication, a host-key mismatch, a credential the remote end refused. |
127 | The target could not be reached at all. |
130 | The operator interrupted it (Ctrl-C / SIGINT). |
125 | upgrade / self update only. The upgrade removed the old binary and could not put anything back, so there is no roksbnkctl at the install path. The message names the .old sidecar and the command to rename it. |
| other | A wrapped tool’s own status, passed through unchanged — terraform, ibmcloud, and --on <target> remote commands all propagate theirs. That includes 125 from any command other than upgrade / self update, which spawn no child. |
130 is worth wiring into CI cleanup: it distinguishes “someone stopped this”
from “it broke”, which a job that retries on failure should treat differently.
125 matters for the same reason, in the opposite direction: an ordinary failed
upgrade is safe to retry, and this one is not — there is nothing left to run. A
wrapper that retries on any non-zero status will loop forever on a machine that
cannot recover until someone renames the sidecar back.
Before v1.50.0 only roksbnkctl init produced it — every other command turned
the same Ctrl-C into 1.
2 is similarly worth separating. It means the command never ran, so retrying
the same invocation will fail the same way; the fix is in the script, not the
environment.
A command that propagates a wrapped tool’s status does not print its own error
on top — terraform’s diagnostics have already gone to stderr, and a second
roksbnkctl: ... line would only add noise. Check the stream, not just the
code.
The interrupt outranks everything else
A failure that happens because of the Ctrl-C reports the interrupt, not the
failure class it landed in: a connect aborted mid-handshake is 130, not
127; a test suite cut off mid-run is 130, not a red 1; a child killed by
the signal propagates 130 (128+SIGINT, the shell convention), not 255.
So $? -eq 130 is reliable — a job does not need to trap SIGINT itself to
tell “someone stopped this” from “it broke”.
--config-file
--config-file parses the supplied YAML strictly — unknown fields are
rejected, not silently dropped — into the workspace config.yaml, and skips
the interview entirely. It is non-interactive when the config is complete.
A config is complete when it has all of:
ibmcloud.regionibmcloud.resource_groupprefixtf_source.type
If any are missing (after --override-from-env runs), init exits with a clear
message naming exactly which — supply them in the file, set them via the
environment, or run init interactively. The API key is not required in the
file; it resolves from the environment, keychain, or config at run time as usual
(see Credentials).
URL inputs
--config-file accepts a local path or an http(s) URL. A URL is fetched
(30 s timeout, 10 MB cap) and treated identically to a local file — a raw-git
URL, a presigned IBM COS URL, or any reachable endpoint. There is no fetch
authentication in this release; use a presigned or public URL.
--override-from-env
--override-from-env overlays config.yaml fields from environment variables
that are set and non-empty, after the file/interview has produced the config.
The environment wins over the seeded value — it is the explicit late-binding
step. This is a fixed field map, not arbitrary templating.
This table is a selection, not the whole surface. The ones below are what most deployments set, chosen because their encoding or precedence needs explaining — a hand-written table of every override would be a second copy of the schema, and this book has been bitten by one of those before. The complete list is generated from the override machinery itself and cannot drift from it:
$ roksbnkctl config env > .env # every variable, commented, with its fieldor the config.yaml cheatsheet, which shows every field beside the variable that sets it. Prefer either of those when you are looking for a specific field; use this table when you want to know how a value is interpreted.
| Environment variable | config.yaml field | Encoding |
|---|---|---|
IBMCLOUD_API_KEY | ibmcloud.api_key_b64 | raw key, base64-encoded |
ROKSBNKCTL_API_KEY_B64 | ibmcloud.api_key_b64 | verbatim (pre-encoded) |
ROKSBNKCTL_PREFIX | prefix | verbatim |
ROKSBNKCTL_REGION | ibmcloud.region | verbatim |
ROKSBNKCTL_RESOURCE_GROUP | ibmcloud.resource_group | verbatim |
ROKSBNKCTL_CLUSTER_NAME | cluster.name | verbatim |
ROKSBNKCTL_CLUSTER_CREATE | cluster.create | bool (true/false/1/0) |
ROKSBNKCTL_OPENSHIFT_VERSION | cluster.openshift_version | verbatim |
ROKSBNKCTL_WORKERS_PER_ZONE | cluster.workers_per_zone | integer |
ROKSBNKCTL_CLUSTER_PUBLIC_GATEWAY | cluster.public_gateway | bool — false builds a disconnected cluster whose workers have no Internet egress. Unset inherits the terraform default (true). |
ROKSBNKCTL_TRUSTED_PROFILE_SA | bnk.trusted_profile.service_account | Kubernetes account allowed to assume the CNE controller’s IBM Cloud Trusted Profile. Must match the account the FLO chart creates. |
ROKSBNKCTL_TRUSTED_PROFILE_ROLES | bnk.trusted_profile.roles | IAM roles for that profile, scoped to the cluster’s own VPC. Comma-separated. An unparseable entry is dropped. |
ROKSBNKCTL_VLAN_PREFIXLEN | bnk.network.vlan_prefixlen | TMM self-IP mask. Independent of the zone CIDRs and never derived from them — a deliberate disagreement, plus static routes, is how a traffic pattern is forced. Out-of-range values are silently ignored and the terraform default stands. |
ROKSBNKCTL_VLAN_PREFIXLEN_EXTERNAL | bnk.network.vlan_prefixlen_external | Overrides the shared mask for the external VLAN only. Unset = inherit. Silently ignored if out of range. |
ROKSBNKCTL_VLAN_PREFIXLEN_INTERNAL | bnk.network.vlan_prefixlen_internal | Same, internal VLAN. The two need not match. |
ROKSBNKCTL_GENERIC_CA_B64 | registry.generic_ca_b64 | Mirror CA, verbatim — already base64, not re-encoded. |
ROKSBNKCTL_GENERIC_CA_SHA256 | registry.generic_ca_sha256 | The out-of-band CA pin. |
ROKSBNKCTL_REACHABILITY_RETRY_SECONDS | bnk.preflight.reachability_retry_seconds | Per-target retry window before the verdict is believed. |
ROKSBNKCTL_REACHABILITY_TIMEOUT_SECONDS | bnk.preflight.reachability_timeout_seconds | Total wait for every node to report. |
ROKSBNKCTL_CLUSTER_NETWORK_MODE | cluster.network_mode | How the workers are attached: single-nic (default) or multi-nic. Create-time only. Exists for exactly this chapter’s case — a runner that never writes a config.yaml would otherwise have no way to ask for it. |
ROKSBNKCTL_CLUSTER_VPC_CIDR | cluster.vpc_cidr | CIDR (/18 or larger) for a new cluster VPC’s per-zone address prefixes. Unset = IBM auto, which is the SAME block for every VPC in the region — set a distinct one per cluster when two share a Transit Gateway, or the gateway silently blackholes one. See Chapter 9a. |
ROKSBNKCTL_TRANSIT_GATEWAY_NAME | resources.transit_gateway (create:false + existing) | a Transit Gateway name or id — cluster up/register attaches the cluster VPC to it (see Sharing a Transit Gateway) |
ROKSBNKCTL_EXISTING_SUBNET_IDS | cluster.existing_subnet_ids | Place the cluster in subnets that already exist. Comma-separated, in zone order — each subnet’s zone is read from the subnet, so a reordered list places the cluster differently. Requires resources.cluster_vpc: {create: false, existing: <vpc-id>}. |
ROKSBNKCTL_FLP_VSI_CREATE_VPC | bnk.flp.vsi.create_vpc | Give the FLP VSI its own VPC instead of placing it in one that exists. This is what lets the proxy be the first thing deployed in an air-gapped estate. Mutually exclusive with ROKSBNKCTL_FLP_VSI_VPC. |
ROKSBNKCTL_FLP_VSI_VPC_NAME | bnk.flp.vsi.vpc_name | Name for that VPC. Empty → flp-vsi-vpc. |
ROKSBNKCTL_FLP_VSI_SUBNET_CIDR | bnk.flp.vsi.subnet_cidr | Its address prefix. Must not overlap anything the consuming clusters can already route to — a transit gateway silently blackholes one of two overlapping VPCs. |
ROKSBNKCTL_CLUSTER_VPC_ID | resources.cluster_vpc (create:false + existing) | verbatim — adopt an existing cluster VPC by ID |
ROKSBNKCTL_TESTING_VPC_NAME | resources.testing_client_vpc_name | verbatim (names the client VPC to create) |
ROKSBNKCTL_TGW_JUMPHOST_CREATE | resources.tgw_jumphost.create | bool — the optional testing jumphost. Defaults off, as the interview does. |
ROKSBNKCTL_CLIENT_VPC_CREATE | resources.client_vpc.create | bool — the testing client VPC. Defaults off; it consumes a Transit Gateway connection. |
ROKSBNKCTL_CLIENT_VPC_NAME | resources.client_vpc.existing | Adopt an existing client VPC instead of creating one — the env equivalent of the interview’s “Existing client VPC name”. Required if you enable the jumphost without creating a VPC. |
ROKSBNKCTL_CERT_MANAGER_CREATE | resources.cert_manager.create | bool — set false to adopt the cert-manager the cluster already runs instead of installing one. An adopted customer cluster very often has one; without this, bnk up fails with namespaces "cert-manager" already exists, which names the collision but not the setting that resolves it. |
ROKSBNKCTL_REGISTRY_COS_CREATE | resources.registry_cos.create | bool — set false when the mirror registry’s object storage already exists (or the registry is external, such as Artifactory or Harbor, and needs none). |
ROKSBNKCTL_CLUSTER_JUMPHOSTS_CREATE | resources.cluster_jumphosts.create | bool — the per-cluster jumphosts. Defaults off. |
ROKSBNKCTL_BIGIP_URL | bnk.cis.bigip_url | verbatim |
ROKSBNKCTL_BIGIP_USERNAME | bnk.cis.bigip_username | verbatim |
ROKSBNKCTL_BIGIP_PASSWORD | bnk.cis.bigip_password_b64 | raw, base64-encoded |
ROKSBNKCTL_ZONE<n>_EXT_VLAN_CIDR … _INTERNAL_SELFIP | bnk.network.zones[n-1] | per-zone VLAN/SNAT/VIP CIDRs + self-IPs (n = 1…3; all six fields required for a zone to apply) |
ROKSBNKCTL_TESTING_SSH_KEY_NAME | resources.testing_ssh_key_name | verbatim |
ROKSBNKCTL_BNKFORGE_CA_B64 | bnkforge.ca_b64 | PEM CA the BNK Forge server’s certificate must chain to, base64-encoded. Pins a self-signed lab cert so the session token is not sent over an unauthenticated connection. Supersedes bnkforge.insecure. |
ROKSBNKCTL_TESTING_JUMPHOST_ALLOWED_CIDRS | resources.testing_jumphost_allowed_cidrs | comma-separated source CIDRs for SSH (:22) to the testing jumphosts. Unset → open (they carry a public floating IP; access is key-only). Set to your public /32 on a shared account. |
ROKSBNKCTL_TESTING_CLIENT_VPC_INBOUND_CIDRS | resources.testing_client_vpc_inbound_cidrs | comma-separated; inbound to the testing client VPC’s default SG. Unset → the RFC-1918 ranges (in-fabric test traffic arrives over the Transit Gateway). |
ROKSBNKCTL_CLUSTER_HTTP_ALLOWED_CIDRS | resources.cluster_http_allowed_cidrs | comma-separated; sources for :80 on the cluster SG. Unset → open (this is the ingress/ALB path). |
ROKSBNKCTL_CLUSTER_VPC_DEFAULT_SG_INBOUND_CIDRS | resources.cluster_vpc_default_sg_inbound_cidrs | comma-separated; inbound (all protocols/ports) to the cluster VPC’s default SG. Unset → open, the historical behaviour. |
ROKSBNKCTL_REGISTRY_TARGET | registry.target | icr | generic |
ROKSBNKCTL_GENERIC_HOST | registry.generic_host | verbatim, e.g. harbor.example.com |
ROKSBNKCTL_GENERIC_REPO_PREFIX | registry.generic_repo_prefix | verbatim — a Harbor project, an Artifactory repo key |
ROKSBNKCTL_GENERIC_USERNAME | registry.generic_username | verbatim |
ROKSBNKCTL_GENERIC_PASSWORD | registry.generic_password_b64 | raw, base64-encoded |
ROKSBNKCTL_LICENSE_MODE | bnk.license_mode | connected | disconnected | f5licenseproxy (see Chapter 10c) |
ROKSBNKCTL_FLO_NAMESPACE | bnk.flo_namespace | Namespace FLO installs into. Default f5-bnk. |
ROKSBNKCTL_FLO_UTILS_NAMESPACE | bnk.flo_utils_namespace | Namespace for the F5 utility components. Default f5-utils. Set both to the same value for one shared namespace — verified against BNK 2.3. |
ROKSBNKCTL_GATEWAY_CLASS_NAME | gateway.class_name | GatewayClass name; blank → gateway-class. GatewayClass is cluster-scoped, so two BNK installs sharing a cluster must not share it. |
ROKSBNKCTL_GATEWAY_CONTROLLER_NAME | gateway.controller_name | Leave blank. Blank derives f5.com/<flo_namespace>-f5-cne-controller — the value the CNE controller answers to — and follows ROKSBNKCTL_FLO_NAMESPACE automatically. Set it only to aim the GatewayClass at a controller this deployment did not install. A wrong value fails silently: the apply succeeds, the GatewayClass is never Accepted, and no traffic flows. |
ROKSBNKCTL_GATEWAY_ROUTE_EXAMPLES | gateway.route_examples | Comma-separated extra route kinds to create working examples of. On BNK 2.3: GRPCRoute, L4Route. Blank leaves an existing deployment byte-identical. TCPRoute is not valid — 2.3 pins Gateway API 1.4.1 standard, which does not contain it; BNK uses its own L4Route for TCP. |
ROKSBNKCTL_GATEWAY_L4_LISTENER_PORT | gateway.l4_listener_port | Port for the TCP listener L4Route needs. Blank → 8080. |
ROKSBNKCTL_FLP_NAMESPACE | bnk.flp.namespace | verbatim (FLP mode only) |
ROKSBNKCTL_FLP_EXTERNAL_URL | bnk.flp.external.url | verbatim — license against a proxy in another cluster |
ROKSBNKCTL_FLP_ROOT_CA_B64 | bnk.flp.external.root_ca_b64 | verbatim; already base64 — re-encoding it hands the CWC a corrupt CA |
ROKSBNKCTL_FLP_MODE | bnk.flp.mode | helm (default) | vsi — how the FLP phase deploys the proxy |
ROKSBNKCTL_FLP_VSI_NAME_PREFIX | bnk.flp.vsi.name_prefix | Prefixes the FLP VSI’s resource names. Blank keeps the legacy unprefixed names — setting it replaces a running proxy. Needed to run more than one FLP in an account, and to make the proxy’s resources visible to cleanup’s <prefix>-* sweep. |
ROKSBNKCTL_FLP_VSI_VPC | bnk.flp.vsi.vpc | an existing VPC id. With _MODE=vsi this arms the standalone, cluster-less appliance — as does _CREATE_VPC=true, which builds one instead. Mutually exclusive. |
ROKSBNKCTL_FLP_VSI_ZONE | bnk.flp.vsi.zone | e.g. us-south-1; blank → the region’s first zone |
ROKSBNKCTL_FLP_VSI_PROFILE | bnk.flp.vsi.profile | VSI instance profile; blank → bx2-4x16 (the FLP’s 4 vCPU / 8 GB floor) |
ROKSBNKCTL_FLP_VSI_SSH_KEY | bnk.flp.vsi.ssh_key | name of an existing IBM Cloud VPC SSH key |
ROKSBNKCTL_FLP_VSI_BOOT_SIZE_GB | bnk.flp.vsi.boot_size_gb | integer; blank → 100 |
ROKSBNKCTL_FLP_VSI_FLOATING_IP | bnk.flp.vsi.floating_ip | bool; an unparseable value leaves the module default (true) rather than pinning false |
ROKSBNKCTL_FLP_VSI_MANAGEMENT_ALLOWED_CIDRS | bnk.flp.vsi.management_allowed_cidrs | comma-separated list — gates the :80 status UI |
ROKSBNKCTL_FLP_VSI_LICENSING_ALLOWED_CIDRS | bnk.flp.vsi.licensing_allowed_cidrs | comma-separated list — gates the :8443 proxy + :22 |
ROKSBNKCTL_FLP_VSI_STATUS_IMAGE | bnk.flp.vsi.status_image | the flp-status web-UI image, e.g. <harbor>/bnk-status/flp-status:v1 |
ROKSBNKCTL_FLP_VSI_STATUS_REGISTRY_HOST | bnk.flp.vsi.status_registry_host | verbatim — the mirror the status image is pulled from |
ROKSBNKCTL_FLP_VSI_STATUS_REGISTRY_CA_B64 | bnk.flp.vsi.status_registry_ca_b64 | verbatim; already base64 — the mirror’s CA, dropped into the VSI’s certs.d |
ROKSBNKCTL_MANIFEST_VERSION | bnk.manifest_version | verbatim — the BNK manifest release |
ROKSBNKCTL_FAR_AUTH_LOCAL_FILE | bnk.far_auth_local_file | path to the FAR auth tarball in the run workspace |
ROKSBNKCTL_SUBSCRIPTION_JWT_LOCAL_FILE | bnk.subscription_jwt_local_file | path to the subscription JWT. Set both local-file vars and the COS download is skipped (use_cos_bucket = false) |
ROKSBNKCTL_COS_INSTANCE | cos.instance | verbatim; blank → bnk-supply-chain |
ROKSBNKCTL_COS_BUCKET | cos.bucket | verbatim; blank → bnk-artifacts |
ROKSBNKCTL_COS_REGION | cos.region | verbatim; blank → us-south |
ROKSBNKCTL_FAR_AUTH_FILE | bnk.far_auth_file | object name in the bucket; blank → f5-far-auth-key.tgz |
ROKSBNKCTL_SUBSCRIPTION_JWT_FILE | bnk.subscription_jwt_file | object name in the bucket; blank → subscription.jwt |
The FLP-VSI block reproduces, from environment variables alone, the standalone
licensing appliance the disconnected-cluster walkthrough
builds from a hand-written config.yaml — for an argv-only runner (a CI
container, BNK Forge’s container engine) that has no shell to write a YAML file
with. bnk.flp.mode: vsi plus a network selects the cluster-less path — either
bnk.flp.vsi.vpc (adopt an existing VPC) or bnk.flp.vsi.create_vpc: true (build
one). The two are mutually exclusive; the mode alone leaves the cluster-required
behaviour.
The supply-chain block matters because the COS fallbacks were renamed in
v1.22.0 (bnk-orchestration → bnk-supply-chain, bnk-schematics-resources →
bnk-artifacts, trial.jwt → subscription.jwt). Before these variables existed
a runner that could not write a config.yaml was pinned to whatever the defaults
happened to be; an account still holding the pre-v1.22 layout can now say so.
The FLP/registry handoff vars are what turn a CI pipeline into a workspace with no config.yaml to
template. The registry four say where the mirror is; the FLP two are the cross-job
handoff — the job that owns the proxy prints them with flp output flp_external_endpoint / flp_root_ca, and the job that installs BNK receives them as
ordinary job outputs. See Flow C in CI.
Notes:
- For the API key,
IBMCLOUD_API_KEYholds the raw key andinitbase64-encodes it intoapi_key_b64(matching how the rest of the tool readsIBMCLOUD_API_KEY). If you already have the base64 form, setROKSBNKCTL_API_KEY_B64instead — it takes precedence and is stored verbatim. initlogs how many overrides were applied and which fields — never the values, so secrets stay out of CI logs.--override-from-envalso applies on the interactive path, so you can answer the interview and still pull the API key from the environment.
--non-interactive — config from the environment alone
--override-from-env overlays env onto a file or interview. --non-interactive
goes further: it builds config.yaml entirely from the environment — no
--config-file, no prompts, no TTY. It is the path for an argv + env container
runner (a CI job, or a BNK Forge container
step) that can pass environment variables and arguments but cannot stage a seed
file.
roksbnkctl -w forge init --non-interactive
It assembles the workspace from the same ROKSBNKCTL_* / IBMCLOUD_API_KEY
variables in the table above, then:
- defaults
tf_source.typetoembedded(the one required field with no env override — an empty type already means “embedded” at render time); - validates completeness and fails fast if a required field is missing
(
ibmcloud.region,ibmcloud.resource_group,prefix,tf_source.type) — it never falls back to a prompt; - logs which fields were applied (never the values).
A complete one-shot, file-free setup:
export IBMCLOUD_API_KEY=… # raw key → ibmcloud.api_key_b64
export ROKSBNKCTL_REGION=eu-de
export ROKSBNKCTL_RESOURCE_GROUP=default
export ROKSBNKCTL_PREFIX=acme-eu
export ROKSBNKCTL_CLUSTER_NAME=acme-eu-roks
export ROKSBNKCTL_CLUSTER_CREATE=true
roksbnkctl -w forge init --non-interactive
roksbnkctl -w forge cluster up --auto
roksbnkctl -w forge bnk up --auto
This is exactly how the all-in-one runner image
is meant to be driven from a pipeline: every step is roksbnkctl <args> with the
config supplied through the environment, and state persisted on the mounted
/work volume (ROKSBNKCTL_HOME=/work/.roksbnkctl).
GitHub Actions CI as an example
roksbnkctl is built to be driven from a pipeline: config is the input, the
environment supplies the values, and one image carries every tool. This chapter
is a worked GitHub Actions example — a single cluster, then a fleet — using the
unattended / --non-interactive flow and the
all-in-one runner image.
The whole model in one breath: GitHub secrets → ROKSBNKCTL_* env →
init --non-interactive builds config.yaml → up reconciles it. No API key
in the repo, no hand-edited config, no interactive prompts.
The runner image as a job container
Run the job in roksbnkctl-tools-runner — it has roksbnkctl plus terraform,
helm, kubectl, oc, and the IBM Cloud CLI on PATH, so the workflow needs nothing
installed:
jobs:
deploy:
runs-on: ubuntu-latest
container:
image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:latest
steps:
- run: roksbnkctl version
Ephemeral runners need remote state
A GitHub-hosted runner is thrown away after the job, so a later teardown run can’t see the terraform state from the deploy run. Put the state in COS so it survives across runs (and across runners) — see Remote state:
state: { backend: s3 } # in config.yaml; HMAC keys come from env (below)
Supply the bucket via config and the HMAC keys via env
(ROKSBNKCTL_COS_HMAC_ACCESS_KEY / _SECRET_KEY).
A single-cluster workflow
name: Deploy BNK
on:
workflow_dispatch:
inputs:
region: { description: IBM Cloud region, default: eu-de }
prefix: { description: Resource prefix, default: ci }
jobs:
deploy:
runs-on: ubuntu-latest
container: { image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:latest }
env:
# secrets → the credential + cluster identity
IBMCLOUD_API_KEY: ${{ secrets.IBMCLOUD_API_KEY }}
ROKSBNKCTL_REGION: ${{ inputs.region }}
ROKSBNKCTL_RESOURCE_GROUP: default
ROKSBNKCTL_PREFIX: ${{ inputs.prefix }}
ROKSBNKCTL_CLUSTER_NAME: ${{ inputs.prefix }}-roks
ROKSBNKCTL_CLUSTER_CREATE: "true"
# adopt shared infrastructure (reuse is config — no extra commands)
ROKSBNKCTL_TRANSIT_GATEWAY_NAME: shared-corp-tgw
# remote state so teardown works from a fresh runner
ROKSBNKCTL_COS_HMAC_ACCESS_KEY: ${{ secrets.COS_HMAC_ACCESS_KEY }}
ROKSBNKCTL_COS_HMAC_SECRET_KEY: ${{ secrets.COS_HMAC_SECRET_KEY }}
steps:
- name: Seed config.yaml from the environment
run: roksbnkctl -w ci init --non-interactive
- name: Deploy (cluster + BNK + testing, per config)
run: roksbnkctl -w ci up --auto
- name: Publish cluster outputs
run: |
roksbnkctl -w ci cluster output --json | tee -a "$GITHUB_STEP_SUMMARY"
echo "roks_cluster_id=$(roksbnkctl -w ci cluster output roks_cluster_id)" >> "$GITHUB_OUTPUT"
init --non-interactive logs which fields it pulled from the environment (never
the values), then up brings up the cluster (create or attach, per
cluster.create) and BNK ∥ Testing — exactly the phases the config enables.
Add roksbnkctl -w ci gateway up --auto if the deployment includes the gateway
phase.
A fleet: deploy many clusters with a matrix
The thing that makes this a CI tool is that the same env-driven flow scales to a fleet — one workspace per cluster, varied by the matrix, all sharing one TGW:
name: Deploy fleet
on: { workflow_dispatch: {} }
jobs:
deploy:
runs-on: ubuntu-latest
container: { image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:latest }
strategy:
fail-fast: false
matrix:
cluster:
- { ws: eu, region: eu-de, prefix: acme-eu }
- { ws: us, region: us-south, prefix: acme-us }
- { ws: jp, region: jp-tok, prefix: acme-jp }
env:
IBMCLOUD_API_KEY: ${{ secrets.IBMCLOUD_API_KEY }}
ROKSBNKCTL_RESOURCE_GROUP: default
ROKSBNKCTL_CLUSTER_CREATE: "true"
ROKSBNKCTL_TRANSIT_GATEWAY_NAME: shared-corp-tgw
ROKSBNKCTL_COS_HMAC_ACCESS_KEY: ${{ secrets.COS_HMAC_ACCESS_KEY }}
ROKSBNKCTL_COS_HMAC_SECRET_KEY: ${{ secrets.COS_HMAC_SECRET_KEY }}
steps:
- env:
ROKSBNKCTL_REGION: ${{ matrix.cluster.region }}
ROKSBNKCTL_PREFIX: ${{ matrix.cluster.prefix }}
ROKSBNKCTL_CLUSTER_NAME: ${{ matrix.cluster.prefix }}-roks
run: |
roksbnkctl -w ${{ matrix.cluster.ws }} init --non-interactive
roksbnkctl -w ${{ matrix.cluster.ws }} up --auto
Each matrix leg is one cluster; reuse-by-name (ROKSBNKCTL_TRANSIT_GATEWAY_NAME)
lets them share one network fabric. To dictate per-AZ addressing or attach a BIG-IP
for CIS, add the matching env vars (ROKSBNKCTL_ZONE<n>_*, ROKSBNKCTL_BIGIP_*) —
the full set is in Unattended setup.
Teardown
A separate manually-triggered (or PR-close) workflow tears a workspace down. Because the state lives in COS, it runs cleanly on a fresh runner:
name: Teardown
on: { workflow_dispatch: { inputs: { ws: { required: true } } } }
jobs:
destroy:
runs-on: ubuntu-latest
container: { image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:latest }
env:
IBMCLOUD_API_KEY: ${{ secrets.IBMCLOUD_API_KEY }}
ROKSBNKCTL_COS_HMAC_ACCESS_KEY: ${{ secrets.COS_HMAC_ACCESS_KEY }}
ROKSBNKCTL_COS_HMAC_SECRET_KEY: ${{ secrets.COS_HMAC_SECRET_KEY }}
steps:
- run: roksbnkctl -w ${{ inputs.ws }} down --auto
(An attached existing cluster — ROKSBNKCTL_CLUSTER_CREATE=false — is not
destroyed; down tears down only what roksbnkctl created.)
Notes
- Pin the image by digest in production CI (
...-runner@sha256:…), not:latest, so a run can’t silently change toolchains. - Secrets never reach the repo or the logs — they enter as job env from GitHub
Secrets, and
initredacts which fields it applied. - This is the same contract bnk-forge uses. The bnk-forge container step is a GitHub-Action-style invocation of this image with these env vars; this chapter is the do-it-yourself version of Registering with BNK Forge.
- Every phase has
status+output.roksbnkctl <phase> status [--json]and<phase> output [name](scoped to that phase’s own attributes) are CI stage gates;roksbnkctl outputmerges them all — e.g.roksbnkctl testing output testing_tgw_jumphost_ip. - Where this is heading.
upalready reconciles cluster + BNK + Testing from config; foldinggatewayin (so a singleupis the whole deployment) plusoutput -o json/validateare the next step — see PRD 17 — Declarative CLI.
The cluster phase (cluster up/down)
A workspace is three phases — Cluster / BNK / Testing. This chapter covers the Cluster phase specifically; the BNK trial and the testing jumphosts are separate sibling phases that deploy in parallel on top. The “trial phase” language in this chapter means the BNK phase, and the jumphosts live in their own Testing phase. See Chapter 8a — The three-phase lifecycle for the full model, parallel
up, androksbnkctl testing up/down.
A roksbnkctl workspace is two phases on top of each other: a durable cluster phase (the ROKS cluster + cluster-shared services that take 30+ minutes to provision) and a short-lived trial phase (the BNK trial that iterates on top in 5-10 minutes). The cluster phase is exposed as its own command pair, roksbnkctl cluster up / roksbnkctl cluster down, so the cluster survives across many BNK trial cycles.
This two-phase shape is the default for every new workspace. A fresh
roksbnkctl upprovisions the cluster phase first, then the trial phase, against separate state directories. Tearing down only the trial — the common iteration case — usesroksbnkctl bnk downand leaves the cluster intact. The unscopedup/downverbs are shape-aware composites that delegate to the right phase commands underneath.
This chapter covers what each phase deploys, why the two state directories are separate, the deploy_bnk=false override that makes “cluster only” work, the cluster-outputs.json artefact written on success, and a worked example. The companion BNK-trial chapter, Chapter 10, covers roksbnkctl bnk up / bnk down for the trial layer.
What’s deployed where
The bundled HCL has roughly two halves. The cluster phase owns the durable, cluster-scoped resources:
- The ROKS cluster itself (VPC + subnets + worker pool)
- A transit gateway (so the test jumphost can reach cluster internals)
- The registry COS (Cloud Object Storage) instance — used by the BNK trial as its FAR image / license / schematic store
cert-manager(Helm release into the cluster)- The TGW jumphost VM (an Ubuntu VSI in the same VPC, used by
--on jumphost)
The trial phase owns the BNK-specific resources:
- F5 Lifecycle Operator (
flo) Helm release cne_instanceKubernetes manifest- BNK license + admin certs
- Various cluster-side bits: ServiceAccounts, RoleBindings, Secrets
Two-phase split: cluster up provisions the first list; roksbnkctl up (the trial) provisions the second.
┌─────────────────────────────────────────────────────────┐
│ cluster phase (durable, reused across many trials) │
│ ROKS cluster + VPC + transit gateway │
│ registry COS instance │
│ cert-manager (Helm) │
│ TGW jumphost │
├─────────────────────────────────────────────────────────┤
│ trial phase (one trial — destroyed by `roksbnkctl down`)│
│ flo (F5 Lifecycle Operator) │
│ cne_instance │
│ license / admin cert / SCC bindings │
└─────────────────────────────────────────────────────────┘
The split exists because ROKS clusters take 30-50 minutes to provision and roughly $0.30/hour to run. Re-creating the cluster every time you want to re-test a BNK trial is wasteful; reusing one cluster for many trials cuts iteration time from “an hour” to “a few minutes”.
The two state directories
To keep cluster state and trial state from tangling, roksbnkctl uses separate Terraform state directories:
~/.roksbnkctl/<workspace>/
state/ # BNK trial state — written by `roksbnkctl up/down`
terraform.tfstate
terraform.tfvars
state-cluster/ # cluster phase state — written by `roksbnkctl cluster up/down`
terraform.tfstate
cluster-phase-override.tfvars
Each phase’s commands read and write only their own state directory. Both phases use the same Terraform source (the bundled HCL) but with different effective tfvars — the trick is the deploy_bnk flag.
The deploy_bnk=false override
The bundled HCL has a top-level deploy_bnk boolean. When true, the BNK trial modules (flo, cne_instance, license) run; when false, they’re skipped and Terraform only provisions the cluster-phase resources.
roksbnkctl cluster up and roksbnkctl cluster down force deploy_bnk = false by writing a small auto-generated tfvars override into the cluster state directory:
# ~/.roksbnkctl/<workspace>/state-cluster/cluster-phase-override.tfvars
# Generated by roksbnkctl. Do not edit by hand.
# Cluster-phase override: BNK trial modules (flo / cne_instance /
# license) are skipped. cert-manager and the testing jumphost still run
# — they're cluster-shared singletons that belong with the cluster.
deploy_bnk = false
This file is layered onto the var-file chain after user-supplied --var-file flags so the override always wins. The user’s terraform.tfvars and --var-file <path> arguments still apply for everything else (region, RG, cluster name, worker count, …) — only deploy_bnk is forced.
roksbnkctl up doesn’t write this override file; its tfvars chain leaves deploy_bnk at the upstream default (true), so the trial modules run.
Reusing an existing VPC (multiple clusters in one VPC)
By default the cluster phase creates a prefix-named VPC for the cluster it builds. But you can point a new cluster at a VPC that already exists — so several clusters (each its own workspace) live in one shared VPC: the shape behind a shared licensing cluster or a common services network.
Decline “Create a new cluster VPC?” in init and it discovers the VPCs in the
chosen region and lets you pick one by number (since v1.26.0):
$ roksbnkctl -w app-b init
...
Create a new cluster VPC? [Y/n] n
→ Discovering existing VPCs...
Existing VPCs in us-south:
1) shared-services-vpc (available)
2) app-a-cluster-vpc (available)
Use an existing cluster VPC — pick a number (0 = none / create a new one): 1
Picking 0 (or a region with no VPCs) falls back to creating one — a cluster must
have a VPC. In config that is (the value is the VPC id, unlike the
transit-gateway adopt-by-name):
resources:
cluster_vpc:
create: false
existing: r006-6fe0b20a-... # the VPC id
which renders use_existing_cluster_vpc = true + existing_cluster_vpc_id. Two
things make this safe for more than one cluster in a VPC: the cluster’s subnets are
named per-cluster (<cluster>-subnet-zoneN) so they don’t collide, and per-zone
public gateways are reused when the VPC already has them (IBM allows exactly
one per zone per VPC). Make sure the VPC has free address-prefix space — each
cluster adds three /24 subnets.
Combine this with an adopted Transit Gateway: clusters that share a VPC share that VPC’s single connection to the gateway, so N clusters can reach each other over one VPC and one gateway.
Teardown order matters in a shared VPC. The cluster that created the VPC owns its per-zone public gateways (IBM allows only one gateway per zone per VPC, so the adopters attach to the owner’s). The owner must be destroyed last — tear the adopters down first.
roksbnkctlenforces this:cluster down/downrefuses to destroy a workspace whose VPC still holds another cluster’s subnets, naming them, rather than failing mid-destroy withThe VPC is in use. (Sincev1.26.1; subnets also attach their gateway inline now, so an adopter’s teardown no longer errors even if the owner’s gateway is already gone.)
cluster-outputs.json — the cluster identity record
When roksbnkctl cluster up apply succeeds, it writes a workspace-scoped JSON file describing the cluster. Most of it comes from Terraform outputs (cluster name, ID, region, RG, VPC, registry COS); the create-time settings — network_mode, and vpc_cidr when this tool created the VPC — come from the workspace config instead, because that is the only place they are knowable and the moment they become settled:
~/.roksbnkctl/<workspace>/cluster-outputs.json
Sample contents:
{
"cluster_name": "bnk-quickstart",
"cluster_id": "cre6h4l20jjsg4kvt3a0",
"region": "us-south",
"resource_group_id": "abc123...",
"vpc_id": "r006-...",
"schema_version": 2,
"network_mode": "single-nic",
"vpc_cidr": "10.242.0.0/16",
"registry_cos_crn": "crn:v1:bluemix:public:cloud-object-storage:global:a/...",
"registry_cos_name": "bnk-quickstart-cos-instance",
"master_url": "https://c106.us-south.containers.cloud.ibm.com:31415",
"openshift_version": "4.14_openshift",
"source": "cluster-up",
"recorded_at": "2026-05-08T14:22:08Z"
}
The source field discriminates between cluster-up (we created it) and cluster-register (we discovered an existing cluster — see Chapter 9). Subsequent commands read this file to learn the workspace’s cluster identity without hitting IBM APIs.
Reading this file from your own tooling
The file is a versioned contract, and the rules for reading it matter more than the field list, because every rule exists to keep a file written by one version readable by another.
schema_versionabsent means 1. Every record written before the field existed describes a real cluster and must stay readable; treating a missing value as0would invalidate all of them.network_modeabsent meanssingle-nic. That is what every cluster built before the field existed actually is.vpc_cidrabsent means unknown, not “no block was set.” It is recorded only when this tool created the VPC; on the adopt path the setting never applied, so recording the configured value would record something that was never true.node_interfacesis declared but written by nothing today. Multi-NIC ROKS has not shipped, so no code path can populate it honestly yet. Empty means unknown — never “this cluster has no extra interfaces”. Both readings are true of every cluster that exists right now, which is exactly what makes the wrong one easy to adopt and expensive to unwind later.- Unknown fields are ignored, deliberately. New keys are only ever added, so an older reader keeps working against a newer file.
For the create-time settings, this file — not config.yaml — is the authority.
See Chapter 8a for what happens
when the two disagree.
roksbnkctl cluster down deletes the file as part of its post-destroy cleanup. roksbnkctl cluster show pretty-prints it for human readers:
roksbnkctl cluster show
workspace: default
source: cluster-up
recorded_at: 2026-05-08T14:22:08Z
cluster_name: bnk-quickstart
cluster_id: cre6h4l20jjsg4kvt3a0
region: us-south
resource_group: abc123...
openshift: 4.14_openshift
master_url: https://c106.us-south.containers.cloud.ibm.com:31415
vpc_id: r006-...
registry_cos: bnk-quickstart-cos-instance
registry_cos_crn: crn:v1:bluemix:public:cloud-object-storage:global:a/...
Worked example: cluster up → kubectl get nodes → cluster down
The cluster-only flow, end to end:
Step 1 — roksbnkctl init
If you don’t have a workspace yet, initialise one. This is the same init flow as the trial path; the cluster commands reuse the workspace’s config.
roksbnkctl init
Step 2 — roksbnkctl cluster up --auto
Provisions the cluster phase only:
roksbnkctl cluster up --auto
Sample output (heavily abridged):
→ terraform plan (cluster phase: deploy_bnk=false forced)
→ Layering user tfvars from ~/.roksbnkctl/default/state-cluster/cluster-phase-override.tfvars (overrides config.yaml-derived values)
→ terraform init
→ terraform apply
module.roks_cluster.ibm_container_vpc_cluster.cluster: Creating...
module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [10m elapsed]
...
module.roks_cluster.ibm_container_vpc_cluster.cluster: Creation complete after 38m12s
module.cert_manager.helm_release.cert_manager: Creation complete after 2m11s
module.testing.tls_private_key.jumphost_shared_key: Creation complete after 0s
module.testing.ibm_is_instance.tgw_jumphost: Creation complete after 1m48s
Apply complete! Resources: 36 added, 0 changed, 0 destroyed.
✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json
✓ Wrote /home/you/.kube/config (chmod 0600)
✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`
Roughly 36 resources land — the cluster phase is about half the size of a full BNK trial. Time-to-ready is dominated by the ROKS cluster itself; everything else after the cluster comes up is fast.
Step 3 — verify the cluster works
The post-apply admin kubeconfig is fetched automatically (unless --no-kubeconfig). kubectl get nodes confirms reachability:
kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# 10.243.0.4 Ready master,worker 3m v1.28.6+5e1b9a1
# 10.243.64.4 Ready master,worker 3m v1.28.6+5e1b9a1
Or the same thing through the internalised verb:
roksbnkctl k get nodes
roksbnkctl status reports cluster identity + reachability:
roksbnkctl status
Workspace: default
Region: us-south
Cluster: bnk-quickstart (attach existing)
TF source: embedded@v1.0.0
Last apply: 2026-05-08 14:22:08 UTC (3m ago)
Kubeconfig: /home/you/.kube/config
Cluster: 2/2 nodes ready
Step 4 — (optional) deploy a BNK trial on top
Now that the cluster is up, roksbnkctl up deploys a BNK trial onto it. It reads cluster-outputs.json and reuses the cluster:
roksbnkctl up --auto
See Chapter 10 — Deploying BNK trials for the trial-phase walkthrough. You can run up / down many times against the same cluster — each cycle is ~5 minutes rather than the ~50 minutes of a fresh-cluster run.
Step 5 — roksbnkctl cluster down --auto
Tear down the cluster phase. cluster down is strictly scoped: it refuses with a hard error on any workspace whose trial state is non-empty, so an out-of-order destroy can’t accidentally orphan BNK resources. Destroy the trial first with roksbnkctl bnk down (or roksbnkctl down for both at once); see Chapter 11 for the full refusal catalogue.
roksbnkctl cluster down --auto
Sample output:
→ terraform destroy (cluster phase)
module.testing.ibm_is_instance.tgw_jumphost: Destroying...
module.cert_manager.helm_release.cert_manager: Destroying...
module.roks_cluster.ibm_container_vpc_cluster.cluster: Destroying...
module.roks_cluster.ibm_container_vpc_cluster.cluster: Still destroying... [5m elapsed]
module.roks_cluster.ibm_container_vpc_cluster.cluster: Destruction complete after 8m16s
Destroy complete! Resources: 36 destroyed.
Post-destroy, cluster-outputs.json is deleted. The workspace directory and its config.yaml survive — re-running cluster up against the same workspace re-creates the cluster with the same name and region.
Why split cluster from trial?
Two-phase is the default because the cost of conflating them is concrete. ROKS clusters take 30-50 minutes to provision and bill at roughly $0.30/hour; a BNK trial on top takes 5-10 minutes. Iterating on the trial — different flo versions, different cne_instance shapes, license bundle revisions — happens far more often than iterating on the cluster underneath. Splitting state means a bnk down / bnk up cycle is a five-minute round-trip instead of an hour.
Three scenarios this shape unlocks:
-
Many BNK trial iterations on one cluster. Run
cluster uponce, then loopbnk up/bnk downagainst the same cluster until you’ve covered all the trial permutations. Thencluster downonce when you’re finished. This is the headline win of the phase split — see Chapter 10 §“Worked example — iterating on a BNK trial”. -
Pre-provisioning for a workshop or demo. You want the cluster ready and warm before the demo starts; you’ll deploy the BNK trial live in front of the audience.
cluster upthe night before;bnk upduring the demo. -
Decoupling cluster lifecycle from trial lifecycle. A long-lived cluster used by multiple team members, where one person owns the cluster phase and others own the BNK trials. Cluster-phase outputs live in
cluster-outputs.json; trials read it. Each trial canbnk up/bnk downwithout affecting the cluster.
For workspaces that just want “create a cluster, deploy BNK on it, test, tear it all down”, the unscoped roksbnkctl up / roksbnkctl down are the right verbs — they’re shape-aware composites that drive the cluster + trial steps in the right order without you having to think about it.
Cross-references
- Chapter 9 — Registering an existing cluster — the alternative to
cluster upwhen you already have a ROKS cluster you wantroksbnkctlto manage. - Chapter 10 — Deploying BNK trials —
roksbnkctl upand thebnk up/bnk downcommand group; the dispatch matrix; iteration walkthrough. - Chapter 11 — Tearing down — phase-aware decision matrix and the refusal-message catalogue.
- Chapter 24 — Day-2 ops —
roksbnkctl k get/apply/logsfor working against the cluster after either phase.
The three-phase lifecycle (Cluster / BNK / Testing)
Transcripts in this chapter are illustrative. They show the shape of the output, not byte-exact captures. Resource counts and timings vary with upstream chart releases and IBM Cloud region behaviour.
As of this release, a roksbnkctl workspace is three independent phases
rather than two. The durable Cluster phase (the ROKS cluster + cluster
VPC + transit gateway + registry COS) sits underneath two independent
sibling phases that deploy on top of it: the BNK phase (cert-manager +
F5 Lifecycle Operator + CNE Instance + License — everything that talks to
the Kubernetes API) and the Testing phase (the jumphost rig — pure IBM
VPC, no Kubernetes).
┌──────────── 1. CLUSTER ────────────┐ (create OR reuse)
│ ROKS cluster + cluster VPC + TGW │ → cluster-outputs.json
│ + registry COS │
└──────────────────┬──────────────────┘
┌─────────┴─────────┐
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ 2. BNK (k8s) │ │ 3. TESTING (VPC) │
│ cert-manager, │ │ jumphosts (pure │
│ flo, cne, license│ │ IBM VPC, no k8s) │
│ needs: cluster │ │ needs: cluster │
│ (k8s API) │ │ VPC + TGW only │
└──────────────────┘ └──────────────────┘
└── parallel; independent up/down ──┘
Why three? BNK depends fully on the cluster (its resources live in the cluster and need the Kubernetes API). Testing depends on the cluster only for the network — the cluster VPC it attaches a jumphost to, and the transit gateway it bridges across regions. The two workloads are otherwise unrelated. Splitting them means:
- You can tear down the BNK trial and keep the jumphosts for the next
iteration (
roksbnkctl bnk downleaves Testing alone) — and the inverse (roksbnkctl testing downleaves BNK alone). - The two unrelated workloads deploy in parallel after the cluster is
ready, shaving wall-clock off a fresh
up.
testingis nottest. This is the one naming collision worth internalising up front:
roksbnkctl testing up/down— a provisioning phase. It creates and destroys the jumphost infrastructure (the TGW jumphost, the per-AZ cluster jumphosts, the optional client VPC). Parallel toclusterandbnk. Writesstate-testing/.roksbnkctl test— runs validation probes (connectivity, DNS, throughput) against an already-deployed environment. Provisions nothing. See Chapter 20.roksbnkctl test hosts— manages the list of hosts thetestprobes target. Still no provisioning.Mnemonic:
testingbuilds the rig;testruns the checks.
The three state directories
Each phase owns its own Terraform state directory so the phases never tangle:
~/.roksbnkctl/<workspace>/
state-cluster/ # Cluster phase — `cluster up/down`
terraform.tfstate
cluster-phase-override.tfvars
state/ # BNK phase — `bnk up/down` (and `up`/`down`/`plan`/`apply`)
terraform.tfstate
bnk-phase-override.tfvars
state-testing/ # Testing phase — `testing up/down`
terraform.tfstate
testing-phase-override.tfvars
cluster-outputs.json # the cluster's identity — the handoff to BNK + Testing
The BNK phase keeps the original state/ directory it has always used
(this is deliberate — see §“Migration”
below); the jumphosts now live in the new state-testing/. Naming is
slightly asymmetric (state/ for BNK vs the explicit state-cluster/ /
state-testing/) for exactly one reason: keeping state/ as BNK means
existing workspaces don’t have to migrate their BNK resources, only evict
the jumphosts.
cluster-outputs.json — the handoff both siblings read
The Cluster phase writes cluster-outputs.json on success (or
cluster register writes it for a pre-existing cluster). Both BNK and
Testing read it to learn the cluster’s identity without re-specifying it:
- BNK reads the cluster id/name (to attach to the existing cluster), the VPC id, and the cluster’s network mode and contract schema version — which it checks against the BNK release it is about to install, before it plans anything (see Chapter 28).
- Testing reads the cluster VPC id (the jumphost attaches to it), the cluster id/name, and the transit gateway name (the Testing module looks the gateway up by name to bridge the client VPC to the cluster).
The transit-gateway name is recorded alongside the id specifically so the Testing phase can run standalone against a registered cluster.
Which side wins
For most settings config.yaml is the input and the record is just an echo of
the result. For create-time settings the relationship inverts: the record
describes what the cluster is, and the config is a request that has to agree
with it. A cluster is never converted in place, so a contradiction is either a
mistake or a request for a different cluster.
That gives three cases, and the middle one is the one worth knowing:
cluster.network_mode | vs the record | result |
|---|---|---|
| set, and matches | agrees | proceeds |
| unset | cannot disagree | defers to the record |
| set, and differs | contradicts | refused, before planning |
Silence is not an assertion. An unset network_mode means the config has no
opinion, not that it insists on single-nic — so it defers to whatever the
cluster actually is. This matters because a config.yaml is not always
hand-written and durable: the BNK Forge modules
regenerate one per step from a curated environment, so a mode set by the step
that created the cluster is absent by the time the step that installs
BNK runs. Reading that absence as a demand for single-nic would refuse a
correct deployment at its second step, over a contradiction nobody expressed.
The refusal in the third row is what stops terraform from planning a replacement of a running cluster — which it will do without complaint, rendered as output that reads like an update.
Parallel up: Cluster first, then BNK || Testing
A fresh roksbnkctl up provisions the cluster serially first (both
siblings depend on it), then launches BNK and Testing concurrently:
roksbnkctl up --auto
→ Cluster phase (serial; both BNK and Testing depend on it)
module.roks_cluster.ibm_container_vpc_cluster.cluster: Creating...
...
Apply complete! Resources: 36 added.
✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json
→ BNK || Testing (parallel)
[bnk] → terraform apply
[testing] → terraform apply
[testing] module.testing.ibm_is_instance.tgw_jumphost: Creating...
[bnk] module.cert_manager.helm_release.cert_manager: Creating...
[testing] module.testing.tls_private_key.jumphost_shared_key: Creation complete
[bnk] module.flo.helm_release.flo: Creating...
[testing] Apply complete! Resources: 9 added.
[bnk] Apply complete! Resources: 32 added.
✓ Auto-registered target jumphost (169.45.91.177)
The two parallel phases’ output is line-prefixed ([bnk] / [testing])
so you can follow each independently. The Cluster phase, being serial, prints
unprefixed.
Without --auto, up plans both phases first (sequentially, so each diff
is cleanly attributed), then asks a separate confirmation for each —
Apply BNK plan? [y/N] and Apply Testing plan? [y/N]. Two concurrent
applies can’t each own an interactive prompt on one terminal, so the approval
is taken up front; the approved phases then apply in parallel. Approving
only one (e.g. y to BNK, N to Testing) brings up just that phase — handy
when you want to redeploy the BNK trial without touching the jumphosts. A
phase whose plan shows no changes is skipped without prompting.
To review a change out-of-band before applying — save the full plan to a file, get
sign-off, then apply exactly that plan — use the dissociated roksbnkctl plan --out <file> / roksbnkctl apply --plan <file> flow (see Chapter 10 — Deploying BNK
trials, §“Reviewing a plan before applying”).
Why Cluster-completes-first rather than starting Testing the moment the cluster VPC exists? The ROKS cluster create dominates wall-clock (~30-50 min); Testing is a handful of VSIs (minutes). Starting Testing against a half-built cluster invites data-source races for a few minutes’ savings. Cluster-first is simpler and correct.
Per-phase up / down
Each phase has its own command pair, all sharing the same flag model
(--auto, --var-file, …):
| Phase | Up | Down | State |
|---|---|---|---|
| Cluster | roksbnkctl cluster up | roksbnkctl cluster down | state-cluster/ |
| BNK | roksbnkctl bnk up | roksbnkctl bnk down | state/ |
| Testing | roksbnkctl testing up | roksbnkctl testing down | state-testing/ |
bnk up and testing up each bootstrap the Cluster phase first if the
workspace has no cluster yet (with a confirmation prompt — the cluster take
~30 min). Against an existing cluster they go straight to their own apply.
bnk down leaves the jumphosts (and the inverse)
The headline capability of the split: the two siblings tear down independently.
# Iterate on BNK while keeping the jumphost rig warm:
roksbnkctl bnk down # destroys cert-manager/flo/cne/license; Testing untouched
roksbnkctl bnk up # redeploy BNK against the same cluster + same jumphosts
✓ BNK phase destroyed. Cluster phase state-cluster/ and Testing phase
state-testing/ are intact. Run `roksbnkctl bnk up` to redeploy.
And the inverse — destroy the jumphost rig but keep BNK running:
roksbnkctl testing down # destroys the jumphosts; cluster + BNK untouched
Because each phase owns a separate state directory, bnk down physically
cannot touch state-testing/, and testing down cannot touch state/.
Jumphost sizing, and why it is worth checking once
testing up picks the jumphost instance profile itself unless
resources.testing_jumphost_profile names one. The intent is the smallest
profile meeting testing_min_vcpu_count (4) and testing_min_memory_gb (8) —
these VMs run curl, iperf3 and a small echo server, nothing more.
Through v1.63.0 it picked the largest instead. The selection sorted profile
names, and sort() is lexicographic, so among the eligible bx2 profiles
"bx2-128x512" < "bx2-16x64" < "bx2-2x8" < "bx2-32x128" < "bx2-4x16"
and the first element was a 128 vCPU / 512 GB machine — roughly $6/hour each, about $440/day for a three-zone cluster jumphost set. Fixed in v1.64.0.
If you already have oversized jumphosts
Check before upgrading:
ibmcloud is instances --output json | jq -r '.[] | select(.name|test("jumphost")) | "\(.name) \(.profile.name)"'
If they are bx2-128x512 (or anything larger than you expect), recreate them
rather than resizing in place:
roksbnkctl testing down
roksbnkctl testing up
From v1.64.0 new jumphosts pin total_volume_bandwidth to 1000 Mbps
(testing_jumphost_total_volume_bandwidth), which keeps a later downsize
possible. Jumphosts created before that release do not carry the pin, and for
them resizing in place is the tempting move and it does not work. total_volume_bandwidth
is set by the API from the profile at creation and carried in state, so shrinking
the profile is rejected —
instance's total volume bandwidth 20000Mbps (specified by total_volume_bandwidth)
must not be greater than max volume bandwidth 3500Mbps
— and the apply stops the VMs before it fails, leaving them stopped on the
old profile. If you are already in that state, recover out of band and then
re-run testing up, which will plan no further change:
ibmcloud is instance-update <vm> --profile bx2-2x8 --total-volume-bandwidth 1000
ibmcloud is instance-start <vm>
Reuse an existing cluster
If you already have a ROKS cluster (yours or a teammate’s), skip the Cluster phase entirely — register it, then deploy BNK and Testing on top:
roksbnkctl cluster register existing-bnk-cluster
roksbnkctl up --auto # Cluster phase skipped; BNK || Testing deploy on the registered cluster
# or per-phase:
roksbnkctl bnk up --auto
roksbnkctl testing up --auto
cluster down is a no-op there — the cluster wasn’t yours to destroy.
Teardown ordering and the cluster down guard
Destroy in reverse of create: the siblings first (in parallel, they’re independent), then the cluster.
roksbnkctl down --auto
This will destroy the BNK, Testing, AND Cluster phases for workspace "default".
Continue? [y/N]: y
→ BNK || Testing (parallel destroy)
[bnk] Destroy complete! Resources: 32 destroyed.
[testing] Destroy complete! Resources: 9 destroyed.
→ Cluster phase (destroy)
Destroy complete! Resources: 36 destroyed.
The bare down takes one composite confirmation naming every present
phase, then runs the parallel sibling-destroy followed by the cluster
destroy — you can’t accidentally confirm one phase and miss another.
roksbnkctl cluster down refuses while BNK or Testing state still has
resources — they reference the cluster VPC/TGW, so destroying the cluster
out from under them would orphan resources:
$ roksbnkctl cluster down
BNK and Testing state exist in this workspace; run `roksbnkctl bnk down` and
`roksbnkctl testing down` first (or `roksbnkctl down` to tear down all phases)
This is a correctness guard, not a prompt — --auto does not bypass it.
cluster-outputs.json is deleted only on cluster down (it’s the cluster’s
identity); bnk down and testing down leave it in place.
Migration from the two-phase layout
Workspaces created before the three-phase split have their jumphosts living
in the BNK state (state/, alongside the BNK modules — that’s how the
old two-phase “trial” worked). The three-phase split needs those jumphosts
in state-testing/ instead. Three cases:
-
Fresh / empty workspaces — nothing to migrate. The first
uplays out all three state directories cleanly. -
Two-phase split workspaces (the common case) — the jumphosts need to move out of
state/intostate-testing/without destroying the live jumphosts. The recommended path preserves them:roksbnkctl testing migrate # terraform state mv: jumphosts → state-testing/This
terraform state mvs everymodule.testing.*resource (including the shared SSH key) fromstate/into a freshstate-testing/, with no cloud churn — the jumphosts keep their IPs and yourknown_hostsstays valid. A subsequentbnk upreconciles the now-jumphost-free BNK state (it plans no jumphost create), andtesting upadopts the moved resources (in-place, not create).If a workspace is in an awkward partial-apply state, the bulletproof fallback is a full
roksbnkctl downthenroksbnkctl up, which re-creates everything in the three-state layout. This does churn the jumphosts (new IPs,known_hostsreset), so thestate mvpath is preferred whenever it’s available.
roksbnkctl nudges you when it detects jumphosts still living in the BNK
state with an empty state-testing/ — it won’t silently rewrite your state.
The optional Gateway phase (gateway up / down)
The BNK phase brings up a licensed, running control plane and now also creates
the cloud-network-mapping ConfigMap and the external/internal VLANs that TMM
needs to program its data plane. The rest of the install guide’s
“Configuration” section — the Gateway API objects, egress SnatPool/Egress, the
per-zone static routes, and the cluster security-group VXLAN rule — is a
separate, optional phase with its own state (state-gateway/):
roksbnkctl up # Cluster → BNK ∥ Testing (Gateway NOT included)
# … verify BNK is healthy: TMM pods Ready, CNEInstance Available …
roksbnkctl gateway up # apply the data-plane config
roksbnkctl gateway down # remove it, leaving cluster/BNK/testing intact
It is never run by the composite up/down — the Gateway config needs a
healthy BNK (its F5SPK* CRDs ship with the BNK manifest, and TMM must be up),
so it is strictly opt-in. It reuses the existing cluster via
cluster-outputs.json (like the Testing phase), forces every other phase’s
creation off in its override, and manages only its own resources. Defaults all
come from the BNK 2.3 install guide; override the HTTPRoute backend, client
subnets, egress mode (snatpool/automap/both) and VXLAN port via the
config.yaml gateway block. cluster down refuses while the Gateway phase
has resources, so tear it down first (gateway down).
The VXLAN security-group rule, and what it opens
The gateway phase adds one inbound rule to the cluster’s own worker security
group (kube-<cluster-id>), permitting the egress VXLAN port (UDP 6789 by
default, gateway.vxlan_port).
This rule is required, not hardening. TMM answers a remote node’s VXLAN from its external-VLAN self-IP, and the workers’ security group does not otherwise admit that source — so without the rule, egress from any node that is not itself running a TMM fails.
Through v1.63.0 the rule permitted 0.0.0.0/0. From v1.64.0 it is scoped to
the per-zone external-VLAN CIDRs (resources/bnk zone config
ext_vlan_cidr), one rule per CIDR, because those are the only addresses a TMM
self-IP can have. That matters because this rule sits on a security group shared
by everything else running on those workers.
Upgrading replaces the rule rather than editing it. The resource moved from
counttofor_each, so terraform removes the single open rule and creates the scoped ones; ordering is not guaranteed, so rungateway upwhen a few seconds of egress disruption is acceptable.
If every zone’s ext_vlan_cidr is empty, gateway up now fails at plan time
rather than silently creating no rule and leaving egress broken with nothing
naming the cause.
Cross-references
- Chapter 8 — The cluster phase — the Cluster phase in depth.
- Chapter 9 — Registering an existing cluster — the reuse-existing-cluster path.
- Chapter 10 — Deploying BNK trials — the BNK
phase (
bnk up/down) in depth. - Chapter 11 — Tearing down — the full destroy decision tree and refusal catalogue.
- Chapter 20 — Connectivity testing — what
roksbnkctl test(nottesting!) actually runs.
Registering an existing cluster
Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the
Workspacestruct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.
roksbnkctl cluster register <name> wires roksbnkctl up to a ROKS cluster that already exists in your IBM Cloud account — one you didn’t provision via cluster up. After a successful register, the workspace behaves exactly as if you’d done cluster up: roksbnkctl up deploys BNK trials onto the registered cluster, roksbnkctl down tears those trials down, roksbnkctl status reports the cluster’s identity, and so on.
This chapter covers when registration is the right answer, what input is required vs auto-discovered, the COS naming convention, the cluster-outputs.json write, and a worked example.
roksbnkctl initcan register for you. Answering no to Create a new ROKS cluster? in theinitinterview lists the account’s running OpenShift clusters and registers the one you pick — it performs the samecluster-outputs.jsonwrite described here, inline. Usecluster register <name>directly when you want to (re)register without re-running the full interview, or when scripting.
cluster upattaches whencluster.create=false. You don’t need a separatecluster registercall: if the workspace’scluster.createisfalse,roksbnkctl cluster upattaches to the existing cluster named incluster.name(the same discovery +cluster-outputs.jsonwrite) instead of running the terraform create. So one command — and one toggle in an automation/CI form — covers both “make a new cluster” and “use the existing one”. Set it non-interactively withROKSBNKCTL_CLUSTER_CREATE/ROKSBNKCTL_CLUSTER_NAME(see Chapter 7a).
When to use this
cluster register is the answer when all of these are true:
- A ROKS cluster already exists in the IBM Cloud account.
- You have IAM access to the cluster’s VPC + container service.
- You want
roksbnkctlto deploy BNK trials onto that cluster. - You don’t want
roksbnkctlto own the cluster’s lifecycle (it shouldn’t beterraform destroy-able from your workstation).
Common scenarios:
-
Your team operates the ROKS cluster centrally. A platform team provisioned the cluster via their own Terraform / Pulumi / IBM Cloud Schematics; you just want to deploy BNK trials onto it. Register it; deploy trials; tear them back down. The cluster itself stays under the platform team’s ownership.
-
You’re attaching to an existing demo cluster. A workshop hosts a shared cluster that participants attach to. Each participant registers it in their own workspace and deploys their own trial — trials are isolated by namespace under the same cluster.
-
You provisioned the cluster manually for testing. You created a one-off cluster via
ibmcloud ks cluster create vpc-gen2 ...and want to move forward withroksbnkctlrather than re-creating it.
If none of those apply — i.e. you want roksbnkctl to own cluster lifecycle end-to-end — use cluster up instead. Register and cluster up are mutually exclusive per workspace; the second one wins.
Required input vs auto-discovery
cluster register takes one positional argument (the cluster name or ID) and one optional flag (--registry-cos-name).
roksbnkctl cluster register <cluster-name-or-id> [--registry-cos-name <cos-instance-name>]
Everything else is auto-discovered via the IBM SDK:
| Field | Source |
|---|---|
cluster_id | ibmcloud ks cluster get <name> (resolved by name → ID) |
region | from the cluster lookup |
resource_group_id | from the cluster lookup |
vpc_id | from the cluster’s provider.vpcs[0].id |
master_url | from the cluster lookup |
openshift_version | from the cluster’s masterKubeVersion |
registry_cos_crn | discovered via the registry COS instance lookup (see below) |
The cluster lookup goes through the same container-service endpoint ibmcloud ks cluster get uses — no host ibmcloud install required. If the named cluster doesn’t exist in the account, the call returns a clear no cluster named <foo> error rather than a 404 stack trace.
A vpc-gen2 cluster is required. Classic infrastructure clusters return successfully but their vpc_id is empty, and cluster register refuses to write a record without one:
Error: cluster "old-classic" has no VPC — roksbnkctl only supports vpc-gen2 clusters
The COS naming convention
roksbnkctl up needs a Cloud Object Storage instance to act as the registry for FAR images, JWT licenses, and schematic state. cluster register verifies that this COS instance exists at registration time so a later up doesn’t fail mid-apply with a missing-instance error.
Default convention
The bundled HCL falls back to <cluster-name>-cos if the user’s tfvars don’t override roks_cos_instance_name. So cluster register defaults to looking up <cluster-name>-cos:
# Cluster name: "canada-roks" → expects COS instance "canada-roks-cos"
roksbnkctl cluster register canada-roks
Override with --registry-cos-name
If your team set roks_cos_instance_name to something else in their tfvars (or named the COS instance via the IBM Cloud console with a different convention), pass --registry-cos-name <name>:
roksbnkctl cluster register canada-roks \
--registry-cos-name canada-roks-bnk-registry
The instance name is case-sensitive and must match exactly — Canada-ROKS-COS and canada-roks-cos are different instances.
What if the COS doesn’t exist yet?
cluster register errors out:
Error: registry COS instance "canada-roks-cos" not found in account: ...
Either run `roksbnkctl cluster up` to create it, or pass --registry-cos-name <name>
if your tfvars uses a different roks_cos_instance_name
You have two options:
-
Create the COS instance in the IBM Cloud console with the conventional name (
<cluster>-cos), then re-run register. The instance can be empty —roksbnkctl upwill populate it with the bucket structure it needs on its first apply. -
Use a different name that already exists in the account, via
--registry-cos-name <name>.
Either way, cluster register won’t write cluster-outputs.json until both the cluster and its registry COS instance exist.
The cluster-outputs.json write
On success, cluster register writes ~/.roksbnkctl/<workspace>/cluster-outputs.json — the same file cluster up writes. The contents look nearly identical; the differences are source, and what is deliberately left empty:
{
"cluster_name": "canada-roks",
"cluster_id": "cre6h4l20jjsg4kvt3a0",
"region": "ca-tor",
"resource_group_id": "abc123...",
"vpc_id": "r038-...",
"schema_version": 2,
"network_mode": "single-nic",
"registry_cos_crn": "crn:v1:bluemix:public:cloud-object-storage:global:a/...",
"registry_cos_name": "canada-roks-cos",
"master_url": "https://c106.ca-tor.containers.cloud.ibm.com:31415",
"openshift_version": "4.14_openshift",
"source": "cluster-register",
"recorded_at": "2026-05-08T14:22:08Z"
}
The source field is cluster-register (vs cluster-up for self-provisioned clusters). Downstream commands that care about provenance — for example, a future roksbnkctl cluster down would refuse to destroy a cluster-register-sourced cluster — read this field. Subnet IDs (subnet_ids) and transit gateway ID (transit_gateway_id) are left blank for registered clusters; the bundled HCL doesn’t need them when roksbnkctl up runs against a pre-existing cluster.
vpc_cidr is also left empty here, and that is deliberate. It records the
block this tool used to create a VPC. On the adopt path the setting never
applied, so writing the configured value would record something that was never
true of the VPC.
That has one consequence worth knowing: because the create-time warning for
cluster.vpc_cidr fires only on a disagreement with a recorded value, a
workspace built by cluster register will never emit it — there is nothing
recorded to disagree with. Silence there means “not knowable”, not “your value
is fine”. See Chapter 26.
network_mode, by contrast, is recorded — from the config, since an adopted
cluster’s mode cannot yet be detected from the IBM API. On a single-NIC cluster
(every cluster today) the default is correct and nothing needs saying. When
multi-NIC ROKS ships, adopting one will mean setting cluster.network_mode to
match what the cluster is, not what you want — the opposite of how the same
field reads on cluster up.
Worked example: register canada-roks
The full flow for attaching to a hypothetical canada-roks cluster.
Step 1 — create or pick a workspace
roksbnkctl ws new canada
roksbnkctl init -w canada
# (interactive — fill in region as ca-tor; cluster.name = canada-roks)
You can also run cluster register against the current workspace; the -w is just for clarity.
Step 2 — cluster register
roksbnkctl -w canada cluster register canada-roks
Sample output:
→ Looking up cluster "canada-roks"
✓ Cluster canada-roks (cre6h4l20jjsg4kvt3a0) — state: normal, masters: 4.14_openshift
✓ VPC r038-... (resource group prod-rg)
→ Verifying registry COS instance "canada-roks-cos"
✓ COS instance canada-roks-cos (abc-123-def-...)
✓ Wrote ~/.roksbnkctl/canada/cluster-outputs.json
If the COS naming was non-conventional:
roksbnkctl -w canada cluster register canada-roks \
--registry-cos-name canada-bnk-registry
Step 3 — verify with cluster show
roksbnkctl -w canada cluster show
workspace: canada
source: cluster-register
recorded_at: 2026-05-08T14:22:08Z
cluster_name: canada-roks
cluster_id: cre6h4l20jjsg4kvt3a0
region: ca-tor
resource_group: abc123...
openshift: 4.14_openshift
master_url: https://c106.ca-tor.containers.cloud.ibm.com:31415
vpc_id: r038-...
registry_cos: canada-roks-cos
registry_cos_crn: crn:v1:bluemix:public:cloud-object-storage:global:a/...
Step 4 — fetch the kubeconfig
cluster register does not automatically download the kubeconfig — it’s a metadata-only operation. Grab it explicitly:
roksbnkctl -w canada kubeconfig --download
# → Fetching admin kubeconfig for "canada-roks"
# ✓ Wrote /home/you/.kube/config (12345 bytes)
Step 5 — use the cluster as if you’d done cluster up
From here, the workflow is identical to a self-provisioned cluster:
# Verify reachability
roksbnkctl -w canada k get nodes
# Deploy a BNK trial onto it
roksbnkctl -w canada up --auto
# Tear the trial back down (cluster survives)
roksbnkctl -w canada down --auto
roksbnkctl up reads cluster-outputs.json and uses the cluster identity directly — no need to re-state cluster name/region/RG in the trial’s tfvars.
When register isn’t enough
Some scenarios where cluster register won’t get you over the line:
- The cluster is in a different IBM Cloud account. API keys are account-scoped; you’d need a key for the cluster’s account.
cluster registerdoesn’t cross account boundaries. - The cluster is private (no public master endpoint).
roksbnkctl upneeds to apply Helm charts and Kubernetes manifests against the master. If the master is only reachable from inside a VPN, route the apply through--on jumphostor wait for the SSH execution backend in Sprint 4. - The cluster is a classic-infrastructure ROKS (not vpc-gen2). Registration refuses; classic clusters aren’t supported.
- The cluster’s worker pool is too small. BNK trials need at least 2 workers with adequate CPU/memory. The upstream HCL provisions appropriately-sized workers; an existing cluster might not.
For the first three, the cluster isn’t a candidate. For the last one, the apply may run but flo / cne_instance will fail to schedule — scale the worker pool first.
Re-registering and unregistering
To re-register with new data (e.g. you renamed the COS instance, or the master URL changed), just run cluster register again — it overwrites cluster-outputs.json in place.
To unregister without destroying anything, delete the file directly:
rm ~/.roksbnkctl/canada/cluster-outputs.json
The workspace’s config.yaml and state/ survive; only the cluster identity record is removed. The next roksbnkctl up will fail with workspace has no cluster-outputs.json until you either re-register or run cluster up.
There’s deliberately no roksbnkctl cluster unregister command. Deleting the JSON is a single-file operation that doesn’t deserve its own subcommand, and the absence of one nudges users toward “destroy the trial first, then deal with the cluster identity” rather than “unregister without thinking about the consequences”.
Cross-references
- Chapter 8 — The cluster phase — the alternative when you want
roksbnkctlto provision the cluster. - Chapter 10 — Deploying BNK trials — what
roksbnkctl updoes on top of a registered (orcluster up’d) cluster. - Chapter 25 — COS supply chain management — the COS instance and bucket layout that
--registry-cos-namepoints at. - Chapter 10a — Air-gapped install — mirror BNK into your own registry; a registered cluster installs from it exactly like a created one.
- Chapter 10c — Licensing with the F5 License Proxy — a registered cluster is a first-class base for the proxy (Flow B), and both clusters in the shared-licensing topology (Flow C) can be registered rather than built.
Sharing a Transit Gateway across clusters
A Transit Gateway (TGW) routes between VPCs. By default roksbnkctl cluster up
creates a prefix-named gateway for the cluster it builds. But you often want the
opposite: one gateway that several clusters share, so they can reach each
other — the shape behind a shared licensing cluster,
a central services VPC, or cross-cluster testing.
roksbnkctl attaches a cluster’s VPC to an existing Transit Gateway — by name or by id, at create time or after — as its own small phase. Each workspace owns its own connection, so N clusters can point at the one gateway.
First: give each cluster VPC its own address block
Decide this before cluster up, because it cannot be changed afterwards.
A Transit Gateway routes on VPC address prefixes. Two attached VPCs claiming the same block make routing ambiguous, and the gateway resolves that by silently dropping traffic for one of them. Nothing logs an error. What you see instead is intermittent image-pull timeouts against a mirror while every security group and network ACL in the path plainly allows the traffic — which is why this is worth a minute up front.
It is easy to hit by accident. IBM’s default address prefix management is auto,
which assigns every VPC in a region the same three per-zone prefixes:
10.241.0.0/18 10.241.64.0/18 10.241.128.0/18
So a second roksbnkctl-created cluster joining a shared gateway collides by construction, not by bad luck. Give each one a distinct block:
cluster:
create: true
name: acme-eu-roks
vpc_cidr: 10.242.0.0/16 # this cluster's own block
or ROKSBNKCTL_CLUSTER_VPC_CIDR=10.242.0.0/16 for CI. The block is split into
three per-zone prefixes (a /16 becomes three /18s), so /18 is the smallest
usable value. Leaving it blank keeps IBM’s auto assignment, and the default
10.241.0.0/16 produces byte-identical prefixes to what auto gives today — so
setting it explicitly on a first cluster changes no addresses.
A worked allocation for a shared gateway:
| VPC | vpc_cidr | Per-zone prefixes |
|---|---|---|
| services (Harbor, FLP) | existing, 10.243.0.0/24 | — |
| first cluster | 10.241.0.0/16 | 10.241.0.0/18, .64.0/18, .128.0/18 |
| second cluster | 10.242.0.0/16 | 10.242.0.0/18, .64.0/18, .128.0/18 |
roksbnkctl checks this for you. cluster up refuses before terraform creates
anything when the VPC it is about to build would overlap a VPC already on the
gateway, and tgw connect refuses before attaching an existing VPC that would
overlap — each naming the other VPC and the colliding prefixes. If the check
cannot reach the API it warns and continues rather than blocking the build.
Already built two overlapping clusters? The prefixes of a live VPC can’t be
edited — moving a subnet’s CIDR replaces the subnet, which destroys the cluster on
it. Either keep only one of them attached at a time (tgw disconnect on the
other; the cluster itself survives), or rebuild one with vpc_cidr set.
Attach when creating (the interview)
Decline to create a gateway, and init discovers the transit gateways already
in your account and lets you pick one by number — name, location, and status —
so you don’t have to remember a name or id (since v1.25.0):
$ roksbnkctl -w prod init
...
Create Transit Gateway? [Y/n] n
→ Discovering existing transit gateways...
Existing transit gateways in this account:
1) shared-tgw (us-south, available)
2) prod-tgw-eu (eu-de, available)
Attach an existing Transit Gateway — pick a number (0 = none / attach later): 1
Picking 0 (or if the account has none) leaves the cluster unattached, to connect
later with tgw connect; if the discovery call fails it falls back to a free-text
name/ID prompt.
cluster up (or cluster register) then attaches the cluster’s VPC to
shared-tgw automatically — no extra step. In config that is:
resources:
transit_gateway:
create: false
existing: shared-tgw # a NAME or an ID — either is resolved
or, for CI, the environment variable ROKSBNKCTL_TRANSIT_GATEWAY_NAME=shared-tgw
(it too accepts a name or an id).
Attach (or detach) after the fact
tgw connect works on a cluster that already exists — one roksbnkctl built, or
one you registered — because it reads the cluster’s VPC from
cluster-outputs.json, not from any terraform state:
roksbnkctl -w prod tgw connect shared-tgw # by name
roksbnkctl -w prod tgw connect 2f28a749-fcc0-… # or by id
Run it in several workspaces with the same gateway and each cluster gets its own connection to it:
roksbnkctl -w app-a tgw connect shared-tgw
roksbnkctl -w app-b tgw connect shared-tgw # same gateway, second cluster
Detach removes only this cluster’s connection — the gateway and every other cluster’s connection stay:
roksbnkctl -w prod tgw disconnect
Status — the connected state, id, and name
tgw status shows the gateway identity and the live connection state, queried
from IBM (not just what was recorded at apply time):
$ roksbnkctl -w prod tgw status
transit_gateway_id: 2f28a749-fcc0-43d7-97dd-403a6b00b56f
transit_gateway_name: shared-tgw
connection_name: prod
connection_id: a1b2c3d4-…
cluster_vpc: r010-072ad88a-…
connection_state: attached
connection_state is the live attachment: attached, pending, or detached
(no connection for this VPC on the gateway). roksbnkctl -w prod cluster config
shows the same block alongside the rest of the cluster identity.
How it works, and what to know
- It’s its own phase (
state-tgw/), separate from cluster / BNK / testing / gateway / FLP. Sotgw connect/disconnectnever touch anything else, and the connection survives abnk downor a re-applied trial. - Name or id is resolved against your account’s gateway list, so an ambiguous name (two gateways sharing it) is an error, not an arbitrary pick — pass the id.
- One VPC, one connection per gateway. Clusters that share a VPC (rare) can’t each hold a separate connection to the same gateway; clusters in distinct VPCs each get their own, which is the normal shared-gateway case.
- A global gateway connects VPCs in any region; a local one only its own
region.
tgw connectattaches to whatever gateway you name — make sure it can reach the region your cluster VPC is in. cluster downrefuses while a connection exists, so the cluster isn’t destroyed out from under its own TGW attachment — runtgw disconnectfirst.roksbnkctl down(the composite) auto-detaches for you (sincev1.26.0): it removes this cluster’s connection after the BNK/Testing teardown and before the cluster, because the connection pins the VPC’s CRN and the VPC delete would otherwise fail (VPC still has an attached transit gateway connection). Only this cluster’s connection is removed — the shared gateway and every other cluster’s connection stay.
Deploying BNK trials on top
roksbnkctl up deploys a BNK trial — F5’s Lifecycle Operator, the CNE Instance, license bundles, and the cluster-side glue that makes them work — onto a ROKS cluster that already exists. “Already exists” means either provisioned by cluster up or registered from a pre-existing cluster.
For workspaces where the cluster and the trial are managed as separate phases (the default — see Chapter 8), the trial layer also gets its own command pair: roksbnkctl bnk up / bnk down. bnk down tears down only the trial; the cluster keeps running, so the next iteration starts in 5-10 minutes instead of an hour. The bnk group is documented in §“The bnk up / bnk down command group” below.
This chapter is the deeper-than-quick-start view of up: what each module does, the ~77-resource shape of a clean apply, the token-rotation observation when you re-run up against an existing cluster, how to read the Terraform plan output, and how the bnk group + the shape-aware composite up / down fit together.
Chapter 7 — Quick start shows the happy path end-to-end with sample output. This chapter goes deeper.
What “deploying BNK” means
A BNK trial is a deliberately small set of Kubernetes resources that share state with a cluster-shared cert-manager and a cluster-scoped registry COS. The components that roksbnkctl up is responsible for landing:
| Component | What it is | Module in the bundled HCL |
|---|---|---|
flo | F5 Lifecycle Operator — the controller that watches CNE Instance CRs and reconciles them into running BIG-IP Next pods | module.flo (helm_release) |
cne_instance | The CR that declares “I want a BIG-IP Next data plane here” — drives flo to provision the TMM pods | module.cne_instance (kubectl_manifest + wait_for) |
license | The License CR (JWT + operation mode) that gates BNK’s runtime — sourced from the registry COS | module.license (kubectl_manifest + wait_for) |
cluster-side bits | Namespaces, Secrets, ServiceAccounts, RoleBindings, SCC bindings, NADs, the cert-manager issuer chain | kubernetes_namespace/kubernetes_secret + kubectl_manifest, across the modules above |
up does not own the cluster, cert-manager’s chart, the registry COS, or the jumphost — those are cluster-phase resources. See Chapter 8 for the split.
The BNK phase is terraform-native. Chart installs are
helm_release(wait = true); the custom resources arekubectl_manifest(from thealekc/kubectlprovider) withwait_forblocks that watch the resource’s real.status; namespaces and Secrets are thekubernetesprovider. The CRs are real Terraform state —plandiffs them,destroydeletes them (finalizer-aware), and drift is detected. See §“The terraform-native deployment model” below.
The 77-resource shape
A clean roksbnkctl up against a fresh cluster lands roughly 77 resources when the cluster phase is bundled in (i.e. cluster up and up were one combined run). Against a pre-existing cluster (cluster up then up), the trial-only count is smaller — roughly the difference, ~41 resources.
The number isn’t load-bearing; it shifts a few resources up or down between upstream chart releases as the charts add/remove Secrets and kubectl_manifest CRs. Treat “77” as a sanity-check tag, not a contract.
A representative breakdown:
Cluster phase (~36 resources, owned by `cluster up`)
ROKS cluster + worker pools ~5
VPC + subnets + security groups ~6
Transit gateway + connections ~4
Registry COS instance + bucket ~3
cert-manager Helm release ~2
TGW jumphost VSI + cloud-init ~16
Trial phase (~41 resources, owned by `roksbnkctl up`)
flo + cis helm_release ~4
cne_instance kubectl_manifest ~1
license kubectl_manifest ~1
namespaces + secrets (kubernetes) ~5
cert issuer chain (kubectl_manifest) ~3
NADs (kubectl_manifest) ~2
SCC / node-labeler kubectl_manifest ~20
IBM IAM trusted profile + COS reads ~5
The kubectl_manifest resources are ordinary Terraform state: a re-run plan shows them as no-ops unless their spec changed — see §“Re-running up is a no-op”.
Apply timing
A clean up against a fresh cluster takes ~50 minutes:
- ROKS cluster provisioning: 30-40 min (the bulk of the wait)
- cert-manager + flo
helm_release(wait = true): ~5 min - cne_instance reconcile: 1-2 min (gated on the CNEInstance’s real
Availablecondition, not a sleep) - license verification: 1-2 min (gated on the License CR’s real
status.state, not a sleep) - Cluster-side bits: applied concurrently with the above
Against a pre-existing cluster (already-up’d or registered), the trial-only run is 5-10 minutes. Most of that is helm_release waiting for flo to stabilise and the two CR wait_for blocks resolving.
Readiness is gated on real signals, not fixed sleeps. The terraform-native path uses
helm_release wait = trueandkubectl_manifestwait_for, which return the instant readiness is observed and no sooner — the CNEInstance’sAvailablecondition and the License CR’sstatus.stateare the actual signals it watches. On a warm cluster this real-readiness gating is the single largest contributor to the fast re-deploy loop.
Re-running up is a no-op
If you re-run roksbnkctl up against an already-deployed BNK trial with no config change, the plan reports 0 to add, 0 to change, 0 to destroy and up skips apply.
Each cluster object is a real Terraform resource:
- A
helm_releaseknows the installed chart version, so an unchanged version is a no-op. - A
kubectl_manifestcarries the rendered manifest as state, so an unchanged spec is a no-op (and a changed spec shows a precise~ update in-placediff of exactly the changed fields). - A
kubernetes_secretis keyed by its data, so an unchanged Secret is a no-op.
So the second up against an unchanged trial genuinely changes nothing. To bump a BNK version or tweak a CNEInstance parameter, edit config.yaml (or a --var-file) and re-run: terraform plan diffs only the changed CNEInstance spec and helm versions — Terraform is the delta engine. Use roksbnkctl plan to preview the diff without applying.
Reading the Terraform plan output
roksbnkctl up runs terraform plan first and prints its output. The plan summary at the end is the most useful part:
Plan: 77 to add, 0 to change, 0 to destroy.
Or, post-rotation:
Plan: 12 to add, 0 to change, 12 to destroy.
The body of the plan shows individual resource changes with one of three markers:
+ create— a new resource. Lines are green in a TTY.<= read— a data source the plan read but did not change. Common fordata "ibm_resource_group"and similar lookups; effectively informational.# destroy— an in-progress destroy of an existing resource. Followed by a+ createif it’s being replaced (e.g. akubectl_manifestwhose spec changed in a way that forces replacement).~ update in-place— a resource whose attributes are being mutated without re-creation.
The <= data sources are the ones that look like:
data "ibm_resource_group" "default" {
name = "Default"
id = "abc123..." (will be read)
}
These are read-only — Terraform is just resolving the resource group’s ID at plan time so downstream modules can reference it. They show up in every plan, including no-op plans.
# destroy lines without a corresponding + create — i.e. resources actually leaving — should make you stop and read carefully. On a re-run of up, this generally means an upstream HCL change removed a resource. It’s rare but not zero.
When up doesn’t apply (no-op runs)
If the plan reports zero changes, up skips apply and prints:
✓ no changes
But it still does two best-effort post-actions:
- Fetch the kubeconfig (unless
--no-kubeconfig). Useful when the cluster exists but you’ve never grabbed the admin kubeconfig on this workstation. - Auto-register the
jumphosttarget. Readstesting_tgw_jumphost_ipandjumphost_shared_keyfrom Terraform outputs and writes atargets:jumphostentry in workspace config. Re-runs are idempotent.
So roksbnkctl up against an unchanged cluster is a useful “re-establish my workstation’s view of this workspace” verb — it can’t hurt anything (no apply runs), and it freshens local artefacts.
The --auto, --no-kubeconfig, --var-file flags
roksbnkctl up [--auto] [--no-kubeconfig] [--var-file <path>]...
| Flag | Effect |
|---|---|
--auto | Skip the “Apply this plan? [y/N]” prompt. Required for non-interactive runs (CI, scripted pipelines). |
--no-kubeconfig | Skip the post-apply kubeconfig fetch. Useful when you’ve already got a kubeconfig and don’t want it overwritten. |
--var-file <path> | Layer extra Terraform var-files onto the chain (repeatable; later wins). Lets you parameterise without editing config.yaml. |
--tf-source <ref> | Override the pinned TF source for this run only. Skip the embedded HCL and use a path or URL instead. Mostly for dev. |
--var-file is the canonical way to stage a non-default deploy. For example, deploying a BNK trial with a non-default cne_instance.replicas:
echo 'cne_replicas = 3' > ./more-replicas.tfvars
roksbnkctl up --auto --var-file ./more-replicas.tfvars
The var-file chain is, in order:
- The auto-generated
terraform.tfvars(rendered fromconfig.yaml). ~/.roksbnkctl/<workspace>/terraform.tfvars.userif present — sibling toconfig.yaml, serving both the trial and cluster phases (a manual raw-tfvars override you place at the workspace root — see Chapter 6 §“Raw terraform-variable overrides”).- Each
--var-fileflag, left-to-right.
Later wins on conflict — same as Terraform’s own ordering. If you find yourself passing the same --var-file on every up / plan / apply / down, drop it once at ~/.roksbnkctl/<ws>/terraform.tfvars.user (sibling to config.yaml) — the lifecycle auto-layers that file on every subsequent call, so you can drop the flag.
Reviewing a plan before applying (plan --out / apply --plan)
plan and apply are separate commands, so you can review a change in full before it touches the cluster — useful for change-control sign-off, or simply when the diff is larger than your terminal scrollback. Save the plan to a file, review it, then apply exactly that plan:
# 1. Save the plan: a binary plan file + a human-readable .txt copy
roksbnkctl plan -w dev --out ./bnk.plan
# ✓ Saved plan: /…/bnk.plan (reviewable copy: /…/bnk.plan.txt)
# Review it, then apply EXACTLY this plan:
# roksbnkctl apply -w dev --plan /…/bnk.plan
# 2. Review bnk.plan.txt (the complete terraform diff — no scrollback limit)
# 3. Apply exactly the reviewed plan — no re-plan
roksbnkctl apply -w dev --plan ./bnk.plan
apply --plan applies the saved plan verbatim: it does not re-plan, and it passes no var-files (the plan already captured every variable). If state or config has drifted since the plan was saved, Terraform refuses the stale plan rather than applying something you didn’t review — that is the change-control guarantee.
Without --plan, a bare roksbnkctl apply re-plans and applies fresh. That is fine for iteration, but it does not guarantee the applied change equals the one you reviewed with plan. Use --out / --plan whenever “what I reviewed is exactly what applied” matters.
The saved-plan flow requires the local execution backend; on a
dockeror remote backend,--out/--planerror clearly rather than being silently ignored.
Apply retries on transient errors
ROKS master endpoints take 1-5 minutes to fully propagate after the cluster reaches Ready. The helm, kubernetes, and alekc/kubectl providers all talk to the master directly; on a fresh cluster, they sometimes race propagation and fail with a connection error (Connection refused, i/o timeout, TLS handshake timeout).
roksbnkctl up has built-in retry: up to 3 apply attempts, with a 60-second sleep between attempts, on any of these heuristic patterns:
exit status 7(curl couldn’t connect)Connection refused/connection refusedi/o timeoutno route to hostnetwork is unreachableno such hostTLS handshake timeoutfailed to dialto download the config doesn't exist
If your apply hits one of these, you’ll see:
→ apply attempt 1 hit a transient-looking failure; waiting 60s and retrying...
Terraform’s idempotence means already-created resources are skipped on the retry; only the failed resources / data sources re-execute. After 3 attempts, up gives up:
✗ apply still failing after 3 attempts — giving up
At that point, fix the underlying cause (usually wait longer or re-run manually) and try again. The retry is for transient races, not persistent failures.
What happens on success
A successful up does five things in order:
- Apply complete.
Apply complete! Resources: 77 added, 0 changed, 0 destroyed. - Fetch the admin kubeconfig from IBM Cloud’s container service API. Written to
$KUBECONFIG(or~/.kube/config) at mode 0600. - Auto-register the
jumphosttarget in workspace config (so--on jumphostworks without manual config — see Chapter 16). - Stamp
terraform.tfstate’s mtime.roksbnkctl statusreads this as “last apply” timestamp. - Exit 0.
The kubeconfig fetch and jumphost registration are best-effort: they log warnings on failure but don’t fail the parent command. up succeeded if Terraform succeeded; the post-apply niceties are conveniences.
The bnk up / bnk down command group
The roksbnkctl bnk group is the trial-only counterpart to roksbnkctl cluster — it operates on the trial state under state/ and leaves the cluster state under state-cluster/ untouched. Because it iterates on the trial without rebuilding the cluster, a bnk down / bnk up round-trip is the 5-10 minute trial-apply window, with the cluster running underneath.
roksbnkctl bnk up
Deploys the BNK trial against the workspace’s registered cluster.
- If the workspace already has a cluster phase (either from
cluster upor fromcluster register),bnk upruns the trial apply directly — same plan, same ~41 resources, same 5-10 minute window as the trial half of a fullup. - If the workspace is empty (no cluster registered yet),
bnk upoffers to bootstrap the cluster phase first with a confirmation prompt, then runs the trial apply. This keeps the new user’s quick-start path one command, even if they typedbnk upinstead ofup.
Sample output of the bootstrap-prompt path:
$ roksbnkctl bnk up
No cluster registered for this workspace.
→ Provisioning the cluster phase first (ROKS cluster + transit gateway +
registry COS + cert-manager + jumphost; ~30 min) before the BNK trial.
Continue? [y/N]: y
→ terraform plan (cluster phase: deploy_bnk=false forced)
...
✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json
→ terraform plan (trial phase)
...
Apply complete! Resources: 41 added, 0 changed, 0 destroyed.
Three prompts fire in the empty-workspace case — one for “do you want to bootstrap the cluster phase,” one for “apply this terraform plan” inside the nested cluster up, and a third when the trial-phase apply prompts. (On a non-empty workspace where bnk up skips the cluster bootstrap, only the latter two fire — and a ShapeClusterOnly/ShapeSplit bnk up is the common iteration case.) For a 30-minute operation we kept the prompts explicit rather than collapsing them. --auto skips all three:
$ roksbnkctl bnk up --auto
roksbnkctl bnk down
Destroys the trial only. The cluster phase keeps running.
- On a split workspace (cluster + trial both present),
bnk downrunsterraform destroyagainst the trial state — ~41 resources, the same as the trial half of a fulldown. - On an empty or cluster-only workspace,
bnk downis a no-op success (exit 0): there’s no trial to destroy, and an orchestrated teardown runs every phase unconditionally, so failing for having nothing to do would fail a teardown that worked.
Sample output against a split workspace:
$ roksbnkctl bnk down --auto
→ terraform destroy (trial phase)
module.license.kubectl_manifest.license: Destroying...
module.cne_instance.kubectl_manifest.cneinstance: Destroying...
module.flo.helm_release.flo: Destroying...
...
Destroy complete! Resources: 41 destroyed.
✓ Trial phase destroyed. Cluster phase ~/.roksbnkctl/default/state-cluster/ is intact.
Run `roksbnkctl bnk up` to deploy another trial against the same cluster.
The shape dispatch matrix
The unscoped roksbnkctl up / down verbs are now shape-aware composites — they detect the on-disk shape of the workspace and delegate to the right phase commands underneath. The full picture for all four shapes and all six commands:
| Command | Empty (nothing applied) | ClusterOnly (cluster up ran) | Split (cluster + trial both applied) |
|---|---|---|---|
up | cluster up → trial up | trial up | cluster up (refresh) → trial up |
down | no-op (exit 0): nothing to destroy | cluster down | trial down → cluster down |
bnk up | confirm + cluster up → trial up | trial up | trial up |
bnk down | no-op (exit 0): no trial | no-op (exit 0): no trial | trial down |
cluster up | cluster up | cluster up (refresh) | cluster up (refresh) |
cluster down | no-op (exit 0): nothing to destroy | cluster down | refuse: trial exists |
The user-facing simplification: the unscoped up / down “just work” against every shape. The phase-scoped commands (bnk, cluster) only operate when the shape allows isolation and refuse loudly with an actionable message otherwise. Refusals always point at the resolution — see Chapter 11 §“Refusal messages” for the full catalogue.
The engineering version of this table — with the implementation details, the ShapeUnknown edge cases, and the rationale — lives in PRD 06 §“Dispatch table”.
Worked example — iterating on a BNK trial
The headline workflow the phase split unlocks. You’re testing different cne_instance parameter combinations against a stable cluster.
# Step 1 — one-time cluster provision (~38 minutes)
roksbnkctl cluster up --auto
# → terraform apply (cluster phase: deploy_bnk=false forced)
# ...
# Apply complete! Resources: 36 added, 0 changed, 0 destroyed.
# ✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json
# Step 2 — first BNK trial (~7 minutes — trial only, cluster is reused)
roksbnkctl bnk up --auto
# → terraform plan (trial phase)
# Plan: 41 to add, 0 to change, 0 to destroy.
# ...
# Apply complete! Resources: 41 added, 0 changed, 0 destroyed.
# Step 3 — poke at the trial, find something to tune
roksbnkctl k get pods -n f5-bnk
roksbnkctl test connectivity
# Step 4 — destroy just the trial (~3 minutes — cluster persists)
roksbnkctl bnk down --auto
# → terraform destroy (trial phase)
# Destroy complete! Resources: 41 destroyed.
# ✓ Trial phase destroyed. Cluster phase ~/.roksbnkctl/default/state-cluster/ is intact.
# Step 5 — edit config.yaml (or a --var-file) to change cne_instance settings
$EDITOR ~/.roksbnkctl/default/config.yaml
# Step 6 — second BNK trial against the same cluster (~7 minutes; the 30-minute
# cluster provision from step 1 does NOT repeat)
roksbnkctl bnk up --auto
# → terraform plan (trial phase)
# ...
# Apply complete! Resources: 41 added, 0 changed, 0 destroyed.
The win is in step 6: the cluster persists across the bnk down / bnk up boundary, so the second trial deploy is ~7 minutes instead of the ~50 minutes a full down → up cycle would cost. Across a day of iteration, that’s the difference between five trial permutations and one.
When you’re done with the whole session:
# Step 7 — tear down the cluster too
roksbnkctl cluster down --auto
# (or `roksbnkctl down` from any starting state — see the dispatch matrix above)
The terraform-native deployment model
The BNK phase has three layers, each owned by a purpose-built Terraform provider. Nothing shells out to curl, kubectl, or helm; nothing sleeps a fixed interval.
1. The install layer — helm_release
cert-manager, FLO, and CIS install as helm_release resources with wait = true. helm_release runs Helm in-process (via the hashicorp/helm provider), tracks the installed chart/version in Terraform state, and — with wait = true — blocks until the release’s workloads report ready. FAR version discovery stays Terraform-side: the manifest chart is still pulled to read the FLO and CIS chart versions, which feed the helm_release.version argument.
The chart installs have prerequisites that must exist first, or helm_release wait = true would hang on ImagePullBackOff:
- Namespaces (
f5-bnk,f5-utils) —kubernetes_namespace. One namespace if you collapse them: settingbnk.flo_namespaceandbnk.flo_utils_namespaceto the same value is supported, and the utils-side namespace (and its duplicate secrets) are then not created at all. - Secrets — the FAR image-pull secret
far-secret(in both namespaces — or once, when they are collapsed to one) and the CISf5-bigip-ctlr-loginlogin secret —kubernetes_secret. These carry the registry credentials the charts pull with, so they are ordered before the charts viadepends_on.
2. The CR layer — kubectl_manifest + wait_for
Every custom resource — the cert-manager issuer chain (ClusterIssuer, the CA Certificate, the CA ClusterIssuer), the two NetworkAttachmentDefinitions, the OpenShift SCC ClusterRoleBindings, the node-labeler Job, the CNEInstance, and the License — is a kubectl_manifest resource from the alekc/kubectl provider. Each manifest body is the CR spec rendered by yamlencode.
Two CRs gate on real readiness via a wait_for block that watches the live object’s .status:
# CNEInstance — wait for FLO to report the instance up
resource "kubectl_manifest" "cneinstance" {
yaml_body = yamlencode(local.cneinstance_manifest)
server_side_apply = true
field_manager = "roksbnkctl"
wait_for {
condition {
type = "Available"
status = "True"
}
}
depends_on = [helm_release.flo] # the FLO chart installs the CRD
}
# License — wait for the CPCL state machine to reach the licensed state
resource "kubectl_manifest" "license" {
yaml_body = yamlencode(local.license_manifest)
server_side_apply = true
field_manager = "roksbnkctl"
wait_for {
field {
key = "status.state"
value = "Verification Complete"
}
}
depends_on = [helm_release.flo, kubectl_manifest.cneinstance]
}
wait_for polls the live object and returns the instant the matched condition/field becomes true. The CNEInstance signal is its Available condition (status.conditions[type=Available].status == True); the License signal is its status.state reaching the licensed value. (The "Verification Complete" literal comes from the architect spike’s CWC-REST evidence and the FAR-shipped License CRD; it is confirmed against a live licensed cluster by the integrator’s e2e-bnk-native.sh run before merge — if the live value differs, staff pins the real one or switches to the status.conditions[] fallback matcher, which is equally expressible.) Resources with no meaningful readiness status — NADs, SCC bindings, issuers — carry no wait_for and simply apply.
3. Namespaces + Secrets — the kubernetes provider
Covered above as the chart prerequisites. They use the hashicorp/kubernetes provider rather than kubectl_manifest because they have typed, first-class resource schemas (kubernetes_namespace, kubernetes_secret) and no plan-time-CRD concern.
Provider wiring and ordering
All three providers are fed from the same data "ibm_container_cluster_config" the modules already use — host, token, and the decoded cluster_ca_certificate. The create_roks_cluster ? 0 : 1 guard that keeps the provider config valid when the cluster doesn’t exist yet at plan time is unchanged.
The only genuinely serial spine is cert-manager helm_release → the CA issuer chain → flo helm_release → CNEInstance → License. Everything else — the NADs, the three Secrets, the node-labeler subtree, and the CNEInstance SCC bindings (19 — 9 FLO-side, 10 utils-side; 18 when the namespaces are collapsed and the duplicate default binding dedupes) — carries no depends_on edge back to the spine, so Terraform’s default parallelism applies them concurrently with the install spine. That parallelism, plus real-readiness gating instead of fixed sleeps, is where the speed comes from.
Why alekc/kubectl
A concept note on the deployment model, for readers who want the why rather than the how.
The plan-time-CRD problem. Applying a CR whose CRD is installed in the same apply (the FLO chart installs the CNEInstance and License CRDs, then we apply those CRs) is the classic Terraform trap. HashiCorp’s own kubernetes_manifest does a typed-schema lookup of the kind at plan time and fails with cannot select exact GVK ... no matches for kind because the CRD doesn’t exist yet. The alekc/kubectl provider’s kubectl_manifest carries the manifest as an opaque YAML string and resolves the GVK at apply time — so it applies a CR whose CRD is created in the same run, no plan-time schema lookup. It gives us real state + drift + finalizer-aware delete + wait_for, without the plan-time-CRD failure and without building a custom provider or a Go reconciler. Ordering (CRD before CR) is expressed with depends_on on the chart’s helm_release — wait_for removes the sleep, not the ordering.
What stays IBM-Cloud-native. The IBM IAM trusted-profile resources (for the CNE controller service account) and the COS reads (the FAR auth archive and the license JWT, fetched via the IBM IAM token + the COS S3 REST API) are IBM-Cloud-native Terraform resources/data sources, not Kubernetes mutations.
The air-gap caveat for alekc/kubectl. alekc/kubectl is a third-party provider on the public Terraform registry (registry.terraform.io/providers/alekc/kubectl). On connected runs it downloads at terraform init like any provider. For air-gapped / offline ROKS installs it must be pre-staged: build a terraform providers mirror bundle on a connected machine and point the offline runner at it via a provider_installation { filesystem_mirror { ... } } block in the Terraform CLI config (or an internal network_mirror). The provider is bounded (~> 2.4) and pinned in .terraform.lock.hcl, which now ships inside the roksbnkctl binary and seeds each workspace’s terraform directory — so a mirrored install resolves the same version, and the recorded checksums verify what the mirror serves.
Note — transcripts, resource counts, and timings are illustrative. The sample
terraform plan/apply/bnk up/bnk downoutput, the~41/~77-resource breakdowns, and the timing figures in this chapter are representative of the terraform-native path, not captured from a live run. The command surface (the resource shapes —helm_release/kubernetes_*_v1/kubectl_manifest+wait_for) is reconciled against the built binary and the landed terraform and is accurate; treat the specific numbers and console lines as orientation, not contract.
Cross-references
- Chapter 7 — Quick start — happy-path walkthrough end-to-end.
- Chapter 8 — The cluster phase — what
cluster upprovisions and the two state directories. - Chapter 11 — Tearing down — phase-aware decision matrix; full refusal-message catalogue; orphan recovery.
- Chapter 13 — Terraform variables — full reference for what you can override via
--var-file. - Chapter 22 — Throughput testing — once BNK is deployed, validating its data plane.
- Chapter 26 — Troubleshooting — long-tail apply failures (SCC violations, propagation lag, kubeconfig 404s) and their fixes.
Air-gapped install: mirroring BNK into a private registry
Some ROKS environments forbid pulling install resources from external repositories
at deploy time. For these, roksbnkctl can replicate every artifact BNK needs
— the F5 charts and images, plus the non-F5 dependencies — from the F5 Artifact
Repository (FAR, repo.f5.com) into a private registry you control, and then
install BNK entirely from there. With the mirror in place, the cluster needs no
egress to repo.f5.com, quay.io, docker.io, or charts.jetstack.io to bring
up BNK.
The mirror target is a registry you own — IBM Container Registry (ICR), the
default, or a generic OCI-compliant registry (Artifactory, Harbor, Quay,
registry:2). Choosing and configuring the target is its own chapter,
Registry targets: ICR and Artifactory; this chapter
is the air-gap flow itself.
The registry surface is CRUD-shaped, modeled on the IBM COS client:
| Command | What it does |
|---|---|
roksbnkctl registry target | Show or set the mirror target — icr / generic (see Registry targets) |
roksbnkctl registry bom | Print the bill-of-materials — every chart + image BNK needs |
roksbnkctl registry replicate | Mirror the BOM into the target registry |
roksbnkctl registry adopt | Record a mirror that already exists, without re-copying (no source access) |
roksbnkctl registry list | What’s present in the target |
roksbnkctl registry diff | BOM vs. target — what’s missing or stale |
roksbnkctl registry verify | Every BOM artifact present + digest-matched |
roksbnkctl registry prune | Remove mirrored artifacts no longer in the BOM |
roksbnkctl registry delete | Delete all replicated artifacts from the target + clear the record |
The bill-of-materials
The BOM comes from the f5-bigip-k8s-manifest that FAR publishes per BNK
release — the same manifest roksbnkctl reads for version discovery. It
enumerates every F5 chart and image (for BNK 2.3, that’s 25 charts + 56 images),
tag-pinned. roksbnkctl unions in the two dependencies the F5 manifest does not
cover — cert-manager (the Jetstack chart + its quay.io images) and the
registry.k8s.io/kubectl node-labeler image — plus the manifest chart itself
(release/f5-bigip-k8s-manifest), because the install needs to read it and a
mirror that lacked it would send every install back to repo.f5.com. So the
mirror is complete:
$ roksbnkctl registry bom
charts repo.f5.com/charts/f5-lifecycle-operator v2.21.13-0.0.28
charts repo.f5.com/charts/f5-tmm 15.430.5-0.2.157
...
images repo.f5.com/images/tmm-img v10.159.3-0.1.5
images quay.io/jetstack/cert-manager-controller v1.17.3
images registry.k8s.io/kubectl v1.36.0
...
81 F5 artifacts + 6 dependency artifacts
The walk-through
$ roksbnkctl cluster up # the durable cluster
$ roksbnkctl registry target icr # choose the target (icr is the default)
$ roksbnkctl registry replicate # mirror the BOM (one-time per BNK version)
$ roksbnkctl registry verify # confirm completeness
$ roksbnkctl bnk up # installs from the mirror
The walk-through runs registry replicate after the Cluster phase for convenience,
but replicate needs no cluster. It is a purely host-side, registry-to-registry
copy: it pulls each artifact by digest from the FAR source (repo.f5.com) and pushes
to your target registry via go-containerregistry,
from wherever roksbnkctl runs. So you can pre-seed the mirror as a standalone
supply-chain step, before any cluster exists — all it requires is a configured
registry: block, the FAR source credential, and network reachability to both
repo.f5.com and your registry:
$ roksbnkctl -w mirror init --config-file mirror.yaml # registry: block + bnk.far_auth_local_file
$ roksbnkctl -w mirror registry target generic # or icr
$ roksbnkctl -w mirror registry replicate # copies FAR → your registry, no cluster
$ roksbnkctl -w mirror registry verify
registry replicate is a deliberate, occasional step — it is not part of the
composite roksbnkctl up (like gateway, you run it explicitly). With no --target,
it uses the workspace’s registry.target (default icr). It copies each artifact by
digest, idempotently — re-running only moves what changed (after a BNK version bump,
registry diff shows exactly the changed artifacts).
How the install reads the mirror
registry replicate records what it copied — and where — in
registry-mirror.json. The BNK phase reads that record and redirects the
install to the target: every chart and image resolves from your private
registry instead of FAR, so BNK comes up with no external pulls. The air-gap
acceptance is exactly this, with egress to the external registries blocked.
The record is only used when it describes the mirror the workspace is
currently configured for — same target kind, repository and host. A record
outlives the mirror it describes: repoint registry.generic_repo_prefix at
another repository, or switch registry.target, and the record on disk still
names the old one. Nothing re-probes on read, so an unchecked record would
redirect the whole install onto a registry nobody asked for, and terraform would
apply it without complaint. bnk up refuses instead, naming both repositories:
the recorded registry mirror does not describe the configured mirror: it was
written for repository "bnk-mirror", the configured repository is "docker-local".
Acting on it would use a mirror this workspace is not configured for.
Re-record it: roksbnkctl -w prod registry replicate (needs the FAR source)
Already filled: roksbnkctl -w prod registry adopt (no source access needed)
The same check gates the node CA-trust install, which would otherwise push the
old mirror’s CA onto every worker. When the configured mirror cannot be resolved
from config at all — an unset generic_host, an ICR region with no known
registry host — the record is left alone: knowing less than the record is not
grounds for discarding it.
The target’s own pull credential is wired in for you:
-
ICR — the cluster authenticates to
*.icr.iowithiamapikey+ the workspace IBM Cloud API key, for both the FLO chart pull and the image pulls. -
Generic OCI — chart and image pulls authenticate with the same basic-auth credential replication used (
registry target generic_username/generic_password), so a private registry needs no anonymous/public project. Concretely: chart pullshelm registry loginwith it, and the pods get amirror-secretdockerconfig built from it, created in every namespace that pulls images — cert-manager, the FLO/BNK namespaces, andkube-systemfor the node-labeler — and referenced from the CNEInstance. You do not create any of it.This is the one place the two mirror kinds differ. An in-cluster/ICR mirror authorizes by RBAC and gets no pull secret; an external one (Harbor, Artifactory) gets
mirror-secret. Dropping the pull secret for every mirror would force people to make their Harbor project world-readable — for a registry holding F5’s proprietary images, not an acceptable requirement.
Doing it all in CI, with nothing installed
Every command in this chapter runs unchanged inside the
tools-runner container, so a pipeline can mirror the
registry and install from it with no roksbnkctl, terraform, helm, kubectl or
ibmcloud on the runner. The registry target itself comes from the environment —
ROKSBNKCTL_REGISTRY_TARGET, ROKSBNKCTL_GENERIC_HOST,
ROKSBNKCTL_GENERIC_REPO_PREFIX, ROKSBNKCTL_GENERIC_USERNAME,
ROKSBNKCTL_GENERIC_PASSWORD — so nothing has to template a config.yaml:
docker run --rm -v "$PWD/state:/work" \
-e IBMCLOUD_API_KEY -e ROKSBNKCTL_REGION=eu-gb \
-e ROKSBNKCTL_PREFIX=ci -e ROKSBNKCTL_CLUSTER_NAME=my-cluster -e ROKSBNKCTL_CLUSTER_CREATE=false \
-e ROKSBNKCTL_REGISTRY_TARGET=generic \
-e ROKSBNKCTL_GENERIC_HOST=harbor.example.com \
-e ROKSBNKCTL_GENERIC_REPO_PREFIX=bnk-mirror \
-e ROKSBNKCTL_GENERIC_USERNAME=admin -e ROKSBNKCTL_GENERIC_PASSWORD \
ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.19.0 -w ci \
init --non-interactive --override-from-env
For the full disconnected picture — a private registry and licensing through a proxy in another cluster, as a two-job pipeline — see Flow C in CI.
Why the install never phones home
Two things have to be true for a mirrored install to be genuinely disconnected,
and roksbnkctl handles both:
- The manifest is mirrored.
f5-bigip-k8s-manifestis the BOM’s own source, so it is easy to overlook — but the install reads it to derive the FLO and CIS chart versions. It is a normal OCI chart, so it is replicated like any other artifact and pulled from the mirror. - FLO reads the manifest from the cluster, not a registry. The F5 Lifecycle
Operator resolves the BNK manifest by listing cluster-scoped
CNEManifestCRs and matchingspec.version; only when none matches does it fall back to pulling the manifest chart from the CNEInstance’sspec.registry.uri.roksbnkctlconverts the manifest into aCNEManifestCR and applies it before the CNEInstance, so FLO finds it in the cluster and never fetches a manifest at all. (Without the CR, that fallback is what breaks a mirrored install: FLO reports “No CNEManifest exists which contains expected manifestVersion” and the CNEInstance never reconciles.)
One mirror, many clusters
Nothing ties a mirror to a single cluster. registry replicate is a supply-chain
step, not a cluster step — the record it writes (registry-mirror.json) just tells
the install where to pull from. So several workspaces can point at the same
registry: replicate once, then every cluster installs from it.
The second workspace still needs a record of its own, though, because bnk up
refuses to render against a mirror it has no record of. Use
registry adopt:
$ roksbnkctl -w cluster-b registry adopt
✓ 10.241.0.4 holds 89 repositories under "bnk-mirror"
✓ recorded the mirror CA from 10.241.0.4 (nodes install it before pulling)
✓ adopted the mirror at 10.241.0.4/bnk-mirror — `bnk up` will render against it
adopt derives the record from the workspace’s own registry config and asks the
mirror what it holds — it needs no access to repo.f5.com at all, which matters
because an air-gapped operator usually has none. Re-running replicate would also
work when the source is reachable, and copies nothing (every artifact is present
and digest-matched), but it cannot run without the source. Add --verify-contents
to adopt when you do have the source and want the mirror digest-checked before it
is recorded.
That pairs naturally with a shared licensing
cluster: one cluster
holds the F5 License Proxy and reaches F5, one registry holds the artifacts, and any
number of air-gapped clusters install from the registry and license through the
proxy — reaching neither repo.f5.com nor F5’s licensing service themselves.
The per-target host, namespace/repository, and credential specifics — including the ICR namespace and a full Artifactory walkthrough — are in Registry targets.
TLS trust. Charts and images are pulled over the target’s HTTPS endpoint. ICR and a public Artifactory carry publicly-trusted certificates. For a custom or self-signed cert, give roksbnkctl the CA itself —
registry target generic_ca <file>records it in the workspace (registry.generic_ca_b64), andbnk upinstalls it on every node before pulling, so you do not have to touch node trust by hand. roksbnkctl will not adopt a self-signed CA it merely discovered over the wire; see Configuration reference for the pin (generic_ca_sha256).
A truly disconnected cluster (no worker egress)
Mirroring removes external pulls at install time, but by default the ROKS cluster is still built with a public gateway on each subnet — its workers keep Internet egress. For a genuinely disconnected cluster with no egress at all, set:
cluster:
create: true
name: dc-roks
public_gateway: false # no public gateways → no worker Internet egress
init also prompts for this (“Attach public gateways for worker Internet egress?”);
answer No for a private cluster. It renders the cluster_public_gateway = false
terraform variable, so no ibm_is_public_gateway is created and no subnet attaches
one. An omitted or true value is the current behavior (egress on).
This is an expert topology. A cluster with no egress can only pull images and reach IBM Cloud services over private paths that
roksbnkctldoes not build for you. Before settingpublic_gateway: falseyou must provide:
- Private reachability to your mirror registry — e.g. the registry in a VPC the cluster reaches over a Transit Gateway, or a Virtual Private Endpoint (VPE). All BNK images/charts then come from the mirror (this chapter).
- VPEs / private service endpoints for the IBM Cloud services the cluster needs (Container Registry if you mirror to ICR, COS for remote state, etc.).
- Licensing without F5 egress — license through an in-cluster or VSI F5 License Proxy (
bnk.license_mode: f5licenseproxy) that itself has the one controlled egress path to F5, or adisconnectedlicense mode, so the cluster never calls F5 directly.The
public_gatewaytoggle only removes the egress path; wiring the private paths above is the operator’s responsibility. Note the cluster master keeps its public service endpoint regardless — this toggle governs worker/subnet egress, not the API endpoint.
bnk up checks the private paths before it installs
Because those private paths are yours to wire, bnk up verifies them rather than
assuming them. Before it plans anything it probes the mirror — and the F5 License
Proxy, when one is configured — from every node, in every availability zone, and
fails if the mirror is unreachable from any of them.
The check has to run on a node. The host running roksbnkctl typically sits on the
services VPC with egress, while the workers are air-gapped behind the gateway; a probe
from there returns a confident green for a mirror the cluster cannot route to. This was
not hypothetical during validation — registry adopt timed out reaching Harbor from the
operator host while all three nodes reached it fine.
Without the gate, an unroutable mirror surfaced as ImagePullBackOff and then a helm
context deadline exceeded about ten minutes later, naming neither the registry nor the
node. A security group or subnet fixed in two zones out of three is a common shape, and
one pod per node is what makes it visible.
A registry with no CA to install — already in the node trust bundle, or publicly signed — is still probed. Being trusted and being reachable are different things, and only the second fails silently.
Both of the gate’s timers are tunable under
bnk.preflight;
raise the retry budget when the Transit Gateway is attached in the same run as the
install. The full output, rationale, and per-topology walkthroughs are in
Appendix A.
Registry targets: ICR and a private Artifactory
The air-gapped install chapter mirrors BNK into a
private registry you control. There are two targets roksbnkctl registry replicate can populate:
registry.target | Backend | Push / pull host |
|---|---|---|
icr (default) | IBM Container Registry | <region>.icr.io/<namespace> |
generic | Any OCI registry — Artifactory, Harbor, Quay, registry:2 | <host>/<repo-prefix> |
Both are ordinary OCI registries: push, image-pull, and chart-pull all use the
same host, with the FAR category nested under a namespace
(<host>/<ns>/images/<name>).
Configuring the target — registry target
Configure the mirror with the registry target command — no config.yaml
editing required. With no arguments it prints the current target and fields;
otherwise the first argument is a backend kind (icr / generic) or a
field name followed by its value. Everything it sets is written to the
workspace config.yaml.
roksbnkctl registry target # show the current target + fields
roksbnkctl registry target icr # select a backend kind
roksbnkctl registry target icr_namespace my-bnk # set a field
roksbnkctl registry replicate --target generic # or override the backend for one run
Recognized fields: icr_host, icr_namespace, generic_host,
generic_repo_prefix, generic_username, generic_password (the last reads
from stdin with --password-stdin).
The default is ICR
An empty registry.target resolves to icr. Set generic explicitly to
mirror into an OCI-compliant registry such as Artifactory.
The OpenShift internal registry was removed as a target. Earlier releases supported an
openshifttarget that mirrored into the cluster’s own internal registry; it has been removed in favor of ICR and OCI-compliant registries. A workspace still carryingregistry.target: openshiftnow errors —unsupported registry target "openshift" (expected icr or generic)— switch it withroksbnkctl registry target icr(orgeneric).
ICR configuration
roksbnkctl registry target icr
roksbnkctl registry target icr_namespace my-bnk # optional — defaults to the workspace prefix
roksbnkctl registry target icr_host de.icr.io # optional — derived from ibmcloud.region
- Host comes from
ibmcloud.region(eu-de→de.icr.io,eu-gb→uk.icr.io,us-south/us-east→us.icr.io, …). Override withicr_hostfor any region the map doesn’t cover. - Namespace is the ICR tenant unit; it defaults to the workspace
prefix. Create it first —roksbnkctldoes not provision the ICR namespace:ibmcloud cr namespace-add my-bnk - Auth is
iamapikey+ your workspace IBM Cloud API key (resolved the usual way). On the cluster, pods pull from<region>.icr.iousing the ROKS global*.icr.iopull secret.
Walkthrough: replicate FAR into a private Artifactory
A generic target works against any standard OCI registry. Here is the full
flow for a private JFrog Artifactory OCI repository.
1 — Provision an Artifactory OCI repository. In Artifactory, create a local
Docker/OCI repository (e.g. bnk-mirror) and an access token with push +
pull on it. Note the registry host (e.g. acme.jfrog.io) and the repository
key.
2 — Configure the workspace. Set the generic target with registry target,
feeding the token via stdin so it never lands in your shell history or argv:
roksbnkctl registry target generic -w prod
roksbnkctl registry target generic_host acme.jfrog.io -w prod
roksbnkctl registry target generic_repo_prefix bnk-mirror -w prod
roksbnkctl registry target generic_username ci-bot -w prod
echo "$ARTIFACTORY_TOKEN" | roksbnkctl registry target generic_password --password-stdin -w prod
roksbnkctl registry target -w prod # confirm (the password shows as "(set)")
For a fully unattended (CI) workspace you can instead supply the token through
ROKSBNKCTL_GENERIC_PASSWORD at init --override-from-env time — see
Unattended setup.
3 — Replicate. Preview, then mirror the bill-of-materials into Artifactory:
roksbnkctl registry diff -w prod # what replicate would copy
roksbnkctl registry replicate -w prod # copy charts + images into acme.jfrog.io/bnk-mirror
roksbnkctl registry verify -w prod # every BOM artifact present + digest-matched
Artifacts land at acme.jfrog.io/bnk-mirror/{images,charts,jetstack,...}/<name>.
4 — Install from the mirror. replicate records the target in
registry-mirror.json, which redirects the BNK install off repo.f5.com onto
the Artifactory mirror automatically:
roksbnkctl bnk up -w prod
The install pulls images and charts from acme.jfrog.io/bnk-mirror, using the same
credential you replicated with — you do not wire the pull secret yourself. Chart pulls
log in with generic_username / generic_password, and the pods get a mirror-secret
dockerconfig built from that credential, created in every namespace that pulls images
(cert-manager, the FLO/BNK namespaces, and kube-system for the node-labeler) and
referenced from the CNEInstance.
So a private repository is fine — the registry does not need an anonymous or public project. The only requirement is network reachability from the cluster.
Licensing from the same mirror. The mirror’s BOM includes charts/f5-license-proxy,
so the F5 License Proxy is replicated alongside BNK and can be installed from your
registry too — including the shared-licensing topology, where one cluster runs the proxy
and other clusters license through it while reaching nothing but this mirror. See
Licensing BNK with the F5 License Proxy.
Removing a mirror
To wipe everything you replicated and revert the install to pulling from FAR:
roksbnkctl registry delete -w prod # confirms first; --force to skip
registry delete removes every artifact recorded in registry-mirror.json from
the target (by digest) and clears the record. Artifacts that fail to delete are
kept in the record so a re-run retries exactly those. For icr the API key needs
Manager (delete) rights on the namespace; for generic the registry must
have deletes enabled. (To remove only artifacts that are no longer in the current
BOM, use registry prune instead.)
Licensing BNK with the F5 License Proxy (FLP)
Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the
Workspacestruct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.
By default roksbnkctl licenses BIG-IP Next for Kubernetes (BNK) with a
subscription JWT — the subscription.jwt (or your production token) staged in the
orchestration COS bucket. In that mode the cluster’s controller reaches F5’s
licensing service directly. Nothing in this chapter changes that default.
Some environments prefer (or require) an F5 License Proxy (FLP): an in-cluster
service that brokers licensing on behalf of every BNK instance in the cluster,
so the workloads themselves never talk to F5 directly. roksbnkctl can deploy
the FLP and point BNK at it, as an opt-in capability — a new flp lifecycle
phase plus a one-line config switch. Both licensing modes are first-class; if you
don’t opt in, the JWT path is byte-for-byte unchanged.
FLP is optional. Everything here is gated behind
bnk.license_mode: f5licenseproxy. Leave it unset (the default) and BNK licenses with the subscription JWT exactly as before — noflpphase, no extra resources.
What the FLP is
The FLP is a Helm-installed workload (the f5-license-proxy chart: the proxy
itself plus a bundled Vault and PostgreSQL). It presents an in-cluster TLS
endpoint that the BNK Cluster-Wide Controller (CWC) calls for licensing; the
proxy forwards to F5’s licensing backend. Because the CWC must trust the proxy’s
TLS, the proxy’s root CA is delivered to the CWC and the BNK License
custom resource is set to operationMode: f5licenseproxy with the proxy’s
service URL.
roksbnkctl automates all of that: roksbnkctl flp up generates the proxy’s CA
and mTLS certs, creates its secrets, installs the chart (pulling from your
registry mirror or FAR), and records the CA + endpoint. A later bnk up reads
that handoff and wires the License CR — you never copy a certificate by hand.
The FLP’s charts and images are part of the standard BNK bill of materials, so
registry replicate already mirrors them into a
private registry — an air-gapped, FLP-licensed install pulls everything,
including the proxy, from a registry you control.
Three topologies
The proxy is a cluster-wide broker, and it is the only component that needs to reach F5. That gives you three ways to arrange it:
| Where the proxy runs | Use it when | |
|---|---|---|
| Flow A | the cluster roksbnkctl creates | the normal case |
| Flow B | a cluster you already own and register | you did not provision the cluster |
| Flow C | a different cluster — one proxy serving many | only one cluster may reach F5; the rest are air-gapped |
Flows A and B are the same shape: proxy and BNK in one cluster. Flow C splits them.
The flp phase
Like testing and gateway, the FLP is an
independent, opt-in phase with its own terraform state (state-flp/). The
composite roksbnkctl up never runs it; you run it explicitly, between the
cluster and BNK:
roksbnkctl flp up # install the proxy into an existing cluster
roksbnkctl flp down # remove it (leaves cluster / BNK / testing intact)
roksbnkctl flp output # print its terraform outputs (endpoint, namespace)
flp up refuses if the workspace has no cluster yet, and records
flp-outputs.json (the proxy’s root CA + service endpoint) on success. flp down is a no-op success when there is no FLP state, so it is safe in a
reverse-order teardown of every phase.
The proxy installs into the f5-license-proxy namespace by default; override
with bnk.flp.namespace (or ROKSBNKCTL_FLP_NAMESPACE).
Opting in
Set the license mode in the workspace config.yaml:
bnk:
manifest_version: 2.3.0-3.2598.3-0.0.170
license_mode: f5licenseproxy # connected (default) | disconnected | f5licenseproxy
flp:
namespace: f5-license-proxy # optional; this is the default
Or answer yes to License via an in-cluster F5 License Proxy (FLP)? in the
roksbnkctl init interview, or set it non-interactively:
export ROKSBNKCTL_LICENSE_MODE=f5licenseproxy
export ROKSBNKCTL_FLP_NAMESPACE=f5-license-proxy # optional
The subscription JWT is still required in FLP mode (the proxy presents it to
F5); it is resolved from COS exactly as in the JWT path — no change to
bnk.subscription_jwt_file or its credential.
Flow A — a cluster you are creating
The full sequence when roksbnkctl provisions the cluster:
# 1. Seed the workspace with FLP mode (interview, config.yaml, or env).
roksbnkctl -w prod init # answer "yes" to the FLP prompt
# 2. Create the cluster.
roksbnkctl -w prod cluster up
# 3. (Optional) mirror BNK — including the FLP — into your private registry.
# Skip this to pull from repo.f5.com directly.
roksbnkctl -w prod registry replicate --target generic
# 4. Install the F5 License Proxy.
roksbnkctl -w prod flp up
# 5. Install BNK — it licenses via the proxy automatically.
roksbnkctl -w prod bnk up
Step 5 reads flp-outputs.json and sets the License CR to
operationMode: f5licenseproxy, its teem*Url at the proxy service, and
delivers the proxy root CA to the CWC. bnk up errors clearly if the flp
phase has not been run.
Flow B — an existing cluster registered in the workspace
You do not have to let roksbnkctl build the cluster. If you already have a ROKS
cluster, register it into the workspace
and install the proxy onto it. Registration looks the cluster up in your IBM Cloud
account and writes the very same cluster-outputs.json that the cluster up path
produces — and that is the only thing the FLP and BNK phases consume. So from
step 3 on, the two flows are character-for-character identical:
# 1. Seed the workspace with FLP mode, and do NOT create a cluster.
roksbnkctl -w prod init # answer "no" to "create a cluster?", "yes" to FLP
# (equivalently, in config.yaml: cluster.create: false + bnk.license_mode: f5licenseproxy)
# 2. Register the cluster you already have.
roksbnkctl -w prod cluster register my-existing-roks
# 3. (Optional) mirror BNK — including the FLP — into your private registry.
roksbnkctl -w prod registry replicate --target generic
# 4. Install the F5 License Proxy into the registered cluster.
roksbnkctl -w prod flp up
# 5. Install BNK, licensed via the proxy.
roksbnkctl -w prod bnk up
roksbnkctl never takes ownership of a registered cluster. flp up adds only
the proxy workload (its namespace, secrets and Helm release); flp down removes
exactly that and nothing else. The cluster itself is never created, modified at the
infrastructure level, or destroyed — cluster down will not touch it.
FLP behind a private registry
The FLP’s chart and its four images are part of the standard BNK bill of
materials, so registry replicate mirrors them
along with everything else. Combining the two gives a genuinely disconnected,
FLP-licensed install: flp up and bnk up pull every chart and image from
your registry, and the CWC brokers licensing through the in-cluster proxy rather
than reaching F5 itself.
Nothing extra is required — point the workspace at the mirror and the FLP phase follows it automatically:
roksbnkctl -w prod registry target generic # your Harbor/Artifactory/ICR
roksbnkctl -w prod registry replicate # mirrors BNK + the FLP
roksbnkctl -w prod flp up # proxy, pulled from the mirror
roksbnkctl -w prod bnk up # BNK, pulled from the mirror
Flow C — a shared licensing cluster
Flows A and B both put the proxy and BNK in the same cluster. But the proxy is a cluster-wide licensing broker, and it is the only thing that needs to reach F5. So you can run it once, in a cluster that has egress, and have BNK installs in other clusters license through it — clusters that reach nothing but your mirror and the proxy.
┌──────────────── services cluster (has egress to F5) ─────────────┐
│ F5 License Proxy ──NodePort 30001──┐ │
└──────────────────────────────────────┼───────────────────────────┘
│ same VPC, or a transit gateway
┌──────────────────────────────────────┼── air-gapped cluster ─────┐
│ BNK + CWC ─────────────────────────┘ │
│ └── charts + images ── your private registry (Harbor/…) │
└──────────────────────────────────────────────────────────────────┘
Neither cluster reaches repo.f5.com, and only the services cluster reaches F5’s
licensing service.
1. The services cluster — expose the proxy
roksbnkctl -w services cluster up # (or `cluster register` an existing one)
# point the workspace at your private registry, then mirror FAR into it
roksbnkctl -w services registry target generic
roksbnkctl -w services registry target generic_host harbor.example.com
roksbnkctl -w services registry target generic_repo_prefix bnk-mirror
roksbnkctl -w services registry target generic_username admin
printf '%s' "$HARBOR_PASSWORD" | \
roksbnkctl -w services registry target generic_password --password-stdin
roksbnkctl -w services registry replicate # BNK *and* the FLP chart — both are in the BOM
roksbnkctl -w services flp up \
--add-node-port-access \
--node-port-source-cidr 10.242.0.0/18,10.242.64.0/18,10.242.128.0/18
Set the registry target before replicating: with none set it defaults to icr, and
you would mirror into IBM Container Registry rather than your own. The password is piped
on stdin so it never lands in argv or your shell history. See
Registry targets for the ICR and Artifactory variants.
The proxy itself is installed from that registry — charts/f5-license-proxy is part
of the mirror BOM, so even the licensing component does not come from repo.f5.com.
The CIDR flag is a list, and it must name every zone. A multi-zone VPC carries one address prefix per zone:
$ ibmcloud is vpc-address-prefixes <vpc> --output json | jq -r '.[] | "\(.zone.name) \(.cidr)"'
eu-gb-1 10.242.0.0/18
eu-gb-2 10.242.64.0/18
eu-gb-3 10.242.128.0/18
List only the first and the proxy will appear to work — a pod you exec into for a test
may well be scheduled in that zone and get an HTTP 200 — while the consuming cluster’s
CWC, scheduled in one of the other two, is dropped at the security group and reports
connect: connection timed out. The symptom is a proxy that answers some pods and
silently times out for others. Pass every prefix, or a supernet covering them.
No BNK is installed here — this cluster exists to run the proxy.
--add-node-port-access does three things you cannot do by hand afterwards:
- Makes every worker answer. The chart’s Service is already
type: NodePort(30001), but it hardcodesexternalTrafficPolicy: Localand runs one replica — so only the node currently hosting the pod answers, and that node changes when the pod reschedules. This flips it toCluster, so any worker IP is a valid endpoint. - Puts the worker IPs in the proxy’s certificate. A remote CWC dials
https://<node-ip>:30001— a literal address, which DNS SANs do not cover. Without IP SANs the TLS handshake fails withbad certificate. - Opens the port. ROKS workers sit in a
kube-<cluster-id>security group that does not admit another cluster.--node-port-source-cidropens only 30001, and only to the CIDRs you name — one security-group rule per prefix. Omit it if a path already exists.
It prints the address to hand over, and records it in flp-outputs.json:
→ FLP reachable from other clusters at https://10.242.0.9:30001
2. The consuming cluster — license against it
Read the endpoint and CA out of the owning workspace:
roksbnkctl -w services flp output # → external_endpoint, root_ca_b64
and point the other workspace at them. It never runs flp up — it does not own a
proxy:
# consuming workspace's config.yaml
bnk:
license_mode: f5licenseproxy
flp:
external:
url: https://10.242.0.9:30001 # from `flp output` → external_endpoint
root_ca_b64: LS0tLS1CRUdJTi… # from `flp output` → root_ca_b64
Point that workspace at the same private registry, and install. bnk up wires the
License CR at the remote proxy and delivers its CA to the CWC, exactly as it would for a
local one:
roksbnkctl -w app cluster register my-app-cluster # or `cluster up` to build one
# the SAME registry as the services workspace
roksbnkctl -w app registry target generic
roksbnkctl -w app registry target generic_host harbor.example.com
roksbnkctl -w app registry target generic_repo_prefix bnk-mirror
roksbnkctl -w app registry target generic_username admin
printf '%s' "$HARBOR_PASSWORD" | \
roksbnkctl -w app registry target generic_password --password-stdin
roksbnkctl -w app registry replicate # copies nothing — the artifacts are already there
roksbnkctl -w app bnk up # licenses via the proxy in the OTHER cluster
registry replicate in the second workspace has nothing left to copy. Run it anyway:
its other job is to write registry-mirror.json, and that record is what redirects the
install off repo.f5.com onto your registry. Without it, bnk up would go to FAR.
(See One mirror, many clusters.)
Why this cluster is genuinely disconnected
Three things have to be true, and bnk up arranges all three — there is nothing to wire
by hand:
- Charts and images come from your registry. Chart pulls log in with the credential
above; the pods get a
mirror-secretdockerconfig built from it, created in every namespace that pulls (cert-manager, the FLO/BNK namespaces,kube-systemfor the node-labeler) and referenced from the CNEInstance. Your registry stays private — it needs no anonymous or public project. - The manifest is already in the cluster. The BNK manifest is mirrored too and
pre-applied as a
CNEManifestCR, so the lifecycle operator resolves it locally instead of fetching it from F5. - Licensing goes to the proxy next door, not to F5.
The result: this cluster talks to your registry and to https://<node-ip>:30001. Nothing
else.
Flow C in CI — the runner container, no host install
Everything above is a roksbnkctl you installed. The same topology runs as a CI
pipeline with nothing installed at all: every step is the
tools-runner container, and the whole workspace is
described by environment variables — the natural shape of CI, where the registry
credential and the proxy handoff are already secrets and job outputs.
Two jobs. The first owns the proxy and emits its address and CA; the second installs BNK into a cluster that reaches nothing but the registry and that proxy.
# .github/workflows/bnk-disconnected.yml
env:
RUNNER: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.19.0
jobs:
licensing-cluster:
runs-on: ubuntu-latest
container: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.19.0
outputs:
flp_url: ${{ steps.flp.outputs.url }} # ← the handoff
flp_ca: ${{ steps.flp.outputs.ca }}
env:
IBMCLOUD_API_KEY: ${{ secrets.IBMCLOUD_API_KEY }}
ROKSBNKCTL_REGION: eu-gb
ROKSBNKCTL_RESOURCE_GROUP: default
ROKSBNKCTL_PREFIX: ci-svc
ROKSBNKCTL_CLUSTER_NAME: ${{ vars.SERVICES_CLUSTER }}
ROKSBNKCTL_CLUSTER_CREATE: "false" # adopt it; do not build it
ROKSBNKCTL_REGISTRY_TARGET: generic
ROKSBNKCTL_GENERIC_HOST: ${{ vars.REGISTRY_HOST }}
ROKSBNKCTL_GENERIC_REPO_PREFIX: bnk-mirror
ROKSBNKCTL_GENERIC_USERNAME: ${{ vars.REGISTRY_USER }}
ROKSBNKCTL_GENERIC_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
steps:
- run: roksbnkctl -w ci-svc init --non-interactive --override-from-env
- run: roksbnkctl -w ci-svc cluster register "$ROKSBNKCTL_CLUSTER_NAME"
- run: roksbnkctl -w ci-svc registry replicate --target generic
- run: roksbnkctl -w ci-svc registry verify
- name: Deploy the proxy, reachable from the other cluster
run: |
roksbnkctl -w ci-svc flp up --auto \
--add-node-port-access \
--node-port-source-cidr "${{ vars.APP_CLUSTER_CIDRS }}"
- id: flp
run: |
echo "url=$(roksbnkctl -w ci-svc flp output flp_external_endpoint)" >> "$GITHUB_OUTPUT"
echo "ca=$(roksbnkctl -w ci-svc flp output flp_root_ca)" >> "$GITHUB_OUTPUT"
application-cluster:
needs: licensing-cluster
runs-on: ubuntu-latest
container: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.19.0
env:
IBMCLOUD_API_KEY: ${{ secrets.IBMCLOUD_API_KEY }}
ROKSBNKCTL_REGION: eu-gb
ROKSBNKCTL_RESOURCE_GROUP: default
ROKSBNKCTL_PREFIX: ci-app
ROKSBNKCTL_CLUSTER_NAME: ${{ vars.APP_CLUSTER }}
ROKSBNKCTL_CLUSTER_CREATE: "false"
ROKSBNKCTL_REGISTRY_TARGET: generic
ROKSBNKCTL_GENERIC_HOST: ${{ vars.REGISTRY_HOST }}
ROKSBNKCTL_GENERIC_REPO_PREFIX: bnk-mirror
ROKSBNKCTL_GENERIC_USERNAME: ${{ vars.REGISTRY_USER }}
ROKSBNKCTL_GENERIC_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
# ── license through the proxy in the OTHER cluster ──
ROKSBNKCTL_LICENSE_MODE: f5licenseproxy
ROKSBNKCTL_FLP_EXTERNAL_URL: ${{ needs.licensing-cluster.outputs.flp_url }}
ROKSBNKCTL_FLP_ROOT_CA_B64: ${{ needs.licensing-cluster.outputs.flp_ca }}
steps:
- run: roksbnkctl -w ci-app init --non-interactive --override-from-env
- run: roksbnkctl -w ci-app cluster register "$ROKSBNKCTL_CLUSTER_NAME"
- run: roksbnkctl -w ci-app registry replicate --target generic # records the mirror
- run: roksbnkctl -w ci-app bnk up --auto
- name: Gate the pipeline on the license
run: roksbnkctl -w ci-app k get license -n f5-utils
The handoff is two environment variables. ROKSBNKCTL_FLP_EXTERNAL_URL and
ROKSBNKCTL_FLP_ROOT_CA_B64 are exactly what flp output prints, so the proxy’s
address and CA travel between jobs as ordinary job outputs. Nothing templates a
config.yaml; init --override-from-env builds the whole workspace from the
environment. (flp_root_ca is already base64, and the variable takes it verbatim —
re-encoding it hands the CWC a corrupt CA.)
Neither job installs anything. No roksbnkctl, no terraform, no helm, no
kubectl, no ibmcloud — the runner image carries them, and roksbnkctl runs the
Kubernetes verbs in-process. That includes the FLP chart’s post-render fix-ups, which
roksbnkctl performs itself (roksbnkctl flp postrender) rather than shelling out to an
interpreter the image does not have.
State lives in the workspace directory, which in a container is ephemeral. For a
pipeline that will run again — to upgrade, or to tear down — put the terraform state in
COS/S3 (roksbnkctl state migrate); see
Remote state and
Ephemeral runners need remote state.
The demo below mounts a volume instead, which is the same idea with a shorter lifetime.
What the last step proves. k get license prints the CR the CWC wrote:
NAME STATE MODE ENTITLEMENT ENVIRONMENT EXPIRY
bnk-license Active f5licenseproxy eval test 2026-08-13T16:11:06Z
Active + f5licenseproxy, in a cluster with no proxy of its own and no route to
repo.f5.com. Fail the job on anything else and the pipeline gates on the license.
What to watch out for
- The two clusters must be able to reach each other — same VPC (simplest: give the
second cluster
resources.cluster_vpc.create: false+existing: <vpc-id>), or a transit gateway between their VPCs. - Worker IPs change when a worker is replaced. The certificate covers the IPs that
existed at
flp uptime, so replacing a node in the services cluster means re-runningflp up(which re-issues the cert) and updating the consuming workspace if you pinned the URL of the node that went away.flp outputlists every worker URL — all are cert SANs — so you can point at another one. A stable VIP would avoid this entirely; NodePort is the trade-off you accept for not needing a load balancer. - The consuming workspace needs no
flpphase at all — nostate-flp/, nothing to tear down.bnk downthere leaves the proxy untouched, serving its other clusters.
Running the FLP as a VSI (mode: vsi)
The flows above run the FLP as an in-cluster Helm deployment. It can instead run as a
standalone Virtual Server Instance — a VM, no Kubernetes — which is the right shape
when the proxy must be the one controlled egress point for an otherwise disconnected
cluster. Select it with bnk.flp.mode: vsi:
bnk:
license_mode: f5licenseproxy
flp:
mode: vsi
vsi:
profile: bx2-4x16 # >= 4 vCPU / 16 GB
zone: us-south-1
reach: floating # "private" (VPC IP) or "floating" (public IP → egress to F5)
allowed_cidrs: [10.242.0.0/18] # sources allowed to reach the proxy (the cluster's workers)
roksbnkctl flp up then provisions the VSI; it terminates in the same flp-outputs.json
(endpoint + root CA) the Helm path produces, so bnk up consumes it unchanged.
Standalone — a licensing appliance in another VPC (no cluster)
By default the FLP VSI joins the cluster’s own VPC (from cluster-outputs.json), so
flp up requires a cluster. Give the proxy a network of its own — set bnk.flp.vsi.vpc to an existing VPC id, or bnk.flp.vsi.create_vpc: true to build one (mutually exclusive) — and flp up
deploys the proxy into that VPC with no cluster at all — a standalone licensing
appliance:
bnk:
license_mode: f5licenseproxy
flp:
mode: vsi
vsi:
vpc: r006-… # existing VPC id — deploy here, no cluster required
zone: us-south-1
reach: floating # the appliance's controlled egress to F5
allowed_cidrs: [10.0.0.0/8] # allow the consuming cluster's workers (e.g. over a TGW)
This is the piece that makes a genuinely disconnected topology clean: put Harbor +
the FLP in a services VPC that has egress, run the ROKS cluster in another VPC with
cluster.public_gateway: false (no worker egress — see
Chapter 10a §“A truly disconnected cluster”), join both
with a Transit Gateway, and point the cluster’s workspace at the proxy with
bnk.flp.external (its flp output endpoint + root CA). The cluster reaches Harbor and
the FLP privately over the TGW; all egress is concentrated in the services VPC, and
the cluster never phones home. A worked end-to-end script is in
disconnected_deployment_demo.sh.
# in a services workspace (no cluster) — deploy the standalone proxy, then read its handoff
$ roksbnkctl -w svc-flp flp up
$ roksbnkctl -w svc-flp flp output flp_external_endpoint
$ roksbnkctl -w svc-flp flp output flp_root_ca # → the disconnected cluster's bnk.flp.external
Tearing down
The FLP is an opt-in phase, so — exactly like gateway — the composite down
does not cover it. Tear it down first, while the cluster is still up:
roksbnkctl -w prod flp down # remove the proxy
roksbnkctl -w prod down # then BNK + the cluster (composite)
Both roksbnkctl down and cluster down refuse while FLP state exists, and
tell you to run flp down first. That guard is deliberate: the proxy’s Helm
release and secrets live in the cluster, so destroying the cluster out from
under them would strand state-flp/ pointing at resources that no longer exist,
and a later flp down could never reconcile.
flp down is a no-op success when there is no FLP state, so it is safe to put
unconditionally in a teardown script. Registered clusters are never destroyed —
flp down removes only the proxy workload.
Reference
| Config key | Meaning |
|---|---|
bnk.license_mode | connected (default), disconnected, or f5licenseproxy |
bnk.flp.namespace | namespace for the proxy (default f5-license-proxy) |
bnk.flp.chart_version | optional override. Left unset, the chart version is read from the BNK manifest (which lists charts/f5-license-proxy for the release), exactly like the FLO and CIS charts — you do not normally pin it. |
bnk.flp.node_port_access | expose the proxy outside its cluster (Flow C). Set by flp up --add-node-port-access; persisted so a later flp up does not tear the exposure down. |
bnk.flp.node_port_source_cidrs | with the above: the CIDRs allowed to reach the NodePort on the worker security group. A list — one address prefix per zone |
bnk.flp.external.url | license against a proxy in another cluster — its external_endpoint. This workspace never runs flp up. |
bnk.flp.external.root_ca_b64 | that proxy’s root_ca_b64, so the CWC can verify its certificate |
Flag (flp up) | Meaning |
|---|---|
--add-node-port-access | expose the proxy for other clusters: externalTrafficPolicy: Cluster, worker IPs as cert SANs, external endpoint recorded |
--node-port-source-cidr | open the NodePort to these CIDRs on the worker security group. Repeatable / comma-separated — pass every zone’s prefix |
| Environment variable | Overrides |
|---|---|
ROKSBNKCTL_LICENSE_MODE | bnk.license_mode |
ROKSBNKCTL_FLP_MODE | bnk.flp.mode ("" | helm | vsi) |
ROKSBNKCTL_FLP_EXTERNAL_URL | bnk.flp.external.url — license via a proxy in another cluster |
ROKSBNKCTL_FLP_ROOT_CA_B64 | bnk.flp.external.root_ca_b64 (verbatim; already base64) |
ROKSBNKCTL_FLP_VSI_CREATE_VPC | bnk.flp.vsi.create_vpc — give the proxy its own VPC |
ROKSBNKCTL_FLP_VSI_VPC_NAME | bnk.flp.vsi.vpc_name |
ROKSBNKCTL_FLP_VSI_SUBNET_CIDR | bnk.flp.vsi.subnet_cidr |
ROKSBNKCTL_FLP_VSI_VPC | bnk.flp.vsi.vpc — adopt an existing VPC (mutually exclusive with _CREATE_VPC) |
ROKSBNKCTL_FLP_VSI_ZONE / _PROFILE / _SSH_KEY / _BOOT_SIZE_GB / _REACH / _FLOATING_IP | the rest of bnk.flp.vsi.* |
ROKSBNKCTL_FLP_VSI_MANAGEMENT_ALLOWED_CIDRS / _LICENSING_ALLOWED_CIDRS | the two per-plane security-group lists (comma-separated) |
ROKSBNKCTL_FLP_NAMESPACE | bnk.flp.namespace |
| File | Written by | Read by |
|---|---|---|
~/.roksbnkctl/<ws>/flp-outputs.json | flp up | bnk up (FLP mode) — or copied into another workspace’s bnk.flp.external |
flp-outputs.json carries endpoint (in-cluster only) and, when exposed,
external_endpoint plus external_endpoints — every worker URL the proxy answers on.
All of them are IP SANs on its certificate, so any one is a valid
bnk.flp.external.url.
BNK 2.4
roksbnkctl drives BNK 2.3 and 2.4 from one build. Which one you get is
decided by bnk.manifest_version and nothing else — there is deliberately no
bnk.line field and no ROKSBNKCTL_BNK_LINE override, because a second way to
say which release this is, is a second thing that can disagree with the manifest
being installed.
bnk:
manifest_version: 2.3.0-3.2598.3-0.0.170 # -> the 2.3 line
# manifest_version: 2.4.0-EA # -> the 2.4 line
The derived value is rendered as a bnk_line terraform variable ("2.3" or
"2.4"), and the modules gate per-release resources with count on it. There is
no forked copy of the tree: the 2.4 changes are additive resources, which count
expresses cleanly and two copies of a file do not.
2.4.0-EAcarries no build suffix, where 2.3 versions have four dot/dash segments (2.3.0-3.2598.3-0.0.170). Anything you write that parses a manifest version should key on themajor.minorprefix rather than the segment count.
What actually changes
2.4 replaces the data-plane configuration model. It runs the CNEInstance with
USE_GATEWAY_SETTINGS=true, and under that flag the CNE controller ignores
the cloud-network-mapping ConfigMap and the entire F5SPK* custom-resource
family that a 2.3 install creates, reading two new CRs instead.
| 2.3 | 2.4 |
|---|---|
cloud-network-mapping ConfigMap | Infra.spec.networks[] + networkAttachments[] |
F5SPKVlan × 2 | Infra.spec.networks[type=vlan] |
F5SPKStaticRoute × N | Infra.spec.staticRoutes[] |
F5SPKSnatpool | GatewaySettings.spec.sourceNATPools[] |
F5SPKEgress × 2 | GatewaySettings.spec.egressConfigs[] + Infra.spec.egressDefaults |
F5BnkGateway | GatewaySettings (the Gateway’s parametersRef target) |
| VIP address arithmetic | IPAM allocation, read from Gateway.status.addresses |
roksbnkctl emits both CRs for you. The addressing comes from the per-zone CIDRs
you already configure — nothing new to fill in for a standard install:
| your existing config | the 2.4 IPAM pool it becomes |
|---|---|
ext_vlan_cidr | external-vlan-ipam — TMM VLAN self-IPs |
int_vip_cidr | vip-listener-ipam — ingress VIPs |
int_snat_cidr | egress-snat-ipam — egress SNAT addresses |
gateway_egress_mode means the same thing on both lines: snatpool becomes a
Pool egress config backed by egress-snat-ipam, automap becomes an Automap
config, and both produces both.
Things 2.4 stops needing
Three of the four items in the 2.4 guide’s own “Changes From BNK Release-2.3”
section are subtractions, which is easy to miss when reading a list of
additions. On the 2.4 line roksbnkctl no longer creates:
- a separate CIS Helm release — CIS is integrated into FLO, and
externalBigip.enabledon the CNEInstance is what turns it on; - the node-labeler Job —
app=f5-tmmlabelling is not required; - the
macvlan-confNetworkAttachmentDefinition — the product creates its own internal NAD asmacvlan-internalondummy0, owned by theF5TmmCR, and that name is reserved; - eighteen of nineteen privileged SCC bindings. FLO grants its components what they need, so the install requires one binding rather than nineteen.
The guide also asks for one privileged SCC binding per application namespace (
oc adm policy add-scc-to-user privileged -n <ns> -z default).roksbnkctldoes not create those: it knows the FLO and utils namespaces and nothing about where you will run workloads. That one stays an operator step.
The Trusted Profile
2.4’s CNE controller authenticates to IBM Cloud with a Trusted Profile and
uses it to write into your VPC — adding the Infra IPAM CIDRs as VPC address
prefixes, creating a routing table, and advertising TMM virtual-IP routes to the
transit gateway.
roksbnkctl creates the profile, links it to the cluster’s service account, and
passes its id to the CNEInstance as
advanced.cneController.env.CLOUD_TRUSTED_PROFILE. Two access policies are
attached: Viewer + Editor on the cluster’s VPC, so the controller can manage
the network objects it creates, and Viewer on the cluster itself, so it can
read the topology it next-hops routes against.
Because those IPAM CIDRs become real VPC address prefixes, they must not collide with what the cluster VPC already allocates — see Transit gateway sharing.
Moving between lines
You cannot flip a deployed workspace from 2.3 to 2.4 in place. bnk up and
gateway up both refuse it, and the refusal names the reasons, because none of
them is visible in a terraform plan:
- GTM object naming differs. 2.3 writes
server_<tmm_selfip>; 2.4 writesserver_<digitalassetid>_<cluster-name>_<ns>_<ip>. Those objects live on an external BIG-IP — outside the cluster and outside terraform state — so nothing here removes them, and leaving both formats in place causes device IP conflicts. - The network model is replaced, not migrated. The
F5SPK*objects a 2.3 install created are simply no longer read, and they stay on the cluster. - The CWC licence secrets survive a teardown and must be cleared before a
reinstall licenses cleanly.
bnk downnow removes them for you.
To run 2.4, use a new workspace and a new install. Changing
CLUSTER_IDENTIFIER on a deployed workspace needs the same manual GTM cleanup,
for the same reason.
mTLS and the Gateway API bundle
A base 2.4 install works with whatever Gateway API bundle OpenShift ships: its
crd-installer logs a graceful skip and leaves the platform’s CRDs alone.
mTLS is the exception. It needs Gateway API 1.5.0 standard, and installing
that means deleting the OpenShift ingress-operator’s
ValidatingAdmissionPolicy and its binding — which the platform recreates
quickly, so it is a race. Opt in:
bnk:
gateway_api_mtls: true # ROKSBNKCTL_GATEWAY_API_MTLS=true
Off by default, because deleting a platform admission policy on a cluster that
does not need the newer bundle is a change nobody asked for. Ignored on 2.3,
where that sweep always runs — there the crd-installer does force its own
CRDs and is blocked without it.
Where the bundle comes from
With gateway_api_mtls on, bnk up installs the upstream
kubernetes-sigs/gateway-api standard-install bundle itself — eight CRDs, a
ValidatingAdmissionPolicy of its own, and no container images — at the release
named by bnk.gateway_api_version (default 1.5.0, the same number the CNE
controller is given as GATEWAY_API_VERSION, so the two cannot disagree). It is
applied server-side with --force-conflicts, inside the window where the
admission sweep is running, and before terraform starts.
The bytes are always checked against a sha256 that ships in roksbnkctl, not in your configuration. A release with no pin in your build is refused rather than fetched unverified.
It is fetched from, in order:
-
The registry mirror, whenever the workspace has one. The bundle is a normal BOM artifact, so
registry bomlists it,registry replicatecopies it andregistry verifyproves it arrived. This is the path a disconnected cluster needs: in the CI flow roksbnkctl runs as a pod inside that cluster, so its egress is the cluster’s andgithub.comis not reachable.The bundle enters the BOM only when
gateway_api_mtlsis on, so a mirror replicated before you set it does not carry the bundle — re-runregistry replicate. -
bnk.gateway_api_bundle_url, if set — an internal proxy of the upstream release, for an estate that blocksgithub.combut has no mirror yet. It moves only where the bytes come from; the pin still applies. -
The upstream release asset, derived from the version.
Conforming to F5’s reference CNEInstance
The 2.4 spec this tool renders is checked against F5’s own reference install —
the live capture in /mnt/d/roksbnkctl-gap-2-3-to-2-4. Every default below is
F5’s value, not an invented one.
Pod placement replaced the node labeller
2.4 removed the node-labeler job, and spec.placement is the mechanism that
took over. For a while this tool had neither: the labeller was gone and
placement was never emitted, so TMM landing one-per-node and spread across zones
was left to the scheduler’s discretion rather than required.
bnk:
tmm_anti_affinity: true # one TMM per node
tmm_anti_affinity_topology_key: kubernetes.io/hostname # the ROKS node label
tmm_zone_spread: true # spread across zones
tmm_zone_topology_key: topology.kubernetes.io/zone # the ROKS zone label
tmm_zone_max_skew: 1
tmm_zone_when_unsatisfiable: DoNotSchedule
tmm_pod_label: f5-tmm
The topology keys are IBM ROKS node labels, and they are settings rather than constants on purpose: hard-coding them is the assumption the node-labeller baked in, and a cluster that labels its topology differently should still be configurable.
Turn both toggles off and no placement key is emitted at all — an operator who
does not want the constraint gets no empty object for the CR to interpret.
The rest of the conformance set
| setting | default | what it is |
|---|---|---|
tmm_replicas | 3 | data-plane replica count |
watch_namespaces | ["All"] | namespaces the controller watches |
tmm_rolling_update | true | pins maxSurge 0 / maxUnavailable 1 |
external_bigip | false | external BIG-IP controller |
external_bigip_login_secret | f5-bigip-ctlr-login | its credentials secret |
cluster_identifier | derived | passed to that controller |
gateway_api_version | 1.5.0 | GATEWAY_API_VERSION on the controller |
demo_mode | false on 2.4 | advanced.demoMode.enabled |
Three of these are worth a sentence each.
tmm_rolling_update is the same shape as the cwc Multi-Attach deadlock: an
unconstrained rolling update on a workload holding a single-attach resource can
wedge, with the old pod holding the volume the new one is waiting for.
gateway_api_version was previously set nowhere, so the controller ran on
whatever the operator defaulted to — v1.4.1 on the verified cluster, against
F5’s 1.5.0. The 2.4 EA guide requires the 1.5 bundle for mTLS. Note that
roksbnkctl still does not install the bundle; see the mTLS caveat above.
demo_mode was being set true on every install, on both lines. On 2.4 it
now defaults to false, matching the reference. On 2.3 it stays true — that is
what has shipped and been exercised there, and changing a working line’s
behaviour without a run to back it would be a guess. Set it explicitly to pin it
on either line.
Every one of these is a config.yaml field, a ROKSBNKCTL_* override, and a row
in Chapter 28 with its line and default.
Hugepages, and why deploymentSize stays Tiny
The defaults aim at a working cluster with no overrides at all. On 2.4 that
means deploymentSize: Tiny, OCP 4.21, and no hugepages — and on ROKS that
is not merely a sensible default, it is the only configuration that runs.
deploymentSize decides how much TMM asks for, hugepages included. Measured on
live 4.21 clusters by changing only that field:
deploymentSize | TMM cpu | TMM memory | hugepages-2Mi |
|---|---|---|---|
Tiny | — | — | none |
Small | 1 | 1536Mi | 4 GiB |
Medium | 4 | 2Gi | 8 GiB |
A stock ROKS worker reports zero, and there is no supported way to change that.
bnk.hugepages does not work on ROKS
This is worth stating plainly, because the setting exists and reads like the answer. It is not. Six approaches were measured and all six fail:
| approach | result |
|---|---|
bnk.hugepages (Node Tuning Operator, [bootloader]) | needs the Machine Config Operator; ROKS has 0 MachineConfigPools |
a Tuned CR with [sysctl] vm.nr_hugepages | ROKS deletes the CR — the create returns a real object, the next read is NotFound |
a privileged pod writing /proc/sys/vm/nr_hugepages | allocates (HugePages_Total: 2048), but kubelet only reads hugepages at startup so the node still advertises 0 |
advanced.tmm.resources with hugepages-2Mi: 0 | TMM schedules, then refuses to start |
advanced.tmm.env.TMM_MEM | accepted, requirement remains |
ibmcloud ks worker-pool | no kernel-argument or sysctl option exists |
IBM manages the worker OS on ROKS, which is what removes the first two. TMM
validates hugepages against the 2 MB page cgroup limit when it starts and exits
if that limit is zero — with exit code 0, so it never looks like a crash:
the pod reports 4/5 Running and the Deployment simply never becomes Available.
Setting bnk.hugepages on ROKS now fails with an explanation naming this,
rather than surfacing as a terraform “provider produced inconsistent result”
error that blames the kubectl provider (issue #203).
So how do you get more capacity?
With more TMM pods on bigger nodes, which is what F5’s own sizing guide
does. Appendix C’s Large cluster reaches ~31 Gbit/s at the Medium profile by
running nine TMM pods on nine larger nodes — capacity comes from pod count and
node size, never from a larger deploymentSize.
cluster:
workers_per_zone: 2
worker_flavor: cx2.16x32
bnk:
cneinstance_size: Tiny # or leave unset — Tiny is the 2.4 default
tmm_replicas: 3 # one TMM per worker node
ROKSBNKCTL_WORKERS_PER_ZONE=2
ROKSBNKCTL_WORKER_FLAVOR=cx2.16x32
ROKSBNKCTL_TMM_REPLICAS=3
All four cluster shapes in Appendix C are verified this way, at 1, 3, 3 and 9 replicas.
cneinstance_size stays configurable rather than hardcoded, deliberately: if
IBM ships a worker profile that supports hugepages, only the platform’s answer
changes and nothing about config.yaml or the ROKSBNKCTL_* surface has to.
What happens if you ask for a size that cannot run
bnk up fails immediately, with the numbers and the alternatives, rather
than waiting fifteen minutes to report Available=False. The scheduler’s
verdict is already in the CNEInstance condition; the wait recognises that no
amount of waiting creates node capacity and stops there.
Tuning the data-plane TCP profile
F5’s approved reference cluster carries a hand-applied sys-default-tcp
F5BigTcpSetting in the BNK namespace. Despite the “sys-default” name it was
applied by engineering rather than created by the product, which is why it is
surfaced here.
bnk:
tcp_settings:
idleTimeout: "300"
congestionControl: high-speed
abc: "true"
ROKSBNKCTL_BNK_TCP_idleTimeout=300
ROKSBNKCTL_BNK_TCP_congestionControl=high-speed
A map rather than 54 config fields, because the CR has 54 of them across bool, int and string. Enumerating each would be 54 rows nobody reads, and it would go stale the moment F5 adds one.
Values are text — a config file and an environment variable cannot carry
anything else — and are coerced on the way out by YAML’s own scalar rules:
300 becomes an integer, true a boolean, and anything else is quoted as a
string. Field names are case-sensitive and used verbatim, because the CR is
case-sensitive and guessing a mapping would silently write a field it ignores.
Setting nothing writes no CR at all. The product manages its own default profile, and emitting an empty one would fight it.
Per-component settings
Field specs for everything below live in Chapter 28 — Configuration reference, which is generated from the struct and test-checked. The
linecolumn there says which release line each field applies to.
The 2.4 CNEInstance takes an environment list for each of its ~26 components. Anything not surfaced as a first-class field is reachable through the passthrough, in YAML or from the environment:
bnk:
advanced:
tmm:
env:
TMM_DEFAULT_MTU: "9000"
cneController:
env:
GSLB_DATACENTER_NAME: "datacenter-eu-gb"
ROKSBNKCTL_BNK_ADV_TMM_ENV_TMM_DEFAULT_MTU=9000
ROKSBNKCTL_BNK_ADV_CNECONTROLLER_ENV_GSLB_DATACENTER_NAME=datacenter-eu-gb
An entry here replaces the default of the same name rather than adding a second copy of it — the CNEInstance spec is read by the lifecycle operator, not by kubelet, so “the last duplicate wins” is not a rule to rely on. That is what makes this the way to change a default rather than only to add to it.
USE_GATEWAY_SETTINGS is deliberately not the example here, though earlier
drafts of this page used it. roksbnkctl sets it on 2.4 itself, because on 2.4
there is no working configuration without it: without the flag the controller
never reads the Infra and GatewaySettings CRs that gateway up applies, and
they sit at Accepted=Unknown forever. You can still override it through this
passthrough — that is what the replace-not-append rule buys you — but you should
not have to.
The component name is matched case-insensitively, and one that roksbnkctl does
not recognize is passed through rather than refused — the component set belongs
to the product, and this tool should not be the reason a newly shipped one is
unreachable.
Checking a 2.4 install
bnk status reports the CNEInstance’s aggregate Available condition plus any
component condition that is not True, which is what turns a timeout into a
diagnosis:
cneinstance: Available=False (ComponentsNotReady)
cneinstance_not_ready: F5TmmAvailable=False
That distinction matters on 2.4. CNEControllerAvailable=True can sit next to
Available=False with TMM at 0/3 and nothing passing traffic.
The gate itself is CNEControllerAvailable on both lines, and deliberately
so: waiting on the aggregate at the gate deadlocks 2.4, because the aggregate
cannot go True until the License CR exists and the License CR is created after
the gate. The aggregate is checked separately, after licensing, which is where
it can actually be satisfied.
If a 2.4 install looks broken in a way that has nothing to do with what you changed, check the
LicenseCR first. The controller refuses to run any resource controller until it is Active, loggingLicense is not enabled.. skip Resource controllers.
gateway status reports the Infra and GatewaySettings CRs alongside the
Gateway, and reads the IPAM-allocated VIP from the Gateway’s status — on 2.4
there is no address to compute, only one to read.
Support status
2.4 is verified against 2.4.0-EA: bnk up reaches all eighteen CNEInstance
conditions with TMM at 5/5 and the licence Active, on a single-NIC OCP 4.20
cluster with connected licensing.
What that run covered, and what it did not, because the difference matters:
| exercised | bnk up on an adopted cluster — manifest 2.4.0-EA, FAR repo.f5.com with the non-GA service account, three workers, single-NIC, connected licensing |
| exercised | the gateway phase — gateway up applies Infra and GatewaySettings, and both reach Accepted, ResolvedRefs and Programmed = True, with the CNEInstance still 18/18 and the licence still Active afterwards |
| exercised | five consecutive clean cluster-lifecycle CLI demo runs on a cluster reproducing F5’s approved reference (6 × cx3d.8x20, OCP 4.21), each a cold start through all six phases |
| not exercised | the CI demo — it is docker-driven and needs its own five runs |
| not exercised | traffic — every check above reads CR conditions, and Accepted / ResolvedRefs / Programmed are all computed without calling IBM Cloud. #313 was a 2.4 install where the CNE controller could not assume its Trusted Profile, so no VPC address prefixes and no routes were created: all four CRs reported Programmed=True and nothing passed traffic in either direction. A condition-based verification cannot see that class of defect |
| not exercised | mTLS, multi-NIC, disconnected licensing, EgressGateway CRs, and the contract boundary |
Every conformance setting was confirmed on the cluster, not inferred from a
plan: deploymentSize: Tiny, tmmReplicas: 3, wholeCluster: false paired with
watchNamespaces: ["All"], pod placement present, demoMode off,
GATEWAY_API_VERSION: 1.5.0, and the three TMM environment settings.
The teardown namespace repair fired in all five runs and in every teardown between them. That is worth stating plainly: the 2.4 finalizer deadlock is not intermittent, so without the repair a 2.4 cluster cannot be reinstalled into at all.
Running the gateway phase is what found the three defects that made it
impossible before it was run: the gateway plan failed outright on 2.4, nothing
set USE_GATEWAY_SETTINGS so no controller ever reconciled the CRs, and the
cneinstance_advanced_env override meant to let you set it by hand was read by
no terraform at all. None of the three was visible in the source — the third
least of all, since the variable’s name appears everywhere it should.
2.4 EA is single-NIC throughout — one NetworkAttachmentDefinition, one
external-vlan network in the Infra CR, and no second data-plane interface
anywhere. An earlier expectation that 2.4 would bring multi-NIC is withdrawn, and
the verified run was single-NIC, so it is evidence for single-NIC and nothing
else.
support_matrix.yaml records the same thing in the same terms. Its rule is worth
repeating, because it is what keeps that file useful: a cell with no coverage is
a claim nobody has checked.
Tearing down
Three-phase note. With the Cluster / BNK / Testing split there is now a fourth destroy verb,
roksbnkctl testing down(destroys the jumphost rig, leaving the cluster and BNK intact), and the bareroksbnkctl downtears down BNK || Testing in parallel, then the cluster.cluster downrefuses while either BNK or Testing state exists. The decision tree below is written for the earlier two-phase shape; see Chapter 8a §“Teardown ordering and thecluster downguard” for the three-phase ordering and guard.
roksbnkctl down, roksbnkctl bnk down, and roksbnkctl cluster down are the three destroy verbs — the inverses of up, bnk up, and cluster up respectively. This chapter covers what each one removes, the ordering constraint between them, the refusal messages you’ll hit if you ask for the wrong one, what survives a destroy, the --auto flag for non-interactive runs, and the workspace-cleanup story.
The phase-aware decision tree
Which verb do you want? The shape of your workspace and your intent both matter. Start here:
I want to keep the cluster and just tear down the BNK trial:
→ roksbnkctl bnk down
I want to tear down everything (cluster + trial):
→ roksbnkctl down
I want to tear down only the cluster (no trial currently deployed):
→ roksbnkctl cluster down
The big rule, stated up front: destroy in reverse of create. Trial first (bnk down), cluster second (cluster down). The unscoped roksbnkctl down does this ordering for you — it runs the trial destroy first and then the cluster destroy. You don’t have to think about ordering; down is the safe default.
The phase-scoped commands (bnk down, cluster down) are the precision tools — they let you keep one phase across many cycles of the other. They also refuse loudly if you ask them to do something that would orphan resources or that the shape doesn’t allow. The full refusal catalogue is in §“Refusal messages catalogue” below; the rule of thumb is that the error message always names the verb that would actually work.
The three destroys
There are three teardown verbs matching the three slices of state:
roksbnkctl down — shape-aware composite
The unscoped down is a shape-aware composite: it detects the on-disk shape of the workspace and dispatches to the right phase destroys in the right order.
roksbnkctl down
| Workspace shape | down does |
|---|---|
| Split (cluster + trial) | trial destroy → cluster destroy |
| ClusterOnly (only cluster applied) | cluster destroy |
| Empty | no-op success: ✓ Nothing to destroy in this workspace — no phase has any state. |
This is the safe default — down always does the right thing regardless of shape.
roksbnkctl bnk down — destroy the BNK trial only
Tears down everything the trial phase created — the flo / cis Helm releases, the cne_instance and license custom resources, and the cluster-side namespaces / Secrets / ServiceAccounts / RoleBindings / SCC bindings — and leaves the cluster running. These are all real terraform resources (helm_release, kubernetes_*, and alekc/kubectl kubectl_manifest), so bnk down is an ordinary terraform destroy that deletes the CRs finalizer-aware.
roksbnkctl bnk down
What survives:
- The ROKS cluster itself
- cert-manager
- The registry COS instance and its bucket contents (FAR images, license artefacts)
- The TGW jumphost
- All cluster-phase Terraform state under
state-cluster/ cluster-outputs.json(the cluster is still registered)- The workspace’s
config.yaml
Roughly 41 resources destroyed on a clean trial-only bnk down. Time is dominated by Helm’s pre-delete hooks and the cne_instance finaliser unwind — usually 2-5 minutes total.
bnk down is a no-op success (exits 0, “nothing to do”) on Empty and ClusterOnly workspaces — there’s no trial state to destroy, so it leaves the cluster alone and returns cleanly. This lets a tool like bnk-forge call bnk down unconditionally in a reverse-order teardown without a missing-trial error blocking the cluster destroy. See §“Refusal messages catalogue” for the exact text.
roksbnkctl cluster down — destroy the cluster phase
Tears down the cluster + cluster-shared services: the ROKS cluster, transit gateway, registry COS instance, cert-manager Helm release, and the TGW jumphost.
roksbnkctl cluster down
What survives:
- The workspace’s
config.yaml ~/.roksbnkctl/<workspace>/state/(now empty of resources but the directory persists)~/.roksbnkctl/<workspace>/state-cluster/Terraform state files (the cluster-side state itself is empty; the directory andterraform.tfstatepersist)
Roughly 36 resources destroyed. The ROKS cluster destroy alone is 5-10 minutes; everything else is fast.
The post-destroy cleanup deletes cluster-outputs.json automatically — the workspace no longer has a registered cluster.
Order matters: trial first, then cluster
The upstream HCL’s resource graph requires this ordering. The trial-phase resources have implicit dependencies on cluster-phase resources (they live in the cluster, after all), and Terraform’s destroy graph traverses dependencies in reverse. If the cluster phase tries to destroy first, the trial phase’s resources are still there — finalisers block the destroy of the cluster’s namespaces, the cluster-side SCC bindings reference SCCs that are in the way, and so on.
roksbnkctl cluster down enforces this ordering with a hard refusal: if the trial state has any resources in it, cluster down errors out and points you at bnk down (or down) instead. Even --auto won’t bypass the guard, because correctness, not confirmation, is the issue. The full refusal text:
$ roksbnkctl cluster down
BNK trial state exists in this workspace; run `roksbnkctl bnk down` first
(or `roksbnkctl down` to tear down both phases)
So in practice, always destroy the trial before the cluster. The unscoped down does this ordering for you on a split workspace; the phase-scoped pair is bnk down then cluster down.
The clean teardown sequence — split workspace, explicit phase commands:
# 1. Destroy the BNK trial
roksbnkctl bnk down --auto
# 2. Now safe to destroy the cluster phase
roksbnkctl cluster down --auto
# 3. (Optional) Delete the workspace itself
roksbnkctl ws delete <name> --force
Or the one-shot equivalent:
# 1. Tear down both phases in order
roksbnkctl down --auto
# 2. (Optional) Delete the workspace itself
roksbnkctl ws delete <name> --force
If you roksbnkctl up against a registered cluster (one you didn’t cluster up yourself), step 2 doesn’t apply — the cluster wasn’t yours to destroy. Just bnk down the trial and stop there, then optionally unregister by deleting cluster-outputs.json.
SSH key cleanup. If
initgenerated a testing SSH key and you accepted the prompt to copy it into~/.ssh/,ws deleteremoves those copied files too, so a generated key doesn’t outlive its workspace. It deletes only the filesinitactually created (recorded inresources.copied_ssh_key_files) — a pre-existing~/.sshkey with the same name is never touched. The confirmation prompt names the files it will remove.
roksbnkctl cleanup — recovering from a failed down
terraform destroy is not always clean. A transient IBM Cloud API error, a resource stuck behind a finaliser, or a down you Ctrl-C’d partway can leave orphaned cloud resources that Terraform no longer tracks — and a re-run of down may not finish the job. The classic symptom is a follow-up up that fails with … name is not unique because a half-deleted security group or VPC is still there.
roksbnkctl cleanup is the recovery tool. It sweeps your IBM Cloud account for every resource named after the workspace prefix (<prefix>-*) and deletes them in dependency order — regardless of what Terraform state says.
roksbnkctl cleanup # scan, list, confirm, delete
roksbnkctl cleanup --dry-run # scan + list only, delete nothing
roksbnkctl cleanup --auto # skip the confirmation prompt
What it deletes
Everything whose name is the workspace prefix or a <prefix>- child, across:
- VPC (per region): virtual server instances (jumphosts), floating IPs, public gateways, subnets, security groups, SSH keys, and VPCs.
- Transit Gateway (and its connections).
- The registry COS instance.
- The ROKS cluster itself.
- The BNK trusted profile (named after the cluster ID — found via
cluster-outputs.json, so it’s still identifiable after the cluster is gone).
Deletion runs in reverse-dependency order (instances → floating IPs → public gateways → subnets → security groups → VPCs → Transit Gateway → registry COS → cluster → trusted profile) so children go before their parents. It is best-effort: a failure on one resource is reported and the sweep continues. Re-run cleanup for anything blocked on an async delete (e.g. a VPC that can’t go until its cluster finishes terminating).
What it will NOT delete: resources this workspace adopted
A name match cannot tell a resource roksbnkctl created from one the
workspace adopted. They look identical, and the adopted name usually matches
the prefix exactly — a workspace with prefix sm-cli adopting
sm-cli-registry-cos is the motivating case.
So cleanup reads the workspace’s adopt decisions and refuses to delete what
they name. Adopted resources are still listed, separately, with the config key
that spared them:
2 resource(s) match the prefix but are ADOPTED by this workspace and will NOT be deleted:
KIND NAME REGION ADOPTED VIA
cos_instance sm-cli-registry-cos resources.registry_cos.existing
cluster sm-cli us-east cluster.name (create: false)
(roksbnkctl never created these; remove the config key if you want cleanup to own them)
The protected set is every path where roksbnkctl reads a resource it never
creates:
| Config key | Kind |
|---|---|
cluster.name with create: false | the ROKS cluster |
resources.transit_gateway.existing | transit gateway |
resources.registry_cos.existing | COS instance |
resources.client_vpc.existing | VPC (by name) |
resources.cluster_vpc.existing | VPC (by ID) |
resources.testing_ssh_key_name | SSH key |
The SSH key has no create toggle because it is never created: the testing
module reaches it only through a data source, so whenever it is named, it is
adopted.
This is why
--autois safe to schedule. Protected resources are removed from the set before the confirmation step, so the delete loop never sees one.--autoskips the prompt, not the protection.
Note that terraform was never the danger here. Adopted resources are read
through data sources, and data sources are never destroyed — so down was
always safe. It is cleanup’s independent sweep that can reach them, precisely
because it does not consult Terraform state at all. That independence is the
feature (it catches what state lost) and was the defect (it caught what state
never owned).
How the Transit Gateway is handled
A gateway cannot be deleted while anything is attached to it, so cleanup detaches its connections first, waits for them to actually clear, and only then deletes the gateway. The detach is asynchronous — IBM leaves the connection in deleting for a few seconds — and deleting the gateway during that window fails with:
412 Precondition Failed: Before you can delete this gateway,
you must delete all attached connections.
Which connections it may detach is a deliberate limit, not an oversight:
- A connection to a VPC this same sweep is deleting is yours;
cleanupremoves it. - Anything else — a VPC under a different prefix, another account’s network, a Direct Link or GRE attachment — belongs to someone else.
cleanuprefuses the gateway and names what is attached, rather than quietly disconnecting a shared gateway’s other tenants.
Connections mid-transition are a third case, and the two directions are not the same:
| status | meaning | what cleanup does |
|---|---|---|
deleting / detaching | departing — going away | waited for, whoever started it. It will be gone either way, so it is never a reason to refuse the gateway |
pending | arriving — still attaching | ownership decides: yours is waited for; someone else’s is refused straight away |
IBM accepts a connection DELETE only from a settled state, so acting on either would fail with 409 invalid_state. The pending case matters more than it sounds: cleanup is most often reached for right after an interrupted up, which is exactly when a connection is still attaching. Sweeping then used to fail and advise a re-run that failed identically until IBM finished attaching — a wait the sweep now does itself.
Ownership is checked before that wait, because it cannot change as a connection settles. A gateway attached to a network outside the sweep is refused on the first listing rather than after the timeout.
If a connection never finishes attaching, the sweep says so rather than timing out twice on something it never had a chance to delete:
transit gateway f5orph-tgw still has a connection attaching after 5m0s:
conn-1 [vpc crn:…, pending] — IBM refuses a DELETE from `pending`, so this
has to finish before the gateway can go. Re-run once it reports `attached`
States that are settled — including failed and suspended — are detached normally. Waiting for attached specifically would mean a broken connection was never cleaned up, which is the opposite of what a sweep is for.
A refusal is not a transient, and an identical re-run will not clear it. Check the region first: cleanup scans only the workspace’s cluster and client regions by default, so the commonest reason a VPC looks foreign is that it is yours and simply was not scanned. Re-run with --all-regions (or --region <name>) and it comes into scope. If the network really is someone else’s, detach it yourself (ibmcloud tg connection-delete <gateway-id> <connection-id>) or delete it, then sweep again.
This matters most for the shared-gateway topology in Sharing a Transit Gateway, where one gateway deliberately outlives the clusters attached to it.
Which regions it scans
By default cleanup scans the workspace’s cluster region plus the testing-client region (from config.yaml’s resources.client_region and cluster-outputs.json). If resources landed in a region not recorded in config, add it explicitly or sweep everything:
roksbnkctl cleanup --region ca-tor # add a specific region to the scan
roksbnkctl cleanup --all-regions # sweep every IBM Cloud region (slower)
Caveats
- It matches purely on the
<prefix>-name convention. If unrelated resources happen to share the workspace prefix, they match too — always review the--dry-run(or the pre-delete confirmation) list first. - It needs a prefix. A workspace with no configured prefix has nothing to match;
cleanupreports there’s nothing to sweep. - It does not touch Terraform state or
config.yaml. After a successful sweep the state files still reference the now-gone resources; runroksbnkctl ws delete <name> --forceto drop the workspace, or re-upto rebuild.
A typical recovery — a down errored, and up now refuses with “not unique”:
roksbnkctl cleanup --dry-run # confirm what's stranded
roksbnkctl cleanup --auto # sweep it
roksbnkctl up --auto # rebuild clean
Refusal messages catalogue
The phase-scoped destroy verbs refuse loudly when the shape doesn’t allow what you’ve asked for. Every refusal names the verb that would actually work. If you hit one in the wild, grep your terminal output for the message text and you should land here:
| Command + shape | Refusal text | Resolution |
|---|---|---|
bnk down on Empty or ClusterOnly | (not a refusal — exits 0) ✓ No BNK trial state to destroy in this workspace — nothing to do. | No-op success — no trial is deployed, so the command returns cleanly and leaves the cluster alone. If you want to destroy the cluster, use roksbnkctl cluster down. |
testing down with no jumphosts | (not a refusal — exits 0) ✓ No testing jumphost state to destroy in this workspace — nothing to do. | No-op success — the testing phase is optional, so down returns cleanly when nothing was provisioned. |
gateway down with no gateway state | (not a refusal — exits 0) ✓ No gateway phase state to destroy in this workspace — nothing to do. | No-op success — the gateway phase is opt-in, so down returns cleanly when it was never applied. |
cluster down on Split | BNK trial state exists in this workspace; run `roksbnkctl bnk down` first (or `roksbnkctl down` to tear down both phases) | Run bnk down first to remove the trial, then cluster down for the cluster — or roksbnkctl down to do both in one shot. |
cluster down on Empty | (not a refusal — exits 0) ✓ No cluster state to destroy in this workspace — nothing to do. | No-op success — the cluster hasn’t been provisioned. |
down on Empty | (not a refusal — exits 0) ✓ Nothing to destroy in this workspace — no phase has any state. | No-op success — no phase has any state. |
any down on an uninitialised workspace | workspace "<name>" is not initialised; run `roksbnkctl init` first | A real error, unlike the empty case above. The two are indistinguishable to the state check but mean opposite things: a mistyped -w reporting a successful teardown is how someone concludes a cluster came down while it is still running. |
What survives a destroy
The contract: roksbnkctl never destroys local state without explicit consent, and never destroys cloud resources outside its Terraform state.
After a successful down:
| Survives | Where |
|---|---|
| Workspace config | ~/.roksbnkctl/<name>/config.yaml |
| Workspace directory + state files | ~/.roksbnkctl/<name>/ (empty state/; state-cluster/ untouched if cluster down not run) |
| OS keychain entry for the API key | per-workspace, named roksbnkctl/<name>/ibmcloud_api_key |
~/.kube/config | left in place |
| The cluster (if only trial was destroyed) | runs and bills as before |
| The registry COS bucket’s contents | FAR images, JWT licenses, schematic state — survive cluster destroy too if the bucket was created outside the bundled HCL |
~/.roksbnkctl/known_hosts | SSH host keys persist; deleting a workspace does not clear them |
Re-running up against a down’d workspace re-creates everything from scratch. The workspace’s config.yaml is preserved precisely so this re-create can use the same inputs without re-prompting.
The COS bucket point is worth highlighting: the bundled HCL provisions the COS instance but generally does not provision the buckets inside it (those are written by post-apply provisioners or by the BNK runtime itself). When cluster down destroys the COS instance, the bucket goes with it — but if the COS instance was created out-of-band (e.g. by a registered cluster’s owner) and roksbnkctl is just attaching, then cluster down doesn’t apply and the COS survives.
--auto for non-interactive runs
All three destroy commands prompt for confirmation by default. On a Split workspace the composite down takes a single up-front confirmation that names both phases, then runs trial → cluster without re-prompting — so you can’t accidentally answer “yes” to the trial and miss the cluster gate:
$ roksbnkctl down
This will destroy BOTH the BNK trial AND the cluster phase for workspace "default" (ROKS + transit gateway + registry COS + cert-manager + jumphost).
Continue? [y/N]:
On LegacySingle the monolithic prompt is unchanged (one state, one destroy):
$ roksbnkctl down # LegacySingle shape
This will destroy workspace "default"'s resources.
Continue? [y/N]:
$ roksbnkctl bnk down
This will destroy workspace "default"'s resources.
Continue? [y/N]:
$ roksbnkctl cluster down
This will destroy the cluster phase for workspace "default" (ROKS + transit gateway + registry COS + cert-manager + jumphost).
Continue? [y/N]:
--auto skips the prompt — required for CI / scripted pipelines:
roksbnkctl down --auto
roksbnkctl bnk down --auto
roksbnkctl cluster down --auto
--auto does not override the shape-based refusals (see §“Refusal messages catalogue” above) — those are correctness guards, not confirmation prompts. If trial state is present, cluster down --auto still refuses.
Like up, transient errors retry
down doesn’t share up’s explicit retry-on-transient-error logic, but Terraform’s destroy is naturally idempotent: re-running down after a partial destroy picks up where the previous run left off. If you see a transient network error during destroy, just re-run:
roksbnkctl down --auto
# (some resources destroyed, then transient error)
roksbnkctl down --auto
# (picks up where it left off, completes)
The same applies to cluster down. ROKS cluster destroy specifically can take longer than expected when the master is propagating its delete state — wait a few minutes and re-try if you see master-not-found errors.
Cleaning up workspaces
A successful down leaves the workspace directory in place. You usually want to clean that up too:
roksbnkctl ws delete <name> --force
One safety rail on ws delete: it refuses if Terraform state still lists resources (unless --force) — which catches the case where you forgot to run down first. You can delete the current workspace directly; the pointer moves to another existing workspace (or clears when none remain), so you don’t need to switch away first (see Chapter 6).
The --force flag overrides the state check — but if you ws delete --force a workspace that still has provisioned cloud resources, you’ll have stranded them. Recover with roksbnkctl cleanup: run it before deleting the workspace (it reads the prefix from config.yaml), or recreate the workspace’s config.yaml with the same prefix and run it after.
The full clean-as-you-go teardown (scripts/e2e-test.sh Phase D destroys; Phase H deletes):
# 1. Destroy the trial
roksbnkctl down --auto
# 2. Destroy the cluster phase
roksbnkctl cluster down --auto
# 3. Delete the workspace — even if it's the current one (the pointer auto-switches)
roksbnkctl ws delete default --force
Cost note: an undestroyed cluster keeps billing
ROKS clusters bill at roughly $0.30/hour per cluster + worker pool — call it $7/day for a 2-worker cluster, plus a few cents/day for the VPC / load balancers / COS / jumphost. A forgotten cluster can rack up real cost over a weekend.
To verify what’s still running in your account:
- IBM Cloud console → Kubernetes → Clusters — every cluster, billing or not.
- IBM Cloud console → VPC Infrastructure → VPCs — networks left over after a partial destroy.
- IBM Cloud console → Resource list — exhaustive view of everything in the account, filterable by RG.
If you find a leaked cluster from a past roksbnkctl run, the right move is to re-attach to it via roksbnkctl cluster register <name> and then cluster down --auto — roksbnkctl cleans up cleanly when it has the cluster in its state. Manually deleting via the console works too but leaves dangling VPCs and security groups that the bundled HCL would have cleaned up.
roksbnkctl status and roksbnkctl cluster show both report the cluster identity recorded in cluster-outputs.json, but they don’t probe for “are there other clusters in this account?” — that’s deliberately not their job. The IBM Cloud console is the canonical source of truth for what’s billing.
Workspace deletion ≠ destroy
A subtle but important distinction. roksbnkctl ws delete removes the local workspace directory and the OS-keychain API key entry. It does not destroy any cloud resources. If you ws delete --force without first running down / cluster down, the cloud resources keep running and you’ve lost the local Terraform state that roksbnkctl would use to destroy them.
In that scenario, recovery is:
- Find the leaked cluster in the IBM Cloud console.
- Recreate the workspace:
roksbnkctl init -w recovery. - Register the existing cluster:
roksbnkctl cluster register <leaked-cluster-name>. - Then run
roksbnkctl cluster down --autoto destroy it cleanly.
The Terraform state is regenerated implicitly during register + plan; the resources roksbnkctl would otherwise have tracked get re-discovered through the IBM SDK lookups. It’s not seamless, but it’s recoverable.
The ws delete --force flag’s “still has resources” check exists exactly to prevent this scenario — don’t bypass it without thinking about the consequences.
Worked example: register an existing cluster, deploy BNK, tear down
End-to-end Part III scenario: somebody on your team already provisioned a ROKS cluster manually via the IBM Cloud console (or via a different terraform tree); you need to deploy BNK on top of it using roksbnkctl, validate, and tear the whole thing down cleanly. The flow exercises Chapter 9, Chapter 10, and this chapter end-to-end.
# 1. Workspace bootstrap — same as a fresh deploy
roksbnkctl init -w preexisting
# (answer prompts for region + resource group; pick the values matching
# the existing cluster's location)
# 2. Register the already-running cluster into the workspace
roksbnkctl cluster register existing-bnk-cluster -w preexisting
# Expected:
# → Discovering cluster "existing-bnk-cluster" via IBM Cloud API ...
# ✓ Cluster ID: <crn>
# ✓ Wrote ~/.roksbnkctl/preexisting/cluster-outputs.json
# ✓ Fetched admin kubeconfig to ~/.kube/config (chmod 0600)
# 3. Verify roksbnkctl sees the cluster
roksbnkctl status -w preexisting
# Expected: cluster Ready, workers count, no BNK pods yet
# 4. Deploy BNK on top — `up` is idempotent over the existing cluster
roksbnkctl up --auto -w preexisting
# Expected: terraform applies the cert-manager + flo + cne_instance +
# license modules only; the roks_cluster module sees the cluster already
# exists and skips. ~10-15 min vs ~50 min for a from-scratch up.
# 5. Validate
roksbnkctl test -w preexisting
# Expected: green across connectivity + dns
# 6. Tear down — destroys the BNK overlay; the registered cluster survives
roksbnkctl down --auto -w preexisting
# Expected:
# → terraform destroy (auto-approved)
# Destroy complete! Resources: N destroyed.
# ✓ Workspace "preexisting" state retained at ~/.roksbnkctl/preexisting/
The destroy count N is the BNK overlay + jumphost only — typically 30-40 resources, not the from-scratch ~77 count. cluster register is a discovery-only path: terraform state holds the overlay modules (cert_manager, flo, cne_instance, license) and the testing jumphost, but not the roks_cluster module, because the cluster pre-existed roksbnkctl. down destroys only what terraform knows about, so the registered cluster survives untouched.
If you also want to release the underlying cluster, you have to tear it down through whatever provisioned it originally (the IBM Cloud console, or the separate terraform tree your teammate used). roksbnkctl cluster down only works against clusters roksbnkctl cluster up created in the first place — see Chapter 8 for the cluster-phase boundary.
The full register → up → test → down loop above is what Phase E + Phase H of the e2e plan exercise; see Chapter 23 for the CI version.
Cross-references
- Chapter 6 — Workspaces —
ws deletemechanics and how the current-workspace pointer auto-switches on delete. - Chapter 8 — The cluster phase — what
cluster upprovisions andcluster downremoves. - Chapter 9 — Registering an existing cluster — the
cluster registermechanics the walkthrough builds on. - Chapter 10 — Deploying BNK trials — what
upprovisions anddownremoves. - Chapter 26 — Troubleshooting — recovery from partial-destroy and orphan-state scenarios.
Workspace config (config.yaml)
This is the teaching chapter. The field specs live in Chapter 28 — Configuration reference. Chapter 28 is generated from the
Workspacestruct and checked by a test, so it cannot drift: go there for a field’s name, type, default, and which BNK line it applies to. This chapter explains what each block is for, which combinations make sense, and the judgement calls a schema table cannot express.
This chapter is the guided tour of the per-workspace config.yaml. If you’ve read Chapter 6 — Workspaces you’ve seen the on-disk layout; this chapter zooms in on the YAML file that drives everything else (init, up, down, cluster up, the test suite, the SSH targets, the execution backends).
You don’t usually edit this file by hand. roksbnkctl init generates it interactively; later runs read it. But because every other knob in the tool reads from here, it’s worth knowing what each block means and what happens when you leave one out. Every section below links to the part of Chapter 28 that lists its fields; what you get here is the reasoning around them.
File location
Each workspace’s config lives at:
~/.roksbnkctl/<workspace>/config.yaml
Override the base directory with the ROKSBNKCTL_HOME env var (test fixtures use this; everyday users shouldn’t need it). The file is created mode 0600, inside a workspace directory created 0700 — owner-only, because this file can hold ibmcloud.api_key_b64 and registry.generic_password_b64, and base64 is obfuscation rather than encryption.
Workspaces created by a release before v1.50.0 are on disk at 0644/0755. They are tightened in place the first time roksbnkctl reads them, and it says so once:
⚠ workspace "prod" was readable by other users on this host; tightened 3 path(s) to owner-only.
config.yaml can hold your IBM Cloud API key and registry password. If this host is shared, rotate them.
The repair fixes the mode from that point on; it does not undo the exposure that already happened. On a shared host, rotate the API key. roksbnkctl doctor reports the current state as the workspace permissions check, and is the only thing that reports a tree that could not be tightened (a read-only mount, or a filesystem with no POSIX modes). On Windows the check is skipped: Go’s Chmod there only toggles the read-only bit, so there is no owner-only mode to assert.
There’s also a global ~/.roksbnkctl/config.yaml at the top level — it holds the current_workspace pointer and other user-wide preferences. That’s a different file with a different schema; this chapter is about the per-workspace one.
When it gets written
| Action | Effect on config.yaml |
|---|---|
roksbnkctl init | Creates the file from interactive prompts. Existing file? Asks before overwriting. |
roksbnkctl init --upgrade-tf | Updates tf_source: only; leaves every other field alone. |
roksbnkctl targets add <name> ... | Adds an entry under targets:. |
roksbnkctl targets remove <name> | Removes the entry. |
roksbnkctl up (post-apply) | Auto-populates targets.jumphost if the upstream HCL emitted a TGW jumphost output. |
| Anything else | Reads the file. Doesn’t write back. |
Direct hand-editing is supported (the file is plain YAML) but discouraged for fields that have dedicated commands — adding an SSH target via roksbnkctl targets add keeps the schema validation in one place.
Top-level structure
prefix: acme-eu # base for every account-scoped resource name
ibmcloud: # IBM Cloud account + auth
region: ca-tor
resource_group: default
api_key_source: keychain
# api_key_b64: <base64-of-api-key> # OPTIONAL fallback when keychain unavailable
cluster: # ROKS cluster identity
create: true
name: acme-eu # equals prefix when init derived the name
openshift_version: "4.20"
workers_per_zone: 2
resources: # per-resource create/adopt toggles
transit_gateway: { create: true }
registry_cos: { create: true }
cert_manager: { create: true }
bnk: { create: true }
tgw_jumphost: { create: true }
cluster_jumphosts: { create: false }
client_vpc: { create: false }
bnk: # BNK trial knobs (optional; falls through to upstream HCL defaults)
cneinstance_size: Small
far_repo_url: repo.f5.com
manifest_version: 2.3.0-3.2598.3-0.0.170
test: # test-suite tuning (optional)
throughput:
duration: 30
streams: 8
connectivity:
extra_hosts:
- https://my.gslb.example.com
tf_source: # where the Terraform HCL comes from
type: embedded # embedded | github | local
targets: # SSH targets (see Chapter 15)
jumphost:
host: 169.45.91.177
user: ubuntu
key_source: tf-output:jumphost_shared_key
exec: # per-tool execution backend defaults (see Chapter 17)
ibmcloud: { backend: local }
iperf3: { backend: k8s }
terraform: { backend: local }
cos: # optional COS supply-chain config
instance: bnk-supply-chain
bucket: bnk-artifacts
Every block except ibmcloud:, cluster:, and tf_source: is optional. Omit a block and the tool falls through to either a documented default or the upstream HCL’s own default for terraform variables. The prefix: field and resources: block are also optional — omit them and the workspace renders a sparse terraform.tfvars that falls through to the upstream module default names.
That sketch is not the whole schema. A workspace can also carry gateway: (the Gateway API resources installed after BNK), registry: (an air-gap mirror), state: (remote Terraform state — Chapter 12a), bnkforge: and agent:. Each has its own chapter, and every field of every block is listed in Chapter 28. The sections below walk the blocks you meet first.
prefix:
prefix: acme-eu
The field spec is Chapter 28 §Workspace.
The init interview asks for a workspace prefix and derives every account-scoped resource name from it — acme-eu becomes cluster acme-eu, VPC acme-eu-cluster-vpc, TGW acme-eu-tgw, COS acme-eu-registry-cos, jumphosts acme-eu-jh-tgw / acme-eu-jh-<zone>. This stops two workspaces in the same IBM Cloud account from colliding on the old shared tf-* default names.
A prefix must be a lowercase DNS-style label — start with a letter, then [a-z0-9-], no trailing hyphen — and 35 characters or fewer. The 35-char cap is the ROKS cluster-name limit, the tightest of all the resource types. roksbnkctl validates the prefix — and every name it derives — at init time and re-prompts on overflow; there is no silent truncation. An empty prefix keeps the legacy sparse render (no name variables emitted at all), so old configs are unaffected. The full derivation table, the per-resource length/charset limits, the source citations, and the override path live in Chapter 13 §“Resource naming & collision avoidance”.
ibmcloud:
ibmcloud:
region: ca-tor
resource_group: default
api_key_source: keychain
api_key_b64: ""
The field spec is Chapter 28 §IBMCloudCfg.
This block is the account context every phase runs against. region is the one field with no default worth having — it decides where the cluster, the VPCs and the COS instance are created, and there is no sensible guess, so init prompts for it and a programmatic load of a config without it fails rather than picking one.
api_key_source pins the credential resolver to a single source. Leaving it empty is the normal case: the resolver then walks its whole chain (env → keychain → the workspace’s own api_key_b64 → a hidden TTY prompt). Pin it when you want a failure rather than a fallback — a CI runner that must use the environment variable and must not silently prompt, say.
api_key_b64 is the last-resort inline copy of the key, for hosts with no usable OS keychain (WSL2 without libsecret is the common one). Base64 is obfuscation, not encryption: anyone holding the file can decode it instantly, so treat config.yaml as a plaintext credential — mode 0600, never committed, rotated if the host is shared. The plaintext field name api_key: is rejected at load time; roksbnkctl refuses to read a workspace config that contains it, and the encoded form is the only inline path. Full discussion in Chapter 14 — Credentials and the resolver chain.
cluster:
cluster:
create: true
name: tf-openshift-cluster
openshift_version: "4.20"
workers_per_zone: 2
public_gateway: true # optional; false = private/disconnected cluster (no egress)
min_worker_vcpu_count: 16 # optional; worker-flavor auto-select floor
min_worker_memory_gb: 64 # optional; worker-flavor auto-select floor
vpc_cidr: 10.242.0.0/16 # optional; address block for a NEW cluster VPC
network_mode: single-nic # optional; single-nic (default) | multi-nic
The field spec is Chapter 28 §ClusterCfg. The rest of this section is the reasoning behind the fields that have some.
Create or adopt. create: true builds a ROKS cluster; create: false adopts one that already exists, looked up by name (a cluster name or ID) in ibmcloud.resource_group. Adopting is the common case: most estates build clusters to their own standards and ask roksbnkctl only for BNK. The cluster is deliberately not part of the resources: block below; it kept its own pair of fields because it predates that block and because a cluster is the one thing a workspace cannot do without. When a prefix is set, init fills cluster.name with the prefix itself — the cluster name carries no suffix, for the reason given in Chapter 13.
Quote the OpenShift version. openshift_version: 4.20 is a YAML float, and 4.20 as a float is 4.2. Write openshift_version: "4.20". Leaving it empty is legitimate and takes IBM’s current default, which moves over time — pin it when a run has to be reproducible.
workers_per_zone is per zone. ROKS spans three availability zones in an MZR region, so 2 is a six-node cluster.
public_gateway: false builds a disconnected cluster, and that is a bigger decision than one boolean. It removes the public gateway from every cluster subnet, so the workers have no Internet egress at all. Everything they used to pull now has to be reachable privately: a mirror registry they can route to, VPEs or private service endpoints for the IBM Cloud services the install touches, and disconnected (or F5 License Proxy) licensing instead of the connected JWT path. Get one of those wrong and the failure shows up much later, as an image pull that hangs. Chapter 10a §“A truly disconnected cluster” is the checklist. Note the scope: this governs worker egress only — the cluster master keeps its public API endpoint, so kubectl still works from your desk.
Worker sizing. Leave worker_flavor unset and the cluster module auto-selects the smallest profile meeting min_worker_vcpu_count and min_worker_memory_gb. The auto-select only considers the bx2 family, so any other profile is unreachable without naming it in worker_flavor — F5’s approved reference cluster runs cx3d.8x20, which no combination of minimums can produce.
vpc_cidr is what lets two clusters share a Transit Gateway. Left empty, the VPC uses IBM’s auto address-prefix management, which hands every VPC in a region the same three prefixes. Two roksbnkctl-built clusters in one region then overlap, and a Transit Gateway joining them cannot route to both — it blackholes one, and what you see is intermittent image-pull timeouts rather than a routing error (issue #46). Give each cluster its own block whenever they must share a gateway, which is the norm for disconnected installs, since the cluster reaches its private mirror across that gateway. The block is split into three per-zone prefixes of /n+2, and each cluster subnet is the first /n+8 of its zone — so a /16 gives 256-address subnets (today’s size), a /17 gives 128, and a /18 gives 64. /18 is therefore the smallest value Terraform accepts, and /16 is the one you want unless address space is tight. It is meaningful only for a created VPC; an adopted VPC keeps its own prefixes, and changing the value after the fact warns rather than silently re-planning.
existing_subnet_ids is the other half of “bring your own network”. Adopting the VPC alone is not enough: in an estate that allocates address space centrally, the subnets carry the ACLs and routing that make them acceptable, and a cluster dropped into freshly-created subnets sits outside all of it. Supply one subnet ID per zone, in zone order. Each subnet’s zone is read from the subnet itself, so a reordered list places the cluster differently rather than failing — check the order. It requires resources.cluster_vpc: {create: false, existing: <vpc-id>}, because a subnet cannot be adopted independently of the VPC containing it. ROKSBNKCTL_EXISTING_SUBNET_IDS sets it from the environment, comma-separated.
network_mode is create-time only, and enforced. Omitted means single-nic, which is what every cluster this tool builds is; the field exists to name that, not to change it. multi-nic requires BNK 2.4+. There is no in-place conversion between the two — Terraform would plan a replacement of a running cluster — so a workspace whose network_mode contradicts the cluster recorded in cluster-outputs.json is refused rather than planned. See Chapter 28 §ClusterCfg for which line each value belongs to.
The cluster: block translates to the terraform variables create_roks_cluster, openshift_cluster_name, roks_cluster_id_or_name, openshift_cluster_version, roks_workers_per_zone, cluster_public_gateway, cluster_vpc_cidr and cluster_network_mode — see Chapter 13 and Chapter 29 for the full mapping.
resources:
resources:
transit_gateway: { create: true }
registry_cos: { create: true }
cert_manager: { create: true }
bnk: { create: true }
tgw_jumphost: { create: true }
cluster_jumphosts: { create: false }
client_vpc: { create: false, existing: shared-client-vpc }
client_region: ca-tor # testing-client region (plain string)
The init interview’s create/adopt answers, one {create, existing} pair per resource. create: true provisions a new, prefix-named resource; create: false declines it, and when a still-enabled resource depends on the declined one, existing: names the pre-existing resource to consume instead.
The field spec — every toggle, plus the jumphost sizing and security-group keys that also live in this block — is Chapter 28 §ResourcesCfg and §ResourceToggle. What the table below adds is the mapping: which terraform variables each toggle drives, and when declining one makes init ask for an existing resource to adopt instead.
| Sub-block | Renders into | Asks for existing when… |
|---|---|---|
transit_gateway | create_roks_transit_gateway (+ roks_transit_gateway_name) | declined and the TGW jumphost is enabled (it needs a TGW) |
registry_cos | create_roks_registry_cos_instance (+ roks_cos_instance_name) | declined |
cert_manager | install_cert_manager | — |
bnk | deploy_bnk | — |
tgw_jumphost | testing_create_tgw_jumphost (+ testing_tgw_jumphost_name) | — |
cluster_jumphosts | testing_create_cluster_jumphosts (+ testing_cluster_jumphost_name_prefix) | — |
client_vpc | testing_create_client_vpc (+ testing_client_vpc_name) | TGW jumphost enabled but you decline creating a new client VPC for it |
cluster_vpc | use_existing_cluster_vpc + existing_cluster_vpc_id, emitted only when adopting | declining it is the adoption — existing: takes the VPC ID, and it is the prerequisite for cluster.existing_subnet_ids |
One extra key, client_region, is a plain string rather than a toggle: the region the testing client (TGW jumphost + client VPC) is installed in, set when init asks where to install the test client. It renders into testing_client_vpc_region (omitted → the terraform default applies) and seeds the regions roksbnkctl cleanup scans.
The block is optional — omit it and the sparse render (upstream module default names) applies; a fresh init writes it in full. For what each of those terraform variables defaults to when a toggle is absent, see Chapter 29.
bnk:
bnk:
cneinstance_size: Small
far_repo_url: repo.f5.com
manifest_version: 2.3.0-3.2598.3-0.0.170
flo_namespace: f5-bnk # optional; FLO namespace
flo_utils_namespace: f5-utils # optional; = flo_namespace ⇒ ONE namespace
gslb_datacenter_name: "" # optional; CNEInstance GSLB datacenter
cert_manager: # optional; cert-manager coordinates
namespace: cert-manager
version: v1.17.3
network: # optional; data-plane subnets + TMM self-IPs
vlan_prefixlen: 24 # self-IP prefix length (F5SPKVlan)
vlan_prefixlen_external: 23 # optional; overrides the shared value
vlan_prefixlen_internal: 26 # optional; the two VLANs need not match
tmm_k8s_routes: 172.17.0.0/18 # pod CIDR TMM routes to
zones: # one entry per AZ (3 total)
- ext_vlan_cidr: 10.155.15.0/24
int_vlan_cidr: 10.254.99.0/24
int_snat_cidr: 10.10.11.0/24
int_vip_cidr: 10.135.15.0/24
external_selfip: 10.155.15.101
internal_selfip: 10.254.99.101
# …zones 2 and 3
flp:
storage_class: "" # optional; FLP PVC StorageClass
Every field here is optional — leave the block out entirely and you get the upstream HCL’s defaults. For a one-page view of every field with its dotted path and the ROKSBNKCTL_* variable that sets it, see the config.yaml cheatsheet — generated from the same struct this chapter teaches, so the two cannot disagree. The field spec is Chapter 28 §BNKCfg, with the sub-blocks broken out as §BNKTrustedProfileCfg, §BNKCertManagerCfg, §BNKFLPCfg and §BNKPreflightCfg. Chapter 28’s line column matters here more than anywhere else in the config: a good part of bnk: is 2.4-only, and manifest_version is the only thing that selects the line.
manifest_version cannot be bumped on a live workspace — bnk up refuses it. The manifest version names the CNEManifest object (bnk-<version>, lowercased), so changing it renames that object. Terraform plans a rename as an in-place update and then aborts with Provider produced inconsistent final plan, and re-running does not converge — the stale value is in terraform state, not on disk.
The failure is not a clean stop. The apply gets partway, having already rolled every pod, and a retry rolls them again: a previous attempt did this three times, exhausted the registry’s pull quota (pull QPS exceeded), and left the cluster with f5-cne-controller, f5-tmm and f5-dssm-sentinel-2 in ImagePullBackOff and no CNEManifest at all (#309). So bnk up now compares the configured version against the one the workspace last applied and refuses before planning, rather than starting an apply that cannot finish.
To move to a new manifest:
roksbnkctl bnk down
roksbnkctl bnk up
That is the supported path, and the one the 2.4.0 GA upgrade was verified with (37 added, 0 changed, 0 destroyed). A cross-line change (2.3 → 2.4) is refused separately and for different reasons — see bnk_line above; that one needs a new workspace, not a reinstall.
cneinstance_size is passed through unvalidated, on purpose. The legal set of sizes is a property of the BNK manifest, not of this tool. Hardcoding a list would go stale the first time F5 adds one and would then refuse a size the product supports, so a size a given manifest does not define is rejected by the operator on the cluster, not here. Tiny is what the BNK 2.4 install guide uses, and on ROKS it is the only size that runs: everything above it requests hugepages (Small 4 GiB, Medium 8 GiB) that the platform has no supported way to allocate. Do not reach for bnk.hugepages — it is a no-op on ROKS and issue #203 explains why. Capacity comes from bnk.tmm_replicas and node size instead; see Appendix C.
gslb_datacenter_name is the whole feature. It is what the CNEInstance calls itself in GSLB, and it is emitted whenever set. There was also a gtm: block carrying the BIG-IP DNS url, username and password; it was removed in #227 because nothing read it — the f5ingress controller binary contains no occurrence of the environment variables it produced, on either BNK line. A config.yaml still carrying bnk.gtm needs no edit: an unrecognised key is ignored.
trusted_profile.service_account is a matcher, not a pointer — and the name differs by BNK line. The IBM Cloud Trusted Profile is the identity the CNE controller assumes to manage the VPC network attachments it creates for TMM, so it works without a stored API key. IBM’s trust relationship compares a pod’s service-account token against crn / namespace / name with EQUALS. Left empty, roksbnkctl derives the name the controller actually runs as, and that is not the same name on both lines:
| Line | Derived service account |
|---|---|
| 2.3 | f5-cne-controller-<flo_namespace>-f5-cne-controller-serviceaccount |
| 2.4 | f5-cne-controller |
On 2.3 it is FLO’s helm construction, <release>-<chart>-serviceaccount with the namespace baked into the release name. On 2.4 the CNE controller chart names the account outright. Both names were read off live installs rather than inferred.
Write a name that matches neither and nothing can assume the profile. The install still reports itself healthy. The controller logs the token exchange failing (BXNIM0398E … no matching rule or link found), then falls back to Cloud environment is not IBM or cloud provider instance is nil, skipping cloud address prefix handling and skips VPC address-prefix and route programming altogether — so nothing reaches the listener addresses and egress SNAT replies have no route back. Meanwhile Infra, GatewaySettings, Gateway and EgressGateway all report Programmed=True, because those conditions are computed without ever calling the cloud, and the profile looks correct in the IBM Cloud console the whole time. A status check cannot detect this; only traffic can (roksbnkctl testing exercises exactly that path).
So set it only if you can also make FLO name the account differently — and roksbnkctl cannot, because FLO creates that account when it reconciles the CNEInstance and the CNEInstance spec has no service-account field. In practice this field exists for estates driving FLO by some other means. The derived name is safe to share across clusters: uniqueness comes from the profile name (which carries the cluster name) and from the link’s cluster CRN, not from the account name.
cert_manager: overrides coordinates, not the decision. bnk.cert_manager.namespace / .version say where and which chart; whether cert-manager is installed at all stays on resources.cert_manager.create. The two are easy to confuse because they share a name.
license_mode: f5licenseproxy additionally requires the FLP phase to be up (roksbnkctl flp up) before bnk up runs, so the install has a proxy to point at. The proxy’s endpoint and root CA are not config — they are produced by flp up and read back from flp-outputs.json.
bnk.network: — data-plane subnets + TMM self-IPs
BNK’s data plane (TMM) needs per-availability-zone VLAN subnets, SNAT/VIP ranges, and self-IPs. Leave bnk.network out entirely and the BNK install-guide defaults apply. Set it to match your cluster’s fabric — roksbnkctl init prompts for every field (opt in at “Customize BNK networking?”, seeded with the values shown below), or hand-write the block.
The field spec is Chapter 28 §BNKNetworkCfg and, for one zone’s six CIDRs, §BNKZoneCfg.
Provide all three zones when you set zones — supplying zones replaces the install-guide defaults entirely (they render cneinstance_network_zones, driving the cloud-network-mapping ConfigMap and the external/internal F5SPKVlan CRs). The YAML above shows zone 1’s defaults; zones 2 and 3 follow the same shape one octet up (10.156.16.0/24 and 10.157.17.0/24 externally), and the full default set is the cneinstance_network_zones default in terraform/modules/cne_instance/modules/cneinstance/variables.tf. Zone names are derived from the region and aren’t configurable.
vlan_prefixlen and tmm_k8s_routes are network-wide, shared across all zones. vlan_prefixlen is independent of your VLAN CIDRs and is never derived from them — nothing validates the two against each other. Usually you want them to agree; a deliberate disagreement, paired with static routes, is how a smaller or larger directly-connected block is forced and the remainder steered. The mask can also differ between the two VLANs via vlan_prefixlen_external / vlan_prefixlen_internal: TMM can front a /23 externally while the internal side is a /26, which one shared scalar could not express. Unset either override to inherit the shared value; ROKSBNKCTL_VLAN_PREFIXLEN_EXTERNAL and ROKSBNKCTL_VLAN_PREFIXLEN_INTERNAL set them from the environment.
tmm_k8s_routes is the pod CIDR TMM installs a route toward (TMM_K8S_ROUTES) so it can reach the backend pods on the internal data path. The default is the ROKS default pod subnet — set it if your cluster’s isn’t.
What each zone CIDR becomes on 2.4. The same values drive a different set of
objects on each line, which is why they are marked “2.3 + 2.4” rather than being
split: on 2.3 they build the cloud-network-mapping ConfigMap and the
F5SPKVlan pair, and on 2.4 they become the Infra CR’s IPAM pools —
ext_vlan_cidr is the external-vlan-ipam pool and the next hop of every
generated route, int_snat_cidr is egress-snat-ipam, and int_vip_cidr is
vip-listener-ipam. The exceptions are the ones chapter 28 marks 2.3:
int_vlan_cidr (2.4 has no internal VLAN), both self-IPs (2.4 allocates TMM’s
VLAN addresses from external-vlan-ipam), and the three vlan_prefixlen*
fields (a 2.4 IPAM pool is a CIDR, and a CIDR states its own mask).
test:
test:
throughput:
image: networkstatic/iperf3:latest
duration: 30
streams: 8
default_mode: north-south
connectivity:
extra_hosts:
- https://my.gslb.example.com
- https://internal.example.test
The field spec is Chapter 28 §TestCfg, broken out as §ThroughputCfg, §ConnectivityCfg and §DNSCfg. duration and streams are iperf3’s -t and -P.
throughput.image is the in-cluster iperf3 server fixture, not the client. roksbnkctl test throughput deploys that image as a pod in the cluster and then runs an iperf3 client against it — and the client’s image is a separate thing, chosen by the execution backend (the docker and k8s backends run F5’s bundled ghcr.io/jgruberf5/roksbnkctl-tools-* tool images; see Chapter 17). Changing throughput.image moves the server end only.
The default server image, networkstatic/iperf3:latest, runs as root. OpenShift’s SCC admission overrides the pod’s user with a project-allocated UID, so it works on ROKS; plain Kubernetes with the restricted Pod Security standard does not, and the pod is refused with a RunAsNonRoot admission error. On a non-OpenShift cluster, point throughput.image at the bundled image instead — ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<version>, which is built to run as UID 1000.
connectivity.extra_hosts adds targets to the reachability probe alongside the canonical IBM/F5 endpoints. It exists mainly for disconnected estates: a probe against a public host there reports a failure that says nothing useful, so give it something the cluster can actually reach.
tf_source:
tf_source:
type: embedded
type | Other fields | Use case |
|---|---|---|
embedded (default) | none | Use the HCL bundled into the roksbnkctl binary via go:embed. The recommended path for users — install one binary, get matched CLI + Terraform together. |
github | repo: "owner/name", ref: "v0.6.1" | Pull a tarball from a GitHub release. Useful for testing forks or pinning to a specific upstream tag. |
local | path: "/abs/path/to/tf-source" | Point Terraform at an on-disk directory. For active development on the HCL itself. |
An empty type is treated as embedded (legacy / forgot-to-set). The field spec for repo / ref / path is Chapter 28 §TFSourceCfg; the table above is about which type to pick, not what the fields are.
roksbnkctl init --upgrade-tf is the helper for bumping the source between versions without retyping the rest of the config — see “Editing by hand vs helpers” below.
targets: — SSH targets
targets:
jumphost:
host: 169.45.91.177
user: ubuntu
key_source: tf-output:jumphost_shared_key
bastion:
host: ops.example.com
user: jgruber
key_path: ~/.ssh/id_ed25519
Each entry has host, user, an optional port, and exactly one of key_path (a PEM file) or key_source (agent, or tf-output:<name> to pull the key from a Terraform output). The field spec is Chapter 28 §TargetCfg.
The deep reference is Chapter 15 — SSH targets, and the user-facing prose is Chapter 16 — The –on flag and SSH jumphosts. This chapter just notes the block’s place in the overall config.
You don’t typically edit this block by hand. roksbnkctl up auto-populates jumphost post-apply, and roksbnkctl targets add ... populates the rest.
exec: — execution-backend defaults
exec:
ibmcloud: { backend: local }
iperf3: { backend: k8s }
terraform: { backend: local }
Per-tool defaults for the --backend system. Each entry is keyed by the tool name (ibmcloud, iperf3, terraform, and others as the matrix grows) and its one field, backend, is specified in Chapter 28 §ExecToolCfg. The table below is the enum of values that field accepts:
| Backend | Notes |
|---|---|
local | os/exec against the host binary. The default for terraform and ibmcloud. |
docker | Runs inside a vendored container image. Frozen toolchain version, no host install. |
k8s | Runs inside the cluster (long-lived ops pod or one-shot Job). Default for iperf3. |
ssh | Runs on a registered SSH target. Format: ssh:<target-name>. |
A --backend <value> flag on the command line overrides the workspace config for that single invocation. The flag wins; the config sets the default.
The iperf3 default is k8s because measuring throughput from a laptop’s internet uplink isn’t useful — you want the test to run from a network location adjacent to or inside the cluster. The local default is wrong for that tool, so the workspace config flips it.
Chapter 17 — Execution backends covers the full backend matrix; Chapter 18 — Choosing a backend per tool is the decision tree.
cos: — COS supply-chain (optional)
cos:
instance: bnk-supply-chain
bucket: bnk-artifacts
region: us-south
upload:
- source: ./local/f5-far-auth-key.tgz
key: f5-far-auth-key.tgz
- source: ./local/subscription.jwt
key: subscription.jwt
The field spec is Chapter 28 §COSCfg and §COSUpload.
This is the orchestration COS — the bucket holding the FAR auth tarball and the subscription JWT, which is a different thing from the registry COS instance the resources: block can create. instance / bucket / region are honoured by both the terraform render (ibmcloud_cos_instance_name / ibmcloud_resources_cos_bucket / ibmcloud_cos_bucket_region) and the registry FAR-file resolver, so a customer-owned bucket is used consistently across both.
The bucket init provisions is not the bare default. COS bucket names share one global namespace, like S3, so a generic base such as bnk-artifacts frequently collides with a bucket some other account already owns. When init provisions the supply chain it therefore appends a short token derived from your account ID — bnk-artifacts-<first 12 of the account id> — and writes that name into cos.bucket. Keying the suffix off the account (rather than the COS instance) makes the name both globally unique and stable and discoverable: a second workspace created from the same account’s API key derives the same name and reuses the bucket the first one provisioned, instead of making a second copy of the same artefacts.
upload performs pre-flight uploads from local files into that bucket before the phases that read them run — useful in CI, where the supply-chain artefacts are produced by the pipeline rather than staged by hand.
The block is optional — if you’ve already populated COS by hand or via the upstream HCL’s roks_cos_instance_name variable, you don’t need it. Chapter 25 — COS supply chain management covers the full workflow.
Behaviour when fields are missing
roksbnkctl falls through three layers in order: workspace config → upstream HCL default → fail.
| Missing field | Behaviour |
|---|---|
prefix | Empty ⇒ legacy sparse terraform.tfvars (no name variables; upstream tf-* module defaults). init prompts for one on a fresh interactive run. |
resources | Empty ⇒ no per-resource toggles rendered; goes with the empty-prefix legacy path. A fresh prefix-driven init writes the full block. |
ibmcloud.region | roksbnkctl init prompts; programmatic loads error with “region is empty”. |
ibmcloud.api_key_source | Resolver walks the full chain (env → keychain → config → prompt). |
ibmcloud.api_key_b64 | Skipped in the resolver chain. |
cluster.name | init prompts; programmatic loads error. |
cluster.openshift_version | Empty string passed to upstream HCL; the module picks the current default. |
cluster.workers_per_zone | Falls through to 1 (upstream default). |
bnk.* | Field is omitted from the generated terraform.tfvars and the upstream HCL default applies. |
tf_source | Treated as type: embedded (legacy default). |
targets.* | Block absent ⇒ roksbnkctl --on jumphost errors with “no target named jumphost”; auto-populated by up. |
exec.* | Per-tool defaults: ibmcloud→local, terraform→local, iperf3→k8s, DNS probe→local. Override per-tool via this block, or per-invocation via --backend. |
cos.* | No pre-flight uploads, and both the terraform render and the registry FAR resolver fall back to the built-in orchestration-COS defaults (bnk-supply-chain / bnk-artifacts / us-south) — which are also the upstream HCL’s. |
The general rule: if you don’t write it in config.yaml, roksbnkctl doesn’t write it into terraform.tfvars, and the upstream HCL’s default = ... clause takes over. The full upstream defaults are listed in Chapter 29.
How --var-file interacts with config.yaml
Both roksbnkctl up and roksbnkctl plan/apply/destroy accept the same --var-file flag terraform itself accepts (repeatable, later files win). The layering rule is:
1. config.yaml-derived terraform.tfvars (written first by roksbnkctl)
2. ~/.roksbnkctl/<ws>/terraform.tfvars.user (optional manual override)
3. --var-file <path> (CLI; repeatable)
Later layers override earlier. Concretely: config.yaml’s cluster.workers_per_zone: 2 writes roks_workers_per_zone = 2 into the generated tfvars. If you then pass --var-file ./bigger.tfvars containing roks_workers_per_zone = 5, terraform sees 5. The config.yaml value didn’t get re-applied; --var-file wins.
The terraform.tfvars.user middle layer is for when you want a workspace-local override that survives across runs without modifying config.yaml — it’s typically used for fields the YAML schema doesn’t model (rare; the schema covers the common knobs). Chapter 13 goes deep on this.
If you want a workspace to always apply a raw terraform.tfvars without re-passing --var-file on every phase command, drop it verbatim at ~/.roksbnkctl/<ws>/terraform.tfvars.user (mode 0600, sibling to config.yaml); subsequent up / plan / apply / down against bare -w <ws> pick it up automatically for both phases. To seed the workspace’s config.yaml itself from a file, use roksbnkctl init -w <ws> --config-file ./config.yaml — see Chapter 6 §“Skip the interview: init --config-file”.
The IBMCLOUD_API_KEY is the one exception that never goes through tfvars on disk. It’s passed as a TF_VAR_ibmcloud_api_key env var on the terraform invocation. --var-file cannot supply the API key — the resolver chain in Chapter 14 is the only path.
Editing by hand vs helpers
Several commands manage subsets of config.yaml so you don’t have to:
| Subset | Helper |
|---|---|
| Whole file (interactive) | roksbnkctl init |
tf_source: only | roksbnkctl init --upgrade-tf |
targets: block | roksbnkctl targets add/remove |
registry: block | roksbnkctl registry target … |
test.connectivity.extra_hosts | roksbnkctl test hosts add/remove/clear |
state: block (local ↔ COS/S3) | roksbnkctl state local / state s3 … / state show |
exec: block (per-tool backend) | roksbnkctl backend set/unset/show |
bnkforge: block | roksbnkctl bnkforge enable/disable/status |
ibmcloud.api_key_b64 | roksbnkctl init (after entering the key, it offers to save) |
When you do edit by hand, the load-time validators run on next roksbnkctl invocation:
- The plaintext-secret heuristic rejects an
api_key:field (it must beapi_key_b64:to be tolerated). - Workspace name validation runs on directory access (workspace names must match
[A-Za-z0-9][A-Za-z0-9_.-]{0,63}). - YAML parse errors surface a line number.
If a hand edit breaks the file, every command that reads the workspace fails fast with the parse error path, so you’ll know within one invocation.
Worked example: bootstrap a workspace from scratch
End-to-end Part IV scenario: brand-new laptop, no roksbnkctl workspaces yet, an IBM Cloud API key in your password manager. Goal: a usable workspace with the key in the OS keychain, the right region + resource group resolved, and terraform.tfvars ready to drive the HCL.
The transcript below is captured against the shipped binary. Prompts are written to stderr, indented two spaces, with the label left-padded; the [default] (or [Y/n] / [y/N]) in brackets is what you get on a bare Enter. Here the operator types acme-eu for the prefix, keeps the cluster + COS + cert-manager + BNK + TGW-jumphost defaults, declines the Transit Gateway (adopting shared-corp-tgw), declines a new client VPC for the jumphost (adopting shared-client-vpc), and declines per-zone cluster jumphosts.
$ roksbnkctl init -w dev
Setting up workspace "dev"
Region [ca-tor]:
→ Verifying IBM Cloud credentials...
✓ IBM Cloud user you@example.com (account 1a2b3c…)
Resource group [default]:
✓ Resource group "default" (id 0d1e2f…)
Workspace prefix (≤ 35 chars) [dev]: acme-eu
Create new ROKS cluster? [Y/n]: y
OpenShift version [4.20]:
Workers per zone [1]: 2
Create registry COS instance? [Y/n]: y
Create Transit Gateway? [Y/n]: n
Install cert-manager? [Y/n]: y
Deploy BIG-IP Next (BNK)? [Y/n]: y
Create TGW test jumphost? [Y/n]: y
Create a new client VPC for it? [y/N]: n
Existing client VPC name : shared-client-vpc
Existing Transit Gateway name : shared-corp-tgw
Create per-zone cluster jumphosts? [y/N]: n
Resolved resource names for prefix "acme-eu":
cluster acme-eu
cluster VPC acme-eu-cluster-vpc
registry COS instance acme-eu-registry-cos
transit gateway shared-corp-tgw (existing)
TGW jumphost acme-eu-jh-tgw
client VPC shared-client-vpc (existing)
✓ Wrote /home/you/.roksbnkctl/dev/config.yaml
(The IBM Cloud API key is resolved before the prompts above via the credentials resolver chain — env, then keychain, then the workspace api_key_b64, then a hidden TTY prompt that offers to save the key. When the key is already in your environment or keychain, no key prompt appears, which is the common re-init case shown here.)
Two ordering details worth noting against the transcript: the OpenShift version and Workers per zone prompts appear only when you answer yes to “Create new ROKS cluster?” (a declined cluster prompts instead for an existing cluster name/ID), and the Existing Transit Gateway name prompt fires after the client-VPC questions — it is only asked when the TGW was declined and the TGW jumphost is enabled (the jumphost rides the gateway, so it needs the existing one’s name).
Three things to note in that flow:
- The prefix prompt validates and re-prompts. Enter a prefix that’s too long, starts with a digit, or has a trailing hyphen, and
initrejects it with the offending resource, its computed length, its limit, and the maximum allowable prefix length — then asks again. In a non-TTY context (CI), an invalid default is a hard error rather than a silent truncation. The cap is 35 characters (the ROKS cluster-name limit). See Chapter 13 §“The length / charset limits”. - Declining a create toggle triggers existing-resource discovery — but only when something still needs it. Here the Transit Gateway was declined, and because the TGW jumphost is enabled (it rides the TGW),
initasks for an existing TGW name (theExisting Transit Gateway nameprompt). The same pattern drives theExisting client VPC namefollow-up. Declining a resource that nothing else depends on (e.g. cluster jumphosts) just turns it off with no follow-up. - The resolved name plan is printed before the workspace is saved, so you see exactly what
roksbnkctlwill ask IBM Cloud to create (and which names it will adopt rather than create) before committing.
The resulting ~/.roksbnkctl/dev/config.yaml:
prefix: acme-eu
ibmcloud:
region: ca-tor
resource_group: default
api_key_source: keychain
cluster:
create: true
name: acme-eu
workers_per_zone: 2
resources:
transit_gateway: { create: false, existing: shared-corp-tgw }
registry_cos: { create: true }
cert_manager: { create: true }
bnk: { create: true }
tgw_jumphost: { create: true }
cluster_jumphosts: { create: false }
client_vpc: { create: false, existing: shared-client-vpc }
tf_source:
type: embedded
That’s the minimum a prefix-driven workspace writes. Everything else (bnk:, test:, targets:, exec:, cos:) is empty and falls through to defaults. A workspace created without a prefix (e.g. an old config) omits both prefix: and resources: and renders the legacy sparse terraform.tfvars. The API key can also be supplied non-interactively from your password manager’s CLI by setting IBMCLOUD_API_KEY in the environment of the init invocation:
op here is the 1Password CLI; the op://... URI is its secret-reference scheme. Any password-manager CLI that prints a secret to stdout works the same way — Bitwarden (bw), gopass, aws secretsmanager get-secret-value, Doppler, etc. — the only thing roksbnkctl cares about is that IBMCLOUD_API_KEY is set in the environment when init runs.
# Alternative: pre-set IBMCLOUD_API_KEY so init resolves it from env rather than prompting
IBMCLOUD_API_KEY=$(op read 'op://Private/IBM Cloud/api-key') roksbnkctl init -w dev
Chapter 14 §“The IBMCLOUD_API_KEY resolver chain” covers the full env → keychain → workspace api_key_b64 → TTY-prompt order; this env-var path is the first link in that chain, so anything init resolves at bootstrap time follows the same precedence later invocations use. Once init has saved the key to the OS keychain (the default sink), no further prompting is needed. init still prompts interactively for the remaining workspace metadata (region, resource group, workspace prefix, and the per-resource create toggles — the cluster name is derived from the prefix when you create a cluster, not prompted) — a fully non-interactive bootstrap is on the v1.x roadmap.
You don’t render terraform.tfvars by hand. roksbnkctl derives it from config.yaml automatically on every up / plan / apply, writing it into the phase’s state directory (~/.roksbnkctl/dev/state/terraform.tfvars). With the prefix: acme-eu set above, that generated file names every account-scoped resource from the prefix:
# ~/.roksbnkctl/dev/state/terraform.tfvars (generated; do not hand-edit)
openshift_cluster_name = "acme-eu"
roks_cluster_vpc_name = "acme-eu-cluster-vpc"
roks_cos_instance_name = "acme-eu-registry-cos"
create_roks_transit_gateway = false
roks_transit_gateway_name = "shared-corp-tgw"
testing_tgw_jumphost_name = "acme-eu-jh-tgw"
testing_client_vpc_name = "shared-client-vpc"
# ...
(The separate roksbnkctl tfvars command is unrelated — it writes a copy of the upstream terraform.tfvars.example starter template to edit by hand, not this config-derived render. See Chapter 13 §“roksbnkctl tfvars — bootstrap a starter”.)
Chapter 13 covers the precedence rules between the generated terraform.tfvars, terraform.tfvars.user (the hand-edit overlay), and --var-file.
Finally, verify the workspace is healthy before the first real up:
# 3. Sanity-check
$ roksbnkctl doctor -w dev
✓ terraform 1.6.2 on PATH
✓ IBMCLOUD_API_KEY resolves via keychain
✓ region "ca-tor" accepts the key (IAM round-trip OK)
✓ resource group "default" exists (id: ...)
✓ workspace dev healthy
From here, roksbnkctl up --auto -w dev is the next step (see Chapter 7 — Quick start). You can layer on bnk:, test:, targets:, exec:, cos: blocks by hand-editing config.yaml whenever you need them — init only writes the minimum to keep first-run friction low.
Cross-references
- Chapter 13 — Terraform variables — the layering between
config.yamlandterraform.tfvars. - Chapter 14 — Credentials and the resolver chain — the
api_key_*fields and how they’re resolved. - Chapter 15 — SSH targets — the
targets:block. - Chapter 17 — Execution backends — the
exec:block. - Chapter 28 — Configuration reference — auto-generated complete field list.
- Chapter 29 — Terraform variable reference — the upstream HCL variables
config.yamltranslates to.
Remote state (COS / S3)
By default, roksbnkctl keeps terraform state in local files under your workspace — ~/.roksbnkctl/<workspace>/<phase>/terraform.tfstate, one per phase. That’s the right thing on a laptop, and it’s fine for a single containerized runner with a mounted volume. It breaks down in two cases:
- Stateless / ephemeral runners — a CI container with no persistent volume loses its state between runs.
- Fleet / parallel runs — local state has no locking, so two runs against the same state can corrupt it.
The fix is an optional remote state backend on IBM Cloud Object Storage (which exposes an S3-compatible API): each phase’s state lives in a bucket, with a lock, so state is decoupled from the container and concurrent runs are safe. This chapter is how to turn it on, and how to move an existing workspace over.
The default is unchanged — omit the state: block and everything works exactly as before. Nothing here is required unless you want remote state.
Turning it on
Turn it on from the CLI — you never hand-edit config.yaml:
roksbnkctl -w acme state s3 \
--endpoint https://s3.us-south.cloud-object-storage.appdomain.cloud \
--bucket acme-bnk-tfstate \
--region us-south \
--key-prefix roksbnkctl
--bucket you create ahead of time; --key-prefix is optional (defaults to the
workspace name). Two more optional flags — --access-key-source /
--secret-key-source — name the env vars the HMAC keys come from (see
Credentials below). Switch back, or inspect, with:
roksbnkctl -w acme state local # back to per-phase local tfstate
roksbnkctl -w acme state show # backend + s3 endpoint/bucket/region/key sources
That writes the state: block into the workspace config.yaml for you (the same
marshaller every other command uses — no hand-edit):
state:
backend: s3
s3:
endpoint: "https://s3.us-south.cloud-object-storage.appdomain.cloud"
bucket: "acme-bnk-tfstate" # you create this ahead of time
region: "us-south"
key_prefix: "roksbnkctl" # optional; defaults to the workspace name
# access_key_source / secret_key_source: env var names (see Credentials)
Three things are your responsibility to provision first; roksbnkctl consumes them and errors clearly if they’re missing:
- The bucket — create it in COS (it is your state store; roksbnkctl never creates or deletes it).
- HMAC credentials — COS’s S3 API authenticates with HMAC access/secret keys (a Service Credential with “Include HMAC Credential” enabled), not the IAM API key roksbnkctl uses elsewhere.
- terraform ≥ 1.10 — required for the native S3 lockfile (below). Selecting
s3on an older terraform fails a preflight check with a clear message; the runner image bundles a new-enough terraform.
Credentials
The HMAC keys are never written into config.yaml, the rendered HCL, or the state object. roksbnkctl reads them from the environment and passes them to terraform as AWS_* env at runtime:
export ROKSBNKCTL_COS_HMAC_ACCESS_KEY="…"
export ROKSBNKCTL_COS_HMAC_SECRET_KEY="…"
access_key_source / secret_key_source name the env vars to read; left unset they default to the two above, falling back to the standard AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY. In CI you just export two secrets.
How state is laid out
All four phases share the one bucket, each at its own key:
<key_prefix>/<workspace>/cluster/terraform.tfstate
<key_prefix>/<workspace>/bnk/terraform.tfstate
<key_prefix>/<workspace>/testing/terraform.tfstate
<key_prefix>/<workspace>/gateway/terraform.tfstate
So the cluster / BNK / testing / gateway states never collide, and the parallel up (BNK ∥ Testing) operates on distinct keys.
Locking — and the single-writer guarantee
COS has no DynamoDB equivalent, so the lock is terraform’s native S3 lockfile (use_lockfile, added in terraform 1.10 — hence the version floor): a small <key>.tflock object alongside each state object. It guarantees one writer per phase key — a second apply against the same phase blocks/fails on the lock rather than corrupting state.
What it does not do: serialize different phases or different workspaces — those are different keys with different locks, by design. So a parallel up is safe, and two operators on two different workspaces don’t contend.
Migrating an existing workspace
A new workspace with state: backend: s3 just works — the first up creates state straight in the bucket. To move an existing local-state workspace over, run roksbnkctl state s3 … (above), export the HMAC keys, then:
roksbnkctl state migrate
It copies each deployed phase’s local state into the bucket (via terraform init -migrate-state), one phase at a time:
→ migrating cluster → s3://acme-bnk-tfstate/roksbnkctl/prod/cluster/terraform.tfstate
✓ cluster migrated
→ migrating bnk → s3://acme-bnk-tfstate/roksbnkctl/prod/bnk/terraform.tfstate
✓ bnk migrated
· testing: remote key … already holds state — skipping (use --force to overwrite)
state migrate: 2 migrated, 1 skipped, 0 failed
Local state files left in place — verify the remote read-back, then remove them when satisfied.
Two safety properties:
- It won’t clobber. Before copying, it HEADs the target key; if the key already holds state, that phase is skipped (pass
--forceto overwrite deliberately). - It leaves your local files in place. Migration is a copy, not a move — verify a
roksbnkctl status/planreads cleanly from the bucket, then delete the localterraform.tfstatefiles when you’re satisfied.
Notes and limits
- Default is local. An absent
state:block (orbackend: local) renders byte-identical to pre-remote-state behavior — no migration, no surprises. - COS is the supported target. Other S3-compatible stores (AWS S3, MinIO) will likely work via the same block, but COS is what’s tested.
- roksbnkctl doesn’t provision the bucket or keys. That’s deliberate — the state store is yours to own and lifecycle.
See also
- Chapter 4 §“Path C — the runner image” — the stateless runner this pairs with.
- Chapter 28 — Configuration reference §
state:— the field-by-field table. - Chapter 6 — Workspaces — the local
~/.roksbnkctl/<workspace>/layout this replaces for state.
Terraform variables (terraform.tfvars)
Variable specs live in Chapter 29 — Terraform variable reference. That chapter is generated from
terraform/variables.tfand checked by a test.
roksbnkctl is a thin orchestration layer over a Terraform HCL bundle. The HCL has its own variables — well over 60 of them — declared in terraform/variables.tf. The workspace’s config.yaml covers the common knobs; for the rest, you reach into terraform.tfvars directly.
This chapter is the surface for that lower layer: where the example file lives, how roksbnkctl tfvars bootstraps a starter, what --var-file does, the layering rule between config.yaml-derived tfvars and your overrides, and the one variable that never goes on disk (ibmcloud_api_key).
Where the bundled HCL lives
The Terraform HCL is bundled into the roksbnkctl binary via go:embed. On first use of a workspace, it gets extracted to:
~/.roksbnkctl/<workspace>/state/tf-source/embedded-terraform/
├── main.tf
├── variables.tf
├── outputs.tf
├── providers.tf
├── versions.tf
├── terraform.tfvars.example
└── modules/
The extracted tree is the base plus a per-release overlay
Extraction writes the base tree above, then layers terraform/lines/<line>/ on
top of it, where <line> is the BNK release derived from bnk.manifest_version
(2.3.0-3.2598.3-0.0.170 → 2.3). Same relative path replaces, new path is
added, nothing is removed — so an overlay can only add to or replace the base,
never quietly delete a resource it declares.
No supported release needs one today, so lines/ ships empty and the extracted
tree is byte-identical to the base. It is documented here anyway, because the
consequence is not obvious and will not announce itself: the extracted tree is no
longer a function of the binary alone, but of the binary and the workspace’s
manifest version. If you are ever comparing your extracted main.tf against the
repo’s and they differ, that is the first thing to check.
Why this rather than a branch per BNK release — which was tried and withdrawn —
is in terraform/lines/README.md.
The short version: a branch forks the whole tool to express a difference that
lives in a handful of .tf files.
That terraform.tfvars.example file is the canonical reference for what’s tunable — every variable with a sensible starter value, grouped by module (ROKS cluster, cert-manager, FLO, CNEInstance, License, testing). terraform/variables.tf (linked at the GitHub canonical URL) is the formal declaration with types, descriptions, and defaults.
You don’t edit the example file in place. Copy or generate from it instead.
roksbnkctl tfvars — bootstrap a starter
The roksbnkctl tfvars subcommand prints a starter terraform.tfvars to stdout, populated from the current workspace state:
$ roksbnkctl tfvars > ~/.roksbnkctl/dev/terraform.tfvars.user
What gets pre-filled:
- Every field from
config.yamlthat maps to a tfvar (cluster name, region, workers, BNK fields, COS fields) cluster_network_mode— rendered unconditionally, unlike the settings around it. Most are omitted when unset so the HCL default stands; this one decides how a cluster is built, and a value that means two different things depending on which side you read it from is worth avoiding. Unset renders"single-nic", which is today’s behaviour exactly.- The cluster’s identity from
cluster-outputs.jsonifcluster uphas already run - A commented-out section for the variables you might want to tune next (jumphost profile, GSLB datacenter, license mode)
What’s deliberately excluded:
ibmcloud_api_key— never on disk (see “The IBMCLOUD_API_KEY exception” below)- Sensitive outputs (BIG-IP passwords, COS HMAC secrets) — left as upstream defaults
The starter is meant to be copied into ~/.roksbnkctl/<ws>/terraform.tfvars.user (the workspace-local override file) or into a --var-file path you keep alongside the workspace.
What you typically edit
The variables that matter for day-to-day BNK trial work, ordered by likely-to-touch:
| Variable | Default | What it controls |
|---|---|---|
openshift_cluster_name | tf-openshift-cluster | Cluster name. Mirrors config.yaml’s cluster.name. |
roks_workers_per_zone | 1 | Worker nodes per AZ. 2 ⇒ 6 workers in a 3-AZ MZR region. |
create_roks_cluster | true | Set false to adopt an existing cluster. Pair with roks_cluster_id_or_name. |
cluster_public_gateway | true | false = private/disconnected cluster (no public gateways, no worker egress). config.yaml’s cluster.public_gateway. Expert — needs private connectivity you provide. |
openshift_cluster_version | "4.20" | OpenShift minor. Quote it — YAML/HCL parses 4.20 as float otherwise. |
cneinstance_deployment_size | Small | Small/Medium/Large. CNEInstance sizing. |
f5_bigip_k8s_manifest_version | upstream pin | Pin a specific BNK manifest chart version. |
far_repo_url | repo.f5.com | FAR Docker/Helm registry. Override for a non-production FAR (an EA repo) or a staging mirror. Reaches every consumer including the standalone FLP VSI, whose chart pulls used to spell the host as a literal and silently ignored this. |
flo_namespace | f5-bnk | Where the F5 Lifecycle Operator runs. |
testing_create_tgw_jumphost | true | Create the testing jumphost in a client VPC over Transit Gateway. |
testing_ssh_key_name | "" (must set) | Existing IBM Cloud SSH key name for jumphost provisioning. |
flo_trusted_profile_sa_name / flo_trusted_profile_roles | the CNE controller’s IBM Cloud identity — modelled as bnk.trusted_profile.* | |
cneinstance_vlan_prefixlen_external / _internal | per-VLAN self-IP masks — modelled as bnk.network.vlan_prefixlen_{external,internal} | |
cneinstance_gslb_datacenter_name | "" | Set when wiring BNK into an F5 BIG-IP GSLB datacenter. |
cneinstance_network_zones | [] (install-guide defaults) | Per-zone VLAN/SNAT/VIP subnets + TMM self-IPs. Mirrors config.yaml’s bnk.network.zones. Supply all three zones or none. |
cneinstance_vlan_prefixlen | 24 | TMM self-IP prefix length (F5SPKVlan spec.prefixlen_v4). bnk.network.vlan_prefixlen. Match your VLAN CIDRs. |
cneinstance_tmm_k8s_routes | 172.17.0.0/18 | Pod CIDR TMM routes to (TMM_K8S_ROUTES). bnk.network.tmm_k8s_routes. Set to your cluster’s pod subnet if non-default. |
license_mode | connected | connected | disconnected. |
For the full list with types and per-field descriptions, see terraform/variables.tf directly — link here — or the auto-generated Chapter 29 — Terraform variable reference.
Resource naming & collision avoidance
Every account-scoped IBM Cloud resource name is derived from a single workspace prefix. Without per-workspace naming, two workspaces creating infrastructure in the same IBM Cloud account would collide at the account level — the second up would hit Provided Name … is not unique / gateway with the same name already exists. The prefix avoids this: roksbnkctl derives the full name set from it, validates each name against its resource type’s length/charset limit at init time, and renders a complete terraform.tfvars that names every resource explicitly.
Prefix → name derivation
Give a workspace the prefix acme-eu, and roksbnkctl generates this name set:
| Resource | tfvars variable | Derived name (prefix = acme-eu) | Suffix |
|---|---|---|---|
| ROKS/OpenShift cluster | openshift_cluster_name | acme-eu | (none — cluster name is the prefix) |
| Cluster VPC | roks_cluster_vpc_name | acme-eu-cluster-vpc | -cluster-vpc |
| Registry COS instance | roks_cos_instance_name | acme-eu-registry-cos | -registry-cos |
| Transit Gateway | roks_transit_gateway_name | acme-eu-tgw | -tgw |
| Client VPC (TGW jumphost) | testing_client_vpc_name | acme-eu-client-vpc | -client-vpc |
| TGW jumphost | testing_tgw_jumphost_name | acme-eu-jh-tgw | -jh-tgw |
| Per-zone cluster jumphosts | testing_cluster_jumphost_name_prefix | acme-eu-jh (module appends -<zone>, e.g. acme-eu-jh-us-south-1) | -jh |
The names are deterministic from the prefix, so roksbnkctl re-derives them on every up / plan / apply — regeneration is always safe, and the rendered terraform.tfvars is a faithful record of exactly what the tool asked IBM Cloud to create.
Why the cluster name takes no suffix
The cluster name deliberately equals the prefix with no suffix appended. This is load-bearing, not an oversight:
- The ROKS/OpenShift cluster name has the tightest limit of any resource here — 35 characters (see the table below). Every other name is an IS resource capped at 63.
- By making the cluster name be the prefix, the prefix’s own length limit becomes that same 35-character cluster limit. A prefix that’s short enough to be a valid cluster name is automatically short enough that every suffixed name fits inside 63 — the worst case is the zone-appended cluster jumphost (
<prefix>-jh-us-south-1), which at a 35-char prefix reaches only 49 of its 63-char budget.
Do not “tidy” the cluster name into
<prefix>-cluster. Adding a suffix to the cluster name would shrink the usable prefix length (the binding 35-char limit would now have to absorb-clustertoo) without buying anything. The no-suffix cluster name is what lets a single prefix-length check guarantee the whole name set is valid.
The length / charset limits
roksbnkctl validates the prefix — and every name it derives — against these per-resource-type constraints. They are pinned from IBM Cloud’s own validators so the tool rejects a bad prefix at init time rather than letting IBM Cloud reject it minutes into an apply:
| Resource kind | Max length | Charset rule | Source |
|---|---|---|---|
| ROKS / OpenShift cluster name | 35 | Start with a letter; letters, digits, and hyphen -; ≤ 35 chars | IBM Cloud provider docs, ibm_container_cluster / ibm_container_vpc_cluster name argument (“must start with a letter, can contain letters, numbers, and hyphen (-), and must be 35 characters or fewer”). |
| IS resource — VPC, VSI, Transit Gateway, SSH key | 63 | Start with a lowercase letter; lowercase letters, digits, and hyphen -; must end alphanumeric; ≤ 63 chars (regex ^[a-z][-a-z0-9]*$ + ends [a-z0-9]) | terraform-provider-ibm ValidateISName (ibm/validate/validators.go). |
| COS / Resource Controller instance | 180 | Permissive server-side; roksbnkctl reuses the lowercase-alphanumeric IS subset since the name derives from an already-validated lowercase prefix | IBM Cloud Resource Controller create-instance name field (“must be 180 characters or less”). |
Because the prefix flows into all three, roksbnkctl enforces the strictest applicable rule on the prefix label itself (lowercase, start with a letter, [a-z0-9-], no trailing hyphen) and the tightest length (35, from the cluster). An over-long prefix is rejected with an actionable message naming the offending resource, its computed length, its limit, and the maximum allowable prefix length — and init re-prompts (TTY) or hard-errors (non-TTY). There is no silent truncation or hashing.
Namespaces are NOT prefixed
Kubernetes namespaces — cert_manager_namespace (cert-manager), flo_namespace (f5-bnk), flo_utils_namespace (f5-utils) — default to their conventional values (and are settable via bnk.flo_namespace / bnk.flo_utils_namespace — including to the same value, for one namespace instead of two) and are never prefixed. They are cluster-internal: two workspaces only ever collide on namespaces if they target the same cluster, in which case a shared namespace is usually what you want (and a prefix would actively break the convention FLO and the BNK charts expect). Only account-scoped IBM Cloud infrastructure names — cluster, VPCs, TGW, COS, jumphosts — are prefixed, because those are what collide at the account level.
Overriding a generated name
Every generated name is just a value in the rendered terraform.tfvars, so the normal layering rule overrides it. To pin a specific name (e.g. to adopt a VPC that doesn’t follow the prefix convention), set the matching variable in ~/.roksbnkctl/<ws>/terraform.tfvars.user or pass it via --var-file:
# Override just the Transit Gateway name for this workspace, permanently
echo 'roks_transit_gateway_name = "shared-corp-tgw"' \
>> ~/.roksbnkctl/<ws>/terraform.tfvars.user
The .user file (or --var-file) layers after the generated terraform.tfvars, so the override wins. This is the same override path documented in §“The layering rule” — the prefix machinery doesn’t change it. Declining a resource at init time and supplying an existing one is the cleaner path for adoption; see Chapter 12 §“Worked example”.
The layering rule
When roksbnkctl up (or plan/apply/destroy) invokes Terraform, it composes three layers of tfvars in this order:
1. terraform.tfvars (rendered by roksbnkctl from config.yaml)
2. terraform.tfvars.user (workspace-local override, optional)
3. --var-file <path> ... (CLI flag, repeatable, later file wins)
Later layers override earlier ones — same rule Terraform itself uses for -var-file chaining.
Concretely:
# config.yaml says cluster.workers_per_zone: 2
# ~/.roksbnkctl/dev/terraform.tfvars.user contains:
# roks_workers_per_zone = 4
# Run with no flag:
roksbnkctl up
# → terraform sees 4 (.user wins over generated .tfvars)
# Pass a CLI override:
roksbnkctl up --var-file ./perf-test.tfvars
# perf-test.tfvars contains: roks_workers_per_zone = 8
# → terraform sees 8 (.var-file wins over .user)
# Multiple --var-files; later wins:
roksbnkctl up \
--var-file ./base.tfvars \
--var-file ./override.tfvars
# → values in override.tfvars win over base.tfvars,
# which both win over .user, which wins over .tfvars
The --var-file flag matches Terraform’s own --var-file exactly — repeatable, paths interpreted relative to the working directory at invocation time.
The IBMCLOUD_API_KEY exception
The upstream HCL declares ibmcloud_api_key as a sensitive variable. Every other tfvar can land in a file on disk; this one never does.
Instead, the API key flows through the resolver chain (env → keychain → config-b64 → prompt — see Chapter 14), and roksbnkctl exports it as TF_VAR_ibmcloud_api_key in the environment of the terraform-exec child process. Terraform reads the env var and injects it as if it had been declared in tfvars, but no plaintext key ever touches the filesystem.
If you put ibmcloud_api_key = "..." in a hand-edited tfvars and run terraform directly (not via roksbnkctl), it works — Terraform itself is happy. But this is not how roksbnkctl runs Terraform, and putting the key in a .tfvars.user or --var-file is strongly discouraged: the file persists on disk, gets backed up, gets committed to git by accident, and gets read by other processes. The env-var path eliminates the on-disk window entirely.
Other secrets in scope:
bigip_password— upstream HCL declares it as a regular string (notsensitive). If you set it in tfvars, the value lands on disk. Treat that file like a credential.- COS HMAC keys — auto-generated by the
roks_clustermodule via the COS service-credentials resource; they live interraform.tfstate(which is itself sensitive —chmod 0600, never commit, treat the workspace as a secret store).
Worked example: bigger cluster for a perf test
Default workspace, default cluster. You want to bump worker count for one perf-test run, then go back.
# 1. Confirm the current value comes from config.yaml
$ grep workers ~/.roksbnkctl/dev/config.yaml
workers_per_zone: 2
# 2. Drop a one-off override into a file
$ cat > ~/perf-cluster.tfvars <<'EOF'
roks_workers_per_zone = 6
roks_min_worker_vcpu_count = 32
roks_min_worker_memory_gb = 128
EOF
# 3. Plan against it (note: --var-file passes through to terraform plan)
$ roksbnkctl plan --var-file ~/perf-cluster.tfvars
# 4. Apply
$ roksbnkctl apply --var-file ~/perf-cluster.tfvars
# 5. Run the throughput test
$ roksbnkctl test throughput
# 6. Roll back: re-apply WITHOUT the var-file
$ roksbnkctl apply
# → terraform sees workers_per_zone=2 again from config.yaml-derived tfvars
Notice step 6 — dropping the --var-file flag is the rollback. Terraform compares its current state to the new desired state (from config.yaml) and scales the worker pool back down. No special “undo” command needed.
For a more permanent override (you want this workspace to always run with bigger nodes), put the contents of perf-cluster.tfvars into ~/.roksbnkctl/dev/terraform.tfvars.user instead. Then every roksbnkctl up/apply picks it up automatically without a CLI flag.
When to edit config.yaml vs .tfvars.user vs --var-file
A rough decision matrix:
| You want to change… | Edit… |
|---|---|
| Cluster identity, region, OpenShift version, worker count | config.yaml (via roksbnkctl init or by hand) |
| BNK chart version, CNEInstance size, FAR repo | config.yaml (the bnk: block) |
A variable not modelled in config.yaml | terraform.tfvars.user (workspace-local, persistent) |
You have a complete ./terraform.tfvars you want this workspace to always use | Copy it to ~/.roksbnkctl/<ws>/terraform.tfvars.user (sibling to config.yaml, mode 0600) so bare -w <ws> commands Just Work for both phases. See Chapter 6 §“Raw terraform-variable overrides”. |
| A one-off override for a single run (perf test, capacity bump) | --var-file ./oneoff.tfvars (CLI) |
| A CI-pipeline variable bundle that’s checked into git | --var-file ./ci-overrides.tfvars (CLI; the file lives in your CI repo, not the workspace) |
The schema in config.yaml covers about a third of the upstream HCL variables — the ones that nearly every workspace needs to set. The other two-thirds (jumphost details, every BNK module’s full surface, the testing module’s full surface) are reachable through the lower layers.
Cross-references
- Chapter 12 — Workspace config — what
config.yamlcovers vs what falls through to tfvars. - Chapter 14 — Credentials and the resolver chain — why
ibmcloud_api_keydoesn’t go in tfvars. - Chapter 29 — Terraform variable reference — auto-generated complete reference for
terraform/variables.tf. - The upstream
terraform/variables.tfsource: https://github.com/jgruberf5/roksbnkctl/blob/main/terraform/variables.tf - The upstream starter file: https://github.com/jgruberf5/roksbnkctl/blob/main/terraform/terraform.tfvars.example
Credentials and the resolver chain
Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the
Workspacestruct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.
roksbnkctl handles four kinds of secrets: an IBM Cloud API key, a kubeconfig, an SSH private key, and the Terraform state file. Each has a different threat model, a different lookup chain, and a different rule for “what’s safe to commit to a workspace”.
This chapter is the user-facing distillation of PRD 04 — credential propagation. PRD 04 is the design surface for developers extending the credential system; this chapter is the operational surface for users who need to know “where does my key live, and how does the tool find it”.
The four secrets in scope
| Credential | Used by | Resolved from |
|---|---|---|
| IBMCLOUD_API_KEY | ibmcloud CLI, terraform’s IBM provider, IBM SDK calls | Env → OS keychain → workspace api_key_b64 → prompt |
| kubeconfig | kubectl/oc passthroughs, roksbnkctl k get/apply/..., terraform’s k8s + helm providers | KUBECONFIG env → ~/.kube/config (kubectl-style) |
| SSH private key | The SSH client backing --on and the ssh:<target> execution backend | Per-target: file path, ssh-agent, or tf-output:<name> |
| Terraform state | The terraform-exec calls inside roksbnkctl up/apply/destroy | Workspace state/terraform.tfstate (filesystem only) |
Each has its own discovery rules. Walk them in turn.
The IBMCLOUD_API_KEY resolver chain
The single most-used credential. Resolved by internal/cred/resolver.go. The resolver walks four sources in order:
1. Environment variables (process-scoped, never persisted)
2. OS keychain (per-user, system-managed)
3. Workspace config api_key_b64 (per-workspace, base64 obfuscation)
4. Interactive prompt (TTY-only)
The first source that yields a non-empty value wins. The chain stops there — the key isn’t re-fetched from a “more authoritative” source on subsequent calls.
Source 1 — Environment
The resolver checks these env vars in order, returning the first non-empty value:
IBMCLOUD_API_KEY # canonical
IC_API_KEY # short alias
TF_VAR_ibmcloud_api_key # terraform passthrough form
TF_VAR_IBMCLOUD_API_KEY # uppercase variant some pipelines use
TF_VAR_IC_API_KEY # uppercase variant of the short alias
Env vars are first because they’re the most explicit path — if you’ve gone to the trouble of setting one, you’ve made a deliberate choice. Pre-existing CI pipelines, automation scripts, and direnv setups all live here. The resolver respects that ordering even when a keychain entry also exists.
Source 2 — OS keychain
roksbnkctl stores per-workspace API keys in the OS-native keychain via github.com/zalando/go-keyring:
| OS | Backend |
|---|---|
| macOS | Keychain (security framework) |
| Linux (with libsecret) | GNOME Keyring / KWallet via Secret Service API |
| Windows | Credential Manager |
| Linux (no libsecret) | Falls back to source 3 (config base64) |
Entries are namespaced under service roksbnkctl, with user <workspace>/ibmcloud_api_key:
# What `roksbnkctl init` writes (no-op shown for clarity)
$ keyring set roksbnkctl dev/ibmcloud_api_key
This is the recommended secure default. The OS handles process isolation; roksbnkctl only sees the value during the brief window between fetch and use.
Source 3 — Workspace api_key_b64
A base64-encoded blob in ~/.roksbnkctl/<workspace>/config.yaml:
ibmcloud:
api_key_b64: ZW5jb2RlZC1hcGkta2V5LXZhbHVl
Important framing: base64 is obfuscation, not encryption. Anyone with read access to the file can decode it instantly:
echo -n "ZW5jb2RlZC1hcGkta2V5LXZhbHVl" | base64 -d
# → encoded-api-key-value
The encoding exists for two reasons:
- Visual. A glancing
cat config.yamldoesn’t surface the literal API key. - Format. API keys can contain
=and other YAML-special characters that complicate inline storage. Base64 normalises them.
api_key_b64 is the fallback when the OS keychain isn’t available — most commonly WSL2 without libsecret, headless Linux servers, and CI runners where bringing up a keychain daemon is more friction than it’s worth. Treat the file like a plaintext credential: never commit it, never share it.
File-mode note:
config.yamlis written mode0600inside a0700workspace directory, and roksbnkctl tightens both on read for workspaces written by an earlier release — you no longer need tochmodit yourself or re-chmod afterinit. Verify withroksbnkctl doctor(theworkspace permissionscheck).Releases before v1.50.0 wrote
0644inside0755, so on any host you share with another account, rotate an API key that has been sitting inconfig.yaml. Tightening the mode now does not un-expose what was readable before. The keychain and env-var paths avoid the question entirely: nothing sensitive lands inconfig.yamlat all.
The plaintext field name api_key: is rejected at workspace-load time:
$ roksbnkctl up
error: ~/.roksbnkctl/dev/config.yaml: plaintext secret detected (offset 47):
workspace config.yaml must not contain credentials — use IBMCLOUD_API_KEY
env var or the OS keychain (see `roksbnkctl init`)
The regex catches api_key, apikey, ibmcloud_api_key, password, token, secret_access_key, hmac_secret. The _b64 suffix is the documented escape — it’s the only inline form the loader tolerates.
Source 4 — Interactive prompt
When sources 1-3 all come up empty AND stdin is a TTY, the resolver prompts:
Enter IBM Cloud API key for workspace "dev": ********
Save the key for future runs? [Y/n]: y
✓ saved to OS keychain
The key is read with echo disabled (via golang.org/x/term). The prompt offers to persist — by default it tries the OS keychain first, falls back to api_key_b64 in config.yaml if the keychain is unavailable.
If stdin isn’t a TTY (CI runner, piped input, daemon process), the resolver errors instead of hanging:
error: no IBM Cloud API key available and stdin is not a TTY (cannot prompt;
set IBMCLOUD_API_KEY or run `roksbnkctl init`)
Pinning a single source
The chain is the default. To force one specific source, set ibmcloud.api_key_source in config.yaml:
ibmcloud:
api_key_source: keychain # env | keychain | config | prompt
This is useful in two scenarios:
- CI:
api_key_source: envmakes a missing env var a hard error rather than falling through to a (locked / non-existent) keychain. - Auditable single-source-of-truth: pinning to
keychaindocuments that this workspace’s key lives in the OS keychain and nowhere else; reading the key from a different source becomes an error rather than a silent fallback.
kubeconfig discovery
Different chain, different rules. roksbnkctl discovers the kubeconfig the same way kubectl does — two sources, in this order:
1. KUBECONFIG environment variable (first existing path in a colon-separated list)
2. ~/.kube/config
This is the kubectl-standard discovery chain, implemented in internal/k8s/client.go::DefaultKubeconfigPath(). Whatever you’ve already taught kubectl to read, roksbnkctl reads too.
cluster up’s post-apply step writes the admin kubeconfig to ~/.kube/config (mode 0600) by default — so the second source in the chain is also the destination of the tool’s own output, and the same KUBECONFIG-overrides-everything rule applies. If KUBECONFIG is set when cluster up runs, the download lands at that path instead.
Note: there is also a
~/.roksbnkctl/<workspace>/state/kubeconfig/directory under the workspace state dir. It’s a Terraform input (kubeconfig_dirtfvar) that the upstream HCL writes per-component sub-files into (cert_manager,cne_instance,flo,license); it is not a kubeconfig file the resolver reads. Don’t confuse the two.
When the file is missing
If neither source yields a kubeconfig, commands that need one error with:
error: no kubeconfig: KUBECONFIG env not set, ~/.kube/config not present.
Run `roksbnkctl cluster up`, `roksbnkctl cluster register <name>`,
or set KUBECONFIG.
The remediation message tells you which path to take. cluster register <name> is the path for an existing cluster you want to adopt without re-creating it (see Chapter 9).
File permissions
cluster up writes ~/.kube/config chmod 0600 (owner read/write only). It contains the cluster admin token; treat it like a credential. Don’t commit it, don’t email it, don’t cat it in screen-shared sessions.
SSH private keys
Per-target, not per-workspace. Each entry under targets: in config.yaml declares exactly one of:
| Source | Form | Notes |
|---|---|---|
| File | key_path: ~/.ssh/id_ed25519 | Standard OpenSSH key formats. Tilde expansion honoured. |
| Agent | key_source: agent | Talks to ssh-agent over $SSH_AUTH_SOCK. Linux/macOS only at v1.0; Windows ssh-agent named-pipe support is on the v1.x roadmap. |
| TF output | key_source: tf-output:jumphost_shared_key | Reads from terraform state at connect time; never written to disk separately. |
The tf-output: form is the auto-discovered jumphost path — the upstream HCL provisions a tls_private_key resource per cluster create, marks it sensitive, and surfaces it as a terraform output. roksbnkctl reads the output via terraform output -raw at SSH-connect time, never persists it, and the key only exists in TF state plus in memory during a connect.
Chapter 15 — SSH targets is the deep reference for the targets: block; this chapter just notes the credential-side framing.
Terraform state
~/.roksbnkctl/<workspace>/state/terraform.tfstate is the workspace’s terraform state file. It contains:
- IBM Cloud admin tokens (cluster admin, COS HMAC credentials)
- Generated TLS private keys (the jumphost shared key referenced above)
- Sensitive outputs (FAR auth bundles, license JWTs)
- Every resource attribute terraform tracks
It is plaintext-credential-equivalent. The file mode is 0600; the parent directory is 0700. Backup the workspace dir intact, never commit it to git, treat compromise of the state file as compromise of every secret it contains.
There is no separate “TF state credential” — the file’s filesystem ACL is the only access control. PRD 04 covers the cross-backend story for moving state into a Docker bind-mount, a Kubernetes Secret, or an SCP’d remote temp directory; at v1.0 the local file is the only path (terraform --backend k8s / ssh are deferred to v1.x; see docs/PLAN.md §“What’s deliberately deferred to post-v1.0”).
What’s safe to commit vs not
A short rule:
SAFE TO COMMIT: nothing in ~/.roksbnkctl/<workspace>/
NOT SAFE: everything in ~/.roksbnkctl/<workspace>/
The longer version, by file:
| Path | Commit? | Why |
|---|---|---|
~/.roksbnkctl/<ws>/config.yaml | No | Even without api_key_b64, this file documents your cluster identity, region, COS bucket — useful inventory for an attacker. |
~/.roksbnkctl/<ws>/config.yaml (with api_key_b64) | Hard no | The base64 value is plaintext-equivalent. Committing it = leaking the key. |
~/.roksbnkctl/<ws>/state/kubeconfig | No | Cluster admin token. |
~/.roksbnkctl/<ws>/state/terraform.tfstate | No | Every secret terraform manages, in plaintext. |
~/.roksbnkctl/<ws>/state/terraform.tfvars | No | Generated; references no secrets directly but documents resource layout. |
~/.roksbnkctl/<ws>/terraform.tfvars.user | Maybe | If you’ve kept secrets out (no bigip_password, no ibmcloud_api_key), it’s just config. Audit before committing. |
~/.roksbnkctl/<ws>/cluster-outputs.json | No | Cluster identity + COS instance name. Not directly a secret but tied to the workspace. |
~/.roksbnkctl/known_hosts | Yes (if you want) | Host-key fingerprints; not a secret. Same threat model as OpenSSH’s ~/.ssh/known_hosts. |
The simplest policy: a .gitignore that excludes the entire ~/.roksbnkctl/ tree. If you really want to share a workspace skeleton with a colleague, send the config.yaml minus api_key_b64 and let them re-run roksbnkctl init against their own account.
How roksbnkctl init writes the API key
Walk through the writeable side of the resolver:
$ roksbnkctl init
Workspace name [default]: dev
IBM Cloud region [ca-tor]:
Enter IBM Cloud API key (input hidden): ********
Save the key for future runs? [Y/n]: y
✓ saved to OS keychain
What just happened:
initprompted for the key. Input echo was off; the key never appeared on screen.- The user said “save”.
SaveAPIKeyForWorkspacetriedSaveAPIKeyToKeychainfirst.- The OS keychain accepted the entry (Linux + libsecret in this case). The success path returned
"OS keychain"andinitprinted the confirmation. - The key was not written to
terraform.tfvars(that’s the resolver’s job at terraform-invoke time, via theTF_VAR_ibmcloud_api_keyenv var).
If step 4 had failed (no keychain, WSL2 without libsecret), SaveAPIKeyForWorkspace would have fallen through to saveAPIKeyToConfig — base64-encoded the key, written it into config.yaml’s api_key_b64 field, returned "config.yaml (base64)". init would have printed:
✓ saved to config.yaml (base64)
warning: base64 is obfuscation, not encryption — chmod 0600 the file
Both destinations work. The keychain path is the recommended default; the config-b64 path is the documented fallback.
The cred-tmpfile and trusted-profile paths
roksbnkctl --backend docker does not leak IBMCLOUD_API_KEY in docker inspect, and roksbnkctl --backend k8s ops install auto-provisions an IBM Cloud trusted profile so the ops pod never sees a static API key (see PRD 04 §“Open questions”). Both have fallbacks for environments where the primary path doesn’t apply.
The tmpfile-bind-mount pattern (docker backend)
The docker backend writes the resolved IBMCLOUD_API_KEY to a 0600 tempfile on the host, bind-mounts that single file read-only at /run/secrets/ibmcloud_api_key inside the container, and points the container at the file via IBMCLOUD_API_KEY_FILE=/run/secrets/ibmcloud_api_key. The value never appears in the container’s stored env metadata — docker inspect <id> shows the path, not the key. The tempfile is owned by the calling user and is removed on backend exit (and on context cancellation, so an interrupted run still cleans up).
You don’t have to do anything to opt in — the pattern is the default for --backend docker. The engineering shape (lifecycle, the inline sh -c shim that re-exports the value into the legacy IBMCLOUD_API_KEY env name for tools that read from env, the why-not-just-use---secret discussion) lives in PRD 04 → “Cred tmpfile-bind-mount pattern (docker backend)”. For most users the takeaway is one line: --backend docker is docker inspect-clean.
The --trusted-profile flag (k8s backend)
New flag on roksbnkctl ops install that controls how the ops pod gets its IBM Cloud credential. Three values:
| Value | What it does | When to use |
|---|---|---|
auto (default) | Try to provision an IBM Cloud trusted profile (roksbnkctl-ops-<workspace>) linked to the ops pod’s ServiceAccount. The pod assumes the profile via its projected SA token and the static API key never lands in any Secret. If your workspace API key doesn’t have IAM iam-identity permissions, automatically fall back to the static-key Secret with a stderr warning that names the missing perm and how to silence it (--trusted-profile=off). | Default for new installs. Production users get the secure path automatically; restricted-IAM users still complete ops install successfully. |
on | Try to provision; fail loudly with a non-zero exit if perms don’t allow. No fallback. | CI / hardened environments where the static-key path is unacceptable and a perm-missing case should block, not warn. |
off | Skip the trusted-profile path entirely; provision the static-key Secret. | Compatibility / debugging — and the documented path for clusters whose IAM admin doesn’t grant iam-identity perms and isn’t expected to. |
Chapter 19 — The in-cluster ops pod walks through the --trusted-profile=auto install flow, the verification commands (oc get serviceaccount roksbnkctl-ops -o yaml showing the trusted-profile annotation), the fallback warning shape, and how ops uninstall cleans up a provisioned profile.
Backend-specific cred propagation
The credential-propagation rules differ per backend. All four backends ship at v1.0:
| Backend | Where creds live | Mechanism |
|---|---|---|
local | The user’s environment | os/exec inherits parent env |
docker | Caller’s env, propagated by reference | docker run --env IBMCLOUD_API_KEY (no =value) — value inherits, never appears in docker inspect |
k8s | Kubernetes Secret in the roksbnkctl-ops namespace | Mounted into the ops pod via envFrom: secretRef; or IAM trusted profile (preferred) |
ssh | Remote env or wrapper script | ssh -o SetEnv=IBMCLOUD_API_KEY=... first; falls back to a 0700 wrapper script with trap rm EXIT |
Each backend’s “where creds live” surface is summarised in Chapter 17 — Execution backends; the design rationale is in PRD 04.
The user-facing invariant across all four: you put the key into one of the resolver chain’s sources, and roksbnkctl figures out the rest. You don’t have to learn four different credential APIs to use four different backends.
The redactor
roksbnkctl writes a fair amount to its own logs (stdout, stderr) — terraform plan output, ibmcloud CLI output, error traces. Anywhere we can plausibly print the IBM API key (because a downstream tool printed it, because an error message included it, because a debug trace dumped a struct), the redactor masks it before the bytes leave the binary.
What gets redacted:
- The IBM Cloud API key value, anywhere it appears in
StdoutorStderrof an exec backend’sRunOpts. Replaced with[REDACTED]. - The same value in
roksbnkctl’s own log output (the lifecycle commands that wrap terraform-exec).
What does not get redacted:
- Output captured by callers via
-o yaml/-o jsonfor resources that legitimately contain the key (e.g., aSecretreturned fromroksbnkctl k get). The redactor doesn’t know about Kubernetes resource semantics; if youk get secret -o yaml, you’ll see the key. (The same is true ofkubectl.) - Output from a tool you ran outside
roksbnkctl(e.g., piping toteeafter invokingterraformdirectly). The redactor only sees bytes that pass through the exec backend’sStdout/Stderrwriters. - The terraform state file. State is on-disk; the redactor is an in-memory stream filter.
The implementation is internal/exec/redact.go — a wrapping io.Writer with byte-comparison redaction and cross-write prefix buffering (so a secret split across two Write calls still gets masked). The matcher uses the resolved API key value verbatim (a known string at run-time) rather than a generic “looks like an IBM API key” pattern, to avoid false positives on legitimate output.
PRD 04’s acceptance criteria require that the API key never appears in docker inspect, ps -ef, kubectl get pods/events -o yaml, or kubectl describe pod. The redactor is the defence-in-depth layer; the per-backend cred-propagation rules are the primary control.
Cross-references
- PRD 04 — credential propagation across execution backends — the full design.
- Chapter 12 — Workspace config — the
ibmcloud:block schema. - Chapter 13 — Terraform variables — why
ibmcloud_api_keydoesn’t go in tfvars. - Chapter 15 — SSH targets — the SSH key sources.
- Chapter 17 — Execution backends — backend-specific cred mechanics.
internal/cred/resolver.go— the resolver implementation: https://github.com/jgruberf5/roksbnkctl/blob/main/internal/cred/resolver.gointernal/config/secrets.go— the keychain + config-b64 helpers: https://github.com/jgruberf5/roksbnkctl/blob/main/internal/config/secrets.go
SSH targets
Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the
Workspacestruct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.
This chapter is the technical reference for the targets: system. Its companion is Chapter 16 — The –on flag and SSH jumphosts, which is the user-facing prose for “how do I run a command on the jumphost”. Chapter 16 introduces targets briefly; this chapter goes deeper into the schema, the key sources, the host-key trust model, the auto-discovery pipeline, and what the ssh execution backend layers on top of the lightweight --on flag.
If you arrived here from Chapter 16 looking for “where do I learn the full surface”, you’re in the right place.
The targets: block schema
Targets live under targets: in ~/.roksbnkctl/<workspace>/config.yaml:
targets:
jumphost:
host: 169.45.91.177
port: 22
user: ubuntu
key_source: tf-output:jumphost_shared_key
bastion:
host: ops.example.com
user: jgruber
key_path: ~/.ssh/id_ed25519
prod-jump:
host: 10.0.0.5
user: ec2-user
key_source: agent
The Go struct backing it is internal/config.TargetCfg:
type TargetCfg struct {
Host string `yaml:"host"`
Port int `yaml:"port,omitempty"` // default 22
User string `yaml:"user"`
KeyPath string `yaml:"key_path,omitempty"` // file path
KeySource string `yaml:"key_source,omitempty"` // "agent" | "tf-output:<name>"
}
| Field | Type | Required | Notes |
|---|---|---|---|
host | string | yes | IP or hostname. Resolved via the standard Go resolver chain (no special DNS handling). |
port | int | no | Defaults to 22. Only set when the remote sshd listens elsewhere. |
user | string | yes | Remote login username. |
key_path | string | one-of | File path to a PEM-encoded private key. Tilde expansion honoured. |
key_source | string | one-of | agent or tf-output:<output-name>. |
Validation rules at load time:
- Exactly one of
key_pathorkey_sourcemust be set. Setting neither, or both, fails the load with a clear error. - The target name (the YAML map key) must be non-empty and stable across YAML round-trips —
roksbnkctl targets show <name>androksbnkctl targets remove <name>look up by this name.
The TargetCfg type lives in internal/config rather than internal/remote to avoid an import cycle: the YAML (de)serialiser needs the wire shape, and the SSH client (internal/remote) needs to consume it. Keeping the shape in config and the runtime Target (parsed key, dialer config, etc.) in remote keeps the dependency direction one-way.
Key sources
The three options for telling roksbnkctl how to find the SSH private key for a target.
key_path: <file>
A PEM-encoded private key on disk:
bastion:
host: ops.example.com
user: jgruber
key_path: ~/.ssh/id_ed25519
Standard OpenSSH formats are accepted: id_rsa, id_ed25519, id_ecdsa, id_dsa (deprecated but supported). Tilde expansion uses os.UserHomeDir() semantics — ~/ → user home, ~user/ is not supported (use an absolute path).
The file is read at SSH-connect time, not at config-load time. A missing or unreadable file fails the connect, not the workspace load. This matters for ergonomics: you can edit a target into config.yaml referencing a key path that doesn’t exist yet, then create the key separately, without roksbnkctl init/use failing in between.
Encrypted keys (passphrase-protected) are not currently supported in the SSH client — the agent path is the recommended workflow for keys that need a passphrase.
key_source: agent
Talks to ssh-agent over $SSH_AUTH_SOCK:
prod-jump:
host: 10.0.0.5
user: ec2-user
key_source: agent
The agent presents whichever keys it currently holds; roksbnkctl tries each in turn against the target’s authorized_keys (via SSH’s standard publickey-authentication exchange). The first key the server accepts is the one that gets used.
This is the right setting when:
- Your team manages keys via 1Password / hardware tokens /
gpg-agentand you don’t want a key file on disk. - You’re on a shared workstation where putting the key file in
~/.ssh/is undesirable. - You’re already using ssh-agent for everything else and want consistent behaviour.
Platform note: ssh-agent integration is Linux/macOS-only. Windows users should use key_path to a file. The restriction is structural to the Go SSH library, which doesn’t wrap the Windows ssh-agent named-pipe protocol — see golang.org/x/crypto/ssh/agent and upstream tracking issues for status; full Windows support is a v2 item.
key_source: tf-output:<output-name>
Reads the key from the workspace’s terraform state output of that name:
jumphost:
host: 169.45.91.177
user: ubuntu
key_source: tf-output:jumphost_shared_key
This is the auto-discovered jumphost path. The upstream HCL provisions a tls_private_key resource per cluster create, marks it sensitive, and surfaces it as a terraform output named jumphost_shared_key. roksbnkctl reads it via the equivalent of terraform output -raw <name> at SSH-connect time.
What this gets you that key_path doesn’t:
- No on-disk key file separate from terraform state. The key only exists in
terraform.tfstate(which is already a sensitive workspace artefact) and in memory during the SSH handshake. - Auto-rotation on cluster re-create. Destroy and re-create the cluster, terraform generates a new
tls_private_key, and the next--on jumphostinvocation picks up the new key without any manual rewriting of the workspace config. - Single source of truth. The key value is in terraform state — the same place every other cluster-generated secret lives.
The terraform output must be a string-typed PEM-encoded private key. terraform output -raw <name> returns the value regardless of the sensitive flag (the flag just suppresses display; the data is still readable to anyone with state access).
Host-key TOFU and ~/.roksbnkctl/known_hosts
roksbnkctl keeps its own known_hosts file at ~/.roksbnkctl/known_hosts. It does not read or write ~/.ssh/known_hosts. The two files are independent.
Why a separate file
Three reasons:
- Isolation.
roksbnkctl’s SSH client is a different program fromssh(1); mixing host-key state between the two creates surprising behaviour (deleting a key from~/.ssh/known_hostsdoesn’t clear it fromroksbnkctl’s view, or vice versa). - Audit. A
roksbnkctl-managed file lets the tool’s behaviour be reasoned about without inspecting the user’s broader SSH state. - Cleanup.
roksbnkctl ws delete <name>could theoretically scrub host-key entries on workspace destroy; mixing into~/.ssh/known_hostswould mean editing a file the tool didn’t own.
The format matches OpenSSH’s ~/.ssh/known_hosts exactly (so future cross-pollination is technically possible), but the filenames are deliberately separate.
TOFU on first connect
The first time you connect to a target, roksbnkctl shows the host key fingerprint and asks whether to trust it:
$ roksbnkctl exec --on jumphost -- whoami
Add 169.45.91.177:22's key (SHA256:abc123def456ghi789jkl0mnopqrstuvwxyz/+=) to ~/.roksbnkctl/known_hosts? [y/N]: y
ubuntu
Answer y and the key is appended. Subsequent connects to the same host:port with the same server key trust silently.
Answer n and the connect aborts with exit code 126.
Mismatch behaviour
If the host key changes — re-provisioned VM, MITM attack, configuration drift — roksbnkctl refuses to connect:
error: host key mismatch: 169.45.91.177:22 known with SHA256:abc123... but
server presented SHA256:zyx987...; if the host was rebuilt, edit
~/.roksbnkctl/known_hosts
Same model OpenSSH uses. The fix is the same: edit (or ssh-keygen -R) the file to remove the stale entry, then re-connect to re-trigger the TOFU prompt.
The default ssh-keygen binary works against ~/.roksbnkctl/known_hosts — pass -f:
ssh-keygen -R 169.45.91.177 -f ~/.roksbnkctl/known_hosts
--insecure-host-key for CI
Automation contexts can’t answer a TOFU prompt. The --insecure-host-key flag skips host-key verification entirely:
roksbnkctl exec --on jumphost --insecure-host-key -- whoami
This is insecure — anyone in the network path can MITM the connection — and is intended only for short-lived CI runs against ephemeral test infrastructure. Don’t use it where session content is sensitive.
The flag is per-invocation, not per-target. There’s deliberately no targets.<name>.insecure_host_key: true config knob: forcing the choice into the call site keeps the security implications visible at every invocation.
When to use it:
- E2E tests against a freshly-provisioned cluster jumphost where the host key is just-generated and changes per run.
- Pipeline runs against ephemeral test VMs that get torn down within minutes.
- Recovery scenarios where the known-hosts file is corrupt and you need to bootstrap.
When not to use it:
- Production jumphosts with stable identity.
- Customer environments where session integrity matters.
- Anything where the SSH session carries secrets you can’t afford to leak to a passive attacker.
roksbnkctl targets — full reference
Four subcommands. Chapter 16 introduces them with worked examples; here’s the complete flag surface.
roksbnkctl targets list
NAME HOST USER KEY
jumphost 169.45.91.177:22 ubuntu tf-output:jumphost_shared_key
bastion ops.example.com:22 jgruber file:~/.ssh/id_ed25519
prod-jump 10.0.0.5:22 ec2-user agent
Flags:
--verbose/-v: also prints whether the target has a known-hosts entry recorded.-o json: machine-readable form. Schema:{"targets": [{"name": ..., "host": ..., "port": ..., "user": ..., "key_source": ...}]}.
The KEY column shows the source descriptor — never the key material. File-backed sources are prefixed file: to distinguish them visually from tf-output: and agent.
roksbnkctl targets show <name>
name: jumphost
host: 169.45.91.177
port: 22
user: ubuntu
key_source: tf-output:jumphost_shared_key
Same restriction: key material itself is never printed.
-o json is supported for scripted callers.
roksbnkctl targets add <name> ...
Required flags: --host, --user, and exactly one of --key-path / --key-source.
# File-backed key
roksbnkctl targets add bastion \
--host ops.example.com \
--user jgruber \
--key-path ~/.ssh/id_ed25519
# ssh-agent
roksbnkctl targets add prod-jump \
--host 10.0.0.5 \
--user ec2-user \
--key-source agent
# Non-default port
roksbnkctl targets add custom \
--host 10.0.0.5 \
--user root \
--key-path ~/.ssh/custom \
--port 2222
# tf-output (rare; usually auto-populated by `up`)
roksbnkctl targets add backup-jump \
--host 10.0.0.6 \
--user ubuntu \
--key-source tf-output:backup_jumphost_key
Refuses to add a target whose name collides with an existing entry — use targets remove <name> first, or pick a different name.
roksbnkctl targets remove <name>
roksbnkctl targets remove bastion
Removes the entry from config.yaml. Does not remove the corresponding host-key line from ~/.roksbnkctl/known_hosts — re-adding the same target later doesn’t re-trigger TOFU. This is deliberate; if you want to wipe the host key too, edit the known-hosts file by hand.
Auto-discovery from terraform outputs
The single most-used target — jumphost — is auto-populated post-roksbnkctl up. The flow:
-
roksbnkctl uprunsterraform applyagainst the workspace’s HCL. -
After successful apply,
roksbnkctlreads three outputs:testing_tgw_jumphost_ip,testing_tgw_jumphost_user,jumphost_shared_key. -
If
testing_tgw_jumphost_ipis non-empty AND not the literal sentinel string"TGW jumphost not created"(which the upstream HCL emits when the testing module is disabled),roksbnkctlwrites ajumphosttarget intoconfig.yaml:targets: jumphost: host: <testing_tgw_jumphost_ip> user: <testing_tgw_jumphost_user || "ubuntu"> key_source: tf-output:jumphost_shared_key -
A confirmation line is printed:
✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`
The auto-population is idempotent — re-running up against an already-jumphost-populated workspace re-writes the same fields. If you’ve manually customised the entry (changed the user, swapped to key_path), the auto-population overwrites your changes. There’s no merge logic; the latest up wins.
If testing_create_tgw_jumphost = false in tfvars, the upstream HCL skips creating the jumphost VM and emits the sentinel output. Auto-population is then a no-op, and you’re free to create your own jumphost (or differently-named) entry via targets add.
Per-AZ cluster jumphosts (jumphost-<zone>)
When testing_create_cluster_jumphosts = true, the deploy builds one cluster jumphost per cluster-VPC availability zone in addition to the single TGW jumphost — each on its own floating IP, all sharing the same key as the TGW jumphost. The same post-up hook that seeds the singular jumphost also auto-registers one target per AZ:
-
After a successful
up,roksbnkctlreads thetesting_cluster_jumphost_ipsterraform output — a map{ zone => floating-IP }. -
For each
zone => fip, it upserts a target namedjumphost-<zone>, reusing the same shared key the singularjumphostuses:targets: jumphost-ca-tor-1: host: <ca-tor-1-fip> user: ubuntu key_source: tf-output:jumphost_shared_key # …one per AZ… -
A summary line is printed:
✓ Auto-registered 3 per-AZ cluster jumphost targets (jumphost-ca-tor-1, jumphost-ca-tor-2, jumphost-ca-tor-3); use `roksbnkctl --on jumphost-<zone> ...`
Verify with roksbnkctl targets list — you should see jumphost plus one jumphost-<zone> per AZ. Each is a first-class --on target (full kubectl/oc/ibmcloud/shell passthrough, no SSH hop): roksbnkctl --on jumphost-ca-tor-2 kubectl get pods.
Preinstalled load generators. Every jumphost — the TGW
jumphostand each per-AZjumphost-<zone>— comes withiperf3andh2load(thenghttp2-clientpackage) already installed by the Testing-phaseuser_data. So bothtest throughput(iperf3) and the performance matrix (iperf3 + h2load) run over--backend ssh:<target>with no--bootstrap— the SSH backend’s apt-bootstrap path (Chapter 17 §“SSH backend”) is needed only for hosts you registered by hand. Like the singularjumphost, registration is best-effort and idempotent — a parse/write failure logs a singlewarning:and does not failup, and re-runningupafter a floating-IP rotation refreshes eachjumphost-<zone>host in place. Whentesting_create_cluster_jumphosts = false(or the output is absent/empty), this is a silent no-op — onlyjumphostis seeded, with no warning noise.
Orphaned-target caveat (option (a) upsert-only). Auto-registration upserts but never prunes. If a later apply removes a zone, or
testing_create_cluster_jumphostsis flipped tofalse, the now-orphanedjumphost-<oldzone>target points at a destroyed host and lingers in your config until you remove it by hand:roksbnkctl targets remove jumphost-ca-tor-3A host re-created on a recycled floating IP will also trip the host-key mismatch refusal — see §“Host-key TOFU and
~/.roksbnkctl/known_hosts” and clear the staleknown_hostsline withssh-keygen -R <fip> -f ~/.roksbnkctl/known_hosts. An automatic-prune (reconcile) mode that removes orphans on the nextupis a deliberate post-v1.5.0 follow-up (it needs unambiguous “this target is auto-managed” ownership semantics so a hand-namedjumphost-myboxis never deleted). See PRD 09.
What is not auto-discovered
The auto-discovery flow registers the TGW jumphost (always) and the per-AZ jumphost-<zone> targets when testing_create_cluster_jumphosts = true. It does not register:
- Per-AZ jumphosts by private IP. There is no top-level
testing_cluster_jumphost_private_ipsterraform output; the private-IP hop pattern in Chapter 16 §“Per-AZ cluster jumphosts” is a documented zero-setup technique, not an auto-registered target. - Any non-jumphost host. Bastions, ops boxes, and the like are always
targets addby hand.
Inspecting what the post-up flow saw
When the auto-population doesn’t happen and you expected it to, check:
roksbnkctl tf output testing_tgw_jumphost_ip
roksbnkctl tf output testing_tgw_jumphost_user
roksbnkctl tf output -json jumphost_shared_key | head -c 50
(The third one returns a JSON-encoded string for a sensitive output; truncate to confirm it’s non-empty without dumping the key.)
If all three are populated and the auto-write didn’t fire, that’s a bug — file an issue with the output values redacted.
What the SSH execution backend adds on top of --on
The --on <target> flag is the lightweight remote-exec path — one SSH session, one command, no remote state. The ssh:<target> execution backend layers more on top, reusing the same internal/remote.Client under internal/exec/ssh.go:
| Capability | What it gives you |
|---|---|
| File materialisation | RunOpts.Files map gets written to /tmp/roksbnkctl.<rand>/<basename> on the remote, available as the working directory for the command. Cleanup via trap 'rm -rf' EXIT in a wrapper. |
| Env passing with fallback | First tries ssh -o SetEnv=KEY=VALUE (requires remote sshd AcceptEnv). On failure, writes a 0700 wrapper script that exports the env and execs the command, with trap 'rm -f $0' EXIT to scrub. |
| Apt bootstrap | If the remote target doesn’t have a tool (iperf3, ibmcloud) installed, the backend can sudo apt-get install it on demand (Ubuntu only). |
| SCP-and-cleanup for kubeconfig | The backend’s recommended path for shipping a kubeconfig to the remote: SCP to a tempdir, run, trap 'rm -rf' EXIT to scrub. |
| Wrapper-script credential propagation | Detailed in PRD 04 § SSH. Brief on-disk window with strict cleanup. |
The targets: schema and the roksbnkctl targets commands are the same surface for both — the backend just uses each target in a heavier-weight way. Anything you set up for --on keeps working under --backend ssh:<target>.
The split between the lightweight --on path and the full ssh backend is deliberate: --on stays simple — one SSH session, one command, no remote state. The backend handles the heavier lifting (file materialisation, package installation, multi-step orchestration).
Cross-references
- Chapter 12 — Workspace config — where
targets:fits in the overall schema. - Chapter 14 — Credentials and the resolver chain — the SSH-key sources from a credential-discipline perspective.
- Chapter 16 — The –on flag and SSH jumphosts — the user-facing prose for “how do I use this”.
- Chapter 17 — Execution backends — where the SSH backend sits in the broader backend matrix.
- PRD 01 — SSH client + –on flag — the design rationale for
targets:and the SSH client. internal/remote/package: https://github.com/jgruberf5/roksbnkctl/tree/main/internal/remoteinternal/cli/targets.go: https://github.com/jgruberf5/roksbnkctl/blob/main/internal/cli/targets.go
The –on flag and SSH jumphosts
The --on <target> flag (most commonly --on jumphost) re-runs a roksbnkctl passthrough command (exec, shell, kubectl, oc, ibmcloud) on a remote SSH host instead of locally. After a successful roksbnkctl up, a jumphost target is auto-populated from the upstream HCL’s terraform outputs, so --on jumphost works with no manual configuration in the common case.
This chapter covers when to reach for --on, the targets: workspace config block, the auto-population behaviour, the roksbnkctl targets command tree for managing your own targets, and how host-key trust is established.
The full design rationale for this feature lives in PRD 01. This chapter is the user-facing distillation.
Why this exists
There are a handful of scenarios where running a command from your laptop is the wrong answer:
- Customer-firewall scenarios. Your customer’s network policy lets the corporate jumphost reach
*.cloud.ibm.combut blocks your laptop’s egress to anything except web traffic.ibmcloud iam oauth-tokensworks from the jumphost; from your laptop it times out. - Air-gapped environments. The cluster lives in a VPC with no public ingress, accessible only via a bastion VM. The cluster API server isn’t reachable from your laptop at all; you need to be inside the network to talk to it.
- Pre-cluster operations. You want to run
ibmcloudcommands against the IBM Cloud API but your workstation doesn’t haveibmcloudinstalled and you’d rather not install it. The jumphost has it; route through there.
--on makes those scenarios one flag rather than “ssh to the jumphost, install your tools there, copy your kubeconfig over manually”. The SSH client is built into roksbnkctl (using golang.org/x/crypto/ssh); no host ssh binary is required.
The targets: workspace config block
Targets are stored in your workspace config at ~/.roksbnkctl/<workspace>/config.yaml under a targets: key:
targets:
jumphost: # auto-populated after `roksbnkctl up`
host: 169.45.91.177
user: ubuntu
key_source: tf-output:jumphost_shared_key
port: 22 # default; can be omitted
bastion: # user-defined
host: ops.example.com
user: jgruber
key_path: ~/.ssh/id_ed25519
prod-jump:
host: 10.0.0.5
user: ec2-user
key_source: agent # use ssh-agent
Each entry has at minimum host and user. Port defaults to 22. Key resolution is determined by exactly one of key_path or key_source — see “Key sources” below.
You don’t typically edit this file by hand. The auto-discovery flow populates jumphost for you, and roksbnkctl targets add ... populates other entries.
Auto-discovery from roksbnkctl up
The upstream HCL provisions a small testing jumphost as part of every cluster apply. Two terraform outputs surface it:
testing_tgw_jumphost_ip— the public IP of the jumphost VM.jumphost_shared_key— the private key (PEM) the jumphost was provisioned with, markedsensitivein the HCL.
After a successful roksbnkctl up, roksbnkctl reads both outputs and writes a jumphost target into your workspace config:
✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`
The auto-registered target uses user: ubuntu (the upstream HCL provisions an Ubuntu cloud image whose default user is ubuntu).
The key_source: tf-output:jumphost_shared_key form means the private key is read from terraform state at SSH-connect time rather than being copied into the workspace config or written to disk separately. The key only ever exists in terraform state and in memory during a connect; destroying and re-creating the cluster generates a new key, and roksbnkctl picks up the new one without any manual intervention.
If your cluster apply produced a testing_tgw_jumphost_ip output of "TGW jumphost not created" (the upstream HCL emits this string when the testing module is disabled) the auto-population is skipped. You can still add a jumphost target manually with roksbnkctl targets add if you have a different bastion in mind.
Key sources
Three ways to tell roksbnkctl how to find the SSH private key:
-
key_path: <path>— a file on disk. Standard OpenSSH key formats are accepted (~/.ssh/id_ed25519,~/.ssh/id_rsa, etc.). Tilde expansion is honoured. -
key_source: agent— talk to the user’sssh-agentover the socket pointed at by$SSH_AUTH_SOCK. The agent presents whichever keys it currently holds;roksbnkctltries each in turn against the target’sauthorized_keys. This is the right setting if your team already manages keys via 1Password / hardware tokens /gpg-agentand you don’t want a key file on disk. Note: ssh-agent integration is Linux/macOS-only; Windows users should usekey_pathinstead. Windows ssh-agent named-pipe support is on the v1.x roadmap. -
key_source: tf-output:<output-name>— read the key from the workspace’s terraform state output of that name. Used by the auto-discoveredjumphosttarget. The terraform output must be a string-typed PEM-encoded private key; sensitive outputs work fine becauseterraform output -raw <name>returns the value regardless of the sensitive flag.
Exactly one of key_path or key_source must be set per target. roksbnkctl targets show <name> will tell you which is in use without printing the key material.
Host-key TOFU on first connect
The first time you connect to a target, roksbnkctl shows the host key fingerprint and asks whether to trust it. The prompt is a single line:
$ roksbnkctl exec --on jumphost -- whoami
Add 169.45.91.177:22's key (SHA256:abc123def456ghi789jkl0mnopqrstuvwxyz/+=) to ~/.roksbnkctl/known_hosts? [y/N]: y
ubuntu
Answer y and the key is appended to ~/.roksbnkctl/known_hosts (the same format as OpenSSH’s ~/.ssh/known_hosts). Subsequent connects trust silently.
Answer n and the connect fails with a clear “host key not trusted” error.
If the host key changes between runs — which would happen on a re-provisioned VM, or could happen as a man-in-the-middle attack — roksbnkctl refuses to connect:
error: host key mismatch: 169.45.91.177:22 known with SHA256:abc123... but server presented SHA256:zyx987...; if the host was rebuilt, edit ~/.roksbnkctl/known_hosts
This is “trust on first use” (TOFU) — the same model OpenSSH uses for new hosts. Exit code is 126 on host-key rejections.
--insecure-host-key for CI
In automation contexts where a TOFU prompt would block forever, pass --insecure-host-key to skip host-key verification entirely:
roksbnkctl exec --on jumphost --insecure-host-key -- whoami
This is insecure — anyone in the network path can MITM the connection — and is intended only for short-lived CI runs against ephemeral test infrastructure. Don’t use it in any context where the SSH session matters for security.
The roksbnkctl targets command tree
Four subcommands for managing target entries:
roksbnkctl targets list
roksbnkctl targets show <name>
roksbnkctl targets add <name> --host ... --user ... --key-path ...
roksbnkctl targets remove <name>
targets list
roksbnkctl targets list
NAME HOST USER KEY
jumphost 169.45.91.177:22 ubuntu tf-output:jumphost_shared_key
bastion ops.example.com:22 jgruber file:~/.ssh/id_ed25519
Prints every target in the current workspace’s config. The KEY column shows the key source — never the key material itself. File-backed keys are prefixed with file: so they’re visually distinct from tf-output: and agent sources.
targets show <name>
roksbnkctl targets show jumphost
name: jumphost
host: 169.45.91.177
port: 22
user: ubuntu
key_source: tf-output:jumphost_shared_key
Prints the full record. Note that the key material itself is never printed — only the source descriptor (file path, ssh-agent, or terraform-output name).
targets add <name> ...
roksbnkctl targets add bastion \
--host ops.example.com \
--user jgruber \
--key-path ~/.ssh/id_ed25519
# or with ssh-agent:
roksbnkctl targets add prod-jump \
--host 10.0.0.5 \
--user ec2-user \
--key-source agent
# or with a non-default port:
roksbnkctl targets add custom \
--host 10.0.0.5 \
--user root \
--key-path ~/.ssh/custom \
--port 2222
Writes the new target into ~/.roksbnkctl/<workspace>/config.yaml. Refuses if a target of that name already exists (use targets remove first).
targets remove <name>
roksbnkctl targets remove bastion
Removes the entry from config.yaml. Does not remove the corresponding line from ~/.roksbnkctl/known_hosts — the host key stays recorded so re-adding the same target later doesn’t re-trigger a TOFU prompt.
Working examples
The everyday verbs:
# Run an arbitrary command on the jumphost
roksbnkctl exec --on jumphost -- whoami
# → ubuntu
roksbnkctl exec --on jumphost -- uname -a
# → Linux jumphost-vm 5.15.0-... #... SMP ... x86_64 GNU/Linux
# Interactive PTY shell
roksbnkctl shell --on jumphost
# → drops you into the jumphost's default shell as the configured user
# → exit returns you to your local prompt
# ibmcloud passthrough — runs `ibmcloud ks cluster ls` on the jumphost
# (handy when your laptop's network can't reach IBM Cloud APIs)
roksbnkctl ibmcloud --on jumphost ks cluster ls
# kubectl passthrough — same pattern
roksbnkctl kubectl --on jumphost get pods -A
# oc passthrough
roksbnkctl oc --on jumphost projects
Per-AZ cluster jumphosts
When your deploy sets testing_create_cluster_jumphosts = true, the upstream HCL builds one cluster jumphost per cluster-VPC availability zone in addition to the single TGW jumphost. These are auto-registered by roksbnkctl up as jumphost-<zone> targets (alongside the singular jumphost) — see Chapter 15 §“Auto-discovery from terraform outputs”. They are first-class --on targets: full passthrough, no hop.
# verify what `up` registered:
roksbnkctl targets list
# → jumphost, jumphost-ca-tor-1, jumphost-ca-tor-2, jumphost-ca-tor-3
# run directly on a specific per-AZ cluster jumphost (no hop, full passthrough):
roksbnkctl kubectl --on jumphost-ca-tor-2 get nodes
roksbnkctl shell --on jumphost-ca-tor-1
Preinstalled perf tools. Every jumphost ships with
iperf3andh2load(thenghttp2-clientpackage) preinstalled by the Testing-phaseuser_data, so the performance matrix andtest throughputrun overssh:<target>with no--bootstrap. The matrix’s locality axis (same-zone / different-zone / different-VPC) is just whichjumphost-<zone>(or the TGWjumphost) a cell names as its client — see Chapter 22a.
Hopping to a cluster jumphost via the registered jumphost (zero-setup, no roksbnkctl state)
The shared key is installed on every jumphost and the private key file is present on each box at /home/ubuntu/.ssh/id_rsa. So from the auto-registered TGW jumphost you can hop to any cluster jumphost by its private IP (the TGW jumphost reaches the cluster VPC over the Transit Gateway) with no key copying and nothing added to roksbnkctl state — handy for a one-off against a host you don’t want to register:
# the per-AZ private IPs are not a top-level terraform output; read them
# from inside the TGW jumphost (it sits in the routed network):
roksbnkctl exec --on jumphost -- \
curl -s http://169.254.169.254/... # or your own inventory source
# run a command on a cluster jumphost, hopping through the TGW jumphost,
# using the shared key already on the box (no scp):
roksbnkctl exec --on jumphost -- \
ssh -i /home/ubuntu/.ssh/id_rsa -o StrictHostKeyChecking=accept-new \
ubuntu@<cluster-jumphost-private-ip> kubectl get nodes
The inner ssh uses the on-box /home/ubuntu/.ssh/id_rsa (the same shared key the auto-registered targets use — one key for all jumphosts), so no key material is copied. StrictHostKeyChecking=accept-new on the inner hop avoids an interactive prompt the outer non-PTY session can’t answer. This is the zero-setup path: nothing is written to ~/.roksbnkctl/. For first-class repeated access prefer the auto-registered jumphost-<zone> targets above.
Orphaned-target caveat. The auto-registered
jumphost-<zone>targets use option-(a) upsert-only registration: if a destroy removes a zone (ortesting_create_cluster_jumphostsis flipped tofalse), the stalejumphost-<oldzone>entry lingers in your config until youroksbnkctl targets remove jumphost-<oldzone>by hand. A re-created host on a recycled floating IP will also trip the host-key TOFU mismatch — see Chapter 15 §“Host-key TOFU and~/.roksbnkctl/known_hosts”. An automatic-prune (reconcile) mode is a tracked post-v1.5.0 follow-up.
Behaviour details worth knowing:
- Streaming I/O. stdout, stderr, stdin all stream in real time — the same as running the command locally. Long-running commands (
oc adm top nodes,ibmcloud ks cluster geton a slow API call) work normally. - Exit code propagation. The remote command’s exit code is the local exit code. A failing remote command produces a non-zero
roksbnkctlexit; a succeeding remote command produces0. CI scripts can rely on this. - TTY auto-detection.
roksbnkctl shell --onauto-allocates a PTY. Other verbs (exec,kubectl,oc,ibmcloud) run without a PTY; if you need a PTY fortopor anotherisatty()-sensitive command, fall back toroksbnkctl shell --on jumphostand run the command from the interactive shell. - Environment passthrough. Only machine-portable values cross the SSH boundary:
IBMCLOUD_API_KEY,IC_API_KEY,IBMCLOUD_REGION, andIBMCLOUD_VERSION_CHECKare propagated to the remote session via SSHSetEnv, soibmcloud iam oauth-tokenson the jumphost authenticates with the same key your local workspace uses. The remote sshd must be configured to acceptAcceptEnv IBMCLOUD_*etc. for this to work; the upstream HCL’s jumphost is already configured for it.KUBECONFIGis not forwarded — it is a local filesystem path, meaningless on the target, and forwarding it would shadow the jumphost’s own cloud-init-provisioned/home/ubuntu/.kube/config. Passthroughkubectl/ocon the target therefore use the target’s own kubeconfig, which is the correct behaviour. Localroksbnkctl kubectlwithout--onstill resolvesKUBECONFIGvia the local chain, unchanged.
What --on doesn’t do (yet)
A few things deliberately deferred to later phases:
- Lifecycle commands (
up,down,plan,apply) reject--onwith a clear error. Running terraform on a remote host has different state-handling considerations and is the job of the SSH execution backend (Chapter 17) —terraformover--backend ssh:<target>is itself deferred to v1.x (state-file portability). - ProxyJump / multi-hop SSH. If your jumphost itself is reached through another bastion, that’s not directly supported. The upstream HCL’s jumphost design lets the TGW jumphost reach cluster-internal VMs natively, so you usually don’t need multi-hop in practice. ProxyJump support is on the v1.x roadmap. (The per-AZ cluster jumphosts themselves are auto-registered as first-class
jumphost-<zone>targets — see §“Per-AZ cluster jumphosts” above; the manualssh -i … via the registered jumphosthop there is the zero-setup alternative for one-offs, not aroksbnkctl-native multi-hop.) ~/.ssh/configparsing. Targets must be defined explicitly in workspace config;roksbnkctldoes not read your existing~/.ssh/config.- Password auth. Keys + agent only. Passwords are not supported and won’t be.
- SCP / SFTP. File transfer is the SSH execution backend’s job (handled via
RunOpts.Filesmaterialisation; see Chapter 17 §“SSH backend”).--ondoes one-shot remote exec only. - Windows ssh-agent. The
key_source: agentpath is Linux/macOS only; Windows users must usekey_pathto a file. Already noted in Key sources above; called out here so a Windows reader who skipped to this section doesn’t miss it.
Cross-reference
Chapter 17 — Execution backends extends the SSH client used here into a full execution backend with file materialisation, env-file fallback for sshd configurations that can’t AcceptEnv, and apt-bootstrap of missing tools on Ubuntu jumphosts. The --on flag stays as the lightweight one-shot path; --backend ssh is the deeper integration. The two share the same internal/remote.Client so what you learn here translates directly.
For the design rationale, edge cases, and open questions, read PRD 01 — this chapter is the user-facing surface; PRD 01 is the developer-facing surface.
Execution backends: local, docker, k8s, ssh
Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the
Workspacestruct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.
roksbnkctl runs a handful of external tools as part of its job — ibmcloud, terraform, iperf3, eventually dig-equivalents and others. By default each tool runs as a child process on your laptop. That’s fine for some tools and wrong for others: iperf3 from your laptop measures your laptop’s internet uplink, not the cluster’s bandwidth. Likewise, terraform via docker (the --backend docker mode covered below) lets you pin a frozen tool version for CI reproducibility without installing it on the host.
The execution-backend system lets you pick where each tool runs without changing the surface command. The same roksbnkctl ibmcloud ks cluster ls invocation can run as a local process, inside a vendored container, inside the cluster, or on a remote SSH host — selected by a flag or a per-tool default in your workspace config.
This chapter is the user-facing reference for all four backends. After the introduction, each backend gets its own deep-dive section covering the mechanics, the credential-propagation rules, the failure modes, and a short “when to use it” callout. Chapter 18 is the decision-tree companion that picks one for a given (tool, scenario) pair.
Architecture at a glance
The four backends sit between roksbnkctl (the binary on your laptop) and the external tools each backend runs. Every backend produces the same observable behaviour from the user’s point of view — the same roksbnkctl ibmcloud ks cluster ls invocation — but routes the actual tool execution to a different network vantage.
graph LR
User[laptop<br/>roksbnkctl binary]
subgraph local[local backend]
L_tf[terraform]
L_ibm[ibmcloud]
L_iperf[iperf3]
L_dns[dns probe]
end
subgraph docker[docker backend]
D_ibm[ibmcloud<br/>frozen image]
D_tf[terraform<br/>frozen image]
end
subgraph k8s[k8s backend]
K_ops[ops pod<br/>ibmcloud]
K_job[one-shot Job<br/>iperf3 / dns]
end
subgraph ssh[ssh:target backend]
S_ibm[ibmcloud<br/>on jumphost]
S_iperf[iperf3<br/>on jumphost]
end
Cluster[ROKS cluster<br/>cert-manager + flo + BNK]
Jump[SSH jumphost<br/>auto-discovered from terraform]
IBMAPI[IBM Cloud API<br/>+ IAM]
User --> local
User --> docker
User --> k8s
User --> ssh
local --> IBMAPI
docker --> IBMAPI
k8s --> Cluster
Cluster --> IBMAPI
ssh --> Jump
Jump --> IBMAPI
Jump -.->|optional<br/>private path| Cluster
classDef bk fill:#f4f4f4,stroke:#666,color:#000;
class local,docker,k8s,ssh bk;
Chapter 18 is the decision-tree companion that picks a backend for a given scenario; the rest of this chapter is the per-backend mechanics.
The four backends at a glance
| Backend | What it does |
|---|---|
| local | os/exec — spawns the tool as a child process, inheriting your env and PATH |
| docker | docker run against a vendored image (ghcr.io/jgruberf5/roksbnkctl-tools-<tool>:<v>); frozen toolchain version |
| k8s | Runs inside the cluster, either in a long-lived ops pod or as a one-shot Job; auth via the pod’s ServiceAccount token |
| ssh | Runs on a registered SSH target via the built-in SSH client; opt-in apt-bootstrap of missing tools on Ubuntu |
Each backend solves a different problem:
- local: fastest startup, simplest mental model, requires the host tool to exist on
PATH. - docker: reproducible across dev machines, no host install needed, frozen at a known-good tool version.
- k8s: network-correct (private IPs reachable, cluster-internal services accessible), zero host install, in-cluster identity via ServiceAccount.
- ssh: pre-cluster ops from a known-IP bastion, customer-firewall workflows, air-gapped environments where the laptop can’t reach IBM Cloud APIs but the jumphost can.
All four implementations conform to the same Go interface (internal/exec.Backend) so callers don’t branch on backend type — they just call backend.Run(ctx, argv, opts) and let the implementation handle the mechanics. That uniformity is what lets the same roksbnkctl ibmcloud ks cluster ls work across all four with no surface-level change.
The --backend CLI flag
Override the per-tool default for a single invocation:
# Local (the implicit default for ibmcloud + terraform)
roksbnkctl ibmcloud ks cluster ls
# Same command, in a vendored docker image
roksbnkctl ibmcloud --backend docker ks cluster ls
# Same command, in the cluster (requires `roksbnkctl ops install` first)
roksbnkctl ibmcloud --backend k8s ks cluster ls
# Same command, on a remote SSH host
roksbnkctl ibmcloud --backend ssh:jumphost ks cluster ls
Format:
--backend local|docker|k8s|ssh:<target>
The ssh:<target> form pins the SSH backend to a specific named target from roksbnkctl targets list (registered via roksbnkctl targets add; see Chapter 16).
The flag is persistent at the root — it works for any command that runs an external tool. Commands that don’t run external tools (like roksbnkctl ws list) ignore it.
The flag wins over the workspace-config default. If config.yaml says iperf3: { backend: k8s } and you pass --backend local, the local backend runs.
Backend-failure semantics
Each backend has a different failure surface. The convention is:
- Backend startup failure (Docker daemon unreachable, k8s API unreachable, SSH connect refused, binary not on PATH for
local) ⇒ exit code127, with a message naming the cause. No silent fallback tolocal. Silent fallback hides intent and produces confusing test results. - Backend mid-run failure (the container started but couldn’t pull a sub-resource; the pod was OOMKilled before the wrapped tool ran; the SSH session died after
apt-get installbut before the tool exec) ⇒ exit code126, distinct from127so CI can tell “we never got going” from “we got going then broke”. - Tool exit code (the actual
ibmcloud/terraform/iperf3exit code, anything in0-125or128-255) ⇒ propagated 1:1, including non-zero codes. - Context cancellation / timeout ⇒ exit code
137(the conventional SIGKILL-on-signal code).
This way, your CI script can tell “the tool said X failed” (typical exit codes) from “we never reached the tool” (127) from “we reached the tool, then the backend died mid-flight” (126) from “we ran out of time” (137).
One code outside the backend surface: 125
roksbnkctl upgrade and roksbnkctl self update exit 125 when an upgrade removed the old binary and could not put anything back — so there is no roksbnkctl at the install path and the only copy is the .old sidecar beside it. The error names the file and the command to rename it.
This is separate from 1 because the two need opposite responses. An ordinary failed upgrade is safe to retry; this one cannot be retried at all, because there is nothing left to run. A wrapper that treats them alike either loops forever on a machine that will never recover, or reports a bricked install as routine.
Note the overlap with the passthrough range above. It is deliberate and bounded, but it is an overlap: a wrapped tool that exits 125 still propagates 125, exactly as the table says. What cannot happen is a single invocation meaning both — upgrade and self update spawn no child process at all (extraction, checksum verification and the install are in-process), so a 125 from those two commands has exactly one meaning, and a 125 from anything else is a child’s status. This is the same bargain already made for 2, which is both “bad invocation” and whatever a child chose to exit with.
In practice this fires on Windows, where a running .exe is locked and the upgrade works by renaming the old binary aside before moving the new one in. A second process holding a handle at that moment — an antivirus scanner opening the freshly written file is the usual one — can make the second rename fail; if the rollback then fails too, this is the result.
Per-tool defaults from exec:
Workspace config carries the per-tool default backend in the exec: block:
# ~/.roksbnkctl/<workspace>/config.yaml
exec:
ibmcloud: { backend: local }
iperf3: { backend: k8s }
terraform: { backend: local }
The defaults shipped today:
| Tool | Default backend | Supported backends | Why |
|---|---|---|---|
terraform | local | local, docker | The terraform-exec local path is the established workflow. State handling is simplest here. The docker backend runs frozen hashicorp/terraform:1.5.7 with a bind-mounted state dir — see § terraform via docker. k8s and ssh are deferred to v1.x. |
ibmcloud | local | local, docker, k8s, ssh:<target> | Most users have it on PATH or are happy installing it. Compliance/firewall scenarios opt in via --backend ssh:jumphost or docker. |
iperf3 | k8s | local, k8s, ssh:<target> | Throughput from a laptop’s uplink isn’t the cluster’s bandwidth. The k8s default runs the iperf3 client adjacent to (or inside) the cluster so the number reflects cluster fabric, not your office Wi-Fi. |
dns | local | local, k8s, ssh:<target> | Single-vantage by default; --gslb-compare fans out across configured vantages for GSLB validation. See Chapter 21. |
Chapter 12 — Workspace config covers the exec: block schema in detail; this chapter just notes its place in the backend system.
Chapter 18 — Choosing a backend per tool is the decision tree for “which backend should I pick for this tool in this scenario”.
Per-backend deep dives
local backend
The default for ibmcloud and terraform. os/exec.CommandContext(ctx, argv[0], argv[1:]...), inheriting the parent process’s environment, PATH, and working directory. Mechanically the simplest of the four — no container, no cluster, no network handshake.
os/exec shape
internal/exec/local.go resolves argv[0] via exec.LookPath, then builds a *exec.Cmd:
bin, err := exec.LookPath(argv[0])
// fall through to argv[0] verbatim if it's an absolute path that LookPath rejects
cmd := exec.CommandContext(ctx, bin, argv[1:]...)
cmd.Env = effectiveEnv // os.Environ() + opts.Env + Credentials.EnvVars()
cmd.Dir = opts.WorkDir // empty → inherit caller's CWD
cmd.Stdin = opts.Stdin
cmd.Stdout = redactor(opts.Stdout, creds)
cmd.Stderr = redactor(opts.Stderr, creds)
The redactor wrap is defense-in-depth — see Chapter 14 §“The redactor”. If a wrapped tool ever prints IBMCLOUD_API_KEY value to stdout (a debug trace, an error message), the redactor replaces it with [REDACTED] before the bytes leave the binary.
Env propagation
Three sources, in order:
- The host process’s environment (
os.Environ()) — your shell’sPATH,HOME,KUBECONFIG, etc. RunOpts.Env— caller-suppliedKEY=VALUEstrings (e.g.,IBMCLOUD_REGION=ca-torfrom the workspace config).Credentials.EnvVars()—IBMCLOUD_API_KEY=…plus the legacyIC_API_KEY=…alias olderibmcloudversions accept.
os/exec documents that for duplicate keys the last entry wins. So caller-supplied vars override host env, and credential vars override caller-supplied — meaning a workspace’s API key always wins over a stale IBMCLOUD_API_KEY in your shell.
The local backend does not scrub the host env. If you have an unrelated AWS_ACCESS_KEY_ID in your shell, the wrapped tool sees it. That’s by design — local is the “trust the user’s shell” path; if you want a hermetic env, switch to docker.
Working directory
RunOpts.WorkDir becomes cmd.Dir. Empty → inherit the caller’s CWD (Cobra’s RootCmd.Run runs from wherever the user invoked roksbnkctl).
When RunOpts.Files is non-empty and WorkDir is empty, the local backend creates a tempdir under os.TempDir(), writes each Files entry as a 0600 file inside, and uses the tempdir as WorkDir. The tempdir is removed via defer after Run returns. This is mostly there for symmetry with the docker / k8s / ssh backends; today’s ibmcloud passthrough never uses it.
Signal handling
exec.CommandContext wires ctx cancellation to the child: when the ctx ticks past its deadline (or the user hits Ctrl-C and the root cobra command cancels), Go sends SIGKILL (the default Cmd.Cancel) to the child. The child has no opportunity to clean up; this is intentional — we’d rather kill a stuck terraform than wait on an indefinite hang.
The kill is process-only, not process-group. If terraform has spawned grandchildren (the IBM provider’s helpers, an SSH key generator, etc.) those grandchildren may outlive the ctx-cancel by a few seconds. We haven’t seen this matter in practice; if it does, a pgid kill is a small follow-up.
Exit-code mapping
| Outcome | Exit code | Source |
|---|---|---|
| Child exits 0 | 0 | child |
Child exits non-zero (e.g., terraform plan saw drift) | child’s exit code, 1-125 or 128-255 | child |
argv[0] not on PATH and not an absolute path | 127 | local backend (POSIX shell convention) |
| Child binary couldn’t be exec’d despite being present (e.g., not executable) | 126 | local backend (mid-run failure: we found the binary but couldn’t spawn it) |
| Ctx cancelled mid-run, child SIGKILL’d | 137 | 128 + SIGKILL |
Note the 126 vs 127 split: 127 means “we never reached the tool” (binary missing, daemon unreachable, SSH refused); 126 means “we reached the tool but the backend itself broke after that point” (couldn’t fork, container created but crashed, pod scheduled but evicted before exec). The local + docker backends split 126 vs 127 per PRD 03 §“Backend interface”. CI scripts that distinguish “test infra broken” from “real test failure” can now key on the difference.
upgrade / self update add one code of their own, listed separately because they wrap no child:
| Outcome | Exit code | Source |
|---|---|---|
| Upgrade left no binary at the install path (rollback also failed) | 125 | exitcode.SelfUpdateStranded |
When to use it
- You have the tool installed and on PATH already.
- You want the fastest startup — no container daemon, no SSH handshake, no cluster API call.
- You’re running
terraformagainst the workspace’s local state (the established workflow). - You’re debugging and want the simplest mental model for “where did that output come from”.
Chapter 18 §“Decision tree” expands these into a per-(tool, scenario) walkthrough.
docker backend
Runs the tool inside a vendored container image, talking to the local docker daemon over its socket. docker on PATH is not required — roksbnkctl uses the official Docker Go SDK (github.com/moby/moby/client) and dials the socket directly.
roksbnkctl ibmcloud --backend docker ks cluster ls
Container shape
Mechanically (the ibmcloud passthrough; iperf3 client is similar with a different image and ports):
docker run --rm \
-v <tempdir>/kubeconfig:/root/.kube/config:ro \ # if Credentials.KubeconfigBytes set
-e IBMCLOUD_API_KEY \ # bare name; value inherits
-e IC_API_KEY \ # legacy alias
ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:<v> \
ks cluster ls
internal/exec/docker.go doesn’t shell out to docker run; it builds a container.Config + container.HostConfig and calls cli.ContainerCreate → ContainerStart → ContainerLogs(stream=true). The bash-style above is the conceptual equivalent.
There’s no workspace-wide bind-mount. Per-invocation mounts come from three sources only:
Credentials.KubeconfigBytes— written to<tempdir>/kubeconfig(mode0600) on the host, bind-mounted as a single file at/root/.kube/configread-only. Single-file mount per PRD 04 §“Anti-patterns” — bind-mounting~/.kube/exposes other clusters’ configs.RunOpts.Files— each name → bytes entry written to<tempdir>/<basename>and bind-mounted at/work/<basename>. The container’sWorkingDiris set to/workso callers can reference files by relative path. (ibmcloudpassthrough doesn’t use this; it lands when the iperf3 client backend wants to shipiperf3.jsonto the pod, or when a future tool wants a config file.)RunOpts.WorkDir— overridesWorkingDirif explicitly set.
The tempdir is removed via defer after Run returns, regardless of exit code or panic.
Credential propagation specifics
Three things matter, all enforced by internal/exec/creds.go::Credentials.DockerArgs(...):
--env IBMCLOUD_API_KEY(bare name, no=value). The docker daemon looks up the value from the daemon’s environment at container-create time, not from argv. So the literal API key string never appears indocker inspect,docker ps -a --format, or the daemon’s container metadata. PRD 04 §“Anti-patterns” calls out the--env IBMCLOUD_API_KEY=$KEYform as a leak vector — we don’t use it. See Chapter 14 —Credentials.DockerArgs()for the full call shape.- Single-file kubeconfig mount, read-only. Not the parent dir. The container can read exactly the kubeconfig you handed it — nothing else under
~/.kube/. - Stdout/stderr through the redactor. Same defense-in-depth as the local backend: if the wrapped tool prints the API key value (rare but possible), the redactor masks it before the bytes leave
roksbnkctl’s process.
:dev tag resolution
The vendored images live at:
| Tool | Image |
|---|---|
ibmcloud | ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:<tag> (vendored from icr.io/ibm-cloud/ibmcloud-cli upstream) |
iperf3 | ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<tag> (Alpine + iperf3) |
h2load | ghcr.io/jgruberf5/roksbnkctl-tools-h2load:<tag> (Alpine + nghttp2; the L7 load generator for test matrix) |
terraform | hashicorp/terraform:<v> (official upstream) |
The h2load image is the L7 generator the performance matrix runs for family: l7 cells. It’s built from tools/docker/h2load/Dockerfile (Alpine’s nghttp2 package, built against OpenSSL so https / TLS-terminate-at-TMM targets work) and published by the tools-images workflow alongside the other per-tool images. Like the iperf3 image it declares USER 1000 so it satisfies the OpenShift restricted-v2 SCC (h2load is a pure client and binds no privileged port). The docker / k8s backends pull this image; the ssh backend uses the jumphost’s apt-installed nghttp2-client instead (preinstalled by the Testing phase — see Chapter 15 §“Per-AZ cluster jumphosts”).
The <tag> for the vendored per-tool images (ibmcloud, iperf3) is resolved at runtime by internal/exec/docker.go::toolImageTag(). It reads the binary’s internal/version.Version (set via ldflags at build time): a release-built binary like v0.10.0 pulls :v0.10.0; a dev build (Version == "dev") pulls :dev. The tag is version-pinned, so a go install of a tagged release pulls a matching tagged image rather than a :dev that may not exist for the published binary. The terraform row is the exception — it points at the upstream hashicorp/terraform image and stays pinned to a specific version (currently 1.5.7) regardless of roksbnkctl’s own version.
The :dev tag is still the local-development idiom: cd tools/docker && make build-all builds and tags every tools image as :dev locally; a dev-build roksbnkctl finds them via the local docker cache without a ghcr.io round-trip.
If you’re cutting a custom tools image and want roksbnkctl to pick it up, the simplest path is docker tag your-image ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:dev locally — the docker backend pulls the local-cached version first.
Auto-remove and ctx-cancel-kill
Two cleanup mechanisms work together:
AutoRemove: trueinHostConfig. The docker daemon removes the container as soon as it exits, regardless of exit code. Nodocker ps -aclutter, no manualdocker rmever required.- Ctx-cancel triggers
ContainerKill. Whenctx.Done()fires, the docker backend issuescli.ContainerKill(ctx, id, "SIGKILL")and waits a few seconds for the daemon to confirm. The--rmthen takes care of removal. Net effect: hitting Ctrl-C during a stuckibmcloud logindoesn’t leave a zombie container behind.
Combined with the daemon’s own watchdog on the container, the worst case is a few seconds of “container is dying” between Ctrl-C and the container disappearing. We haven’t seen leaked containers in dev or CI.
Image build pipeline
Image versions are tagged in lock-step with roksbnkctl releases; the GitHub Actions workflow that builds + pushes them runs on every release tag. See Chapter 31 — Building from source for the build pipeline details.
terraform via docker
terraform is the second tool routed through the docker backend (alongside ibmcloud). The shape is similar to ibmcloud (docker run against a vendored image, single-file mounts for sensitive data, no creds in argv) but with two terraform-specific concerns: state persistence across runs, and host-user UID alignment so state files written inside the container stay readable on the host.
State persistence via bind-mount
Terraform’s local state file lives at terraform.tfstate in the working directory. For the docker backend the working directory has to be a host-side path bind-mounted into the container, not a container-internal path that disappears on --rm. The docker backend bind-mounts the workspace’s state directory into the container:
docker run --rm \
-v ~/.roksbnkctl/<workspace>/state:/state \
--workdir /state/tf-source/embedded-terraform \
--user $(id -u):$(id -g) \
hashicorp/terraform:1.5.7 \
apply -auto-approve
Concretely:
- Host source:
~/.roksbnkctl/<workspace>/state/— the same directory the local terraform backend writes state to today, so switching between--backend localand--backend dockeragainst the same workspace doesn’t fork state. - Container target:
/state— the bind-mount root inside the container. - Container working directory:
/state/tf-source/<source>/(e.g.,/state/tf-source/embedded-terraform/for the default embedded source) — the same path the local backend resolves to, so terraform sees the samemain.tfeither way. - The HCL is bind-mounted alongside state. The embedded HCL is materialised at run time into
~/.roksbnkctl/<workspace>/state/tf-source/<source>/(chapter 31 covers the embedded-source layout); sincestate/is the bind-mount root, bothterraform.tfstateand the HCL tree land inside the container together. There’s no separate HCL projection.
The bind-mount is read-write — terraform needs to write terraform.tfstate, rotate terraform.tfstate.backup, and populate the .terraform/ cache. Combined with --rm, the file lifecycle is: container creates state, container exits, --rm removes the container, state files persist on the host. Subsequent runs (re-mounted at the same host path) pick up where the prior run left off.
Image: hashicorp/terraform:1.5.7
The image is the official upstream hashicorp/terraform published by HashiCorp on Docker Hub, pinned to a literal version in internal/exec/docker.go’s toolImages map (currently 1.5.7). The pin is intentional — the embedded HCL has been validated against this terraform version, and the docker backend’s whole point is reproducibility. Bumping the pin is a deliberate change to the binary and lands as a release.
The vendored per-tool images (ibmcloud, iperf3) get their tag from the roksbnkctl binary’s own version (see § :dev tag resolution above). Terraform is the exception — the binary’s version doesn’t follow upstream terraform’s release cadence, so the pin stays literal.
The UID/GID alignment gotcha
Linux Docker containers run as root by default. With a root-owned container writing into a bind-mount, the resulting host files end up owned by root — and any subsequent local-backend terraform apply (or even a cat ~/.roksbnkctl/<ws>/state/terraform.tfstate) hits permission errors. The docker backend works around this by passing --user $(id -u):$(id -g) explicitly:
docker run --rm \
--user 1000:1000 \ # host's caller-uid:caller-gid
-v ~/.roksbnkctl/dev-tor/state:/state \
--workdir /state/tf-source/embedded-terraform \
hashicorp/terraform:1.5.7 \
apply -auto-approve
The container process runs as the host user, so files written into the bind-mount are owned by the host user — same as a local-backend terraform apply would have produced. Switching backends mid-debug doesn’t strand state files behind a permission wall.
The UID/GID values are read from the host process at run time (Go’s os.Getuid() / os.Getgid()). On macOS this is mostly cosmetic — Docker Desktop’s VM normalises ownership on the host bind-mount automatically — but it’s required for clean Linux behaviour, so the backend always passes the flag.
Supported commands
The terraform docker backend honours --backend docker for the four lifecycle commands:
roksbnkctl up --backend docker [--var-file <path>] [--auto]
roksbnkctl plan --backend docker [--var-file <path>]
roksbnkctl apply --backend docker [--var-file <path>] [--auto]
roksbnkctl down --backend docker [--var-file <path>] [--auto]
Flags that the local terraform backend honours (--var-file, --auto, plus the -w/--workspace selector) plumb through to the docker backend identically — the backend’s job is to spawn hashicorp/terraform:1.5.7 with the right argv; it doesn’t filter or rewrite the lifecycle commands’ flags. (--auto is roksbnkctl’s shorthand for terraform’s -auto-approve; the wrapper renames it for terseness and consistency across up/apply/down.)
roksbnkctl up --backend docker is the apply-with-auto-approve shorthand the existing local lifecycle uses; --backend docker switches the spawn target without changing the command shape.
Deferred: k8s and ssh terraform backends
--backend k8s and --backend ssh:<target> for terraform are not in v1.0. The blocker is state-handling: the local backend keeps state on the host filesystem, the docker backend bind-mounts the same path, but k8s (run terraform in a one-shot Job) and ssh:<target> (run terraform on a remote host) need a story for shipping state between the run vantage and the canonical workspace state dir. Designs under consideration include a versioned ConfigMap/Secret pair for k8s and an scp-pre-and-post atomic move for ssh; both are deferred to v1.x once the trade-offs have settled (see docs/PLAN.md §“What’s deliberately deferred to post-v1.0”).
PRD 03 §“State concerns” is the design spec; trying --backend k8s against terraform errors at parse time:
$ roksbnkctl up --backend k8s
error: terraform doesn't support backend `k8s` at v1.0 (state-handling design
open; tracked in PRD 03 § State concerns); supported: local, docker
When to use it
- You’re on a clean dev machine without
ibmcloudinstalled and don’t want to install it. - You need a frozen tool version for CI reproducibility.
- You’re debugging a “works on my machine” issue and want to factor out the host install variable.
When docker is the wrong call:
- The tool needs network access that your laptop has but the container doesn’t (rare; default bridge networking usually preserves laptop’s egress).
- You’re running
iperf3and want a network-locality benefit —dockerdoesn’t give you that vslocal. Usek8sinstead. - You’re running a DNS probe and want a different network vantage — same network identity as the host, no value-add. The DNS subcommand rejects
--backend dockerby design. - You’re on Windows. Linux/macOS docker daemons are in scope; Windows Docker Desktop coverage is deferred to a future round.
k8s backend
Runs the wrapped tool inside the cluster. Two distinct execution patterns share the same Backend.Run interface:
| Pattern | Used for | Lives in | Lifetime |
|---|---|---|---|
| Long-lived ops pod | ad-hoc ibmcloud commands, future interactive shells | roksbnkctl-ops namespace | manually managed via roksbnkctl ops install/uninstall |
| One-shot Job | iperf3 client runs, future terraform runs, future DNS probes | roksbnkctl-test namespace | per-invocation; auto-deleted after ttlSecondsAfterFinished: 60 |
The split mirrors the two latency budgets. Long-lived pods amortise the pod-startup cost across many invocations — perfect for ibmcloud iam oauth-tokens which you might run twenty times in a debugging session. One-shot Jobs are clean (no leftover state, no concurrency questions) — perfect for iperf3 -c <server> which runs once, emits its JSON, and exits.
Long-lived ops pod pattern
The pod is named roksbnkctl-ops in the roksbnkctl-ops namespace. roksbnkctl ops install deploys it (see Chapter 19 for the full lifecycle). The image bundles ibmcloud CLI plus kubectl as backup; future iterations may add oc, terraform, etc. The container inside the pod is named tools.
Backend.Run(ctx, argv, opts) for the ops-pod path is essentially:
exec, _ := remotecommand.NewSPDYExecutor(restConfig, "POST",
clientset.CoreV1().RESTClient().Post().
Resource("pods").Namespace("roksbnkctl-ops").Name("roksbnkctl-ops").
SubResource("exec").
VersionedParams(&corev1.PodExecOptions{
Container: "tools",
Command: argv,
Stdin: opts.Stdin != nil,
Stdout: true,
Stderr: true,
TTY: opts.TTY,
}, scheme.ParameterCodec).URL())
exec.StreamWithContext(ctx, remotecommand.StreamOptions{
Stdin: opts.Stdin, Stdout: redactor(opts.Stdout, creds), Stderr: redactor(opts.Stderr, creds), Tty: opts.TTY,
})
The exit code comes back via the SPDY channel’s metav1.Status — the executor surfaces it as a exec.CodeExitError. We propagate that as the backend’s exit code, same as local propagates exec.ExitError.ExitCode().
opts.WorkDir is ignored for the ops pod path. The pod’s WorkingDir is fixed at container-spec time (/work); per-exec working-dir changes would require recreating the pod. Callers that need a specific cwd should cd <dir> && it into argv (the local backend’s symmetric escape hatch).
One-shot Job pattern
For each invocation, the backend builds a batchv1.Job spec, applies it, streams logs from the Job’s pod, waits for completion, reads the exit code from the pod’s container status, and lets ttlSecondsAfterFinished clean up.
Skeleton:
apiVersion: batch/v1
kind: Job
metadata:
generateName: roksbnkctl-iperf3-client- # randomized; multiple runs don't collide
namespace: roksbnkctl-test
spec:
ttlSecondsAfterFinished: 60 # auto-delete the Job + its Pod 60s after completion
backoffLimit: 0 # no retries; the test reports failure once and stops
template:
spec:
restartPolicy: Never
containers:
- name: iperf3-client
image: ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<v>
command: ["iperf3", "-c", "<server-svc>", "-J"]
envFrom:
- secretRef:
name: roksbnkctl-job-creds-<random> # projected per invocation
volumeMounts:
- name: files
mountPath: /work
volumes:
- name: files
projected:
sources:
- secret:
name: roksbnkctl-job-files-<random> # one Secret per invocation, holds RunOpts.Files
Three details to call out:
- Projected Secret for cred propagation.
Credentials.IBMCloudAPIKey(when set) becomes a one-shot Secret, mounted viaenvFrom: secretRef. Per PRD 04 §“In-cluster pod” this beats argv (which would show inkubectl describe pod) and beats inlineenv:blocks (which surface inkubectl get pod -o yaml). The Secret carries the samettlSecondsAfterFinished-equivalent lifecycle: when the Job’sttlSecondsAfterFinisheddeletes the Job, the owning controller’s GC sweeps the Secret too viaownerReferences. - Log streaming via
client-go. Once the Job’s pod is inRunningstate,clientset.CoreV1().Pods(ns).GetLogs(name, &corev1.PodLogOptions{Follow: true}).Stream(ctx)returns anio.ReadCloserthat we copy through the redactor intoopts.Stdout. The stream stays open until the pod terminates or ctx cancels. - Exit-code extraction. When the pod transitions to
SucceededorFailed, we readpod.Status.ContainerStatuses[0].State.Terminated.ExitCodeand return that as the backend’s exit code. AFailedpod withExitCode: 0(rare; usually OOMKilled or evicted) maps to backend exit code126— backend mid-run failure rather than tool failure.
The roksbnkctl-test namespace is a fresh namespace dedicated to one-shot test workloads. It’s separate from roksbnkctl-ops (the long-lived pod’s home) so RBAC can be scoped tighter — see Chapter 19 §“RBAC”.
iperf3 server side
Worth calling out because it’s the asymmetric piece. The iperf3 test deploys a server bare Pod + Service into roksbnkctl-test, then runs the client as the one-shot Job described above:
| Side | Resource | Lifetime |
|---|---|---|
| Server | roksbnkctl-iperf3 bare Pod + Service (LoadBalancer for --mode north-south; ClusterIP for --mode east-west) | torn down after the client Job completes |
| Client | one-shot Job | ttlSecondsAfterFinished: 60 |
The bare-Pod (rather than Deployment) shape is intentional — the iperf3 server is single-shot, scoped to one test, torn down on completion; the controller-managed replica machinery a Deployment provides is unused and would only confuse the cleanup story. Service type is driven by --mode: north-south measures laptop-to-cluster bandwidth and needs a publicly reachable endpoint (LoadBalancer); east-west measures node-to-pod and stays in-cluster (ClusterIP). See Chapter 22 — Throughput testing for the user-facing flag surface.
The client Job’s argv is iperf3 -c <server-cluster-ip-or-lb> -J. The -J JSON flows back via log streaming, parsed in internal/test/throughput.go, surfaced as roksbnkctl test throughput JSON output.
The server pod’s securityContext is set to satisfy OpenShift’s restricted-v2 SCC: runAsNonRoot: true, allowPrivilegeEscalation: false, seccompProfile.type: RuntimeDefault, capabilities.drop: [ALL]. iperf3 listens on port 5201 (unprivileged) so no root is needed. The manifest the k8s backend emits sets all four fields to satisfy the SCC.
When to use it
- You’re running
iperf3and want a number that reflects cluster fabric, not your office Wi-Fi. - You’re running
ibmcloudfrom a network that can reach the cluster but not*.cloud.ibm.comdirectly. The ops pod has both lines of sight; your laptop has only one. - You want a cluster-side ad-hoc shell for debugging —
roksbnkctl exec --backend k8s -- bash(when implemented) drops into the ops pod.
When k8s is the wrong call:
- The cluster doesn’t exist yet (
roksbnkctl ops installrequires a working kubeconfig). Uselocalorsshfor pre-cluster ops. - You haven’t run
roksbnkctl ops install. Run it first; it’s a one-time setup per cluster. - You’re running
terraform—--backend k8sfor terraform is deferred to a future release pending a state-handling design (see PRD 03 §“State concerns”).
Chapter 19 is the full reference for the cluster-side mechanics: namespace, ServiceAccount, ClusterRole, Secret, lifecycle.
ssh backend
Runs the wrapped tool on a registered SSH target. Builds on the internal/remote.Client (the same SSH client backing the --on flag); this section assumes you’ve read Chapter 16 for the target-config and host-key TOFU framing.
roksbnkctl ibmcloud --backend ssh:jumphost ks cluster ls
roksbnkctl ibmcloud --backend ssh:bastion --bootstrap iam oauth-tokens
Per-tool apt-bootstrap and the --bootstrap flag
Before exec’ing the wrapped tool, the SSH backend probes whether it’s installed:
ssh <target> 'command -v <tool>'
Exit 0 → tool present, proceed. Non-zero → tool missing. What happens next depends on --bootstrap:
-
Without
--bootstrap(the default). The backend errors with exit127and a clear message:error: tool `iperf3` not found on ssh target jumphost; re-run with --bootstrap to install via apt-get, or pre-install on the target manuallyNo
sudo apt-getever runs. The backend won’t surprise the user with package-manager invocations or sudo password prompts on a remote they didn’t expect mutation on. -
With
--bootstrap. The backend runs the per-tool bootstrap recipe. For Ubuntu (the only OS supported this round), the recipe is roughly:# ibmcloud needs IBM's apt repo + GPG key first curl -fsSL https://download.clis.cloud.ibm.com/Linux/Ubuntu/repo.gpg | sudo apt-key add - echo 'deb https://download.clis.cloud.ibm.com/Linux/Ubuntu jammy main' \ | sudo tee /etc/apt/sources.list.d/ibmcloud.list sudo -n apt-get update -y sudo -n apt-get install -y ibmcloud-cliiperf3is simpler — no repo addition, justsudo -n apt-get install -y iperf3.
The opt-in default reflects PRD 03 open question §“--bootstrap opt-in for SSH”: silent sudo apt-get on a remote host is the kind of surprise that erodes operator trust, especially when the remote is shared between teams. Make the user say “yes, install for me”.
Bootstrap failure modes split between the two backend-failure exit codes per §“Backend-failure semantics”: 127 when we never got going (couldn’t reach the repo, no apt mapping, tool missing without --bootstrap); 126 when we got partway in and then something broke (sudo / OS-detect / install).
| Failure | Exit | What you see |
|---|---|---|
--bootstrap not set and tool missing | 127 | “tool <name> not found on ssh target <target>; re-run with –bootstrap to install via apt-get, or pre-install on the target manually” |
sudo requires a password (NOPASSWD not configured) | 126 | sudo: a password is required → “the SSH user needs passwordless sudo for apt-get install. Configure <user> ALL=(ALL) NOPASSWD: /usr/bin/apt-get in /etc/sudoers, or pre-install <pkg> manually.” |
Non-Ubuntu OS (lsb_release -is doesn’t return Ubuntu) | 126 | “auto-install only supports Ubuntu. Pre-install <pkg> on the target (RHEL: yum install <pkg>).” |
| Network unreachable from target (apt-get can’t reach the repo) | 127 | “target can’t reach the package repo. Check the target’s egress policy or pre-install <pkg> manually.” |
| No apt mapping for the requested tool | 126 | “no bootstrap recipe known for tool <name>; the SSH backend only auto-installs ibmcloud + iperf3 today” |
File materialisation
RunOpts.Files entries are written to a per-invocation tempdir on the remote. The tempdir is /tmp/roksbnkctl.<random>/ where <random> is a fresh 16-byte hex string per Run:
# pseudo-flow
ssh <target> 'mkdir -m 0700 /tmp/roksbnkctl.<rand>'
scp <local-temp>/<basename> <target>:/tmp/roksbnkctl.<rand>/<basename>
ssh <target> '
trap "rm -rf /tmp/roksbnkctl.<rand>" EXIT
cd /tmp/roksbnkctl.<rand>
<argv...>
'
The trap … EXIT is shell-builtin; it fires on normal exit, on set -e failure, on SIGINT (Ctrl-C), on SIGTERM. So even if the user kills their roksbnkctl invocation mid-run, the remote tempdir is cleaned up by the wrapper script’s own trap before the SSH session terminates.
The 0700 mode on the tempdir ensures only the SSH user can read it during the brief on-disk window. On shared bastions (multi-user jumphosts) this matters — and it’s why we materialise to /tmp (which the user owns) rather than /var/tmp or some shared scratch path.
Kubeconfig follows the same pattern: Credentials.KubeconfigBytes becomes <tempdir>/kubeconfig, the wrapper exports KUBECONFIG=<tempdir>/kubeconfig, the trap removes the file on exit. PRD 04 §“Kubeconfig options for SSH backend” calls this “Option A” — scp-and-cleanup. We picked it over the in-memory <() process-substitution alternative because it’s robust across remote shells and sshd configs.
Env propagation: SetEnv vs wrapper script
OpenSSH supports two ways to pass an env var to a remote command:
ssh -o SetEnv=KEY=VALUE target …— client tells the server “please add this to the env”. Works only if the server’ssshd_confighasAcceptEnv KEYmatching. Most stock sshd configs don’t enableAcceptEnvfor arbitrary keys.- Wrapper script with
export KEY=VALUE— the script writes the env var into its own process beforeexec "$@". Works regardless of sshd config, but the value lives briefly in a 0700 file on the remote.
The SSH backend tries SetEnv first. On the first connect to a new target, it sends a sentinel env var (ROKSBNKCTL_SETENV_TEST=ok) and runs echo "$ROKSBNKCTL_SETENV_TEST". If the output is ok, SetEnv works on this target — the result is cached in workspace state, and subsequent runs use SetEnv directly.
If the sentinel doesn’t surface, sshd silently dropped it (it logs refused setenv request on the server side, but clients don’t see that). The backend falls back to a wrapper script:
#!/bin/sh
# /tmp/roksbnkctl.<rand>/wrap.sh, mode 0700, owner-readable only
trap 'rm -f "$0"' EXIT
set +o history
export IBMCLOUD_API_KEY='<value>'
exec "$@"
Then: ssh <target> /tmp/roksbnkctl.<rand>/wrap.sh ibmcloud iam oauth-tokens.
The wrapper-script path is the validator Issue 4 carry-over — the same shape --on uses for env passing today. Risks (file content includes the secret) are mitigated by:
- Mode
0700so only the SSH user can read. set +o historyso the value doesn’t leak into shell history.trap 'rm -f "$0"' EXITdeletes the wrapper as soon as it exits — including on Ctrl-C, since the trap covers SIGINT/SIGTERM by virtue of being in the script’s main process.- The key is never in argv, so
ps -efon the remote doesn’t show it.
roksbnkctl targets show <name> reports which mechanism the target uses (e.g., env propagation: SetEnv (AcceptEnv ok) or env propagation: wrapper script (sshd refused SetEnv)) so users can choose to enable AcceptEnv server-side if they prefer.
Bootstrap failure modes (consolidated)
| Symptom | Cause | Remediation |
|---|---|---|
sudo: a password is required | NOPASSWD sudo not configured | Add <ssh-user> ALL=(ALL) NOPASSWD: /usr/bin/apt-get to /etc/sudoers.d/roksbnkctl on the target |
auto-install only supports Ubuntu | /etc/os-release ID is not ubuntu | Pre-install the tool manually; RHEL: sudo yum install <pkg>; Alpine: sudo apk add <pkg> |
target can't reach the package repo | Target’s egress policy blocks download.clis.cloud.ibm.com (or upstream Ubuntu mirrors) | Pre-install or open egress; doctor’s --backend ssh:<target> flags this |
tool not found on ssh target …; re-run with --bootstrap | --bootstrap not passed and tool missing | Re-run with --bootstrap, or pre-install on the target |
When to use it
- You’re running
ibmcloudfrom a customer-firewalled office where the corporate jumphost can reach IBM Cloud APIs but your laptop can’t. - You’re working in an air-gapped environment where
roksbnkctlruns on your laptop but the IBM Cloud API conversations have to happen from a specific bastion’s IP. - You want a low-overhead remote-exec path that doesn’t require a cluster (the
k8sbackend’s prereq).
When ssh is the wrong call:
- The target lacks the tool and you don’t want to mutate it. Skip
--bootstrap; the backend errors clearly without installing anything. - The target isn’t Ubuntu and you don’t want to pre-install. Bootstrap won’t work; pre-install or use
local/docker/k8s. - You’re running
iperf3to measure cluster bandwidth. SSH puts the client somewhere on the network path to the cluster but not necessarily adjacent to it —k8sis the right answer for that case.
Chapter 16 covers the lighter-weight --on jumphost predecessor that uses the same targets: config block. The SSH backend is the heavier-duty form: file materialisation, env propagation hardening, opt-in bootstrap. Chapter 18 is the decision tree.
The Backend interface
For the curious, the Go interface every backend conforms to:
package exec
type Backend interface {
Run(ctx context.Context, argv []string, opts RunOpts) (int, error)
Name() string
}
type RunOpts struct {
Stdin io.Reader
Stdout, Stderr io.Writer
Env []string // KEY=VALUE pairs
WorkDir string // best-effort; some backends ignore (k8s)
TTY bool // request PTY where supported
Files map[string][]byte // files materialized at exec time
Credentials *Credentials // routed via PRD 04's per-backend mechanism
}
type Credentials struct {
KubeconfigBytes []byte
IBMCloudAPIKey string
}
All four implementations satisfy this interface. Call sites in cli/cluster.go, cli/test.go, etc., get a Backend from the registry and call Run(...) — no branching on backend type. The uniformity is what makes the system extensible without rewriting callers each time a backend lands.
The Credentials struct is the bridge between the resolver chain (env → keychain → config-b64 → prompt) covered in Chapter 14 and the per-backend propagation rules in PRD 04. Each backend translates the struct into the mechanism appropriate to where it runs: env vars for local, --env KEY (no =value) for docker, secretKeyRef for k8s, SetEnv or wrapper script for ssh.
Cross-references
- PRD 03 — pluggable execution backends — the design rationale and full per-backend spec.
- PRD 04 — credential propagation — the cred-passing rules every backend implements.
- Chapter 12 — Workspace config — the
exec:block schema. - Chapter 14 — Credentials and the resolver chain — how creds reach the backend in the first place.
- Chapter 16 — The –on flag and SSH jumphosts — the lightweight remote-exec predecessor to the SSH backend.
- Chapter 18 — Choosing a backend per tool — the decision tree.
- Chapter 19 — The in-cluster ops pod — deploy-time mechanics for the
k8sbackend.
Choosing a backend per tool
Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the
Workspacestruct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.
Chapter 17 covered the mechanics of each backend. This chapter is the decision tree: given a tool and a scenario, which of local / docker / k8s / ssh is the right call.
If you’re searching for “which backend should I use”, you’ve landed on the right page.
The four backends in one line each
| Backend | One-line summary | Deep dive |
|---|---|---|
local | os/exec on your laptop; fastest, requires the tool on PATH | § Local backend |
docker | docker run against a vendored image; frozen tool version, no host install | § Docker backend |
k8s | inside the cluster (long-lived ops pod or one-shot Job); cluster-correct network identity | § K8s backend |
ssh:<target> | on a registered SSH target; opt-in apt-bootstrap on Ubuntu | § SSH backend |
If you’re skimming, the cheat-sheet is:
localwhen you have the tool installed and the host’s network identity is correct for the call.dockerwhen you don’t have the tool and don’t want to install it, or you need a frozen version for CI.k8swhen the call’s network position matters and the cluster is the right vantage point.ssh:<target>when the call needs to originate from a specific external host (a customer bastion, an air-gapped bridge).
Inside the all-in-one runner image, use
local. When you run roksbnkctl from the runner container, every tool it dispatches is already onPATHin the image — solocalis the right call and thedocker/k8sbackends (which fetch a tool from elsewhere) are redundant there (anddockerwould mean docker-in-docker). The decision tree below is for the binary-on-host case, where those backends earn their keep.
The rest of this chapter is the longer version.
Per-tool default backends
Every tool has a default backend baked into roksbnkctl. Workspace config (exec: block) can override the default per workspace; --backend overrides for a single invocation.
| Tool | Default | Resolved by |
|---|---|---|
ibmcloud | local | internal/cli/cluster.go::resolveBackendSpecWith("ibmcloud", flagOverride) |
iperf3 | k8s | internal/cli/test.go::resolveBackendSpecWith("iperf3", flagOverride) |
terraform | local | internal/cli/lifecycle.go::resolveBackendSpecWith("terraform", flagOverride) |
The defaults reflect “what’s the right answer for the most common scenario”:
ibmclouddefaults tolocalbecause most users have it on PATH or are happy installing it. The compliance + firewall scenarios wheresshordockerare better are the minority of calls.iperf3defaults tok8sbecause throughput from a laptop’s uplink isn’t the cluster’s bandwidth. The k8s backend places the iperf3 client in (or adjacent to) the cluster so the number reflects fabric, not Wi-Fi. Laptop-uplink-to-cluster is a real measurement too, but it’s the special case — opt in via--backend local.terraformdefaults tolocalbecause the terraform-exec local path is the established workflow. State handling is simplest there. Frozen-version CI runs use--backend docker; non-local network-locality use cases (cluster-side, SSH-bastion-side) are deferred to a future release pending a state-handling design — see PRD 03 §“State concerns”.
To change a default per workspace, use the backend command — no config.yaml hand-edit:
roksbnkctl -w acme backend set ibmcloud ssh:bastion
roksbnkctl -w acme backend set terraform docker
roksbnkctl -w acme backend show # effective backend per tool + where it came from
roksbnkctl -w acme backend unset terraform # revert to the built-in default
That writes the exec: block for you:
exec:
ibmcloud: { backend: ssh:bastion }
terraform: { backend: docker }
Chapter 12 §“exec:” covers the schema. The --backend CLI flag overrides whatever is in exec: for a single invocation.
Per-tool supported-backend matrix
Not every tool supports every backend. The authoritative matrix at v1.0:
| Tool | local | docker | k8s | ssh:<target> |
|---|---|---|---|---|
ibmcloud | yes (default) | yes (frozen image) | yes (long-lived ops pod) | yes |
iperf3 | yes (opt-in: laptop vantage) | not supported (same network identity as local) | yes (default) | yes |
terraform | yes (default) | yes (frozen image) | deferred to v1.x (state-file handling) | deferred to v1.x (state-file handling) |
| DNS probe | yes (default for laptop vantage) | not supported (same network identity as local) | yes (cluster vantage) | yes (remote vantage) |
kubectl / oc | internalised — runs via the Go client, not via a host binary | n/a | n/a | n/a |
dig | internalised — DNS probe replaces dig for in-tree work | n/a | n/a | n/a |
Legend:
- yes — supported; same surface command works on this backend.
- yes (default) — this backend is the per-tool default; pass
--backend otherto override. - not supported — rejected at CLI parse time with a clear error pointing at the right alternative.
- deferred to v1.x — a real design constraint, not a gap; see the cell text and PRD 03 §“State concerns”.
- internalised —
roksbnkctlperforms the operation via its own embedded library, not by shelling out; no backend selection applies.
The “no” entries are intentional design decisions, not gaps:
iperf3overdockeris rejected because a Docker container running locally has the same network identity as the host — same NAT egress, same uplink, same observed bandwidth as--backend local. The user’s mental model would be “I picked docker, so the iperf3 must be hermetic now” but the throughput number wouldn’t actually differ. Better to refuse and force the user to picklocal(deliberate laptop measurement) ork8s(cluster measurement).- DNS probe over
dockeris rejected for the same reason. DNS resolution from a Docker container with default bridge networking goes through the same resolver as the host. There’s no GSLB-relevant network-locality difference. The probe subcommand errors with “DNS probe doesn’t benefit from docker; use local instead” when--backend dockeris passed. terraformoverk8sandsshis deferred to v1.x. The state file is sensitive (admin tokens, generated TLS keys, license bundles); moving it into a Kubernetes Secret or scp’ing it pre/post-run requires a state-handling design that hasn’t shipped yet. PRD 03 §“State concerns” lays out the considerations; the roadmap entry lives indocs/PLAN.md§“What’s deliberately deferred to post-v1.0”.
Passing an unsupported (tool, backend) pair errors at the CLI layer before the backend is invoked:
$ roksbnkctl test throughput --backend docker
error: iperf3 doesn't support backend `docker` (same network identity as `local`,
no value-add); supported: local, k8s, ssh:<target>
Decision tree
Pick the question that matches your scenario.
“I want to measure cluster bandwidth”
Use --backend k8s. The default for iperf3 is already k8s — the explicit flag is redundant unless you’ve overridden the default in workspace config:
roksbnkctl test throughput
# equivalent to:
roksbnkctl test throughput --backend k8s
The k8s backend deploys a server-side Deployment + LoadBalancer Service in roksbnkctl-test, runs the iperf3 client as a one-shot Job in the same namespace, collects the JSON output from the client pod’s logs, and tears down both. The bandwidth number reflects the cluster fabric.
If you instead want to measure your laptop’s uplink to the cluster:
roksbnkctl test throughput --backend local --endpoint <cluster-LB-ip>:5201
That’s a deliberately different measurement — useful when you suspect office Wi-Fi, not cluster fabric, is the bottleneck.
“I’m doing GSLB DNS validation”
Use both local and k8s. F5 BIG-IP Next’s GSLB returns different answers depending on the requesting resolver’s IP — geographic affinity, datacenter routing, health-check state. To validate that the GSLB is actually doing this, query from multiple network vantage points and compare.
The multi-vantage probe ships at v1.0 via roksbnkctl test dns --gslb-compare:
roksbnkctl test dns \
--target www.example.com \
--type A \
--server gslb-vip.f5.example.com \
--gslb-compare
--gslb-compare fans out to every configured vantage (local for your office IP, k8s for the cluster’s egress IP, ssh:<region-bastion> for a bastion in another region) in parallel and emits a single comparison JSON with a gslb_divergence boolean. Different answers across vantages are expected in a healthy GSLB; identical answers might mean the GSLB rules aren’t taking effect for the resolver positions you queried from.
Chapter 21 — DNS testing for GSLB is the full reference.
“I need to run ibmcloud from a customer-firewalled office”
Use --backend ssh:<bastion>. Your customer’s network policy lets the corporate jumphost reach *.cloud.ibm.com but blocks your laptop. The SSH backend ships your kubeconfig to the bastion (single file, mode 0600, removed via trap on session exit), runs ibmcloud there, streams the output back:
roksbnkctl ibmcloud --backend ssh:bastion ks cluster ls
If ibmcloud isn’t installed on the bastion, you’ll get a clear error:
error: tool `ibmcloud` not found on ssh target bastion; re-run with --bootstrap to install
via apt-get, or pre-install on the target manually
Re-run with --bootstrap if you want roksbnkctl to sudo apt-get install -y ibmcloud-cli on the bastion. The opt-in default reflects “we don’t surprise users with sudo apt-get on a remote they didn’t expect mutation on” — see Chapter 17 §“SSH backend” for the bootstrap mechanics.
“I’m in CI and want a frozen toolchain version”
Use --backend docker. The vendored images are tagged in lock-step with roksbnkctl releases — ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:v1.0.0 is the exact same ibmcloud binary every CI run sees, regardless of when the runner image was built or what apt-get happens to ship that day:
roksbnkctl ibmcloud --backend docker iam oauth-tokens
roksbnkctl up --backend docker # terraform inside hashicorp/terraform:<v>
For CI specifically, also pin ibmcloud.api_key_source: env in workspace config so the API key resolution is unambiguous (no keychain fallback to confuse a non-interactive runner) — see Chapter 14 §“Pinning a single source”.
“I’m on a clean dev machine without ibmcloud installed”
Use --backend docker. No apt-get install ibmcloud-cli, no IBM repo + GPG key dance, no upstream-package-version mismatch — docker pull ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:dev is the only setup, and roksbnkctl does that for you on first invocation.
Alternatively, if your laptop is the dev machine and you’ll run ibmcloud more than once, just install it. The local backend has lower per-invocation startup latency than docker (no container create/start/log-attach), so once you’ve paid the install cost the local path is faster for the rest of the session.
“I want a cluster-side ad-hoc shell”
Use --backend k8s with the long-lived ops pod. Once roksbnkctl ops install has run, --backend k8s for ibmcloud (or any future tool) routes through kubectl exec -n roksbnkctl-ops ops -- <argv>. The pod stays alive between invocations, so the second and subsequent commands skip pod-startup latency.
roksbnkctl ops install
roksbnkctl ibmcloud --backend k8s iam oauth-tokens
roksbnkctl ibmcloud --backend k8s ks cluster ls
roksbnkctl ibmcloud --backend k8s account list
Chapter 19 is the full reference for the ops pod lifecycle.
“I’m pre-cluster — there’s no cluster yet”
Use local or ssh:<target>. The k8s backend prereq is a working kubeconfig pointing at a running cluster; before roksbnkctl up has succeeded, that doesn’t exist. For pre-cluster ibmcloud + terraform calls (account inspection, IAM tinkering, the cluster-create itself), local and ssh:bastion are the only two paths.
When not to use a backend
Common foot-guns, in rough order of how often they come up:
--backend k8s without roksbnkctl ops install
The ops pod must exist before the k8s backend can route ibmcloud calls through it. First-time use:
roksbnkctl ops install # one-time setup per cluster
roksbnkctl ibmcloud --backend k8s ks cluster ls
If you skip the install, the backend errors with a clear remediation:
error: ops pod not found in roksbnkctl-ops namespace; run `roksbnkctl ops install` first
Chapter 19 covers the install/show/uninstall lifecycle.
--backend docker for a network-locality test
iperf3 and the DNS probe both reject --backend docker because a local Docker container has the same network identity as the host (default bridge networking). The probe wouldn’t measure anything different. The CLI errors at parse time:
$ roksbnkctl test throughput --backend docker
error: iperf3 doesn't support backend `docker` (same network identity as `local`,
no value-add); supported: local, k8s, ssh:<target>
If you actually want a hermetic-tools throughput test, --backend k8s is the right answer.
--backend ssh:host without --bootstrap on a fresh target
If ibmcloud (or iperf3) isn’t installed on the target, the SSH backend won’t silently sudo apt-get for you — --bootstrap is opt-in. The first call on a fresh target tells you exactly what’s needed:
error: tool `ibmcloud` not found on ssh target bastion; re-run with --bootstrap to install
via apt-get, or pre-install on the target manually
Re-run with --bootstrap if mutation is OK; otherwise pre-install via your config-management of choice (Ansible, Salt, baked-in-image).
--backend ssh:host to a non-Ubuntu target with --bootstrap
The apt-bootstrap recipe is Ubuntu-only this round. RHEL / CentOS / Alpine targets need pre-installation via yum / dnf / apk — --bootstrap errors out cleanly:
error: auto-install only supports Ubuntu. Pre-install `ibmcloud-cli` on the target
(RHEL: `yum install ibmcloud-cli`)
Once the tool is installed, --backend ssh:host works without --bootstrap.
--backend k8s for terraform
Deferred to v1.x. The terraform tool’s k8s + ssh backends require a state-handling design that hasn’t shipped — moving the state file into a Kubernetes Secret or scp’ing it pre/post-run is fiddly enough to be a feature in its own right (PRD 03 §“State concerns”; roadmap in docs/PLAN.md §“What’s deliberately deferred to post-v1.0”). For now, terraform supports local and docker only. If the network-locality use case (running terraform from a customer VPC for IP-egress reasons) is blocking, file an issue.
Mixing --on and --backend ssh:<target>
--on <target> is the Chapter 16 lightweight remote-exec — it runs the passthrough shape (exec, shell, kubectl, oc, ibmcloud) on the target by literally re-running the command via SSH. --backend ssh:<target> is the heavier-duty form — it routes through the Backend interface, which means file materialisation, env propagation hardening, opt-in apt-bootstrap, and the redactor are all wired in.
You generally want one or the other, not both. The supported precedence is “--backend ssh:<target> wins”; passing both flags on the same invocation surfaces a warning. If you’re calling roksbnkctl ibmcloud …, prefer --backend ssh:<target> for the same target — you get the better cred-handling story automatically.
Workspace config + --backend flag interaction
Recap of Chapter 12 §“exec:”:
The flag wins. If ~/.roksbnkctl/<ws>/config.yaml says:
exec:
iperf3: { backend: k8s }
…and you run roksbnkctl test throughput --backend local, the local backend runs. The flag is the per-invocation override; the workspace config is the per-workspace default.
If neither is set, the per-tool default from the previous section applies (iperf3 → k8s, ibmcloud → local, terraform → local). The resolution order is exact:
--backendflagexec.<tool>.backendin workspace config- Per-tool baked-in default
There’s no fallback chain inside this resolution — if you pass --backend k8s and the cluster is unreachable, the backend errors with “cluster API unreachable” (exit 127). It does not fall through to local. Silent fallback hides intent and produces confusing CI results; the failure-mode discipline in Chapter 17 §“Backend-failure semantics” applies here too.
Summary table
The decision-tree contents collapsed into one table:
| If you want to… | Backend | Notes |
|---|---|---|
| Measure cluster bandwidth | k8s | iperf3 client + server in cluster (the default) |
| Measure laptop-uplink-to-cluster bandwidth | local | deliberate; not the iperf3 default |
| GSLB DNS cross-vantage compare | local + k8s (--gslb-compare) | multiple vantages in parallel |
ibmcloud from a customer-firewalled office | ssh:bastion | with --bootstrap if first call on fresh Ubuntu |
| Frozen-version CI for any tool | docker | image tag matches roksbnkctl release |
Cluster-side ad-hoc ibmcloud debugging | k8s | requires roksbnkctl ops install first |
| Pre-cluster ibmcloud / terraform | local or ssh | k8s requires a working cluster |
terraform up on a clean dev machine | local (default) or docker | k8s + ssh deferred |
| Air-gapped: laptop can’t reach IBM Cloud, bastion can | ssh:bastion | with kubeconfig propagation |
| Just learning the tool | local | simplest mental model |
Worked example: bare-metal + jumphost office workflow
End-to-end Part V scenario: you’re an F5 SE running a customer POC from a corporate-firewalled office. The laptop can’t reach *.cloud.ibm.com directly (the office proxy blocks it) but a customer-provisioned Ubuntu jumphost at 10.20.30.40 can. The jumphost was already auto-discovered by an earlier roksbnkctl up against this customer’s account, so targets list shows it. You need to: install the in-cluster ops pod, run ibmcloud from the bastion, and run a throughput test from inside the cluster — all without installing tools locally.
# 1. Verify jumphost is registered + reachable
$ roksbnkctl targets list -w customer
NAME HOST KEY_SOURCE STATUS
jumphost 10.20.30.40 workspace/state reachable
# 2. Run ibmcloud from the jumphost (Sprint 1 --on flag, lightweight)
$ roksbnkctl ibmcloud --on jumphost ks cluster ls
OK
Name ID State Created ...
customer-cluster c4abc123def456 normal 3 days ago ...
# 3. For the same call routed through the Backend interface (cred-handling
# hardened, redactor wired, opt-in apt-bootstrap available), use --backend
$ roksbnkctl ibmcloud --backend ssh:jumphost ks cluster ls
# Same output; different code path. Prefer --backend ssh:<target> for
# everything except quick interactive shells where --on is faster to type.
# 4. Install the in-cluster ops pod (one-time per cluster)
$ roksbnkctl ops install
✓ Namespace roksbnkctl-ops created
✓ ServiceAccount + Role + RoleBinding applied
✓ Secret roksbnkctl-ibm-creds applied (envFrom secretRef)
✓ Pod roksbnkctl-ops Running (2.3s)
# 5. Same ibmcloud call routed through the ops pod (k8s backend)
$ roksbnkctl ibmcloud --backend k8s iam oauth-tokens
IAM token: Bearer eyJ...
# (The token comes from inside the cluster — different egress IP from the
# jumphost's, useful when IAM policy is IP-conditional.)
# 6. Throughput test using the cluster vantage (default for iperf3)
$ roksbnkctl test throughput
→ Deploying iperf3 server pod into namespace "roksbnkctl-test"
✓ Pod ready (iperf3-server-...)
→ Deploying iperf3 client Job in the same namespace
✓ Client Job complete
✓ throughput: 8.92 Gbits/sec (mean over 10s)
→ Tearing down iperf3 fixture
✓ pod, service, and Job deleted
# 7. Throughput test from the jumphost into the cluster (north-south, real
# customer-network bandwidth — not laptop wifi)
$ roksbnkctl test throughput --backend ssh:jumphost --mode north-south
✓ throughput: 936 Mbits/sec (jumphost → cluster LB; customer's WAN)
# 8. Persist the per-tool routing in workspace config (one-time)
$ cat >> ~/.roksbnkctl/customer/config.yaml <<'YAML'
exec:
ibmcloud: { backend: ssh:jumphost }
iperf3: { backend: k8s }
terraform: { backend: local }
YAML
# Subsequent runs skip the --backend flag — every ibmcloud call routes via
# the jumphost automatically; every iperf3 runs in-cluster.
The point of this walkthrough: with no tools installed locally beyond roksbnkctl itself, you’ve reached the IBM Cloud control plane from the customer’s bastion (compliance-correct egress), exercised the cluster fabric for throughput, and persisted the routing per workspace. The same laptop with the same roksbnkctl binary handles a different customer’s POC by pointing at a different workspace; nothing on the laptop is workspace-specific.
Chapter 19 covers the ops-pod lifecycle in detail; Chapter 22 covers the north-south vs east-west modes.
Cross-references
- Chapter 12 — Workspace config — the
exec:block schema. - Chapter 14 — Credentials and the resolver chain — how creds reach each backend.
- Chapter 16 — The
--onflag and SSH jumphosts — the lightweight remote-exec predecessor to the SSH backend. - Chapter 17 — Execution backends — per-backend mechanics.
- Chapter 19 — The in-cluster ops pod — the cluster-side prerequisite for
--backend k8s. - Chapter 22 — Throughput testing — iperf3-specific flags.
- PRD 03 — pluggable execution backends — the design spec.
The in-cluster ops pod
The k8s execution backend has two execution patterns: a long-lived ops pod for ad-hoc commands, and one-shot Jobs for throughput tests, DNS probes, and other per-invocation workloads. Chapter 17 §“K8s backend” covered the interface mechanics — how Backend.Run dispatches into either pattern.
This chapter is the reference for the pod itself: what roksbnkctl ops install deploys, what RBAC it grants, where credentials live, how to rotate them, and how to debug when something goes wrong.
If you’ve never run roksbnkctl ops install, you can read this chapter front-to-back; otherwise the § Operability section near the end is the troubleshooting jump-off point.
What the ops pod is
A long-lived pod in the roksbnkctl-ops namespace, running an image bundled with the tools roksbnkctl may want to invoke cluster-side: ibmcloud CLI plus kubectl as a fallback, with oc and terraform reserved for future iterations.
The pod sits idle waiting for kubectl exec calls. Each roksbnkctl ibmcloud --backend k8s … invocation routes through client-go’s SPDY executor, runs the wrapped tool inside the existing pod, streams stdout/stderr back, and returns the exit code. No pod create/start latency between invocations — a session of twenty ibmcloud commands pays the startup cost once.
Compared to the one-shot Job pattern (used for iperf3 and the DNS probe), the ops pod trades a bit of resource-usage idle-state for substantially lower per-call latency. It’s the right shape when you want to debug interactively or run many small commands.
roksbnkctl ops install
Idempotent setup. Run once per cluster; re-run any time you want to refresh the image, rotate the API key Secret, or recover from a partial uninstall.
roksbnkctl ops install
What it does, step by step:
1. Create the namespace
apiVersion: v1
kind: Namespace
metadata:
name: roksbnkctl-ops
labels:
app.kubernetes.io/name: roksbnkctl
app.kubernetes.io/component: ops-pod
The roksbnkctl-ops namespace is dedicated to the long-lived pod. Separate from roksbnkctl-test (where one-shot Jobs run) so RBAC can be scoped per namespace — see § RBAC below.
2. Create the ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
name: roksbnkctl-ops
namespace: roksbnkctl-ops
The pod runs as this SA. Its projected token is auto-mounted at /var/run/secrets/kubernetes.io/serviceaccount/, which is what the bundled kubectl uses for in-cluster authentication. The IBM Cloud API key (a separate credential) reaches the pod through a Kubernetes Secret — see § Credential propagation below.
3. Create the ClusterRole + ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: roksbnkctl-ops
rules:
- apiGroups: [""]
resources: ["pods", "pods/exec", "pods/log"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]
resourceNames: ["roksbnkctl-ibm-creds"]
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "create", "delete"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: roksbnkctl-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: roksbnkctl-ops
subjects:
- kind: ServiceAccount
name: roksbnkctl-ops
namespace: roksbnkctl-ops
The full manifest lives at internal/exec/k8s_install.yaml (embedded into the binary). § RBAC walks through what each rule is for.
4. Create or update the credential Secret
Note. This step describes the static-key Secret applied under
--trusted-profile=offand under theauto-fallback when IAM perms don’t allow the trusted-profile path. Under--trusted-profile=autosuccess the Secret is still applied with empty data fields (placeholder soenvFrom: secretRefalways resolves); the cred propagation happens via the trusted-profile annotation on the SA + theIAM_PROFILE_IDenv var instead. See §“Trusted-profile flow” below.
apiVersion: v1
kind: Secret
metadata:
name: roksbnkctl-ibm-creds
namespace: roksbnkctl-ops
annotations:
helm.sh/resource-policy: keep # don't sweep on accidental destroy
type: Opaque
stringData:
IBMCLOUD_API_KEY: <resolved-key-value>
The key value comes from the workspace’s resolver chain (env → keychain → config-b64 → prompt) — see Chapter 14 for the resolution order. The Secret carries two keys (IBMCLOUD_API_KEY and the legacy alias IC_API_KEY) both populated from the same resolved value, so older ibmcloud CLI versions that look for the IC_ name find it.
If the Secret already exists (re-running ops install after a key rotation), roksbnkctl does a client-side Get + Update: the Secret’s data is overwritten with the freshly resolved value, the roksbnkctl.io/rotated-at annotation is stamped with the current timestamp, and the rest of the Secret’s metadata is left untouched. roksbnkctl ops show surfaces last cred rotation: <timestamp> by reading that annotation.
5. Create the Pod
apiVersion: v1
kind: Pod
metadata:
name: roksbnkctl-ops
namespace: roksbnkctl-ops
labels:
app: roksbnkctl-ops
roksbnkctl.io/managed: "true"
spec:
serviceAccountName: roksbnkctl-ops
restartPolicy: Always
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: tools
image: ${OPS_IMAGE} # resolved from roksbnkctl's version at install time
imagePullPolicy: IfNotPresent
command: ["sleep", "infinity"]
envFrom:
- secretRef:
name: roksbnkctl-ibm-creds
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
capabilities:
drop: ["ALL"]
resources:
requests: { cpu: 50m, memory: 64Mi }
limits: { cpu: 500m, memory: 256Mi }
Three details to call out:
command: ["sleep", "infinity"]— the pod’s own command. EachBackend.Runinvocation issues akubectl execagainst this idle process, which means the pod’s main process never exits as long as the pod is healthy.securityContextis set explicitly for OpenShift’srestricted-v2SCC. Pod-levelrunAsNonRoot+seccompProfile.type: RuntimeDefault; container-levelallowPrivilegeEscalation: false+capabilities.drop: [ALL]+runAsNonRoot— the same fields the iperf3 server pod sets, for the same reason.envFrom: secretRef— the API key reaches the pod’s env without ever touching the pod manifest’s argv orenv:block.kubectl describe pod roksbnkctl-opsshows the secret reference name but not the value, per PRD 04 §“In-cluster pod”.
6. Wait for readiness
roksbnkctl ops install waits for Pod.Status.Phase == Running and the container’s Ready condition before returning. Default timeout is 60 seconds; longer for clusters with slow image pulls (the ghcr.io image is ~80 MiB). Failures surface a kubectl describe pod roksbnkctl-ops excerpt for context.
Trusted-profile flow
The static-key Secret described above is the fallback. The default ops install invocation auto-provisions an IBM Cloud IAM trusted profile linked to the ops pod’s ServiceAccount, so the static API key no longer needs to land in any Kubernetes Secret. PRD 04 → “Trusted-profile auto-provisioning (k8s backend)” is the design reference; this section is the operational walkthrough.
ops install --trusted-profile=auto provisions the profile, and the in-pod ibmcloud login wrap detects the SA’s trusted-profile annotation and authenticates via the projected SA token at runtime.
roksbnkctl ops install --trusted-profile=auto
--trusted-profile=auto is the default — running roksbnkctl ops install with no flag picks the auto path. Naming the flag explicitly is useful in scripts that pin behaviour or in docs that want to read unambiguously:
$ roksbnkctl ops install --trusted-profile=auto
✓ Provisioned IAM trusted profile roksbnkctl-ops-canada-roks (iam-Profile-9f2…)
✓ created namespace roksbnkctl-ops
✓ created sa roksbnkctl-ops/roksbnkctl-ops
✓ created secret roksbnkctl-ops/roksbnkctl-ibm-creds
✓ created clusterrole roksbnkctl-ops
✓ created crb roksbnkctl-ops
✓ created pod roksbnkctl-ops/roksbnkctl-ops
→ Waiting for ops pod to be Ready (60s timeout)
✓ Ops pod is Ready (trusted profile roksbnkctl-ops-canada-roks)
Re-runs against an existing install emit updated <kind> … / <kind> … exists instead of created for each resource that already matches the desired state. The trusted-profile provisioning line above is the single line internal/cli/ops.go emits for the whole IBM IAM-side flow (perm probe + profile create + compute-resource link + SA annotation) — the work happens silently inside resolveTrustedProfileForInstall; the one line you see is the receipt.
What just happened, in order (the binary doesn’t narrate these steps but they’re what’s actually going on):
- IAM perm probe.
ops installcalls IBM IAM Identity to confirm the resolved API key hasiam-identityperms. On403, the flag value drives the next step:autofalls back to the static-key Secret with a warning (see §“--trusted-profile=autofalling back” below);onerrors out with a non-zero exit. - Profile creation. Names the profile
roksbnkctl-ops-<workspace>so multiple workspaces against the same IBM Cloud account don’t race for a single shared name. The compute-resource link binds the profile to your cluster’s OIDC issuer URL + theroksbnkctl-ops/roksbnkctl-opsServiceAccount specifically — other SAs on the same cluster can’t assume the profile. - Policy attachment. No default policies are attached — the profile inherits whatever IAM policies your account has set up for trusted profiles in general (typically nothing, until you grant). A future cycle will surface
ibmcloud.trusted_profile.policiesas a workspace-config block; tracked under v1.x deferred. If you need the profile to actually authorise specific actions (Container Registry pulls, Cloud Object Storage reads), grant the policies via IBM Cloud Console oribmcloud iam trusted-profile-policy-createafterops installreturns. - SA annotation. The ServiceAccount gets
iam.cloud.ibm.com/trusted-profile: roksbnkctl-ops-<workspace>plus theroksbnkctl.io/trusted-profile-managed: "true"marker that signalsops uninstallto delete the profile during cleanup. - Pod creation. The pod’s container always has
envFrom: secretRef: roksbnkctl-ibm-creds; what changes between modes is the Secret’s contents. Under--trusted-profile=autosuccess the Secret is created with empty data —IBMCLOUD_API_KEYis the empty string — plus an extraIAM_PROFILE_IDenv var pointing at the provisioned profile’s ID, and a projected ServiceAccount-token volume mounted at/var/run/secrets/tokens/token(audienceiam) so the pod has a cluster-issued JWT the IBM IAM endpoint will accept. The in-podibmcloud loginwrap detects the SA’s trusted-profile annotation and runsibmcloud login -a https://cloud.ibm.com --cr-token @/var/run/secrets/tokens/token --profile "$IAM_PROFILE_ID" -r "${IBMCLOUD_REGION:-us-south}" --quiet— the--cr-token @<path>form reads the projected SA token from disk; IBM IAM validates that JWT against the trusted profile’sROKS_SAcompute-resource link (the linkinternal/ibm/trusted_profile.go::ensureLinkprovisions). The static API key never transits the pod env. Under--trusted-profile=off(or theauto-fallback) the Secret carries the resolved API key, no projected token volume is mounted, and the wrap runsibmcloud login --apikey "$IBMCLOUD_API_KEY".
Verifying the profile is in use
The ServiceAccount carries the truth-of-record annotation:
$ oc get serviceaccount roksbnkctl-ops -n roksbnkctl-ops -o yaml
# or, kubectl-equivalent via the bundled passthrough:
$ roksbnkctl k get sa roksbnkctl-ops -n roksbnkctl-ops -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
iam.cloud.ibm.com/trusted-profile: Profile-ccba11f2-3b1f-4b1a-b8a4-aeed2b7b3320 # ← the IBM IAM Profile ID
roksbnkctl.io/trusted-profile-managed: "true" # ← ops uninstall will delete it
roksbnkctl.io/provisioned-at: "2026-05-13T14:08:33Z"
name: roksbnkctl-ops
namespace: roksbnkctl-ops
End-to-end smoke test of the runtime cred flow:
$ roksbnkctl --backend k8s ibmcloud iam oauth-tokens
IAM token: Bearer eyJ…
The token is fresh-each-call: the in-pod wrap detects the SA’s iam.cloud.ibm.com/trusted-profile annotation, trades the pod’s projected SA token for an IAM token against the trusted profile, and returns it to the caller. No static API key transits the pod env.
The first invocation may take 30–60 seconds after ops install returns because IBM IAM needs to pick up the cluster’s OIDC issuer URL before it will accept the projected SA token as proof for the profile. The wrap absorbs this with a 3-attempt × 20s-backoff retry — up to ~40s of waiting inside the wrap before it surfaces the failure. On triple-fail the wrap prints trusted-profile login failed after 3 attempts: <captured-stderr> to your terminal (the captured stderr will include the underlying ibmcloud login diagnostic — typically the “Unable to authenticate” / FAILED banner shape). If your first smoke test produces that line, give IAM a few more seconds and re-run. After the first successful call the wrap’s auth state is cached for the pod’s lifetime, so subsequent roksbnkctl --backend k8s ibmcloud <subcommand> invocations don’t re-pay the propagation window.
--trusted-profile=auto falling back
auto falls back to the static-key Secret when any of three pre-conditions for trusted-profile provisioning aren’t met. The warning prints first (the fallback decision is made before any cluster-side resource is applied), then the rest of the install proceeds with the static-key shape:
$ roksbnkctl ops install
warning: IAM perm 'iam-identity' missing; using static-key Secret. Pass `--trusted-profile=off` to silence.
✓ created namespace roksbnkctl-ops
✓ created sa roksbnkctl-ops/roksbnkctl-ops
✓ created secret roksbnkctl-ops/roksbnkctl-ibm-creds
✓ created clusterrole roksbnkctl-ops
✓ created crb roksbnkctl-ops
✓ created pod roksbnkctl-ops/roksbnkctl-ops
→ Waiting for ops pod to be Ready (60s timeout)
✓ Ops pod is Ready (static-key Secret)
The three warning shapes (in source order — internal/cli/ops.go resolveTrustedProfileForInstall):
| Trigger | Warning |
|---|---|
Workspace has no registered cluster yet (cluster-outputs.json missing — run cluster up or cluster register first) | warning: trusted-profile mode 'auto' needs a registered cluster (<err>); falling back to static-key Secret. Pass –trusted-profile=off to silence. |
| Registered cluster lookup against the IBM Cloud API failed (network, key auth, cluster deleted out-of-band) | warning: trusted-profile mode 'auto' couldn't look up cluster (<err>); falling back to static-key Secret. Pass –trusted-profile=off to silence. |
API key lacks IAM iam-identity permission (the most common fallback) | warning: IAM perm 'iam-identity' missing; using static-key Secret. Pass –trusted-profile=off to silence. |
All three are non-fatal; the install completes and the pod works normally. The warnings are terse on purpose — the actionable detail belongs in this chapter, not in every stderr line. Three ways to clear them permanently:
- Run
cluster uporcluster registerfirst if the warning names the missing cluster registration.ops installre-run after the registration completes will detect the cluster and switch to the trusted-profile path. - Ask your IAM admin to grant
iam-identityOperator role on the API key (or use a different key that already has it) if the warning names the missing IAM perm. Re-runops install— the install detects the changed perm posture on re-run and replaces the static-key Secret with a trusted-profile binding. - Opt out via
--trusted-profile=off(next subsection) if you don’t want the warning every install.
--trusted-profile=off
Explicit opt-out. Skips the IAM perm check entirely and provisions the static-key Secret:
$ roksbnkctl ops install --trusted-profile=off
✓ created namespace roksbnkctl-ops
✓ created sa roksbnkctl-ops/roksbnkctl-ops
✓ created secret roksbnkctl-ops/roksbnkctl-ibm-creds
✓ created clusterrole roksbnkctl-ops
✓ created crb roksbnkctl-ops
✓ created pod roksbnkctl-ops/roksbnkctl-ops
→ Waiting for ops pod to be Ready (60s timeout)
✓ Ops pod is Ready (static-key Secret)
Use cases:
- Byte-for-byte parity tests — for scripts whose assertions match a static-key
ops showoutput verbatim, or parity tests against an existing deployment. - Air-gapped clusters that can’t reach the IBM IAM API at runtime — without that connectivity, the pod can’t trade its projected SA token for an IAM token, so the trusted-profile path is non-functional regardless of perms.
- Cred rotation runbooks that already automate the static-key path and aren’t yet ready to switch to the projected-token model.
The third value, --trusted-profile=on, is the inverse — it forces the trusted-profile path and refuses to fall back on perm-missing, returning a non-zero exit with the same warning text. Use it in CI to surface IAM-perm regressions explicitly.
Cleanup on ops uninstall
roksbnkctl ops uninstall --confirm honors the roksbnkctl.io/trusted-profile-managed: "true" annotation on the SA and deletes the IBM Cloud trusted profile alongside the cluster-side objects:
$ roksbnkctl ops uninstall --confirm
✓ deleted trusted profile roksbnkctl-ops-canada-roks
✓ deleted pod roksbnkctl-ops
✓ deleted secret roksbnkctl-ibm-creds
✓ deleted serviceaccount roksbnkctl-ops
✓ deleted clusterrolebinding roksbnkctl-ops
✓ deleted clusterrole roksbnkctl-ops
✓ deleted namespace roksbnkctl-ops
✓ deleted namespace roksbnkctl-test
The trusted profile is deleted first, before the cluster-side objects, so even if the cluster API becomes unreachable mid-uninstall the IBM Cloud-side state isn’t left orphaned. The Secret is always deleted regardless of mode (it’s always rendered by ops install, just with empty data under --trusted-profile=auto success).
The trusted-profile delete is best-effort — if the calling user’s API key has lost iam-identity perms in the meantime (or the key itself has rotated and the new key doesn’t have those perms), the cluster-side objects still delete and a warning line is printed instructing the user to delete the profile manually via the IBM Cloud console. The annotation remains correct documentation of what was provisioned; roksbnkctl ops install on a fresh cluster will pick a fresh profile name unconditionally so an orphaned profile from a prior install doesn’t collide.
--trusted-profile=off installs leave no trusted profile to clean up — ops uninstall --confirm just deletes the cluster-side objects + the static-key Secret.
roksbnkctl ops show
Reports current state without making any changes:
$ roksbnkctl ops show
namespace: roksbnkctl-ops
pod: roksbnkctl-ops
phase: Running
ready: true
image: ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:v0.9.0
rbac subject: system:serviceaccount:roksbnkctl-ops:roksbnkctl-ops
trusted-profile: Profile-ccba11f2-3b1f-4b1a-b8a4-aeed2b7b3320
secret: roksbnkctl-ibm-creds (rotated 2026-05-10T11:03:17Z)
What each line surfaces:
- Pod phase + readiness —
Running+trueis green; anything else means the pod is unhealthy andBackend.Runcalls will fail. The container count is exactly one (tools);ready: trueis a single bool, not a2/2-style ratio. - Image — the
:v…tag matches theroksbnkctlrelease the image was published with (resolved at install time from the binary’s version; see Chapter 17 §:devtag resolution). Mismatched against yourroksbnkctl --versionmeans re-runningops installwill pull the matching image. - RBAC subject — the SA the pod runs as.
kubectl describe clusterrole roksbnkctl-opsprints the full ruleset (the ClusterRoleBinding is named the same as the role). - Trusted-profile line — reads the SA’s
iam.cloud.ibm.com/trusted-profileannotation. The value is the IBM IAM Profile ID (Profile-<uuid>, the canonical IAM identifier —runOpsInstallannotates withtp.IDrather than the friendlyroksbnkctl-ops-<workspace>name so the value is grep-friendly against IBM Cloud IAM audit logs; the parenthetical form on the✓ Provisioned IAM trusted profile …install line cross-references both). Present + non-empty meansops installran with--trusted-profile=autoor=onand the provisioning succeeded; the runtime cred flow is going via the projected SA token. Under--trusted-profile=off(orauto-fallback when IAM perms were missing) the line readstrusted-profile: (none — static-key Secret path). - Secret line — the cred Secret’s name + the
roksbnkctl.io/rotated-atannotation thatops installstamps each time the Secret is applied. Always emitted (the Secret manifest is always rendered — empty data under trusted-profile success, populated under the static-key paths). If the Secret resource is missing entirely on the cluster, the line readssecret: (missing: …).
The current output is a fixed seven-line key/value block; a structured --output json mode is on the v1.x roadmap once ops show grows additional fields (image-id hash, env-hash reconciliation against the live pod, etc.). See docs/PLAN.md §“What’s deliberately deferred to post-v1.0”.
roksbnkctl ops uninstall
Full removal. Run when decommissioning the cluster, or when you want a clean re-install. The command is a destructive-action gate: by default it prints a preview of what would be deleted and exits successfully; the actual deletion only runs with --confirm.
$ roksbnkctl ops uninstall
Would delete (re-run with --confirm to proceed):
- Pod roksbnkctl-ops/roksbnkctl-ops
- Secret roksbnkctl-ops/roksbnkctl-ibm-creds
- ServiceAccount roksbnkctl-ops/roksbnkctl-ops
- ClusterRole/ClusterRoleBinding roksbnkctl-ops
- Namespace roksbnkctl-ops
- Namespace roksbnkctl-test
$ roksbnkctl ops uninstall --confirm
✓ deleted pod roksbnkctl-ops
✓ deleted secret roksbnkctl-ibm-creds
✓ deleted serviceaccount roksbnkctl-ops
✓ deleted clusterrolebinding roksbnkctl-ops
✓ deleted clusterrole roksbnkctl-ops
✓ deleted namespace roksbnkctl-ops
✓ deleted namespace roksbnkctl-test
Note that the cluster-scoped objects (ClusterRole, ClusterRoleBinding) get cleaned too — they’re not garbage-collected by namespace deletion since they live above the namespace. roksbnkctl ops uninstall --confirm makes this explicit so a stale roksbnkctl-ops ClusterRole can’t outlive a namespace removed via kubectl delete ns.
Both managed namespaces (roksbnkctl-ops and roksbnkctl-test) are deleted. The roksbnkctl-test namespace is where one-shot Job pods (iperf3 client, future probes) land — by the time you’re running uninstall you’ve already concluded those test workloads are finished, so removing the namespace alongside the ops-pod surface keeps the cluster clean.
When to run uninstall:
- Cluster decommission — the cluster is going away, clean up cluster-scoped objects before destroying it.
- Cred rotation when paranoid — the rotation story (next section) doesn’t require uninstall, but if you’re worried about old secrets persisting in etcd snapshots, an uninstall + re-install regenerates the Secret cleanly.
- Image upgrade with a major manifest change — if the embedded
k8s_install.yamlevolves (new RBAC rule, security-context tweak),uninstall+installis the cleanest way to apply.
RBAC: the ClusterRole rules
The full ClusterRole rule set (transcribed from internal/exec/k8s_install.yaml):
| API group | Resources | Verbs | Why |
|---|---|---|---|
batch | jobs | get, list, watch, create, delete | One-shot Job lifecycle (iperf3 client, future probes). The backend creates the Job, watches it, reads logs, deletes it. |
"" (core) | pods | get, list, watch | The backend lists/watches pods to find Job-spawned pods + to wait for the ops pod’s Ready state. No create/delete — pods are owned by their Jobs (or by ops install’s user-side privilege), not by the pod’s SA. |
"" (core) | pods/log | get, list | Log streaming from one-shot Job pods (the bytes the wrapped tool wrote to stdout/stderr). |
"" (core) | pods/exec | create, get | kubectl exec is a create against the pods/exec subresource — the SPDY-channel verb the long-lived ops-pod path uses. |
"" (core) | secrets (named roksbnkctl-ibm-creds) | get | The pod reads the cred Secret directly only if a future workflow opts to (kubelet’s projection of envFrom: secretRef runs as kubelet, not as this SA). The resourceNames filter keeps the SA from reading any other Secret in the namespace — least-privilege per PRD 04 §“In-cluster pod”. |
Notably not granted:
podscreate / delete — the SA can list and watch pods but can’t create or delete them. Pod lifecycle is mediated by Jobs (which the SA does manage) and by the ops pod itself (whichops installcreates with the user’s privilege, not the SA’s).secretscreate / update / delete / list — the pod never writes Secrets, and can’t even list to discover which Secrets exist in the namespace. The install-time Secret creation is done by the user invokingops install(whose kubeconfig has cluster-admin or comparable), not by the pod’s SA. Combined with theresourceNamesfilter onget, this is the tightest practical surface that still lets the pod consume its own cred.services,deployments,namespaces— the SA can’t touch these at all. The iperf3 server fixture (when the throughput test runs) is provisioned byroksbnkctl test throughputrunning on the caller’s side using the user’s kubeconfig, not by anything inside the ops pod.clusterroles,clusterrolebindings— the pod never modifies its own RBAC.*cluster-admin — explicitly avoided. The pod has exactly the verbs it needs and nothing else.
This matches PRD 04 §“Least privilege per backend” and PRD 03 §“K8s”: the ops pod is a powerful tool but its blast radius is bounded.
To audit the rules on a running cluster:
kubectl describe clusterrole roksbnkctl-ops
kubectl auth can-i --as=system:serviceaccount:roksbnkctl-ops:roksbnkctl-ops \
'*' '*' --all-namespaces # should print mostly "no"
Credential propagation
Note. What follows is the static-key propagation path — the fallback rather than the default.
--trusted-profile=autoinstalls assume an IBM Cloud trusted profile via the pod’s projected SA token and the static API key never lands in a Kubernetes Secret. See §“Trusted-profile flow” above for that path. The hop-by-hop description below describes what happens under--trusted-profile=off(and under theauto-fallback when IAM perms don’t allow the trusted-profile path).
The IBMCLOUD_API_KEY reaches the wrapped tool in three hops:
resolver chain (env → keychain → config-b64 → prompt)
↓ on the laptop, at `roksbnkctl ops install` time
Kubernetes Secret roksbnkctl-ibm-creds in roksbnkctl-ops namespace
↓ applied by `ops install` via kubectl-equivalent
Pod env (IBMCLOUD_API_KEY=…) via `envFrom: secretRef`
↓ kubelet reads Secret, sets env on container start
Wrapped tool (`ibmcloud iam oauth-tokens`) reads from os.Getenv
Three properties this gives you:
- The key never appears in argv.
kubectl describe pod roksbnkctl-opsshowsenvFrom: secretRef: name: roksbnkctl-ibm-creds, not the value.kubectl get pod roksbnkctl-ops -o yamlshows the same. - The key never appears in the pod’s own logs. The wrapped tool uses the env var; the env var name (not value) is what the pod’s startup logs print.
- The redactor is the defense-in-depth backstop. If the wrapped tool ever prints the value (e.g.,
ibmcloud --debug), the SPDY stream from the pod is wrapped throughinternal/exec/redact.gobefore reaching the caller’s stdout — same as the local + docker backends.
The Secret carries two keys today — IBMCLOUD_API_KEY and the legacy IC_API_KEY alias older ibmcloud versions accept — both populated from the same resolved value. Names are stable; embedded in internal/exec/k8s_install.yaml. Future cluster-side credentials (an AWS access key, a GCP service-account JSON) will add new keys to the same Secret rather than spinning up new Secrets, simplifying RBAC.
Rotation: rotating the API key
Note. Under
--trusted-profile=auto/=on(default), there’s nothing to rotate — the ops pod’s IAM tokens are short-lived and the IBM IAM endpoint refreshes them transparently each time the SDK trades the projected SA token. Key rotation only matters when the install ran with--trusted-profile=offor fell back to the static-key Secret because the resolved key lackediam-identityperms. The procedure below covers that static-key case.
When the IBM Cloud API key changes (key rotation, account takeover, key compromise), you need to update the cluster-side Secret. The flow:
# 1. Update the local resolver chain — pick whichever source you populated
# initially (the chain order is: env > keychain > config-b64; see chapter 14):
export IBMCLOUD_API_KEY=<new-key> # env (one-shot)
# or update the keychain entry directly: `keyring` / `secret-tool` / Keychain.app
# or edit ~/.roksbnkctl/<workspace>/config.yaml's api_key_b64 field
# 2. Re-run ops install — this re-resolves the key, updates the cluster
# Secret, and rolls the pod
roksbnkctl ops install
What ops install does on re-run:
- The Secret
roksbnkctl-ibm-credsis updated with the new value via a client-side Get + Update (the existing Secret’sdatais overwritten, theroksbnkctl.io/rotated-atannotation is refreshed, the rest of the metadata is left alone). - The pod’s env, however, is set at container-start time — kubelet reads the Secret value when the pod is created, not on every Secret update. So an updated Secret doesn’t propagate to the running pod’s env until the pod is recreated.
ops installtherefore deletes and recreates the bare ops pod after the Secret update. New pod → kubelet reads the updated Secret → env contains the new value. (Re-creation takes a few seconds for the image cache hit; up to ~30 seconds on a cold cluster.)
The ops pod is a bare Pod, not a Deployment or DaemonSet, so kubectl rollout restart won’t work on it (rollout restart only operates on controller resources). The canonical way to force a fresh pod is roksbnkctl ops install (idempotent — it’ll handle the delete-and-recreate). If you really want to do it by hand:
kubectl delete pod roksbnkctl-ops -n roksbnkctl-ops
# then re-run `roksbnkctl ops install` to create the replacement;
# the bare pod has no controller, so nothing else will recreate it.
roksbnkctl ops show will report phase: Pending briefly during recreation, then Running + ready: true once kubelet finishes projecting the updated Secret.
Operability
Things to know when something’s wrong.
Where pod logs go
roksbnkctl k logs -n roksbnkctl-ops roksbnkctl-ops
# or
kubectl logs -n roksbnkctl-ops roksbnkctl-ops
The pod’s main process is sleep infinity, so the log is mostly empty. Each kubectl exec invocation runs in its own ephemeral process — those processes’ stdout/stderr go back through the SPDY channel to the caller, not into the pod’s log. So kubectl logs is helpful for debugging pod startup (image pull failures, SCC denials, OOMKills) but not for “what did ibmcloud iam oauth-tokens actually print” — that’s just the caller’s stdout.
For a paper trail of recent invocations, capture roksbnkctl ibmcloud --backend k8s … 2>&1 | tee /tmp/ibmcloud.log on the calling side.
Debugging a stuck ops install
ops install waits up to 60 seconds for the pod to become Ready. If it times out:
roksbnkctl k describe -n roksbnkctl-ops pod/roksbnkctl-ops
roksbnkctl k get -n roksbnkctl-ops events --sort-by=.lastTimestamp | tail -20
Common causes:
| Symptom | Cause | Fix |
|---|---|---|
ImagePullBackOff | ghcr.io rate limit, or image tag doesn’t exist | check roksbnkctl --version, ensure ghcr.io is reachable from the cluster |
CreateContainerConfigError referencing the Secret | Secret was deleted between Secret apply and Pod create (race) | re-run roksbnkctl ops install (idempotent) |
RunContainerError with SCC denial | the cluster’s PodSecurity admission rejected the manifest | kubectl get events will name the missing field; usually means an OpenShift cluster expects the restricted-v2 profile and a manifest field is wrong — file an issue with the event message |
Pod stuck in Pending with no Events | cluster is at capacity / out of CPU | scale the cluster or trim resources; the pod requests 50m CPU + 128Mi mem, very small |
Cluster API outage during ops install
If the kube-apiserver becomes unreachable mid-install (transient cloud-provider issue, kubeconfig expired, network partition), ops install fails fast at whichever step hit the apiserver:
✓ applied namespace roksbnkctl-ops
applying secret roksbnkctl-ops/roksbnkctl-ibm-creds ... ERROR: Get "https://...": dial tcp: i/o timeout
The install is partial at that point — earlier steps succeeded, later steps didn’t. ops install is idempotent, so just re-run once the apiserver is back; the steps that already completed are no-ops the second time, the steps that didn’t will run.
If the apiserver is permanently gone (cluster destroyed): ops uninstall will fail the same way, since it also needs the apiserver. In that case the cluster-scoped objects (ClusterRole, ClusterRoleBinding) become orphans you can clean up manually if you ever rebuild the cluster, or ignore if you’re done with this cluster’s identity entirely.
Verifying the install end-to-end
A one-liner sanity check:
roksbnkctl ibmcloud --backend k8s iam oauth-tokens
If the SA/Secret/RBAC/Pod chain is healthy, this prints a fresh OAuth token. If it errors, the error message names which link in the chain broke (pod not found, Secret missing, exec denied, ibmcloud CLI exit non-zero).
Chapter 26 — Troubleshooting covers the broader “ops pod is unhappy” failure modes alongside other end-user troubleshooting.
Cross-references
- PRD 03 — pluggable execution backends, §“K8s” — the ops-pod design rationale.
- PRD 04 — credential propagation, §“In-cluster pod” — Secret-based propagation rules.
- Chapter 14 — Credentials and the resolver chain — where the
IBMCLOUD_API_KEYvalue comes from before it lands in the Secret. - Chapter 17 §“K8s backend” — the interface mechanics this chapter complements.
- Chapter 18 — Choosing a backend per tool — when
--backend k8sis the right call. internal/exec/k8s_install.yaml— the embedded RBAC manifests: https://github.com/jgruberf5/roksbnkctl/blob/main/internal/exec/k8s_install.yamlinternal/cli/ops.go— theroksbnkctl ops install/show/uninstallcommand implementation: https://github.com/jgruberf5/roksbnkctl/blob/main/internal/cli/ops.go
Connectivity testing
roksbnkctl test connectivity answers one question: can my workspace reach the HTTP/HTTPS endpoints I care about right now?
It’s the simplest of the three test suites — no cluster fixtures, no remote vantage, no JSON parsing harness. Each configured URL gets one HTTP GET, the suite reports pass/fail, and the runner exits 0 if every probe passed.
Use it as the first sanity check after roksbnkctl up, as a CI smoke step against a known-good fixture set, or as the “is it me or is it the network” baseline before reaching for curl -v or openssl s_client.
What the connectivity suite does
For each configured URL the runner:
- Adds an
https://scheme if you didn’t write one. - Issues a single
GETwith a 10-second timeout and the user-agentroksbnkctl/test. - Records the HTTP status code, the wall-clock duration, and (for HTTPS) the negotiated TLS version.
- Marks the probe pass if the status code is in
[200, 400)(any 2xx or 3xx); fail for anything else, any TLS error, any DNS error, any timeout. - Aggregates the per-URL results into a suite result; the suite passes only when every URL passed.
That’s it. No retries, no expected-body matching, no configurable status assertions, no L4 reachability — those are deliberate non-goals (see § When connectivity is the wrong tool below).
Configuring extra_hosts
The list of URLs to probe lives in your workspace config under test.connectivity.extra_hosts:
# ~/.roksbnkctl/<workspace>/config.yaml
test:
connectivity:
extra_hosts:
- https://my-bnk-cis-controller.example.com
- https://bigip-next-admin.example.com:8443
- https://gslb.example.com
- my-bare-host.example.com # scheme defaults to https://
The schema is intentionally minimal — extra_hosts is a []string of URLs (or bare hostnames; https:// is added when no scheme is present). One entry per line. The order in the file is the order the runner probes.
You don’t have to hand-edit the YAML to populate this list — roksbnkctl test hosts {list,add,remove,clear} reads and writes the same slice. See § Managing test hosts via the CLI below.
There’s no per-host method, no per-host expected-status, and no per-host TLS-trust override today. If you need to assert something more specific than “does HTTP work” — a particular status code, a custom header, a body match — curl is the right tool, not roksbnkctl test connectivity. A richer per-host schema is queued for v1.x; the v1.0 surface holds the YAML simple on purpose.
Chapter 12 — Workspace config covers the full test: block; this chapter expands the connectivity slice.
What extra_hosts typically holds
Three classes of URL show up most often in a real workspace:
- The BNK CIS controller — confirms the data-plane front-end is reachable and is returning a sane status code.
- The F5 BIG-IP Next admin endpoint — confirms the management plane is reachable from your seat (often
:8443rather than:443). - The GSLB VIP that fronts the application — confirms the routed name actually serves a 2xx; pair with
roksbnkctl test dnsfor the GSLB-aware DNS-side validation.
What doesn’t belong in extra_hosts: anything you only care about on a specific TLS error, anything that needs a request body, anything where pass/fail is more nuanced than “got a 2xx or 3xx”. Those are curl jobs, not connectivity-suite jobs.
Managing test hosts via the CLI
The YAML in § Configuring extra_hosts is the persistent backing store, but you don’t have to hand-edit config.yaml to populate it. roksbnkctl test hosts mirrors the roksbnkctl targets ergonomic (see Chapter 15 — SSH targets) and gives you four subcommands that read and write the test.connectivity.extra_hosts slice directly:
roksbnkctl test hosts list # print current extra_hosts, one per line
roksbnkctl test hosts add <url> [<url>...] # append; idempotent (no-op if already present)
roksbnkctl test hosts remove <url> [<url>...] # remove; tolerant of absent (no error)
roksbnkctl test hosts clear # remove ALL (confirmation prompt, --auto skip)
All four route through the same workspace marshaller targets add uses, so a test hosts add write lands in ~/.roksbnkctl/<workspace>/config.yaml under test.connectivity.extra_hosts — operators who already know the YAML field see the connection. Run cat ~/.roksbnkctl/<workspace>/config.yaml after any mutation to confirm.
Argv discipline matches the rest of the v1.7.x surface: list and clear take cobra.NoArgs (a stray positional is rejected at parse time, not silently consumed); add and remove take cobra.MinimumNArgs(1). A typo like roksbnkctl test hosts list extra fails before any RunE runs.
Worked example
Start from an empty workspace, add a URL, list, then run the connectivity probe — the exact sequence an operator hits the first time the no hosts configured to probe; add via roksbnkctl test hosts add <url> error fires from roksbnkctl test connectivity or no-flag roksbnkctl test dns:
$ roksbnkctl test hosts list
$ echo $?
0
$ roksbnkctl test hosts add https://docs.f5.com
added https://docs.f5.com
$ roksbnkctl test hosts add https://docs.f5.com
already present: https://docs.f5.com
$ roksbnkctl test hosts list
https://docs.f5.com
$ roksbnkctl test connectivity
running connectivity ...
PASS https://docs.f5.com 200 OK in 187ms
connectivity PASS (1/1 passed)
Note: the byte-for-byte output above is illustrative — it reflects the surface the Sprint 24 CLI ships and is what tech-writer’s drift sweep will re-capture against the built binary before the
v1.7.1cut. The shape is stable (listempty → zero bytes + exit 0;addagainst already-present → “already present:” log + exit 0); minor wording deltas in the log lines may surface in the GREEN-verdict re-capture.
--output json is supported on list for scripted callers — empty list emits [], populated list emits a JSON array preserving insertion order:
$ roksbnkctl test hosts list -o json
["https://docs.f5.com","https://bigip-next-admin.example.com:8443"]
clear
clear requires confirmation (defaults to No) because it wipes the entire slice — the same pattern roksbnkctl down and roksbnkctl cluster down use, not the per-target targets remove pattern:
$ roksbnkctl test hosts clear
This will remove ALL configured test hosts (2 entries). Continue? [y/N]: y
cleared 2 entries
$ roksbnkctl test hosts clear --auto
cleared 0 entries
--auto skips the prompt for automation contexts. Mirrors the --auto semantics elsewhere in the binary.
Scope of the CLI
The CLI manages only test.connectivity.extra_hosts — the slice shared by roksbnkctl test connectivity and no-flag roksbnkctl test dns. The test.dns.{default_target,resolvers} and test.throughput.* fields have working flag-driven equivalents (--target, --server, --duration, …) and aren’t covered by test hosts; populate those by hand-editing config.yaml for now. A broader test config surface is a candidate for a future sprint.
YAML-comment preservation through the marshaller is best-effort and depends on the workspace round-trip — if you keep hand-written comments in config.yaml, prefer the YAML path for additions that need to land alongside specific comments; use the CLI for the routine add/remove/list/clear flow.
The --insecure flag
Self-signed certs are common in pre-production BNK deployments — the F5 BIG-IP Next admin endpoint, the CIS controller, an internal GSLB VIP that hasn’t yet been re-fronted with a public CA cert. By default Go’s TLS stack rejects them and the probe fails with x509: certificate signed by unknown authority.
Pass --insecure to skip certificate verification for the run:
roksbnkctl test connectivity --insecure
What --insecure does:
- Sets
tls.Config.InsecureSkipVerify = trueon the HTTP client used by the connectivity suite. - Applies for the duration of one invocation only.
- Affects every URL probed in that run.
What --insecure does not do:
- It does not change L4 / DNS behaviour. A name that won’t resolve still fails; a host that drops TCP still fails.
- It is not per-host — there’s no
--insecure-only=foo.example.com. Once set, the run skips verification for everything inextra_hosts. - It is not persisted. Setting it in one invocation does not affect the next.
- It is not the same as a config-level
insecure_tls: trueper host. The schema doesn’t have that knob; the only way to skip cert verification today is the session-wide flag.
If you need different TLS-trust posture per endpoint (one URL strict, another lenient), run two invocations with two different extra_hosts lists in two workspaces — that’s the workaround until per-host trust lands.
Reading the output
Default output is human-readable on stderr; pass -o json for machine-readable on stdout.
Human-readable
$ roksbnkctl test connectivity
running connectivity ...
PASS https://my-bnk-cis-controller.example.com 200 OK in 142ms
PASS https://bigip-next-admin.example.com:8443 302 Found in 88ms
FAIL https://gslb.example.com Get "...": dial tcp: i/o timeout
connectivity FAIL (2/3 passed)
$ echo $?
1
A 3xx redirect counts as pass — the runner doesn’t follow redirects, but the redirect itself is a successful HTTP response, which is what the suite measures. If you specifically need the final 200 after a redirect chain, curl -L is the tool.
JSON
$ roksbnkctl test connectivity -o json
{
"schema": "roksbnkctl.v1",
"command": "test",
"suite": "connectivity",
"timestamp": "2026-05-10T14:32:01.123Z",
"duration_ms": 235,
"overall": "fail",
"results": [
{
"suite": "connectivity",
"name": "https://my-bnk-cis-controller.example.com",
"status": "pass",
"detail": "200 OK in 142ms",
"duration_ms": 142,
"extra": { "status_code": 200, "tls_version": "TLS 1.3" }
},
{
"suite": "connectivity",
"name": "https://gslb.example.com",
"status": "fail",
"detail": "Get \"https://gslb.example.com\": dial tcp: i/o timeout",
"duration_ms": 10003
}
]
}
Exit code follows the same rules as the human-readable form: 0 on overall: pass, 1 on overall: fail. CI runners can branch on the exit code; richer assertions (e.g., “I tolerate one fail out of five”) need to consume the JSON.
Running connectivity inside roksbnkctl test all
Connectivity is one of the suites the bare roksbnkctl test (or roksbnkctl test all) command dispatches. The runner walks every configured suite, prints per-suite summaries on stderr, and exits non-zero if any suite failed:
$ roksbnkctl test
running connectivity ...
PASS https://bnk-cis.dev-tor.example.com 200 OK in 174ms
running dns ...
PASS bnk-cis.dev-tor.example.com resolved 1 address(es)
connectivity PASS (1/1 passed)
dns PASS (1/1 passed)
PASS overall (2/2 suites passed)
In -o json mode, roksbnkctl test all emits an all-shape envelope with one suites[] entry per suite. CI assertions can pin to either the suite-level overall or to a specific probe’s status:
roksbnkctl test all -o json | jq -e '.suites[] | select(.suite=="connectivity") | .overall == "pass"'
The bare roksbnkctl test defaults to the all suite. To run connectivity in isolation:
roksbnkctl test connectivity # explicit suite
roksbnkctl test connectivity --insecure # session-wide TLS skip
Exit codes and CI integration
exit 0 → every probe passed (every URL returned 2xx or 3xx)
exit 1 → any probe failed (non-2xx/3xx, TLS error, DNS error, timeout)
There’s no third “infra error” exit code from the connectivity suite specifically — the suite is straight Go HTTP, no external tooling, no backend dispatch. If roksbnkctl test connectivity exits non-zero, the cause is in the response from one of your configured URLs.
For a CI step that tolerates a known-flaky endpoint while still failing on the others, consume the JSON instead of relying on the exit code:
roksbnkctl test connectivity -o json \
| jq -e '[.results[] | select(.name | test("flaky-staging.example") | not) | .status] | all(. == "pass")'
Worked example: probing a BNK deployment
A typical post-up config for a BNK trial — cover the data-plane VIP, the admin endpoint, and the GSLB front:
# ~/.roksbnkctl/dev-tor/config.yaml
test:
connectivity:
extra_hosts:
- https://bnk-cis.dev-tor.bnkfun.example.com # BNK CIS controller (data plane)
- https://bigip-next-admin.dev-tor.bnkfun.example.com:8443 # F5 BIG-IP Next admin
- https://gslb-vip.dev-tor.bnkfun.example.com # the GSLB front
Then:
$ roksbnkctl test connectivity --insecure
running connectivity ...
PASS https://bnk-cis.dev-tor.bnkfun.example.com 200 OK in 174ms
PASS https://bigip-next-admin.dev-tor.bnkfun.example.com:8443 302 Found in 91ms
PASS https://gslb-vip.dev-tor.bnkfun.example.com 200 OK in 211ms
connectivity PASS (3/3 passed)
--insecure is needed here because BNK’s admin endpoint and the GSLB VIP are fronted by a self-signed cert in dev. Once the trial moves to a production cert chain, drop the flag — the strict path is what you want for staging and prod.
When connectivity is the wrong tool
roksbnkctl test connectivity is “does HTTP work”. For anything finer-grained, reach for the right tool:
| Scenario | Use this instead |
|---|---|
| You want to see the full TLS handshake, the cert chain, the SNI resolution, the negotiated cipher | openssl s_client -connect host:port -servername host |
| You want headers, redirect-following, body matching, a specific status assertion | curl -v -L --fail-with-body <url> |
| You want to confirm L4 reachability on a specific port, no HTTP layer | nc -vz host port (or bash -c 'echo > /dev/tcp/host/port') |
| You want to confirm DNS resolution from a specific resolver, especially across vantages for GSLB | roksbnkctl test dns |
| You want to see what answer a name returns from inside the cluster vs from your laptop | roksbnkctl test dns --gslb-compare |
| You want to measure bandwidth between two endpoints | roksbnkctl test throughput |
The connectivity suite is intentionally a thin probe. When the answer to “is it broken” is “yes” and you need to know why, the suite has done its job — it’s flagged the URL — and the next step is one of the tools above.
Cross-references
- Chapter 12 — Workspace config — full
test:block schema, includingconnectivity.extra_hosts. - Chapter 15 — SSH targets — the
roksbnkctl targetsergonomic precedent thatroksbnkctl test hostsmirrors. - Chapter 21 — DNS testing for GSLB — when “the URL fails” actually means “the name doesn’t resolve from this vantage”.
- Chapter 22 — Throughput testing — the bandwidth-measurement companion suite.
- Chapter 26 — Troubleshooting — common patterns for diagnosing connectivity failures across BNK / ROKS deployments.
DNS testing for GSLB
Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the
Workspacestruct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.
roksbnkctl test dns is the diagnostic surface for DNS-driven traffic management — the kind of behaviour an F5 BIG-IP Next GSLB deployment depends on, where the answer a name returns isn’t a single global truth but a function of who’s asking from where.
This is the longest chapter in the testing section because the question it answers is the most subtle. Connectivity testing tells you “the URL works”; throughput testing tells you “the path is fast”. Both assume the name resolved. When the GSLB is the thing under test, “the name resolved” is itself the question — and the answer changes depending on the network vantage of whoever’s asking.
The flag surface, the JSON output, and the multi-vantage workflow on this page are all what v1.0 ships. The design rationale lives in PRD 03 §“DNS probe (GSLB-aware)”; read that for the why, this chapter for the how.
Pointing the deployment at a BIG-IP DNS. This chapter covers testing GSLB behaviour. The deployment side is one setting:
bnk.gslb_datacenter_name, which names the datacenter the CNEInstance registers as.There was also a
bnk.gtm.{url,username,password_b64}block for the BIG-IP DNS connection. It was removed in #227: the environment variables it produced appear nowhere in the f5ingress controller binary on either BNK line, so it configured nothing. The GTM side of the relationship is configured on the GTM, not from here.
Three vantages, one comparison
--gslb-compare is the flagship workflow: a single roksbnkctl test dns invocation fans out across local, k8s, and (optionally) ssh:<target> vantages in parallel, asks each one to resolve the same name, and reports whether the answers diverged.
graph TB
subgraph runner[roksbnkctl test dns --gslb-compare]
cmd[fan-out parallel<br/>per-vantage probe]
cmp[divergence detector<br/>+ JSON aggregate]
end
subgraph vantage_local[local vantage]
L[laptop resolver<br/>public DNS / GSLB outside-the-cluster answer]
end
subgraph vantage_k8s[k8s vantage]
K[ops-pod resolver<br/>cluster CoreDNS / cluster-routed GSLB answer]
end
subgraph vantage_ssh[ssh:jumphost vantage]
S[remote-host resolver<br/>third network path's GSLB answer]
end
GSLB[F5 BIG-IP Next GSLB<br/>per-resolver dispatch rule]
cmd --> L
cmd --> K
cmd --> S
L --> GSLB
K --> GSLB
S --> GSLB
GSLB -. region A IP .-> L
GSLB -. region B IP .-> K
GSLB -. region C IP .-> S
L --> cmp
K --> cmp
S --> cmp
cmp -->|gslb_divergence: true/false| out[stdout JSON<br/>roksbnkctl.dns.v1]
The point of the diagram: a single roksbnkctl invocation probes from network positions a dig from your laptop can’t reach. The cluster vantage answers from the cluster’s egress IP; the SSH vantage answers from a third network path. Comparing the three is exactly the assertion “is the GSLB rule taking effect” needs.
The design rationale for this shape lives in PRD 03 §“DNS probe (GSLB-aware)”. The rest of this chapter is the user-facing surface.
The GSLB problem
F5 BIG-IP Next’s GSLB (Global Server Load Balancing) returns different DNS answers depending on the requesting resolver’s IP address. The discrimination is a feature, not a bug — that’s the whole point of GSLB:
- Geographic affinity: a user in the US gets a US datacenter IP; a user in the EU gets an EU datacenter IP. The dispatch rule is per-region.
- Datacenter routing: a request from a known partner CIDR gets a private VIP; a request from the public internet gets a public VIP.
- Health-check state: when the primary pool member goes unhealthy, GSLB starts handing out the secondary pool member’s IP — but only after the resolver’s TTL on the prior answer expires.
- Anycast vs unicast: a name fronted by an anycast resolver fleet may return the same answer everywhere; the same name fronted by GSLB returns a per-region answer.
Validating GSLB means validating that the right answer comes back from the right vantage. From your laptop in Toronto, you should see the US datacenter IP. From a workload running in the EU, you should see the EU datacenter IP. From an east-Asia bastion, you should see whatever your GSLB rule says east-Asia gets.
The standard dig www.example.com from your laptop only ever tells you what your laptop’s resolver, talking to its configured upstream, gets back. That’s one vantage. To validate GSLB you need several, and you need to be able to compare the answers.
Why per-vantage probing matters
A concrete worked example. Suppose your GSLB rule says:
“Users in the US get the IP for
dc1.example.com(169.45.91.10). Users in the EU get the IP fordc2.example.com(52.123.45.67).”
You run from your laptop in the US:
$ dig +short www.example.com
169.45.91.10
That confirms the US rule. But it tells you nothing about the EU rule. To verify the EU rule you’d have to actually be in the EU — or, more practically, run the query from a network vantage that the GSLB will treat as EU.
roksbnkctl test dns makes that vantage selection a flag:
# From the laptop (your home / office / coffee-shop IP)
roksbnkctl test dns --target www.example.com --type A --backend local
# From inside the cluster (the cluster's egress IP — often a different region)
roksbnkctl test dns --target www.example.com --type A --backend k8s
# From a registered SSH target in the EU
roksbnkctl test dns --target www.example.com --type A --backend ssh:eu-bastion
The --gslb-compare flag fans out across the configured backends in parallel and emits a single JSON report that calls out whether the answers diverged across vantages — exactly the assertion you need for “is the GSLB rule taking effect”.
The roksbnkctl test dns flag surface
roksbnkctl test dns \
[--target <name>] \
[--type <record-type>] \
[--server <server-spec>] \
[--iterations <N>] \
[--backend <local|k8s|ssh:<target>>] \
[--gslb-compare] \
[-o json]
| Flag | Default | Notes |
|---|---|---|
--target | the workspace’s test.dns.default_target if set, otherwise required | The DNS name to query. FQDN preferred (the trailing dot is added if missing). |
--type | A | Any record type the underlying miekg/dns library accepts via dns.StringToType. Common picks: A, AAAA, CNAME, MX, NS, TXT, SRV, SOA, PTR, CAA, DS, DNSKEY, ANY. The full table also includes HTTPS, SVCB, TLSA, SSHFP, URI, NAPTR, RRSIG, NSEC/NSEC3, LOC, etc. |
--server | system | Where to send the query. Literal IP, host:port, the keyword system (use the host’s /etc/resolv.conf), the keyword cluster (use the cluster’s CoreDNS — --backend k8s only), or a name from the workspace’s test.dns.resolvers map. |
--iterations | 1 | How many queries to send to the same server. The runner reports per-query RTT plus p50/p95/p99 across the run. |
--backend | per-tool default (see § Backend selection) | local, k8s, or ssh:<target>. Docker is rejected — see § Why --backend docker is rejected. |
--gslb-compare | off | Fan out across all configured vantages and emit a comparison JSON. See § The --gslb-compare workflow. |
-o json | text | Switch from human-readable text on stderr to JSON on stdout. Two schemas: roksbnkctl.dns.v1.vantage for single-vantage runs (a flat document), roksbnkctl.dns.v1 for --gslb-compare (wraps one or more vantages plus a gslb_divergence boolean). |
The probe library is github.com/miekg/dns — the same DNS implementation CoreDNS uses. Replacing the standard library’s net.Resolver got us three things we couldn’t get otherwise:
- Full record-type surface:
net.Resolveronly exposes a fixed subset; GSLB validation often needsCAA(cert provisioning),DS/DNSKEY(DNSSEC), orSOA(authority chain) whichnet.Resolverdoesn’t. - Per-query server selection: standard library hides the upstream resolver behind whatever
/etc/resolv.confsays; we need to be able to point at a specific GSLB VIP for the query. - Per-query RTT measurement:
miekg/dns’sExchange()returnstime.Durationdirectly. No timing-overhead fudging from running queries serially through the resolver chain.
When --backend k8s is selected, the probe self-execs as a one-shot Job in the roksbnkctl-test namespace — no separate image. roksbnkctl is its own probe runner; the Job’s pod runs roksbnkctl test dns ... with --backend local from the cluster’s network vantage. See Chapter 17 §“K8s backend” for the Job mechanics.
Server resolution
--server accepts five forms:
Literal IP or host:port
roksbnkctl test dns --target www.example.com --type A --server 8.8.8.8
roksbnkctl test dns --target www.example.com --type A --server 8.8.8.8:53
roksbnkctl test dns --target www.example.com --type A --server gslb-vip.example.com:53
Bare hosts default to port 53. IPv6 literals must be bracketed: [2001:4860:4860::8888]:53.
system
roksbnkctl test dns --target www.example.com --type A --server system
Reads /etc/resolv.conf from the host running the probe (so for --backend local that’s your laptop; for --backend k8s that’s the Pod’s /etc/resolv.conf, which CoreDNS owns; for --backend ssh:<target> that’s the target’s /etc/resolv.conf). This is the default if --server is omitted entirely.
cluster
roksbnkctl test dns --target www.example.com --type A --server cluster --backend k8s
Identical to system when running with --backend k8s (CoreDNS is what the Pod’s /etc/resolv.conf points at). Allowed only with --backend k8s — using --server cluster from a local or ssh vantage errors at parse time, since “cluster CoreDNS” isn’t a meaningful concept from outside.
Named resolver from workspace config
roksbnkctl test dns --target www.example.com --type A --server gslb-vip
Looks up gslb-vip in test.dns.resolvers (see next section). Useful for checking the same name against several different upstream resolvers without remembering each IP.
Workspace config: test.dns
Two new keys land this sprint under the existing test: block:
# ~/.roksbnkctl/<workspace>/config.yaml
test:
dns:
default_target: www.example.com
resolvers:
google: "8.8.8.8:53"
cloudflare: "1.1.1.1:53"
gslb-vip: "169.45.91.5:53"
| Field | Type | Default | Notes |
|---|---|---|---|
dns.default_target | string | empty | The name roksbnkctl test dns queries when --target isn’t passed. Lets you keep the per-workspace canonical name out of every CLI invocation. |
dns.resolvers | map[string]string | empty | Named resolvers usable as --server <name>. Values are <host>:<port> (port required; mirrors the --server literal-IP form). |
Both are optional. With neither set, --target is required on every invocation and --server only accepts literal IPs / system / cluster.
Chapter 12 §“test:” is the full workspace-config reference; this is the GSLB-relevant subset.
Backend selection for the probe
The probe runs from one network vantage at a time per --backend:
--backend local: in-process. Runs in theroksbnkctlbinary itself, so the network vantage is your laptop’s. No cluster prereq, no SSH prereq.--backend k8s: a one-shot Job inroksbnkctl-test. The Job’s pod runs the bundled tools imageghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:<tag>— the same image the in-cluster ops pod uses, which carries bothibmcloudandroksbnkctlon PATH. (If a Job fails to pull,kubectl describe podwill name theroksbnkctl-tools-ibmcloudimage — there is no separateroksbnkctl-cliimage to look for.) The Job’s command isroksbnkctl test dns --target ... --type ... --server ... --backend local -o json; the stdout is collected via the k8s backend’s log-stream path. Vantage is the cluster’s egress IP.--backend ssh:<target>: scps theroksbnkctlbinary onto the target if it’s missing (or skips if it’s already there, marker-file gated), then runs the sameroksbnkctl test dns ... --backend local -o jsonover SSH. The vantage is the target’s IP.--backend docker: rejected — see below.
The default backend per roksbnkctl invocation (when --backend is omitted and there’s no exec.dns.backend in workspace config) is local. To run GSLB cross-vantage you generally pass --gslb-compare, which fans out instead of picking a single vantage.
Chapter 17 §“K8s backend” has the one-shot-Job mechanics; Chapter 17 §“SSH backend” has the file-materialisation and bootstrap story. Both apply to the DNS probe verbatim.
The --gslb-compare workflow
Pass --gslb-compare to fan out across all configured vantages in parallel and emit a single comparison JSON:
roksbnkctl test dns \
--target www.example.com \
--type A \
--server gslb-vip.example.com \
--gslb-compare \
-o json
What happens:
- The runner enumerates configured vantages:
localalways;k8swhen a kubeconfig is reachable on the host (the probe runs as a one-shot Job inroksbnkctl-test— the long-lived ops pod isn’t required); plus every entry in the workspace’stargets:block, each asssh:<name>. - Each vantage runs the probe in sequence (one at a time; the run completes when the slowest vantage returns). The query (target, type, server) is identical; only the backend differs. Worst-case wall time with three vantages and the default 2-second per-query timeout is ~6 seconds.
- Per-vantage results are collected with their full RTT distribution and answer set.
- The runner compares the answer sets across vantages. If they differ,
gslb_divergenceis set totruein the output and a human-readable summary names the diverging vantages. - The output is a single
roksbnkctl.dns.v1JSON document wrapping onevantages[]entry per backend.
gslb_divergence: true is not a failure signal — for a healthy GSLB it’s the expected outcome. The exit code is 0 whenever every per-vantage probe succeeded (got an answer, even if the answers differ). The exit code is 1 when any per-vantage probe failed (NXDOMAIN, SERVFAIL, timeout).
JSON output schema
There are two distinct JSON shapes depending on whether --gslb-compare was passed. Both are versioned, both pin against PRD 03 §“DNS probe”, and both can be consumed by CI:
roksbnkctl.dns.v1.vantage— single-vantage probe. A flat document describing one vantage’s result.roksbnkctl.dns.v1— multi-vantage--gslb-compare. Wraps an array of per-vantage entries plus agslb_divergenceboolean.
Single-vantage output (roksbnkctl.dns.v1.vantage)
roksbnkctl test dns --target www.cloudflare.com --type A --server 8.8.8.8 \
--iterations 10 --backend local -o json
{
"schema": "roksbnkctl.dns.v1.vantage",
"backend": "local",
"server": "8.8.8.8:53",
"iterations": 10,
"rtt_ms": { "p50": 12.4, "p95": 18.1, "p99": 22.7 },
"answers": [
{ "name": "www.cloudflare.com.", "type": "A", "ttl": 60, "rdata": "104.16.132.229" },
{ "name": "www.cloudflare.com.", "type": "A", "ttl": 60, "rdata": "104.16.133.229" }
],
"rcode": "NOERROR",
"authoritative": false,
"truncated": false
}
This is the per-vantage shape — no target / type wrapper at the top level (the caller already knows what they queried), no vantages[] array, no gslb_divergence field.
Multi-vantage output (roksbnkctl.dns.v1, divergence detected)
roksbnkctl test dns --target www.example.com --type A --server gslb-vip.example.com \
--gslb-compare -o json
{
"schema": "roksbnkctl.dns.v1",
"target": "www.example.com",
"type": "A",
"vantages": [
{
"schema": "roksbnkctl.dns.v1.vantage",
"backend": "local",
"server": "169.45.91.5:53",
"iterations": 1,
"rtt_ms": { "p50": 14.2, "p95": 14.2, "p99": 14.2 },
"answers": [
{ "name": "www.example.com.", "type": "A", "ttl": 30, "rdata": "169.45.91.10" }
],
"rcode": "NOERROR",
"authoritative": true,
"truncated": false
},
{
"schema": "roksbnkctl.dns.v1.vantage",
"backend": "k8s",
"server": "169.45.91.5:53",
"iterations": 1,
"rtt_ms": { "p50": 8.7, "p95": 8.7, "p99": 8.7 },
"answers": [
{ "name": "www.example.com.", "type": "A", "ttl": 30, "rdata": "10.20.30.40" }
],
"rcode": "NOERROR",
"authoritative": true,
"truncated": false
}
],
"gslb_divergence": true,
"gslb_divergence_summary": "answers differ between local (169.45.91.10) and k8s (10.20.30.40) — GSLB returning location-specific records as expected"
}
The comparison document embeds the per-vantage shape unchanged inside vantages[] — each entry still carries "schema": "roksbnkctl.dns.v1.vantage" so a downstream parser can validate per-vantage entries against the same schema independent of whether they came in standalone or as part of a comparison.
Schema field reference
Per-vantage shape (roksbnkctl.dns.v1.vantage):
| Path | Type | Meaning |
|---|---|---|
schema | string | Always roksbnkctl.dns.v1.vantage. |
backend | string | local, k8s, or ssh:<target>. |
server | string | The resolver address actually used (literal, system-resolvconf result, or named-resolver lookup). |
iterations | int | How many queries went to that vantage. Mirrors --iterations. |
rtt_ms | object | { p50, p95, p99 } across the iterations. Single-iteration runs report the same number for all three. |
answers[] | array | The RRs returned. name is the FQDN, type is the record type, ttl is from the response, rdata is the RR’s data (IP for A/AAAA, target for CNAME, etc.). |
rcode | string | The DNS response code: NOERROR, NXDOMAIN, SERVFAIL, REFUSED, TIMEOUT. |
authoritative | bool | Whether the AA flag was set in the response. |
truncated | bool | Whether the TC flag was set. (The probe automatically retries truncated UDP responses over TCP; the field reflects the final response.) |
error | string | Present only when the probe could not get a usable response. Carries the underlying Go error string. |
Comparison shape (roksbnkctl.dns.v1, emitted by --gslb-compare):
| Path | Type | Meaning |
|---|---|---|
schema | string | Always roksbnkctl.dns.v1. |
target | string | The queried name, normalised to FQDN (trailing dot included). |
type | string | The record type queried. |
vantages[] | array | One per-vantage entry (each conforms to roksbnkctl.dns.v1.vantage). |
gslb_divergence | bool | True when the answer sets across vantages[] differ. |
gslb_divergence_summary | string | Present only when gslb_divergence: true. Human-readable explanation naming the diverging vantages and answers. |
Both schemas are stable at v1.0 — additive changes (new optional fields) are allowed within v1; field renames or removals would bump to .v2. v1.0 includes the optional edns_client_subnet object on each per-vantage entry, emitted when the resolver echoes an EDNS Client Subnet option (RFC 7871) in its response — most GSLB-aware authoritative servers do; vanilla recursive resolvers don’t, in which case the field is omitted from the JSON via omitempty. Sub-fields are family (1 = IPv4, 2 = IPv6), source_netmask, scope_netmask, and address. Useful for confirming the GSLB actually saw your client’s geographic scope rather than the resolver’s IP.
RTT measurement and --iterations
Per-query RTT is captured directly from miekg/dns’s Exchange(), which returns the round-trip duration without the runner having to wrap a stopwatch around the call. That keeps the timing honest — no scheduling jitter from a goroutine yield between time.Now() and the actual UDP send.
--iterations N runs the same query against the same server N times serially and reports p50/p95/p99 across the samples. Use cases:
- Detecting health-check flapping: if a GSLB pool member is on the edge of its health threshold, the answer can flip back and forth across iterations. A single query catches one snapshot; ten iterations show the ratio.
- Detecting anycast routing changes: when an anycast resolver changes which BGP path serves your AS, RTT can jump 30-100ms. p99 catches the worst case; p50 stays steady.
- Establishing a baseline before a change: run with
--iterations 30before a GSLB rule change, again after, and compare distributions.
For --backend k8s and --backend ssh:<target>, RTT is measured inside the remote vantage. The number reflects the resolver-to-resolver path from that vantage, not the laptop-to-cluster (or laptop-to-jumphost) transit. That’s the correct measurement: when you ask “how slow is the GSLB from inside the cluster”, you mean cluster-side latency, not how long it took your laptop to wait for the SSH-tunnelled answer.
Sample F5 BIG-IP Next GSLB scenarios
Three concrete scenarios you’ll hit when validating a real BNK deployment.
Scenario 1: Geographic affinity working as expected
You’ve configured a GSLB rule that returns dc1 (169.45.91.10) for US queries and dc2 (52.123.45.67) for EU queries. You’re in the US, your jumphost is in eu-de (Frankfurt).
roksbnkctl test dns \
--target www.example.com \
--type A \
--server gslb-vip.example.com \
--gslb-compare \
-o json
Expected output:
{
"schema": "roksbnkctl.dns.v1",
"target": "www.example.com",
"type": "A",
"vantages": [
{ "backend": "local", "answers": [{ "rdata": "169.45.91.10" }], "rcode": "NOERROR" },
{ "backend": "ssh:eu-jumphost", "answers": [{ "rdata": "52.123.45.67" }], "rcode": "NOERROR" }
],
"gslb_divergence": true,
"gslb_divergence_summary": "answers differ between local (169.45.91.10) and ssh:eu-jumphost (52.123.45.67) — GSLB returning location-specific records as expected"
}
gslb_divergence: true is the assertion you wanted. There are two ways to key CI on it:
# Option A: parse the JSON yourself
roksbnkctl test dns ... --gslb-compare -o json | jq -e '.gslb_divergence == true'
# Option B: built-in --require-divergence flag (the binary returns
# non-zero exit when --gslb-compare finds NO divergence)
roksbnkctl test dns ... --gslb-compare --require-divergence
Both forms produce a non-zero exit when gslb_divergence is false — the --require-divergence flag is the same assertion baked into the binary so CI scripts don’t need a jq dependency. Pick whichever fits your pipeline.
If gslb_divergence flips to false, something has changed — the GSLB rule was disabled, the geographic dispatch broke, or the jumphost moved out of the EU range. Surface that as a CI failure.
Scenario 2: Health-check-driven failover
A BNK pool member backing www.example.com has an active health check. You want to verify that GSLB stops returning that member’s IP when the health check fails.
Steps:
# 1. Baseline: while the member is healthy, both vantages return its IP
roksbnkctl test dns --target www.example.com --type A --server gslb-vip.example.com \
--gslb-compare -o json > /tmp/before.json
# 2. Take the member offline (BNK admin UI / API; outside roksbnkctl's surface)
# 3. Wait for the GSLB's TTL to expire (in this example, 30 seconds)
sleep 35
# 4. Probe again
roksbnkctl test dns --target www.example.com --type A --server gslb-vip.example.com \
--gslb-compare -o json > /tmp/after.json
# 5. Diff the answer sets
diff <(jq '.vantages[].answers' /tmp/before.json) <(jq '.vantages[].answers' /tmp/after.json)
You expect the IP in vantages[].answers[].rdata to change from the failed member’s IP to the secondary’s IP across both vantages — and to do so on roughly the same TTL boundary across vantages. If one vantage flips and the other doesn’t, the GSLB’s health-check propagation is asymmetric — useful diagnostic data.
Scenario 3: Anycast vs unicast detection
You suspect a name is fronted by a public anycast resolver fleet (Cloudflare’s 1.1.1.1, Google’s 8.8.8.8) rather than a real GSLB. Anycast returns the same answer everywhere; GSLB returns per-region answers.
roksbnkctl test dns --target www.cloudflare.com --type A --server 8.8.8.8 \
--gslb-compare -o json
{
"schema": "roksbnkctl.dns.v1",
"target": "www.cloudflare.com",
"vantages": [
{ "backend": "local", "answers": [{ "rdata": "104.16.132.229" }] },
{ "backend": "k8s", "answers": [{ "rdata": "104.16.132.229" }] }
],
"gslb_divergence": false
}
gslb_divergence: false despite probing from two vantages → the answer is anycast, not GSLB-dispatched. Useful when handing off to a customer who’s claiming “the GSLB isn’t routing me right” — you can prove the name they’re querying isn’t actually under GSLB control.
Why --backend docker is rejected
A Docker container running locally on the user’s laptop has the same network identity as the host (default bridge networking). The container’s egress NATs out via the host’s interface, so a DNS query from inside the container reaches the upstream resolver from the host’s IP.
For GSLB validation, that means --backend docker would give an answer identical to --backend local. No new vantage. The CLI rejects the combination at parse time:
$ roksbnkctl test dns --backend docker --target www.example.com --type A
error: DNS probe doesn't benefit from --backend docker (same network identity
as --backend local, no GSLB-relevant vantage difference). Use --backend
local, --backend k8s, or --backend ssh:<target> instead.
This is by design and called out in PRD 03 §“DNS probe”. If you want frozen-toolchain DNS testing for CI reproducibility, the roksbnkctl binary itself is already a single static binary — pinning it to a specific version pins the probe.
Integration with extra_hosts
If you’ve configured connectivity.extra_hosts for your workspace and want a quick “does each of those names resolve” check:
roksbnkctl test dns
With no --target and no --gslb-compare, the runner falls back to today’s behaviour: probe each host in connectivity.extra_hosts, single-vantage, single-iteration, with the host’s system resolver. This is the same shape as the connectivity suite’s reachability probe — no GSLB awareness, no per-server selection — and is the right tool for “did I typo a hostname in my config” rather than “is the GSLB doing what I configured”.
For real GSLB validation you almost always want --gslb-compare plus an explicit --target and --server. The extra_hosts fallback is the carry-over from earlier roksbnkctl releases, kept for compatibility.
If the no-flag invocation exits with no hosts configured to probe; add via roksbnkctl test hosts add <url>, the slice is empty for this workspace. Populate it with roksbnkctl test hosts add <url> (see Chapter 20 §“Managing test hosts via the CLI”) instead of hand-editing config.yaml.
Chapter 20 — Connectivity testing covers connectivity.extra_hosts in full.
Worked example: GSLB divergence troubleshooting
End-to-end Part VI scenario: a customer’s BNK GSLB rule says “users in the US get IP A, users in the EU get IP B”. They claim the rule isn’t working — EU users keep landing on the US datacenter. You have roksbnkctl configured with a local laptop (your office), the customer’s cluster (ops pod installed), and a customer-provided EU bastion as an SSH target named eu-bastion. Goal: prove or disprove the divergence, point at where to look next.
# 1. Baseline — three-vantage probe of the GSLB-fronted name
$ roksbnkctl test dns \
--target www.example.com \
--type A \
--server gslb-vip.example.com \
--gslb-compare \
-o json | tee /tmp/gslb-baseline.json | jq .
{
"schema": "roksbnkctl.dns.v1",
"target": "www.example.com",
"type": "A",
"server": "gslb-vip.example.com",
"gslb_divergence": false,
"vantages": [
{
"backend": "local",
"answers": [{"rdata": "192.0.2.10"}],
"rtt_ms": {"p50": 18.2, "p95": 22.1}
},
{
"backend": "k8s",
"answers": [{"rdata": "192.0.2.10"}],
"rtt_ms": {"p50": 6.4, "p95": 9.8}
},
{
"backend": "ssh:eu-bastion",
"answers": [{"rdata": "192.0.2.10"}],
"rtt_ms": {"p50": 24.7, "p95": 31.4}
}
]
}
# Note: `gslb_divergence: false` means ALL THREE vantages got 192.0.2.10
# back. That's the smoking gun — the customer is right. The GSLB is
# returning the same answer regardless of where the resolver is.
# 2. Confirm via `dig` from the EU bastion directly (sanity check that
# roksbnkctl isn't somehow masking the answer)
$ roksbnkctl exec --on eu-bastion -- dig +short @gslb-vip.example.com www.example.com
192.0.2.10
# Same answer. The probe isn't lying.
# 3. Check the GSLB rule's resolver-IP detection
$ roksbnkctl ibmcloud --backend ssh:eu-bastion ks cluster get \
--cluster customer-cluster --json | jq '.serviceEndpoints'
# (Inspect the cluster's egress IP that GSLB sees for k8s vantage queries)
# 4. Repeat the probe with a known-good test name (one with a working
# anycast resolver, to rule out probe-side bugs)
$ roksbnkctl test dns \
--target www.cloudflare.com \
--type A \
--server 8.8.8.8 \
--gslb-compare
✓ local : 104.16.124.96 (RTT 12ms)
✓ k8s : 104.16.124.96 (RTT 6ms)
✓ ssh:eu-bastion: 104.16.124.96 (RTT 28ms)
gslb_divergence: false (anycast — expected)
# 5. Hand off the artefact to the GSLB owner with three pieces of evidence:
# - /tmp/gslb-baseline.json (the divergence-false JSON)
# - The cluster's resolver-IP-as-seen-by-GSLB from step 3
# - Confirmation that the probe machinery itself works (step 4)
What this walkthrough lets you say with confidence: “the GSLB is returning 192.0.2.10 to all three vantages including the EU bastion. The rule isn’t dispatching. Check the rule’s resolver-IP-CIDR-match clause against the EU bastion’s egress IP <X>.” That’s a falsifiable claim a GSLB engineer can act on.
Common follow-up failure modes (covered in Chapter 26):
- Probe says diverged but customer says it’s not — likely the customer’s testing point is behind a NAT that masks their network position; their resolver looks like a different region to the GSLB.
- Probe says NOT diverged but customer says it IS — likely the customer is querying through a CDN-fronted resolver that anycasts the request. Re-probe with
--server <gslb-vip-directly>to skip the resolver chain. - All vantages return SERVFAIL — the GSLB-VIP isn’t reachable from any of them. Check connectivity (Chapter 20) before re-running.
Cross-references
- PRD 03 §“DNS probe (GSLB-aware)” — design rationale, the full schema spec, the rejected-by-design list.
- PRD 05 §“Phase L-DNS” — the end-to-end test sequence that validates each row of this chapter against a live cluster.
- Chapter 12 §“test:” — workspace-config schema, including
test.dns.resolversandtest.dns.default_target. - Chapter 14 — Credentials and the resolver chain — how
roksbnkctlitself reaches the cluster / SSH target the probe runs on. - Chapter 17 §“K8s backend” — the one-shot-Job mechanics the probe reuses for
--backend k8s. - Chapter 17 §“SSH backend” — file materialisation and bootstrap for
--backend ssh:<target>. - Chapter 18 §“I’m doing GSLB DNS validation” — the decision-tree row that picks backends for a GSLB scenario.
- Chapter 20 — Connectivity testing — the simpler “does HTTP work” companion suite.
- Chapter 20 §“Managing test hosts via the CLI” —
roksbnkctl test hosts {list,add,remove,clear}, the CLI path for populating theextra_hostsslice the no-flagtest dnsinvocation walks. - Chapter 22 — Throughput testing — the bandwidth-measurement companion suite.
miekg/dnsupstream — the underlying DNS library, used by CoreDNS and a long list of other reference implementations.
Throughput testing
roksbnkctl test throughput measures TCP bandwidth between an iperf3 client and an iperf3 server, with at least one side running adjacent to (or inside) the cluster so the number reflects something useful — cluster fabric, the inbound path through a LoadBalancer (the iperf3 north-south mode, default), the outbound path from a jumphost, or pod-to-pod (east-west).
The heavy lifting (server pod lifecycle, OpenShift SCC compliance, in-cluster client Job, log streaming) lives in Chapter 17 §“K8s backend”. This chapter is the user-facing flag surface, the mode selection, and the output-interpretation guide.
What the suite measures
Plain TCP throughput, plus jitter and retransmits, between two endpoints both running iperf3:
- The server runs in the cluster — a single bare Pod plus a Service, deployed in the
roksbnkctl-testnamespace. Service type isClusterIPfor east-west,LoadBalancerfor north-south. See Chapter 17 §“iperf3 server side” for the manifest details. - The client runs wherever you point the backend — by default in the cluster as a one-shot Job, alternatively on your laptop or on a registered SSH target.
- Output is iperf3’s native
-JJSON, parsed and surfaced asroksbnkctl test throughputJSON.
The suite is appropriate for “is the cluster fabric healthy”, “is the BNK data path delivering the bandwidth I expect from outside”, and “is this jumphost the bottleneck between my office and the cluster”. It is not a precision benchmark — TCP throughput is sensitive to MTU, NIC offloads, kernel tunables, and the iperf3 server’s own resource limits, none of which the suite tries to control.
The two modes
Mode is selected by --mode. The default is north-south.
roksbnkctl test throughput --mode north-south # default
roksbnkctl test throughput --mode east-west
--mode north-south
Measures the inbound path from outside the cluster to a Pod inside it. The server’s Service is a LoadBalancer, so the cluster provisions an external endpoint (an IBM Cloud LB on ROKS, an external IP / hostname on bare-metal k8s). The client connects to that endpoint.
Use cases:
- “Is the BNK ingress path delivering the bandwidth I expect”
- “Is my office Wi-Fi or my home connection the bottleneck”
- “Is the cluster’s egress capacity what the cloud provider promised”
Combine with --backend local (run the client on your laptop) when you specifically want to measure the laptop-to-cluster path. Combine with --backend ssh:<jumphost> when you want a known-stable measurement vantage from a jumphost in a known IP block — useful when laptop Wi-Fi is suspect.
--mode east-west
Measures the intra-cluster fabric — Pod-to-Pod or host-to-Pod. The server’s Service is ClusterIP, reachable only from inside the cluster. The default --backend k8s runs the client adjacent to the server (a one-shot Job in the same namespace), so the number reflects the CNI’s pod-to-pod throughput.
Use cases:
- “Is the cluster’s network plugin healthy”
- “Are the worker nodes hitting the link rate the underlying fabric promises”
- “Has the BNK CIS deployment regressed cluster-internal throughput”
Today’s east-west still allows --backend local (the client runs on the host and reaches the ClusterIP via NodePort-equivalent access if the kubeconfig is the same one a kubectl port-forward would use), but the number is a host-to-cluster-via-NodePort hybrid in that case rather than a true pod-to-pod measurement. True pod-to-pod east-west — both sides scheduled to specific pods, optionally pinned to different nodes via --cross-node — is the v1.x refinement; today the in-cluster Job client gets you most of the way there.
Per-tool default backend
The default backend for iperf3 is k8s. From the per-tool defaults table in Chapter 18 §“Per-tool default backends”:
| Tool | Default backend | Why |
|---|---|---|
iperf3 | k8s | Throughput from a laptop’s uplink isn’t the cluster’s bandwidth. Default to running adjacent to the cluster so the number reflects fabric, not Wi-Fi. |
The default holds whether or not you’ve set exec.iperf3.backend in workspace config. To override per-invocation:
roksbnkctl test throughput --backend local # client on laptop
roksbnkctl test throughput --backend ssh:jumphost # client on jumphost
roksbnkctl test throughput --backend k8s # default; explicit
--backend docker is rejected by the throughput suite. A Docker container running locally has the same network identity as the host (default bridge networking), so the client’s view of the network is identical to --backend local. The CLI errors at parse time:
$ roksbnkctl test throughput --backend docker
error: --backend docker isn't supported for iperf3 — docker shares the host
network namespace by default and gives no network-locality benefit over
local. Use --backend local or --backend k8s instead
Chapter 18 §“Throughput testing” is the decision-tree row that walks through the (mode, backend) matrix.
When local or ssh:<target> makes sense:
local: you’re deliberately measuring laptop-to-cluster bandwidth (north-south from your seat). Typical use is debugging “the dashboard feels slow from my desk” — you want to confirm the office uplink, not the cluster fabric, is the bottleneck.ssh:<target>: you have a registered SSH target with a known IP (often a customer jumphost in a specific datacenter) and want a bandwidth measurement from that vantage. The SSH backend ensuresiperf3is on the target (auto-installs viaaptwith--bootstrapon Ubuntu; see Chapter 17 §“SSH backend”).
The bundled image and the runAsNonRoot constraint
The iperf3 server pod’s securityContext is set to satisfy OpenShift’s restricted-v2 SCC:
securityContext:
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: iperf3
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
capabilities:
drop: ["ALL"]
iperf3 listens on port 5201 (unprivileged) so root isn’t needed. The bundled image at ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<v> declares USER 1000 in its Dockerfile, matching the pod’s runAsUser: 1000.
Two things follow:
-
Stock images that run as root will fail admission. The default in workspace config is
networkstatic/iperf3:latest, which runs as root. On OpenShift / on any cluster withrestricted-v2PodSecurity admission, that image will fail withforbidden: violates PodSecurity "restricted:v1.x". The fix is to switch to the bundled image:# ~/.roksbnkctl/<workspace>/config.yaml test: throughput: image: ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:v0.9.0The bundled image is the
--backend k8sdefault for the client Job regardless; the workspace override only affects the server pod’s image. Keep them in sync to avoid version skew during a debug session. -
A custom workspace-overridden image must respect
runAsNonRoot. If you pointtest.throughput.imageat your own iperf3 image, that image must not require root to start. iperf3 itself doesn’t need privilege; if your image does, drop theUSER rootline and rebuild.
Chapter 17 §“iperf3 server side” goes deeper on the SCC story — what the four securityContext fields do, why each is required, and how to debug an admission failure.
OpenShift SCC failure mode
If your throughput pod fails to start with one of:
Forbidden: violates PodSecurity "restricted:v1.x"unable to validate against any security context constraint: ... restricted-v2runAsNonRoot is required
…then either the configured image runs as root (use the bundled ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<v> image instead — set test.throughput.image in workspace config) or the cluster’s PodSecurity admission is stricter than restricted-v2 (the manifest the k8s backend builds satisfies restricted-v2 but not privileged; if your cluster requires privileged for the test namespace, that’s a cluster policy question outside the suite’s control).
Chapter 17 §“iperf3 server side” is the canonical source for the manifest’s securityContext. If you’re hand-rolling an iperf3 image for the suite, it’s the spec to match.
Reading the output
Default output is human-readable on stderr; -o json switches to JSON on stdout.
Human-readable
$ roksbnkctl test throughput
→ Deploying iperf3 fixture
→ Waiting for iperf3 server pod ready
→ Waiting for LoadBalancer endpoint (can take 30–90s on IBM Cloud)
✓ iperf3 endpoint: 169.45.91.10:5201
running throughput ...
PASS iperf3 north-south → 169.45.91.10:5201 (k8s) 3.41 Gbps received, 0% retransmits in 30s
throughput PASS (1/1 passed)
✓ iperf3 fixture removed
JSON
iperf3’s -J output is rich (sender, receiver, per-stream stats, CPU usage). The roksbnkctl wrapper preserves the iperf3 JSON in the probe’s detail field so all of iperf3’s data survives, while the suite-level shell follows the roksbnkctl.v1 schema:
roksbnkctl test throughput -o json
{
"schema": "roksbnkctl.v1",
"command": "test",
"suite": "throughput",
"timestamp": "2026-05-10T14:32:01Z",
"duration_ms": 31420,
"overall": "pass",
"results": [
{
"suite": "throughput",
"name": "iperf3 north-south → 169.45.91.10:5201 (k8s)",
"status": "pass",
"duration_ms": 30015,
"detail": "{ ...full iperf3 -J JSON, including sum_received, sum_sent... }"
}
]
}
The fields you’ll most often want from the embedded iperf3 JSON, in order of usefulness:
| Field | What it tells you |
|---|---|
end.sum_received.bits_per_second | The throughput number you should report. iperf3 measures both sender and receiver and the receiver number is the right one to quote — it accounts for retransmits and any path losses. |
end.sum_sent.bits_per_second | Sender-side throughput. If sent ≫ received, packets were dropped on the path. If sent ≈ received, the path is healthy. |
end.sum_sent.retransmits | TCP retransmits over the run. A handful is normal; double-digit-percent of streams indicates congestion or a bad NIC. |
end.streams[].sender.jitter_ms | Per-stream jitter. Useful for diagnosing variable-latency paths. |
end.cpu_utilization_percent.host_total | Whether the client CPU was the bottleneck. >80% suggests the iperf3 client maxed out CPU before the network did — increase the iperf3 server’s stream count (a server-pod knob, not a roksbnkctl flag) to spread load, or run on a beefier client. |
Example interpretation:
sum_received: 3.41 Gbps → headline number
sum_sent: 3.42 Gbps → very close, healthy path
retransmits: 127 → normal-low
vs
sum_received: 1.21 Gbps → headline number
sum_sent: 2.95 Gbps → ≫ received; >50% of bytes lost or retransmitted
retransmits: 18743 → heavy
The second shape is what a saturated link or a flaky NIC looks like. The first is a healthy gigabit-class path.
Tuning knobs in workspace config
# ~/.roksbnkctl/<workspace>/config.yaml
test:
throughput:
image: ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:v0.9.0
duration: 30 # iperf3 -t flag, seconds
streams: 8 # iperf3 -P flag, parallel streams
default_mode: north-south
The defaults (30s, 8 streams, north-south) are a reasonable starting point for “is the BNK data path healthy”. For deeper diagnosis:
- Bump
durationto 60-90s if the path is variable and you want a stable average. - Bump
streamsto 16 or 32 if the path’s bandwidth-delay product is high (long-haul links benefit from more parallelism). - Drop
streamsto 1 if you’re specifically testing single-flow throughput (e.g., reproducing a customer’s “single-stream upload feels slow” complaint).
Chapter 12 — Workspace config lists the full schema.
Cleanup and --keep
By default the suite tears down the iperf3 server pod and Service after the client run completes. If a test fails and you want to poke at the fixture (kubectl exec into the server, hand-run iperf3 -c from a third location, etc.), pass --keep:
roksbnkctl test throughput --keep
# ... fixture stays up; debug to your heart's content ...
kubectl delete -n roksbnkctl-test pod/roksbnkctl-iperf3 svc/roksbnkctl-iperf3
The fixture is in the roksbnkctl-test namespace (same namespace the k8s backend uses for one-shot Jobs). It’s a bare Pod plus a Service; nothing else lingers when you delete the two resources.
Cross-references
- Chapter 17 §“K8s backend” — server-side mechanics (manifest, SCC, log streaming, exit-code extraction).
- Chapter 17 §“iperf3 server side” — the asymmetric server-pod-plus-client-Job shape.
- Chapter 18 §“I want to measure cluster bandwidth” — the decision-tree entry that picks
(mode, backend)for your scenario. - Chapter 12 §“test:” — workspace-config schema for
test.throughput.*. - Chapter 20 — Connectivity testing — the simpler “does HTTP work” companion suite.
- Chapter 21 — DNS testing for GSLB — the DNS validation companion.
- PRD 03 §“iperf3” — the design spec.
The performance matrix
Throughput testing runs a single measurement: one iperf3 client against one iperf3 server. roksbnkctl test matrix runs a whole set of them at once. You write a matrix.yaml listing the measurements you want — each one a row — and the command runs them all and prints the results in one report.
Reach for test throughput to check a single path; reach for test matrix to characterise a deployment — throughput, request rates, and latency from several network locations — in one repeatable command instead of a dozen steps by hand. (Don’t confuse it with roksbnkctl testing up, which builds the jumphosts. test only runs tests.)
The matrix runs against an environment that is already deployed: a complete Cluster + BNK deploy with the Testing phase jumphosts up, and — for the HTTP tests — the Gateway phase in place.
The example numbers in this chapter are illustrative. The
--dry-runplan is exercised in CI and its transcript is real, but a full live run has not yet been measured against a production ROKS cluster.
The two families
Each row (a “cell”) picks a family:
iperf3— raw TCP throughput (Layer 4).l7— HTTP load, driven by h2load (Layer 7).
The runner turns each cell into a tool command and runs it from wherever you point it — locally, or on a jumphost over SSH.
iperf3 — TCP throughput
iperf3 measures plain TCP throughput against a host:port target (an L4Route VIP — BNK’s own L4 CRD, gateway.k8s.f5net.com/v1; the Gateway API standard channel BNK installs has no TCPRoute). The length field sets the TCP block size, which stands in for message size: "128" for tiny messages, "512K" for bulk transfer. The other knobs map straight to iperf3 flags:
| Cell field | iperf3 flag | Meaning |
|---|---|---|
length | -l | block size ("128", "512K") |
bytes | -n | send a fixed number of bytes instead of running for a time |
duration | -t | run length in seconds |
streams | -P | parallel TCP streams |
A cell becomes, for example, iperf3 -c 10.240.0.10 -J -p 5201 -t 30 -P 8 -l 512K. The result reports throughput_gbps and retransmits (the same parser as Chapter 22).
l7 — HTTP load
h2load drives HTTP traffic at a URL. The scheme picks the path: an http:// URL hits the plain HTTP route; an https:// URL hits the TLS route, where TMM terminates TLS (h2load accepts the self-signed certificate, so there’s nothing extra to configure). The response size is whatever the URL returns — the test’s backend serves /128, /5k, and /512k, so you point a cell at the size you want.
h2load reports requests per second, a transfer rate, and request latency as min / max / mean (it does not report percentiles).
The l7.mode field is a shortcut that fills in sensible h2load flags; anything you set yourself overrides it:
| Mode | What it stresses | Defaults (when you leave the knobs unset) |
|---|---|---|
cps | new connections — short connections, roughly one request each | --h1, -m 1, -c 50, -n = clients×200 |
tps | sustained requests — reuse connections, many requests each | -c 50, -m 100 (or 1 with --h1), -n = clients×2000 |
throughput | payload — a few connections pulling a large body | -c 8, -m 1, -n = clients×200 |
A cps cell becomes h2load -c 50 -m 1 --h1 -n 10000 http://10.240.0.10/128; a tps cell written as l7: { mode: tps, duration: 30 } becomes h2load -c 50 -m 100 -D 30 …. Set any of clients, streams, threads, requests, duration, http1 to override the defaults. The result reports req_per_sec, throughput_mbps, request counts (requests_total / succeeded / failed), status-code buckets, and req_time_{min,max,mean}_ms.
The locality axis = jumphost placement
There is no “location” setting. Where the traffic comes from is simply which jumphost a cell names as its client. Each vsi endpoint’s target is an SSH target name the Testing phase registered: jumphost (in the client VPC — a different VPC from the cluster), or jumphost-<zone> (in the cluster VPC, in the zone you pick).
| To measure traffic from… | Name this jumphost as the cell’s client |
|---|---|
| same VPC, same zone as TMM | jumphost-<tmm-zone> |
| same VPC, a different zone | jumphost-<other-zone> |
| a different VPC | jumphost |
The runner runs the tool on that jumphost over SSH (see Chapter 15 and Chapter 16). Both tools — iperf3 and h2load — are already installed on every jumphost, so there’s nothing to set up first. A cell with no client runs the tool locally.
The matrix.yaml schema
This is a workspace-sibling file — under ~/.roksbnkctl/<workspace>/, next to config.yaml but separate from it, so you can keep and diff a campaign’s grid in git on its own. The runner looks for --file, then <workspace>/matrix.yaml, then ./matrix.yaml. It has four top-level keys, shown here against the example grid (internal/test/testdata/matrix.example.yaml):
# gateway — names your already-deployed BNK gateway so the test's HTTP/TCP
# routes can attach to it. Only needed when fixtures.routes is true.
gateway:
app_namespace: bnk-apps # namespace the test's objects are created in
name: bnk-gateway # the existing Gateway the routes attach to
http_section: http # a listener on that Gateway, by section name
https_section: https # the TLS listener
tcp_section: tcp # the TCP listener (for the iperf3 route)
# fixtures — the temporary objects the test creates and then removes
# (kept in place if you pass --keep).
fixtures:
iperf3_server: true # the TCP server for the iperf3 cells
http_backend: true # an nginx server returning /128 /5k /512k
routes: true # the HTTP/TCP routes wiring the gateway to them
# endpoints — named (where, what) anchors the cells reference.
endpoints:
vsi-same-zone: { kind: vsi, target: jumphost-jp-osa-1 } # client in TMM's zone
vsi-diff-zone: { kind: vsi, target: jumphost-jp-osa-3 } # another zone, same VPC
vsi-diff-vpc: { kind: vsi, target: jumphost } # the client VPC
tmm-tcp: { kind: address, host: 10.240.0.10, port: 5201 } # iperf3 target
tmm-http-128: { kind: url, url: "http://10.240.0.10/128" }
tmm-https-512k:{ kind: url, url: "https://10.240.0.10/512k" }
# cells — the grid. Each is one row in the report.
cells:
- { name: "L4 512K VSI->TMM diff-zone", family: iperf3, client: vsi-diff-zone, server: tmm-tcp, length: "512K", duration: 30, streams: 8 }
- { name: "L7 http CPS 128B", family: l7, client: vsi-diff-vpc, server: tmm-http-128, l7: { mode: cps } }
- { name: "L7 https THRU 512K", family: l7, client: vsi-diff-vpc, server: tmm-https-512k, l7: { mode: throughput, duration: 30 } }
gateway: — the existing-gateway identity
Names your already-deployed gateway so the test’s routes can attach to it. Only read when fixtures.routes is true.
| Field | Notes |
|---|---|
app_namespace | namespace the test’s objects are created in (required for routes) |
name | the existing Gateway the routes attach to (required for routes) |
http_section / https_section / tcp_section | which listeners on that Gateway the http / https / tcp routes attach to, by section name. They must already exist. Leave one empty to skip that route. |
class_name / controller_name / bnkgateway_name / flo_namespace | optional descriptive fields; not needed to attach routes |
endpoints: — named anchors
kind | Fields | Is |
|---|---|---|
vsi | target | a jumphost the test sends traffic from (the client) |
address | host, port (default 5201) | an iperf3 target — a host:port (an iperf3 cell’s server) |
url | url | an HTTP target — a full http(s):// URL (an l7 cell’s server) |
cells: — the grid
| Field | Applies to | Notes |
|---|---|---|
name | both | required; the report row label and the --only glob target |
family | both | iperf3 | l7 |
client | both | a vsi endpoint; empty → run locally |
server | both | an address endpoint for iperf3, a url endpoint for l7 |
length / bytes / duration / streams | iperf3 | iperf3 -l / -n / -t / -P |
l7.mode | l7 | cps | tps | throughput (required for l7) |
l7.{clients,streams,threads,requests,duration,http1} | l7 | h2load -c / -m / -t / -n / -D / --h1; override the mode defaults |
The file is checked before anything runs: a client must be a vsi with a target; an iperf3 server must be an address with a host; an l7 server must be a url and the cell must set a mode; and routes needs gateway.name, app_namespace, and at least one listener section.
What the test sets up (and cleans up)
For a real run the test creates a few temporary objects, runs the measurements, then deletes them again (unless you pass --keep). Three toggles under fixtures — all skipped under --dry-run:
iperf3_server— the TCP server the iperf3 cells measure against (the same onetest throughputuses).http_backend— an nginx server that returns fixed-size bodies at/128,/5k, and/512kfor the HTTP cells.routes— the HTTP and TCP routes (plus a TLS route and a self-signed certificate, when an https section is named) that wire your Gateway to those backends.
Everything the test creates is labelled app.kubernetes.io/managed-by: roksbnkctl-matrix, so cleanup is a single delete by label. The TLS certificate is self-signed and generated fresh each run — fine for a load test.
One prerequisite for the HTTP/TCP routes: your Gateway must already have the listener sections the routes attach to (http / https / tcp). The test adds the routes and backends, not the listeners — those come from the Gateway phase. If a named section doesn’t exist, the route has nowhere to attach and that cell’s traffic has no path. Run with --keep and check the route status if a cell isn’t reaching its backend.
The workflow
--dry-run — see the plan first
--dry-run expands the grid and prints, for each cell, the resolved client, server, and exact tool command — plus the objects it would create — and makes no cluster calls. Use it to confirm placement and commands before a long run:
$ roksbnkctl test matrix --dry-run --file matrix.example.yaml
→ matrix file: matrix.example.yaml
matrix plan — 10 cell(s)
[1] L4 128B VSI->TMM same-zone
family : iperf3
client : ssh:jumphost-jp-osa-1
server : 10.240.0.10:5201
run : iperf3 -c 10.240.0.10 -J -p 5201 -t 30 -l 128
[2] L4 512K VSI->TMM same-zone
family : iperf3
client : ssh:jumphost-jp-osa-1
server : 10.240.0.10:5201
run : iperf3 -c 10.240.0.10 -J -p 5201 -t 30 -P 8 -l 512K
[5] L7 http CPS 128B VSI->TMM
family : l7
client : ssh:jumphost
server : http://10.240.0.10/128
run : h2load -c 50 -m 1 --h1 -n 10000 http://10.240.0.10/128
[8] L7 https CPS 128B VSI->TMM
family : l7
client : ssh:jumphost
server : https://10.240.0.10/128
run : h2load -c 50 -m 1 --h1 -n 10000 https://10.240.0.10/128
--- fixtures (would apply) ---
apiVersion: apps/v1
kind: Deployment
metadata:
name: matrix-http-backend
namespace: bnk-apps
labels:
app.kubernetes.io/managed-by: roksbnkctl-matrix
...
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: matrix-httproute
namespace: bnk-apps
spec:
parentRefs:
- name: bnk-gateway
namespace: bnk-apps
sectionName: http
rules:
- backendRefs:
- name: matrix-http-backend
port: 80
...
(Abridged; the full plan prints all ten cells of the example grid and the complete set of objects.)
--only '<glob>' narrows the plan (or a real run) to cells whose name matches a glob — --only 'L7*' for just the HTTP rows, handy when iterating on one family.
The real run and the report
Drop --dry-run against a complete deployment. The runner creates the fixtures, runs each cell from its jumphost, removes the fixtures (unless --keep), and prints the report. -o selects the shape:
-
-o text(default) — the grid on stderr. -
-o md— the grid on stdout. iperf3 rows show Gbit/s and retransmits; HTTP rows show req/s, Mbit/s, and mean latency:## matrix (pass, 184230ms) | Cell | Family | Status | Throughput | req/s | mean ms | notes | |---|---|---|---|---|---|---| | L4 512K VSI->TMM diff-zone | iperf3 | pass | 9.21 Gbit/s | — | — | 312 retransmits | | L7 http CPS 128B VSI->TMM | l7 | pass | 18.4 Mbit/s | 21340 | 2.34 | — |(Numbers illustrative.)
-
-o json— theroksbnkctl.v1schema: one result per cell, each with its family-specific fields (plus theclient/serverit used). Exit code is non-zero if any cell failed, so it drops into CI:{ "schema": "roksbnkctl.v1", "command": "test", "suite": "matrix", "cells": [ { "suite": "l7", "name": "L7 http CPS 128B VSI->TMM", "status": "pass", "extra": { "req_per_sec": 21340.0, "throughput_mbps": 18.4, "req_time_mean_ms": 2.34, "client": "ssh:jumphost", "server": "http://10.240.0.10/128" } } ], "overall": "pass" }
Flag summary
| Flag | Effect |
|---|---|
--file <path> | the matrix.yaml (default: <workspace>/matrix.yaml, then ./matrix.yaml) |
--only <glob> | run only cells whose name matches the glob |
--dry-run | print the plan and the objects it would create; no cluster calls |
--keep | leave the test’s objects in place after the run |
-o json|text|md | report shape |
Cross-references
- Chapter 22 — Throughput testing — the single iperf3 measurement the matrix’s
iperf3family builds on; start there for the server-pod and SCC mechanics. - Chapter 15 — SSH targets — the
jumphost/jumphost-<zone>targets the locality axis names. - Chapter 16 — The –on flag and SSH jumphosts — the per-AZ jumphost fleet (with iperf3 + h2load preinstalled).
- Chapter 17 — Execution backends — how the
ssh:<target>dispatch and the bundled tools images work. - Chapter 28 — Configuration reference §“matrix.yaml” — the schema as a reference table.
- Chapter 27 — Command reference §
roksbnkctl test matrix— the generated flag surface.
The E2E test plan
roksbnkctl ships a layered end-to-end test suite that exercises the full surface — install, lifecycle, four execution backends, internalised kubectl, the DNS probe, the cred-leak audit, and a mixed-mode lifecycle — against a live IBM Cloud account. This chapter is the user-facing guide: what the suite is, how to run it locally, what each phase validates, what it costs, and how it’s re-run when (not if) part of it flakes.
The design rationale lives in PRD 05; read that for the why. This chapter is the how and what.
What the E2E suite is
The suite is 14 automated phases organised into two tiers, plus Phase J as a manual integrator step (kubectl internalisation requires sudo mv of the host kubectl/oc binaries; that mutation is too disruptive to automate, so PRD 05 §J leaves it as a release-checklist item):
| Tier | Phases | What it covers | Driver script |
|---|---|---|---|
| Baseline | A, B, C, D, E, F, G, H | install, init, plan, up, post-apply checks, test suites, down | scripts/e2e-test.sh |
| Backends + extras | I, K, L, L-DNS, M, N | SSH backend, docker backend, k8s backend + ops pod, DNS probe with GSLB compare, cred-leak audit, mixed-mode lifecycle | scripts/e2e-test-backends.sh |
| Manual | J | kubectl internalisation (PATH-stripped, integrator-driven) | per-release checklist |
A combined driver, scripts/e2e-test-full.sh, runs both automated tiers in sequence: A-H first to bring up + exercise + tear down the baseline cluster, then I-N which provisions a fresh cluster via Phase N’s mixed-mode-lifecycle step. The two drivers stay decoupled — each can be run standalone — at the cost of an extra cluster apply (~70min wall-time, ~5-7h combined). Cluster-sharing across the two drivers (the PRD-envisioned design) is queued for v1.x; see PRD 05 §“Test infrastructure”.
Phase coverage at a glance
| Phase | Tier | Validates | PRD |
|---|---|---|---|
| A | baseline | doctor + init | — |
| B | baseline | plan (read-only) | — |
| C | baseline | targets list, registration | 01 |
| D | baseline | up lifecycle — provision + deploy BNK | — |
| E | baseline | post-apply checks (status, k get, logs) | 02 |
| F | baseline | test connectivity (HTTP probes) | — |
| G | baseline | test throughput (iperf3) | — |
| H | baseline | down — destroy + cleanup | — |
| I | backends | SSH backend / --on jumphost, host-key TOFU | 01 |
| J | manual | kubectl internalisation (PATH-stripped — requires sudo mv) | 02 |
| K | backends | docker backend (ibmcloud + iperf3 client) | 03 § Docker |
| L | backends | k8s backend (ops pod + ibmcloud + iperf3) | 03 § K8s |
| L-DNS | backends | DNS probe + GSLB cross-vantage compare | 03 § DNS |
| M | backends | cred-leak audit (docker inspect, k8s events, ssh tempfiles) | 04 |
| N | backends | mixed-mode lifecycle (each tool on a different backend) | all of the above |
Phase J is an integrator-driven manual step; the per-release checklist in docs/E2E_TEST.md covers its procedure (PATH-strip kubectl + oc, then re-run the baseline driver’s Phase E to confirm roksbnkctl k get/apply/describe/... still works against the cluster).
How to run it locally
The three driver scripts all live under scripts/:
# Baseline only (A-H) — ~90 minutes
./scripts/e2e-test.sh
# Backends + extras only (I-N + L-DNS) — requires a live cluster (run after
# Phase D of e2e-test.sh, or provision via `roksbnkctl up` in a separate
# workspace first)
./scripts/e2e-test-backends.sh
# Combined — A-H baseline, then I-N + L-DNS against a fresh cluster the
# backends driver brings up via Phase N's mixed-mode-lifecycle step,
# ~5-7 hours total (two separate cluster applies)
./scripts/e2e-test-full.sh
Pre-requisites
| Pre-req | Required for | Notes |
|---|---|---|
| IBM Cloud account with API key | every phase | IBMCLOUD_API_KEY env var, or roksbnkctl init writes a keychain entry |
terraform binary on PATH | phases B-H | the only strictly-required host tool for the baseline |
| Docker daemon | phase K | dockerd or colima or Rancher Desktop — anything that publishes a docker socket |
kind binary | phase L on CI | the in-CI k8s backend uses a kind cluster; on a real run it uses the ROKS cluster from D |
| An SSH bastion or jumphost | phase I, N | provisioned automatically by phase D’s terraform when testing_create_tgw_jumphost = true (the default) |
| Adequate disk for terraform plan output | phases B-H | ~200 MB for the embedded module’s state |
Everything else (kubectl, oc, dig, iperf3) is internalised by the binary — phase J explicitly verifies the suite passes with kubectl and oc moved out of PATH.
Resuming a partial run
Every phase is re-runnable. The driver scripts respect a PHASE_FROM= env var:
# Restart from phase G (skipping A-F, which already ran)
PHASE_FROM=G ./scripts/e2e-test.sh
# Same for the backend driver
PHASE_FROM=L ./scripts/e2e-test-backends.sh
The phase pointer is read at startup and the script fast-forwards past every step before it. Assertion phases that hit external APIs (DNS resolvers, IBM Cloud control plane) include jitter and retry on the typical transient failure shapes — short DNS timeouts, IAM 5xx blips, etc. See PRD 05 §“Risks” for the retry policy.
Run logs
Each driver writes a single combined log per run:
scripts/e2e-test.sh→/tmp/roksbnkctl-e2e/run-<timestamp>.logscripts/e2e-test-backends.sh→/tmp/roksbnkctl-e2e-backends/run-<timestamp>.logscripts/e2e-test-full.sh→ both of the above (the combined runner re-uses each child driver’s log directory)
Logs are preserved on both success and failure; clean them up manually when disk pressure warrants. On a CI machine that runs the suite nightly, the logs are the only forensics you get — keep them for at least 7 days. Per-phase log splitting is a v1.x consideration.
Dry-run
DRY_RUN=1 short-circuits every roksbnkctl invocation to a no-op that prints the command it would have run. Useful for re-validating the script wiring after edits without paying the 30-minute cluster-apply tax. The validator agent’s “is the test plan still well-formed” check runs in this mode.
What each phase validates
Phase A — init
roksbnkctl init prompts for region, resource group, cluster name, and BNK version, then writes ~/.roksbnkctl/<workspace>/config.yaml. The phase asserts the file exists, contains no plaintext API key (the rejection regex in internal/config/workspace.go catches that), and that roksbnkctl doctor reports green for terraform and informational for kubectl/oc.
Phase B — plan
roksbnkctl plan runs terraform init (downloads providers, ~30s) and terraform plan (computes the resource diff, ~30-60s on a clean workspace). No infrastructure is provisioned. The phase asserts the plan reports ~77 resources to add (the exact count is the upstream HCL’s full set of cluster + cert-manager + flo + cne_instance + license + testing resources).
Phase C — targets and registration
roksbnkctl targets list against a fresh workspace returns empty. roksbnkctl cluster register <existing-cluster> (optional, skipped if the workspace is provisioning new infra) ties an existing ROKS cluster’s COS instance + bucket discovery into the workspace.
Phase D — up lifecycle
The dominant cost phase. roksbnkctl up --auto runs terraform apply against the embedded HCL — provisioning ~77 resources: VPC, subnets, transit gateway, ROKS cluster, cert-manager, FLO, CNEInstance, License, jumphost. Expect 30-50 minutes on a clean apply, 5-15 minutes longer when IBM Cloud’s control plane is slow. The phase asserts terraform exits zero and the admin kubeconfig was fetched and written to $KUBECONFIG.
Post-apply, the phase auto-registers the jumphost target (per Chapter 16 §“Auto-discovery from roksbnkctl up”) so subsequent phases can --on jumphost without manual config.
Phase E — post-apply checks
roksbnkctl status shows the deployed BNK components. roksbnkctl k get nodes lists 3 worker nodes Ready. roksbnkctl logs flo (the F5 Lifecycle Operator) prints recent log lines. The phase asserts each command exits zero and that the cluster’s BNK install (FLO + CNE Instance + License) is in a healthy state.
Phase F — test connectivity
roksbnkctl test connectivity walks the workspace’s test.connectivity.extra_hosts list and probes each URL. Pass criteria: every URL returned a 2xx (or the expected status, when Chapter 20’s richer assertion shape lands).
Phase G — test throughput
roksbnkctl test throughput deploys the iperf3 server-pod fixture, runs a 30-second client measurement, and tears the fixture down. Pass criteria: bandwidth > 100 Mbps (a conservative floor — actual numbers on IBM Cloud are typically 1-5 Gbps), retransmits < 5% of streams, fixture removed afterwards.
Phase H — down
roksbnkctl down --auto runs terraform destroy. Pass criteria: all ~77 resources destroyed (terraform reports Destroy complete!), no orphan IBM Cloud resources detectable via roksbnkctl ibmcloud resource search.
Phase I — SSH backend / --on jumphost
roksbnkctl exec --on jumphost -- whoami returns root (the jumphost auto-provisioned with cloud-init’s root user). roksbnkctl ibmcloud --on jumphost iam oauth-tokens validates IBMCLOUD_API_KEY propagation over SSH. A negative test mutates ~/.roksbnkctl/known_hosts to a wrong fingerprint and asserts the next call exits 126 with a clear “host key mismatch” error. Phase I is the user-facing acceptance test for PRD 01.
Phase J — kubectl internalisation
The phase strips kubectl and oc out of PATH (via env-var sanitisation, not filesystem moves — see PRD 05 §“Open questions”) and verifies that roksbnkctl k get nodes, roksbnkctl k apply -f, roksbnkctl k describe, roksbnkctl k exec, roksbnkctl k port-forward, and roksbnkctl k delete all work against the cluster. A supplementary byte-equivalence step (run separately, not gated on PATH stripping) diffs kubectl get nodes -o yaml against roksbnkctl k get nodes -o yaml and asserts the diff (excluding managedFields, resourceVersion, creationTimestamp) is empty.
Phase K — docker backend
roksbnkctl ibmcloud --backend docker iam oauth-tokens pulls the roksbnkctl-tools-ibmcloud image on first call and runs the ibmcloud CLI inside it. The phase asserts the API key is not baked into the image (via docker history inspection) and not exposed in the running container’s env (via docker inspect). Chapter 17 §“docker backend” covers the credential-passing mechanism.
Phase L — k8s backend and ops pod
roksbnkctl ops install creates the roksbnkctl-ops namespace, deploys the long-lived ops pod, projects the IBM API key as a Kubernetes Secret, and binds the pod’s ServiceAccount to a least-privilege ClusterRole. Subsequent steps run roksbnkctl ibmcloud --backend k8s iam oauth-tokens (executes inside the ops pod) and roksbnkctl test throughput --backend k8s (the iperf3 server pod and client Job both run in-cluster). RBAC assertions confirm the SA can create Jobs in roksbnkctl-test but cannot delete Pods in default — least-privilege is enforced. Chapter 19 is the ops-pod reference.
Phase L-DNS — DNS probe and GSLB compare
The DNS phase exercises the miekg/dns-backed probe:
- Single-vantage A and AAAA lookups against
8.8.8.8 - NXDOMAIN negative test (asserts rcode=
NXDOMAIN) - Iterated probe (10 queries to the same server, RTT p50/p95/p99 reported)
- K8s-backend probe (runs as a Job in
roksbnkctl-test, the binary self-execs in-cluster, RTT reflects in-cluster network path) --server cluster(uses the pod’s/etc/resolv.conf, validates CoreDNS visibility)--gslb-comparehappy path (fans out local + k8s, asserts the answer schema)--gslb-comparedivergence (target a geo-resolved name where laptop and cluster IPs hit different DCs, assertsgslb_divergence: true)- Docker rejection negative (asserts the parse-time rejection error for
--backend docker --target ...)
LD9 (SSH vantage) is exercised only when a jumphost is configured; LD5-LD8 are the must-pass set.
Phase M — cred-leak audit
Cross-cutting check that runs after I-L — confirms no credential value leaked during any prior phase. Concrete assertions:
docker history <ibmcloud-tool-image>— noIBMCLOUD_API_KEY=...ENV layerdocker inspect <last-container>— no API key value in envkubectl get events -n roksbnkctl-ops -o yaml— grepping for the API key value returns nothingkubectl logs <ops-pod>— grepping for the API key value returns nothing (the redactor masks any tool output that prints it)ssh jumphost ls /tmp/roksbnkctl.*— empty (the SSH backend’s trap cleans up tempfiles on exit)sshd auth.log—Accepted publickeylines present; theSetEnvvar name (IBMCLOUD_API_KEY) is logged but not the value~/.roksbnkctl/*/state/*.loghost-side logs — no API key value
The audit is the single most important gate on the v1.0 release. A leak in any of M1-M7 is a stop-ship. See PRD 04 for the threat model.
Phase N — mixed-mode lifecycle
A realistic scenario: workspace config routes each tool to its preferred backend, then a full up + test + down cycle runs end-to-end. Concretely, exec.terraform=local, exec.ibmcloud=ssh:jumphost, exec.iperf3=k8s — three different backends in one lifecycle. The phase asserts state is preserved across the per-tool dispatch (the workspace’s terraform state file is touched only by the local-backend terraform; the API key projected into the k8s ops pod is the same one resolved for the SSH dispatch) and that down cleanly destroys everything.
How CI runs it
.github/workflows/ci.yml runs unit + integration on every PR — go test ./... plus the testcontainers-go-backed integration tests. The full e2e suite is too expensive (4-6 hours, $5-10 of IBM Cloud spend per run) to gate on every PR.
The full e2e suite is run manually, out of band — on demand against main — rather than gated on every PR. It provisions real IBM Cloud infrastructure, so it is launched deliberately (with IBMCLOUD_API_KEY set) and its log artefacts archived against the run.
The release-cut policy is: don’t tag vX.Y.Z until the most recent manual-trigger run on main is green for three consecutive nights. (This project is trunk-based — there are no release branches. A per-BNK-release branch model was tried and withdrawn; what replaced it is the per-line terraform overlay in Chapter 13.) This catches the flakes that don’t reproduce locally — most of which are IBM Cloud control-plane blips rather than real regressions.
Cost and time
A full scripts/e2e-test-full.sh run currently costs:
| Resource | Approximate cost (USD) |
|---|---|
| ROKS cluster (3 workers, ~5 hours uptime) | $3-6 |
| 1-2 LoadBalancer Service objects (for north-south throughput) | $0.50-1 |
| COS instance + objects for the supply chain | $0.10-0.20 |
| Egress bandwidth (throughput tests, image pulls) | $0.20-0.50 |
| Total per run | $5-10 |
Per-phase time estimates:
| Phase | Wall time |
|---|---|
| A (init) | <1 minute |
| B (plan) | 1-2 minutes |
| C (targets / register) | <1 minute |
| D (up) | 30-50 minutes (the dominant cost) |
| E (post-apply checks) | 1-2 minutes |
| F (connectivity) | 1-2 minutes |
| G (throughput) | 1-3 minutes |
| H (down) | 15-25 minutes |
| I (SSH backend) | 2-3 minutes |
| J (kubectl internal) | 3-5 minutes |
| K (docker backend) | 3-5 minutes (first call pulls the image, +1-2 minutes) |
| L (k8s backend + ops pod) | 3-5 minutes |
| L-DNS (DNS probe + GSLB compare) | 2-4 minutes |
| M (cred audit) | <1 minute |
| N (mixed-mode lifecycle) | 30-50 minutes (full up + down again) |
| Total | ~4-6 hours |
Phase N is the second-dominant cost — it runs a complete up/down cycle on top of D’s. Contributors who want a shorter test loop should skip N (PHASE_FROM= past it) and rely on D + I-M coverage; full N is a release-gate concern, not a per-PR concern.
Re-runnability
Every phase is re-runnable via PHASE_FROM=. The driver scripts are idempotent in two senses:
- Phase ordering: phases later than
PHASE_FROM=<X>run unconditionally; phases at or beforeXare skipped. The script doesn’t try to remember whether earlier phases succeeded — that’s the user’s job (the per-phase log files are the evidence). - Per-phase actions: each phase’s individual
stepcalls are themselves idempotent where possible.roksbnkctl upon an already-applied workspace is a no-op (terraform plan reports zero changes).roksbnkctl ops installon an already-installed cluster is a no-op (the namespace + RBAC exist). The redactor + cred-resolver short-circuit cleanly on repeated invocations.
External-API steps include jitter+retry per PRD 05 §“Risks”: DNS resolvers occasionally return SERVFAIL on first query and succeed on the second, IBM IAM occasionally 5xxs during high-load periods, and the in-cluster ops pod can take a few seconds to be Running after ops install returns. Each of these is retried with a short exponential-backoff jitter rather than failing the phase.
The intended workflow on a flake is:
# Phase L flaked on "ops pod not yet Running"
# Re-run from L; everything before is preserved
PHASE_FROM=L ./scripts/e2e-test-backends.sh
If the same phase flakes on consecutive PHASE_FROM= runs, it’s a real bug — open an issue with the per-phase log attached.
Cross-references
- Chapter 26 — Troubleshooting — symptom → root cause → fix entries for the failure modes phase D through phase N can surface.
- Chapter 17 — Execution backends — the four-backend matrix that phases I, K, L exercise.
- Chapter 19 — The in-cluster ops pod — what phase L installs and what RBAC it carries.
- Chapter 21 — DNS testing for GSLB — the probe behaviour phase L-DNS exercises.
- Chapter 22 — Throughput testing — the iperf3 fixture phase G uses.
- PRD 05 — the design spec for the suite.
Day-2 ops: status, logs, k get/apply/exec
This is the chapter to read after the cluster is up and BNK is deployed and you’re now living with the result. It opens with roksbnkctl status — the workspace-level read of what’s deployed — then covers the per-resource verbs: read pod state, tail logs, apply a manifest, port-forward to a service, exec into a pod. These per-resource verbs are native Go via client-go, so you don’t need kubectl on PATH for the everyday workflow.
The full design rationale lives in PRD 02. This chapter is the user-facing surface — the canonical “what’s the kubectl-equivalent in roksbnkctl?” reference.
Why internalise
Three reasons, in order of weight:
- Single binary.
roksbnkctlis meant to be the one thing you install. The only required external prerequisite for the happy path isterraform. Everything else —kubectl,oc,iperf3,dig— is either built-in or an optional escape hatch. - No version skew. The vendored
client-gomatches the kube API the bundled HCL targets. You can’t accidentally usekubectl1.20 against a 1.28 cluster and have its print column heuristics go sideways. - First-class output formats.
cli-runtimegives byte-identical-o yaml/-o json/-o jsonpathoutput tokubectl. The validator agent’s golden-file tests ininternal/k8s/golden_test.goassert this for representative resources.
roksbnkctl status
The one-shot read of a workspace’s posture. Always best-effort — every section reports its own missing pieces so a partial state still produces useful output rather than a hard error.
$ roksbnkctl status
Workspace: canada-roks
Region: ca-tor
Resource group: default
Cluster: canada-roks (attach existing)
TF source: jgruberf5/ibmcloud_terraform_bigip_next_for_kubernetes_2_3@v1.3.0
Cluster phase: deployed (last apply 2026-05-13 14:08:33 MST)
BNK trial: deployed (last apply 2026-05-13 14:15:01 MST)
Kubeconfig: /home/you/.kube/config
Cluster: 2/2 nodes ready
The header rows (workspace, region, resource group, cluster identity, TF source pin, kubeconfig path, cluster reachability) are the same across every workspace shape. The per-phase deployment lines below them are shape-dependent: roksbnkctl status reads each phase’s terraform.tfstate mtime independently and emits one line per phase. The mapping from workspace shape to per-phase output:
The two Cluster: lines are by design: the first (in the header block) reports cluster identity — which cluster you’re targeting and whether the workspace creates it or attaches to an existing one. The second (the trailer line) reports cluster reachability — node count and ready count from a live API call. The label is reused because both pieces of information are about “the cluster”; the column to the right disambiguates.
TF source: reflects the workspace’s tf_source.type: github renders as <Repo>@<Ref> (the canonical happy-path shape — e.g., jgruberf5/ibmcloud_terraform_bigip_next_for_kubernetes_2_3@v1.3.0); local renders as local:<Path>; embedded or unset renders as (unset). The samples below use the github shape since it’s what most readers will see.
ShapeEmpty — fresh workspace, neither phase deployed
$ roksbnkctl status
Workspace: dev
Region: us-south
Resource group: default
Cluster: (unset) (attach existing)
TF source: jgruberf5/ibmcloud_terraform_bigip_next_for_kubernetes_2_3@v1.3.0
Cluster phase: not deployed
BNK trial: not deployed
Kubeconfig: (none — run `roksbnkctl kubeconfig --download`)
Both phases report not deployed — the state directories either don’t exist or hold zero-resource state files. Running roksbnkctl cluster up (or roksbnkctl up for the monolithic path) advances the workspace to ShapeClusterOnly.
ShapeClusterOnly — cluster phase deployed, no BNK trial yet
$ roksbnkctl status
Workspace: canada-roks
Region: ca-tor
Resource group: default
Cluster: canada-roks (attach existing)
TF source: jgruberf5/ibmcloud_terraform_bigip_next_for_kubernetes_2_3@v1.3.0
Cluster phase: deployed (last apply 2026-05-13 14:08:33 MST)
BNK trial: not deployed
Kubeconfig: /home/you/.kube/config
Cluster: 2/2 nodes ready
The Cluster phase line reads the mtime of <state-cluster-dir>/terraform.tfstate; the BNK trial line reads <state-dir>/terraform.tfstate and falls back to not deployed when the trial state is empty or missing. Running roksbnkctl bnk up advances the workspace to ShapeSplit.
ShapeSplit — both phases deployed (the steady state)
$ roksbnkctl status
Workspace: canada-roks
Region: ca-tor
Resource group: default
Cluster: canada-roks (attach existing)
TF source: jgruberf5/ibmcloud_terraform_bigip_next_for_kubernetes_2_3@v1.3.0
Cluster phase: deployed (last apply 2026-05-13 14:08:33 MST)
BNK trial: deployed (last apply 2026-05-13 14:15:01 MST)
Kubeconfig: /home/you/.kube/config
Cluster: 2/2 nodes ready
Each phase has its own mtime; the timestamps move independently. Re-running roksbnkctl bnk down then roksbnkctl bnk up updates the BNK trial line without touching the Cluster phase line — useful for confirming which phase you most recently exercised.
The shape detection logic lives in internal/config/tfstate.go::DetectShape; the per-phase emission in runStatus is in internal/cli/inspect.go. See PRD 06 §“status command integration” for the design rationale.
The k command tree
All internalised verbs live under roksbnkctl k:
roksbnkctl k get # fetch resources
roksbnkctl k describe # human-readable detail
roksbnkctl k apply # server-side apply from file/dir
roksbnkctl k delete # delete with cascade options
roksbnkctl k logs # pod or component logs
roksbnkctl k exec # exec into a pod (SPDY)
roksbnkctl k port-forward # forward local ports to a pod (SPDY)
Two of those have top-level shortcuts for muscle-memory convenience — the verbs you’d type a hundred times a day:
roksbnkctl get ↔ roksbnkctl k get
roksbnkctl logs ↔ roksbnkctl k logs
apply, exec, delete, describe, and port-forward only work under the k prefix.
Two verbs are deliberately not aliased to avoid shadowing existing top-level commands:
roksbnkctl applyis the existing top-level lifecycle verb that runsterraform applyagainst the workspace. Adding a secondapplywould shadow it and breakroksbnkctl up/roksbnkctl applymuscle memory. Useroksbnkctl k apply -f ...explicitly for the Kubernetes-side server-side apply.roksbnkctl execruns a command on the host with the workspace’s env loaded (the host-exec verb — see Chapter 16, specifically the “Working examples” section).roksbnkctl k execruns in a pod. The split keeps both meanings unambiguous without surprising name-collision behaviour.
kubectl/oc passthroughs stay as escape hatches
The roksbnkctl kubectl <args...> and roksbnkctl oc <args...> passthroughs are preserved alongside the internalised verbs. They shell out to the host binary (with the workspace’s KUBECONFIG and credentials loaded) for anything outside the internalised subset.
-wselects the cluster here too. The passthroughs resolve the same workspace-scoped kubeconfig and context thekverbs do, so-w acme-eu kubectl get nodeslistsacme-eu’s nodes. This was not always true: before v1.42.0 they took the ambientKUBECONFIGand then fell back to~/.roksbnkctl/forge/kubeconfig.yaml— a single file every workspace shares — so two workspaces on different clusters silently retargeted each other (#55). The output looked entirely believable, which is what made it dangerous fordelete.Two deliberate limits. An explicit
--contextor--kubeconfigin your arguments always wins — being told which cluster you mean is not something the tool overrides. And a workspace with no known cluster keeps the historical ambient behaviour rather than failing. If the workspace does know its cluster but no kubeconfig on disk can authenticate to it, you get the same error thekverbs give, telling you to runkubeconfig --download— rather than a silent fall-back to whatever the shared file points at.
roksbnkctl shellandroksbnkctl execpin the same kubeconfig. Neither can pin a context, though — they hand you an arbitrary command line, andkubectlreads no environment variable for the context — so when the pinned file’scurrent-contextnames a different cluster they say so on stderr instead of letting the next command address it.Across the
--onSSH boundary nothing is pinned: a kubeconfig path and a context name are local facts that mean nothing on the remote host, which is exactly why that boundary scrubs them (Chapter 16).
When to reach for the passthrough:
| Use case | Why passthrough |
|---|---|
kubectl rollout (status/history/undo/restart) | Out of scope for v1.0; PRD 02 explicitly defers |
kubectl scale / kubectl autoscale | Out of scope; passthrough is fine |
kubectl edit / kubectl patch | Low frequency for BNK ops; out of scope for v1.0 |
kubectl auth can-i / RBAC introspection | Out of scope; passthrough is fine |
kubectl drain / cordon / taint | Cluster admin operations; not roksbnkctl’s role |
kubectl run / kubectl create | Imperative resource creation; use k apply -f instead |
oc adm / oc image / OpenShift admin verbs | Niche enough to defer; passthrough |
| Niche flag combos | Anything not in the internalised verb’s flag set |
If kubectl is missing from PATH, the passthrough errors with:
Error: kubectl not on PATH; use `roksbnkctl k get/apply/...` for the in-process path,
or install kubectl
Same for oc. The doctor check treats both as informational rather than warnings — see Chapter 5 — Doctor.
Worked examples
The verbs in everyday order. Every example below assumes roksbnkctl k and accepts the top-level alias where one exists.
roksbnkctl k get
The most-used verb. Resource type + optional name + optional flags:
# All pods in the default namespace
roksbnkctl k get pods
# Pods in a specific namespace
roksbnkctl k get pods -n f5-bnk
# Pods across all namespaces
roksbnkctl k get pods -A
# A specific pod by name
roksbnkctl k get pod flo-controller-abc123 -n f5-bnk
# Label selector
roksbnkctl k get pods -A -l app.kubernetes.io/name=f5-lifecycle-operator
# Cluster-scoped resources (no namespace)
roksbnkctl k get nodes
roksbnkctl k get storageclasses
Output formats — these match kubectl byte-for-byte:
roksbnkctl k get pods -n f5-bnk -o yaml
roksbnkctl k get pods -n f5-bnk -o json
roksbnkctl k get pods -n f5-bnk -o wide
roksbnkctl k get pods -n f5-bnk -o name
roksbnkctl k get pods -n f5-bnk -o jsonpath='{.items[*].metadata.name}'
roksbnkctl k get pods -n f5-bnk -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'
Plural / singular / shortname handling comes from the cluster’s RESTMapper via the discovery client, so pod, pods, po all work and pick up CRDs without a hardcoded list. roksbnkctl k get cneinstances (a BNK CRD) works as soon as the CRD is registered with the API server — no rebuild required.
Using the top-level alias:
roksbnkctl get pods -A
roksbnkctl k describe
Delegates to k8s.io/kubectl/pkg/describe — the same library kubectl uses internally. Output is identical to kubectl describe:
roksbnkctl k describe pod flo-controller-abc123 -n f5-bnk
roksbnkctl k describe node 10.243.0.4
roksbnkctl k describe service flo-webhook -n f5-bnk
roksbnkctl k describe cneinstance my-instance -n f5-bnk
The describe output’s “Events” section is especially useful for debugging stuck resources — pod scheduling failures, image pull errors, finaliser hangs all surface here.
roksbnkctl k apply
Server-side apply (SSA) with field-manager roksbnkctl. Inputs:
# Single file
roksbnkctl k apply -f pod.yaml
# Directory of YAMLs (recurses *.yaml)
roksbnkctl k apply -f manifests/
# Kustomize base (auto-detected if kustomization.yaml is present)
roksbnkctl k apply -f my-kustomize-base/
# stdin
cat pod.yaml | roksbnkctl k apply -f -
# Apply into a specific namespace (overrides metadata.namespace if absent)
roksbnkctl k apply -f manifests/ -n f5-bnk
# Force conflicts (SSA force-conflicts=true)
roksbnkctl k apply -f manifests/ --force
There is no top-level roksbnkctl apply alias for this verb — roksbnkctl apply is the lifecycle command that runs terraform apply. Always use roksbnkctl k apply for the Kubernetes-side apply.
Differences from kubectl apply:
- Always SSA. Field manager is
roksbnkctl. Client-side apply is not supported. - Kustomize auto-detect. A directory containing
kustomization.yamlis built viasigs.k8s.io/kustomize/apibefore applying — no-kflag needed. --forcemaps to SSA’sforce-conflicts=true. Without it, conflicts with another field manager produce a clean error rather than silently winning.
For a vanilla kubectl apply -f workflow, the behaviour is functionally identical. For workflows that depend on client-side three-way merge or specific --server-side flag combinations, fall back to the passthrough.
roksbnkctl k delete
Cascade-aware deletion via the dynamic client:
# Delete by name
roksbnkctl k delete pod flo-controller-abc123 -n f5-bnk
# Cascade: orphan, background (default), foreground
roksbnkctl k delete deployment flo -n f5-bnk --cascade=foreground
# Force (bypass graceful deletion; immediate)
roksbnkctl k delete pod stuck-pod -n f5-bnk --force
# Custom grace period (seconds)
roksbnkctl k delete pod my-pod -n f5-bnk --grace-period=5
Use --cascade=foreground when you want to wait for owned resources (Pods owned by a Deployment, etc.) to be deleted before the parent disappears — useful for tearing down BNK trial CRs cleanly so finalisers run in order.
roksbnkctl k logs and roksbnkctl logs
Two paths, one verb. The component-aware path is a BNK-specific label-selector shortcut; the raw pod-name path takes a pod name directly.
Component-aware (by label selector):
roksbnkctl logs flo # F5 Lifecycle Operator (label selector under the hood)
roksbnkctl logs cis # F5 BNK CIS controller
roksbnkctl logs cert-manager # cert-manager
roksbnkctl logs cneinstance # BIG-IP TMM data plane pods
Raw pod-name (by pod name):
roksbnkctl k logs flo-controller-abc123 -n f5-bnk
Common flags (both paths):
-f, --follow # stream live (kubectl logs -f)
-c, --container <name> # specific container in a multi-container pod
--previous # logs from the previous instance (after a crash)
--since=10m # only logs in the last 10 minutes
--tail=100 # last N lines only
Top-level alias:
roksbnkctl logs flo -f --since=5m
If the named first arg matches one of the well-known BNK components (flo, cis, cert-manager, cneinstance), the component-aware path is used; otherwise it’s treated as a pod name. The component map lives in internal/cli/inspect.go and is keyed off the upstream chart’s default labels.
roksbnkctl k exec
SPDY exec into a pod. Same semantics as kubectl exec:
# One-shot command
roksbnkctl k exec flo-controller-abc123 -n f5-bnk -- ls -la /
# stdin attached
roksbnkctl k exec flo-controller-abc123 -n f5-bnk -i -- cat /etc/hostname
# Interactive PTY (the bash-style use)
roksbnkctl k exec flo-controller-abc123 -n f5-bnk -i -t -- bash
# Specific container in a multi-container pod
roksbnkctl k exec flo-controller-abc123 -n f5-bnk -c sidecar -- env
The -i and -t flags map directly to kubectl exec’s -i (stdin) and -t (PTY). For top / bash / interactive Python sessions, pass both.
There is no roksbnkctl exec (top-level) alias — roksbnkctl exec runs on the host. See “Disambiguating roksbnkctl exec” in PRD 02.
roksbnkctl k port-forward
SPDY port-forward to a pod:
# Forward localhost:8080 → pod's :80
roksbnkctl k port-forward flo-controller-abc123 -n f5-bnk 8080:80
# Multiple ports
roksbnkctl k port-forward flo-controller-abc123 -n f5-bnk 8080:80 8443:443
# Random local port (let the kernel pick)
roksbnkctl k port-forward flo-controller-abc123 -n f5-bnk :80
Ctrl+C closes the tunnel cleanly — no orphaned local listeners. The forward survives idle (reads/writes are bidirectional); it’s torn down only on signal or pod restart.
For a Service rather than a Pod, port-forward via the Service’s underlying pod or use kubectl port-forward svc/<name> through the passthrough — Service-targeted port-forwarding is currently passthrough-only.
Output format compatibility
The biggest user-visible promise: -o yaml / -o json / -o wide / -o jsonpath produce the same bytes as kubectl, modulo a small set of timestamp-and-resourceVersion fields that change between calls.
Concretely, the validator agent’s golden-file tests at internal/k8s/golden_test.go capture kubectl get <resource> -o yaml and roksbnkctl k get <resource> -o yaml against a live ROKS cluster and diff them, ignoring:
metadata.managedFields(ordering varies between callers; not user-visible)metadata.resourceVersion(monotonic counter; changes on every read)metadata.creationTimestamp(set server-side; not under our control)
Anything else differing is a test failure. The covered resources are Node, Pod, Service, ConfigMap — representative both of cluster-scoped (Node) and namespace-scoped (Pod, Service, ConfigMap), and of the typed-client (Node, Pod, Service) and dynamic-client (anything via cli-runtime’s resource.Builder) paths.
Run them locally with:
make test-live
…against a KUBECONFIG that points at a real ROKS cluster. They’re not part of the unit-test CI run because they need a live cluster; the integrator runs them before tagging a release. Documented in CONTRIBUTING.md.
OpenShift extensions
Beyond the core kubectl-equivalent verbs, ROKS clusters surface OpenShift-specific resource types — Project, Route, ImageStream, BuildConfig. roksbnkctl k get discovers these natively today via the dynamic client + RESTMapper path (the cluster advertises them through the API discovery doc; the deferred-discovery mapper picks them up):
roksbnkctl k get projects # OpenShift projects (vs Kubernetes namespaces)
roksbnkctl k get routes -n f5-bnk # OpenShift Routes (vs Ingress)
roksbnkctl k get imagestreams -n f5-bnk # OpenShift ImageStreams
roksbnkctl k get buildconfigs # BuildConfigs (mostly empty in BNK trials)
Same verb shape (get / describe / delete); the dynamic-client + RESTMapper combination handles type discovery without needing a per-type Go-side scheme registration.
Phase 2.1 of PRD 02 adds typed clients via github.com/openshift/client-go for nicer printing and describe integration of these resources. This is on the v1.x roadmap (see docs/PLAN.md §“What’s deliberately deferred to post-v1.0”). Until typed clients land, roksbnkctl k get/describe still works against OpenShift CRDs — just with the generic unstructured printer. If you want richer per-type output today, fall back to the oc passthrough:
roksbnkctl oc get projects # typed-client output today
roksbnkctl oc describe route f5-bnk-svc # typed Route fields
How doctor treats kubectl / oc
How roksbnkctl doctor reports the host binaries (covered in Chapter 5):
kubectl— informational (no warning when missing).oc— also informational (no warning when missing).
A fresh dev box without kubectl / oc installed should run roksbnkctl doctor and see green-or-informational across the board for the everyday workflow. The host binaries aren’t required for the everyday workflow; they’re nice-to-have for the passthroughs.
kubectl muscle-memory cheat sheet
A reader migrating from kubectl should be able to use this section as a Rosetta Stone:
kubectl ... | roksbnkctl ... |
|---|---|
kubectl get pods | roksbnkctl get pods (or roksbnkctl k get pods) |
kubectl get pods -A | roksbnkctl get pods -A |
kubectl get pods -o yaml | roksbnkctl get pods -o yaml |
kubectl describe pod <name> | roksbnkctl k describe pod <name> |
kubectl apply -f manifests/ | roksbnkctl k apply -f manifests/ |
kubectl apply -k overlay/ | roksbnkctl k apply -f overlay/ (auto-detects kustomize) |
kubectl delete pod <name> | roksbnkctl k delete pod <name> |
kubectl logs <pod> -f | roksbnkctl logs <pod> -f (or roksbnkctl k logs <pod> -f) |
kubectl exec -it <pod> -- bash | roksbnkctl k exec <pod> -i -t -- bash |
kubectl port-forward <pod> 8080:80 | roksbnkctl k port-forward <pod> 8080:80 |
kubectl rollout status deploy/foo | roksbnkctl kubectl rollout status deploy/foo (passthrough) |
kubectl edit deployment foo | roksbnkctl kubectl edit deployment foo (passthrough) |
kubectl scale deployment foo --replicas=3 | roksbnkctl kubectl scale deployment foo --replicas=3 (passthrough) |
oc projects | roksbnkctl k get projects (works today via dynamic-client) or roksbnkctl oc projects for typed-client output |
The general pattern: if it’s get / describe / apply / delete / logs / exec / port-forward against a typed or unstructured Kubernetes resource, the internalised verb is the right answer. Anything else, fall back to the passthrough.
Cross-references
- Chapter 5 — Doctor — the kubectl/oc downgrade in context.
- Chapter 6 — Workspaces — the
KUBECONFIGresolution chain that powers everyk <verb>. - Chapter 8 — Cluster/trial phase split — the workspace-shape concept that drives the per-phase
statuslines. - Chapter 10 — Deploying BNK trials on top — the verb that advances
ShapeClusterOnly→ShapeSplit(and back viabnk down). - Chapter 11 — Tearing down — independent teardown of the cluster and trial phases.
- Chapter 16 — The
--onflag —--onplus the passthroughs for customer-firewalled scenarios. - PRD 02 — the design rationale and acceptance criteria for the work in this chapter.
- PRD 06 §“
statuscommand integration” — the design rationale for the per-phasestatusoutput.
Registering the cluster with BNK Forge
BNK Forge is a separate platform — a FastAPI/React
app — for operating BIG-IP Next for Kubernetes across a fleet of clusters. When you run BNK Forge co-located with your ROKS deployments,
roksbnkctl can hand each cluster it provisions straight to BNK Forge, so the
cluster shows up in the Forge fleet the moment cluster up finishes.
This is opt-in and best-effort: it never blocks or fails a deploy. If you don’t enable it (the default), nothing changes.
Quick start
Three commands cover everything. None of them edits config.yaml by hand —
enable writes the opt-in for you, exactly like registry target or targets add elsewhere in the tool:
roksbnkctl -w acme-eu bnkforge enable # auto-register on every `cluster up`
roksbnkctl -w acme-eu bnkforge register # register the current cluster right now
roksbnkctl -w acme-eu bnkforge status # show config + readiness (url, session, cluster id)
enable turns on the post-cluster up hook; register is the on-demand path
(no re-up needed) for a cluster that already exists. Both talk to BNK Forge v3’s
REST API directly — there is no external CLI to install; see
Prerequisites.
Why credential-backed
The naive way to register a cluster with an external platform is to hand it a kubeconfig. ROKS admin kubeconfigs are short-lived — they carry an embedded token that expires — so a stored kubeconfig goes stale and the platform loses access until someone re-uploads a fresh one.
roksbnkctl registers credential-backed instead. It points BNK Forge at an
IBM Cloud credential template (your IBM Cloud API key, stored once in Forge)
plus the cluster’s id and region. BNK Forge then re-derives the kubeconfig on
demand from that credential whenever it needs cluster access. Nothing
perishable is stored, so the registration doesn’t go stale.
roksbnkctl BNK Forge
----------- ---------
cluster id + region ───────▶ credential template (IBM Cloud API key)
│
▼
derive kubeconfig on demand (never stale)
Prerequisites
-
A co-located BNK Forge install that supports IBM ROKS credential-backed registration — i.e. a Forge that can re-derive a ROKS kubeconfig from an IBM Cloud credential template. (This backend support is being added on a BNK Forge branch; an older Forge that only accepts a static kubeconfig won’t work with this flow.)
-
The Forge URL and credentials.
BNK_FORGE_URL,BNK_FORGE_USERandBNK_FORGE_PASSWORD(orbnkforge.url/--url/--username). The session token that results is cached in the OS keychain and reused until it expires, so the password is normally used once.roksbnkctl bnkforge statusshows what is resolved.The password can also be stored as
bnkforge.password_b64, base64-encoded, the same shape as every other machine credential this tool takes (ibmcloud.api_key_b64,bnk.cis.bigip_password_b64,registry.generic_password_b64). Precedence isBNK_FORGE_PASSWORD, then--password, then the field, then the prompt — so a stored value never prevents a runner overriding it.base64 is obfuscation, not encryption. Anyone holding the file holds the password. Storing it trades a long-lived secret on disk for not being asked, which is the right trade for an unattended runner and the wrong one for a laptop. Use
scripts/generate_b64_password.shto produce the value without it reaching your shell history:$ ./scripts/generate_b64_password.sh bnkforge.password_b64 Password: Re-enter: # config.yaml — under the bnkforge block password_b64: aHVudGVyMg==On a terminal,
bnkforge registerprompts for anything missing. The automatic post-uphook runs non-interactively: with no cached token and no env credentials it declines with a one-line note rather than blocking a deploy on a prompt. -
An IBM Cloud API key resolvable for the workspace (the usual env / keychain / config chain — see Chapter 14). It becomes the credential template Forge derives kubeconfigs from.
No external CLI. Nothing is looked up on
PATH, and there is no~/.bnk-forge/config.json. Earlier roksbnkctl versions shelled out to a v1bnk-forgebinary; v3 has no such CLI, soroksbnkctlspeaks to the REST API itself.
The bnkforge commands
| Command | What it does |
|---|---|
bnkforge enable [--url U] [--project P] | Persist register: true (+ optional overrides) so every cluster up registers the cluster. |
bnkforge disable | Persist register: false — turn the auto-hook off. Local only: it does not unregister anything. |
bnkforge status | Show the effective config (register / url / project), whether a Forge session token is cached, and whether a cluster id is recorded. |
bnkforge register [--url U] [--project P] [--force] | Register the current workspace’s cluster now, regardless of the opt-in. Surfaces errors (the auto-hook swallows them). --force takes over a cluster held by another Forge project — see Registration is non-destructive. |
bnkforge unregister [--url U] [--project P] | Remove the cluster from its Forge project — the inverse of register. Every “not there” case exits 0, so it is safe on a teardown path. |
bnkforge ssh-credential --host H --key K | Give Forge the SSH private key for an appliance this workspace built (the FLP), and turn on the project’s infrastructure access. See Giving Forge an appliance’s SSH key. |
enable, disable, and register all accept --url (override the Forge URL) and
--project (target Forge project). On enable those are persisted; on
register they’re one-off overrides that aren’t written to config.yaml.
What enable writes
enable writes a bnkforge: block into the workspace config.yaml for you (the
same marshaller every other roksbnkctl command uses — no hand-edit):
bnkforge:
register: true
url: https://forge.example.com # only if you passed --url
project: "42" # only if you passed --project
| Field | Type | Default | Set by |
|---|---|---|---|
register | bool | false | bnkforge enable / disable. false / omitted ⇒ no-op. |
url | string | (BNK_FORGE_URL) | bnkforge enable --url. Falls back to the BNK_FORGE_URL env var. |
project | string | (the workspace name) | bnkforge enable --project. Empty ⇒ a Forge project named after the workspace, created if absent. |
The block is additive — an absent bnkforge: block (any existing config.yaml)
loads unchanged.
What happens on cluster up
Forge kubeconfig for module-based registration. Independently of the credential-backed flow below, every
cluster upalso writes a portable, self-contained cert-based kubeconfig to$ROKSBNKCTL_HOME/forge/kubeconfig.yaml(one cluster entry with the public server + CA-if-any, and the cluster’s admin client certificate/key). ROKS is OpenShift: its API server authenticates via client certs or OAuth tokens and rejects raw IBM IAM bearer tokens (401), so the forge kubeconfig carries the admin client cert rather than a token. A BNK Forge that registers from a declared kubeconfig file can point at this path; roksbnkctl keeps it current automatically by re-fetching the admin kubeconfig as the certs near expiry (seekubeconfig --refresh). This is separate from, and complementary to, the credential-backedbnkforge registerflow documented below.
When register: true, roksbnkctl runs a post-apply hook right after it fetches
the admin kubeconfig (the same spot whether the apply made changes or was a
no-op). The hook is exactly what bnkforge register runs, with errors swallowed.
It:
- Checks the opt-in. No
bnkforge.register: true⇒ it does nothing. - Reads the cluster identity from
cluster-outputs.json(cluster id, region, name). No recorded cluster id yet ⇒ skipped with a note. - Resolves the IBM Cloud API key non-interactively (env → keychain → config). This is what gets stored as the credential template, so Forge can re-derive the kubeconfig on demand instead of holding a perishable one.
- Authenticates to Forge over its REST API. A cached session token from the OS
keychain is reused when still valid; otherwise it logs in with
BNK_FORGE_USER/BNK_FORGE_PASSWORD— orbnkforge.password_b64— and caches the new token. The auto-hook runs non-interactively, so with no valid token, no env credentials and no stored password it declines rather than blocking a deploy on a prompt. - Ensures the credential template and the project, then registers the cluster by id + region + template.
Sample output on a successful run:
→ Registering cluster "acme-eu-roks" with BNK Forge (https://forge.example.com)…
✓ Registered cluster "acme-eu-roks" with BNK Forge (project "acme-eu", forge cluster id 17).
Registration is non-destructive
Re-registering used to DELETE any same-named cluster and re-POST it. Within one project that was called idempotent, and very nearly is — except the cluster id changed, so anything referencing it broke and any scan history attached to the old id was discarded. Across projects it was worse: the lookup was project-scoped, so a cluster held by another project was invisible, and the POST either moved it silently or failed with a bare exit 1 that named nothing.
Since v1.42.0 (#54):
| The cluster name is held by | What happens |
|---|---|
| this project | Updated in place — the cluster id is preserved |
| another project | Refused, naming the owning project and both ids. --force takes it over |
| nobody | Created |
$ roksbnkctl -w acme-eu bnkforge register
Error: cluster is registered to another BNK Forge project: "acme-eu-roks" is held by
project "platform-team" (id 93, cluster id 35). Re-registering would move it, changing
its cluster id and removing it from that project's view. Pass --force to take it over
deliberately, or unregister it there first
--force is a real move — the cluster is removed from the owning project and
re-created in yours, so its id changes. That is inherent to moving it, and is what
--force opts into.
The automatic post-
uphook never forces. An unattended step silently taking another project’s cluster is precisely the harm the refusal exists to prevent. Only an explicitroksbnkctl bnkforge register --forcecan do it.
Two behaviours worth knowing when Forge is older or your account is narrowly scoped:
- An older Forge build with no
PUTfor the cluster resource falls back to the historical delete-and-recreate, so registration still works; the id changes, which is the cost of the older server. The fallback fires only on404/405— a transient500is reported as an error rather than escalated into a destructive retry. - A project you cannot read produces a warning, not a failure. The cross-project scan is a second opinion on top of a direct query of your project; refusing to register because some other project was unreadable would block a least-privilege Forge account from its own work.
Registering on demand (no re-up)
roksbnkctl bnkforge register registers the current workspace’s cluster right
now, without enabling the auto-hook or re-running cluster up. Use it to
register a cluster that was provisioned before you enabled the feature, or one you
registered with an existing-cluster record:
roksbnkctl -w acme-eu bnkforge register
roksbnkctl -w acme-eu bnkforge register --project 42 # one-off project override
It reads the cluster id + region from cluster-outputs.json, resolves your IBM
Cloud API key the usual way, and runs exactly what the post-up hook runs — the
only difference is that errors are surfaced rather than swallowed.
Unregistering on teardown
bnkforge disable reads like the inverse of register, but it is not: it clears
the local register flag and never contacts the server. A cluster already
registered stays on Forge’s Kubernetes page, in a project that outlives everything
it was created alongside, pointing at a cluster that may no longer have BNK on it —
or may no longer exist.
roksbnkctl bnkforge unregister is the actual inverse:
roksbnkctl -w acme-eu bnkforge unregister
✓ unregistered cluster "acme-eu" (id 13) from BNK Forge project "acme-eu"
Absence is success. A destroy runs when things are already half dismantled, and may run twice, so each of these reports and exits 0 rather than failing:
| Situation | Output |
|---|---|
| No project of that name | ✓ no BNK Forge project "acme-eu" — nothing to unregister |
| Project exists, cluster does not | ✓ cluster "acme-eu" is not registered in project "acme-eu" — nothing to do |
| Cluster already deleted (404) | treated as done |
Only a genuine server failure surfaces as an error. Treating absence as failure would make the command unusable in exactly the situation it exists for.
It also never creates anything. register ensures the project exists; a
teardown asking “is this still here?” must not bring it into being, or it would
leave behind the very thing being removed.
Registration is not removed automatically on
cluster down. The post-uphook registers, but there is no matching hook on the way down — runbnkforge unregisteras part of your teardown (this is what the BNK Forge blueprints do in their destroy step).
Doing it by hand against the REST API
There is no CLI to drive, but the flow is four ordinary REST calls, so it is
reproducible with curl when you need to see exactly what Forge receives. Look up
the cluster’s identity first:
roksbnkctl -w acme-eu cluster show # cluster_id, region, cluster_name
| Step | Call |
|---|---|
| 1. Log in | POST /api/auth/login → {"token": …} |
| 2. Credential template | GET/POST /api/credential-templates — provider must be lowercase ibm, your API key, plus region and ibm_cos_instance_name |
| 3. Project | GET/POST /api/projects, then PUT /api/projects/{id} to set the ROKS/IBM platform |
| 4. Register | POST /api/projects/{id}/k8s/clusters, or PUT /api/k8s/clusters/{id} to update in place |
Step 3’s PUT matters: without it Forge shows the project’s platform as Unknown.
Step 4 is the one that changed in v1.42.0 — roksbnkctl prefers the in-place PUT
so the cluster id survives, and only falls back to DELETE + POST against a Forge
build that has no PUT route.
Giving Forge an appliance’s SSH key
Registering the cluster is not the same as giving Forge a way into the appliances the workspace builds. Without the second step a perfectly healthy F5 License Proxy reports itself unreachable:
infrastructure_private_key_available: false
infrastructure_access_status: recovery_required
infrastructure_access_message: Infrastructure SSH private key is unavailable.
Run state recovery/repair to rematerialize
access metadata.
Nothing can be recovered there. The credential was never created.
The two “SSH keys” are different things
This is the part that makes the state above confusing:
bnk.flp.vsi.ssh_keynames an existing IBM Cloud VPC key, which puts a public key on the VSI. That is your access to the appliance, and it works.- Forge separately needs the private half, so it can reach the appliance itself. Nothing supplies that as part of building the VSI.
Registering it
roksbnkctl -w acme-eu bnkforge ssh-credential \
--host 52.116.1.2 \
--ssh-username ubuntu \
--key ~/.ssh/id_rsa
| Flag | Meaning |
|---|---|
--host | The address Forge reaches the appliance on. For an FLP that is the floating IP. |
--ssh-username | The SSH user on the appliance (default ubuntu). Note the prefix: --username is the Forge login, as on every other bnkforge subcommand. |
--key | The private key file. Unencrypted — Forge stores the key and cannot prompt for a passphrase. |
--expect-fingerprint | Optional SHA256 fingerprint the key must match before anything is sent (e.g. the VPC key’s). |
--name | Credential name in Forge. Default <project>-ssh. |
--port | SSH port. Default 22. |
Use the floating IP, not the endpoint
flp status reports something like https://10.243.1.4:8443. That is a
services-VPC address, and Forge sits outside that VPC — a credential built
from it can never connect. Passing a URL is refused rather than accepted:
--host "https://10.243.1.4:8443" looks like an endpoint URL, not a host
Pass the bare floating IP or hostname; the port is separate (--port)
Why the key is checked before it is stored
A credential that cannot log in is worse than no credential: Forge then reports infrastructure access as configured, and every later failure points somewhere other than the key. Two checks stand in the way:
--expect-fingerprintcompares the private key’s SHA256 fingerprint — the same valuessh-keygen -l -E sha256and IBM Cloud print — locally, before anything leaves the machine.- Forge itself tests the SSH connection before storing, and answers
400when it fails. That is the stronger of the two: the fingerprint proves the key is the one the VPC knows about, the connection test proves it opens the box.
What it writes
| Step | Call |
|---|---|
| 1. The credential | GET/POST/PUT /api/ssh-credentials — name, host, port, user, and the private key |
| 2. Infrastructure access | POST /api/cloud-auth/ssh/configure — sets infra_enabled, infra_host, infra_ssh_username, infra_auth_type |
| 3. The project link | PUT /api/projects/{id} — sets ssh_credential_id |
Step 2 is the one that is easy to miss. PUT /api/projects/{id} accepts
ssh_credential_id and ignores the infra_* fields — they are not that
route’s to set. With only the credential linked, infra_enabled stays false and
the appliance is still unreachable.
The command reads the project back afterwards and names anything that did not stick, rather than reporting success on a write it did not verify.
TLS: pin the CA, don’t disable verification
Forge installs commonly carry a self-signed certificate, and --insecure makes
that connect. What it costs is easy to miss: the session token is sent on
every request, so disabling verification leaves the connection encrypted but
unauthenticated. Anyone positioned on the path can present a certificate for
the Forge host, terminate TLS, and read the token. Encryption without
authentication protects the traffic from a passive observer and not from an
active one.
That would matter less if the setting were transient, but bnkforge.insecure: true persists in config.yaml — it is typically set once for a lab and then
forgotten, including when the same workspace is later pointed at a production
Forge. So every request made with it prints a warning naming the host:
⚠ BNK Forge TLS verification is DISABLED for forge.lab.example.com:8443 — the API token
is sent over a connection that is encrypted but NOT authenticated.
Anyone able to intercept it can present any certificate for that host and read the token.
Pin the Forge CA instead — `roksbnkctl bnkforge enable --forge-ca <file>` — then drop --insecure.
Pin the CA instead. For a self-signed Forge you generated the CA, so you already hold it — and pinning it authenticates the connection rather than abandoning authentication:
roksbnkctl bnkforge enable --url https://forge.lab.example.com:8443 --forge-ca ./forge-ca.pem
That stores the PEM as bnkforge.ca_b64 (base64 only for single-line YAML
safety — a certificate is public data) and clears insecure. --forge-ca also
works as a one-shot on bnkforge register and unregister. When both are set
the pin wins and the certificate is verified.
Non-interactively, set ROKSBNKCTL_BNKFORGE_CA_B64 (base64 -w0 forge-ca.pem).
roksbnkctl bnkforge status reports which of the three is in force:
tls: verified (pinned CA from bnkforge.ca_b64)
tls: verified (system roots)
tls: NOT VERIFIED (insecure: true) — the API token is sent over an unauthenticated connection; pin a CA with `bnkforge enable --forge-ca <file>`
If the pinned CA is wrong, the connection fails at the handshake with an
x509 error rather than silently falling back — which is the point.
Behaviour notes
- Best-effort — it never blocks or fails the deploy. Whatever happens with
Forge,
cluster upsucceeds if the cluster came up. Worst case you get a one-line note and register later withroksbnkctl bnkforge register. - Re-registering preserves the cluster id. Running
cluster upagain on an unchanged cluster re-runs the hook harmlessly: the cluster is updated in place, so references to its Forge id and any scan history attached to it survive. Before v1.42.0 the id changed on every re-register (#54). - The auto-hook never forces. A cluster held by another Forge project is left
alone and the note says so; only an explicit
bnkforge register --forcemoves it. - Registering later. If it was skipped (no session, no credentials, no TTY), fix
the prerequisite and run
roksbnkctl bnkforge register— no re-upneeded.
Troubleshooting
-
no BNK Forge URL (set bnkforge.url, BNK_FORGE_URL, or --url)— nothing toldroksbnkctlwhere Forge lives.roksbnkctl bnkforge statusshows what resolved. -
no BNK Forge username/no BNK Forge password— the cached session token is absent or expired and there is nothing to log in with. SetBNK_FORGE_USERandBNK_FORGE_PASSWORD, storebnkforge.password_b64, or runroksbnkctl bnkforge registerat a terminal to be prompted. On a CI runner the env var and the stored field are the only paths — there is no prompt to fall back to. -
cluster is registered to another BNK Forge project— the name is held by a different project; the message names it and both ids. Either unregister it there, or pass--forceto move it deliberately (its cluster id changes, because a move re-creates it). This refusal is the point: silently taking another project’s cluster is the harm. -
Forge shows the credential template but nothing uses it — check its
provider. Forge matchesprovider == "ibm"case-sensitively in several places and lowercases none of them, so a template created with"IBM"is accepted by the API and then matches nothing: credentials are never injected into a deployment, blueprint inputs sourced fromcredential_templatenever resolve, and the “IBM templates must carry an API key” validation never fires. Nothing errors. roksbnkctl wrote"IBM"before v1.56.0; re-runningbnkforge registerwith a current build repairs an existing template rather than only fixing new ones. -
HTTP 403/ lacks permission to manage credential templates — your BNK Forge session’s role can’t create or list credential templates. Log in as an operator/admin, or have one pre-create the template.A
403on a project’s cluster list is different and is not fatal: the cross-project ownership scan warns and continues, so a least-privilege account can still register into its own project. -
no cluster id recorded in cluster-outputs.json yet— the cluster identity wasn’t written. Runroksbnkctl cluster register <name>(or re-runcluster up) socluster-outputs.jsonis populated, then register.
Cross-references
- Chapter 8 — The cluster phase — where the post-apply hook fires.
- Chapter 9 — Registering an existing cluster
— how
cluster-outputs.json(the cluster id + region the hook reads) gets populated for clustersroksbnkctldidn’t provision. - Chapter 14 — Credentials and the resolver chain — how the IBM Cloud API key passed to Forge is resolved.
- Chapter 28 — Configuration reference
— the
bnkforge:field schema.
Two behaviours that exist because of how Forge drives this tool
The Forge modules run one
container per roksbnkctl phase, argv-only, each step re-running
init --override-from-env --non-interactive from a curated environment over
a deployment-scoped shared /work, against a digest-pinned runner image.
Two of roksbnkctl’s checks are shaped by that, and both would otherwise break
blueprints:
An unknown BNK line warns instead of refusing. manifest_version is a
free-text input on the bnk-install module, the support matrix ships inside the
binary, and the image is pinned by digest — so a user selecting a release newer
than the pinned build would hit a hard refusal on a combination never known to be
wrong, with no action available to them. It warns and proceeds instead.
An unset network_mode defers to the recorded cluster. Each step’s
environment carries only what that step needs, so a mode set by cluster-create
is absent when bnk-install regenerates config.yaml. Reading that
absence as a demand for single-nic would refuse a correct multi-NIC deployment
at its second module. Only an explicit contradiction is refused.
The practical rule for blueprint authors: only the module that CREATES a cluster needs to know the network mode. Everything downstream reads the record.
COS supply chain management
BIG-IP Next for Kubernetes (BNK) pulls its runtime artefacts — the F5 Application Runtime (FAR) container images, the JWT licence used at install + renewal time, the f5-bigip-k8s-manifest Helm chart, and the schematic JSON the deployer renders — from IBM Cloud Object Storage (COS). The COS bucket is the supply chain: it’s how artefacts produced upstream (F5 build pipeline, licence-issuing service, schematic generator) reach the cluster.
roksbnkctl cos is the management surface for that supply chain. Three command levels — cos instance, cos bucket, cos object — cover the full CRUD on COS resources without touching the ibmcloud CLI; everything (most visibly cos object put for uploads and cos object get for downloads) goes through the IBM Cloud Go SDKs (go-sdk-core, platform-services-go-sdk, ibm-cos-sdk-go).
What COS is in this stack
COS is IBM’s S3-compatible object store. Two layers matter here:
- Instance: a service instance under Resource Controller. Instances are global; they don’t pin to a region. A workspace typically uses one COS instance per environment (dev, staging, prod) and shares it across multiple buckets.
- Bucket: an S3-style bucket with regional affinity, hosted on a COS instance. Buckets carry the storage class (
standard,vault,cold,smart) and the access policy (HMAC keys, service-instance creds, public ACLs — the latter are off-limits for BNK supply-chain use).
The BNK supply chain reads from one COS bucket per cluster’s BNK install. The bucket holds:
| Object | What it is | Consumed by |
|---|---|---|
f5-far-auth-key.tgz | FAR repository pull credentials — the F5-internal artefact key that lets FLO download FAR container images | flo module at install time |
subscription.jwt (or production equivalent) | BNK subscription JWT — the licence the CNE Instance presents | flo and license modules |
schematic-<v>.json | The deployer’s schematic JSON for the deployed BNK version | informational, not directly mounted into the cluster |
| (optional) FAR image tarballs | Pre-pulled FAR images for air-gapped installs | flo when running in disconnected mode |
The bucket structure is defined by the upstream HCL — concretely by the ibmcloud_resources_cos_bucket variable. When you leave cos.bucket unset, init provisions (or discovers and reuses) an account-scoped bucket named bnk-artifacts-<first-12-of-account-id>: COS bucket names share one global namespace (like S3), so the account suffix keeps the name unique and lets a second workspace run from the same account find the bucket the first one created — no duplicate, no re-upload. The instance defaults to bnk-supply-chain. (The bare bnk-artifacts used in the manual cos examples below is just an illustrative name; supply your own.)
Local files instead of COS (no bucket needed)
COS is not mandatory for the two artefacts the BNK phase actually needs at apply time — the FAR auth tarball and the subscription JWT. You can point the workspace at local files and skip COS entirely:
bnk:
far_auth_local_file: /path/to/f5-far-auth-key.tgz
subscription_jwt_local_file: /path/to/subscription.jwt
When both are set, roksbnkctl reads them at render time — extracting the FAR _json_key_base64 service account from the tarball in Go — and injects the content directly into the flo and license modules (use_cos_bucket = false), so no COS instance, bucket, or S3 endpoint is contacted. This is useful when the account has no orchestration COS, or when the COS S3 endpoint isn’t reachable from where you run roksbnkctl.
roksbnkctl init wires this up automatically: it checks COS first, and on any COS error (including the dial tcp … no such host you get when a cluster region such as eu-fr2 has no COS S3 endpoint) it falls back to prompting for these local files and records them on the workspace. You can also set them by hand. When they’re unset, the COS path below is used unchanged.
The three command levels
roksbnkctl cos instance {create|delete|list}
roksbnkctl cos bucket {create|delete|list|get} --instance <name-or-CRN>
roksbnkctl cos object {put|get|delete|list} --instance <name-or-CRN>
All three layers resolve credentials through the standard credential resolver chain — env var, OS keychain, workspace api_key_b64, prompt. There’s no separate “COS credential”; the IBM API key authenticates against Resource Controller (instance ops) and IAM-signed S3 requests (bucket and object ops).
cos instance
Manages COS service instances at the account level via Resource Controller.
# Create a Standard-plan instance under the workspace's resource group
roksbnkctl cos instance create bnk-supply-chain --plan standard
# Override the plan by catalog UUID when roksbnkctl hasn't mapped the tier
roksbnkctl cos instance create bnk-supply-chain --plan-id <uuid>
# List instances in the account
roksbnkctl cos instance list
# Delete an instance (default: recursive — removes bound HMAC keys, service creds)
roksbnkctl cos instance delete bnk-supply-chain
roksbnkctl cos instance delete bnk-supply-chain --no-recursive --auto
| Flag | Default | Notes |
|---|---|---|
--plan | standard | Friendly name (standard, lite); maps to a Resource Controller plan UUID internally. |
--plan-id | — | Catalog UUID — bypasses the friendly-name mapping. Use when IBM ships a plan tier roksbnkctl hasn’t seen yet. |
--target | global | COS instances are global; this is left as a flag for forward compatibility. |
--no-recursive | (off) | On delete, do NOT remove bound HMAC keys and service credentials. Hardly ever what you want. |
--auto | (off) | On delete, skip the y/N confirmation. |
The resource group is read from the workspace’s ibmcloud.resource_group field (defaulting to default when unset).
cos bucket
Manages buckets within a named instance. The --instance flag is required for every bucket and object call — buckets aren’t globally unique, only unique within an instance.
# Create a standard-class bucket
roksbnkctl cos bucket create bnk-artifacts \
--instance bnk-supply-chain \
--class standard
# List buckets on the instance
roksbnkctl cos bucket list --instance bnk-supply-chain
# Delete (the bucket must be empty first; cos object delete --recursive isn't implemented yet)
roksbnkctl cos bucket delete bnk-artifacts --instance bnk-supply-chain
# Snapshot the entire bucket to a local directory (recursive, streaming, byte-for-byte).
# Useful before rotating supply-chain artefacts — keeps a known-good set you can roll back to.
roksbnkctl cos bucket get bnk-artifacts ./snapshot \
--instance bnk-supply-chain
| Flag | Default | Notes |
|---|---|---|
--instance | (required) | Instance name or CRN — the CRN starts with crn:v1: and is used as-is; a bare name is looked up via Resource Controller. |
--region | workspace region | The IBM Cloud region the bucket is pinned to. Override only when you’re crossing regions deliberately. |
--class | standard | Storage class: standard (frequently accessed), vault (infrequent), cold (archive), smart (auto-tiered). The BNK supply chain uses standard because FLO reads at install + every restart. |
--no-clobber | (off) | On cos bucket get: skip any object whose local path already exists. Default overwrites. Files-only flag; ignored by other cos bucket verbs. |
cos object
Manages objects (files) within a bucket. The key syntax is <bucket>/<key/with/slashes> — the parser splits on the first slash, so bucket/dir/file.tgz parses as bucket bucket, key dir/file.tgz.
# Upload (streaming; multipart auto-engages for large files)
roksbnkctl cos object put bnk-artifacts/f5-far-auth-key.tgz \
./local/f5-far-auth-key.tgz \
--instance bnk-supply-chain
# Download (streaming)
roksbnkctl cos object get bnk-artifacts/f5-far-auth-key.tgz \
./downloaded.tgz \
--instance bnk-supply-chain
# Delete
roksbnkctl cos object delete bnk-artifacts/old-subscription.jwt \
--instance bnk-supply-chain
# List (with an optional key prefix)
roksbnkctl cos object list bnk-artifacts \
--instance bnk-supply-chain
roksbnkctl cos object list bnk-artifacts/schematics/ \
--instance bnk-supply-chain
The list output is a tab-separated KEY SIZE MODIFIED table — pipe through column -t for readability or cut -f1 to extract just the keys.
The BNK supply chain shape
A typical bnk-artifacts bucket after a clean install looks like:
$ roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain
KEY SIZE MODIFIED
f5-far-auth-key.tgz 2412 2026-05-08T14:12:33Z
subscription.jwt 1857 2026-05-08T14:12:34Z
schematic-2.3.0-3.2598.3-0.0.170.json 18432 2026-05-08T14:13:01Z
Three pieces of metadata in the upstream HCL (terraform/variables.tf) pin the bucket layout:
| HCL variable | Default | Object |
|---|---|---|
f5_cne_far_auth_file | f5-far-auth-key.tgz | FAR pull credentials |
f5_cne_subscription_jwt_file | subscription.jwt | Subscription JWT |
f5_bigip_k8s_manifest_version | 2.3.0-3.2598.3-0.0.170 | Schematic filename inferred from this |
Changing any of these in terraform.tfvars (or the workspace bnk: block, which renders into tfvars) changes which COS keys FLO will look for. The HCL doesn’t auto-discover key names — they’re literal.
For air-gapped installs where the cluster can’t reach repo.f5.com, additional pre-pulled FAR image tarballs go in the same bucket and the far_repo_url variable points at a COS-backed proxy. That topology is out of scope for v1.0; the supply chain shape described here is the connected-mode happy path.
Multipart upload and streaming download
FAR image tarballs run 1-5 GB. cos object put streams the input file into the bucket in 5 MB parts using S3-style multipart uploads:
- For files under 5 MB, a single-part
PutObjectis used (the SDK’s default). - For files over 5 MB, the SDK auto-engages multipart upload — the file is split into 5 MB parts, each uploaded in parallel (up to 4 concurrent parts), and finalised with a
CompleteMultipartUploadcall.
The split is transparent — there’s no --multipart flag to set. The SDK handles it under the hood. If you want to verify multipart is happening for a specific file, watch with roksbnkctl --verbose cos object put … and the SDK’s debug logging surfaces the part count.
cos object get is similarly streaming: the SDK pipes the body straight to the destination file without buffering it in memory. Multi-gigabyte downloads on a memory-constrained jumphost are safe.
If a multipart upload is interrupted (network drop, ^C), the partial-upload state lingers on COS until cleaned up. Today roksbnkctl doesn’t expose a “list and abort orphan multipart uploads” command — that’s a v1.x addition. The workaround is to use ibmcloud cos list-multipart-uploads directly via Chapter 17’s docker backend or the IBM Cloud console.
Workspace config integration
The workspace cos: block is optional — if the bucket is already populated (manually, or by an external CI pipeline), the block can be omitted entirely. When set, it triggers an auto-upload at roksbnkctl up time so the FAR pull and licence land before FLO needs them.
# ~/.roksbnkctl/<workspace>/config.yaml
cos:
instance: bnk-supply-chain
bucket: bnk-artifacts
upload:
- source: ./local/f5-far-auth-key.tgz
key: f5-far-auth-key.tgz
- source: ./local/subscription.jwt
key: subscription.jwt
The block maps directly to internal/config/workspace.go::COSCfg:
| Field | Type | Purpose |
|---|---|---|
instance | string | COS instance name or CRN. Looked up via Resource Controller at runtime. |
bucket | string | Bucket name within the instance. |
upload | list of {source, key} | Optional pre-flight uploads. source is a host filesystem path (relative or absolute); key is the destination object key in the bucket. |
Pre-flight uploads run before terraform apply, so FLO sees the artefacts when it pulls. Idempotent: re-running up re-uploads, which COS treats as overwrite — safe.
When the supply chain matters
Three lifecycle moments where the COS bucket is in play:
Install time
roksbnkctl up provisions FLO, which queries the bucket for f5-far-auth-key.tgz and subscription.jwt. Missing either object → FLO fails to start → terraform apply retries (per internal/cli/lifecycle.go::applyWithRetry) for ~3 attempts before erroring. The fix is always “put the missing object in the bucket and re-run up”; the lifecycle retry hides transient bucket-policy propagation lag but won’t paper over a genuinely-empty bucket.
Upgrade time
When bnk.manifest_version (or the f5_bigip_k8s_manifest_version HCL variable) bumps, FLO pulls a new FAR image tarball and re-renders the CNE Instance. If the new manifest version references a FAR image that isn’t already in repo.f5.com’s public registry (rare, but happens for pre-release builds), the bucket holds the air-gapped fallback. Standard upgrades — the connected-mode case — don’t touch the bucket; they just pull from repo.f5.com using the credentials in f5-far-auth-key.tgz.
Licence rotation
When the trial expires or a production licence arrives, swap subscription.jwt for the new file:
roksbnkctl cos object put bnk-artifacts/subscription.jwt \
./new-license.jwt \
--instance bnk-supply-chain
# Force FLO to re-read the licence (delete the CNE Instance's License resource;
# FLO's reconciler re-creates it from the updated JWT)
roksbnkctl k delete license -n f5-bnk --all
FLO picks up the new JWT within 60-90 seconds. No roksbnkctl up re-run required for licence rotation alone.
Worked example: rotating COS supply-chain assets
End-to-end Part VII scenario: the FAR auth key on file is about to expire, a new one arrived from the F5 distribution side, and you need to rotate it without taking BNK down. The same flow handles licence-JWT rotation (swap subscription.jwt for the production JWT) and FAR-image-tarball uploads for air-gapped clusters. Cross-link to Chapter 14 for the API-key half of the rotation story; this walkthrough focuses on the COS object half.
# 1. Sanity-check the current state
roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain
# 2. Upload the new auth key (overwrites the existing file)
roksbnkctl cos object put bnk-artifacts/f5-far-auth-key.tgz \
./new-far-auth-key.tgz \
--instance bnk-supply-chain
# 3. Verify the upload
roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain
# Expected: the f5-far-auth-key.tgz row's MODIFIED timestamp is now
# 4. (optional, air-gapped only) Upload the FAR image tarball
roksbnkctl cos object put bnk-artifacts/far-2.3.0-images.tgz \
./far-2.3.0-images.tgz \
--instance bnk-supply-chain
# 5. Force FLO to re-read the supply chain
roksbnkctl k delete pod -n f5-bnk -l app=flo
# (FLO's controller restarts; the new pod re-pulls f5-far-auth-key.tgz on first reconcile)
# 6. Verify FLO is healthy with the new key
roksbnkctl logs flo
# Expected: no "failed to pull FAR image: unauthorized" lines
The third step is the verification gate. If FLO’s logs still show auth failures after the pod restart, the new auth key was rejected by repo.f5.com — re-issue the key on the F5 side, not in the bucket.
Cross-references
- Chapter 12 §“
cos:— COS supply-chain (optional)” — the workspace-config block this chapter operationalises. - Chapter 13 — Terraform variables —
f5_cne_far_auth_file,f5_cne_subscription_jwt_file,ibmcloud_cos_instance_name,ibmcloud_resources_cos_bucketare the HCL handles. - Chapter 14 — Credentials — the API-key resolution that auths every
coscall. - Chapter 24 — Day-2 ops —
roksbnkctl logs flois the post-rotation verification command. - Chapter 26 — Troubleshooting — bucket-policy propagation lag, missing-object failure shapes.
Troubleshooting
Common failure modes you’ll hit when running roksbnkctl against real IBM Cloud accounts, organised as symptom → root cause → fix. The entries here are mined from the issue logs accumulated over Sprints 0-5 plus the failure shapes documented in PRD 05 §“Risks”.
Use the page as a lookup table. If your symptom isn’t here, Chapter 23 — The E2E test plan lists what every phase asserts; reverse-engineering from the assertions can narrow your diagnosis. For deeper-than-here debugging, the per-phase log files under /tmp/roksbnkctl-e2e-backends/ are the first stop.
Install and init
Symptom: roksbnkctl init errors with plaintext secret detected
Root cause: an existing ~/.roksbnkctl/<workspace>/config.yaml has a credential value sitting in a field whose name matches the rejection regex (api_key, password, token, secret_access_key, hmac_secret). The rejection is a deliberate safety net — see Chapter 14 §“What’s safe to commit vs not”.
Fix: move the credential into IBMCLOUD_API_KEY (env var) or the OS keychain (roksbnkctl init writes it via zalando/go-keyring). For a single-user dev box, the supported plaintext-on-disk channel is ibmcloud.api_key_b64 — base64-encoded, which doesn’t trip the regex.
Symptom: roksbnkctl init interactive prompts loop forever asking for the API key
Root cause: you’re running under CI / a non-TTY shell and roksbnkctl can’t read stdin. The interactive prompt fallback is the last step in the credential resolver chain and it doesn’t gracefully skip when stdin is closed.
Fix: set IBMCLOUD_API_KEY in the env, or pre-populate the keychain entry. For batch / CI runs, the documented invocation is:
IBMCLOUD_API_KEY=$(cat /path/to/secret) roksbnkctl init -w my-workspace
Pre-setting IBMCLOUD_API_KEY skips the API-key prompt (it’s the first link in the resolver chain). init still prompts for the remaining workspace metadata (region, resource group, cluster name) on TTY-bound stdin — a fully non-interactive bootstrap is on the v1.x roadmap.
Symptom: doctor reports terraform: not found on a fresh dev box
Root cause: terraform is the only strictly-required host tool for v1.0 (everything else is internalised). Doctor checks PATH; if your shell session hasn’t sourced the install location it’ll miss.
Fix: install terraform via your package manager (brew install terraform, apt-get install terraform, etc.) and re-source the shell, or set the TERRAFORM_BIN env var pointing at the binary explicitly.
Symptom (Windows): roksbnkctl install succeeds but roksbnkctl version says the command is not recognized
Root cause: roksbnkctl version (like any command) only resolves if the binary sits in a directory on the Windows %PATH%. If the install landed the binary somewhere off %PATH% (for example ~/.local/bin, a Unix convention that is never on the Windows %PATH%), the copy succeeds but the command doesn’t resolve.
Fix: roksbnkctl install (and the install.ps1 one-liner) install into a directory already on %PATH% — preferring %LOCALAPPDATA%\Microsoft\WindowsApps (on the per-user PATH by default, no admin) — so the binary resolves immediately in the same session. Remove any stray copy left in a non-PATH directory with Remove-Item "$env:USERPROFILE\.local\bin\roksbnkctl.exe". If a run ever lands in a non-PATH directory, it prints the PowerShell one-liner ([Environment]::SetEnvironmentVariable('Path', ...,'User')) to add it.
roksbnkctl up lifecycle
Symptom: bnk up refuses with cluster "…" already has BNK installed … but workspace "…" has no terraform state for it
Root cause: exactly what it says — the cluster is serving a BNK install that this workspace’s state knows nothing about. This is the normal shape when a cluster moves between deployments: BNK Forge gives each project its own deployment-scoped volume, so the second project to adopt a cluster legitimately has no state for the first project’s install.
Before v1.42.0 there was no refusal. An empty state planned a full install — resources_to_add=64 over a cluster that already had all 64 — and ground on for about thirteen minutes before exiting 1 without naming the cause or the cluster, three times over with retries (#53). The refusal replaces that with a fast, named failure.
Fix: pick one —
- run from the workspace that installed it, whose state can converge or remove the install;
- remove BNK from the cluster first, then re-run here;
- if the install is genuinely abandoned, clear it from the cluster deliberately rather than letting an apply collide with it.
roksbnkctl -w <ws> bnk status shows what is actually on the cluster.
The check is narrow by construction: it fires only when this workspace’s BNK state holds no managed resources and the cluster is actually serving an install. A workspace that owns an install converges exactly as before and never reaches it. If the cluster cannot be reached, or what the state holds cannot be determined — including an s3 remote backend whose state cannot be listed — the guard stays silent rather than inventing a refusal.
There is deliberately no
bnk adoptyet. Deriving and validating a complete install record from a live cluster is a larger change than a guard, and wants its own testing against real installs.
Symptom: terraform apply errors timeout while waiting for state to become 'normal'
Root cause: IBM Cloud’s control plane is occasionally 5-15 minutes slow propagating cluster state — a known transient. The cluster was created; the API just hasn’t caught up to reporting it as Ready.
Fix: roksbnkctl up retries the apply automatically up to 3 attempts with a 60-second sleep between (see applyWithRetry in internal/cli/lifecycle.go). If all three retries fail, just re-run roksbnkctl up manually — terraform’s state is durable, and the second attempt skips every resource that’s already provisioned.
Symptom: roksbnkctl up returns success but roksbnkctl k get nodes says No resources found
Root cause: the ROKS cluster’s worker nodes take 5-10 minutes to provision after the cluster’s master endpoint returns Ready. Terraform considers the cluster “applied” as soon as the master is up; the workers come up asynchronously.
Fix: wait 5-10 minutes and re-run. If you want a deterministic gate, watch the IBM Cloud console’s cluster page until the worker count matches workers_per_zone × zones, then proceed. There’s no roksbnkctl wait command.
Symptom: roksbnkctl up post-apply hook fails fetching the admin kubeconfig with a 404
Root cause: the IBM Cloud kubeconfig API (/global/v2/applications/kubeconfig) returns 404 for ~30-60 seconds after the cluster create call returns. The cluster exists but the kubeconfig endpoint hasn’t materialised.
Fix: the binary retries with exponential backoff and usually succeeds within a minute. If it still 404s after the retry budget, run roksbnkctl kubeconfig --download -w <workspace> to retry just the fetch without re-applying.
Symptom: Error: Inappropriate value for attribute "kubeconfig_dir": directory does not exist
Root cause: the upstream HCL’s IBM provider doesn’t MkdirAll for the kubeconfig output directory; it expects the parent dir to exist already. The variable’s default (/work/.bnk/scratch/kubeconfig) is the in-container path; on a direct-on-host run it’s a path that doesn’t exist.
Fix: roksbnkctl writes a workspace-scoped override (kubeconfig_dir = ~/.roksbnkctl/<ws>/state/kubeconfig) and creates the dir at apply time. If you’re hand-rolling terraform without roksbnkctl up, mkdir -p ~/.roksbnkctl/<ws>/state/kubeconfig first.
Symptom: cert-manager (or another chart) fails with could not download chart: ... open <HOME>/.cache/helm/repository/<hash>-index.yaml: no such file or directory
Root cause: a helm_release whose repository is a bare chart-repo URL (cert-manager off https://charts.jetstack.io) resolves through Helm’s anonymous-repo path, which downloads the repo index into helmpath.CachePath("repository") — i.e. $HELM_CACHE_HOME/repository, defaulting to $HOME/.cache/helm/repository. Crucially, that path is governed by HELM_CACHE_HOME, not HELM_REPOSITORY_CACHE (which only covers the named-repo / helm repo CLI path). In a runner whose $HOME resolves to an empty or non-writable path — how the bnk-forge tools-runner image launches the phases — that directory can’t be created, so the download fails even though charts.jetstack.io is perfectly reachable (the index downloads on the fly once the dir is writable — no pre-helm repo add is needed). The sibling symptom is a non-fatal warning: creating <HOME>/.kube: mkdir <HOME>: permission denied from the post-apply admin-kubeconfig fetch.
Fix: roksbnkctl exports writable, $ROKSBNKCTL_HOME-relative Helm homes before invoking terraform/helm — HELM_CACHE_HOME, HELM_CONFIG_HOME, HELM_DATA_HOME (plus HELM_REPOSITORY_CACHE / HELM_REPOSITORY_CONFIG / HELM_REGISTRY_CONFIG) under $ROKSBNKCTL_HOME/.helm/, and KUBECONFIG at $ROKSBNKCTL_HOME/.kube/config when $HOME/.kube isn’t writable — and pre-creates those dirs (see prepareToolEnv in internal/tf/terraform.go). The terraform helm provider runs as a plugin subprocess that inherits this env, so the redirect reaches the chart download. $ROKSBNKCTL_HOME is the persisted workspace tree (/work/.roksbnkctl under bnk-forge), so the cache is writable and survives across the phased invocations. Each var is set only when unset, so you can still point the cache at an air-gap mirror by exporting your own value. Set ROKSBNKCTL_DEBUG=1 to log the resolved HELM_CACHE_HOME / KUBECONFIG before each phase’s terraform exec. No action needed on a normal host run — $HOME there is already writable.
Symptom: terraform destroy leaves orphan IBM Cloud resources (LBs, security groups, VPEs)
Root cause: ROKS occasionally leaves dangling cluster-owned resources after the cluster itself is destroyed — the destroy returns success but the IBM Cloud account still shows a load balancer or a Virtual Private Endpoint Gateway tagged with the deleted cluster’s ID.
Fix: run roksbnkctl cleanup. It sweeps the account for every resource named after the workspace prefix (<prefix>-*) — VPC instances/subnets/security-groups/floating-IPs/public-gateways/VPCs, the Transit Gateway, registry COS, the cluster, and the BNK trusted profile — and deletes them in dependency order. Preview first with roksbnkctl cleanup --dry-run; add --all-regions if resources landed in a region not recorded in config.yaml. (For one-off resource types not covered by the prefix sweep, roksbnkctl ibmcloud is load-balancers | grep <cluster-name> + ibmcloud is load-balancer-delete <id> still works as a manual fallback.)
Teardown
bnk down sits for four minutes, then the namespace hangs in Terminating
⚠ 7 BNK custom resource(s) in f5-bnk did not finalize within 4m0s.
⚠ namespace "f5-bnk" was stuck Terminating; cleared F5 finalizers on 2 object(s).
Error: context deadline exceeded
Fixed in v1.59.0. If you see this, you are on v1.58.0 or earlier.
The teardown drains BNK’s custom resources before destroying the namespace, so that the F5 Lifecycle Operator is still running to finalize them — a finalizer that outlives its controller blocks the namespace forever. Three things made that drain unable to finish:
- It deleted the components FLO manages.
cnecontrollers,f5tmms,dssms,afms,downloadersandcnemanifestsare declared by theCNEInstance, and FLO recreates any of them within about five seconds for as long as theCNEInstanceexists. Deleting theCNEInstanceis the uninstall; the components go with it. - Two product defaults cannot be drained at all.
f5-big-tcp-settings/sys-default-tcpis refused by BNK’s own admission webhook (Default TCP parameters CR cannot be deleted!) andf5-big-dns-caches/security-fqdn-dns-resolveris recreated on sight. Neither carries a finalizer, so the namespace delete removes them; waiting for them achieves nothing. f5validate-<ns>blocks its own removal. It hasfailurePolicy: Failand is served from the namespace being destroyed, so once the controller stops answering the API server refuses everyDELETEof ak8s.f5net.comobject. Removing it does not hold — FLO restores it in under a second.
If you are stuck on an older version, the manual escape is to delete the
CNEInstance and let FLO remove the rest, rather than deleting the components
individually:
kubectl -n f5-bnk delete cneinstance --all
kubectl get validatingwebhookconfiguration | grep f5validate # delete if the namespace will not terminate
The teardown says a custom resource “did not finalize”
The message names the object. Two cases, and they need opposite responses:
cneinstances/...— theCNEInstancefinalizer tears down the whole product, so this is legitimately slow work rather than a stall. Let it run.- anything else — check whether it is being recreated:
kubectl -n f5-bnk get <kind> <name> -o jsonpath='{.metadata.uid}'before and after a delete. A new UID means a controller replaced it, and no amount of waiting will remove it while its controller is running.
Cluster-scoped objects remain after a successful bnk down
Expected. A BNK 2.4 install leaves 95 CustomResourceDefinitions, 10
APIServices and a ValidatingAdmissionPolicy behind, none of which a namespace
delete touches. They are shared with any other BNK install on the cluster, so
removing them is a deliberate operator decision rather than a side effect of one
workspace’s teardown. Reinstalling over them works — verified across consecutive
install/uninstall cycles at the same manifest version.
scripts/testing/cluster-residue-check.sh reports what is left, and excludes
cert-manager: it is shared infrastructure, and removing its CRDs would delete
every Certificate and Issuer on the cluster including ones BNK never created.
What must not remain is a ValidatingWebhookConfiguration served from the BNK
namespace. It is named for one namespace and points at a Service that no longer
exists, and because it matches by namespaceSelector, recreating a namespace of
the same name re-arms a webhook whose backend will never answer.
Workspaces
Symptom: after roksbnkctl ws delete <name>, which workspace is current?
Behaviour (since v1.9.0): you can delete the current workspace directly — ws delete moves the current_workspace pointer to another existing workspace (alphabetically first), or clears it when you delete the last one. There’s no “switch first” requirement and no phantom default fallback.
If a command then reports no workspace selected: you deleted the last workspace, so nothing is current. Create one with roksbnkctl init, or point at an existing one with roksbnkctl ws use <name>. (If the message is instead workspace "<name>" is not initialised, the current_workspace pointer references a directory that was removed by hand — ws use <other> repoints it.) Documented in Chapter 6 — Workspaces.
Symptom: workspace "<name>" has terraform-managed resources; pass --force to delete anyway
Root cause: the workspace’s terraform.tfstate is non-empty — live infrastructure exists. roksbnkctl ws delete refuses to orphan the resources by removing the state file out from under them.
Fix: run roksbnkctl down -w <name> --auto first to destroy the resources, then roksbnkctl ws delete <name> (no --force needed once state is empty). If you genuinely want to abandon the infra and clean up by hand later, roksbnkctl ws delete --force skips the check.
Backends
Symptom: --backend docker errors with Cannot connect to the Docker daemon
Root cause: dockerd isn’t running, or your user isn’t in the docker group, or you’re on a system that needs a separate rootless-docker socket path.
Fix:
- Linux with system docker:
sudo systemctl start docker; add yourself to thedockergroup (sudo usermod -aG docker $USER) and log out + back in. - Linux with rootless docker:
systemctl --user start docker; setDOCKER_HOST=unix:///run/user/$(id -u)/docker.sock. - macOS / Windows: launch Docker Desktop / Rancher Desktop.
Verify with docker info | head -1 — if that fails, roksbnkctl --backend docker will too.
Symptom: --backend k8s errors with ops pod not found in roksbnkctl-ops namespace
Root cause: you haven’t run roksbnkctl ops install against the target cluster. The k8s backend dispatches into a long-lived ops pod that has to be provisioned first.
Fix:
roksbnkctl ops install
Verify with roksbnkctl k get pod -n roksbnkctl-ops — the pod should be Running. See Chapter 19 for the install model.
Symptom: --backend ssh:<target> errors with tool not found: iperf3 — run with --bootstrap to apt-install
Root cause: the SSH target doesn’t have the tool installed, and roksbnkctl doesn’t auto-install without explicit opt-in (because apt-installing on a production jumphost without consent is rude).
Fix: pass --bootstrap once per fresh target:
roksbnkctl --backend ssh:jumphost --bootstrap test throughput
The bootstrap step runs apt-get install -y <tool> (or the equivalent for ibmcloud — adding the IBM apt repo first). Subsequent calls skip the install check and run normally. See Chapter 17 §“SSH backend” for the bootstrap mechanism.
Symptom: --backend ssh:jumphost errors host key mismatch for jumphost (got SHA256:..., known_hosts has SHA256:...)
Root cause: the jumphost was re-provisioned (terraform destroy + apply) and now has a fresh host key, but ~/.roksbnkctl/known_hosts still has the old fingerprint. TOFU refuses to silently accept the change — that’s the threat model the prompt exists to defend against.
Fix: if you know the re-provision is legitimate, delete the stale entry:
ssh-keygen -R '<jumphost-ip>' -f ~/.roksbnkctl/known_hosts
# Or for the whole roksbnkctl known_hosts:
rm ~/.roksbnkctl/known_hosts
The next roksbnkctl --on jumphost call will TOFU-prompt with the new fingerprint. For CI use the --insecure-host-key flag, which records the key on first contact without prompting.
OpenShift and PodSecurity
Symptom: throughput test pod fails admission: violates PodSecurity "restricted:v1.x": runAsNonRoot != true
Root cause: the throughput suite’s default iperf3 image is networkstatic/iperf3:latest which runs as root. OpenShift’s restricted-v2 SCC rejects root pods.
Fix: set the workspace config to use the bundled image, which is built USER 1000:
# ~/.roksbnkctl/<workspace>/config.yaml
test:
throughput:
image: ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:v0.9.0
Chapter 22 §“The bundled image and the runAsNonRoot constraint” is the full backstory. The same chapter’s §“OpenShift SCC failure mode” lists the three error-message variants OpenShift produces.
Symptom: roksbnkctl ops install errors ServiceAccount "roksbnkctl-ops" forbidden: violates PodSecurityPolicy
Root cause: rare — the cluster is running PodSecurityPolicy (the deprecated predecessor to PodSecurity admission) and the ops pod’s ServiceAccount doesn’t have the SCC binding it needs.
Fix: the ops manifest assumes restricted-v2 is acceptable. If your cluster forces privileged, that’s a cluster-policy question outside roksbnkctl’s control — talk to your cluster admin about granting restricted-v2 to the roksbnkctl-ops namespace.
Symptom: ImagePullBackOff on the ops pod or throughput pod
Root cause: most commonly, the cluster can’t reach the image registry. Three sub-causes:
- The cluster’s egress NAT doesn’t route to
ghcr.io(the image host forroksbnkctl-tools-*). - The image tag doesn’t exist for the version you’re running (e.g., you built
roksbnkctlfrommainat a commit between releases, and:devisn’t published). - ghcr.io itself is rate-limiting unauthenticated pulls (rare; usually only an issue for shared CI hosts hitting ghcr.io en masse).
Fix:
- Check egress with
roksbnkctl k exec <ops-pod> -- curl -sI https://ghcr.io— if that hangs, you have a network path issue, not a roksbnkctl issue. - Check the tag with
docker manifest inspect ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<version>— if 404, pin to a tagged release version in workspace config rather than running frommainhead. - For rate-limit issues, pre-pull images to a local registry mirror and override the workspace
test.throughput.imageto point there.
DNS
Symptom: roksbnkctl test dns returns NXDOMAIN against an internal GSLB record that you know exists
Root cause: your laptop’s resolver chain doesn’t have a route to the internal GSLB VIP. The default --server system uses your /etc/resolv.conf, which resolves against your office or ISP resolver — neither of which knows about the cluster-private GSLB.
Fix: query the GSLB VIP explicitly, or query from inside the cluster:
# Query the GSLB VIP directly
roksbnkctl test dns --target www.example.com --type A --server 169.45.91.5
# Or run the probe from inside the cluster (the cluster's resolvers reach the GSLB)
roksbnkctl test dns --target www.example.com --type A --backend k8s --server cluster
Chapter 21 §“Server resolution” is the full --server reference.
Symptom: --gslb-compare always reports gslb_divergence: false against a target you expect to diverge
Root cause: the chosen target’s GSLB rule isn’t differentiating your local vantage (laptop) from your k8s vantage (cluster). Two common shapes:
- The name is fronted by an anycast resolver fleet (Cloudflare, Google Public DNS) — same answer everywhere by design.
- Your laptop and your cluster are both in the same geographic region from GSLB’s perspective (both in North America hitting the same datacenter).
Fix: pick a target known to be geo-resolved (www.google.com is the canonical “different IPs from different regions” example), or add an SSH-based vantage (--backend ssh:eu-bastion) to bring in a third region. Chapter 21 §“GSLB cross-vantage compare” covers the multi-vantage workflow.
Symptom: roksbnkctl test dns --backend docker errors DNS probe doesn't benefit from docker
Root cause: design choice. Docker containers share the host’s network namespace by default, so a docker-backend probe has the same network identity as a --backend local probe — no GSLB-relevant vantage difference.
Fix: use --backend local, --backend k8s, or --backend ssh:<target> instead.
Cluster registration
Symptom: roksbnkctl cluster register <name> errors cluster not found
Root cause: the cluster name doesn’t exist in the workspace’s resource group, or the API key doesn’t have visibility into the resource group.
Fix: verify the name with roksbnkctl ibmcloud ks cluster ls --output json | jq '.[].name', and verify the resource-group scope in workspace config matches where the cluster lives. If the cluster is in a different resource group, set ibmcloud.resource_group in the workspace config to that group.
Symptom: register succeeds but roksbnkctl k get nodes immediately errors Unauthorized
Root cause: the kubeconfig was fetched but the auth token has already expired, or the IAM-based token that the kubeconfig embeds doesn’t match the API key that’s currently in env. Common after a 1-hour idle window.
Fix:
roksbnkctl kubeconfig --download --cluster <name>
The token refresh is automatic on every up/apply, but register against a cluster you didn’t just provision sometimes lands you with a stale token in the kubeconfig.
COS supply chain
Symptom: FLO fails to start with failed to pull FAR image: 403 Forbidden
Root cause: the f5-far-auth-key.tgz object in the bucket has stale credentials (the F5-side pull key was rotated, but the bucket still has the old one).
Fix: re-issue the key on the F5 side and upload to COS:
roksbnkctl cos object put bnk-artifacts/f5-far-auth-key.tgz \
./new-f5-far-auth-key.tgz \
--instance bnk-supply-chain
# Restart FLO so it re-reads
roksbnkctl k delete pod -n f5-bnk -l app=flo
See Chapter 25 §“Worked example” for the full flow.
Symptom: cos object put for a 3 GB file errors midway with RequestTimeout
Root cause: the multipart upload SDK encountered a transient COS HTTP timeout on one of the part uploads. Multipart uploads aren’t currently resumed from the failure point — they restart from zero.
Fix: re-run the cos object put. If it fails reproducibly on the same part, the underlying network is the problem (your egress link is saturated, or COS is having a regional outage — check the IBM Cloud status page).
Symptom: cos bucket delete errors Bucket not empty
Root cause: COS requires buckets to be empty before delete; there’s no --recursive flag on bucket delete today.
Fix: list and delete each object, then delete the bucket:
roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain | \
awk 'NR>1 {print $1}' | \
xargs -I{} roksbnkctl cos object delete "bnk-artifacts/{}" --instance bnk-supply-chain
roksbnkctl cos bucket delete bnk-artifacts --instance bnk-supply-chain
Don’t forget to abort any pending multipart uploads first — they don’t appear in the standard object list but they do prevent bucket deletion. The workaround for now is ibmcloud cos list-multipart-uploads followed by ibmcloud cos abort-multipart-upload until v1.x lands a native command.
Shared VPC, Transit Gateway, and quotas
Symptom: roksbnkctl up fails at apply with a VPC or Transit Gateway quota error (Quota: 20, or cannot add more than 10 gateways)
Root cause: IBM Cloud caps VPCs at 20 per region and Transit Gateways at 10 per account (defaults). A full region/account fails the create — often ~40 minutes into a cluster build.
Fix: run roksbnkctl doctor first — the ibm cloud quota check reports VPCs-in-region and account-wide Transit Gateways vs the limits and warns when you’re at the wall. The init interview shows the same count at the create-VPC / create-TGW prompts; answer no and adopt an existing VPC/gateway (the picker lists them), or request a quota increase from IBM. See Reusing an existing VPC and Sharing a Transit Gateway.
Symptom: image pulls from the mirror time out intermittently, but every security group and ACL allows the traffic
Root cause: two VPCs on the same Transit Gateway have overlapping address prefixes. A gateway routes on VPC address prefixes; when two attached VPCs claim the same block the route is ambiguous and the gateway silently drops traffic for one of them. Nothing logs an error, which is what sends people into hours of firewall archaeology. The classic tell is that connectivity is partial or flaps — some pulls land, some hang — rather than failing cleanly.
It happens by accident because IBM’s default address-prefix management is auto, which gives every VPC in a region the identical three per-zone prefixes (10.241.0.0/18, 10.241.64.0/18, 10.241.128.0/18). A second roksbnkctl-created cluster joining a shared gateway therefore collides by construction.
Confirm it — list the prefixes of each VPC on the gateway and look for repeats:
ibmcloud tg connections <gateway-id> --output json | jq -r '.[]|select(.network_type=="vpc")|.network_id'
ibmcloud is vpc-address-prefixes <vpc-id> # per VPC from that list
Fix: the prefixes of a live VPC cannot be edited — moving a subnet’s CIDR replaces the subnet, destroying the cluster on it. Either detach one (roksbnkctl -w <other-workspace> tgw disconnect --auto; the cluster survives), or rebuild one cluster on its own block, set before cluster up:
cluster:
vpc_cidr: 10.242.0.0/16 # ROKSBNKCTL_CLUSTER_VPC_CIDR for CI
Set it before
cluster up, not after. Editingvpc_cidrin a workspace whose cluster already exists does not re-address anything — an existing VPC’s prefixes cannot be changed without replacing its subnets. It now says so:! cluster.vpc_cidr is a CREATE-time setting and it has changed. Cluster "x" was created from 10.242.0.0/16; this workspace now says 10.99.0.0/16. The new value is ignored — ... A future release will refuse this rather than ignore it; ...The warning fires only on a real disagreement with the block the cluster was created from, so a workspace that set it once and left it alone stays silent. A workspace built by
cluster registeralso stays silent, because the adopt path records no block to disagree with — see Chapter 9.
Prevention (v1.39.0+): cluster up refuses before terraform creates anything when the VPC it is about to build would overlap one already on the gateway, and tgw connect refuses before attaching — each naming the other VPC and the colliding prefixes. See Give each cluster VPC its own address block.
Symptom: cluster ... was created as a single-nic cluster, but this workspace now asks for multi-nic
Root cause: cluster.network_mode is a create-time setting and it changed
after the cluster was built. There is no in-place conversion between modes.
Why it refuses rather than warns: terraform would happily plan this — as a destroy and create of the running cluster, rendered as ~60 lines that read like an update. The refusal happens before planning, so you never get the chance to approve it by mistake.
Fix: set cluster.network_mode back to what the cluster is, or build the
other mode as a new cluster in a new workspace. What the cluster actually is
comes from cluster-outputs.json, not from config.yaml.
Note that removing the setting is not the same as changing it: an unset
network_mode defers to the record and is never refused. See
Chapter 8a §Which side wins.
Symptom: BNK 2.3 does not support multi-nic clusters (it supports single-nic)
Root cause: the BNK release — derived from bnk.manifest_version — cannot
drive a cluster of that network mode. 2.3 does not express the multi-NIC network
attachments or CNEInstance options.
Fix: use a cluster built in a mode this line supports. There is currently
no line to move to for multi-nic — 2.4 was expected to add it and, on the
evidence of its EA install guide, does not, so its row was corrected to
single-nic. See Chapter 28.
The pairing is checked at plan time precisely so this is a message rather than a
failed apply against real infrastructure.
Symptom: ! unknown BNK release line: BNK 2.5 is not in this build's support matrix
This is a warning, and the run continues. It is expected, not a defect.
Root cause: the support matrix ships inside the binary, so a release newer than your build is unknown to it. That is missing information, not a known incompatibility — refusing on it would make every build refuse every BNK release that shipped after it, which is unhelpful when the binary is pinned (the BNK Forge modules pin the runner image by digest, so “use a newer build” is not an action available to whoever is choosing the release).
What it means: the combination is unverified here, not blessed. Upgrade
roksbnkctl if you want it actually checked.
Symptom: down fails with The VPC is in use and cannot be deleted. Subnets [...]
Root cause: this workspace created the VPC, and another workspace that adopted it still has a cluster (its subnets) inside. The VPC owner must be torn down last — deleting it first can’t remove a non-empty VPC, and also deletes the shared per-zone public gateways the other cluster still depends on.
Fix (v1.26.1+): down refuses this up front, naming the foreign subnets. Tear down the workspace(s) sharing the VPC first, then re-run down on the owner. See the teardown-order note in Reusing an existing VPC.
Symptom: down fails with UnsetSubnetPublicGateway ... the specified subnet has no public gateway
Root cause: an adopter cluster’s teardown tried to detach its subnets from a shared public gateway that the VPC-owning cluster had already deleted (the owner was torn down first — see above).
Fix: v1.26.1+ clusters can’t hit this — subnets attach their gateway inline, so deleting the subnet detaches implicitly. For an older cluster stuck mid-teardown, drop the dead attachment from state and retry: cd ~/.roksbnkctl/<ws>/state-cluster && terraform state rm 'module.roks_cluster.module.cluster.ibm_is_subnet_public_gateway_attachment.cluster_subnet_gateway_zone1[0]' (repeat for zone2/zone3), then roksbnkctl -w <ws> cluster down.
Symptom: roksbnkctl bnk up hangs on licensing / the License CR never reaches Active
Root cause: BNK licensing completes only when the License CR reports status.state: Active (or the LicenseActive condition), which roksbnkctl polls for deterministically (~15-minute bound). A hang usually means the FAR supply chain is wrong — a missing/expired JWT, or the JWT uploaded under the wrong object key — or, in f5licenseproxy mode, the F5 License Proxy isn’t reachable.
Fix: confirm the JWT is present under the exact object key subscription.jwt (roksbnkctl cos object list <bucket> --instance bnk-supply-chain) and not expired; for FLP mode check roksbnkctl flp status before bnk up. See Licensing BNK with the FLP.
Since v1.41.0 the reachability gate catches the network half of this before the install starts, so a licensing hang that is really an unreachable proxy now surfaces as a named per-node failure instead — see the next entry.
Symptom: bnk up fails the reachability gate on a cluster whose network you believe is correct
✗ kube-…-0000013a -> registry (10.243.0.4:443): dns=skipped-ip tcp=FAILED
(tcp connect failed -- refused, filtered, or no route)
Root cause: most often the gate is right and something (a security group, an ACL, a missing Transit Gateway attachment, or overlapping VPC address prefixes) genuinely blocks the path from that node.
But there is one case where the network is correct and the probe is early. A Transit Gateway attachment is asynchronous: IBM programs the routes some time after the connection reports attached. A workspace that creates the VPC, joins the gateway, and installs in a single run can probe a path that is correct but not yet live. A probe 73 seconds after attach has failed on a path that was healthy minutes later, while a sibling cluster on the same gateway and the same blueprint passed simply by landing on the other side of route programming (#57).
Fix: read the attempt count in the detail first — it is what tells the two apart:
-
(still failing after 19 attempts over 180s)— it kept trying for the whole budget. That is a real break; go look at the security groups, the TGW attachment, and the VPC prefixes. -
No attempt clause, or a low count — it gave up early. Raise the budget:
bnk: preflight: reachability_retry_seconds: 600or
ROKSBNKCTL_REACHABILITY_RETRY_SECONDS=600on a CI runner. See Chapter 28 §bnk.preflight.
Check dns before tcp: dns=FAILED is a resolver or hostname problem and no retry budget will fix it.
Re-running gives you a fresh answer. Since v1.42.0 the gate rolls its DaemonSet on every run, so the verdict always belongs to the run you just started. Before that, a re-run after fixing the routing re-read the original pod’s log and showed the same failure with nothing to indicate it was a recording.
Networking
Symptom: roksbnkctl test connectivity reports Get "https://...": dial tcp: i/o timeout for an internal-only URL
Root cause: connectivity probes run from --backend local by default. From your laptop, internal-only URLs (cluster-private VIPs, internal GSLB names) aren’t reachable.
Fix: route the probe through the cluster’s network — either via --backend k8s (when it lands for the connectivity suite — currently k8s-backend is iperf3 + DNS only; connectivity stays local for v1.0) or via an SSH target inside the cluster’s VPC (--backend ssh:cluster-jumphost).
Symptom: roksbnkctl test connectivity fails with x509: certificate signed by unknown authority against a self-signed internal endpoint
Root cause: the URL’s TLS cert isn’t in the host’s trust store.
Fix: pass --insecure (session-wide; skips TLS validation for every probe in the run). The flag is deliberately session-wide rather than per-host — see Chapter 20 §“Mixed TLS-trust posture”. For mixed trust posture across multiple internal endpoints, run two separate test connectivity invocations, one per trust group.
CI-specific
Symptom: nightly e2e run fails on phase D with Error: Provider configuration is missing
Root cause: a terraform init cache invalidation under ~/.roksbnkctl/<ws>/state/.terraform/ left a partial provider download. Happens after a CI worker is recycled mid-init.
Fix: rm -rf ~/.roksbnkctl/<ws>/state/.terraform/ then re-run roksbnkctl up. Terraform-init re-downloads the providers cleanly. For CI workers that get recycled often, add a pre-step that purges .terraform/ before each run.
Symptom: cred audit (phase M) reports IBMCLOUD_API_KEY found in docker inspect output
Root cause: real stop-ship — credentials leaked into a docker container’s runtime env. Check internal/exec/docker.go::buildEnvArgv for any code path that passes the credential by value (-e IBMCLOUD_API_KEY=<value>) rather than by reference (-e IBMCLOUD_API_KEY — let docker pull from the caller’s env).
Fix: file an issue immediately, do not tag a release until this is green. Phase M is the v1.0 release gate; a leak here means the redactor or the cred-passing logic regressed. See PRD 04 for the threat model.
Getting more help
When the symptom isn’t on this page:
- Re-run with
--verbose(-v) — the verbose output usually surfaces the root cause directly. - Check
/tmp/roksbnkctl-e2e-backends/<phase>-<ts>.logfor the per-phase trail. - Cross-reference Chapter 23 — The E2E test plan — the phase-by-phase pass criteria usually narrow down where the breakage lives.
- File an issue on github.com/jgruberf5/roksbnkctl with the verbose output, the
roksbnkctl --versionstamp, and the per-phase log if there is one.
Command reference
Auto-generated by go run ./tools/refgen/cobra-md > book/src/27-command-reference.md. Re-run on every CLI surface change.
roksbnkctl deploys F5 BIG-IP Next for Kubernetes (BNK) onto IBM Cloud ROKS, manages the COS supply chain BNK depends on, and runs built-in connectivity, DNS, and throughput tests against the deployed environment.
The 4-command lifecycle: roksbnkctl init Interactive setup; writes the workspace config roksbnkctl up Provision (or attach) and deploy BNK roksbnkctl test Run connectivity, DNS, and throughput tests roksbnkctl down Tear down BNK (and the cluster if cluster up provisioned it)
See https://jgruberf5.github.io/roksbnkctl/book/ for the canonical user guide.
Global flags
These flags apply to every command. They are declared on the root command and inherited by every subcommand.
| Flag | Type | Default | Description |
|---|---|---|---|
--backend | string | — | execution backend: local | docker | k8s | ssh:<target> (default: per-tool from workspace exec: block, else local) |
--bootstrap | bool | false | for –backend ssh:<target>: auto-install missing tools on Ubuntu via apt-get (requires passwordless sudo on the target) |
--insecure-host-key | bool | false | skip the host-key TOFU prompt; record on first contact (CI use) |
--no-color | bool | false | disable colored output |
--on | string | — | run on the named SSH target instead of locally (roksbnkctl targets list to see options) |
--output / -o | string | text | output format: text | json |
--quiet / -q | bool | false | suppress all but errors |
--verbose / -v | bool | false | verbose output |
--workspace / -w | string | — | workspace name (default: current; first run creates ‘default’) |
roksbnkctl agent
Drive this workspace with an agentic CLI (personas + AGENTS.md)
roksbnkctl agent [claude|gemini|aider|openai|pi|opencode|agy] [flags]
Agentic mode. roksbnkctl embeds no LLM — bring your own coding-agent CLI.
roksbnkctl agent init Scaffold AGENTS.md + personas/ + journal/ into the workspace
roksbnkctl agent List supported CLIs + this workspace’s default
roksbnkctl agent <cli> Launch <cli> against the workspace\n roksbnkctl agent <cli> –show Print the invocation instead of running it
Personas (act as exactly one at a time): solution-architect (customer interface, owns scope), cloud-operator (runs the lifecycle), test-engineer (validation probes), doc-specialist (the report). See personas/ after init.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--show | bool | false | print the invocation instead of running the agent |
roksbnkctl agent init
Scaffold the agentic-mode files into the workspace
Copies AGENTS.md, CLAUDE.md, the persona role contracts, and a decisions.md seed into the workspace dir, and creates an empty journal/. Safe to re-run: existing files are left untouched (your edits survive).
← back to roksbnkctl agent
roksbnkctl apikey
Print the workspace’s resolved IBM Cloud API key to stdout (a secret)
Resolve the workspace’s IBM Cloud API key via the standard chain and print it to stdout — the scripting seam for driving the ibmcloud CLI off a workspace:
ibmcloud login –apikey “$(roksbnkctl -w <ws> apikey)”
Resolution order (the same roksbnkctl itself uses):
- environment — IBMCLOUD_API_KEY / IC_API_KEY / TF_VAR_* (including any loaded from $PWD/.env, which roksbnkctl reads at startup)
- OS keychain (service “roksbnkctl”, account “
<workspace>/ibmcloud_api_key”) - workspace config.yaml — ibmcloud.api_key_b64 (base64-decoded)
Non-interactive: it never prompts — if no key is found it exits non-zero with an actionable error.
WARNING: this writes the credential to stdout. Pipe or redirect it; never echo it into logs or CI output.
roksbnkctl apply
Apply Terraform without re-prompting (assumes config.yaml exists)
roksbnkctl apply [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the confirmation prompt |
--no-kubeconfig | bool | false | skip the post-apply admin kubeconfig fetch |
--plan | string | — | apply exactly this saved plan file (from plan --out) instead of re-planning |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
roksbnkctl backend
Configure the per-tool execution backend default (writes config.yaml for you)
Set or inspect the per-tool execution-backend default — the exec: block — from the CLI, instead of hand-editing config.yaml.
show effective backend for each tool, and where it came from
set <tool> <be> persist exec.<tool>.backend (be: local | docker | k8s | ssh:<target>)
unset <tool> remove the override; the tool reverts to its built-in default
The per-invocation --backend flag still wins over whatever is persisted here.
roksbnkctl backend set
Persist a per-tool execution-backend default
roksbnkctl backend set <tool> <backend>
← back to roksbnkctl backend
roksbnkctl backend show
Show the effective execution backend for each tool
← back to roksbnkctl backend
roksbnkctl backend unset
Remove a per-tool backend override (revert to the built-in default)
roksbnkctl backend unset <tool>
← back to roksbnkctl backend
roksbnkctl bnk
BNK trial lifecycle (sits on top of a cluster)
Manage the BNK trial resources (flo, cne_instance, license) as a short-lived layer on top of a shared, durable cluster.
Commands: roksbnkctl bnk up Deploy the BNK trial (auto-provisions the cluster if missing) roksbnkctl bnk down Destroy the BNK trial, leaving the cluster in place
Use roksbnkctl cluster ... to manage the cluster phase itself.
roksbnkctl bnk down
Destroy the BNK trial, leaving the cluster in place
roksbnkctl bnk down [flags]
Destroys only the BNK trial resources, leaving the cluster phase
intact for the next bnk up to attach to. The common iteration loop:
roksbnkctl bnk down && roksbnkctl bnk up # ~5 min trial reset roksbnkctl down # full teardown, ~30 min
Exits 0 (“nothing to do”) when there’s no trial state (cluster-only or
empty), so it’s safe in a reverse-order teardown of every phase. Refuses
on legacy single-state workspaces (use roksbnkctl down there).
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the destroy confirmation |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl bnk
roksbnkctl bnk output
Print the BNK phase’s own terraform outputs (text or –json; [name] = one raw value)
roksbnkctl bnk output [name] [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
--show-sensitive | bool | false | reveal sensitive output values (default redacted) |
← back to roksbnkctl bnk
roksbnkctl bnk status
Live runtime status of the BNK trial phase (namespaces + component readiness)
roksbnkctl bnk status [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
← back to roksbnkctl bnk
roksbnkctl bnk up
Deploy the BNK trial; provisions the cluster first if missing
roksbnkctl bnk up [flags]
Provisions the BNK trial against the existing cluster phase. If
the workspace has no cluster registered yet, bnk up bootstraps the
cluster phase first (with a confirmation prompt — the cluster
provision takes ~30 min) before the trial apply.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip confirmation prompts (cluster-bootstrap + apply) |
--no-kubeconfig | bool | false | skip the post-apply admin kubeconfig fetch |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl bnk
roksbnkctl bnkforge
Configure + drive BNK Forge cluster registration
Configure and drive registration of this workspace’s cluster with a co-located BNK Forge (v3) install — without hand-editing config.yaml.
enable turn on auto-registration on cluster up (writes config.yaml)
disable turn it back off
status show the effective config + readiness
register register this workspace’s cluster with BNK Forge right now
Registration uses BNK Forge v3’s REST API (it has no CLI). It is credential- backed: an IBM Cloud credential template is stored in Forge so it re-derives the kubeconfig on demand — nothing perishable is stored. The password is read from BNK_FORGE_PASSWORD or an interactive prompt (never persisted); the resulting session token is cached in the OS keychain.
roksbnkctl bnkforge disable
Turn off BNK Forge auto-registration (LOCAL only — does not unregister)
Clears the workspace’s auto-register flag, so cluster up stops registering
with BNK Forge.
This is a LOCAL setting and never contacts the server. A cluster already
registered stays on Forge’s Kubernetes page — to remove it, use
roksbnkctl bnkforge unregister.
← back to roksbnkctl bnkforge
roksbnkctl bnkforge enable
Enable BNK Forge auto-registration on cluster up (writes config.yaml for you)
roksbnkctl bnkforge enable [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--forge-ca | string | — | PEM file holding the CA the Forge server’s certificate must chain to (pins a self-signed lab cert; supersedes –insecure) |
--insecure | bool | false | DISABLE TLS verification — the API token is then sent over an unauthenticated connection; prefer –forge-ca |
--project | string | — | target BNK Forge project name (ensured-or-created) |
--url | string | — | BNK Forge server URL (e.g. https://forge.example.com) |
--username | string | — | BNK Forge login username |
← back to roksbnkctl bnkforge
roksbnkctl bnkforge register
Register this workspace’s cluster with BNK Forge now (no re-up needed)
roksbnkctl bnkforge register [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--force | bool | false | take over a cluster registered to ANOTHER Forge project (moves it, and its cluster id changes) |
--forge-ca | string | — | PEM file holding the CA the Forge server’s certificate must chain to (pins a self-signed lab cert; supersedes –insecure) |
--insecure | bool | false | DISABLE TLS verification — the API token is then sent over an unauthenticated connection; prefer –forge-ca |
--password | string | — | BNK Forge password (prefer BNK_FORGE_PASSWORD env or the prompt) |
--project | string | — | target BNK Forge project name (overrides config) |
--url | string | — | BNK Forge server URL (overrides config) |
--username | string | — | BNK Forge login username (overrides config) |
← back to roksbnkctl bnkforge
roksbnkctl bnkforge ssh-credential
Give BNK Forge the SSH private key for an appliance this workspace built
roksbnkctl bnkforge ssh-credential [flags]
Stores an appliance’s SSH private key in BNK Forge and attaches it to the workspace’s project, so Forge can reach the appliance itself.
bnk.flp.vsi.ssh_key names an IBM Cloud VPC key, which puts the PUBLIC half on
the VSI — that is operator access, and it already works. Forge separately needs
the PRIVATE half, and nothing supplied it, so a healthy FLP reports:
infrastructure_private_key_available: false
infrastructure_access_status: recovery_required
Nothing can be recovered there — the credential was never created.
–host must be the address FORGE can reach, which for an FLP is the FLOATING IP.
flp status reports a services-VPC endpoint that Forge sits outside of, so a
credential built from it can never connect.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--expect-fingerprint | string | — | SHA256 fingerprint the key must match (e.g. the VPC key’s) — a credential that cannot log in is worse than none |
--forge-ca | string | — | PEM file holding the CA the Forge server’s certificate must chain to (pins a self-signed lab cert; supersedes –insecure) |
--host | string | — | address FORGE reaches the appliance on — the FLOATING IP, not the endpoint flp status prints |
--insecure | bool | false | DISABLE TLS verification — the API token is then sent over an unauthenticated connection; prefer –forge-ca |
--key | string | — | PRIVATE key file Forge will use (unencrypted; Forge stores it and cannot prompt) |
--name | string | — | credential name in Forge (default: <project>-ssh) |
--password | string | — | BNK Forge password (prefer BNK_FORGE_PASSWORD env or the prompt) |
--port | int | 22 | SSH port on the appliance |
--project | string | — | BNK Forge project to attach the credential to |
--ssh-username | string | ubuntu | SSH user on the appliance |
--url | string | — | BNK Forge server URL (overrides config) |
--username | string | — | BNK Forge login username (overrides config) |
← back to roksbnkctl bnkforge
roksbnkctl bnkforge status
Show this workspace’s BNK Forge registration config + readiness
← back to roksbnkctl bnkforge
roksbnkctl bnkforge unregister
Remove this workspace’s cluster from its BNK Forge project
roksbnkctl bnkforge unregister [flags]
Removes the cluster this workspace registered, so a teardown can undo what
bnkforge register did.
Without it a workspace can create a registration it has no way to remove:
bnkforge disable only clears the local auto-register flag and never contacts
the server, so the cluster stays on Forge’s Kubernetes page pointing at
something that may no longer exist.
Absence is success. No project, no cluster of that name, or a cluster already gone all report and exit 0, so this is safe to run from a destroy path that may run twice or run late.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--forge-ca | string | — | PEM file holding the CA the Forge server’s certificate must chain to (pins a self-signed lab cert; supersedes –insecure) |
--insecure | bool | false | DISABLE TLS verification — the API token is then sent over an unauthenticated connection; prefer –forge-ca |
--password | string | — | BNK Forge password (prefer BNK_FORGE_PASSWORD) |
--project | string | — | BNK Forge project holding the cluster |
--url | string | — | BNK Forge base URL |
--username | string | — | BNK Forge username |
← back to roksbnkctl bnkforge
roksbnkctl cleanup
Delete orphaned IBM Cloud resources left by a failed down
roksbnkctl cleanup [flags]
cleanup sweeps the current workspace’s IBM Cloud account for resources named
after the workspace prefix (<prefix>-*) and deletes them in dependency order.
Use it to recover from a ‘down’ that errored partway and stranded resources (jumphosts, security groups, floating IPs, subnets, VPCs, the Transit Gateway, the registry COS instance, the ROKS cluster, and the BNK trusted profile).
This is a DESTRUCTIVE, best-effort sweep keyed purely on the <prefix>- name
convention. It always lists what it found and asks before deleting (unless
–auto); –dry-run lists without deleting. Re-run if some deletes fail (e.g. a
VPC waiting on an async cluster delete).
A Transit Gateway is deleted only once its connections are detached, and cleanup detaches only connections to VPCs it is also deleting. A gateway still attached to anything else — a VPC under another prefix, a Direct Link, a GRE tunnel — is REFUSED rather than silently disconnected from its other tenants. Re-running will not clear that: detach it yourself, or delete those networks.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--all-regions | bool | false | sweep every IBM Cloud region (slower; catches resources in regions not recorded in config) |
--auto / -y | bool | false | skip the confirmation prompt |
--dry-run | bool | false | list orphaned resources without deleting |
--region | stringSlice | [] | additional region(s) to sweep (repeatable); defaults to the cluster + client regions |
roksbnkctl cluster
ROKS cluster lifecycle (separate from BNK trials)
Manage the ROKS cluster as a durable, reusable resource that sits underneath your BNK trials.
Commands: roksbnkctl cluster up Create the ROKS cluster (+ transit gateway, registry COS) roksbnkctl cluster down Destroy the cluster and everything cluster-scoped roksbnkctl cluster register Discover an already-existing cluster and persist its identity roksbnkctl cluster config Print the recorded cluster identity from cluster-outputs.json roksbnkctl cluster status Live runtime status (endpoints + node readiness)
Each roksbnkctl up against this workspace will reuse the registered
cluster (reading cluster-outputs.json) so multiple BNK trials can share
one cluster.
roksbnkctl cluster config
Print the recorded cluster identity (cluster-outputs.json)
Aliases: show
roksbnkctl cluster config [flags]
Prints the cluster identity recorded at cluster up / register time
(~/.roksbnkctl/<workspace>/cluster-outputs.json): cluster ID, endpoints, VPC,
transit gateway, registry COS, and the create-time settings the cluster was
built with (network mode, and the VPC address block when this tool created the
VPC). This is the RECORDED config, and for the create-time settings it is
AUTHORITATIVE — config.yaml describes what is wanted, this describes what
exists. For live runtime state (node readiness) use roksbnkctl cluster status.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
← back to roksbnkctl cluster
roksbnkctl cluster down
Destroy the cluster phase (ROKS + cluster-shared services)
roksbnkctl cluster down [flags]
Tears down everything roksbnkctl cluster up created. Refuses to run
if any BNK trial state exists for this workspace — destroy those first
with roksbnkctl down to avoid orphaned BNK resources.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the destroy confirmation |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl cluster
roksbnkctl cluster output
Print the Cluster phase’s own terraform outputs (text or –json; [name] = one raw value)
roksbnkctl cluster output [name] [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
--show-sensitive | bool | false | reveal sensitive output values (default redacted) |
← back to roksbnkctl cluster
roksbnkctl cluster register
Discover an existing ROKS cluster and persist its identity
roksbnkctl cluster register [cluster-name-or-id] [flags]
Looks up an existing ROKS cluster in your IBM Cloud account,
verifies its registry COS instance exists, and writes the cluster’s
identity to ~/.roksbnkctl/<workspace>/cluster-outputs.json.
Subsequent roksbnkctl up runs in this workspace will pick up the
registered cluster automatically — no need to repeat its identity in
trial tfvars.
The registry COS instance is found by probing the names roksbnkctl and the
upstream HCL actually use: “<prefix>-registry-cos” (what cluster up
creates), “<cluster>-registry-cos”, then the upstream fallback
“<cluster>-cos”. Pass –registry-cos-name to override (e.g. if your tfvars
sets roks_cos_instance_name to something else).
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--prompt | bool | false | prompt for the cluster name even if one is given as an argument |
--registry-cos-name | string | — | registry COS instance name (default: probe “<prefix>-registry-cos”, “<cluster>-registry-cos”, “<cluster>-cos”) |
← back to roksbnkctl cluster
roksbnkctl cluster status
Live runtime status of the Cluster phase (endpoints + node readiness)
roksbnkctl cluster status [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
← back to roksbnkctl cluster
roksbnkctl cluster up
Provision the ROKS cluster (and cluster-shared services) only
roksbnkctl cluster up [flags]
Creates the durable cluster-shared infrastructure only — the ROKS
cluster, transit gateway, and registry COS. cert-manager and the BNK trial
modules (flo, cne_instance, license) deploy in the BNK phase; the jumphosts
deploy in the testing phase. On success, writes the cluster’s identity to
~/.roksbnkctl/<workspace>/cluster-outputs.json so subsequent roksbnkctl up
runs can deploy the BNK and testing phases onto this cluster.
Uses a separate state directory (~/.roksbnkctl/<workspace>/state-cluster/)
so it doesn’t tangle with BNK or testing state.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the confirmation prompt before apply |
--no-kubeconfig | bool | false | skip the post-apply admin kubeconfig fetch |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl cluster
roksbnkctl config
Print an example config.yaml or .env, or convert one into the other
Prints the workspace input in either form, for piping to a file.
With no –from flag it prints an ANNOTATED template — every setting with the comment explaining it, which is what you want when starting from nothing:
roksbnkctl config yaml > config.yaml roksbnkctl config env > .env
With –from-yaml or –from-env it prints the POPULATED equivalent, without comments — the same settings expressed in the other form:
roksbnkctl config env –from-yaml config.yaml > .env roksbnkctl config yaml –from-env .env > config.yaml
The conversion carries what the input actually sets. A setting left at its default is omitted rather than written out, so the result stays a statement of intent instead of a snapshot of every default at the moment it ran – except the handful of REQUIRED fields, which are emitted empty so a missing one is visible rather than silently absent.
roksbnkctl config env
Print an annotated .env of ROKSBNKCTL_* overrides, or the .env equivalent of a config.yaml
roksbnkctl config env [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--from-env | string | — | read settings from a .env file and print the populated equivalent (no comments) |
--from-yaml | string | — | read settings from a config.yaml and print the populated equivalent (no comments) |
← back to roksbnkctl config
roksbnkctl config yaml
Print an annotated config.yaml, or the config.yaml equivalent of a .env
roksbnkctl config yaml [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--from-env | string | — | read settings from a .env file and print the populated equivalent (no comments) |
--from-yaml | string | — | read settings from a config.yaml and print the populated equivalent (no comments) |
← back to roksbnkctl config
roksbnkctl cos
Manage IBM Cloud Object Storage (instances, buckets, objects)
roksbnkctl cos provides full CRUD on the COS supply chain BNK depends on: COS instances (via Resource Controller), buckets, and keyed objects (FAR pull keys, JWT licenses, etc.). All calls go through the IBM Go SDKs — no ibmcloud CLI dependency.
roksbnkctl cos bucket
Manage COS buckets
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--instance | string | — | COS instance name or CRN (required) |
← back to roksbnkctl cos
roksbnkctl cos bucket create
Create a bucket on the named instance
roksbnkctl cos bucket create <bucket> [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--class | string | standard | storage class (standard, vault, cold, smart) |
--region | string | — | bucket region (default: workspace region) |
← back to roksbnkctl cos bucket
roksbnkctl cos bucket delete
Delete a bucket (must be empty)
roksbnkctl cos bucket delete <bucket>
← back to roksbnkctl cos bucket
roksbnkctl cos bucket get
Recursively download every object in a bucket to a local directory
roksbnkctl cos bucket get <bucket> <local-dir> [flags]
Recursively download every object in <bucket> to <local-dir>.
<local-dir> is created if it does not exist (mkdir -p semantics, mode
0755). Object keys map to nested subdirectories — a key foo/bar/baz.json
lands at <local-dir>/foo/bar/baz.json. Streaming download per object
(no whole-object in-memory buffering); text and binary are both copied
through verbatim.
Default behavior is overwrite (the operator just asked to download); pass –no-clobber to skip objects whose local target already exists, matching cp -n.
With –output json, emits one JSON object per file completed: {“key”,“local_path”,“size”,“outcome”}. The final counts line goes to stderr in text mode; the JSON stream concludes with one summary record of shape {“counts”:{…}}.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--no-clobber | bool | false | skip objects whose local target already exists (cp -n semantics) |
← back to roksbnkctl cos bucket
roksbnkctl cos bucket list
List buckets on the named instance
← back to roksbnkctl cos bucket
roksbnkctl cos instance
Manage COS instances (service instances under Resource Controller)
← back to roksbnkctl cos
roksbnkctl cos instance create
Create a COS instance
roksbnkctl cos instance create <name> [flags]
Create a COS service instance under the workspace’s resource group.
–plan accepts a friendly name (standard | lite); –plan-id takes a catalog UUID directly when IBM ships a tier roksbnkctl hasn’t mapped yet. –target defaults to “global” (COS instances are global; buckets carry the regional affinity).
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--plan | string | standard | service plan name (standard | lite) |
--plan-id | string | — | service plan UUID (overrides –plan; for plans roksbnkctl hasn’t mapped) |
--target | string | global | target region (default: global; COS instances are global) |
← back to roksbnkctl cos instance
roksbnkctl cos instance delete
Delete a COS instance (and its bound resources unless –no-recursive)
roksbnkctl cos instance delete <name> [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the confirmation prompt |
--no-recursive | bool | false | do NOT delete bound resources (HMAC keys, service credentials) |
← back to roksbnkctl cos instance
roksbnkctl cos instance list
List COS instances in the current account
← back to roksbnkctl cos instance
roksbnkctl cos object
Manage objects in COS buckets
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--instance | string | — | COS instance name or CRN (required) |
← back to roksbnkctl cos
roksbnkctl cos object delete
Delete an object
roksbnkctl cos object delete <bucket>/<key>
← back to roksbnkctl cos object
roksbnkctl cos object get
Download an object (streaming)
roksbnkctl cos object get <bucket>/<key> <local-file>
← back to roksbnkctl cos object
roksbnkctl cos object list
List objects (optionally under a prefix)
roksbnkctl cos object list <bucket>[/<prefix>]
← back to roksbnkctl cos object
roksbnkctl cos object put
Upload an object (multipart for large files, streaming)
roksbnkctl cos object put <bucket>/<key> <local-file>
← back to roksbnkctl cos object
roksbnkctl doctor
Check prerequisites and report missing pieces
roksbnkctl doctor [flags]
Verifies the host has what roksbnkctl needs.
Required (hard fail on missing):
- terraform on PATH (the local backend’s workhorse for
roksbnkctl up)
Informational (the binary internalises each surface; missing → no warning):
- kubectl / oc — internalised via client-go (
roksbnkctl k *) - ibmcloud — bundled image, run via –backend docker / –backend ssh:
<target> - iperf3 — bundled image, run via –backend k8s
- dig — DNS probe internalised via miekg/dns
A stock dev box with only terraform installed should produce exit 0
and zero warnings.
Pass –target <name> to additionally probe an SSH target (runs whoami).
Pass –backend k8s | ssh:<target> for per-backend prereq checks.
Exits non-zero only when a required check fails (warnings don’t block).
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--backend | string | — | additionally run per-backend checks: k8s | ssh:<target> |
--target | string | — | additionally probe the named SSH target with whoami |
roksbnkctl down
Destroy everything in the workspace — terraform destroy
roksbnkctl down [flags]
Tears down the workspace’s phases in reverse-dependency order: BNK and Testing in parallel, then the cluster. Two behaviours worth knowing:
- If the cluster was attached to an EXISTING (shared) Transit Gateway, down auto-detaches that connection first — removing only this cluster’s connection; the shared gateway and every other cluster’s connection are left intact.
- If this workspace CREATED a VPC that another cluster’s subnets still live in (the shared-VPC topology), down refuses: the VPC owner must be torn down LAST, so tear the workspaces sharing the VPC down first.
The Gateway and FLP phases are separate and optional; if present, tear them down first (down reports which command to run).
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the destroy confirmation |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
roksbnkctl exec
Run a single command with cluster context loaded
roksbnkctl exec [command...]
roksbnkctl flp
F5 License Proxy phase — optional in-cluster licensing proxy
Manage the F5 License Proxy (FLP) as an optional, independent phase on an existing cluster: it deploys the f5-license-proxy chart (vault + postgresql + proxy) pulling from the configured registry (Harbor mirror or FAR), and records its root CA + service endpoint so a subsequent BNK install can license in FLP mode. Runs in its own state (state-flp/), separate from cluster/BNK/testing/gateway.
Commands: roksbnkctl flp up Install the F5 License Proxy (needs a cluster) roksbnkctl flp down Remove the F5 License Proxy, leaving the other phases intact
Typical flow (FLP-licensed BNK):
roksbnkctl cluster up # or cluster register for an existing cluster
roksbnkctl registry replicate # mirror FAR into your registry (optional)
roksbnkctl flp up # install the proxy
roksbnkctl bnk up # with bnk.license_mode: f5licenseproxy
FLP is opt-in: without it, BNK licenses with a subscription JWT as before.
roksbnkctl flp down
Remove the F5 License Proxy, leaving the other phases
roksbnkctl flp down [flags]
Destroys only the FLP-phase resources (state-flp/), leaving cluster, BNK, testing and gateway intact, and clears flp-outputs.json.
Exits 0 (“nothing to do”) when there’s no FLP state, so it’s safe in a reverse-order teardown of every phase.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the destroy confirmation |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl flp
roksbnkctl flp output
Print the FLP phase’s own terraform outputs (text or –json; [name] = one raw value)
roksbnkctl flp output [name] [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
--show-sensitive | bool | false | reveal sensitive output values (default redacted) |
← back to roksbnkctl flp
roksbnkctl flp status
Show the F5 License Proxy’s live status (services, listener, F5/TEEM, CNE fields)
roksbnkctl flp status [flags]
Fetches /api/status from the FLP’s status service (the flp-status container, which runs alongside the proxy on the VSI pod or in the cluster) and renders it.
The service URL is derived from the workspace’s flp-outputs.json (the FLP endpoint host, port 80) unless –url is given. Output honors -o json.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--url | string | — | flp-status base URL (default: derived from the FLP endpoint, port 80) |
← back to roksbnkctl flp
roksbnkctl flp up
Install the F5 License Proxy (a cluster must exist)
roksbnkctl flp up [flags]
Deploys the F5 License Proxy into the existing cluster (read from
cluster-outputs.json) in its own state (state-flp/), and writes flp-outputs.json
(root CA + endpoint) for a later bnk up in f5licenseproxy mode.
Refuses when no cluster exists yet.
With –add-node-port-access the proxy is also reachable from OUTSIDE its own cluster, so a BNK install in a DIFFERENT cluster (same VPC, or across a transit gateway) can license through it — a “shared licensing cluster”, where only the cluster running the proxy needs egress to F5. That additionally puts the worker node IPs in the proxy’s server certificate (without them the remote cluster’s controller rejects the TLS handshake) and records an external endpoint in flp-outputs.json. Feed that endpoint + CA to the other workspace via its bnk.flp.external block.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--add-node-port-access | bool | false | expose the proxy outside its cluster (NodePort + node-IP cert SANs) so a BNK install in another cluster can license through it |
--auto | bool | false | skip the confirmation prompt before apply |
--node-port-source-cidr | stringSlice | [] | with –add-node-port-access: open the NodePort on the worker security group to this CIDR. REPEATABLE — a multi-zone VPC has one address prefix PER ZONE, and a consuming pod scheduled in an unlisted zone is silently dropped. Pass every zone’s CIDR (or a supernet covering them). |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl flp
roksbnkctl gateway
Gateway-phase (data-plane config) lifecycle — optional ingress/egress setup
Manage the BNK data-plane configuration as an optional, independent phase on top of a healthy BNK deployment: the Gateway API objects (GatewayClass, F5BnkGateway, Gateway, HTTPRoute), the egress SnatPool + Egress CRs, the per-zone static routes, and the cluster security-group VXLAN rule. Runs in its own state (state-gateway/), separate from cluster/BNK/testing.
Commands: roksbnkctl gateway up Apply the data-plane config (needs a healthy BNK) roksbnkctl gateway down Remove the data-plane config, leaving cluster/BNK/testing intact
Run this only after roksbnkctl up has brought up the cluster + BNK and
the BNK is healthy (TMM pods Ready). The composite up/down never runs
this phase — it is opt-in.
roksbnkctl gateway down
Remove the BNK data-plane config, leaving cluster/BNK/testing
roksbnkctl gateway down [flags]
Destroys only the Gateway-phase resources (state-gateway/), leaving the cluster, BNK and testing phases intact.
Exits 0 (“nothing to do”) when there’s no gateway state, so it’s safe in a reverse-order teardown of every phase.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the destroy confirmation |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl gateway
roksbnkctl gateway output
Print the Gateway phase’s own terraform outputs (text or –json; [name] = one raw value)
roksbnkctl gateway output [name] [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
--show-sensitive | bool | false | reveal sensitive output values (default redacted) |
← back to roksbnkctl gateway
roksbnkctl gateway status
Runtime status of the Gateway phase (CR readiness + assigned address)
roksbnkctl gateway status [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
← back to roksbnkctl gateway
roksbnkctl gateway up
Apply the BNK data-plane config (BNK must be healthy)
roksbnkctl gateway up [flags]
Applies the Gateway API + SnatPool + Egress + static-route CRs and the VXLAN security-group rule against the existing cluster + BNK (read from cluster-outputs.json). Runs in its own state (state-gateway/).
Refuses when no cluster/BNK exists yet.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the confirmation prompt before apply |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl gateway
roksbnkctl get
Get one or more resources (pods, nodes, services, CRDs, …)
roksbnkctl get <resource> [name] [-n <ns> | -A] [-l <selector>] [-o <fmt>] [flags]
Fetches Kubernetes resources via client-go, no host kubectl required.
The resource argument accepts plurals, singulars, and short names from RESTMapper (pods/pod/po, services/svc, deployments/deploy). Multiple types can be comma-separated:
roksbnkctl k get pods,services -n f5-bnk roksbnkctl k get nodes -o yaml roksbnkctl k get pods -A -l app.kubernetes.io/name=f5-lifecycle-operator roksbnkctl k get pod my-pod -n default -o jsonpath=‘{.status.phase}’
CRDs work via dynamic discovery without a hardcoded list — roksbnkctl k get cneinstances resolves the same way kubectl does.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--all-namespaces / -A | bool | false | list across all namespaces |
--namespace / -n | string | — | namespace scope (default: current-context’s namespace) |
--output / -o | string | — | output format: yaml | json | wide | name | jsonpath=… | go-template=… |
--selector / -l | string | — | label selector (e.g. ‘app=foo,tier!=cache’) |
roksbnkctl ibmcloud
Passthrough to local ibmcloud with workspace API key + region loaded
roksbnkctl ibmcloud [args...]
roksbnkctl init
Interactive setup; writes the workspace config.yaml
roksbnkctl init [flags]
roksbnkctl init walks through the prompts (region, resource group, cluster,
BNK version) and writes ~/.roksbnkctl/<workspace>/config.yaml.
On first run with no -w flag, creates and uses the ‘default’ workspace. Re-run with –upgrade-tf to bump the pinned Terraform source to its latest release.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--config-file | string | — | path or http(s) URL to a workspace config.yaml to seed (non-interactive when complete; see init example) |
--non-interactive | bool | false | build config.yaml from environment variables ALONE — no prompts, no –config-file (for argv+env runners; pair with the ROKSBNKCTL_* env vars) |
--override-from-env | bool | false | after seeding, overlay config.yaml fields from environment variables (e.g. IBMCLOUD_API_KEY → ibmcloud.api_key_b64) |
--tf-source | string | — | override TF source (path or URL); relative local paths are resolved to absolute before being pinned into config.yaml |
--upgrade-tf | bool | false | resolve and pin the latest TF release into config.yaml |
roksbnkctl init example
Print an annotated example config.yaml to stdout (a template, or for piping)
Writes an annotated config.yaml to stdout — the canonical declarative input, with the required fields filled and every optional axis documented (cluster create-or-attach, BYO infrastructure reuse, BNK install, gateway, registry mirror, remote state). config.yaml is the single input to a workspace; the embedded terraform is internal.
Create a starting config or inspect the knobs with ordinary pipes:
roksbnkctl init example > config.yaml roksbnkctl -w demo init –config-file config.yaml roksbnkctl init example | grep -E ‘manifest_version|cluster_vpc’
← back to roksbnkctl init
roksbnkctl install
Copy the running roksbnkctl binary into a directory on PATH
roksbnkctl install [flags]
Install the roksbnkctl binary you’re currently running into a directory
on $PATH so you can invoke it as roksbnkctl from any working
directory.
Default destination: Linux/macOS, in order of preference: $HOME/.local/bin (preferred — typically writable without sudo) $HOME/bin (older convention; still on PATH for some setups) /usr/local/bin (system-wide; usually needs sudo) Windows: a writable directory already on %PATH% — preferring %LOCALAPPDATA%\Microsoft\WindowsApps (on the per-user PATH by default, no admin) — so the binary resolves immediately. Falls back to %LOCALAPPDATA%\Programs\roksbnkctl (with a PATH hint) if none is usable.
Override the destination with –dir.
Idempotent: if the running binary already lives at the destination, prints a message and exits 0. Use –force to overwrite (useful right after a local rebuild that landed at the install path).
Examples: roksbnkctl install # default — ~/.local/bin roksbnkctl install –dir ~/bin # specific user dir sudo roksbnkctl install –dir /usr/local/bin # system-wide
Note: this is distinct from roksbnkctl self update, which
pulls the latest GitHub release tarball over the network.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--dir | string | — | destination directory (default: a PATH dir — ~/.local/bin on Unix, %LOCALAPPDATA%\Microsoft\WindowsApps on Windows) |
--force | bool | false | overwrite even if destination resolves to the running binary |
roksbnkctl journal
Append-only timeline + report for agentic-mode handoffs
Manage the workspace journal (<workspace>/journal/), the append-only
timeline the agentic-mode personas use to hand off to one another.
roksbnkctl journal add “<note>” Append a note to today’s journal
roksbnkctl journal list List entries chronologically with summaries
roksbnkctl journal report Assemble report.md from decisions + journal
roksbnkctl journal add
Append a note to today’s journal entry
roksbnkctl journal add <message>
← back to roksbnkctl journal
roksbnkctl journal list
List journal entries (chronological) with one-line summaries
← back to roksbnkctl journal
roksbnkctl journal report
Assemble report.md from decisions.md + the journal timeline
← back to roksbnkctl journal
roksbnkctl k
Kubernetes verbs (kubectl-internalised; no host kubectl required)
roksbnkctl k <verb> runs the BNK-relevant kubectl/oc verb subset
natively in-process via client-go, with no host kubectl/oc binary
required. Output formatting matches kubectl byte-for-byte for
-o yaml/json/wide/name/jsonpath/go-template.
Verbs: k get list/show resources k describe human-friendly resource detail (delegates to kubectl/pkg/describe) k apply server-side apply for files, dirs, kustomize bases, or stdin k delete delete with cascade + grace period control k logs pod or component logs (extends roksbnkctl logs) k exec exec into a pod via SPDY k port-forward forward a local port to a pod via SPDY
The existing roksbnkctl kubectl / roksbnkctl oc passthroughs remain as escape hatches for verbs not internalised here (edit, patch, rollout, scale, etc.) — they require kubectl/oc on PATH.
roksbnkctl k apply
Server-side apply YAML/JSON manifests, directories, or kustomize bases
roksbnkctl k apply -f <file-or-dir> [-n <ns>] [--force] [flags]
Server-side apply with field-manager ‘roksbnkctl’.
-f <file> single YAML/JSON file (multi-doc YAML supported)
-f <dir> directory: kustomization.yaml-detected → krusty build;
otherwise recursive *.yaml / *.yml
-f - stdin (multi-doc YAML)
–force passes through to SSA’s force-conflicts flag, identical to kubectl apply –server-side –force-conflicts.
Examples:
roksbnkctl k apply -f deploy.yaml -n f5-bnk roksbnkctl k apply -f manifests/ cat deploy.yaml | roksbnkctl k apply -f -
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--filename / -f | string | — | file, directory, or ‘-’ for stdin |
--force | bool | false | force-conflicts on server-side apply (kubectl apply –force-conflicts) |
--namespace / -n | string | — | namespace for namespaced resources without an explicit namespace field |
← back to roksbnkctl k
roksbnkctl k delete
Delete resources by name or label selector
roksbnkctl k delete <resource> [name] [-n <ns> | -A] [-l <selector>] [--force] [--grace-period N] [--cascade orphan|background|foreground] [flags]
Deletes resources via the dynamic client. Cascade options match kubectl’s:
–cascade=background delete the object; controller cleans dependents async (default) –cascade=foreground block until dependents are gone –cascade=orphan delete only the object, leave dependents
Examples:
roksbnkctl k delete pod my-pod -n f5-bnk roksbnkctl k delete pods -l app=stale –force –grace-period=0 roksbnkctl k delete deployment foo –cascade=foreground
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--all-namespaces / -A | bool | false | delete across all namespaces |
--cascade | string | background | cascade: orphan|background|foreground |
--force | bool | false | force-delete: implies –grace-period=0 unless overridden |
--grace-period | int | -1 | graceful termination period (seconds); -1 = use resource default |
--namespace / -n | string | — | namespace scope |
--selector / -l | string | — | label selector |
← back to roksbnkctl k
roksbnkctl k describe
Show detailed human-readable resource info (events, conditions, related objects)
roksbnkctl k describe <resource> [name] [-n <ns> | -A] [-l <selector>] [flags]
Delegates to k8s.io/kubectl/pkg/describe — the same library kubectl/oc use internally, so output is byte-equivalent.
Examples:
roksbnkctl k describe pod my-pod -n f5-bnk roksbnkctl k describe nodes roksbnkctl k describe deployment f5-lifecycle-operator -n f5-bnk
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--all-namespaces / -A | bool | false | describe across all namespaces |
--namespace / -n | string | — | namespace scope |
--selector / -l | string | — | label selector |
--show-events | bool | true | include the Events block (kubectl default: true) |
← back to roksbnkctl k
roksbnkctl k exec
Exec into a pod via SPDY (kubectl-equivalent in-process)
roksbnkctl k exec <pod> [-n <ns>] [-c <container>] [-i] [-t] -- <cmd> [args...] [flags]
Opens an exec stream against the named pod over SPDY. The semantics mirror kubectl exec:
-i / –stdin attach stdin to the remote process -t / –tty allocate a PTY (use for top, bash-style interactive work) -c / –container pick a container in a multi-container pod
Examples:
roksbnkctl k exec my-pod – ls /tmp roksbnkctl k exec my-pod -it – bash roksbnkctl k exec my-pod -c sidecar – cat /etc/hostname
Note: this is the cluster-side exec. The host-side equivalent is
‘roksbnkctl exec <cmd>’ — distinct on purpose (PRD 02 §“Disambiguating
roksbnkctl exec”, Option B).
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--container / -c | string | — | container name in a multi-container pod |
--namespace / -n | string | — | namespace scope (default: default) |
--stdin / -i | bool | false | attach stdin |
--tty / -t | bool | false | allocate a PTY |
← back to roksbnkctl k
roksbnkctl k get
Get one or more resources (pods, nodes, services, CRDs, …)
roksbnkctl k get <resource> [name] [-n <ns> | -A] [-l <selector>] [-o <fmt>] [flags]
Fetches Kubernetes resources via client-go, no host kubectl required.
The resource argument accepts plurals, singulars, and short names from RESTMapper (pods/pod/po, services/svc, deployments/deploy). Multiple types can be comma-separated:
roksbnkctl k get pods,services -n f5-bnk roksbnkctl k get nodes -o yaml roksbnkctl k get pods -A -l app.kubernetes.io/name=f5-lifecycle-operator roksbnkctl k get pod my-pod -n default -o jsonpath=‘{.status.phase}’
CRDs work via dynamic discovery without a hardcoded list — roksbnkctl k get cneinstances resolves the same way kubectl does.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--all-namespaces / -A | bool | false | list across all namespaces |
--namespace / -n | string | — | namespace scope (default: current-context’s namespace) |
--output / -o | string | — | output format: yaml | json | wide | name | jsonpath=… | go-template=… |
--selector / -l | string | — | label selector (e.g. ‘app=foo,tier!=cache’) |
← back to roksbnkctl k
roksbnkctl k logs
Stream pod logs (kubectl-equivalent direct path)
roksbnkctl k logs <pod-name> [-n <ns>] [-c <container>] [-f] [--previous] [--since 5m] [--tail N] [flags]
Streams logs for a named pod. Differs from the top-level
‘roksbnkctl logs <component>’ in that this takes a literal pod name —
matching kubectl’s surface — while the component variant maps a known
BNK component name to a label selector.
Both forms honour -n, -c, -f, –previous, –since, –tail.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--container / -c | string | — | container name in a multi-container pod |
--follow / -f | bool | false | follow log output |
--namespace / -n | string | — | namespace scope (default: default) |
--previous | bool | false | fetch logs from the previous container instance |
--since | string | — | only return logs newer than this duration (e.g. 5s, 2m, 1h) |
--tail | int64 | -1 | tail the last N lines (-1 = full log) |
← back to roksbnkctl k
roksbnkctl k port-forward
Forward local port(s) to a pod via SPDY
Aliases: port_forward
roksbnkctl k port-forward <pod> [-n <ns>] <local-port>[:<remote-port>] [...] [flags]
Forwards one or more local TCP ports to ports on the named pod. Equivalent to kubectl port-forward; signal handling closes the tunnel cleanly on Ctrl+C.
Port spec:
- 8080 local 8080 → pod 8080 8080:80 local 8080 → pod 80
- 80 ephemeral local port → pod 80
Examples:
roksbnkctl k port-forward my-pod 8080:80 roksbnkctl k port-forward my-pod -n f5-bnk 9090:9090 8080:80
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--namespace / -n | string | — | namespace scope (default: default) |
← back to roksbnkctl k
roksbnkctl kubeconfig
Print the kubeconfig path (or contents with –export)
roksbnkctl kubeconfig [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--cluster | string | — | cluster name or ID for –download (default: workspace cluster.name) |
--download | bool | false | fetch admin kubeconfig from IBM Cloud and save to ~/.kube/config |
--export | bool | false | print kubeconfig contents instead of path |
--refresh | bool | false | force-refresh the forge kubeconfig now (re-fetch the admin client certs) |
roksbnkctl kubectl
Passthrough to local kubectl with workspace KUBECONFIG loaded
roksbnkctl kubectl [args...]
roksbnkctl logs
Tail logs for a BNK component (flo, cis, cert-manager, cneinstance)
roksbnkctl logs <component> [flags]
Looks up the named BNK component, finds its pod(s) by label, and streams logs to stdout. With –follow, streams live. With multiple matching pods, tails the first and prints a hint about using roksbnkctl kubectl for per-pod selection.
The component → namespace/selector map is hardcoded for v1 against the upstream TF chart’s default labels; if your install renamed namespaces or relabelled, fall back to:
roksbnkctl kubectl logs -n <ns> <pod>
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--container / -c | string | — | container name in a multi-container pod |
--follow / -f | bool | false | follow log output |
--namespace / -n | string | — | override the component’s default namespace |
--previous | bool | false | fetch logs from the previous container instance |
--since | string | — | only return logs newer than this duration (e.g. 5s, 2m, 1h) |
--tail | int64 | -1 | tail the last N lines (-1 = full log) |
roksbnkctl oc
Passthrough to local oc with workspace KUBECONFIG loaded
roksbnkctl oc [args...]
roksbnkctl ops
Manage the in-cluster ops pod (k8s execution backend)
roksbnkctl ops manages the long-lived ops pod the k8s execution backend exec’s tools into. The pod runs in the roksbnkctl-ops namespace with a least-privilege ServiceAccount + ClusterRole, and gets its IBM Cloud API key from a Secret apply-time-templated from the workspace credential.
Subcommands: install apply the embedded manifests (idempotent) show print pod + Secret + RBAC status uninstall delete every roksbnkctl.io/managed object created by install
roksbnkctl ops install
Apply (or update) the in-cluster ops fixtures
roksbnkctl ops install [flags]
Applies the embedded namespaces, ServiceAccount, Secret, ClusterRole, ClusterRoleBinding, and ops Pod. Idempotent: re-running with a new API key updates the Secret and rolls the Pod.
Credential mode is selected via –trusted-profile (auto|on|off):
auto (default) — provision an IBM Cloud IAM trusted profile linked to the ops pod’s ServiceAccount when the resolved API key has ‘iam-identity’ perms; otherwise fall back to the static-key Secret with a stderr warning. on — require the trusted-profile path; fail loudly if perms don’t allow. off — skip the trusted-profile path; install the v1.0.x static-key Secret.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--trusted-profile | string | auto | IBM IAM trusted profile mode: auto (default; provision when perms allow, fall back to static-key Secret), on (require trusted profile), off (static-key Secret only) |
← back to roksbnkctl ops
roksbnkctl ops show
Print the ops pod’s status, image, RBAC subject, and Secret rotation timestamp
← back to roksbnkctl ops
roksbnkctl ops uninstall
Delete the ops fixtures (namespaces, RBAC, Pod, Secret)
roksbnkctl ops uninstall [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--confirm | bool | false | actually perform the uninstall (otherwise prints what would be deleted) |
← back to roksbnkctl ops
roksbnkctl output
Print the merged outputs across all phases (cluster + bnk + testing + gateway + flp)
roksbnkctl output [name] [flags]
Print the union of every phase’s own outputs — each read from its owning
phase’s state, so values are the populated ones and never conflict. This is the
“everything” view; the per-phase <phase> output commands scope to just that
phase’s managed attributes.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
--show-sensitive | bool | false | reveal sensitive output values (default redacted) |
roksbnkctl plan
Read-only; show what roksbnkctl up would change
roksbnkctl plan [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--out | string | — | save the plan to <file> (binary plan + a readable <file>.txt) for later apply --plan |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
roksbnkctl registry
Air-gap registry mirror — replicate BNK artifacts into a private registry (PRD 11)
Manage the air-gap registry mirror: replicate every chart + image a BNK install needs (the f5-bigip-k8s-manifest set plus the non-F5 dependencies) from the F5 Artifact Repository (repo.f5.com) into a private target — IBM Container Registry (ICR) or a generic OCI registry (Artifactory / Harbor / Quay) — so an air-gapped cluster installs BNK from a registry it controls.
Commands:
roksbnkctl registry target Show or set the mirror target (icr|generic)
roksbnkctl registry bom Build + print the bill-of-materials
roksbnkctl registry list List the artifacts the last replicate recorded
roksbnkctl registry diff Show what replicate would copy (BOM vs. mirror)
roksbnkctl registry replicate Copy the BOM into the mirror (registry-to-registry; no cluster)
roksbnkctl registry verify Confirm every BOM artifact is present + digest-matched
roksbnkctl registry prune Remove mirrored artifacts no longer in the BOM
roksbnkctl registry delete Delete ALL replicated artifacts from the target
registry bom works entirely offline against the FAR manifest. The other verbs
need a configured registry: block and network reachability to the target registry —
replicate also needs the FAR source (repo.f5.com). NONE require a Kubernetes
cluster: replicate copies registry-to-registry (via go-containerregistry) from wherever
roksbnkctl runs, so you can pre-seed the mirror as a standalone supply-chain step before
any cluster exists.
roksbnkctl registry adopt
Record a mirror this workspace did not populate, so bnk up can use it
roksbnkctl registry adopt [flags]
Writes registry-mirror.json for a mirror that already exists.
bnk up refuses to render against a mirror the workspace has no record of —
otherwise BNK would be pointed at far_repo_url, which an air-gapped cluster cannot
reach. Until now only registry replicate wrote that record, which means a
workspace could only use a mirror it had populated itself.
That is the wrong constraint for how mirrors are actually used. The registry is filled once, as a supply-chain step, and then many installs pull from it — often from a different workspace, a different host, or a different team. Those installs were forced to re-run replicate purely to re-derive a record, which needs the FAR source reachable at install time. An air-gapped operator frequently does not have that, and it is the whole point of having mirrored in the first place.
adopt derives the record from the configured registry target: the chart and image hosts, the repo namespace, the manifest version, and the mirror CA all come from the workspace config, so no source access is needed. It then asks the MIRROR what it holds under the configured prefix — a sanity check that catches a typo in the prefix or an empty registry, without pretending to prove the contents are correct.
Pass –verify-contents when the source IS reachable and you want proof rather than assertion: it builds the BOM and digest-checks every artifact before recording, and the record then carries the full artifact inventory.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--force | bool | false | record the mirror even when it holds nothing under the configured prefix |
--verify-contents | bool | false | digest-check every BOM artifact before recording (needs the FAR source) |
← back to roksbnkctl registry
roksbnkctl registry bom
Build + print the BNK bill-of-materials from the FAR manifest
roksbnkctl registry bom [flags]
Pulls the f5-bigip-k8s-manifest for the configured BNK release, parses it into the full artifact set (charts + images), unions the non-F5 dependencies, and prints the resulting BOM. Needs no target — it only reads from FAR.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--far-repo-url | string | — | FAR registry host (default: workspace bnk.far_repo_url, else repo.f5.com) |
--include-deps | bool | false | force-include the non-F5 dependency artifacts (cert-manager, node-labeler) |
--json | bool | false | emit the BOM as JSON (overrides –output) |
--manifest-version | string | — | BNK manifest version (default: workspace bnk.manifest_version) |
--no-include-deps | bool | false | exclude the non-F5 dependency artifacts |
--source-sa-b64 | string | — | FAR _json_key_base64 service account (default: workspace registry.source_service_account_b64) |
← back to roksbnkctl registry
roksbnkctl registry delete
Delete ALL replicated artifacts from the target registry
roksbnkctl registry delete [flags]
Removes every artifact roksbnkctl replicated (recorded in registry-mirror.json) from the configured target, then clears the mirror record so the BNK install reverts to pulling from FAR. Destructive — pass –force to skip the confirmation.
Deletion is by digest where recorded (the reliable form for a registry manifest DELETE). Artifacts that fail to delete are kept in the record so a re-run retries them. For target=icr the API key needs Manager (delete) rights on the namespace; for target=generic the registry must have deletes enabled.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--force | bool | false | skip the confirmation prompt |
--target | string | — | mirror target backend: icr|generic (default: workspace registry.target, else “icr”) |
← back to roksbnkctl registry
roksbnkctl registry diff
Show BOM vs. mirror drift (what replicate would copy)
roksbnkctl registry diff [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--far-repo-url | string | — | FAR registry host (default: workspace bnk.far_repo_url, else repo.f5.com) |
--include-deps | bool | false | force-include the non-F5 dependency artifacts (cert-manager, node-labeler) |
--manifest-version | string | — | BNK manifest version (default: workspace bnk.manifest_version) |
--no-include-deps | bool | false | exclude the non-F5 dependency artifacts |
--source-sa-b64 | string | — | FAR _json_key_base64 service account (default: workspace registry.source_service_account_b64) |
← back to roksbnkctl registry
roksbnkctl registry list
List artifacts currently recorded in the mirror
Lists the artifacts the last replicate recorded in registry-mirror.json.
← back to roksbnkctl registry
roksbnkctl registry prune
Remove mirrored artifacts no longer in the BOM
roksbnkctl registry prune [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--far-repo-url | string | — | FAR registry host (default: workspace bnk.far_repo_url, else repo.f5.com) |
--include-deps | bool | false | force-include the non-F5 dependency artifacts (cert-manager, node-labeler) |
--manifest-version | string | — | BNK manifest version (default: workspace bnk.manifest_version) |
--no-include-deps | bool | false | exclude the non-F5 dependency artifacts |
--source-sa-b64 | string | — | FAR _json_key_base64 service account (default: workspace registry.source_service_account_b64) |
--target | string | — | mirror target backend: icr|generic (default: workspace registry.target, else “icr”) |
← back to roksbnkctl registry
roksbnkctl registry replicate
Copy the BOM into the mirror (registry-to-registry; no cluster needed)
roksbnkctl registry replicate [flags]
Prepares the target registry (auth + repository namespace), then copies every BOM artifact into it, idempotently. Records the result in registry-mirror.json so the BNK install can be redirected to the mirror.
Runs entirely host-side: it pulls each artifact by digest from the FAR source (repo.f5.com) and pushes to the target registry via go-containerregistry — no Kubernetes cluster is involved. So you can pre-seed the mirror as a standalone supply-chain step, before creating any cluster, from any host that can reach both the FAR source and the target registry. Requires a configured registry: block and the FAR source credential (registry.source_service_account_b64, the workspace FAR auth, or –source-sa-b64).
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--concurrency | int | 0 | parallel copy workers (default: 4) |
--far-repo-url | string | — | FAR registry host (default: workspace bnk.far_repo_url, else repo.f5.com) |
--include-deps | bool | false | force-include the non-F5 dependency artifacts (cert-manager, node-labeler) |
--insecure-capture-ca | bool | false | adopt a self-signed mirror CA over an UNAUTHENTICATED connection (trust-on-first-use); prefer –registry-ca or –registry-ca-fingerprint |
--manifest-version | string | — | BNK manifest version (default: workspace bnk.manifest_version) |
--no-include-deps | bool | false | exclude the non-F5 dependency artifacts |
--registry-ca | string | — | PEM CA the mirror serves TLS with, for air-gap node trust (preferred: the file you generated; else registry.generic_ca_b64) |
--registry-ca-fingerprint | string | — | expected SHA-256 of the mirror CA (“sha256:ab:cd…” or bare hex), authenticating a captured CA out of band |
--source-sa-b64 | string | — | FAR _json_key_base64 service account (default: workspace registry.source_service_account_b64) |
--target | string | — | mirror target backend: icr|generic (default: workspace registry.target, else “icr”) |
← back to roksbnkctl registry
roksbnkctl registry target
Show or set the registry mirror target and its fields
roksbnkctl registry target [icr|generic | <field> <value>] [flags]
Configure the registry replication target without hand-editing config.yaml.
With no arguments, prints the current target + configured fields. Otherwise the first argument is either a backend KIND (sets registry.target) or a FIELD name (set with a following value):
Kinds: icr | generic Fields: icr_host icr_namespace generic_host generic_repo_prefix generic_username generic_password generic_ca (a PEM file) generic_ca_sha256
Examples: roksbnkctl registry target # show current config roksbnkctl registry target icr # use IBM Container Registry roksbnkctl registry target icr_namespace bnk-test roksbnkctl registry target generic # use a generic OCI registry roksbnkctl registry target generic_host art.example.com roksbnkctl registry target generic_repo_prefix bnk roksbnkctl registry target generic_username ci-bot echo “$TOKEN” | roksbnkctl registry target generic_password –password-stdin roksbnkctl registry target generic_ca /opt/harbor/certs/harbor.crt
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--password-stdin | bool | false | read the generic registry password from stdin (for registry target generic_password) |
← back to roksbnkctl registry
roksbnkctl registry verify
Confirm every BOM artifact is present + digest-matched in the mirror
roksbnkctl registry verify [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--far-repo-url | string | — | FAR registry host (default: workspace bnk.far_repo_url, else repo.f5.com) |
--include-deps | bool | false | force-include the non-F5 dependency artifacts (cert-manager, node-labeler) |
--manifest-version | string | — | BNK manifest version (default: workspace bnk.manifest_version) |
--no-include-deps | bool | false | exclude the non-F5 dependency artifacts |
--source-sa-b64 | string | — | FAR _json_key_base64 service account (default: workspace registry.source_service_account_b64) |
--target | string | — | mirror target backend: icr|generic (default: workspace registry.target, else “icr”) |
← back to roksbnkctl registry
roksbnkctl self
Manage the roksbnkctl binary itself
roksbnkctl self update
Pull the latest roksbnkctl release matching the host arch
Downloads the latest GitHub release for this platform, verifies its SHA256 against the release’s checksums.txt, and replaces the running binary in place. Works on Linux, macOS, and Windows.
This is the latest-only, interactive form; use roksbnkctl upgrade to pin a
specific --version or to skip the prompt with --yes.
Requires write permission on the binary’s directory (typical install under /usr/local/bin needs sudo; brew/scoop should use their own upgrade verb).
← back to roksbnkctl self
roksbnkctl shell
Interactive bash with KUBECONFIG, IBMCLOUD_API_KEY, and region pre-loaded
roksbnkctl shell drops into a $SHELL subshell with the workspace’s KUBECONFIG, IBMCLOUD_API_KEY, IC_API_KEY, and IBMCLOUD_REGION exported so locally-installed kubectl / oc / ibmcloud commands work without further setup. Exits when the subshell does.
roksbnkctl state
Manage where terraform state lives (local vs COS/S3 remote backend)
roksbnkctl state local
Use local per-phase terraform state (the default) — writes config.yaml for you
← back to roksbnkctl state
roksbnkctl state migrate
Copy each phase’s local terraform state into the configured COS/S3 backend
roksbnkctl state migrate [flags]
Migrates a workspace’s terraform state from local files into the COS/S3 remote backend declared in config.yaml’s state: block (PRD 16).
Pre-requisite: set state.backend = “s3” (+ state.s3.{endpoint,bucket,region}) and export the COS HMAC keys (ROKSBNKCTL_COS_HMAC_ACCESS_KEY / ROKSBNKCTL_COS_HMAC_SECRET_KEY, or AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) before running.
For each deployed phase (cluster / bnk / testing / gateway), this runs terraform init -migrate-state to copy its local state to the per-phase key in the bucket. It refuses to overwrite a key that already holds state (pass –force to override). The local state files are left in place; verify the remote read-back before deleting them.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--force | bool | false | migrate even if the remote key already holds state (overwrites it) |
← back to roksbnkctl state
roksbnkctl state s3
Use the COS/S3 remote terraform-state backend — writes the state: block for you
roksbnkctl state s3 [flags]
Switch this workspace to the COS/S3 remote state backend (PRD 16) by writing its state: block to config.yaml — no hand-edit.
You still provision the bucket + HMAC credentials yourself: the HMAC access/
secret keys are NEVER written to config.yaml — only the names of the env vars
they come from are (–access-key-source / –secret-key-source, defaulting to
ROKSBNKCTL_COS_HMAC_ACCESS_KEY / ROKSBNKCTL_COS_HMAC_SECRET_KEY). After this,
export those env vars and run roksbnkctl state migrate to move existing state.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--access-key-source | string | — | env var holding the HMAC access key (default ROKSBNKCTL_COS_HMAC_ACCESS_KEY) |
--bucket | string | — | pre-provisioned bucket name (required) |
--endpoint | string | — | COS S3 endpoint URL (required) |
--key-prefix | string | — | state key prefix (default: the workspace name) |
--region | string | — | COS location / region (required) |
--secret-key-source | string | — | env var holding the HMAC secret key (default ROKSBNKCTL_COS_HMAC_SECRET_KEY) |
← back to roksbnkctl state
roksbnkctl state show
Show the workspace’s terraform-state backend config
← back to roksbnkctl state
roksbnkctl status
Summary of the workspace: cluster, components, last apply
roksbnkctl status [flags]
roksbnkctl status reports a quick read of the workspace:
- workspace name + region
- configured cluster name
- pinned Terraform source
- per-phase deployment status (Cluster, BNK trial, Testing, Gateway)
- kubeconfig path (if any)
- cluster reachability (node count + ready count)
v1.x will add per-BNK-component readiness (flo, cis, cert-manager, cneinstance) once the component-discovery shape is finalised.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
roksbnkctl targets
Manage SSH targets used by –on
Targets are named SSH endpoints stored under the workspace’s
targets: block. They become reachable via the persistent –on flag
on commands like roksbnkctl exec, roksbnkctl shell, roksbnkctl kubectl, etc.
A jumphost target is auto-populated after a successful roksbnkctl up
when the upstream HCL provisions one (testing_tgw_jumphost outputs).
roksbnkctl targets add
Add or update a target
roksbnkctl targets add <name> --host H --user U [--port P] [--key-path P | --key-source S] [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--host | string | — | host or IP |
--key-path | string | — | path to a PEM private key |
--key-source | string | — | key source — “agent” or “tf-output:<name>” |
--port | int | 0 | ssh port (default 22) |
--user | string | — | remote user |
← back to roksbnkctl targets
roksbnkctl targets list
List all targets in the current workspace
← back to roksbnkctl targets
roksbnkctl targets remove
Remove a target
roksbnkctl targets remove <name>
← back to roksbnkctl targets
roksbnkctl targets show
Show detail for one target
roksbnkctl targets show <name>
← back to roksbnkctl targets
roksbnkctl terraform
Read-only passthrough to terraform against the workspace’s managed state
Aliases: tf
roksbnkctl terraform [subcommand] [args...]
roksbnkctl terraform runs a READ-ONLY terraform subcommand against the workspace’s managed state, with terraform’s working directory and TF_DATA_DIR resolved for you (no need to cd into ~/.roksbnkctl/… or export TF_DATA_DIR).
This command is read-only by allowlist. Permitted subcommands: output, show, state list, state show, state pull, providers, version, graph, validate, fmt -check
Everything else — including apply, destroy, init, plan, import, taint, untaint, state rm/mv/replace-provider, and any -auto-approve — is rejected before terraform runs. Mutations go exclusively through roksbnkctl up / plan / apply / down (or cluster/bnk up/down); running terraform apply/destroy outside the orchestration skips the rendered tfvars, the apply-retry wrapper, the post-apply kubeconfig fetch, the applied-tfvars snapshot, and the auto-jumphost seeding, and desyncs the managed state.
Use –phase cluster to target the cluster-phase state
(/.roksbnkctl//.roksbnkctl/<ws>/state-cluster/); the default is the trial/single
state (<ws>/state/). –on is not supported: the managed
terraform state lives on this workstation, not on a jumphost.
Examples: roksbnkctl terraform output testing_cluster_jumphost_ssh_commands roksbnkctl terraform state list roksbnkctl –phase cluster terraform show
roksbnkctl test
Run deployment validation tests (default: all)
roksbnkctl test [suite] [flags]
roksbnkctl test runs deployment validation against the current workspace.
Suites: connectivity HTTP/HTTPS reachability of deployed BNK services dns DNS resolution of ingress and service hostnames throughput iperf3 measurements (north-south by default; v1.x) all run all of the above (default if no suite is specified)
Honors -o json with the roksbnkctl.v1 schema. Exit code 0 on all-pass, non-zero on any-fail — CI-friendly.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--insecure | bool | false | skip TLS certificate validation (connectivity only) |
roksbnkctl test connectivity
HTTP/HTTPS reachability against configured hosts
← back to roksbnkctl test
roksbnkctl test dns
DNS resolution probe (single-vantage, GSLB-compare, or workspace-driven)
roksbnkctl test dns [flags]
roksbnkctl test dns runs DNS probes against configured resolvers.
Two modes:
Workspace-driven (no flags) — resolves each host listed under
test.connectivity.extra_hosts via the std-lib resolver. Same as
Sprint 0–4 behaviour; preserves CI invocations using the legacy
roksbnkctl.v1 schema.
Flag-driven (any of –target/–type/–server/–gslb-compare set) —
uses the embedded miekg/dns probe (no external dig install needed).
Single-vantage emits roksbnkctl.dns.v1.vantage; –gslb-compare
emits roksbnkctl.dns.v1 with a gslb_divergence boolean across
all configured backends (local + k8s + ssh:<targets>).
Use –backend local|k8s|ssh:<target> to pick a single vantage point;
–gslb-compare fans out across all available vantages. PRD 03 §“DNS
probe (GSLB-aware)”.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--gslb-compare | bool | false | fan out the probe across all configured backends (local + k8s + ssh:<targets>) and emit a comparison JSON with gslb_divergence |
--iterations | int | 1 | number of repeated queries; >1 enables RTT distribution |
--require-divergence | bool | false | with –gslb-compare: exit non-zero if NO divergence is observed (CI assertion that GSLB is doing something) |
--server | string | — | resolver: <ip>[:<port>] | system | cluster | <named-from-workspace> (default: system) |
--target | string | — | DNS name to query (overrides workspace test.dns.default_target) |
--timeout | duration | 2s | per-query timeout |
--type | string | A | record type: A | AAAA | CNAME | MX | NS | TXT | SRV | SOA | PTR | CAA | DS | DNSKEY | ANY |
← back to roksbnkctl test
roksbnkctl test hosts
Manage test.connectivity.extra_hosts in the workspace config
Test hosts are URLs probed by roksbnkctl test connectivity and
(in workspace-driven mode) roksbnkctl test dns. They are stored under
test.connectivity.extra_hosts in the workspace’s config.yaml.
This command group is the first-class CLI for managing that slice —
without it, the only path is hand-editing
~/.roksbnkctl/<workspace>/config.yaml. Mirrors roksbnkctl targets’
ergonomics: idempotent add/remove, -o json on list.
← back to roksbnkctl test
roksbnkctl test hosts add
Append URLs to test.connectivity.extra_hosts (idempotent)
roksbnkctl test hosts add <url> [<url> ...]
Appends each <url> to test.connectivity.extra_hosts. Idempotent —
adding an already-present URL is a no-op (logs to stderr; exit 0).
Each <url> is validated via the std-lib url.Parse; non-URLs are
rejected with an actionable error naming the offending arg. Insertion
order is preserved.
← back to roksbnkctl test hosts
roksbnkctl test hosts clear
Remove ALL entries from test.connectivity.extra_hosts
roksbnkctl test hosts clear [flags]
Clears test.connectivity.extra_hosts. Confirmation prompt defaults to
No; pass --auto to skip the prompt (matches the
roksbnkctl down / cluster down confirmation pattern).
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the confirmation prompt |
← back to roksbnkctl test hosts
roksbnkctl test hosts list
List configured test hosts (one per line; -o json for array)
Prints the workspace’s test.connectivity.extra_hosts one per line on stdout. Empty list emits zero bytes + exit 0 (NOT an error; distinguishes “nothing configured” from “command failed” by exit code).
With -o json, emits the slice as a JSON array ([] when empty).
← back to roksbnkctl test hosts
roksbnkctl test hosts remove
Remove URLs from test.connectivity.extra_hosts (idempotent)
roksbnkctl test hosts remove <url> [<url> ...]
Removes each <url> from test.connectivity.extra_hosts. Idempotent —
removing an absent URL is a no-op (logs to stderr; exit 0). Preserves
the order of remaining entries.
← back to roksbnkctl test hosts
roksbnkctl test list
List available test suites
← back to roksbnkctl test
roksbnkctl test matrix
Run the declarative BNK-on-ROKS performance grid (iperf3 L4 + h2load L7)
roksbnkctl test matrix [flags]
roksbnkctl test matrix runs a declarative performance grid against an already-deployed cluster + BNK. It reads a matrix.yaml describing cells (endpoint-pair × test-family) and runs each, emitting a report shaped like the BNK-on-ROKS perf plan.
Two families: iperf3 — L4 TCP throughput over an L4Route VIP, with content-size knobs (length: “128” vs “512K”) — the L4 analog of the plan’s 128 B / 512 KB payload axis. l7 — h2load against an HTTPRoute, http and https (TLS terminate at TMM); cps / tps / throughput modes.
The locality axis (same-zone / different-zone / different-VPC) is implicit
in which jumphost (“vsi”) endpoint a cell names as its client, so the
per-AZ jumphost targets the Testing phase auto-registers (jumphost,
jumphost-<zone>) are the traffic-source fleet. The Testing phase
preinstalls both generators (iperf3 + h2load/nghttp2-client) on every
jumphost, so the ssh runs need no –bootstrap.
This command provisions only ephemeral, runner-owned fixtures (an iperf3 server, an HTTP file backend, and optional L4Route/HTTPRoute/TLS objects that attach to the EXISTING Gateway by name) and tears them down after (unless –keep). It never touches Terraform or the gateway-phase objects.
roksbnkctl test matrix –dry-run expand + print the plan, no cluster calls roksbnkctl test matrix –only ‘L7*’ run a subset (glob on cell name) roksbnkctl test matrix -o md Markdown grid report
Honors -o json|text|md with the roksbnkctl.v1 schema. Exit code is non-zero if any cell fails.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--dry-run | bool | false | expand the grid and print the plan + fixtures; no cluster calls |
--file | string | — | path to matrix.yaml (default: <workspace>/matrix.yaml, then ./matrix.yaml) |
--keep | bool | false | leave fixtures running after the run |
--only | string | — | glob on cell name; run only matching cells |
← back to roksbnkctl test
roksbnkctl test throughput
iperf3 throughput; deploys server pod automatically (v1.x)
roksbnkctl test throughput [flags]
Deploys an iperf3 server in the test namespace and runs the client either from the roksbnkctl host (–mode north-south, default) or from a second in-cluster pod (–mode east-west).
Not yet implemented — landing in v1.x once the internal/k8s client-go fixture lifecycle is wired.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--cross-node | bool | false | force east-west client and server onto different nodes |
--keep | bool | false | leave the iperf3 server pod running after the test |
--mode | string | north-south | throughput mode: north-south | east-west |
← back to roksbnkctl test
roksbnkctl testing
Testing-phase (jumphost) lifecycle — provisions the test rig
Manage the testing jumphost infrastructure (TGW jumphost, per-AZ cluster jumphosts, client VPC) as an independent phase that sits beside the BNK phase on top of a shared cluster. Pure IBM VPC — no Kubernetes.
Commands: roksbnkctl testing up Provision the jumphosts (needs a cluster: VPC + transit gateway) roksbnkctl testing down Destroy the jumphosts, leaving the cluster and BNK intact roksbnkctl testing status Live jumphost IPs / SSH commands + reachability
This is the provisioning phase. To RUN connectivity/DNS/throughput probes
against a deployed environment, use roksbnkctl test (and
roksbnkctl test hosts to manage the probe target list) — those
provision nothing.
roksbnkctl testing down
Destroy the testing jumphosts, leaving the cluster and BNK
roksbnkctl testing down [flags]
Destroys only the testing jumphost resources (state-testing/), leaving
the cluster phase and the BNK phase intact. The inverse of bnk down:
each phase tears down independently. Exits 0 (“nothing to do”) when there’s no
testing state, so it’s safe in a reverse-order teardown of every phase.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the destroy confirmation |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl testing
roksbnkctl testing output
Print the Testing phase’s own terraform outputs (text or –json; [name] = one raw value)
roksbnkctl testing output [name] [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
--show-sensitive | bool | false | reveal sensitive output values (default redacted) |
← back to roksbnkctl testing
roksbnkctl testing status
Live runtime status of the Testing phase (jumphost IPs + SSH reachability)
roksbnkctl testing status [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--json | bool | false | output JSON (CI-friendly) |
← back to roksbnkctl testing
roksbnkctl testing up
Provision the testing jumphosts (cluster must exist)
roksbnkctl testing up [flags]
Provisions the testing jumphosts against the existing cluster’s VPC +
transit gateway (read from cluster-outputs.json). Runs in its own state
(state-testing/), independent of the BNK phase — bnk down leaves
the jumphosts and testing down leaves BNK.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the confirmation prompt before apply |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl testing
roksbnkctl tfvars
Emit the upstream TF’s terraform.tfvars.example for editing
roksbnkctl tfvars [flags]
Resolves the workspace’s pinned TF source (downloading the tarball if not yet cached) and writes its terraform.tfvars.example as a starting point you can edit and pass to roksbnkctl up.
Default writes to ./terraform.tfvars in the current directory.
Pass -o <path> to write elsewhere, or -o - to print to stdout.
Refuses to overwrite an existing destination unless –force is set.
Workflow: roksbnkctl init # pins a TF source roksbnkctl tfvars # writes ./terraform.tfvars from the upstream example $EDITOR ./terraform.tfvars roksbnkctl up –var-file ./terraform.tfvars
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--force | bool | false | overwrite the destination if it already exists |
--output / -o | string | ./terraform.tfvars | destination file (or - for stdout) |
roksbnkctl tgw
Attach the cluster to an existing Transit Gateway (share one across clusters)
Connect this workspace’s cluster VPC to an EXISTING IBM Transit Gateway, by name or id — at create time or after. Multiple workspaces can point at the same gateway; each owns its own connection.
Commands:
roksbnkctl tgw connect <name-or-id> Attach the cluster VPC to the gateway
roksbnkctl tgw disconnect Detach (leaves the gateway + other clusters)
roksbnkctl tgw status Gateway id/name + live connection state
Runs in its own state (state-tgw/), separate from cluster/BNK/testing/gateway/FLP, and works against a created OR a registered cluster.
roksbnkctl tgw connect
Attach the cluster VPC to an existing Transit Gateway
roksbnkctl tgw connect [transit-gateway-name-or-id] [flags]
Attaches this workspace’s cluster VPC to an existing Transit Gateway.
The gateway may be given by NAME or by ID — either is resolved against your
account. Pass it as the argument (persisted to config for later re-applies), or
set resources.transit_gateway (create: false, existing: <name-or-id>) in
config.yaml / the init interview.
Idempotent and shareable: run it in several workspaces with the same gateway and each cluster gets its own connection to the one gateway.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the confirmation prompt before apply |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl tgw
roksbnkctl tgw disconnect
Detach the cluster VPC from its Transit Gateway
roksbnkctl tgw disconnect [flags]
Removes ONLY this cluster’s connection to the Transit Gateway. The gateway itself and every other cluster’s connection to it are left intact.
Exits 0 (“nothing to do”) when there’s no connection.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the destroy confirmation |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
← back to roksbnkctl tgw
roksbnkctl tgw status
Show the Transit Gateway id/name and the live connection state
← back to roksbnkctl tgw
roksbnkctl uninstall
Remove the installed roksbnkctl binary from ~/.local/bin (opposite of install)
roksbnkctl uninstall [flags]
Delete the roksbnkctl binary that roksbnkctl install copied onto $PATH.
By default it removes <install-dir>/roksbnkctl, where <install-dir> is the same
directory install uses (~/.local/bin, then ~/bin). Override with –dir.
It refuses to delete the binary you are currently running on Windows (a running .exe cannot be removed there) — delete it manually or run uninstall from a different binary. On Linux/macOS removing the running (installed) binary is fine.
Examples: roksbnkctl uninstall # remove ~/.local/bin/roksbnkctl roksbnkctl uninstall –dir ~/bin sudo roksbnkctl uninstall –dir /usr/local/bin
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--dir | string | — | directory to remove roksbnkctl from (default: ~/.local/bin or ~/bin) |
--yes / -y | bool | false | skip the confirmation prompt |
roksbnkctl up
Provision (or attach) and deploy BNK — terraform plan + apply
roksbnkctl up [flags]
roksbnkctl up validates credentials, resolves the pinned Terraform source, runs plan, and (after confirmation, unless –auto) applies. Idempotent and resumable: a partial failure is recovered by re-running ‘roksbnkctl up’.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--auto | bool | false | skip the confirmation prompt before apply |
--no-kubeconfig | bool | false | skip the post-apply admin kubeconfig fetch |
--tf-source | string | — | override TF source for this run only (path or URL; relative local paths resolved against the invocation CWD) |
--var-file | stringArray | [] | extra TF var-file (repeatable; later files override earlier) |
roksbnkctl upgrade
Upgrade (or pin) the roksbnkctl binary to a GitHub release
roksbnkctl upgrade [flags]
Downloads a roksbnkctl release for this OS/arch from GitHub, verifies its SHA256 against the release’s checksums.txt, and replaces the running binary in place. With no –version it upgrades to the latest release; –version pins a specific release (and may downgrade or reinstall).
Works on Linux, macOS, and Windows. On Windows the running .exe cannot be
overwritten, so it is moved aside to <binary>.old and the new binary takes its
place; the .old file is removed automatically on the next run.
Requires write permission on the binary’s directory (a /usr/local/bin install needs sudo; Homebrew/Scoop installs should use their own upgrade verb).
Note: release binaries are not yet code-signed, so on a host with an application-allowlist policy (e.g. Windows Device Guard/WDAC) the freshly downloaded binary may be blocked until its hash is trusted.
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--version | string | — | release to install, e.g. v1.20.1 (default: latest) |
--yes / -y | bool | false | skip the confirmation prompt |
roksbnkctl version
Print version, commit, and build date
roksbnkctl workspaces
Manage roksbnkctl workspaces (per-environment config + state bundles)
Aliases: ws
Each workspace lives under ~/.roksbnkctl/<name>/ with its own config.yaml
and state. The current_workspace pointer in ~/.roksbnkctl/config.yaml decides
which one commands run against; -w/–workspace overrides for one invocation.
roksbnkctl workspaces current
Print the current workspace name
← back to roksbnkctl workspaces
roksbnkctl workspaces delete
Delete a workspace (refuses if state is non-empty unless –force)
roksbnkctl workspaces delete <name> [flags]
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--force | bool | false | delete even if Terraform state lists provisioned resources |
← back to roksbnkctl workspaces
roksbnkctl workspaces list
List workspaces and their states
← back to roksbnkctl workspaces
roksbnkctl workspaces new
Create a new (empty) workspace skeleton — run roksbnkctl init -w <name> to populate
roksbnkctl workspaces new <name>
← back to roksbnkctl workspaces
roksbnkctl workspaces use
Set the current workspace pointer
roksbnkctl workspaces use <name>
← back to roksbnkctl workspaces
Configuration reference
Field-by-field schema for the workspace config.yaml, generated from the
Workspace struct
in internal/config/workspace.go. This chapter is the single source of truth for
what a field is called, what type it takes, and which BNK line it applies to.
For a one-page version carrying the full dotted paths and each field’s
ROKSBNKCTL_* override, see the
config.yaml cheatsheet
— generated from the same struct, so the two cannot disagree.
Chapter 12 — Workspace config is the teaching chapter: use it to learn the shape. Use this one to look up a specific field. Every other chapter links here rather than restating fields, because a field restated in four places is a field that will disagree with itself by the next release.
The line column says which BNK release line a field applies to. Most apply to
both. A field marked 2.4 has no effect on a 2.3 install and vice versa — the line
itself is selected by bnk.manifest_version and nothing else.
Required means the field has no omitempty: it is always rendered, and its zero
value is meaningful. It does not mean you must type it into your config.yaml.
Default is carried on the field as a default:"..." struct tag. A dash means no
default is declared — either the zero value applies, or the default is computed at
run time and belongs in the prose rather than a table cell.
Workspace
Workspace is ~/.roksbnkctl/<name>/config.yaml. Mirrors the per-workspace example in docs/PRD.md. Note that there is no api_key field — secrets live in env vars or the OS keychain, never in this struct. Plaintext keys in the YAML are rejected at load time by rejectPlaintextSecrets.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
ibmcloud | IBMCloudCfg | 2.3 + 2.4 | — | yes | IBMCloud is the account context every phase runs against: region, resource group, and the API key (base64-obfuscated, never plaintext). |
cluster | ClusterCfg | 2.3 + 2.4 | — | yes | Cluster describes the ROKS cluster — either one to CREATE, or one that already exists and is being adopted (create: false plus its name). |
bnk | BNKCfg | 2.3 + 2.4 | — | no | BNK configures the BIG-IP Next for Kubernetes install itself: the manifest version (which also decides whether the 2.3 or 2.4 model is rendered), licensing, the per-zone network mapping, and the CNEInstance settings that place the TMM pods. |
gateway | GatewayCfg | 2.3 + 2.4 | — | no | Gateway configures the Gateway API resources installed after BNK. |
registry | *RegistryCfg | 2.3 + 2.4 | — | no | Registry selects where images and charts are pulled from: F5’s Artifact Repository directly, or a mirror that a disconnected cluster can reach — IBM Container Registry, or any OCI registry (generic) such as Artifactory or Harbor. |
test | TestCfg | 2.3 + 2.4 | — | no | Test configures the connectivity, DNS and throughput probes roksbnkctl test runs, and the jumphost they run FROM. |
tf_source | TFSourceCfg | 2.3 + 2.4 | — | yes | TFSource selects which Terraform the binary applies: the embedded tree shipped inside this binary (the default, and the only version matched to it), a GitHub release, or a local path for testing a fork. |
cos | *COSCfg | 2.3 + 2.4 | — | no | COS is the IBM Cloud Object Storage bucket the FAR service-account credential is read from, for estates that stage it there rather than passing a local file. |
targets | map[string]TargetCfg | 2.3 + 2.4 | — | no | Targets are named remote hosts (ssh:<target>) that Exec backends can run tools on — a jumphost inside the VPC, say, when the operator’s workstation cannot reach the cluster directly. |
state | StateCfg | 2.3 + 2.4 | — | no | State controls where Terraform state is kept and how it is locked. |
bnkforge | *BNKForgeCfg | 2.3 + 2.4 | — | no | BNKForge points at a BNK Forge server, which takes over a durable cluster once roksbnkctl has built it. |
agent | *AgentCfg | 2.3 + 2.4 | — | no | Agent configures roksbnkctl agent, which hands the workspace to a coding agent. |
prefix | string | 2.3 + 2.4 | — | no | Prefix is the workspace’s account-scoped resource-name base. |
resources | *ResourcesCfg | 2.3 + 2.4 | — | no | Resources carries the per-resource create toggles (and the existing-resource name/ID for any declined-but-still-depended-on resource). |
exec | map[string]ExecToolCfg | 2.3 + 2.4 | — | no | Exec is the per-tool execution-backend config block introduced in Sprint 3 (PRD 03). |
AdvancedComponentCfg
AdvancedComponentCfg is one component’s advanced settings. Only env is surfaced today; the CR’s advanced block carries more, and this is the shape that lets those arrive without moving what already works.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
env | map[string]string | 2.3 + 2.4 | — | no | Env sets environment variables on that component’s containers, name → value. |
AgentCfg
AgentCfg configures roksbnkctl’s agentic mode (the agent command). It is purely advisory metadata for launching an external coding-agent CLI against the workspace’s scaffolded AGENTS.md + personas/ — roksbnkctl embeds no LLM. nil/absent = agent defaults to claude and the CLI’s own endpoint config.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
default | string | 2.3 + 2.4 | — | no | Default is the agentic CLI roksbnkctl agent (no arg) reports as this workspace’s default — claude | gemini | aider | openai | pi | opencode. |
llm_endpoint | string | 2.3 + 2.4 | — | no | LLMEndpoint is an optional OpenAI-/Anthropic-compatible base URL woven into the printed invocation (cloud vendor, local vLLM, etc.). |
BNKCISCfg
BNKCISCfg configures the BNK CIS controller’s BIG-IP target. All optional. BigIPPasswordB64 stores the password base64-encoded (obfuscation, NOT encryption — like ibmcloud.api_key_b64); the raw value is rendered to terraform.tfvars as bigip_password at apply time.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
bigip_url | string | 2.3 + 2.4 | — | no | BigIPURL is the management address of the classic BIG-IP that the Container Ingress Services controller configures, e.g. “https://10.1.1.5”. |
bigip_username | string | 2.3 + 2.4 | — | no | BigIPUsername is the account CIS authenticates to that BIG-IP as. |
bigip_password_b64 | string | 2.3 + 2.4 | — | no | BigIPPasswordB64 is that account’s password, base64-encoded — obfuscation, not encryption. |
BNKCertManagerCfg
BNKCertManagerCfg overrides cert-manager’s install coordinates. All optional; the create/skip decision stays on resources.cert_manager.create.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
namespace | string | 2.3 + 2.4 | — | no | Namespace cert-manager installs into (rendered as cert_manager_namespace). |
version | string | 2.3 + 2.4 | — | no | Version pins the cert-manager Helm chart (rendered as cert_manager_version). |
BNKCfg
| key | type | line | default | required | description |
|---|---|---|---|---|---|
cneinstance_size | string | 2.3 + 2.4 | — | no | CNEInstanceSize is the CNEInstance deploymentSize: Tiny, Small, Medium, Large or Max. |
far_repo_url | string | 2.3 + 2.4 | repo.f5.com | no | FARRepoURL is the F5 Artifact Repository charts are pulled from. |
manifest_version | string | 2.3 + 2.4 | — | no | ManifestVersion pins the BNK release, e.g. “2.4.0-EA”. |
far_auth_file | string | 2.3 + 2.4 | — | no | FarAuthFile is the FAR auth tarball’s filename in the orchestration COS bucket; rendered as the f5_cne_far_auth_file tfvar + used by registry to resolve the FAR _json_key_base64 service account. |
subscription_jwt_file | string | 2.3 + 2.4 | — | no | SubscriptionJWTFile is the subscription/license JWT’s filename in the orchestration COS bucket; rendered as the f5_cne_subscription_jwt_file tfvar. |
far_auth_local_file | string | 2.3 + 2.4 | — | no | FarAuthLocalFile / SubscriptionJWTLocalFile point at LOCAL files instead of COS objects. |
subscription_jwt_local_file | string | 2.3 + 2.4 | — | no | SubscriptionJWTLocalFile is the F5 subscription JWT as a LOCAL file, the companion to FarAuthLocalFile above. |
trusted_profile | *BNKTrustedProfileCfg | 2.3 + 2.4 | — | no | TrustedProfile tunes the IBM Cloud Trusted Profile the CNE controller assumes to manage its own cluster’s VPC. |
flo_namespace | string | 2.3 + 2.4 | f5-bnk | no | FLONamespace / FLOUtilsNamespace override the namespaces the F5 Lifecycle Operator and its utility components install into (rendered as flo_namespace / flo_utils_namespace). |
flo_utils_namespace | string | 2.3 + 2.4 | f5-utils | no | FLOUtilsNamespace is where FLO’s SHARED utility components install — coremond, the observer, the licence — as opposed to FLONamespace, which holds the BNK instance itself. |
storage_class_name | string | 2.3 + 2.4 | — | no | StorageClassName is the StorageClass the CNEInstance’s persistent volumes use, TMM’s included. |
gateway_api_mtls | bool | 2.4 | — | no | GatewayAPIMTLS opts into the Gateway API bundle BNK 2.4 needs for mTLS (#170). |
tmm_replicas | int | 2.4 | 3 | no | TMMReplicas is the number of f5-tmm data-plane replicas. |
watch_namespaces | []string | 2.4 | All | no | WatchNamespaces are the namespaces the CNE controller watches. |
tmm_anti_affinity | *bool | 2.4 | true | no | TMMAntiAffinity requires f5-tmm pods onto different nodes. |
tmm_anti_affinity_topology_key | string | 2.4 | kubernetes.io/hostname | no | TMMAntiAffinityTopologyKey is the node label the anti-affinity rule spreads across — the IBM ROKS per-node label. |
tmm_zone_spread | *bool | 2.4 | true | no | TMMZoneSpread spreads f5-tmm pods across zones. |
tmm_zone_topology_key | string | 2.4 | topology.kubernetes.io/zone | no | TMMZoneTopologyKey is the IBM ROKS zone label the spread constraint uses. |
tmm_zone_max_skew | int | 2.4 | 1 | no | TMMZoneMaxSkew is maxSkew for the zone topology-spread constraint. |
tmm_zone_when_unsatisfiable | string | 2.4 | DoNotSchedule | no | TMMZoneWhenUnsatisfiable is DoNotSchedule or ScheduleAnyway. |
tmm_pod_label | string | 2.4 | f5-tmm | no | TMMPodLabel is the app label value the placement rules select TMM by. |
tmm_rolling_update | *bool | 2.4 | true | no | TMMRollingUpdate pins TMM’s rolling update to maxSurge 0 / maxUnavailable 1 — the same shape as the cwc Multi-Attach deadlock, where an unconstrained rolling update on a single-attach resource wedges. |
external_bigip | *bool | 2.4 | false | no | ExternalBigIP enables the external BIG-IP controller. |
external_bigip_login_secret | string | 2.4 | f5-bigip-ctlr-login | no | ExternalBigIPLoginSecret holds the external BIG-IP credentials. |
cluster_identifier | string | 2.4 | — | no | ClusterIdentifier is passed to the external BIG-IP controller. |
gateway_api_version | string | 2.4 | 1.5.0 | no | GatewayAPIVersion is GATEWAY_API_VERSION for the CNE controller. |
gateway_api_bundle_url | string | 2.4 | — | no | GatewayAPIBundleURL is where the Gateway API standard-install bundle is FETCHED from when gateway_api_mtls is on (#185). |
demo_mode | *bool | 2.3 + 2.4 | false on 2.4, true on 2.3 | no | DemoMode sets advanced.demoMode.enabled. |
whole_cluster | *bool | 2.3 + 2.4 | false on 2.4, true on 2.3 | no | TCPSettings overrides fields on the data-plane F5BigTcpSetting CR. |
hugepages | *HugepagesCfg | 2.3 + 2.4 | — | no | Hugepages optionally allocates hugepages on the worker pool via the OpenShift Node Tuning Operator. |
tcp_settings | map[string]string | 2.3 + 2.4 | — | no | TCPSettings overrides individual fields on the data-plane F5BigTcpSetting CR by name, e.g. {“congestionControl”: “bbr”, “delayedAcks”: “true”} — the key is the CR’s own spec field. |
tcp_settings_name | string | 2.3 + 2.4 | sys-default-tcp | no | TCPSettingsName is the F5BigTcpSetting to write. |
advanced | map[string]AdvancedComponentCfg | 2.3 + 2.4 | — | no | Advanced carries per-component environment passthrough for the 2.4 CNEInstance’s advanced.<component>.env[] lists (#175). |
tmm_resources | map[string]map[string]string | 2.3 + 2.4 | — | no | TMMResources overrides the TMM pod’s resource requests/limits, rendered as the CNEInstance’s advanced.tmm.resources (#203). |
gslb_datacenter_name | string | 2.3 + 2.4 | — | no | GSLBDatacenterName sets the optional CNEInstance GSLB datacenter name (rendered as cneinstance_gslb_datacenter_name). |
cert_manager | *BNKCertManagerCfg | 2.3 + 2.4 | — | no | CertManager overrides cert-manager’s namespace + chart version. |
network | *BNKNetworkCfg | 2.3 + 2.4 | — | no | Network holds the optional per-zone subnet CIDRs + TMM self-IPs for the cloud-network-mapping ConfigMap and the external/internal F5SPKVlan CRs (BNK install-guide “Configuration”). |
cis | *BNKCISCfg | 2.3 + 2.4 | — | no | CIS holds the BIG-IP management endpoint + credentials the BNK CIS controller (k8s-bigip-ctlr) uses. |
license_mode | string | 2.3 + 2.4 | connected | no | LicenseMode selects the License CR operationMode: “connected” (default when empty), “disconnected”, or “f5licenseproxy”. |
flp | *BNKFLPCfg | 2.3 + 2.4 | — | no | FLP holds settings for the optional F5 License Proxy phase. |
preflight | *BNKPreflightCfg | 2.3 + 2.4 | — | no | Preflight tunes the pre-install checks bnk up runs before it plans anything. |
BNKFLPCfg
BNKFLPCfg configures the F5 License Proxy (FLP) phase deployment. All optional; nil block means FLP is off. It never carries secrets — the FLP generates its own certs, and its subscription JWT is the same one resolved from COS.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
mode | string | 2.3 + 2.4 | — | no | Mode selects HOW the FLP phase deploys the proxy: “” | “helm” → the f5-license-proxy Helm chart into the ROKS cluster (default). |
vsi | *BNKFLPVSICfg | 2.3 + 2.4 | — | no | VSI configures the mode: vsi deployment backend. |
namespace | string | 2.3 + 2.4 | — | no | Namespace the FLP is installed into (helm mode). |
chart_version | string | 2.3 + 2.4 | — | no | ChartVersion pins the f5-license-proxy chart. |
storage_class | string | 2.3 + 2.4 | — | no | StorageClass is the dynamic StorageClass for the FLP’s PVCs (rendered as flp_storage_class). |
node_port_access | bool | 2.3 + 2.4 | — | no | NodePortAccess exposes the proxy OUTSIDE its own cluster, so a BNK install in a DIFFERENT cluster (same VPC, or across a transit gateway) can license through it — the “shared licensing cluster” topology, where only the cluster running the proxy needs egress to F5. |
node_port_source_cidrs | []string | 2.3 + 2.4 | — | no | NodePortSourceCIDRs, when set with NodePortAccess, opens the proxy’s NodePort on the cluster’s worker security group to these CIDRs — the consuming cluster’s subnets. |
external | *BNKFLPExternalCfg | 2.3 + 2.4 | — | no | External points a workspace at a FOREIGN proxy — one deployed by a DIFFERENT workspace/cluster. |
BNKFLPExternalCfg
BNKFLPExternalCfg addresses an F5 License Proxy that this workspace does not own. Both fields come from the owning workspace’s roksbnkctl flp output — the URL is its externally-reachable endpoint and the CA is its (base64) root CA, which the CWC must trust to complete the TLS handshake to the proxy.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
url | string | 2.3 + 2.4 | — | no | URL of the proxy, e.g. https://10.240.64.5:30001 — reachable from THIS cluster’s pods. |
root_ca_b64 | string | 2.3 + 2.4 | — | no | RootCAB64 is the proxy’s root CA, base64-encoded (as flp output emits it). |
BNKFLPForwardProxyCfg
BNKFLPForwardProxyCfg describes an egress forward proxy for the FLP VSI’s calls to F5’s licensing backend (product-s.apis.f5.com).
| key | type | line | default | required | description |
|---|---|---|---|---|---|
host | string | 2.3 + 2.4 | — | no | Host is the forward proxy’s address. |
port | int | 2.3 + 2.4 | — | no | Port is the forward proxy’s port. |
protocol | string | 2.3 + 2.4 | — | no | http (default) | https. |
BNKFLPVSICfg
BNKFLPVSICfg configures the mode: vsi FLP backend — a standalone VSI running the f5-license-proxy stack as a podman pod. All fields optional; sensible defaults apply.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
name_prefix | string | 2.3 + 2.4 | — | no | NamePrefix prefixes the FLP VSI’s IBM Cloud resource names — instance, subnet, security group, floating IP, public gateway, boot volume, and the VPC when this module creates one. |
vpc | string | 2.3 + 2.4 | — | no | VPC is an existing VPC id to deploy the standalone FLP VSI into, WITHOUT any ROKS cluster. |
create_vpc | bool | 2.3 + 2.4 | — | no | CreateVPC builds the proxy its OWN VPC, address prefix and public gateway rather than placing it in one that already exists (#60). |
vpc_name | string | 2.3 + 2.4 | — | no | VPCName names the VPC created when CreateVPC is set. |
subnet_cidr | string | 2.3 + 2.4 | — | no | SubnetCIDR is the address prefix for that VPC. |
profile | string | 2.3 + 2.4 | — | no | Profile is the IBM Cloud VSI instance profile. |
zone | string | 2.3 + 2.4 | — | no | Zone the VSI lands in (e.g. us-south-1). |
boot_size_gb | int | 2.3 + 2.4 | — | no | BootSizeGB is the boot volume size. |
management_allowed_cidrs | []string | 2.3 + 2.4 | — | no | ManagementAllowedCIDRs are the source CIDRs permitted to reach the :80 flp-status web UI (read-only status). |
licensing_allowed_cidrs | []string | 2.3 + 2.4 | — | no | LicensingAllowedCIDRs are the source CIDRs permitted to reach the :8443 licensing proxy (and :22 SSH). |
allowed_cidrs | []string | 2.3 + 2.4 | — | no | AllowedCIDRs is DEPRECATED — a legacy single list. |
ssh_key | string | 2.3 + 2.4 | — | no | SSHKey is the name of an existing IBM Cloud VPC SSH key (RSA) to attach to the FLP VSI, so an operator can SSH in to inspect/recover the licensing appliance (podman pod, Vault, logs). |
floating_ip | *bool | 2.3 + 2.4 | — | no | FloatingIP attaches an operator floating IP to the FLP VSI for remote management — running roksbnkctl flp status and reaching the :80 web UI + the :8443 proxy from a machine OUTSIDE the VPC. |
status_image | string | 2.3 + 2.4 | — | no | StatusImage, when set, runs the flp-status web UI as a container in the FLP pod (mobile-friendly status page + /api/status + live logs on :80, no auth — a read-only private status endpoint). |
status_registry_host | string | 2.3 + 2.4 | — | no | StatusRegistryHost + StatusRegistryCAB64 make the VSI trust a self-signed mirror so it can pull StatusImage: cloud-init drops the (base64) CA into /etc/containers/certs.d/<host>/ca.crt before the pod comes up. |
status_registry_ca_b64 | string | 2.3 + 2.4 | — | no | StatusRegistryCAB64 is that registry’s CA, base64-encoded, written to /etc/containers/certs.d/<host>/ca.crt on the VSI. |
forward_proxy | *BNKFLPForwardProxyCfg | 2.3 + 2.4 | — | no | ForwardProxy optionally routes the VSI’s egress to F5 licensing through an HTTP forward proxy (air-gapped/egress-controlled networks). |
BNKForgeCfg
BNKForgeCfg is the optional integration with a co-located BNK Forge (v3) install. When Register is true, cluster up registers the just-provisioned ROKS cluster with BNK Forge via its REST API — credential-backed, so BNK Forge derives the kubeconfig on demand from an IBM Cloud credential template rather than storing a perishable one. Best-effort: registration never blocks or fails the deploy. nil/absent (legacy config) = no-op.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
register | bool | 2.3 + 2.4 | — | no | Register opts the workspace in. |
url | string | 2.3 + 2.4 | — | no | URL is the BNK Forge server base URL (e.g. https://forge.example.com). |
project | string | 2.3 + 2.4 | — | no | Project is the target BNK Forge project NAME. |
username | string | 2.3 + 2.4 | — | no | Username is the BNK Forge login user. |
password_b64 | string | 2.3 + 2.4 | — | no | PasswordB64 is that user’s password, base64-encoded — obfuscation, NOT encryption, exactly as bnk.cis.bigip_password_b64 and registry.generic_password_b64 are. |
insecure | bool | 2.3 + 2.4 | — | no | Insecure skips TLS verification against the Forge server. |
ca_b64 | string | 2.3 + 2.4 | — | no | CAB64 pins the CA the Forge server’s certificate must chain to, PEM, base64-encoded. |
BNKNetworkCfg
BNKNetworkCfg is the optional cloud-network-mapping / VLAN zone data.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
zones | []BNKZoneCfg | 2.3 + 2.4 | — | no | Zones is the per-availability-zone network mapping, in zone order — one entry per zone the cluster spans. |
vlan_prefixlen | *int | 2.3 | 24 | no | VLANPrefixLen is the self-IP prefix length (spec.prefixlen_v4) TMM applies to its external and internal self-IPs on the F5SPKVlan CRs — the size of the L2 subnet TMM treats as directly connected on each VLAN. |
vlan_prefixlen_external | *int | 2.3 | — | no | VLANPrefixLenExternal / VLANPrefixLenInternal override VLANPrefixLen for one VLAN. |
vlan_prefixlen_internal | *int | 2.3 | — | no | VLANPrefixLenInternal is the same override for the INTERNAL VLAN. |
tmm_k8s_routes | string | 2.3 + 2.4 | 172.17.0.0/18 | no | TMMK8SRoutes is the Kubernetes pod CIDR TMM installs a route toward (advanced.tmm.env TMM_K8S_ROUTES), so TMM can reach backend pods on the internal data path. |
BNKPreflightCfg
BNKPreflightCfg tunes the per-node reachability gate. These are exposed because the right values are a property of the ENVIRONMENT, not of roksbnkctl. A Transit Gateway attachment is asynchronous — IBM programs the routes some time after the connection reports attached — and how long that takes varies by account, region and gateway. A probe run ~73s after attach saw both targets unreachable and refused an install whose path was healthy minutes later (issue #57), while a sibling cluster on the same gateway passed simply by landing on the other side of route programming. A site that consistently sees slower propagation should raise the budget rather than rediscover the race; a site with a static, long-established gateway can lower it to fail faster.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
reachability_retry_seconds | *int | 2.3 + 2.4 | — | no | ReachabilityRetrySeconds is how long a target may keep failing before the verdict is believed. |
reachability_timeout_seconds | *int | 2.3 + 2.4 | — | no | ReachabilityTimeoutSeconds is how long to wait for the probe DaemonSet to report from every node. |
BNKTrustedProfileCfg
BNKTrustedProfileCfg is the IBM Cloud Trusted Profile the CNE controller assumes at runtime — the identity that lets it manage the VPC network attachments it creates for TMM, without a stored API key. Both fields default in the HCL rather than here, so an absent block renders nothing and the shipped terraform decides — the same rule every other optional BNK field follows.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
service_account | string | 2.3 + 2.4 | — | no | ServiceAccount is the Kubernetes service account the profile is LINKED to: which account may assume it. |
roles | []string | 2.3 + 2.4 | [Viewer, Editor] | no | Roles granted to the profile, scoped to the cluster’s OWN VPC. |
BNKZoneCfg
BNKZoneCfg is one availability zone’s subnet CIDRs + TMM self-IPs. Field order/names match the terraform cneinstance_network_zones object.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
ext_vlan_cidr | string | 2.3 + 2.4 | — | yes | ExtVLANCIDR is the zone’s EXTERNAL VLAN — the client side of the data plane, where traffic arrives. |
int_vlan_cidr | string | 2.3 | — | yes | IntVLANCIDR is the zone’s INTERNAL VLAN — the pod side, where BNK reaches the workloads it fronts. |
int_snat_cidr | string | 2.3 + 2.4 | — | yes | IntSNATCIDR is the pool BNK source-NATs to when it talks to pods, so the return traffic comes back through TMM rather than routing around it. |
int_vip_cidr | string | 2.3 + 2.4 | — | yes | IntVIPCIDR is the range virtual servers are allocated from on the internal side. |
external_selfip | string | 2.3 | — | yes | ExternalSelfIP is TMM’s own address on the external VLAN. |
internal_selfip | string | 2.3 | — | yes | InternalSelfIP is TMM’s own address on the internal VLAN. |
COSCfg
COSCfg points roksbnkctl at the IBM Cloud Object Storage that holds the FAR auth key + subscription JWT (the “orchestration” COS). Empty fields fall back to the built-in defaults (bnk-supply-chain / bnk-artifacts / us-south). These are honoured BOTH by the terraform render (ibmcloud_cos_instance_name / ibmcloud_resources_cos_bucket / ibmcloud_cos_bucket_region) AND by the registry FAR-file resolver, so a customer-owned COS bucket is used consistently across both.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
instance | string | 2.3 + 2.4 | bnk-supply-chain | no | Instance is the IBM Cloud Object Storage service instance holding the bucket. |
bucket | string | 2.3 + 2.4 | bnk-artifacts | no | Bucket is the bucket the FAR service-account credential is read from, for estates that stage it centrally rather than passing a local file. |
region | string | 2.3 + 2.4 | us-south | no | Region is the bucket’s region, which need not match ibmcloud.region. |
resource_group | string | 2.3 + 2.4 | — | no | ResourceGroup is the IBM Cloud resource group holding Instance. |
upload | []COSUpload | 2.3 + 2.4 | — | no | Upload lists local files to place into that bucket before the phases that read them run. |
COSUpload
| key | type | line | default | required | description |
|---|---|---|---|---|---|
source | string | 2.3 + 2.4 | — | yes | Source is the local file to upload. |
key | string | 2.3 + 2.4 | — | yes | Key is the object name it is stored under in the bucket. |
ClusterCfg
| key | type | line | default | required | description |
|---|---|---|---|---|---|
create | bool | 2.3 + 2.4 | — | yes | Create decides whether this workspace BUILDS the ROKS cluster or adopts one that already exists. |
name | string | 2.3 + 2.4 | — | yes | Name is the ROKS cluster’s name — the one to create, or the existing one to adopt when Create is false. |
openshift_version | string | 2.3 + 2.4 | — | no | OpenShiftVersion pins the OpenShift version for a CREATED cluster (e.g. “4.20”). |
workers_per_zone | int | 2.3 + 2.4 | 1 | no | WorkersPerZone is the worker count PER ZONE, not in total: ROKS spans three availability zones, so 2 here is a six-node cluster. |
public_gateway | *bool | 2.3 + 2.4 | true | no | PublicGateway controls whether the cluster subnets attach a public gateway for worker Internet egress. |
vpc_cidr | string | 2.3 + 2.4 | — | no | VPCCIDR is the block the cluster VPC’s per-zone address prefixes are carved from — “10.241.0.0/16” becomes 10.241.0.0/18, 10.241.64.0/18, 10.241.128.0/18. |
network_mode | string | 2.3 + 2.4 | single-nic | no | NetworkMode selects how the cluster’s worker nodes are attached: “single-nic” (the default, and today’s only behaviour) or “multi-nic”. |
existing_subnet_ids | []string | 2.3 + 2.4 | — | no | ExistingSubnetIDs places the cluster in subnets that ALREADY EXIST, one per zone in zone order, instead of creating them (#61). |
worker_flavor | string | 2.3 + 2.4 | — | no | WorkerFlavor names the worker profile EXACTLY, e.g. “cx3d.8x20”. |
min_worker_vcpu_count | int | 2.3 + 2.4 | 16 | no | MinWorkerVCPUCount is the vCPU floor for the worker-flavor auto-select: the cluster module picks the smallest bx2 profile meeting this and MinWorkerMemoryGB. |
min_worker_memory_gb | int | 2.3 + 2.4 | 64 | no | MinWorkerMemoryGB is the memory floor for the same auto-select. |
ConnectivityCfg
| key | type | line | default | required | description |
|---|---|---|---|---|---|
extra_hosts | []string | 2.3 + 2.4 | — | no | ExtraHosts are additional targets the connectivity probe tries, beyond the defaults. |
DNSCfg
DNSCfg drives the Sprint 5 flag-driven DNS probe (PRD 03 §“DNS probe (GSLB-aware)” §“Server resolution”). The map’s keys are the names users pass to --server <name> and the values are concrete “<ip>[:<port>]” strings the miekg/dns client dials. DefaultTarget is used when –target isn’t passed on the command line. Example: test: dns: resolvers: google: “8.8.8.8:53” cloudflare: “1.1.1.1:53” gslb-vip: “169.45.91.5:53” default_target: “www.example.com”
| key | type | line | default | required | description |
|---|---|---|---|---|---|
resolvers | map[string]string | 2.3 + 2.4 | — | no | Resolvers names the DNS servers to query, as name → “<ip>[:<port>]”. |
default_target | string | 2.3 + 2.4 | — | no | DefaultTarget is the resolver queried when a probe names none. |
ExecToolCfg
ExecToolCfg is one entry under workspace.Exec — the chosen backend for a given tool.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
backend | string | 2.3 + 2.4 | — | yes | Backend is the execution-backend spec: “local” | “docker” | “k8s” | “ssh:<target>”. |
GatewayCfg
| key | type | line | default | required | description |
|---|---|---|---|---|---|
app_namespace | string | 2.3 + 2.4 | f5-app | no | AppNamespace is the namespace the example application and its Gateway resources are created in. |
class_name | string | 2.3 + 2.4 | gateway-class | no | ClassName is the GatewayClass name. |
controller_name | string | 2.3 + 2.4 | — | no | ControllerName is the GatewayClass controllerName. |
backend_service | string | 2.3 + 2.4 | nginx-service | no | BackendService is the Kubernetes Service the example HTTPRoute forwards to. |
backend_port | int | 2.3 + 2.4 | 80 | no | BackendPort is the port on that Service. |
egress_mode | string | 2.3 + 2.4 | snatpool | no | EgressMode selects how return traffic is source-addressed: “snatpool”, “automap”, or “both”. |
client_subnet_local | []string | 2.3 + 2.4 | — | no | ClientSubnetLocal lists client CIDRs reachable on the same VPC as the cluster — routed directly rather than over the transit gateway. |
client_subnet_remote | []string | 2.3 + 2.4 | — | no | ClientSubnetRemote lists client CIDRs reached ACROSS the transit gateway, e.g. the testing client VPC. |
vxlan_port | int | 2.3 + 2.4 | 6789 | no | VXLANPort is the UDP port for the VXLAN overlay between TMM and the nodes. |
route_examples | []string | 2.3 + 2.4 | — | no | RouteExamples names extra route kinds to create WORKING examples of, alongside the HTTPRoute the gateway phase already creates. |
l4_listener_port | int | 2.3 + 2.4 | 8080 | no | L4ListenerPort is the port for that TCP listener. |
HugepagesCfg
HugepagesCfg allocates hugepages on the worker pool through the OpenShift Node Tuning Operator. BNK’s deploymentSize decides how much TMM asks for: Tiny requests none, Small requests 4Gi of hugepages-2Mi. A stock ROKS worker reports zero — including F5’s approved reference cluster, which runs Tiny for exactly this reason. So any size above Tiny needs this, or nodes prepared some other way. APPLYING THIS REBOOTS WORKERS. The profile sets a bootloader kernel argument, and the Machine Config Operator rolls the pool to apply it — draining and restarting each node in turn. On a live cluster that is a maintenance event, not a configuration change.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
enabled | bool | 2.3 + 2.4 | — | yes | Enabled allocates hugepages. |
size | string | 2.3 + 2.4 | 2M | no | Size is the hugepage size, e.g. “2M” or “1G”. |
count | int | 2.3 + 2.4 | 2048 | no | Count is the number of pages PER NODE. |
node_role | string | 2.3 + 2.4 | worker | no | NodeRole is the machineconfiguration.openshift.io/role the profile applies to. |
profile_name | string | 2.3 + 2.4 | bnk-hugepages | no | ProfileName names the Tuned profile and CR. |
IBMCloudCfg
| key | type | line | default | required | description |
|---|---|---|---|---|---|
region | string | 2.3 + 2.4 | — | yes | Region is the IBM Cloud region everything in this workspace is created in or adopted from, e.g. “us-east”. |
resource_group | string | 2.3 + 2.4 | default | yes | ResourceGroup is the IBM Cloud resource group resources are placed in, and the one an adopted cluster is looked up in. |
api_key_source | string | 2.3 + 2.4 | — | no | env | keychain | config | prompt — see secrets.go. |
api_key_b64 | string | 2.3 + 2.4 | — | no | APIKeyB64 stores the API key base64-encoded inline in the workspace config. |
RegistryCfg
RegistryCfg configures the Sprint 29 air-gap registry mirror (PRD 11): which target the roksbnkctl registry replicate populates and which namespace it uses, plus the optional source/target credentials. All fields are optional — an absent block (nil) means the mirror is not configured and the BNK install pulls directly from FAR (far_repo_url). Additive + omitempty, so existing config.yaml files load unchanged.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
target | string | 2.3 + 2.4 | — | no | Target selects the mirror backend: “icr” (IBM Container Registry — the DEFAULT when unset) or “generic” (any OCI-compliant registry — Artifactory / Harbor / Quay / registry:2). |
icr_host | string | 2.3 + 2.4 | — | no | ICRHost overrides the IBM Container Registry host for target=icr (e.g. “de.icr.io”). |
icr_namespace | string | 2.3 + 2.4 | — | no | ICRNamespace is the ICR namespace artifacts nest under for target=icr. |
generic_host | string | 2.3 + 2.4 | — | no | GenericHost is the OCI registry host for target=generic (e.g. “artifactory.example.com”). |
generic_repo_prefix | string | 2.3 + 2.4 | — | no | GenericRepoPrefix is the repository path artifacts nest under for target=generic (e.g. an Artifactory repo key). |
generic_username | string | 2.3 + 2.4 | — | no | GenericUsername / GenericPasswordB64 are the static basic-auth credential for target=generic (an Artifactory user + access token). |
generic_password_b64 | string | 2.3 + 2.4 | — | no | GenericPasswordB64 is that user’s password or access token, base64-encoded — obfuscation, not encryption. |
generic_ca_b64 | string | 2.3 + 2.4 | — | no | GenericCAB64 is the mirror’s CA chain, PEM, base64-encoded — the AUTHORITATIVE copy, recorded from the file that generated it rather than learned from the network. |
generic_ca_sha256 | string | 2.3 + 2.4 | — | no | GenericCASHA256 pins the mirror’s CA by SHA-256 (hex; a “sha256:” prefix and colons are accepted). |
namespace | string | 2.3 + 2.4 | — | no | Namespace is the mirror project the artifacts land in. |
include_deps | *bool | 2.3 + 2.4 | — | no | IncludeDeps unions the non-F5 dependency artifacts (Jetstack cert-manager chart + images, the registry.k8s.io/kubectl node-labeler image) into the BOM. |
source_service_account_b64 | string | 2.3 + 2.4 | — | no | SourceServiceAccountB64 is the FAR _json_key_base64 service-account JSON, base64-encoded, used as the replication SOURCE credential for repo.f5.com. |
ResourceToggle
ResourceToggle is one create/reuse decision: Create=true provisions the resource (under its prefix-derived name); Create=false reuses an existing one named by Existing (when a live dependent consumes it).
| key | type | line | default | required | description |
|---|---|---|---|---|---|
create | bool | 2.3 + 2.4 | — | yes | Create decides whether roksbnkctl provisions this resource or adopts one that already exists. |
existing | string | 2.3 + 2.4 | — | no | Existing is the name or ID of the resource to adopt when Create is false. |
ResourcesCfg
ResourcesCfg holds the per-resource create toggles for a prefix-driven workspace (Sprint 26). The cluster itself is NOT here — it reuses the existing ClusterCfg.Create / ClusterCfg.Name (Name doubles as the existing id/name when Create=false, as today). Each toggle carries an Existing name/ID used when Create=false and a live dependent still needs to reference the resource by name.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
transit_gateway | ResourceToggle | 2.3 + 2.4 | — | yes | TransitGateway controls the transit gateway the cluster VPC attaches to. |
registry_cos | ResourceToggle | 2.3 + 2.4 | — | yes | RegistryCOS controls the IBM Cloud Object Storage bucket that backs a mirror registry. |
cert_manager | ResourceToggle | 2.3 + 2.4 | — | yes | CertManager controls whether cert-manager is INSTALLED or adopted. |
bnk | ResourceToggle | 2.3 + 2.4 | — | yes | BNK controls whether the BIG-IP Next for Kubernetes install runs at all. |
tgw_jumphost | ResourceToggle | 2.3 + 2.4 | — | yes | TGWJumphost controls the optional testing jumphost in the client VPC. |
cluster_jumphosts | ResourceToggle | 2.3 + 2.4 | — | yes | ClusterJumphosts controls the per-cluster jumphosts. |
client_vpc | ResourceToggle | 2.3 + 2.4 | — | yes | ClientVPC controls the testing client VPC that the jumphost lives in. |
cluster_vpc | ResourceToggle | 2.3 + 2.4 | — | yes | ClusterVPC controls the ROKS cluster’s OWN VPC. |
client_region | string | 2.3 + 2.4 | — | no | ClientRegion is the region the testing client (TGW jumphost + client VPC) is installed in. |
testing_client_vpc_name | string | 2.3 + 2.4 | tf-testing-vpc | no | TestingClientVPCName names the testing client VPC when ClientVPC.Create is true (rendered as testing_client_vpc_name). |
testing_ssh_key_name | string | 2.3 + 2.4 | — | no | TestingSSHKeyName is the IBM Cloud VPC SSH key name attached to the testing jumphosts (rendered as testing_ssh_key_name). |
testing_jumphost_profile | string | 2.3 + 2.4 | — | no | Jumphost sizing. |
testing_min_vcpu_count | int | 2.3 + 2.4 | 4 | no | TestingMinVCPUCount is the vCPU floor for the jumphost auto-select. |
testing_min_memory_gb | int | 2.3 + 2.4 | 8 | no | TestingMinMemoryGB is the memory floor for the same auto-select. |
testing_jumphost_allowed_cidrs | []string | 2.3 + 2.4 | — | no | Security-group source CIDRs, following the flp_vsi module’s split between a management plane and an in-fabric plane. |
testing_client_vpc_inbound_cidrs | []string | 2.3 + 2.4 | — | no | TestingClientVPCInboundCIDRs gates the client VPC’s DEFAULT security group, which the testing phase widens to all protocols and ports. |
cluster_http_allowed_cidrs | []string | 2.3 + 2.4 | — | no | ClusterHTTPAllowedCIDRs gates :80 on the cluster security group — the ingress/ALB path, which is meant to be publicly reachable, so empty means open. |
cluster_vpc_default_sg_inbound_cidrs | []string | 2.3 + 2.4 | — | no | ClusterVPCDefaultSGInboundCIDRs gates the cluster VPC’s DEFAULT security group, which the cluster phase widens to all protocols and ports. |
copied_ssh_key_files | []string | 2.3 + 2.4 | — | no | CopiedSSHKeyFiles lists the ~/.ssh basenames roksbnkctl init ACTUALLY wrote when the user accepted the “copy the private key to ~/.ssh” prompt (only files it created — pre-existing files are skipped, never recorded). |
StateCfg
GatewayCfg carries optional overrides for the Gateway phase (the BNK data-plane config). Every field is optional — unset values fall back to the terraform gateway module’s BNK install-guide defaults. Rendered as gateway_* tfvars. The phase itself is driven by roksbnkctl gateway up/down, not a toggle here. StateCfg selects where terraform state lives (PRD 16). Backend “” or “local” (the default) keeps per-phase local tfstate under the workspace dir — the original behaviour. “s3” stores each phase’s state in an S3-compatible bucket (IBM COS), so a stateless runner / parallel CI needs no shared volume, with native lockfile locking (terraform >= 1.10). Additive + omitempty — an absent state: block loads as the local default.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
backend | string | 2.3 + 2.4 | — | no | “” | “local” | “s3”. |
s3 | *StateS3Cfg | 2.3 + 2.4 | — | no | S3 configures an S3-compatible backend (IBM COS) for terraform state, so a workspace is not tied to one machine’s disk. |
StateS3Cfg
StateS3Cfg configures the COS/S3 remote backend. The HMAC access/secret keys are NOT stored here — KeySource names the env var they come from (env-first), and roksbnkctl injects them as AWS_ env to the terraform child, never into the rendered HCL or the state object.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
endpoint | string | 2.3 + 2.4 | — | yes | COS S3 endpoint URL. |
bucket | string | 2.3 + 2.4 | — | yes | pre-provisioned bucket. |
region | string | 2.3 + 2.4 | — | yes | COS location / region. |
key_prefix | string | 2.3 + 2.4 | — | no | default: the workspace name. |
access_key_source | string | 2.3 + 2.4 | — | no | env var name; default ROKSBNKCTL_COS_HMAC_ACCESS_KEY. |
secret_key_source | string | 2.3 + 2.4 | — | no | env var name; default ROKSBNKCTL_COS_HMAC_SECRET_KEY. |
TFSourceCfg
TFSourceCfg picks where Terraform’s source tree comes from. Type drives which other fields apply: - embedded — uses the HCL bundled into the roksbnkctl binary via Go’s embed directive. No other fields needed. This is the default and what most users want; install one binary, get CLI + matched TF together. - github — downloads a tarball release from a GitHub repo. Repo (“owner/name”) and Ref (release tag) required. For testing forks or pinning to a specific upstream tag. - local — points Terraform at a directory on disk. Path required. For active development on the HCL itself. An empty Type (legacy / forgot-to-set) is treated as embedded.
| key | type | line | default | required | description |
|---|---|---|---|---|---|
type | string | 2.3 + 2.4 | — | yes | Type selects where the Terraform comes from: “embedded” (the tree compiled into this binary — the default, and the only one guaranteed to match it), “github” (a released tree), or “local” (a path on disk, for testing a fork). |
repo | string | 2.3 + 2.4 | — | no | Repo is the owner/name to fetch from when Type is “github”. |
ref | string | 2.3 + 2.4 | — | no | Ref is the tag, branch or commit to fetch when Type is “github”. |
path | string | 2.3 + 2.4 | — | no | Path is the local directory holding the Terraform tree when Type is “local”. |
TargetCfg
TargetCfg is the on-disk shape of one entry under targets: in the workspace config. Lives in this package (rather than internal/remote) to avoid an import cycle: workspace.go needs to (de)serialise it, internal/remote needs to consume it. Keeping the wire shape here and the runtime Target type in internal/remote keeps the dep direction clean (remote → config, never the reverse).
| key | type | line | default | required | description |
|---|---|---|---|---|---|
host | string | 2.3 + 2.4 | — | yes | Host is the target’s address, reached over SSH. |
port | int | 2.3 + 2.4 | 22 | no | Port is its SSH port. |
user | string | 2.3 + 2.4 | — | yes | User is the SSH login. |
key_path | string | 2.3 + 2.4 | — | no | file path (PEM). |
key_source | string | 2.3 + 2.4 | — | no | “agent” | “tf-output:<name>”. |
TestCfg
| key | type | line | default | required | description |
|---|---|---|---|---|---|
throughput | ThroughputCfg | 2.3 + 2.4 | — | no | Throughput configures the iperf3 bandwidth probe. |
connectivity | ConnectivityCfg | 2.3 + 2.4 | — | no | Connectivity configures the reachability probes — the hosts roksbnkctl test tries to reach, and from where. |
dns | DNSCfg | 2.3 + 2.4 | — | no | DNS configures the name-resolution probes, including which resolvers to ask. |
ThroughputCfg
| key | type | line | default | required | description |
|---|---|---|---|---|---|
image | string | 2.3 + 2.4 | networkstatic/iperf3:latest | no | default: networkstatic/iperf3:latest. |
duration | int | 2.3 + 2.4 | 30 | no | seconds; default 30. |
streams | int | 2.3 + 2.4 | 8 | no | parallel; default 8. |
default_mode | string | 2.3 + 2.4 | north-south | no | north-south | east-west. |
Terraform variable reference
Auto-generated by go run ./tools/refgen/tfvars-md > book/src/29-terraform-variable-reference.md. Re-run on every terraform/variables.tf change.
Every variable below is settable via terraform.tfvars, -var, -var-file, or (for sensitive values) the corresponding TF_VAR_<name> environment variable. Variables with _required_ defaults must be set explicitly. See Chapter 13 for how roksbnkctl threads these through the workspace config.
Root module variables
Source: terraform/variables.tf
| Variable | Type | Default | Description | Sensitive |
|---|---|---|---|---|
ibmcloud_api_key | string | required | IBM Cloud API key | yes |
ibmcloud_cluster_region | string | "ca-tor" | IBM Cloud region for all cluster resources | no |
ibmcloud_resource_group | string | "default" | IBM Cloud resource group name | no |
create_roks_cluster | bool | true | Create a new ROKS cluster. When false, supply roks_cluster_id_or_name instead. | no |
cluster_public_gateway | bool | true | Attach a public gateway to each cluster subnet for worker Internet egress. true (default) keeps current behavior; false builds a private, disconnected cluster with no egress (operator must supply private connectivity — VPEs / private service endpoints — for image pulls and IBM Cloud services). | no |
roks_cluster_id_or_name | string | "" | ID or name of an existing ROKS cluster — used when create_roks_cluster = false | no |
create_roks_transit_gateway | bool | true | Create Transit Gateway and VPC connections | no |
create_roks_registry_cos_instance | bool | true | Create Cloud Object Storage instance for the OpenShift image registry | no |
deploy_cert_manager | bool | true | When true, the cert_manager module’s helm/null_resource bring-up runs. Forced false by writeBnkPhaseOverrideAt when cluster-outputs.json exists (cluster phase already deployed cert_manager; trial phase consumes it via outputs that resolve to null on the bnk-phase apply, and downstream gates fall back to "direct-apply"). | no |
roks_cluster_vpc_name | string | "tf-cluster-vpc" | Name of the cluster VPC | no |
openshift_cluster_name | string | "tf-openshift-cluster" | Name of the OpenShift cluster | no |
openshift_cluster_version | string | "4.21" | OpenShift cluster version as a BARE major.minor prefix (e.g. “4.20”). Empty uses the latest available. | no |
roks_workers_per_zone | number | 1 | Number of worker nodes per availability zone | no |
roks_min_worker_vcpu_count | number | 16 | Minimum vCPU count when auto-selecting the worker node flavor | no |
roks_min_worker_memory_gb | number | 64 | Minimum memory in GB when auto-selecting the worker node flavor | no |
roks_cos_instance_name | string | "tf-openshift-cos-instance" | Name of the COS instance for the OpenShift image registry | no |
roks_transit_gateway_name | string | "tf-tgw" | Name of the Transit Gateway. Must reference an existing TGW when create_roks_transit_gateway = false and testing_create_tgw_jumphost = true. | no |
use_existing_cluster_vpc | bool | false | Reuse an existing cluster VPC instead of creating one. roksbnkctl sets this true in the second (bnk/testing) phase when cluster-outputs.json exists; the cluster phase leaves it false (create). | no |
existing_cluster_vpc_id | string | "" | ID of the existing cluster VPC (used only when use_existing_cluster_vpc = true) — sourced from cluster-outputs.json vpc_id. | no |
install_cert_manager | bool | true | Install cert-manager. When false, cert_manager_namespace is passed directly to flo. | no |
cert_manager_namespace | string | "cert-manager" | Kubernetes namespace for cert-manager | no |
cert_manager_version | string | "v1.17.3" | cert-manager Helm chart version | no |
ibmcloud_cos_resource_group | string | "" | Resource group holding the supply-chain COS instance. Empty resolves the workspace’s own resource group, which is the pre-#295 behaviour. Set it when the supply chain lives centrally (typically default) and the workspace was placed elsewhere. | no |
ibmcloud_cos_bucket_region | string | "us-south" | IBM Cloud region where the COS bucket is located | no |
ibmcloud_cos_instance_name | string | "bnk-supply-chain" | IBM Cloud COS instance name | no |
ibmcloud_resources_cos_bucket | string | "bnk-artifacts" | IBM Cloud COS bucket containing FAR auth key and JWT files | no |
deploy_bnk | bool | true | Deploy BIG-IP Next for Kubernetes — creates flo, cne_instance, and license. When false all three modules are skipped. | no |
far_repo_url | string | "repo.f5.com" | FAR repository URL for Docker and Helm images | no |
far_chart_repo_url | string | "" | Chart-pull host for the air-gap mirror (helm_release repository + manifest pull). Empty falls back to far_repo_url. | no |
far_image_repo_url | string | "" | Image-pull host for the air-gap mirror (image.repository + CNEInstance spec.registry.uri). Empty falls back to far_repo_url. | no |
use_registry_mirror | bool | false | Pull charts + images from the registry mirror instead of FAR. The far-secret dockerconfig is dropped; how pods then authenticate depends on the mirror: an in-cluster/ICR mirror authorizes by RBAC and needs no pull secret, while an EXTERNAL mirror (Harbor, Artifactory) gets a mirror-secret dockerconfig built from registry_mirror_username/password. A private mirror therefore needs no anonymous/public project. | no |
registry_mirror_username | string | "" | Basic-auth username for an EXTERNAL registry mirror (e.g. a Harbor robot/admin). Empty → the mirror is the in-cluster/ICR registry, which authenticates via the kube token / IAM key instead. | no |
registry_mirror_password | string | "" | Basic-auth password/token for an external registry mirror. When set (with use_registry_mirror), chart and image pulls authenticate to the mirror with these credentials instead of the in-cluster kube token. | yes |
f5_bigip_k8s_manifest_version | string | "2.3.0-3.2598.3-0.0.170" | Version of the f5-bigip-k8s-manifest chart (FLO and CIS versions are extracted from this) | no |
f5_cne_far_auth_file | string | "f5-far-auth-key.tgz" | FAR auth key filename in the COS bucket (.tgz) | no |
f5_cne_subscription_jwt_file | string | "subscription.jwt" | Subscription JWT filename in the COS bucket — used by flo and license | no |
use_cos_bucket | bool | true | Download the FAR auth tarball + subscription JWT from the orchestration COS. false = use the injected far_service_account_b64 / f5_cne_subscription_jwt content instead (local files). | no |
far_service_account_b64 | string | "" | FAR _json_key_base64 service account (base64 of the .json), injected when use_cos_bucket = false. Empty on the COS path. | yes |
f5_cne_subscription_jwt | string | "" | Subscription/license JWT token content, injected when use_cos_bucket = false. Empty on the COS path (downloaded from COS instead). | yes |
bnk_line | string | "2.3" | BNK release line driving per-release resource gating (2.3 or 2.4). Derived by roksbnkctl from bnk.manifest_version — set it by hand only for a standalone terraform run. | no |
flo_namespace | string | "f5-bnk" | Kubernetes namespace for the F5 Lifecycle Operator | no |
flo_utils_namespace | string | "f5-utils" | Kubernetes namespace for F5 utility components — used by flo, cne_instance, and license. Set equal to flo_namespace to install into ONE namespace. | no |
bigip_username | string | "admin" | BIG-IP username for the CIS controller | no |
bigip_password | string | "admin" | BIG-IP password for the CIS controller | yes |
bigip_url | string | "192.168.1.245" | BIG-IP URL for the CIS controller | no |
flo_trusted_profile_id | string | "" | IBM Cloud Trusted Profile ID created by flo — wired automatically from flo output; set here to override | no |
flo_trusted_profile_sa_name | string | "" | Kubernetes service account name the CNE controller’s IBM Cloud Trusted Profile is linked to — i.e. which service account may ASSUME the profile and act on the VPC. | no |
flo_trusted_profile_roles | list(string) | ["Viewer", "Editor"] | IAM roles granted to the CNE controller’s Trusted Profile, scoped to the cluster’s OWN VPC (serviceName=is, vpcId=<cluster vpc>). | no |
flo_cluster_issuer_name | string | "" | Kubernetes ClusterIssuer name created by flo — wired automatically from flo output; set here to override | no |
cneinstance_network_attachments | list(string) | ["ens3-ipvlan-l2", "macvlan-conf"] | Network attachment names for cne_instance — wired automatically from flo output; set here to override | no |
cneinstance_deployment_size | string | "" | Deployment size for CNEInstance (Tiny, Small, Medium, Large). EMPTY takes the line default: Small on 2.3, Tiny on 2.4 (what the 2.4 install guide and F5’s reference cluster use). Tiny is what the BNK 2.4 install guide uses; it is passed through unvalidated, so a size a given manifest does not define is rejected by the operator, not here. | no |
cneinstance_gslb_datacenter_name | string | "" | GSLB datacenter name for CNEInstance (optional) | no |
cneinstance_network_zones | list(object({ | [] | Per-zone subnet CIDRs + TMM self-IPs (empty = use install-guide defaults) | no |
cneinstance_vlan_prefixlen_external | number | 0 | External VLAN self-IP prefix length. 0 (default) → cneinstance_vlan_prefixlen, so a deployment that does not care keeps one knob and one value. | no |
cneinstance_vlan_prefixlen_internal | number | 0 | Internal VLAN self-IP prefix length. 0 (default) → cneinstance_vlan_prefixlen. See the external variable for why the two can differ. | no |
cneinstance_vlan_prefixlen | number | 24 | TMM self-IP prefix length (spec.prefixlen_v4) for the external/internal F5SPKVlan CRs | no |
cneinstance_tmm_k8s_routes | string | "172.17.0.0/18" | Pod CIDR TMM routes to (advanced.tmm.env TMM_K8S_ROUTES). Default is the ROKS default pod subnet. | no |
license_mode | string | "connected" | License operation mode (connected, disconnected, or f5licenseproxy) | no |
flp_license_server_url | string | "" | Base URL of the in-cluster F5 License Proxy service (FLP mode only; e.g. https://f5-license-proxy.<ns>.svc.cluster.local:8443) | no |
license_server_root_ca | string | "" | PEM of the FLP root CA, written into the licenseserver-rootca Secret so CWC trusts the proxy (FLP mode only) | no |
testing_create_tgw_jumphost | bool | true | Create a jumphost in a client VPC connected to the cluster via the Transit Gateway | no |
testing_create_cluster_jumphosts | bool | false | Create one jumphost per availability zone directly inside the cluster VPC | no |
testing_ssh_key_name | string | "" | Name of the IBM Cloud SSH key to inject into all jumphosts | no |
testing_jumphost_profile | string | "" | Instance profile for all jumphosts (leave empty to auto-select based on min_vcpu_count and min_memory_gb) | no |
testing_jumphost_total_volume_bandwidth | number | 1000 | Volume bandwidth (Mbps) pinned on each jumphost. IBM computes this from the profile at creation and terraform carries the stored value into a later profile change, so an unmanaged value taken from an oversized profile makes a downsize impossible (#316). 1000 is ample for jumphosts, which run curl/iperf3 and no disk-heavy work, and is below the ceiling of every profile the auto-select can pick. | no |
testing_min_vcpu_count | number | 4 | Minimum vCPU count when auto-selecting the jumphost instance profile | no |
testing_min_memory_gb | number | 8 | Minimum memory in GB when auto-selecting the jumphost instance profile | no |
testing_create_client_vpc | bool | false | Create a new client VPC for the TGW jumphost. When false, testing_client_vpc_name must reference an existing VPC. | no |
testing_client_vpc_name | string | "tf-testing-vpc" | Name of the client VPC — created when testing_create_client_vpc = true, or looked up when false | no |
testing_client_vpc_region | string | "ca-tor" | IBM Cloud region for the client VPC and TGW jumphost | no |
testing_tgw_jumphost_name | string | "tf-testing-jumphost-tgw" | Name of the TGW-connected jumphost instance | no |
testing_cluster_jumphost_name_prefix | string | "tf-testing-jumphost-cluster" | Name prefix for cluster jumphosts — zone name is appended (<prefix>-<zone>) | no |
kubeconfig_dir | string | "/work/.bnk/scratch/kubeconfig" | Parent directory where ibm_container_cluster_config writes admin kubeconfigs. Each submodule appends its name as a subdir. Default is the bnk runner image’s /work mount; override for direct-on-host runs. | no |
scratch_dir | string | "/work/.bnk/scratch" | Persistent scratch directory for FLO’s FAR/manifest cross-apply artifacts. Default is the bnk runner image’s /work mount; override for direct-on-host runs. | no |
deploy_gateway | bool | false | Master toggle for the Gateway phase. Off in every other phase’s override; on only for gateway up. | no |
deploy_flp | bool | false | Master toggle for the F5 License Proxy phase. Off in every other phase’s override; on only for flp up. | no |
flp_namespace | string | "f5-license-proxy" | Namespace the F5 License Proxy is installed into (FLP phase). | no |
flp_chart_version | string | "" | Pin for the f5-license-proxy chart version (empty → registry latest). | no |
flp_storage_class | string | "ibmc-vpc-block-metro-10iops-tier" | Dynamic StorageClass for the FLP’s PVCs (FLP phase). Default = ROKS VPC block. | no |
gateway_app_namespace | string | "f5-app" | Application namespace the Gateway + HTTPRoute serve (created by the gateway module) | no |
gateway_class_name | string | "gateway-class" | GatewayClass name. Set it to run more than one BNK GatewayClass in a cluster — GatewayClass is cluster-scoped, so two installs sharing the name collide. | no |
gateway_route_examples | list(string) | [] | Extra route kinds to create working examples of, alongside the default HTTPRoute. Valid on BNK 2.3: GRPCRoute, L4Route. Empty (default) leaves an existing deployment byte-identical. L4Route also adds a TCP listener to the Gateway, because an L4Route cannot attach to an HTTP one. | no |
gateway_l4_listener_port | number | 8080 | Port for the TCP listener added when gateway_route_examples includes L4Route | no |
gateway_controller_name | string | "" | GatewayClass controllerName. Empty → derived as f5.com/<flo_namespace>-f5-cne-controller, which is the value the CNE controller answers to. Set it only to point the GatewayClass at a controller this deployment did not install; a wrong value fails silently (the GatewayClass is never Accepted and the apply still succeeds). | no |
gateway_backend_service | string | "nginx-service" | HTTPRoute backend Service name in the app namespace | no |
gateway_backend_port | number | 80 | HTTPRoute backend Service port | no |
gateway_egress_mode | string | "snatpool" | Egress SNAT strategy: snatpool (default), automap, or both | no |
gateway_client_subnet_local | list(string) | [] | Local-VSI client subnet CIDRs the static routes reach (cluster-VPC clients; one route per entry × zone). Empty = no local client routes. gateway up auto-derives these from the cluster jumphost subnets when unset (PRD 12). | no |
gateway_client_subnet_remote | list(string) | [] | Remote-VSI client subnet CIDRs the static routes reach (client-VPC clients over the TGW; one route per entry × zone). Empty = no remote client routes. | no |
gateway_vxlan_port | number | 6789 | Egress VXLAN UDP port (also opened on the cluster security group) | no |
flp_node_port_access | bool | false | Expose the F5 License Proxy outside its own cluster (NodePort + worker-node-IP cert SANs), so a BNK install in a different cluster can license through it. | no |
flp_node_port_source_cidrs | list(string) | [] | With flp_node_port_access: open the proxy’s NodePort on the worker security group to these CIDRs. A LIST — a multi-zone VPC has one address prefix per zone, and a consuming pod in an unlisted zone is silently dropped. | no |
roksbnkctl_binary | string | "" | Absolute path to the roksbnkctl binary. helm invokes it as the f5-license-proxy chart’s post-renderer (roksbnkctl flp postrender), so the FLP install needs no interpreter on the host. roksbnkctl sets this automatically via TF_VAR_roksbnkctl_binary; empty falls back to roksbnkctl on PATH. | no |
deploy_flp_vsi | bool | false | Deploy the F5 License Proxy as a standalone VSI (podman pod, no k8s) instead of the helm chart. Set true only by the FLP phase in mode: vsi. | no |
flp_status_image | string | "" | Optional flp-status web UI image for the standalone FLP VSI (mirror/public ref). Empty = no status UI. | no |
flp_status_registry_host | string | "" | Registry host:port whose CA to trust so the FLP VSI can pull flp_status_image (e.g. Harbor’s <ip>). | no |
flp_status_registry_ca_b64 | string | "" | Base64 CA cert for flp_status_registry_host. | no |
flp_vsi_ssh_key | string | "" | Existing IBM Cloud VPC SSH key name to attach to the standalone FLP VSI (operator access). Empty = no key. | no |
flp_vsi_profile | string | "bx2-4x16" | VSI instance profile for the FLP (>= 4 vCPU / 8 GB). | no |
flp_vsi_zone | string | "" | Zone for the FLP VSI. Empty → <region>-1. | no |
flp_vsi_boot_size_gb | number | 100 | Boot volume size (GB) for the FLP VSI (>= 80). | no |
flp_vsi_floating_ip | bool | true | Attach an operator floating IP to the FLP VSI for remote management (flp status + web UI + 8443 from another machine). Not the CWC endpoint. Reachability still gated by flp_vsi_allowed_cidrs. Default true. | no |
flp_vsi_management_allowed_cidrs | list(string) | [] | Source CIDRs for the FLP VSI’s :80 flp-status web UI (read-only). Empty → 0.0.0.0/0 (open). | no |
flp_vsi_licensing_allowed_cidrs | list(string) | [] | Source CIDRs for the FLP VSI’s :8443 proxy (+ :22 SSH). Empty → RFC-1918 private ranges. | no |
flp_vsi_allowed_cidrs | list(string) | [] | DEPRECATED — legacy single list; seeds both management + licensing when set. Prefer the two per-plane variables. | no |
flp_prod_jwks_b64 | string | "" | Optional override: base64 of F5’s public prod_jwks.txt. Empty → the flp_vsi module extracts it from the f5-license-proxy chart. | no |
flp_forward_proxy_host | string | "" | — | no |
flp_forward_proxy_port | number | 0 | — | no |
flp_forward_proxy_protocol | string | "http" | — | no |
deploy_tgw_connection | bool | false | Attach the cluster’s VPC to an existing Transit Gateway. On only for the tgw phase; a no-op everywhere else. | no |
tgw_connection_target | string | "" | Existing Transit Gateway to attach the cluster VPC to, by NAME or ID. Multiple clusters passing the same value share one gateway. | no |
tgw_connection_name | string | "" | Name for this cluster’s connection on the gateway (unique per gateway; prefix-derived so shared-gateway clusters don’t collide). | no |
cluster_absent | bool | false | True only in the standalone FLP-VSI phase: no ROKS cluster exists or will be adopted, so all cluster data-source lookups + kube providers across modules are skipped (count=0). | no |
cluster_vpc_cidr | string | "" | CIDR block the cluster VPC’s per-zone address prefixes are carved from (e.g. “10.241.0.0/16” → 10.241.0.0/18, 10.241.64.0/18, 10.241.128.0/18). | no |
use_existing_cluster_subnets | bool | false | Place the cluster in subnets that already exist instead of creating them. Requires use_existing_cluster_vpc — a subnet cannot be adopted independently of its VPC. | no |
existing_cluster_subnet_ids | list(string) | [] | Subnet ids to place the cluster in, one per zone, in zone order. Used only when use_existing_cluster_subnets = true. Their zones are read from the subnets themselves. | no |
flp_vsi_create_vpc | bool | false | Build the F5 License Proxy its own VPC instead of placing it in an existing one. | no |
flp_vsi_name_prefix | string | "" | Prefix for the standalone FLP VSI’s resource names (e.g. bnk-ci → bnk-ci-flp-vsi). Empty (the default) keeps the legacy unprefixed names so an existing proxy is NOT replaced on upgrade. Set it to run more than one FLP in an account, and to bring the resources into roksbnkctl cleanup’s <prefix>-* sweep. | no |
flp_vsi_vpc_name | string | "" | Name for the VPC created when flp_vsi_create_vpc = true. | no |
flp_vsi_subnet_cidr | string | "10.250.0.0/24" | Address prefix for the VPC created when flp_vsi_create_vpc = true. | no |
cluster_network_mode | string | "single-nic" | How the cluster’s worker nodes are attached: single-nic (default) or multi-nic. | no |
testing_jumphost_allowed_cidrs | list(string) | [] | Source CIDRs allowed to SSH (:22) to the testing jumphosts. Empty → 0.0.0.0/0 (open; access is key-only). Narrow to the operator’s public /32 on a shared account. | no |
testing_client_vpc_inbound_cidrs | list(string) | [] | Source CIDRs allowed inbound to the testing client VPC’s default security group. Empty → the RFC-1918 private ranges (in-fabric test traffic arrives over the Transit Gateway). | no |
cluster_http_allowed_cidrs | list(string) | [] | Source CIDRs allowed to reach :80 on the cluster security group. Empty → 0.0.0.0/0 (the ingress/ALB path is meant to be publicly reachable). | no |
cluster_vpc_default_sg_inbound_cidrs | list(string) | [] | Source CIDRs allowed inbound (all protocols/ports) to the cluster VPC’s default security group. Empty → 0.0.0.0/0, the historical behaviour. Narrow to your private ranges unless a workload in this VPC needs a public source. | no |
cneinstance_tmm_resources | map(map(string)) | {} | Overrides the TMM pod’s resource requests/limits, rendered as the CNEInstance’s advanced.tmm.resources (#203). | no |
cneinstance_advanced_env | map(map(string)) | {} | Per-component environment passthrough for the BNK 2.4 CNEInstance’s advanced.<component>.env[] lists (#175). | no |
cneinstance_tmm_replicas | number | 3 | Number of f5-tmm data-plane replicas (2.4). Reference: 3. | no |
cneinstance_watch_namespaces | list(string) | ["All"] | Namespaces the CNE controller watches (2.4). Reference: ["All"]. | no |
cneinstance_tmm_anti_affinity | bool | true | Require f5-tmm pods onto DIFFERENT NODES (2.4). | no |
cneinstance_tmm_zone_spread | bool | true | Spread f5-tmm pods across zones with maxSkew 1, DoNotSchedule (2.4). Reference: on. | no |
cneinstance_tmm_rolling_update | bool | true | Pin TMM’s rolling update to maxSurge 0 / maxUnavailable 1 (2.4). | no |
cneinstance_external_bigip | bool | false | Enable the external BIG-IP controller (2.4). Reference: true. | no |
cneinstance_external_bigip_login_secret | string | "f5-bigip-ctlr-login" | Secret holding external BIG-IP credentials (2.4). Reference: f5-bigip-ctlr-login. | no |
cneinstance_cluster_identifier | string | "" | CLUSTER_IDENTIFIER passed to the external BIG-IP controller (2.4). Empty derives from the cluster name. | no |
cneinstance_gateway_api_version | string | "1.5.0" | GATEWAY_API_VERSION for the CNE controller (2.4). Reference: 1.5.0. | no |
gateway_api_bundle_url | string | "" | Where the Gateway API standard-install bundle is FETCHED from when the workspace has no registry mirror recorded (2.4 + gateway_api_mtls). Empty derives the upstream release asset for cneinstance_gateway_api_version. | no |
cneinstance_demo_mode | string | "" | advanced.demoMode.enabled. | no |
cneinstance_tmm_pod_label | string | "f5-tmm" | Value of the app label the placement rules select f5-tmm pods by (2.4). Reference: f5-tmm. | no |
cneinstance_tmm_anti_affinity_topology_key | string | "kubernetes.io/hostname" | Node label the TMM anti-affinity rule spreads across (2.4). | no |
cneinstance_tmm_zone_topology_key | string | "topology.kubernetes.io/zone" | Node label the TMM zone spread uses (2.4). The IBM ROKS zone label. Reference: topology.kubernetes.io/zone. | no |
cneinstance_tmm_zone_max_skew | number | 1 | maxSkew for the TMM zone topology-spread constraint (2.4). Reference: 1. | no |
cneinstance_tmm_zone_when_unsatisfiable | string | "DoNotSchedule" | whenUnsatisfiable for the TMM zone spread (2.4): DoNotSchedule or ScheduleAnyway. Reference: DoNotSchedule. | no |
cneinstance_tcp_settings | map(string) | {} | F5BigTcpSetting field overrides, as a flat map of field name to value. | no |
cneinstance_tcp_settings_name | string | "sys-default-tcp" | Name of the F5BigTcpSetting CR to write. Reference: sys-default-tcp. | no |
cneinstance_whole_cluster_override | string | "" | spec.wholeCluster, as a tri-state. | no |
roks_worker_flavor | string | "" | Exact worker-node flavor, e.g. “cx3d.8x20”. Empty auto-selects. | no |
cneinstance_hugepages | bool | false | Allocate hugepages on the worker pool via the OpenShift Node Tuning Operator. | no |
cneinstance_hugepages_size | string | "2M" | Hugepage size, e.g. 2M or 1G. TMM requests hugepages-2Mi, so 2M is the matching size. | no |
cneinstance_hugepages_count | number | 2048 | Hugepages PER NODE. 2048 x 2M = 4Gi, which is what deploymentSize Small was observed to request. | no |
cneinstance_hugepages_node_role | string | "worker" | machineconfiguration.openshift.io/role the Tuned profile applies to. | no |
cneinstance_hugepages_profile_name | string | "bnk-hugepages" | Name of the Tuned profile and CR. | no |
cneinstance_storage_class_name | string | "" | StorageClass for the CNEInstance’s persistent volumes, TMM’s included. Empty leaves the CR’s own default, which resolves to the cluster default class. | no |
Module: cert_manager
Source: terraform/modules/cert_manager/variables.tf
| Variable | Type | Default | Description | Sensitive |
|---|---|---|---|---|
ibmcloud_api_key | string | required | IBM Cloud API Key | yes |
ibmcloud_cluster_region | string | "ca-tor" | IBM Cloud region where the cluster resides | no |
ibmcloud_resource_group | string | "default" | IBM Cloud Resource Group name (leave empty to use account default) | no |
roks_cluster_name_or_id | string | required | Name or ID of the existing OpenShift ROKS cluster to deploy BNK onto | no |
cert_manager_namespace | string | "cert-manager" | Kubernetes namespace for cert-manager | no |
cert_manager_version | string | "v1.17.3" | cert-manager Helm chart version | no |
cert_manager_image_repository | string | "" | Override the cert-manager controller image repository (air-gap mirror image host). Empty leaves the chart default. | no |
create_roks_cluster | bool | false | When true, cluster is being created by roks_cluster — skip plan-time cluster credential fetch | no |
deploy_cert_manager | bool | true | When true, manage the cert_manager helm/null_resource bring-up. Set false in the bnk-phase override when cluster-outputs.json exists — cluster phase already provisioned cert_manager and the second phase must NOT re-manage it (would attempt kubectl delete namespace cert-manager on a subsequent bnk down). | no |
roks_cluster_dependency_id | string | null | roks_cluster sentinel ID — when set, defers runtime_config fetch to apply time after roks_cluster completes | no |
kubeconfig_dir | string | "/work/.bnk/scratch/kubeconfig/cert_manager" | Persistent, writable dir for ibm_container_cluster_config kubeconfig downloads. Defaults to a host-bind-mounted, module-scoped path under .bnk/scratch. | no |
registry_mirror_username | string | "" | Basic-auth username for an external registry mirror (a private Harbor/Artifactory). Empty → no mirror pull secret is created. | no |
registry_mirror_password | string | "" | Basic-auth password for an external registry mirror. When set with image_repository, cert-manager’s pods pull with a dockerconfig secret built from it instead of anonymously. | yes |
cluster_absent | bool | false | True in the standalone FLP-VSI phase: no ROKS cluster exists, so all cluster data-source lookups + kube providers are skipped (count=0). | no |
Module: cne_instance
Source: terraform/modules/cne_instance/variables.tf
| Variable | Type | Default | Description | Sensitive |
|---|---|---|---|---|
bnk_line | string | "2.3" | BNK release line driving per-release resource gating (“2.3” or “2.4”). | no |
ibmcloud_api_key | string | required | IBM Cloud API Key | yes |
ibmcloud_cluster_region | string | "ca-tor" | IBM Cloud region where the cluster resides | no |
ibmcloud_resource_group | string | "default" | IBM Cloud Resource Group name (leave empty to use account default) | no |
roks_cluster_name_or_id | string | required | Name or ID of the existing OpenShift ROKS cluster to deploy BNK onto | no |
far_repo_url | string | "repo.f5.com" | FAR Repository URL for Docker and Helm registry | no |
far_image_repo_url | string | "" | Image-pull host for the mirror (CNEInstance spec.registry.uri). Empty falls back to far_repo_url. | no |
use_registry_mirror | bool | false | Pull charts + images from the registry mirror instead of FAR. The far-secret dockerconfig is dropped; how pods then authenticate depends on the mirror: an in-cluster/ICR mirror authorizes by RBAC and needs no pull secret, while an EXTERNAL mirror (Harbor, Artifactory) gets a mirror-secret dockerconfig built from registry_mirror_username/password. A private mirror therefore needs no anonymous/public project. | no |
flo_namespace | string | "f5-bnk" | Namespace for F5 Lifecycle Operator | no |
flo_utils_namespace | string | "f5-utils" | Namespace for F5 utility components | no |
f5_bigip_k8s_manifest_version | string | "2.3.0-3.2598.3-0.0.170" | Version of f5-bigip-k8s-manifest chart - used by flo, cneinstance modules | no |
flo_trusted_profile_sa_name | string | "" | The CNE controller service account; must match what the flo module linked. | no |
flo_trusted_profile_id | string | "" | IBM IAM Trusted Profile ID for provisioning VPC routes | no |
flo_cluster_issuer_name | string | "" | mTLS certificate issuer name | no |
cneinstance_deployment_size | string | "" | Deployment size for CNEInstance (Tiny, Small, Medium, Large). EMPTY takes the line default: Small on 2.3, Tiny on 2.4 (what the 2.4 install guide and F5’s reference cluster use). Tiny is what the BNK 2.4 install guide uses. | no |
cneinstance_gslb_datacenter_name | string | "" | GSLB datacenter name for CNEInstance (optional) | no |
cneinstance_network_attachments | list(string) | ["ens3-ipvlan-l2", "macvlan-conf"] | The Multus Network Attachment Definitions for the CNEInstance TMM deployments | no |
cneinstance_network_zones | list(object({ | [] | Per-zone subnet CIDRs + TMM self-IPs (empty = use the install-guide defaults) | no |
cneinstance_vlan_prefixlen_external | number | 0 | External VLAN self-IP prefix length; 0 inherits cneinstance_vlan_prefixlen. | no |
cneinstance_vlan_prefixlen_internal | number | 0 | Internal VLAN self-IP prefix length; 0 inherits cneinstance_vlan_prefixlen. | no |
cneinstance_vlan_prefixlen | number | 24 | TMM self-IP prefix length (spec.prefixlen_v4) for the external/internal F5SPKVlan CRs | no |
cneinstance_tmm_k8s_routes | string | "172.17.0.0/18" | Pod CIDR TMM routes to (advanced.tmm.env TMM_K8S_ROUTES). Default is the ROKS default pod subnet. | no |
create_roks_cluster | bool | false | When true, cluster is being created by roks_cluster — skip plan-time cluster credential fetch | no |
roks_cluster_dependency_id | string | null | roks_cluster sentinel ID — when set, defers runtime_config fetch to apply time after roks_cluster completes | no |
flo_dependency_id | string | null | flo_ready sentinel ID — pass module.flo.flo_ready_id to defer cne_instance until flo completes and CRDs are registered | no |
deploy_bnk | bool | true | Deploy BIG-IP Next for Kubernetes — when false the inner cneinstance module is disabled and no CNEInstance resources are created | no |
kubeconfig_dir | string | "/work/.bnk/scratch/kubeconfig/cne_instance" | Persistent, writable dir for ibm_container_cluster_config kubeconfig downloads. Defaults to a host-bind-mounted, module-scoped path under .bnk/scratch. | no |
registry_mirror_password | string | "" | Basic-auth password for an external registry mirror. When set with use_registry_mirror, the CNEInstance references the mirror-secret pull secret instead of pulling anonymously. | yes |
roksbnkctl_binary | string | "" | Absolute path to the roksbnkctl binary; the CNE-instance phase invokes roksbnkctl tfx <verb> in place of host curl (no interpreter, so cmd.exe execs it on Windows). Empty falls back to roksbnkctl on PATH. | no |
cluster_absent | bool | false | True in the standalone FLP-VSI phase: no ROKS cluster exists, so all cluster data-source lookups + kube providers are skipped (count=0). | no |
cneinstance_tmm_resources | map(map(string)) | {} | TMM resource requests/limits override; rendered as advanced.tmm.resources (#203). | no |
cneinstance_advanced_env | map(map(string)) | {} | Per-component advanced.<component>.env passthrough (#175). See the root variable of the same name. | no |
cneinstance_tmm_replicas | number | 3 | Number of f5-tmm data-plane replicas (2.4). Reference: 3. | no |
cneinstance_watch_namespaces | list(string) | ["All"] | Namespaces the CNE controller watches (2.4). Reference: ["All"]. | no |
cneinstance_tmm_anti_affinity | bool | true | Require f5-tmm pods onto DIFFERENT NODES (2.4). | no |
cneinstance_tmm_zone_spread | bool | true | Spread f5-tmm pods across zones with maxSkew 1, DoNotSchedule (2.4). Reference: on. | no |
cneinstance_tmm_rolling_update | bool | true | Pin TMM’s rolling update to maxSurge 0 / maxUnavailable 1 (2.4). | no |
cneinstance_external_bigip | bool | false | Enable the external BIG-IP controller (2.4). Reference: true. | no |
cneinstance_external_bigip_login_secret | string | "f5-bigip-ctlr-login" | Secret holding external BIG-IP credentials (2.4). Reference: f5-bigip-ctlr-login. | no |
cneinstance_cluster_identifier | string | "" | CLUSTER_IDENTIFIER passed to the external BIG-IP controller (2.4). Empty derives from the cluster name. | no |
cneinstance_gateway_api_version | string | "1.5.0" | GATEWAY_API_VERSION for the CNE controller (2.4). Reference: 1.5.0. | no |
cneinstance_demo_mode | string | "" | advanced.demoMode.enabled. | no |
cneinstance_tmm_pod_label | string | "f5-tmm" | Value of the app label the placement rules select f5-tmm pods by (2.4). Reference: f5-tmm. | no |
cneinstance_tmm_anti_affinity_topology_key | string | "kubernetes.io/hostname" | Node label the TMM anti-affinity rule spreads across (2.4). | no |
cneinstance_tmm_zone_topology_key | string | "topology.kubernetes.io/zone" | Node label the TMM zone spread uses (2.4). The IBM ROKS zone label. Reference: topology.kubernetes.io/zone. | no |
cneinstance_tmm_zone_max_skew | number | 1 | maxSkew for the TMM zone topology-spread constraint (2.4). Reference: 1. | no |
cneinstance_tmm_zone_when_unsatisfiable | string | "DoNotSchedule" | whenUnsatisfiable for the TMM zone spread (2.4): DoNotSchedule or ScheduleAnyway. Reference: DoNotSchedule. | no |
cneinstance_tcp_settings | map(string) | {} | F5BigTcpSetting field overrides, as a flat map of field name to value. | no |
cneinstance_tcp_settings_name | string | "sys-default-tcp" | Name of the F5BigTcpSetting CR to write. Reference: sys-default-tcp. | no |
cneinstance_whole_cluster_override | string | "" | spec.wholeCluster, as a tri-state. | no |
cneinstance_hugepages | bool | false | Allocate hugepages on the worker pool via the OpenShift Node Tuning Operator. | no |
cneinstance_hugepages_size | string | "2M" | Hugepage size, e.g. 2M or 1G. TMM requests hugepages-2Mi, so 2M is the matching size. | no |
cneinstance_hugepages_count | number | 2048 | Hugepages PER NODE. 2048 x 2M = 4Gi, which is what deploymentSize Small was observed to request. | no |
cneinstance_hugepages_node_role | string | "worker" | machineconfiguration.openshift.io/role the Tuned profile applies to. | no |
cneinstance_hugepages_profile_name | string | "bnk-hugepages" | Name of the Tuned profile and CR. | no |
cneinstance_storage_class_name | string | "" | StorageClass for the CNEInstance’s persistent volumes. See the leaf module variable of the same name (#189). | no |
Module: flo
Source: terraform/modules/flo/variables.tf
| Variable | Type | Default | Description | Sensitive |
|---|---|---|---|---|
bnk_line | string | "2.3" | BNK release line driving per-release resource gating (“2.3” or “2.4”). | no |
ibmcloud_api_key | string | required | IBM Cloud API Key | yes |
ibmcloud_cluster_region | string | "ca-tor" | IBM Cloud region where the cluster resides | no |
ibmcloud_resource_group | string | "default" | IBM Cloud Resource Group name (leave empty to use account default) | no |
roks_cluster_name_or_id | string | required | Name or ID of the existing OpenShift ROKS cluster to deploy BNK onto | no |
far_repo_url | string | "repo.f5.com" | FAR Repository URL for Docker and Helm registry | no |
far_chart_repo_url | string | "" | Chart-pull host for the mirror (helm_release repository + manifest pull). Empty falls back to far_repo_url. | no |
far_image_repo_url | string | "" | Image-pull host for the mirror (image.repository). Empty falls back to far_repo_url. | no |
use_registry_mirror | bool | false | Pull charts + images from the registry mirror instead of FAR. The far-secret dockerconfig is dropped; how pods then authenticate depends on the mirror: an in-cluster/ICR mirror authorizes by RBAC and needs no pull secret, while an EXTERNAL mirror (Harbor, Artifactory) gets a mirror-secret dockerconfig built from registry_mirror_username/password. A private mirror therefore needs no anonymous/public project. | no |
registry_mirror_username | string | "" | Basic-auth username for an external registry mirror (Harbor). Empty → in-cluster/ICR mirror. | no |
registry_mirror_password | string | "" | Basic-auth password/token for an external registry mirror; chart pulls authenticate with it when set. | yes |
f5_bigip_k8s_manifest_version | string | "2.3.0-3.2598.3-0.0.170" | Version of the f5-bigip-k8s-manifest chart (FLO/CIS versions are extracted from this) | no |
use_cos_bucket | bool | true | Fetch FAR auth key and JWT from IBM Cloud Object Storage instead of local variables | no |
far_service_account_b64 | string | "" | FAR _json_key_base64 service account (base64 of the .json), injected when use_cos_bucket = false | yes |
ibmcloud_cos_bucket_region | string | "us-south" | IBM Cloud region where the COS bucket is located | no |
ibmcloud_cos_instance_name | string | "bnk-supply-chain" | IBM Cloud COS instance name | no |
ibmcloud_resources_cos_bucket | string | "bnk-artifacts" | IBM Cloud COS bucket containing the FAR auth key and JWT files | no |
f5_cne_far_auth_file | string | "f5-far-auth-key.tgz" | FAR auth key filename in the COS bucket (.tgz) | no |
flo_trusted_profile_sa_name | string | "" | Service account the CNE controller Trusted Profile is linked to. | no |
flo_trusted_profile_roles | list(string) | ["Viewer", "Editor"] | IAM roles for the CNE controller Trusted Profile, scoped to the cluster VPC. | no |
flo_namespace | string | "f5-bnk" | Namespace for F5 Lifecycle Operator | no |
flo_utils_namespace | string | "f5-utils" | Namespace for F5 utility components | no |
cert_manager_namespace | string | "cert-manager" | Kubernetes namespace for cert-manager - used by cert-manager, flo modules | no |
bigip_username | string | "admin" | BIG-IP username for CIS controller login | no |
bigip_password | string | "admin" | BIG-IP password for CIS controller login | yes |
bigip_url | string | "https://192.168.1.245" | BIG-IP URL for CIS controller login | no |
create_roks_cluster | bool | false | When true, cluster is being created by roks_cluster — skip plan-time cluster credential fetch | no |
roks_cluster_dependency_id | string | null | roks_cluster sentinel ID — when set, defers runtime_config fetch to apply time after roks_cluster completes | no |
cert_manager_dependency_id | string | null | cert_manager ready sentinel ID — when set, blocks flo inner module until cert-manager CRDs are available | no |
deploy_bnk | bool | true | Deploy BIG-IP Next for Kubernetes — when false the inner flo module is disabled and no FLO resources are created | no |
kubeconfig_dir | string | "/work/.bnk/scratch/kubeconfig/flo" | Persistent, writable dir for ibm_container_cluster_config kubeconfig downloads. Defaults to a host-bind-mounted, module-scoped path under .bnk/scratch. | no |
scratch_dir | string | "/work/.bnk/scratch" | Persistent scratch directory for FAR/manifest cross-apply artifacts. Default is the bnk runner image’s /work mount. | no |
roksbnkctl_binary | string | "" | Absolute path to the roksbnkctl binary; the FLO phase invokes roksbnkctl tfx <verb> in place of host curl/tar (no interpreter, so cmd.exe execs it on Windows). Empty falls back to roksbnkctl on PATH. | no |
cluster_absent | bool | false | True in the standalone FLP-VSI phase: no ROKS cluster exists, so all cluster data-source lookups + kube providers are skipped (count=0). | no |
ibmcloud_cos_resource_group | string | "" | Resource group holding the supply-chain COS instance. Empty resolves the workspace’s own resource group (#295). | no |
Module: flp
Source: terraform/modules/flp/variables.tf
| Variable | Type | Default | Description | Sensitive |
|---|---|---|---|---|
deploy_flp | bool | false | Deploy the F5 License Proxy. Off in every other phase’s override; on only for flp up. When false the module is a complete no-op. | no |
create_roks_cluster | bool | false | When true the cluster is being created in this same apply (FLP never runs then); gates the plan-time cluster-config fetch. | no |
roks_cluster_name_or_id | string | "" | Name or id of the existing ROKS cluster to deploy the FLP into. | no |
roks_cluster_dependency_id | any | null | Dependency handle from the cluster phase, deferring apply until the cluster is ready. | no |
kubeconfig_dir | string | "" | Directory the ibm_container_cluster_config data source writes the kubeconfig into. | no |
ibmcloud_api_key | string | "" | IBM Cloud API key (COS/IAM auth + provider config). | yes |
ibmcloud_cluster_region | string | "" | Region of the ROKS cluster (ibm provider). | no |
ibmcloud_resource_group | string | "" | Resource group containing the COS instance (empty = default group). | no |
ibmcloud_cos_instance_name | string | "" | COS service instance holding the FAR auth tarball + subscription JWT. | no |
ibmcloud_resources_cos_bucket | string | "" | COS bucket holding the FAR auth tarball + subscription JWT. | no |
ibmcloud_cos_bucket_region | string | "" | Region of the COS bucket. | no |
f5_cne_far_auth_file | string | "f5-far-auth-key.tgz" | FAR auth tarball object key in the COS bucket (the _json_key_base64 SA lives inside). | no |
f5_cne_subscription_jwt_file | string | "subscription.jwt" | Subscription JWT object key in the COS bucket — seeds flp-jwt-secret. | no |
scratch_dir | string | "/tmp/roksbnkctl-flp" | Working directory for the FAR-auth download/extract. | no |
far_repo_url | string | "repo.f5.com" | FAR registry host (fallback when no mirror). | no |
far_chart_repo_url | string | "" | Mirror host for chart pulls (empty → coalesces to far_repo_url). | no |
far_image_repo_url | string | "" | Mirror host for image pulls (empty → coalesces to far_repo_url). | no |
use_registry_mirror | bool | false | Pull charts + images from the registry mirror instead of FAR. The far-secret dockerconfig is dropped; how pods then authenticate depends on the mirror: an in-cluster/ICR mirror authorizes by RBAC and needs no pull secret, while an EXTERNAL mirror (Harbor, Artifactory) gets a mirror-secret dockerconfig built from registry_mirror_username/password. A private mirror therefore needs no anonymous/public project. | no |
registry_mirror_username | string | "" | Basic-auth username for an EXTERNAL registry mirror (e.g. a Harbor robot/admin). Empty → in-cluster/ICR mirror (kube-token/IAM auth). | no |
registry_mirror_password | string | "" | Basic-auth password/token for an external registry mirror. When set (with use_registry_mirror), chart + image pulls authenticate with it instead of the kube token. | yes |
flp_namespace | string | "f5-license-proxy" | Namespace to install the F5 License Proxy into. | no |
flp_chart_version | string | "" | Pin the f5-license-proxy chart version. Empty (the default) → resolved from the BNK manifest, which lists charts/f5-license-proxy for the release — same as the FLO and CIS charts. Set this only to override the manifest. | no |
f5_bigip_k8s_manifest_version | string | "" | BNK manifest version. The f5-license-proxy chart version is read out of this manifest when flp_chart_version is empty. | no |
flp_node_port_access | bool | false | Expose the proxy OUTSIDE its own cluster so a BNK install in a different cluster can license through it. The chart’s Service is already type NodePort (30001), but it hardcodes externalTrafficPolicy: Local — with one replica only the node running the pod answers — so this flips it to Cluster and adds the worker node IPs to the proxy’s server certificate (without them the remote CWC rejects the TLS handshake). | no |
flp_node_port_source_cidrs | list(string) | [] | With flp_node_port_access: open the proxy’s NodePort on the cluster’s worker security group to these CIDRs (the consuming cluster’s subnets). A LIST, because a multi-zone VPC carries one address prefix per zone — allowing only one means a consuming pod scheduled in another zone is silently dropped at the security group. Empty leaves the security group untouched. | no |
flp_storage_class | string | "ibmc-vpc-block-metro-10iops-tier" | Dynamic StorageClass for the FLP’s PVCs. The chart ships hostPath PVs (incompatible with ROKS multi-node/non-root); a post-renderer drops them and repoints the PVCs here, so the CSI driver provisions block volumes chowned to fsGroup. Default is the ROKS VPC block default. | no |
roksbnkctl_binary | string | "" | Absolute path to the roksbnkctl binary, which helm invokes as the f5-license-proxy chart’s POST-RENDERER (roksbnkctl flp postrender). roksbnkctl sets this to its own path automatically via TF_VAR_roksbnkctl_binary; empty falls back to roksbnkctl on PATH for a direct terraform apply. Replaces a generated python script, which made python3 an undeclared runtime dependency of the FLP phase — absent in the tools-runner container. | no |
ibmcloud_cos_resource_group | string | "" | Resource group holding the supply-chain COS instance. Empty resolves the workspace’s own resource group (#295). | no |
Module: flp_vsi
Source: terraform/modules/flp_vsi/variables.tf
| Variable | Type | Default | Description | Sensitive |
|---|---|---|---|---|
deploy_flp_vsi | bool | false | Master toggle — when false the whole module is a no-op (count=0). Set true only by the FLP phase in mode: vsi. | no |
ibmcloud_api_key | string | required | IBM Cloud API key (provider + COS/IAM REST auth). | yes |
ibmcloud_cluster_region | string | required | Region of the ROKS cluster / where the FLP VSI is provisioned. | no |
ibmcloud_resource_group | string | "" | Resource group for the VSI + network (empty = account default). | no |
flp_vsi_profile | string | "bx2-4x16" | VSI instance profile (>= 4 vCPU / 8 GB). | no |
flp_vsi_zone | string | "" | Zone for the VSI (e.g. us-south-1). Empty → <region>-1. | no |
flp_vsi_boot_size_gb | number | 100 | Boot volume size in GB (>= 80). | no |
flp_vsi_floating_ip | bool | true | Attach an operator floating IP to the FLP VSI for remote management — running roksbnkctl flp status and reaching the :80 web UI + :8443 proxy from another machine. NOT the CWC endpoint (the cluster always reaches the proxy privately). The floating IP is added to the leaf-cert SAN; reachability is still gated by flp_vsi_allowed_cidrs. Default true. | no |
flp_vsi_management_allowed_cidrs | list(string) | [] | Source CIDRs allowed to reach the :80 flp-status web UI (read-only status). Empty → 0.0.0.0/0 (open — the page carries no secrets). | no |
flp_vsi_licensing_allowed_cidrs | list(string) | [] | Source CIDRs allowed to reach the :8443 licensing proxy (and :22 SSH). Empty → the RFC-1918 private ranges (the cluster reaches the proxy privately over the VPC / Transit Gateway). | no |
flp_vsi_allowed_cidrs | list(string) | [] | DEPRECATED — legacy single list. When set, seeds BOTH flp_vsi_management_allowed_cidrs and flp_vsi_licensing_allowed_cidrs. Prefer the two per-plane variables. Empty → the per-plane defaults apply. | no |
existing_cluster_vpc_id | string | "" | The cluster VPC id (from cluster-outputs.json) the FLP VSI joins so the CWC reaches it directly. | no |
flp_vsi_name_prefix | string | "" | Prefix for the FLP VSI’s resource names, e.g. "bnk-ci" yields bnk-ci-flp-vsi. Empty (the default) keeps the legacy unprefixed names, so an existing proxy is not replaced on upgrade. Set it to run more than one standalone FLP in an account, and to make the resources visible to roksbnkctl cleanup. | no |
far_repo_url | string | "repo.f5.com" | FAR repository host for the FLP chart pulls. Same value as the root far_repo_url. | no |
flp_image_registry | string | "" | FAR image host prefix. Empty (the default) derives <far_repo_url>/images, so the image host follows the chart host instead of pinning repo.f5.com independently. | no |
f5_bigip_k8s_manifest_version | string | required | BNK manifest version — the f5-license-proxy chart/image tag is resolved from it (like the helm path) when flp_chart_version is empty. | no |
flp_chart_version | string | "" | Pin the f5-license-proxy chart/image tag. Empty → resolved from the BNK manifest. | no |
flp_vault_image_tag | string | "2.0.0" | Tag for the vault image. | no |
f5_cert_url | string | "https://product.apis.f5.com/ee/v1" | — | no |
f5_entitlement_url | string | "https://product-s.apis.f5.com/ee/v1" | — | no |
f5_initial_config_url | string | "https://product-s.apis.f5.com/ee/v1" | — | no |
mode_of_operation | string | "connected" | — | no |
flp_forward_proxy_host | string | "" | — | no |
flp_forward_proxy_port | number | 0 | — | no |
flp_forward_proxy_protocol | string | "http" | — | no |
ibmcloud_cos_instance_name | string | "bnk-supply-chain" | — | no |
ibmcloud_resources_cos_bucket | string | "bnk-artifacts" | — | no |
ibmcloud_cos_bucket_region | string | "us-south" | — | no |
f5_cne_far_auth_file | string | "f5-far-auth-key.tgz" | — | no |
f5_cne_subscription_jwt_file | string | "subscription.jwt" | — | no |
scratch_dir | string | "/tmp/flp-vsi-scratch" | Working dir for the FAR-auth download/extract + chart pull. | no |
flp_prod_jwks_b64 | string | "" | Base64 of F5’s public prod_jwks.txt (JWT signature verification). Supplied by the FLP phase, which extracts it from the f5-license-proxy chart. | no |
roksbnkctl_binary | string | "" | Absolute path to the roksbnkctl binary; the FLP-VSI phase invokes roksbnkctl tfx <verb> in place of host curl/tar (no interpreter, so cmd.exe execs it on Windows). Empty falls back to roksbnkctl on PATH. | no |
use_cos_bucket | bool | true | True: pull the FAR tarball + subscription JWT from COS. False (disconnected): use far_service_account_b64 + f5_cne_subscription_jwt supplied by the root from local files. | no |
far_service_account_b64 | string | "" | Base64 FAR service account (from bnk.far_auth_local_file), used when use_cos_bucket=false. | no |
f5_cne_subscription_jwt | string | "" | Subscription JWT contents (from bnk.subscription_jwt_local_file), used when use_cos_bucket=false. | no |
flp_vsi_ssh_key | string | "" | Existing VPC SSH key name to attach to the FLP VSI (operator access). Empty = no key. | no |
flp_vsi_create_vpc | bool | false | Build the proxy its own VPC, address prefix and public gateway instead of placing it in one that already exists. Default false keeps existing workspaces byte-identical. | no |
flp_vsi_vpc_name | string | "" | Name for the VPC created when flp_vsi_create_vpc = true. Empty uses flp-vsi-vpc. | no |
flp_vsi_subnet_cidr | string | "10.250.0.0/24" | Address prefix for the VPC created when flp_vsi_create_vpc = true. Must not overlap anything the consuming clusters can already route to. | no |
ibmcloud_cos_resource_group | string | "" | Resource group holding the supply-chain COS instance. Empty resolves the workspace’s own resource group (#295). | no |
Module: gateway
Source: terraform/modules/gateway/variables.tf
| Variable | Type | Default | Description | Sensitive |
|---|---|---|---|---|
bnk_line | string | "2.3" | BNK release line driving per-release resource gating (“2.3” or “2.4”). | no |
ibmcloud_api_key | string | required | IBM Cloud API key | yes |
ibmcloud_cluster_region | string | required | IBM Cloud region the cluster runs in (zone names derive as <region>-1/2/3) | no |
roks_cluster_name_or_id | string | required | Existing ROKS cluster name or id the Gateway phase configures | no |
kubeconfig_dir | string | "" | Directory the IBM provider writes the cluster kubeconfig into | no |
create_roks_cluster | bool | false | Always false for the Gateway phase (it reuses an existing cluster) | no |
roks_cluster_dependency_id | string | null | Cluster-ready handle (apply-time ordering); unused gate kept for symmetry | no |
deploy_gateway | bool | false | Master toggle — when false the whole Gateway phase is a no-op (count=0) | no |
flo_namespace | string | "f5-bnk" | BNK namespace (F5BnkGateway / SnatPool / Egress / StaticRoute live here) | no |
app_namespace | string | "f5-app" | Application namespace the Gateway + HTTPRoute serve (created by this module) | no |
cneinstance_network_zones | list(object({ | [ { ext_vlan_cidr = "10.155.15.0/24" int_vlan_cidr = "10.254.99.0/24" int_snat_cidr = "10.10.11.0/24" int_vip_cidr = "10.135.15.0/24" external_selfip = "10.155.15.101" internal_selfip = "10.254.99.101" }, { ext_vlan_cidr = "10.156.16.0/24" int_vlan_cidr = "10.254.100.0/24" int_snat_cidr = "10.10.21.0/24" int_vip_cidr = "10.136.16.0/24" external_selfip = "10.156.16.101" internal_selfip = "10.254.100.101" }, { ext_vlan_cidr = "10.157.17.0/24" int_vlan_cidr = "10.254.101.0/24" int_snat_cidr = "10.10.31.0/24" int_vip_cidr = "10.137.17.0/24" external_selfip = "10.157.17.101" internal_selfip = "10.254.101.101" }, ] | Per-zone subnet CIDRs (empty = install-guide defaults). Shared with the BNK phase. | no |
gateway_class_name | string | "gateway-class" | GatewayClass name | no |
gateway_controller_name | string | "" | GatewayClass controllerName. Empty (the default) DERIVES it from flo_namespace as “f5.com/<flo_namespace>-f5-cne-controller”, which is the only value the CNE controller answers to — it must equal the CNEInstance’s own name (modules/cne_instance/…/main.tf builds that as “<flo_namespace>-f5-cne-controller”). This used to be the literal “f5.com/f5-bnk-f5-cne-controller”, i.e. the DEFAULT namespace baked in as a constant, so any workspace with a non-default bnk.flo_namespace got a GatewayClass no controller ever accepted and a Gateway that never programmed. Set explicitly only to point at a controller this module did not deploy. | no |
gateway_bnkgateway_name | string | "bnkgateway-cloud1" | F5BnkGateway name (referenced by the Gateway parametersRef) | no |
gateway_name | string | "http-gw" | Gateway (gateway.networking.k8s.io) name | no |
gateway_listener_port | number | 80 | Gateway HTTP listener port | no |
gateway_vip_start_host | number | 100 | Host number in int_vip_cidr for F5BnkGateway listener startAddress | no |
gateway_vip_end_host | number | 120 | Host number in int_vip_cidr for F5BnkGateway listener endAddress | no |
gateway_route_name | string | "http-route" | HTTPRoute name | no |
gateway_backend_service | string | "nginx-service" | HTTPRoute backend Service name in the app namespace | no |
gateway_backend_port | number | 80 | HTTPRoute backend Service port | no |
gateway_route_examples | list(string) | [] | Extra route kinds to create WORKING examples of, alongside the default HTTPRoute. Empty (the default) changes nothing about an existing deployment. | no |
gateway_l4_listener_port | number | 8080 | Port for the TCP listener added when gateway_route_examples includes L4Route | no |
gateway_egress_mode | string | "snatpool" | Egress SNAT strategy: snatpool (default; creates the SnatPool + snatpool Egress), automap (automap Egress only), or both. | no |
gateway_snatpool_name | string | "egress-snat-vx102" | F5SPKSnatpool name | no |
gateway_snat_host | number | 111 | Host number in int_snat_cidr for each zone’s SNAT address | no |
gateway_egress_app_interface | string | "eth0" | Application pod interface the Egress intercepts | no |
gateway_egress_tmm_interface | string | "external-vlan" | TMM interface name the Egress VXLAN binds to (matches the external VLAN) | no |
gateway_egress_node_interface | string | "ens3" | Node interface name for the Egress VXLAN | no |
gateway_egress_mtu | number | 1460 | Egress VXLAN MTU | no |
gateway_vxlan_port | number | 6789 | Egress VXLAN UDP port (also opened on the cluster security group) | no |
gateway_client_subnet_local | list(string) | [] | Local-VSI client subnet CIDRs the static routes reach (cluster-VPC clients; one F5SPKStaticRoute per entry × zone). Empty = no local client routes. | no |
gateway_client_subnet_remote | list(string) | [] | Remote-VSI client subnet CIDRs the static routes reach (client-VPC clients over the TGW; one F5SPKStaticRoute per entry × zone). Empty = no remote client routes. | no |
gateway_static_route_gw_host | number | 1 | Host number in ext_vlan_cidr used as each zone’s static-route gateway | no |
gateway_infra_name | string | "cloud-infra" | Name of the BNK 2.4 Infra CR (gateway.k8s.f5.com/v1alpha1) in the FLO namespace. | no |
gateway_settings_name | string | "gw-settings-common" | Name of the BNK 2.4 GatewaySettings CR in the FLO namespace. | no |
gateway_nad_name | string | "ens3-ipvlan-l2" | NetworkAttachmentDefinition the 2.4 Infra CR’s external VLAN attaches to. | no |
gateway_egress_vxlan_port | number | 6789 | egressDefaults.port on the 2.4 Infra CR. The CRD default is 4789; the install guide’s worked example uses 6789, which is what this defaults to so a stock install matches the vendor’s own manifest. | no |
gateway_egress_vxlan_subnet | string | "" | egressDefaults.subnet on the 2.4 Infra CR. Empty leaves the controller’s default of 10.0.0.0/16. | no |
Module: license
Source: terraform/modules/license/variables.tf
| Variable | Type | Default | Description | Sensitive |
|---|---|---|---|---|
flo_namespace | string | "f5-bnk" | Namespace the CNEInstance lives in, for the 2.4 post-licensing health check (#167). | no |
bnk_line | string | "2.3" | BNK release line (‘2.3’ or ‘2.4’) for per-release gating. See PRD 18 §4. | no |
ibmcloud_api_key | string | required | IBM Cloud API Key | yes |
ibmcloud_cluster_region | string | "ca-tor" | IBM Cloud region where the cluster resides | no |
ibmcloud_resource_group | string | "default" | IBM Cloud Resource Group name (leave empty to use account default) | no |
ibmcloud_cos_bucket_region | string | "us-south" | IBM Cloud region where the COS bucket is located | no |
ibmcloud_cos_instance_name | string | "bnk-supply-chain" | IBM Cloud COS instance name | no |
ibmcloud_resources_cos_bucket | string | "bnk-artifacts" | IBM Cloud COS bucket containing the FAR auth key and JWT files | no |
roks_cluster_name_or_id | string | required | Name or ID of the existing OpenShift ROKS cluster to deploy BNK onto | no |
flo_utils_namespace | string | "f5-utils" | Namespace for F5 utility components | no |
f5_cne_subscription_jwt_file | string | "subscription.jwt" | Subscription JWT filename in the COS bucket | no |
use_cos_bucket | bool | true | Download the subscription JWT from COS. false = use the injected jwt_token content (local file). | no |
jwt_token | string | "" | Subscription/license JWT content, injected when use_cos_bucket = false. Empty on the COS path. | yes |
license_mode | string | "connected" | License operation mode (connected, disconnected, or f5licenseproxy) | no |
flp_license_server_url | string | "" | Base URL of the in-cluster F5 License Proxy (FLP mode only) | no |
license_server_root_ca | string | "" | PEM of the FLP root CA, written to the licenseserver-rootca Secret (FLP mode only) | no |
create_roks_cluster | bool | false | When true, cluster is being created by roks_cluster — skip plan-time cluster credential fetch | no |
roks_cluster_dependency_id | string | null | roks_cluster sentinel ID — when set, defers runtime_config fetch to apply time after roks_cluster completes | no |
cneinstance_dependency_id | string | null | cneinstance_ready_id from ws4 — when set, ensures License CRD is available before applying License CR | no |
deploy_bnk | bool | true | Deploy BIG-IP Next for Kubernetes — when false the inner license module is disabled and no License resources are created | no |
kubeconfig_dir | string | "/work/.bnk/scratch/kubeconfig/license" | Persistent, writable dir for ibm_container_cluster_config kubeconfig downloads. Defaults to a host-bind-mounted, module-scoped path under .bnk/scratch. | no |
roksbnkctl_binary | string | "" | Absolute path to the roksbnkctl binary; the license phase invokes roksbnkctl tfx <verb> in place of host curl (no interpreter, so cmd.exe execs it on Windows). Empty falls back to roksbnkctl on PATH. | no |
cluster_absent | bool | false | True in the standalone FLP-VSI phase: no ROKS cluster exists, so all cluster data-source lookups + kube providers are skipped (count=0). | no |
ibmcloud_cos_resource_group | string | "" | Resource group holding the supply-chain COS instance. Empty resolves the workspace’s own resource group (#295). | no |
Module: roks_cluster
Source: terraform/modules/roks_cluster/variables.tf
| Variable | Type | Default | Description | Sensitive |
|---|---|---|---|---|
ibmcloud_api_key | string | required | IBM Cloud API key | yes |
ibmcloud_cluster_region | string | required | IBM Cloud region for all cluster resources | no |
ibmcloud_resource_group | string | "default" | IBM Cloud resource group name | no |
create_roks_cluster | bool | true | Create a new ROKS cluster. When false, supply roks_cluster_id_or_name instead. | no |
cluster_public_gateway | bool | true | Attach public gateways for worker Internet egress. true (default) = current behavior; false = private/disconnected cluster (no egress). | no |
roks_cluster_id_or_name | string | "" | ID or name of an existing ROKS cluster — used when create_roks_cluster = false | no |
create_roks_transit_gateway | bool | true | Create Transit Gateway and VPC connections | no |
create_roks_registry_cos_instance | bool | true | Create Cloud Object Storage instance for the OpenShift image registry | no |
roks_cluster_vpc_name | string | "tf-cluster-vpc" | Name of the cluster VPC | no |
openshift_cluster_name | string | "tf-openshift-cluster" | Name of the OpenShift cluster | no |
openshift_cluster_version | string | "4.20" | OpenShift cluster version as a BARE major.minor prefix (e.g. “4.20”). Empty uses the latest available. | no |
roks_workers_per_zone | number | 1 | Number of worker nodes per availability zone | no |
roks_min_worker_vcpu_count | number | 16 | Minimum vCPU count when auto-selecting the worker node flavor | no |
roks_min_worker_memory_gb | number | 64 | Minimum memory in GB when auto-selecting the worker node flavor | no |
roks_cos_instance_name | string | "tf-openshift-cos-instance" | Name of the COS instance for the OpenShift image registry | no |
roks_transit_gateway_name | string | "tf-tgw" | Name of the Transit Gateway | no |
use_existing_cluster_vpc | bool | false | Reuse an existing cluster VPC instead of creating one (forwarded to module.cluster). | no |
existing_cluster_vpc_id | string | "" | ID of the existing cluster VPC (used only when use_existing_cluster_vpc = true; forwarded to module.cluster). | no |
kubeconfig_dir | string | required | Directory where ibm_container_cluster_config writes the admin kubeconfig. Must be writable; set explicitly to avoid the provider’s HOME-derived default, which resolves empty under the roksbnkctl runner. | no |
roksbnkctl_binary | string | "" | Absolute path to the roksbnkctl binary; the cluster phase invokes roksbnkctl tfx <verb> in place of host curl/kubectl (no interpreter, so cmd.exe execs it on Windows). roksbnkctl sets this via TF_VAR_roksbnkctl_binary; empty falls back to roksbnkctl on PATH. | no |
cluster_absent | bool | false | True only in the standalone FLP-VSI phase: no ROKS cluster exists or will be adopted, so all cluster data-source lookups + kube providers across modules are skipped (count=0). | no |
cluster_vpc_cidr | string | "" | CIDR block the cluster VPC’s per-zone address prefixes are carved from (e.g. “10.241.0.0/16” → 10.241.0.0/18, 10.241.64.0/18, 10.241.128.0/18). | no |
use_existing_cluster_subnets | bool | false | Place the cluster in subnets that already exist instead of creating them. Requires use_existing_cluster_vpc — a subnet cannot be adopted independently of its VPC. | no |
existing_cluster_subnet_ids | list(string) | [] | Subnet ids to place the cluster in, one per zone, in zone order. Used only when use_existing_cluster_subnets = true. Their zones are read from the subnets themselves. | no |
cluster_http_allowed_cidrs | list(string) | [] | Source CIDRs allowed to reach :80 on the cluster security group. Empty → 0.0.0.0/0. | no |
cluster_vpc_default_sg_inbound_cidrs | list(string) | [] | Source CIDRs allowed inbound (all protocols/ports) to the cluster VPC’s default security group. Empty → 0.0.0.0/0. | no |
roks_worker_flavor | string | "" | Exact worker-node flavor, e.g. “cx3d.8x20”. Empty auto-selects. | no |
bnk_line | string | "2.3" | BNK release line (2.3 / 2.4), used to pick a VALIDATED default worker flavor per line. | no |
Module: testing
Source: terraform/modules/testing/variables.tf
| Variable | Type | Default | Description | Sensitive |
|---|---|---|---|---|
ibmcloud_api_key | string | required | IBM Cloud API Key | yes |
ibmcloud_cluster_region | string | "ca-tor" | IBM Cloud region where the referenced cluster resides | no |
ibmcloud_resource_group | string | "" | IBM Cloud Resource Group name (leave empty to use account default) | no |
roks_cluster_name_or_id | string | required | Name or ID of the existing OpenShift ROKS cluster | no |
testing_create_tgw_jumphost | bool | true | Create a jumphost in a client VPC and (optionally) connect it to the cluster via a Transit Gateway | no |
testing_create_cluster_jumphosts | bool | false | Create one jumphost per availability zone directly inside the cluster VPC | no |
testing_ssh_key_name | string | "" | Name of the SSH key to inject into all jumphosts. Must exist in client_vpc_region (for TGW jumphost) and in ibmcloud_cluster_region (for cluster jumphosts) | no |
testing_jumphost_profile | string | "" | Instance profile for all jumphosts (leave empty to auto-select from min_vcpu_count and min_memory_gb) | no |
testing_jumphost_total_volume_bandwidth | number | 1000 | Volume bandwidth (Mbps) pinned on each jumphost. IBM computes this from the profile at creation and terraform carries the stored value into a later profile change, so an unmanaged value taken from an oversized profile makes a downsize impossible (#316). 1000 is ample for jumphosts, which run curl/iperf3 and no disk-heavy work, and is below the ceiling of every profile the auto-select can pick. | no |
testing_min_vcpu_count | number | 4 | Minimum vCPU count when auto-selecting the instance profile | no |
testing_min_memory_gb | number | 8 | Minimum memory in GB when auto-selecting the instance profile | no |
testing_create_client_vpc | bool | false | Create a new client VPC for the TGW jumphost. When false, client_vpc_name must reference an existing VPC | no |
testing_client_vpc_name | string | "tf-testing-vpc" | Name of the client VPC — created when create_client_vpc = true, or looked up when create_client_vpc = false | no |
testing_client_vpc_region | string | "ca-tor" | IBM Cloud region for the client VPC and TGW jumphost | no |
testing_transit_gateway_name | string | "" | Name of an existing Transit Gateway to connect the client VPC to (leave empty to skip TGW attachment) | no |
testing_tgw_jumphost_name | string | "tf-testing-jumphost-tgw" | Name of the TGW-connected jumphost instance (used as prefix for subnet, gateway, security group, and floating IP) | no |
testing_cluster_jumphost_name_prefix | string | "tf-testing-jumphost-cluster" | Name prefix for cluster jumphosts — zone name is appended (<prefix>-<zone>) | no |
roks_cluster_dependency_id | string | null | roks_cluster sentinel ID — when set, defers cluster/TGW data source reads to apply time after roks_cluster completes | no |
create_roks_cluster | bool | false | Set to true when the ROKS cluster is being created in this run — skips cluster-VPC-derived data sources that require a pre-existing cluster | no |
cluster_vpc_id | string | "" | ID of the cluster VPC — pass module.roks_cluster.roks_cluster_vpc_id directly; avoids deriving via worker-pool subnet chain which is deferred to apply time | no |
cluster_absent | bool | false | True only in the standalone FLP-VSI phase: no ROKS cluster exists, so cluster data-source lookups are skipped (count=0). | no |
testing_jumphost_allowed_cidrs | list(string) | [] | Source CIDRs allowed to SSH (:22) to the testing jumphosts. Empty → 0.0.0.0/0 (open — the jumphosts carry a public floating IP and are reached from wherever the operator happens to be; access is key-only). Narrow this to the operator’s public /32 on a shared account. | no |
testing_client_vpc_inbound_cidrs | list(string) | [] | Source CIDRs allowed inbound to the testing client VPC’s DEFAULT security group. Empty → the RFC-1918 private ranges, which is where in-fabric test traffic arrives from (the cluster reaches the client VPC over the Transit Gateway). Set explicitly only if a test needs a public source. | no |
Module: tgw_connection
Source: terraform/modules/tgw_connection/variables.tf
| Variable | Type | Default | Description | Sensitive |
|---|---|---|---|---|
deploy_tgw_connection | bool | false | Attach the cluster’s VPC to an existing Transit Gateway. Off in every other phase’s override; on only for tgw connect. When false the module is a complete no-op. | no |
ibmcloud_api_key | string | "" | IBM Cloud API key (IAM auth + provider config). | yes |
ibmcloud_cluster_region | string | "" | Region of the cluster VPC (ibm provider + the ibm_is_vpc lookup). | no |
cluster_vpc_id | string | "" | ID of the cluster’s VPC — the network attached to the Transit Gateway. Resolved to a CRN here; the connection needs the CRN, but cluster-outputs.json records the id. | no |
transit_gateway | string | "" | The EXISTING Transit Gateway to attach to, by NAME or by ID. Resolved against the account’s gateway list, so either works. Multiple clusters passing the same value share one gateway; each workspace owns its own connection. | no |
connection_name | string | "" | Name for THIS cluster’s connection on the gateway. Must be unique per gateway, so it is prefix-derived — two clusters sharing one Transit Gateway get distinct connection names. | no |
Glossary
Plain-English definitions of the terms used across the book. Project-specific concepts, IBM-Cloud-specific products, OpenShift / Kubernetes admission concepts, and the F5 BIG-IP Next networking vocabulary all live here. Entries are deliberately one or two sentences; the deep-dive lives in the linked chapter where applicable.
A — D
api_key_b64
Base64-encoded IBM Cloud API key stored inline in the workspace config.yaml. Obfuscation, not encryption — anyone with file-read access decodes instantly. The field name deliberately doesn’t match the plaintext-secret rejection regex. See Chapter 14 §“Source 3”.
Backend (--backend)
The execution context for a tool dispatch. One of local (os/exec on the host), docker (containerised), k8s (in-cluster ops pod or Job), or ssh:<target> (a registered SSH endpoint). See Chapter 17.
BNK BIG-IP Next for Kubernetes. F5’s Kubernetes-native CNF deployment of BIG-IP, made up of FLO (F5 Lifecycle Operator) + CNE Instance + License + CIS. The reason this CLI exists. See Chapter 1.
CIS Two unrelated CISes appear in this stack. Inside the cluster, CIS is F5’s Container Ingress Services — the F5 controller that watches Kubernetes Ingress + Route resources and programs the BIG-IP data plane. At the IBM Cloud account level, CIS is Cloud Internet Services — IBM’s DNS, CDN, WAF, and DDoS-protection product. Context disambiguates; when in doubt, “F5 CIS” vs “IBM Cloud CIS”.
ClusterIP (k8s)
A Service type that gives a Service an internal cluster IP, reachable only from inside the cluster. Used by the throughput suite’s east-west mode.
ClusterRole (k8s)
A cluster-scoped RBAC role granting verbs on resources. The ops pod’s least-privilege ClusterRole grants jobs.create in roksbnkctl-test but not pods.delete in default.
CNE Instance
A Custom Resource defined by FLO. Represents one deployed instance of the BNK data plane (TMM pods + control plane). Sizing is Small/Medium/Large. See Chapter 10.
Cobra
github.com/spf13/cobra — the Go CLI library roksbnkctl is built on. The command tree at internal/cli/ is a cobra command tree.
COS IBM Cloud Object Storage — S3-compatible object store. The BNK supply chain bucket lives on a COS instance. See Chapter 25.
cred resolver chain
The ordered lookup for IBMCLOUD_API_KEY: env var → OS keychain → workspace api_key_b64 → interactive prompt. The chain stops at the first source that yields a non-empty value. See Chapter 14.
CRN
Cloud Resource Name — IBM’s globally-unique resource identifier. Starts with crn:v1: and encodes account, region, service, and resource ID. Most roksbnkctl cos commands accept either a friendly name (resolved at runtime) or a CRN.
E — J
east-west
Network direction term: traffic between two endpoints inside the cluster (pod-to-pod, service-to-service). The throughput suite’s --mode east-west measures CNI fabric throughput. See Chapter 22.
Embedded HCL
The Terraform source tree compiled into the roksbnkctl binary via Go’s //go:embed directive. The default tf_source is embedded. Rebuilding the binary picks up HCL changes. See Chapter 31 §“The embedded HCL”.
envFrom (k8s)
A Pod spec field that references a Secret or ConfigMap and projects all of its keys as environment variables into the container. The k8s backend’s ops pod uses envFrom: secretRef: roksbnkctl-ibm-creds to receive the API key without listing it in the manifest plaintext.
extra_hosts
The workspace config’s list of additional URLs to probe under roksbnkctl test connectivity. The value is a bare []string of URLs; per-host method/expected-status overrides are deferred. See Chapter 20.
FAR
F5 Application Runtime — the container-image distribution of the BIG-IP Next data plane. FLO pulls FAR images from repo.f5.com using the auth key in the COS supply-chain bucket.
FLO F5 Lifecycle Operator — the Kubernetes operator that owns the CNE Instance + License + supporting resources. The control plane piece of BNK. (The acronym sometimes also surfaces as “F5 Logging Operator” — context disambiguates; in this book it always means Lifecycle Operator.)
FQDN
Fully Qualified Domain Name — the absolute form of a DNS name ending with a trailing dot (www.example.com.).
FAR auth key
The credential tarball (f5-far-auth-key.tgz) that FLO uses to pull FAR images from repo.f5.com. Lives in the COS supply-chain bucket. Rotated periodically; see Chapter 25 §“Licence rotation”.
ghcr.io
GitHub Container Registry — where the roksbnkctl-tools-* images are published. The k8s backend pulls from ghcr.io/jgruberf5/roksbnkctl-tools-{ibmcloud,iperf3}.
GSLB Global Server Load Balancing — DNS-driven traffic management where the answer a name returns depends on the requesting resolver’s network vantage. The thing Chapter 21 is built to validate.
--gslb-compare
The DNS-probe flag that fans out across all configured backends in parallel and emits a comparison JSON with gslb_divergence: true|false. The signature workflow for “is the GSLB rule taking effect”. See Chapter 21 §“The –gslb-compare workflow”.
HCL
HashiCorp Configuration Language — the syntax of Terraform .tf files. The upstream HCL is bundled into the binary; see Embedded HCL.
ibmcloud
Two senses. The IBM Cloud CLI binary (which roksbnkctl ibmcloud … passes through to or replaces, depending on the backend). And the YAML block in config.yaml (ibmcloud:) holding region, resource group, and API key source.
ImagePullBackOff (k8s)
A Pod status indicating the image couldn’t be pulled from the registry. Usually a network or auth problem; sometimes a tag-doesn’t-exist problem. See Chapter 26 §“ImagePullBackOff…”.
iperf3 mode (--mode)
The throughput-suite flag selecting north-south (LoadBalancer Service, client outside the cluster) or east-west (ClusterIP Service, client inside the cluster). See Chapter 22 §“The two modes”.
JWT
JSON Web Token — the signed-token format BNK uses for the subscription licence (subscription.jwt in the COS supply-chain bucket).
K — N
k (roksbnkctl k <verb>)
The internalised kubectl subtree. roksbnkctl k get/apply/describe/delete/exec/logs/port-forward — built on k8s.io/client-go directly so no host kubectl binary is required. See Chapter 24.
kubeconfig
The Kubernetes client-configuration file (clusters, contexts, credentials). Defaults to ~/.kube/config. roksbnkctl up auto-fetches the admin kubeconfig post-apply.
LoadBalancer (k8s Service type)
A Service type that provisions an external endpoint (a cloud LB on managed Kubernetes; an external IP on bare-metal CNI). Used by the throughput suite’s north-south mode and by BNK’s exposed VIPs.
Long-lived ops pod
The k8s backend’s persistent execution context. Deployed by roksbnkctl ops install; subsequent --backend k8s dispatches kubectl exec into the same pod rather than starting a fresh Pod each call. Contrasted with the one-shot Job pattern used for iperf3 and DNS probes. See Chapter 19.
BNK line / release line — The <major>.<minor> of a BNK release, e.g. 2.3. Not configured: derived from bnk.manifest_version. Selects the terraform overlay (Chapter 13) and gates the support-matrix check. An unset manifest_version derives the line from the built-in default rather than failing, because that field has always been optional.
Network mode (single-nic | multi-nic) — How a cluster’s worker nodes are attached. Fixed when the cluster is created and never converted in place. single-nic is the default and describes every cluster built with this tool today. multi-nic needs both IBM support that has not shipped and a BNK line that expresses it — and no shipped line does: 2.4 was expected to and, on the evidence of its EA install guide, does not.
Contract schema version — The schema_version in cluster-outputs.json, currently 2. Absent means 1. Declares the shape of the handoff record so a reader can tell what it is looking at.
Support matrix — internal/config/support_matrix.yaml: which BNK lines drive which network modes, and which contract versions each can read. Checked before planning. A known line with an unsupported mode is refused; a line the matrix has never heard of warns and proceeds, since that usually means the binary predates the release rather than that the pairing is wrong.
Manifest version (f5_bigip_k8s_manifest_version)
The version pin on the f5-bigip-k8s-manifest Helm chart. Transitively pins both the FLO and CIS versions (both are extracted from the manifest chart). See Chapter 13. It also determines the derived BNK line above, so it selects the terraform overlay and gates the support-matrix check — a larger blast radius than the name suggests, and the reason there is no separate line field that could disagree with it.
mdBook
rust-lang/mdBook — the static-site generator the book is built with. Markdown source under book/src/, HTML output under book/book/. See Chapter 31 §“The book build”.
miekg/dns
github.com/miekg/dns — the Go DNS library the GSLB probe is built on. Same library CoreDNS uses; gives roksbnkctl test dns full record-type coverage and per-query server selection. See Chapter 21 §“The roksbnkctl test dns flag surface”.
north-south
Network direction term: traffic crossing the cluster boundary — from outside the cluster to a pod inside, or vice versa. The throughput suite’s --mode north-south measures inbound LoadBalancer-path throughput. See Chapter 22.
NXDOMAIN
DNS response code indicating “this name does not exist”. roksbnkctl test dns against a non-existent name exits 1 with rcode=NXDOMAIN.
O — R
--on <target>
The persistent CLI flag dispatching an ibmcloud/exec/shell/kubectl/oc passthrough over SSH to a named target instead of running it locally. The other half of the SSH-client + --on feature alongside the SSH backend. See Chapter 16.
OpenShift Red Hat’s enterprise Kubernetes distribution. ROKS = managed OpenShift on IBM Cloud.
Ops pod
Shorthand for the long-lived k8s-backend execution pod deployed in the roksbnkctl-ops namespace by roksbnkctl ops install. See Chapter 19.
passthrough
A command that proxies its argv to an underlying tool. roksbnkctl ibmcloud … passes through to the ibmcloud CLI; roksbnkctl kubectl … passes through to kubectl. Passthroughs run on whatever backend is selected (local by default).
PRD
Product Requirements Document. The project uses numbered PRDs under docs/prd/ to coordinate larger feature work. See Chapter 32 §“The PRD process”.
PHASE_FROM=
The env-var resume mechanism on the e2e driver scripts. PHASE_FROM=L ./scripts/e2e-test-backends.sh fast-forwards past phases A-K. See Chapter 23 §“Resuming a partial run”.
RBAC Role-Based Access Control — the Kubernetes authorization model. The ops pod has a least-privilege RBAC binding; see ClusterRole.
restricted-v2
The default OpenShift PodSecurity policy / SCC at admission. Rejects pods that run as root, allow privilege escalation, or hold the ALL capability set. All roksbnkctl-managed pods (ops pod, iperf3 server, DNS probe Job) are written to satisfy restricted-v2. See Chapter 22 §“The bundled image and the runAsNonRoot constraint”.
redactor
The output-stream wrapper at internal/exec/redact.go that masks the IBM API key value in any subprocess’s stdout/stderr before it reaches the user’s terminal or the log. The defence-in-depth net for credential leaks. See Chapter 14 §“The redactor”.
ROKS
Red Hat OpenShift on IBM Cloud — IBM’s managed OpenShift offering. The cluster roksbnkctl up provisions. See Chapter 2.
runAsNonRoot
A Pod / container securityContext field. Required true by restricted-v2. Images that have USER root in the Dockerfile fail admission with this set.
RTT
Round-Trip Time — measured in milliseconds for each DNS query. roksbnkctl test dns -o json surfaces p50/p95/p99 across the run.
S — Z
Schematic JSON The deployer-rendered JSON document describing a BNK deployment. Lives in the COS supply-chain bucket; not consumed at install time, kept for forensics.
SCC
Security Context Constraint — OpenShift’s pod-admission policy. restricted-v2 is the default; pods that violate it (e.g., by running as root) are rejected by the admission controller. See Chapter 22.
Secret (k8s)
A namespaced resource holding key/value data, typically base64-encoded credentials. The k8s backend creates roksbnkctl-ibm-creds in the roksbnkctl-ops namespace at ops install time.
secretRef (k8s)
The Pod spec form that references a Secret for environment-variable projection. Used together with envFrom for the ops pod’s credential injection.
Service (k8s sense)
A Kubernetes resource that provides a stable endpoint for accessing one or more Pods. Types: ClusterIP (default), NodePort, LoadBalancer, ExternalName. See ClusterIP, LoadBalancer.
SPDY
Speedy (protocol). The websocket-like, multiplexed-stream protocol Kubernetes uses for exec and port-forward. roksbnkctl k exec is a SPDY client implementation on top of k8s.io/client-go’s SPDY executor.
SSH backend
The --backend ssh:<target> execution path. Runs the tool on a registered SSH endpoint via the internal/remote.Client wrapper. See Chapter 17 §“SSH backend”.
TGW Transit Gateway — IBM Cloud’s VPC-to-VPC connectivity service. The upstream HCL provisions a TGW between the cluster VPC and the testing-client VPC so the jumphost can reach the cluster’s internal endpoints.
tfvars (terraform.tfvars)
Variable-value file for Terraform — assigns concrete values to the HCL’s variable blocks. roksbnkctl auto-renders one from config.yaml; user overrides layer on top via terraform.tfvars.user and --var-file. See Chapter 13.
tf_source
The workspace config.yaml block selecting where the Terraform source comes from: embedded (compiled into the binary; the default), github (downloaded tarball), local (an on-disk directory). See Chapter 12 §“tf_source:”.
TLS (--insecure)
Transport Layer Security. The --insecure flag on roksbnkctl test connectivity skips TLS certificate validation for every probe in the run (session-wide, not per-host).
TMM Traffic Management Microkernel — the BIG-IP data-plane process. BNK runs TMM as a Pod; the CNE Instance specifies how many and at what size.
TOFU
Trust On First Use — the SSH-style host-key acceptance pattern. On first connection to a new SSH target, roksbnkctl prompts the user to verify the fingerprint; subsequent connections check against the saved fingerprint in ~/.roksbnkctl/known_hosts. A fingerprint mismatch refuses to connect. See Chapter 16 §“Host-key handling”.
Trusted Profile An IBM IAM construct that lets a Kubernetes ServiceAccount assume IBM Cloud permissions. FLO uses one to authenticate against IBM Cloud APIs without storing an API key in the cluster.
TTL
Time To Live — DNS-record cache duration in seconds. roksbnkctl test dns -o json surfaces each answer’s TTL.
v1.0 The release this book is the launch deliverable for. All E2E phases pass on a clean dev box; doctor green-by-default with terraform-only required.
VPE Virtual Private Endpoint — IBM Cloud’s private-network access point for managed services. Sometimes left dangling after a cluster destroy (see Chapter 26 §“orphan IBM Cloud resources”).
VPC Virtual Private Cloud — IBM Cloud’s network-isolation primitive. The cluster lives in one VPC; the testing client jumphost lives in another, connected via TGW.
VSI Virtual Server Instance — IBM Cloud’s general-purpose VM. The jumphosts are VSIs.
workspace
A named slot under ~/.roksbnkctl/<name>/ containing one config.yaml, one Terraform state directory, and (usually) one kubeconfig. The kubectl-style multi-environment isolation primitive. See Chapter 6.
ws / workspace
The CLI subtree managing workspaces. roksbnkctl ws new/use/list/delete.
Cross-references
- Chapter 1 — BNK context.
- Chapter 2 — ROKS context.
- Chapter 14 — credentials terminology.
- Chapter 17 — backend terminology.
- Chapter 21 — DNS / GSLB terminology.
- Chapter 22 — throughput / SCC terminology.
Building from source
This chapter is for contributors and operators who want to build roksbnkctl themselves — whether to test an unreleased change, to verify a release artefact, or to embed a custom HCL fork into the binary.
For users who just want to install the binary, Chapter 4 — Installation is the right page. This chapter is the build-side companion.
Go version requirement
The minimum Go version is the one pinned in go.mod:
go 1.25.0
We pin to a recent toolchain for two reasons:
- The IBM Cloud Go SDKs (
go-sdk-core/v5,platform-services-go-sdk,ibm-cos-sdk-go) andk8s.io/client-gov0.30+ both make liberal use of Go’s modern generics — pre-1.21 toolchains won’t build. - Several dependencies (
miekg/dns,docker/docker) test on the current and previous minor only; we follow upstream.
Install Go via your package manager (brew install go, apt-get install golang-1.25, etc.) or from go.dev/dl. Verify with go version.
Quick build
The shortest path from a fresh clone to a working binary:
git clone https://github.com/jgruberf5/roksbnkctl.git
cd roksbnkctl
go build -o roksbnkctl ./cmd/roksbnkctl
./roksbnkctl --version
go build produces a static binary in the working directory. Cross-compilation to a different OS/arch needs GOOS / GOARCH set:
GOOS=linux GOARCH=amd64 go build -o roksbnkctl-linux-amd64 ./cmd/roksbnkctl
GOOS=darwin GOARCH=arm64 go build -o roksbnkctl-darwin-arm64 ./cmd/roksbnkctl
GOOS=windows GOARCH=amd64 go build -o roksbnkctl.exe ./cmd/roksbnkctl
A full multi-platform build is easier through goreleaser:
goreleaser release --snapshot --clean
# Output lands in dist/
The --snapshot --clean flags produce a local build without trying to publish to GitHub. The release shape is described in .goreleaser.yml — Linux + macOS × amd64 + arm64, plus a Windows compile-only check.
Build via the Makefile
The repo’s Makefile wraps the common build steps and stamps version metadata into the binary:
make build # builds bin/roksbnkctl with -ldflags version stamping
make test # go test ./...
make vet # go vet ./...
make tidy # go mod tidy
make test-short # go test -short ./...
make test-integration # testcontainers-go-backed integration tests (needs Docker)
make test-cred-audit # the security-spine regression suite
make lint # gofmt + vet + staticcheck (if installed)
The version stamp comes from three ldflags variables baked into internal/cli:
var (
Version = "dev"
Commit = "none"
BuildDate = "unknown"
)
make build passes -X github.com/jgruberf5/roksbnkctl/internal/cli.Version=$VERSION (and the others) so roksbnkctl --version reports the actual git-rev and build timestamp rather than the placeholders. Set VERSION explicitly when stamping a release:
VERSION=v1.0.0 make build
The embedded HCL
The Terraform source tree at terraform/ is compiled into the binary via Go’s //go:embed directive. The embed declaration lives at the repo root in embedded.go (and is wired through internal/tf/ to be served as the tf_source: embedded provider).
Two implications:
- Rebuilding the binary picks up HCL changes. If you’re hacking on the HCL,
make buildproduces a binary that ships your changes embedded. No separate “deploy the HCL” step. - The HCL is read-only at runtime. The binary extracts it to a temporary directory on first use; the extracted copy is what terraform operates on. The original embedded source is immutable.
For users who want to not use the embedded HCL, the tf_source: github or tf_source: local options in the workspace config bypass it entirely. See Chapter 12 §“tf_source:”.
The bundled tools images
The tools/docker/ directory holds Dockerfiles for the images the docker and k8s backends use, plus the release-time book builder:
tools/docker/
├── Makefile
├── ibmcloud/
│ └── Dockerfile # roksbnkctl-tools-ibmcloud
├── iperf3/
│ └── Dockerfile # roksbnkctl-tools-iperf3
└── mdbook/
└── Dockerfile # roksbnkctl-tools-mdbook (release-time book builder)
tools/docker/Makefile builds the three images locally as :dev:
cd tools/docker
make build-ibmcloud # builds roksbnkctl-tools-ibmcloud:dev
make build-iperf3 # builds roksbnkctl-tools-iperf3:dev
make build-mdbook # builds roksbnkctl-tools-mdbook:dev
make build-all # all three
The :dev tag is what a from-source roksbnkctl resolves to when the binary’s Version is dev. A tag-released binary (v1.0.0) resolves to ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:v1.0.0 instead — the resolver logic lives in internal/exec/ (SetToolImageTag is wired in internal/cli/root.go::init). See Chapter 17 §“:dev tag resolution”.
The GitHub Actions workflow tools-images.yml builds and pushes the published images via a strategy.matrix.image over [ibmcloud, iperf3, mdbook]. Every push to main republishes the :dev tag for each image; every v* tag push publishes :<tagname> + :latest. workflow_dispatch is also wired for manual runs. The mdbook image is part of the matrix, so edits to tools/docker/mdbook/Dockerfile are republished automatically: the next push to main republishes ghcr.io/jgruberf5/roksbnkctl-tools-mdbook:dev with no manual make -C tools/docker build-mdbook + docker push step on the release-cut host.
The book build
The book is built with mdBook. Install:
cargo install mdbook
# or
brew install mdbook
The book source lives under book/src/ with book.toml as the config. Common operations:
make book-serve # mdbook serve book/ --open
# opens http://localhost:3000 with live-reload
make book # mdbook build book/
# static HTML at book/book/
make book-clean # rm -rf book/book
The published site at https://jgruberf5.github.io/roksbnkctl/book/ is built and deployed by .github/workflows/book.yml on every push to main. The workflow runs mdbook build book/ and pushes the output to the gh-pages branch via peaceiris/actions-gh-pages.
For PR-time verification, .github/workflows/spellcheck.yml runs cspell on book/src/**/*.md — a warning, not a gate, but worth eyeballing the output before merging.
The auto-generated chapters
Two reference chapters are generated rather than hand-written. The generators live under tools/refgen/:
# Chapter 27 — command reference (walks the cobra command tree)
go run ./tools/refgen/cobra-md > book/src/27-command-reference.md
# Chapter 29 — terraform variable reference (parses terraform/variables.tf)
go run ./tools/refgen/tfvars-md > book/src/29-terraform-variable-reference.md
When to re-run:
- Chapter 27: any change to the cobra command tree under
internal/cli/orcmd/roksbnkctl/— new commands, renamed flags, editedLong:/Example:strings. - Chapter 29: any change to
terraform/variables.tfor any submodulevariables.tfreferenced from the root — new variables, default-value changes, edited descriptions.
Both generators emit deterministic output — the same input HCL or cobra tree always produces the same markdown — so you can commit the rendered output to source control without worrying about spurious diff churn.
Cross-compile matrix
goreleaser covers the canonical matrix:
| OS | Architecture | Status |
|---|---|---|
| Linux | amd64 | Fully supported |
| Linux | arm64 | Fully supported |
| macOS | amd64 (Intel) | Fully supported |
| macOS | arm64 (Apple Silicon) | Fully supported |
| Windows | amd64 | Compile-only; SSH TTY support degraded |
| Windows | arm64 | Compile-only; same caveat |
| FreeBSD | amd64 | Not tested |
The Windows caveat is real: golang.org/x/crypto/ssh’s PTY allocation isn’t complete on Windows, so roksbnkctl shell --on jumphost falls back to a non-TTY shell. The other commands (exec, ibmcloud, kubectl) work fine on Windows.
Output from goreleaser release --snapshot --clean lands in dist/:
dist/
├── roksbnkctl_linux_amd64_v1/
│ └── roksbnkctl
├── roksbnkctl_linux_arm64/
│ └── roksbnkctl
├── roksbnkctl_darwin_amd64_v1/
│ └── roksbnkctl
├── roksbnkctl_darwin_arm64/
│ └── roksbnkctl
└── ...
Each archive bundles the binary plus LICENSE, README.md, and the rendered book/book/ directory (when the snapshot is built from a tagged commit).
Release process
Tagged releases are cut on the main branch:
# Update CHANGELOG.md with the release notes
git tag -a v1.0.0 -m "v1.0.0 — book launch + full E2E coverage"
git push origin v1.0.0
The push triggers release.yml, which runs goreleaser release to:
- Cross-compile the binary for the supported OS/arch matrix.
- Build the matching tools images and push to
ghcr.io/jgruberf5/roksbnkctl-tools-*:<tag>. - Attach the binaries, checksums (
checksums.txt), and the rendered book PDF (if mdbook-pdf is configured) to the GitHub release. - Generate release notes from the CHANGELOG and the commits since the previous tag.
The release-gate criteria — what has to hold before tagging — are documented in PLAN.md §“v1.0 (M4)”. The most important: full E2E green for 3 consecutive nights on main.
Cross-references
- Chapter 4 — Installation — for users who just want the binary, not the source.
- Chapter 17 §“:dev tag resolution” — how a from-source binary picks tool images.
- Chapter 32 — Extending roksbnkctl — once you’ve built, what to actually hack on.
docs/PLAN.md— the release-gate policy.
Extending roksbnkctl
This chapter is the hacking guide for contributors. It covers the four most common extension shapes — adding a new execution backend, a new test suite, a new tool to an existing backend, a new chapter to the book — plus the PRD process the project uses to coordinate larger changes and the four-agent sprint-dispatch pattern Sprints 0-6 ran on.
For building the binary, see Chapter 31 — Building from source. For using the binary, see the rest of the book.
Adding a new execution backend
A backend is anything implementing the Backend interface in internal/exec/backend.go. The four backends shipped at v1.0 — local, docker, k8s, ssh:<target> — are each a single file under that package.
The end-to-end shape:
-
Implement the interface. Create
internal/exec/<your-backend>.go. The contract isRun(ctx context.Context, argv []string, opts RunOpts) (int, error). Honouropts.Stdin/Stdout/Stderr,opts.WorkDir,opts.Env,opts.Credentials,opts.HostMounts, andopts.RunAsUser. Return the subprocess exit code as the first return; second is for backend-side errors (couldn’t start, ctx cancelled, etc.). -
Register it. Call
exec.Register(name string, b Backend)from the package’sinit()block. TheResolveBackend(spec string)function ininternal/exec/backend.godispatches--backend <name>to the registered backend. -
Handle credentials safely. Read PRD 04 before touching
opts.Credentials. The cardinal rule: never pass credential values via argv — they end up inpsoutput, container metadata, and process accounting. Pass by reference (env var by name, projected Secret, SSHSetEnv) and let the runtime do the value plumbing. -
Wire the redactor. Wrap
opts.Stdoutandopts.Stderrwithinternal/exec.NewRedactorbefore handing them to the subprocess. The redactor masks any credential value that leaks into the tool’s stdout/stderr. Thelocalanddockerbackends do this in their wrappers; copy the pattern. -
Add a doctor check. Doctor’s per-backend availability check needs to recognise your backend. Add an entry under
internal/cli/doctor_backend.goreporting whether your backend’s prerequisites are satisfied (e.g., “is the daemon running”, “is the SDK reachable”). Green-by-default on a stock dev box is the goal — yellow-skip rather than red-fail when prerequisites are missing. -
Add per-backend cred-audit assertions. The cred-leak audit at
internal/exec/audit_test.go(and Phase M of the e2e plan) needs to know what surfaces your backend produces — container inspection, process listing, log files. Add aTestCredAudit_<YourBackend>subtest asserting the API key value never appears in any of them. -
E2E phase. Add a new phase to
scripts/e2e-test-backends.shwith concrete pass/fail criteria. Cross-link from PRD 05 so the test plan stays the source of truth. -
Documentation. Add a deep-dive subsection to Chapter 17 — Execution backends and a decision-tree entry to Chapter 18 — Choosing a backend per tool. Without docs the backend doesn’t exist for users.
A backend PR that lands all eight steps is a complete contribution; one that lands the code but skips the audit and docs will get a “please come back with…” review comment.
Adding a new test suite
The test subtree (roksbnkctl test <suite>) holds three suites at v1.0: connectivity, dns, throughput. Adding a fourth (e.g., tls-handshake, latency, tcp-flowstate) follows a five-step recipe:
-
Implement the runner. Create
internal/test/<suite>.go. The suite produces results in theroksbnkctl.<suite>.v1JSON schema — pick a top-level shape consistent with the existing suites (ProbeResultfor single-probe,ProbeSuiteResultfor an aggregate withresults[]). -
Wire a subcommand. Add
internal/cli/test_<suite>.gowith a cobra command undertest. The flag surface should mirror the existing suites’ patterns —--target,--iterations,-o json,--backend(when the suite is backend-aware). -
Pick the backends. Most test suites are backend-aware (the suite runs from a network vantage that the backend selects). DNS and throughput accept
local/k8s/ssh:<target>and rejectdocker; connectivity is currentlylocal-only. Decide which backends make sense for your suite — the deciding question is “does the vantage change the answer?”. -
Wire the JSON schema constant. Add
roksbnkctl.<suite>.v1to your suite’s output. CI assertions diff against this — bumping the version is a breaking change, document it in CHANGELOG. -
Add an E2E phase. New phase under PRD 05 and corresponding script section in
scripts/e2e-test-backends.sh. -
Documentation. New chapter or major section in Part VI of the book (currently chapters 20-23). Cross-link from Chapter 23 — The E2E test plan and Chapter 18 — Choosing a backend per tool.
The DNS probe is the canonical worked example — read internal/test/dns.go + internal/cli/test.go to see all six steps in their landed form.
Adding a new tool to an existing backend
The docker, k8s, and ssh backends each maintain a map of tool-name → image / package. Adding a new tool (e.g., mtr, tcpdump, helm) means an entry in each backend’s map.
Docker backend
internal/exec/docker.go::toolImages maps tool names to image specs:
var toolImages = map[string]string{
"ibmcloud": "ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud",
"iperf3": "ghcr.io/jgruberf5/roksbnkctl-tools-iperf3",
"terraform": "hashicorp/terraform:1.5.7",
"<your>": "<your-image-ref>",
}
Tag resolution is handled by SetToolImageTag (set in internal/cli/root.go::init) — a :dev tag for a from-source binary, :<release-tag> for a tagged release. If your image needs its ENTRYPOINT bypassed (e.g., for image-specific argv mangling), add a jobToolCmdOverride entry.
K8s backend
internal/exec/k8s.go holds two patterns — long-lived ops pod (for tools that share state, like ibmcloud) and one-shot Job (for tools that produce a single output, like iperf3 or DNS probes). New tools pick one pattern:
- Ops pod: add the tool’s image to the ops pod’s container spec at install time, or
kubectl execinto the existing ops pod and run the host-installed binary. - One-shot Job: build a Pod template using the same image conventions as iperf3, run, stream logs, capture exit code, delete. The Job pattern is the right call for tools where the result is the only thing that matters.
SSH backend
internal/exec/ssh.go maintains a map of tool names to apt-package names for the --bootstrap auto-install:
// toolPackage carries apt-repo metadata + package name; see the
// production form in internal/exec/ssh.go for the full struct shape
// (IBM repo URL + GPG key + apt-source line for ibmcloud-cli, etc.).
var toolPackages = map[string]toolPackage{
"ibmcloud": { /* IBM apt repo + key + "ibmcloud-cli" */ },
"iperf3": { /* plain ubuntu-main "iperf3" */ },
"<your>": { /* repo + key + "<deb-package>" */ },
}
The bootstrap step runs apt-get install -y <packages> on the SSH target when the tool isn’t already on PATH. Non-Debian targets are out of scope for v1.0; the bootstrap fails clearly with a message pointing at the manual-install path.
For each backend, the implementation work is small (one map entry). The doctor checks, e2e coverage, and docs are the bulk — same shape as adding a new backend, scaled to the smaller surface.
Supporting a new BNK release
Two version axes vary independently here — the BNK release, and the IBM platform capability the cluster was built with. Adding a release means telling the tool about the new cell, not forking anything.
1. Add a row to internal/config/support_matrix.yaml.
- bnk: "2.5"
contract: [1, 2] # schema versions this line can read
network_modes: [single-nic, multi-nic]
notes: >-
What changed, and anything a reader needs to know before trusting the row.
The line itself is derived from bnk.manifest_version (2.5.0-… → 2.5), so
there is no new config field and nothing that can disagree with the manifest
actually being installed.
2. Add terraform/lines/2.5/ only if the HCL genuinely differs. It mirrors
the base tree and is layered on top at extraction: same path replaces, new path
is added, nothing is removed. A difference a variable can express belongs in
the base as a variable — an overlay is a fork of a file, and two copies of a file
drift. An empty lines/ is the correct state whenever the base serves every
supported release, which is the case today.
3. Add the CI coverage for the cell. From the matrix’s own header: “Adding a cell means adding a CI plan for it. A cell with no coverage is a claim nobody has checked.”
4. Mark it provisional until it has actually run. A row written before the
release ships is an expectation, not an observation. Say so in the row — the
2.4 row does — so that a plan-time pass isn’t mistaken for verification.
Note what is not on this list: cutting a branch. That was the first answer and
it was withdrawn. A branch per release forks the whole tool — CLI fixes, IBM
module updates, docs, CI — to express a difference that lives in a handful of
.tf files, and it makes every subsequent change pick a target branch.
Adding a new chapter to the book
The book is mdBook with markdown source under book/src/. Adding a chapter:
- Create
book/src/<NN>-<slug>.md— the file. Numbered prefix for sort order. - Add the chapter to
book/src/SUMMARY.md— the TOC. Use the existing parts (Concepts, Getting Started, Cluster Lifecycle, …) or add a new part if it doesn’t fit. - Run
make book-serveto live-preview athttp://localhost:3000with auto-reload. - Cross-link from related chapters at the bottom (the “Cross-references” section every chapter ends with).
- Push.
.github/workflows/book.ymlre-deploys togh-pageson every merge tomain.
The book follows a consistent style:
- Lower-case prose, sentence-case section headers.
- Code blocks for any command, inline
codefor filenames and identifiers. - Short paragraphs, one idea each.
- Examples should be runnable as written.
- PRD references use the full GitHub URL (
https://github.com/jgruberf5/roksbnkctl/blob/main/docs/prd/03-EXECUTION-BACKENDS.md) to avoid the published-book 404 issue surfaced in Sprint 1.
The PRD process
The project uses numbered Product Requirements Documents under docs/prd/ for larger feature work — anything that touches multiple files, spans more than one sprint, or has open design questions that need to be settled before code lands.
When a feature warrants a PRD vs. a direct PR:
| Use a PR | Use a PRD |
|---|---|
| Single-file change | Multi-file change across internal/{exec,cli,config,…} |
| Bug fix | New subsystem (a new backend, a new test suite) |
| Doc fix | New surface that needs a stable contract (a JSON schema, a workspace-config field) |
| Refactor with no behaviour change | A change that needs threat-model thinking (creds, network, multi-tenancy) |
| Drive-by polish | Anything cross-cutting >50 LOC |
The PRD lifecycle:
- Draft: open as a markdown file under
docs/prd/NN-<TITLE>.md. The structure should follow the existing PRDs (00-OVERVIEW, 01-SSH, 02-KUBECTL, 03-BACKENDS, 04-CREDS, 05-E2E): goal, approach, file-by-file plan, test plan, acceptance criteria, open questions. - Review: open a PR adding the PRD. Discuss in the PR. Open questions get resolved by edit or by punting to a follow-up issue.
- Implement: the PRD becomes the implementation plan. Per-sprint tasks land in
docs/PLAN.mdreferencing the PRD by number. - Land: code PRs reference the PRD; the PRD itself is the spec, code is the implementation. When the implementation diverges from the PRD, the PRD gets updated to match — never the other way around (the binary’s behaviour is the source of truth).
The PLAN.md per-sprint planning rhythm interleaves code + tests + docs per sprint. Each sprint’s prompts translate the PLAN into concrete agent tasks.
The four-agent sprint dispatch
Larger sprints (Sprints 3-6) are dispatched as four parallel agents:
- Architect — designs the surface, drafts the book chapters that explain it, files architect-side issues.
- Staff engineer — writes the production Go and shell code, modifies the bundled HCL when needed.
- Tech-writer — reviews the architect’s chapters for accuracy, fluency, and cross-link integrity. Files tech-writer-side issues.
- Validator — writes / extends the e2e test scripts and CI workflows, files validator-side issues.
Each agent runs independently from its own prompt — one per agent — against the same repo snapshot. An integrator at the end folds the four agents’ outputs together, resolves the issues each filed against the others, and commits the aggregate.
When to dispatch four agents vs. just open a PR:
| Direct PR | Four-agent sprint |
|---|---|
| Single feature, single sprint, <10 files | Multi-feature sprint with code + docs + tests scope |
| Bug fix | New PRD landing |
| Drive-by improvement | Sprint-gate milestone work |
| You’re the only contributor | Coordinating with reviewers who’d otherwise serialise |
prompts/README.md documents the agent-coordination pattern. The sprint dispatch is the project’s way of running review-and-implementation in parallel rather than serial — it works when the surfaces are well-separated (code vs docs vs tests don’t conflict on file ownership) and the integrator has enough context to merge the four lanes.
Worked example: adding a new execution backend
End-to-end Part IX scenario: you want to add a podman backend (rootless container runtime as an alternative to docker) so users on Fedora/RHEL hosts that ship podman by default don’t have to install Docker just to use the --backend docker workflow. Same surface, different daemon. The walkthrough below tracks the eight-step recipe above with concrete file paths and a diff-shaped sketch of each change.
# 1. Implement the interface — new backend file
cat > internal/exec/podman.go <<'GO'
package exec
import (
"context"
"os/exec"
)
type podmanBackend struct{}
func (p *podmanBackend) Run(ctx context.Context, argv []string, opts RunOpts) (int, error) {
args := append([]string{"run", "--rm"}, dockerStyleArgs(opts)...)
args = append(args, opts.Image)
args = append(args, argv...)
cmd := exec.CommandContext(ctx, "podman", args...)
return runWithRedactor(cmd, opts)
}
func init() {
Register("podman", &podmanBackend{})
}
GO
# 2. Add the tool image mapping (podman uses the same OCI images as docker)
# Edit internal/exec/podman.go — add a toolImages map analogous to docker.go,
# or share the docker.go map by exporting it. The two registries are
# compatible; you'd typically share.
# 3. Doctor check — internal/cli/doctor_backend.go
# Add a `checkPodmanBackend()` function that runs `podman info` once with a
# 2s timeout. Green if exit 0, yellow if podman not found, red if podman
# present but daemon unreachable.
# 4. Wire credentials — re-use the docker backend's cred-propagation logic
# (the `-e VAR` pattern works identically for podman). Pass opts.Credentials
# by env-var reference, never by argv. See internal/exec/docker.go::
# dockerStyleArgs for the pattern to copy.
# 5. Add cred-audit test
cat > internal/exec/podman_audit_test.go <<'GO'
package exec_test
func TestCredAudit_Podman(t *testing.T) {
// Run a no-op command via the podman backend with a known API key,
// then inspect `podman inspect`'s output for the key value. Assert
// the value never appears in the container's labels, env, or args.
}
GO
# 6. E2E phase — extend scripts/e2e-test-backends.sh
# Add Phase P (or extend Phase K) with a parallel sequence to K2-K6 but
# using --backend podman. Cross-link to PRD 05.
# 7. Documentation — chapters 17 + 18
# - Chapter 17: add a "Podman backend" section parallel to "Docker backend",
# noting it's rootless-by-default and a drop-in alternative.
# - Chapter 18: add a row to the per-tool matrix; add a decision-tree entry
# ("I'm on a podman-only host"); update the at-a-glance table.
# 8. Run the full test suite
go build ./...
go vet ./...
go test ./...
DRY_RUN=1 ./scripts/e2e-test-backends.sh
The PR should land all eight steps in one commit-set. A reviewer will look for: registered init(), doctor check, cred-audit test, e2e phase, and the two chapter additions. Without the audit + docs, the PR isn’t complete — see the cardinal rule at the top of the Adding a new execution backend section.
The same pattern applies to a new test suite, a new tool on an existing backend, or a new chapter — the eight-step recipe is the long version; the worked example is the copy-paste short version. Pick the shape that matches your contribution.
Cross-references
- Chapter 17 — Execution backends — the four-backend matrix you’re extending.
- Chapter 19 — The in-cluster ops pod — the k8s-backend pattern your new tool might join.
- Chapter 20-22 — the three existing test suites your new suite would join.
- Chapter 23 — The E2E test plan — where your new phase belongs.
- Chapter 31 — Building from source — the build-side counterpart to the hacking side.
- PRD 00 — Overview — the PRD index.
docs/PLAN.md— the per-sprint planning rhythm.prompts/README.md— the four-agent dispatch pattern.
Agentic mode
Agentic mode lets you drive a ROKS + BNK trial with an AI coding agent (Claude
Code, Gemini CLI, aider, …) acting under role-scoped personas, while
roksbnkctl stays a deterministic tool. The binary embeds no LLM — you
bring your own agent CLI; roksbnkctl provides the shared knowledge base
(AGENTS.md), the role contracts (personas/), and a coordination substrate
(journal/, decisions.md) scaffolded into the workspace.
It’s markdown plus a launcher. Nothing about agentic mode changes how the lifecycle commands behave.
Quick start
roksbnkctl -w acme agent init # scaffold AGENTS.md + personas/ + journal/ into the workspace
roksbnkctl -w acme agent # list supported CLIs + the workspace default
roksbnkctl -w acme agent claude # print the invocation to launch Claude Code there
agent init writes into the workspace dir (~/.roksbnkctl/<workspace>/,
alongside config.yaml):
AGENTS.md shared reference every persona reads first
CLAUDE.md one-line @AGENTS.md include for Claude Code
personas/ the four role contracts
decisions.md decision-log seed
journal/ append-only timeline (handoffs live here)
Re-running agent init never clobbers your edits — existing files are left
as-is.
The personas
Act as exactly one at a time. Each persona file defines an identity, ordered goals, a tool allowlist, what’s explicitly not allowed, a handoff protocol, and a phase checklist.
| Persona | Role | Touches infra? | Talks to customer? |
|---|---|---|---|
solution-architect | Owns scope (config.yaml) + decisions.md; drives the phases | No (delegates) | Yes (the only one) |
cloud-operator | Runs the lifecycle (cluster up, bnk up, …), gates destructive steps | Yes | No |
test-engineer | Runs connectivity/DNS/throughput probes, records evidence | No (read + test) | No |
doc-specialist | Reads everything, writes report.md | No | No |
The boundaries are enforced by convention (the allowlists in each persona file), not by code — so the personas work with any agent CLI.
How they coordinate
config.yaml is the contract (scope/source of truth). The personas hand off
through the append-only journal:
roksbnkctl -w acme journal add "cloud-operator: cluster up complete, nodes Ready"
roksbnkctl -w acme journal list # chronological, with one-line summaries
roksbnkctl -w acme journal report # assemble report.md from decisions.md + the journal
Never edit another persona’s journal entry — append a follow-up instead. The
solution-architect requests infra work by journaling a request (with the
consent tier); the cloud-operator executes and journals the result; the
doc-specialist turns the timeline into the customer-facing report.md.
Choosing the CLI + endpoint
roksbnkctl agent <cli> supports claude, gemini, aider, openai, pi,
opencode, and agy (Google). Set the workspace default and an optional LLM
endpoint in config.yaml:
agent:
default: claude
llm_endpoint: "" # OpenAI-/Anthropic-compatible base URL (cloud vendor or local vLLM); blank = the CLI's own config
The agy recipe seeds the persona with -i, which runs an initial prompt and
then continues interactively. That is deliberate: pi and opencode auto-load
AGENTS.md from the working directory, and their recipes rely on it, but agy
does not advertise that behaviour — so its recipe passes the instruction rather
than assuming the file is read.
roksbnkctl agent <cli> launches the agent, in the workspace directory, with
your terminal attached. Add --show to print the invocation instead of running
it — useful for putting it in a script, or for openai, whose recipe is guidance
for whichever OpenAI-compatible REPL you use rather than a single command.
$ roksbnkctl agent agy # launches agy in the workspace
$ roksbnkctl agent agy --show # prints the invocation, runs nothing
roksbnkctl still embeds no LLM and sends nothing anywhere itself: it execs your
CLI, exactly as roksbnkctl kubectl execs kubectl. Your CLI uses your credentials
and endpoint.
Do not pipe the --show output into a shell. roksbnkctl agent <cli> --show | bash looks like the obvious shortcut and is silently wrong: bash’s standard
input is the pipe carrying the recipe, so the agent’s first turn is the remaining
comment lines of its own recipe. Nothing errors — the session simply begins as
though you had typed them. If you need to run a printed recipe, use
eval "$(roksbnkctl agent <cli> --show)", which leaves your terminal on standard
input.
Running without --show is refused when standard output is not a terminal. An
agent session needs one, and capturing it — $(...) or a pipe — would send the
model’s output somewhere it does not belong.
Safety model
The personas inherit roksbnkctl’s existing guard-rails rather than replacing
them:
- Destructive commands (
cluster up,down,cluster down,cleanup) still prompt unless--autois passed — and the persona contracts forbid passing--autotocluster up/cleanupwithout journaled consent. - Secret-tier commands (
roksbnkctl apikey) and reading key/JWT/token material are off-limits to every persona except where a role genuinely needs it. forge/kubeconfig.yamlcarries the cluster’s admin client certificate/key (ROKS/OpenShift rejects raw IAM tokens) — treat it as secret-tier and never copy it into deliverables.
See the scaffolded AGENTS.md for the full operator reference (the phased
lifecycle, the destructive-command gate contract, and the field-tested
gotchas).
Appendix A — The four cluster topologies, end to end
BNK on ROKS comes in four shapes, and every one of them is the same roksbnkctl pipeline with different answers to two questions:
- Does roksbnkctl build the cluster, or adopt one that already exists?
- Do the workers have Internet egress, or not?
That is the whole taxonomy. This appendix walks all four end to end, twice: once driven by hand from the CLI, once driven as a CI pipeline by Argo Workflows — the same binary, the same commands, the same workspace layout, only the thing typing them changes.
| # | Topology | Cluster | Worker egress | Artifacts from | Licensing | Transit Gateway |
|---|---|---|---|---|---|---|
| A | New + connected | roksbnkctl creates it | yes (public gateway) | repo.f5.com directly | F5 directly | roksbnkctl creates its own |
| B | New + disconnected | roksbnkctl creates it | no | private mirror (Harbor) | F5 License Proxy | joins an existing shared one |
| C | Existing + connected | you already have it | yes | repo.f5.com directly | F5 directly | already attached |
| D | Existing + disconnected | you already have it | no | private mirror (Harbor) | F5 License Proxy | already attached |
A and C need nothing else. They reach F5 over the Internet, so there is no mirror to fill and no proxy to build. If that is your case, skip straight to 1A or 1C — Part 0 does not apply to you.
B and D need the services infrastructure first — a Harbor registry holding a mirror of F5’s artifacts, and a standalone F5 License Proxy, both reachable from the cluster over a Transit Gateway. That is Part 0, and it is built once and shared by every disconnected cluster afterwards.
Related chapters: Air-gapped install, The F5 License Proxy, Sharing a Transit Gateway, Registering an existing cluster.
The topology — what reaches the Internet, and what doesn’t
graph TB
subgraph net["Internet"]
FAR["repo.f5.com<br/>FAR artifacts"]
TEEM["F5 TEEM + licensing"]
end
subgraph svc["SERVICES VPC — built once, shared (topologies B + D)<br/>public gateway = the only egress"]
Harbor["Harbor VSI<br/>OCI mirror · addressed by PRIVATE IP<br/>▶ the CLI operator runs HERE"]
FLP["FLP VSI<br/>F5 License Proxy"]
end
TGW{{"Transit Gateway · global<br/>private RFC1918 only"}}
subgraph conn["A / C — CONNECTED cluster VPC<br/>public_gateway: true"]
CW["workers WITH egress"]
end
subgraph disc["B / D — DISCONNECTED cluster VPC<br/>public_gateway: false"]
DW["workers, no egress"]
Priv["IBM private service endpoints<br/>ICR · IAM · COS · master<br/>161.26.0.0/16 · 166.8.0.0/14"]
end
Harbor -->|egress| FAR
FLP -->|egress| FAR
FLP -->|egress| TEEM
CW -->|"direct"| FAR
CW -->|"direct"| TEEM
Harbor --- TGW
FLP --- TGW
TGW --- DW
DW -->|"images + charts"| Harbor
DW -->|"licensing"| FLP
DW --> Priv
classDef vpc fill:#eef3fb,stroke:#4b6ea9,color:#000;
classDef inet fill:#fdf2f2,stroke:#c0392b,color:#000;
class svc,conn,disc vpc;
class net inet;
Reachability summary
| Component | Internet? | How it reaches what it needs |
|---|---|---|
| Harbor VSI (B, D) | Yes — FAR pull only | public gateway (services VPC) |
| FLP VSI (B, D) | Yes — F5 licensing + TEEM only | public gateway (services VPC) |
| Connected workers (A, C) | Yes | their own public gateway |
| Disconnected workers (B, D) → Harbor / FLP | No | private IPs over the Transit Gateway |
| Any workers → ICR / IAM / COS / master | No | IBM private service-endpoint CIDRs (161.26.0.0/16, 166.8.0.0/14) — routable without a public gateway |
Operator (roksbnkctl, CLI) | on the Harbor VSI for B/D; anywhere with egress for A/C | — |
Why no operator-built VPEs are needed for the platform. A
public_gateway: falseROKS VPC cluster still reaches its IBM Cloud dependencies privately: system-image pulls use private ICR (private.icr.io), IAM/COS use their private endpoints, and the master is reached over the private service endpoint ROKS provisions itself. All of those resolve into IBM Cloud’s service-endpoint ranges (161.26.0.0/16,166.8.0.0/14), which the VPC routes without a public gateway. Calico SNATs pod egress to the node, and the node uses those implicit routes. So the platform works private-only out of the box; you only add explicit VPEs for a specific service instance you want on a dedicated private endpoint (e.g. a particular COS bucket), not for the core ICR/IAM/COS/master path.
roksbnkctl does not build the services VPC, Harbor, or the FLP host image. It provisions no registries and no standalone VPCs (VPC creation only happens inside the cluster phase). Part 0 uses the
ibmcloudCLI for those prerequisites; everything after it isroksbnkctl.
Prerequisites
All four topologies:
- An IBM Cloud API key with VPC, Kubernetes Service and Transit Gateway authority.
terraform >= 1.10onPATH(the floor is enforced — see Chapter 12a), plusroksbnkctlandkubectl.roksbnkctl doctorshould be healthy.- The FAR supply chain:
f5-far-auth-key.tgzandsubscription.jwt, either staged as files or — better for CI — uploaded once into a COS bucket (see the box in S2).
Additionally for B and D (disconnected):
- An existing global Transit Gateway (a global gateway spans regions; a local one does not).
- An RSA VPC SSH key (IBM Cloud VPC rejects ed25519).
- The
ibmcloudCLI with thevpc-infrastructureandtg-cliplugins, plusjq, to build the services VPC in Part 0.
Versions this appendix was tested against
| Component | Tested | Floor, and why |
|---|---|---|
roksbnkctl / roksbnkctl-tools-runner | v1.42.0 | ≥ v1.36.0 for registry adopt (B and D cannot work without it); ≥ v1.37.0 for bnkforge unregister; ≥ v1.39.0 for cluster.vpc_cidr; ≥ v1.40.1 fixes a variable validation that raised Invalid index on every terraform 1.10 plan; ≥ v1.42.0 retries the reachability gate, which matters most for topology B — it builds the VPC and joins the gateway in the same run as the install. |
| terraform | 1.10.5 (shipped inside the runner) | ≥ 1.10, enforced. Do not assume newer behaviour — 1.10 and 1.15 differ in ways that have shipped bugs here. |
| OpenShift (ROKS) | 4.20 (recommended default) | cluster.openshift_version: "4.20" pins the minor; the latest patch within it is selected automatically. The end-to-end run recorded here was on 4.18.51 — the pipeline is version-agnostic, but that is the build the timings and screenshots come from. Check ibmcloud ks versions for what your account offers; IBM’s own default moves ahead of this. |
| Argo Workflows (Part 2) | v4.0.8 | ≥ v3.4 for the emissary executor and sidecars:. |
| Kubernetes hosting Argo (Part 2) | k3s v1.36.3+k3s1 | any conformant cluster with an RWO StorageClass. |
Addressing — give every VPC its own block
Read this before creating anything, because it cannot be changed afterwards.
A Transit Gateway routes on VPC address prefixes. Two attached VPCs claiming the same block make the route ambiguous, and the gateway resolves that by silently blackholing one of them — no error, no log line. It surfaces later as intermittent image-pull timeouts against the mirror while every security group and ACL in the path plainly allows the traffic.
It is easy to hit by accident: IBM’s default address-prefix management is auto, which gives
every VPC in a region the identical three per-zone prefixes
(10.241.0.0/18, 10.241.64.0/18, 10.241.128.0/18). So a second disconnected cluster on a
shared gateway collides by construction.
Set cluster.vpc_cidr (env ROKSBNKCTL_CLUSTER_VPC_CIDR) per cluster. A worked allocation:
| VPC | Block | Per-zone prefixes |
|---|---|---|
| services (Harbor + FLP) | existing, auto | 10.241.0.0/18, .64.0/18, .128.0/18 |
| first disconnected cluster | 10.245.0.0/16 | 10.245.0.0/18, .64.0/18, .128.0/18 |
| second disconnected cluster | 10.247.0.0/16 | 10.247.0.0/18, .64.0/18, .128.0/18 |
The block is split three ways, so /18 is the smallest usable value. Leaving it blank keeps
auto, which is fine for exactly one cluster per gateway. Since v1.39.0 both cluster up and
tgw connect refuse a detectable overlap up front and name the conflicting VPC — but only a
distinct block prevents it. Full write-up: Chapter 9a.
Topology A creates its own gateway and is its only member, so the default is safe there.
Part 0 — The services infrastructure (topologies B and D only)
Built once with the ibmcloud CLI, then shared by every disconnected cluster. If you only
ever run connected clusters, skip this entire part.
export IBMCLOUD_API_KEY=…
export TGW_NAME=my-global-tgw
export SVC_REGION=us-east SVC_ZONE=us-east-1
export SVC_PREFIX=bnk-svc RESOURCE_GROUP=default
export SSH_KEY_NAME=bnk-rsa-key # an RSA key already in IBM Cloud VPC
export HARBOR_ADMIN_PASSWORD=… # you choose it; it is a credential
ibmcloud login --apikey "$IBMCLOUD_API_KEY" -r "$SVC_REGION" -g "$RESOURCE_GROUP" -q
TGW_ID=$(ibmcloud tg gateways --output json | jq -r --arg n "$TGW_NAME" '.[]|select(.name==$n)|.id')
S1 — Services VPC + Harbor, attached to the TGW
# VPC + subnet + public gateway (the only egress) + 22/443 inbound
SVC_VPC_ID=$(ibmcloud is vpc-create "${SVC_PREFIX}-vpc" --resource-group-name "$RESOURCE_GROUP" --output json | jq -r .id)
SVC_VPC_CRN=$(ibmcloud is vpc "$SVC_VPC_ID" --output json | jq -r .crn)
SUBNET_ID=$(ibmcloud is subnet-create "${SVC_PREFIX}-subnet" "$SVC_VPC_ID" --zone "$SVC_ZONE" \
--ipv4-address-count 256 --resource-group-name "$RESOURCE_GROUP" --output json | jq -r .id)
PGW_ID=$(ibmcloud is public-gateway-create "${SVC_PREFIX}-pgw" "$SVC_VPC_ID" "$SVC_ZONE" --output json | jq -r .id)
ibmcloud is subnet-update "$SUBNET_ID" --pgw "$PGW_ID"
SG=$(ibmcloud is vpc "$SVC_VPC_ID" --output json | jq -r .default_security_group.id)
for p in 22 443; do ibmcloud is security-group-rule-add "$SG" inbound tcp --port-min $p --port-max $p; done
# Attach to the gateway — the ONLY path from cluster workers to Harbor.
CONN_ID=$(ibmcloud tg connection-create "$TGW_ID" --name "${SVC_PREFIX}-conn" \
--network-type vpc --network-id "$SVC_VPC_CRN" --output json | jq -r .id)
until [ "$(ibmcloud tg connection "$TGW_ID" "$CONN_ID" --output json | jq -r .status)" = attached ]; do sleep 6; done
Verify
attachedbefore moving on. If this connection silently fails, nothing complains untilbnk updies ~10 minutes in withcert_manager: context deadline exceeded— the nodes could not reach the mirror, and only nodes with a cached image survived.
Harbor’s TLS certificate must carry the floating IP in its SAN, so reserve the IP before rendering cloud-init:
ibmcloud is floating-ip-reserve "${SVC_PREFIX}-harbor-fip" --zone "$SVC_ZONE"
HARBOR_FIP=$(ibmcloud is floating-ips --output json | jq -r --arg n "${SVC_PREFIX}-harbor-fip" '.[]|select(.name==$n)|.address')
HARBOR_FIP_ID=$(ibmcloud is floating-ips --output json | jq -r --arg n "${SVC_PREFIX}-harbor-fip" '.[]|select(.name==$n)|.id')
export HARBOR_FIP HARBOR_ADMIN_PASSWORD HARBOR_VERSION=v2.11.1
envsubst '${HARBOR_FIP} ${HARBOR_VERSION} ${HARBOR_ADMIN_PASSWORD}' \
< scripts/demos/disconnected-cluster-cli-demo/harbor-cloud-init.yaml.tmpl > /tmp/harbor.yaml
VSI=$(ibmcloud is instance-create "${SVC_PREFIX}-harbor" "$SVC_VPC_ID" "$SVC_ZONE" bx2-4x16 "$SUBNET_ID" \
--image ibm-ubuntu-22-04-5-minimal-amd64-17 --keys "$SSH_KEY_NAME" \
--resource-group-name "$RESOURCE_GROUP" --user-data @/tmp/harbor.yaml --output json)
HARBOR_VSI_ID=$(echo "$VSI" | jq -r .id)
VNI=$(echo "$VSI" | jq -r '.primary_network_attachment.virtual_network_interface.id')
ibmcloud is virtual-network-interface-floating-ip-add "$VNI" "$HARBOR_FIP_ID"
# The primary IP is 0.0.0.0 at create time — poll until it binds, or you bake 0.0.0.0
# into generic_host and every mirrored image resolves to https://0.0.0.0/…
until HARBOR_PRIVATE_IP=$(ibmcloud is instance "$HARBOR_VSI_ID" --output json \
| jq -r '.primary_network_interface.primary_ip.address') \
&& [ -n "$HARBOR_PRIVATE_IP" ] && [ "$HARBOR_PRIVATE_IP" != 0.0.0.0 ]; do sleep 5; done
echo "Harbor private=$HARBOR_PRIVATE_IP floating=$HARBOR_FIP"
Give cloud-init 8–15 minutes, then create the projects:
until curl -sk --max-time 8 "https://${HARBOR_FIP}/api/v2.0/systeminfo" | grep -q harbor_version; do sleep 30; done
for pj in bnk-mirror bnk-status; do
curl -sk -u "admin:${HARBOR_ADMIN_PASSWORD}" -X POST "https://${HARBOR_FIP}/api/v2.0/projects" \
-H 'Content-Type: application/json' -d "{\"project_name\":\"$pj\",\"public\":true}"
done
bnk-mirroris public (anonymous pull). Harbor is network-isolated behind the gateway, so the network is the security boundary. Anonymous pull avoids a per-namespace pull-secret / ServiceAccount ordering race that can leave fresh cert-manager pods inImagePullBackOff.
Finally, take the CA from the file that generated it — roksbnkctl refuses to adopt a self-signed CA it merely discovered over the wire:
ssh -i <key> ubuntu@"$HARBOR_FIP" 'sudo cat /opt/harbor/certs/harbor.crt' > harbor-ca.crt
HARBOR_CA_B64=$(base64 -w0 < harbor-ca.crt)
Put the operator on the VSI
For the CLI path, the Harbor VSI is the natural operator host: it is the only box with both Internet egress (to pull FAR) and private reach to Harbor.
This is not a convenience — it is a requirement. bnk up pulls the BNK manifest and the
charts host-side (that is deliberate: pulling the archives itself keeps the terraform helm
provider from doing an OCI login, whose credential-store step breaks on Windows). So the operator
host must reach the mirror’s private address too. Running from a laptop that is not on the
transit gateway gets as far as the manifest pull and then stops:
helm pull oci://10.243.0.4/bnk-mirror/release/f5-bigip-k8s-manifest
dial tcp 10.243.0.4:443: i/o timeout
The node reachability gate does not cover this — it answers for the nodes, which is a different question from whether the operator can pull.
A stock Ubuntu VSI has none of the toolchain, so stage it once:
scp -i <key> "$(command -v roksbnkctl)" ubuntu@<HARBOR_FIP>:/home/ubuntu/roksbnkctl
# terraform >= 1.10 — the floor roksbnkctl enforces (native S3 state locking, Chapter 12a).
# Do not settle for whatever the distro ships; 1.9.x is rejected.
curl -fsSLo /tmp/tf.zip https://releases.hashicorp.com/terraform/1.10.5/terraform_1.10.5_linux_amd64.zip
sudo apt-get install -y unzip && unzip -o -q /tmp/tf.zip -d /tmp && sudo install -m0755 /tmp/terraform /usr/local/bin/
curl -fsSL https://get.helm.sh/helm-v3.16.3-linux-amd64.tar.gz -o /tmp/helm.tgz
tar xzf /tmp/helm.tgz -C /tmp && sudo install -m0755 /tmp/linux-amd64/helm /usr/local/bin/
Then either scp the two FAR artifacts across, or skip staging entirely with the COS supply
chain below — which is what CI does, since a container has no local files.
S2 — Mirror FAR → Harbor
registry replicate pulls from repo.f5.com (the VSI’s egress) and pushes to Harbor at its
private IP.
# mirror.yaml (on the VSI)
ibmcloud: { region: us-east, resource_group: default }
prefix: mirror
tf_source: { type: embedded }
cluster: { create: false, name: none }
bnk:
manifest_version: 2.3.0-3.2598.3-0.0.170
far_auth_local_file: /root/f5-far-auth-key.tgz
subscription_jwt_local_file: /root/subscription.jwt
registry:
target: generic # ← see the warning below
generic_host: <HARBOR_PRIVATE_IP> # local to the VSI, private to the cluster
generic_repo_prefix: bnk-mirror
generic_username: admin
generic_password_b64: <base64 of the Harbor admin password>
generic_ca_b64: <HARBOR_CA_B64> # from the file, not the wire
$ roksbnkctl -w mirror init --config-file mirror.yaml
$ roksbnkctl -w mirror registry bom # the bill of materials to mirror
$ roksbnkctl -w mirror registry replicate # FAR → Harbor (89 artifacts)
$ roksbnkctl -w mirror registry verify
registry.target: genericis not optional, and a--targetflag is not a substitute.registry adopt— which topologies B and D run — has no--targetflag; it readsregistry.targetfrom the workspace config. Left unset it defaults toicr,bnk uprendersus.icr.io/<prefix>/…, and every pull on an air-gapped cluster fails withunauthorized: Authorization required. Passing--target generictoreplicatefills Harbor correctly but writes nothing the lateradoptreads, so the two silently disagree.
Local files vs. the COS supply chain. The config above points
bnk.far_auth_local_file/bnk.subscription_jwt_local_fileat files on the operator VSI — simplest when driving by hand. The alternative is to upload both once into a COS bucket and read them from there. This is mandatory for CI, where a container has no local files:roksbnkctl cos object put bnk-artifacts-<acct>/f5-far-auth-key.tgz ./f5-far-auth-key.tgz --instance bnk-supply-chain roksbnkctl cos object put bnk-artifacts-<acct>/subscription.jwt ./subscription.jwt --instance bnk-supply-chainthen drop the two
*_local_filekeys and add:cos: { instance: bnk-supply-chain, bucket: bnk-artifacts-<acct>, region: us-south } bnk: far_auth_file: f5-far-auth-key.tgz # object keys in the bucket above subscription_jwt_file: subscription.jwtThe commands do not change — only the config keys. See Chapter 25 — the COS supply chain.
S3 — Standalone FLP licensing appliance
The FLP is a self-contained F5 licensing appliance — a VSI running the f5-license-proxy
stack as a podman pod, with no cluster. It is the one box with controlled egress to F5: it
pulls its own images from repo.f5.com and brokers licences (and sends TEEM telemetry) on the
cluster’s behalf, over a private IP the disconnected cluster reaches via the gateway.
# flp.yaml (on the VSI)
ibmcloud: { region: us-east, resource_group: default }
prefix: flp
tf_source: { type: embedded }
cluster: { create: false, name: none }
bnk:
manifest_version: 2.3.0-3.2598.3-0.0.170
far_auth_local_file: /root/f5-far-auth-key.tgz
subscription_jwt_local_file: /root/subscription.jwt
flp:
mode: vsi # mode:vsi + a non-empty vpc selects the cluster-less path
vsi:
vpc: <SVC_VPC_ID>
zone: us-east-1
profile: bx2-4x16
ssh_key: <SSH_KEY_NAME>
reach: private
floating_ip: true # operator address for `flp status` and the :80 UI
$ roksbnkctl -w flp flp init --config-file flp.yaml
$ roksbnkctl -w flp flp up --auto # builds the VSI + the licence-proxy pod
$ roksbnkctl -w flp flp output # → flp_external_endpoint + root CA
Keep both outputs — every disconnected cluster needs them as bnk.flp.external.url and
bnk.flp.external.root_ca_b64.
The root CA is already base64. Pass it verbatim. Re-encoding it hands the CWC a corrupt CA and licensing fails in a way that looks like a network problem.
The status web UI (optional)
flp status queries a small status service on port 80 of the FLP. It is an optional add-on:
it only exists if you built and pushed its image and set flp.vsi.status_image. The licence proxy
works perfectly without it — do not let its absence fail a pipeline.
Part 1 — Driving it from the CLI
Four topologies, four subsections. Each is a complete config plus the commands, and each one is
one roksbnkctl workspace (-w <name>) — a workspace’s terraform state describes exactly one
cluster, so never point two clusters at the same one.
1A — New VPC + connected cluster (CLI)
The simplest shape, and the one to read first. Workers have egress, so BNK pulls charts and
images straight from repo.f5.com and licenses directly against F5. Nothing is mirrored and
nothing is proxied — there is no registry: block and no flp: block, and their absence is
what makes it connected.
# connected.yaml
ibmcloud: { region: us-east, resource_group: default }
prefix: bnk-conn
tf_source: { type: embedded }
cluster:
create: true
name: bnk-conn # a NEW cluster is named from the prefix; see the note below
openshift_version: "4.20"
workers_per_zone: 1
public_gateway: true # ← CONNECTED: worker Internet egress
resources:
transit_gateway: { create: true } # its own gateway; it is the only member
registry_cos: { create: true } # REQUIRED when creating — see the warning
cert_manager: { create: true }
bnk: { create: true }
tgw_jumphost: { create: false }
cluster_jumphosts: { create: false }
client_vpc: { create: false }
bnk:
manifest_version: 2.3.0-3.2598.3-0.0.170
far_auth_local_file: /root/f5-far-auth-key.tgz
subscription_jwt_local_file: /root/subscription.jwt
# license_mode defaults to `connected` — F5 directly
$ roksbnkctl -w bnkconn init --config-file connected.yaml
$ roksbnkctl -w bnkconn cluster up --auto # ~45–55 min
$ roksbnkctl -w bnkconn bnk up --auto # ~10 min
$ roksbnkctl -w bnkconn bnk status
⚠
registry_cos: { create: true }is mandatory when creating a cluster. ROKS-on-VPC refuses to provision without a COS instance backing its internal image registry (E7278). That is IBM Cloud COS over the private service-endpoint range — needed even air-gapped.create: falsefails the cluster create outright.
A new cluster is named from
prefix, not fromcluster.name.cluster.nameonly selects which cluster to adopt (1C/1D). If you set a prefix ofbnk-conn, the cluster comes up asbnk-conn— and that is the name 1C must later register.
1B — New VPC + disconnected cluster (CLI)
Requires Part 0. Three things make
it disconnected, and all three are visible in the config: public_gateway: false, a registry:
block pointing at Harbor, and license_mode: f5licenseproxy with the FLP endpoint.
It also joins an existing gateway rather than creating one — a new gateway of its own would isolate it from the very mirror it exists to install from.
# disconnected.yaml (on the Harbor VSI)
ibmcloud: { region: us-south, resource_group: default }
prefix: bnk-disco
tf_source: { type: embedded }
cluster:
create: true
name: bnk-disco
openshift_version: "4.20"
workers_per_zone: 1
public_gateway: false # ← DISCONNECTED: no worker egress
vpc_cidr: 10.242.0.0/16 # ← its own block; see "Addressing" above
resources:
transit_gateway: { create: false, existing: my-global-tgw } # JOIN the mirror's gateway
registry_cos: { create: true }
cert_manager: { create: true }
bnk: { create: true }
tgw_jumphost: { create: false }
cluster_jumphosts: { create: false }
client_vpc: { create: false }
registry:
target: generic
generic_host: <HARBOR_PRIVATE_IP> # reached over the TGW
generic_repo_prefix: bnk-mirror
generic_username: admin
generic_password_b64: <base64 of the Harbor admin password>
generic_ca_b64: <HARBOR_CA_B64>
bnk:
manifest_version: 2.3.0-3.2598.3-0.0.170
far_auth_local_file: /root/f5-far-auth-key.tgz
subscription_jwt_local_file: /root/subscription.jwt
license_mode: f5licenseproxy
flp:
external:
url: <flp_external_endpoint from S3>
root_ca_b64: <flp_root_ca from S3> # already base64 — verbatim
$ roksbnkctl -w bnkdisco init --config-file disconnected.yaml
$ roksbnkctl -w bnkdisco cluster up --auto # ~45–55 min; TGW connect runs at the end
$ roksbnkctl -w bnkdisco tgw status
$ roksbnkctl -w bnkdisco registry adopt # ← record the mirror in THIS workspace
$ roksbnkctl -w bnkdisco bnk up --auto
$ roksbnkctl -w bnkdisco bnk status
registry adoptis the step people miss.bnk uprefuses to render against a mirror this workspace has no record of — otherwise BNK would be pointed atfar_repo_url, which an air-gapped cluster cannot reach. Onlyregistry replicatewrites that record, and replicating again would need the FAR source this cluster has no route to.adoptwrites the record from the registry config alone, with no source access. That is the entire reason it exists.If you mirrored in a different workspace on the same host (S2 used
-w mirror), copying the record works too:cp ~/.roksbnkctl/mirror/registry-mirror.json ~/.roksbnkctl/bnkdisco/.
Node CA trust is automatic. Before pulling, CRI-O on each node must trust Harbor’s self-signed cert or every pull fails
x509. The usual OpenShift mechanism does not work on ROKS —image.config.openshift.io/clusteris HostedCluster-managed and a ValidatingAdmissionPolicy denies edits.bnk upinstalls the CA into each node’s/etc/containers/certs.d/<HARBOR_PRIVATE_IP>/ca.crtvia a privileged DaemonSet using a node-cached image (so it needs no egress), and gates the install on the CA landing on every node. Because Harbor is addressed by an IP, thecerts.dkey is that IP — no node/etc/hostsentry needed.
Reachability is checked from every node first
The same DaemonSet also probes the mirror and the licence proxy from every node before
anything tries to pull, and bnk up fails if the mirror is unreachable from any of them:
→ installing registry CA trust on all nodes (10.243.0.4) and checking reachability
F5-License-Proxy: 3/3 nodes reachable
✓ kube-…-0000013a -> F5-License-Proxy (10.243.1.4:8443): dns=skipped-ip tcp=ok
registry: 3/3 nodes reachable
✓ kube-…-0000013a -> registry (10.243.0.4:443): dns=skipped-ip tcp=ok
✓ registry CA installed on all nodes; 10.243.0.4 is trusted and reachable
It has to run on a node, not on the operator. The operator host sits on the services VPC
with egress; the workers are air-gapped behind the gateway. A check from where roksbnkctl runs
answers a different question and will return a confident green for a mirror the cluster cannot
route to. Because a DaemonSet is one pod per node, this covers every availability zone without
enumerating them — which matters, since a security group or subnet fixed in only two of three
zones is a common shape and otherwise shows up merely as some pods failing to pull.
DNS and TCP are reported separately because the fixes differ: dns=FAILED is a resolver or
hostname problem, tcp=FAILED is routing, security groups, or overlapping VPC address prefixes
(which a gateway blackholes silently — Addressing).
The mirror is required: an unreachable one fails the install rather than deferring it to
ImagePullBackOff and a helm context deadline exceeded that names neither the registry nor the
node. The licence proxy is reported but not fatal, since a staged bring-up may legitimately point
at an endpoint that is not up yet.
Two details worth knowing:
- No CA is needed for the probe to run. A registry already trusted by the node bundle, or one with a publicly-signed certificate, skips the CA install but is still checked — being trusted and being reachable are different things, and only the second fails silently.
- Every node must report. A node whose pod has not answered yet is not counted as a pass; the check waits for the DaemonSet’s full complement and fails if it cannot hear from them all, because a silent node is indistinguishable from a broken one.
It retries, and the budget is yours to set
A Transit Gateway attachment is asynchronous. IBM programs the routes some time after the
connection reports attached, so a cluster built and installed in one run can probe a path that
is correct but not yet live. This has been seen for real: a probe 73 seconds after attach failed
on both targets, bnk up refused, and the same path was healthy minutes later — while a sibling
cluster on the same gateway and the same blueprint passed, purely by landing on the other side of
route programming.
“Unreachable” and “not reachable yet” are different claims, and one TCP failure cannot tell them apart. So each target is retried until its budget is spent; a success ends the retry immediately, so a healthy environment never pays for it. A failure then says how hard it tried:
✗ kube-…-0000013a -> registry (10.243.0.4:443): dns=skipped-ip tcp=FAILED
(tcp connect failed -- refused, filtered, or no route
(still failing after 19 attempts over 180s))
Both timers are workspace settings, because the right values are a property of the environment rather than of the tool:
bnk:
preflight:
reachability_retry_seconds: 180 # per target, before the verdict is believed
reachability_timeout_seconds: 480 # for every node to report
| Env var | Default | When to change it | |
|---|---|---|---|
reachability_retry_seconds | ROKSBNKCTL_REACHABILITY_RETRY_SECONDS | 180 | Raise when the gateway is attached in the same run as the install, or the fabric programs routes slowly. Set 0 for a static environment where a failure is never a race — one shot, fail immediately. |
reachability_timeout_seconds | ROKSBNKCTL_REACHABILITY_TIMEOUT_SECONDS | 480 | Raise alongside a large retry budget. It is clamped to stay above the retry budget regardless, since a wait shorter than the retry would give up while the probe was still working. |
The env vars matter for the CI path, where the workspace is built by
init --non-interactive --override-from-env and there is no config.yaml to edit.
Each run gets a fresh verdict
The probe runs once per pod and then holds the pod Ready. That means the result bnk up reads is
only as fresh as the pod that wrote it — so the gate rolls the DaemonSet on every run, even
when the CA and the target list are unchanged. Without that, a re-run after fixing the routing
re-read the original pod’s log and showed the same failure, with nothing to indicate it was a
recording. The cost is one pod restart per node against a node-cached image, and the CA install
is idempotent.
Keep
SSL_CERT_FILEpointed at Harbor’s cert when running on the VSI — the chart pulls happen host-side:export SSL_CERT_FILE=/opt/harbor/certs/harbor.crt.
1C — Existing connected cluster (CLI)
roksbnkctl provisions nothing here. cluster register records the cluster’s identity — VPC,
endpoints, registry COS — into cluster-outputs.json, which is what activates bnk up’s
existing-cluster path.
# adopt-connected.yaml
ibmcloud: { region: us-east, resource_group: default }
prefix: bnk-conn
tf_source: { type: embedded }
cluster:
create: false # ← ADOPT
name: my-running-cluster # ← the cluster to adopt, by name or id
resources:
registry_cos: { create: false } # it already has one
cert_manager: { create: true }
bnk: { create: true }
tgw_jumphost: { create: false }
cluster_jumphosts: { create: false }
client_vpc: { create: false }
bnk:
manifest_version: 2.3.0-3.2598.3-0.0.170
far_auth_local_file: /root/f5-far-auth-key.tgz
subscription_jwt_local_file: /root/subscription.jwt
$ roksbnkctl -w bnkconn init --config-file adopt-connected.yaml
$ roksbnkctl -w bnkconn cluster register my-running-cluster # writes cluster-outputs.json
$ roksbnkctl -w bnkconn kubeconfig --download # for kubectl
$ roksbnkctl -w bnkconn bnk up --auto
$ roksbnkctl -w bnkconn bnk status
cluster upalso adopts. Ifcluster.createisfalse,roksbnkctl cluster upattaches to the cluster named incluster.name— the same discovery andcluster-outputs.jsonwrite — instead of running the terraform create. One command covers both, which is what makes a single CI form able to drive either. See Chapter 9.
Re-installing over a live BNK? Watch the CWC.
f5-spk-cwcuses a single-replica RWO PVC with aRollingUpdatestrategy, so a re-install can deadlock: the new pod cannot mount the volume until the old one releases it, and the old one is not torn down first (Multi-Attach). Patch the Deployment tostrategy: Recreatewhilebnk upruns. A fresh install has no prior CWC and does not hit this — it is specific to re-use. Part 2 automates it as a sidecar.
1D — Existing disconnected cluster (CLI)
1C plus the mirror and the proxy. Requires Part 0.
Take the 1C config and add the registry: block and the bnk.flp block exactly as in
1B, keeping cluster: { create: false, name: … } and
registry_cos: { create: false }.
$ roksbnkctl -w bnkdisco init --config-file adopt-disconnected.yaml
$ roksbnkctl -w bnkdisco cluster register my-air-gapped-cluster
$ roksbnkctl -w bnkdisco kubeconfig --download
$ roksbnkctl -w bnkdisco tgw connect my-global-tgw # idempotent; skip if already attached
$ roksbnkctl -w bnkdisco registry adopt
$ roksbnkctl -w bnkdisco bnk up --auto
$ roksbnkctl -w bnkdisco bnk status
An adopted cluster usually already runs cert-manager. OpenShift estates commonly install it as a day-1 add-on, and
bnk upthen stops withnamespaces "cert-manager" already exists. That message names the collision but not the fix: setresources.cert_manager: { create: false }to adopt the one that is there. From CI, where there is noconfig.yamlto edit, the same thing is-e ROKSBNKCTL_CERT_MANAGER_CREATE=false. Check withkubectl get ns cert-managerbefore the firstbnk uprather than after it.
Check the addressing before attaching. The existing cluster’s VPC must be on the same global gateway as Harbor and its address prefixes must not overlap the services VPC, or the gateway blackholes one of them.
tgw connectrefuses a detectable overlap and names the conflicting VPC. An existing VPC’s prefixes cannot be changed in place — moving a subnet’s CIDR replaces the subnet and destroys the cluster on it — so the remedy is to detach the other VPC, or rebuild this cluster on its own block.
Verifying any of the four
$ kubectl get pods -n f5-bnk # all Running
$ kubectl get pods -n f5-utils # the utility stack, incl. f5-spk-cwc
$ kubectl get license -n f5-utils # STATE: Active
$ kubectl get cneinstance -n f5-bnk -o \
jsonpath='{.items[0].status.conditions[?(@.type=="Available")].status}' # True
The one check that distinguishes connected from disconnected is where the images came from:
$ kubectl -n cert-manager get pods \
-o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}'
quay.io/jetstack/cert-manager-controller:v1.17.3 # A / C — connected
10.241.0.4/bnk-mirror/jetstack/cert-manager-controller:… # B / D — from the mirror
Teardown (CLI)
Per cluster workspace, then the shared services last:
$ roksbnkctl -w <ws> bnk down --auto # BNK only; the cluster stays
$ roksbnkctl -w <ws> tgw disconnect --auto # detach BEFORE destroying the cluster
$ roksbnkctl -w <ws> cluster down --auto # the cluster + its VPC (+ its own TGW for topology A)
Then, only when no disconnected cluster is left:
$ roksbnkctl -w flp flp down --auto
$ ibmcloud is instance-delete "${SVC_PREFIX}-harbor" --force
$ ibmcloud tg connection-delete "$TGW_ID" "$CONN_ID"
$ ibmcloud is vpc-delete "$SVC_VPC_ID" --force # after its subnet / pgw / floating IP are gone
Order matters, twice over.
cluster downrefuses while a TGW connection exists — the connection pins the VPC’s CRN and the VPC delete would fail — hencetgw disconnectin the middle. And disconnecting is not deleting: it removes only this cluster’s connection, so the shared gateway, the mirror’s connection and every other cluster’s stay. Topologies C and D adopted their clusters, socluster downwill not destroy them; remove BNK withbnk downand leave the cluster alone.
Part 2 — Driving it from CI with Argo Workflows
The same four topologies, same binary, same commands — as a pipeline. Each step is its own pod
with its own status, logs and retries (the per-step visibility a single Job cannot give), and
the workspace lives on a persistent PVC shared across the steps.
The complete, runnable manifests are in
scripts/demos/blueprint-workflows-ci-demo.
This part explains their shape; the repository has the YAML.
Everything is an environment variable
There is no config.yaml anywhere in Part 2 — the whole workspace is built from the environment
by init --non-interactive --override-from-env, because that is the only shape available to a
container module or an argv-only runner: no shell, no prompts, nowhere to stage a file.
| Carrier | Holds |
|---|---|
bnk-env ConfigMap | every non-secret ROKSBNKCTL_* setting — readable in the Argo UI and via kubectl get cm bnk-env -o yaml |
bnk-secrets Secret | IBMCLOUD_API_KEY, ROKSBNKCTL_GENERIC_PASSWORD, and friends — never rendered, logged or printed |
Adoption is environment-only too. A cluster the customer already built usually already has some
of what bnk up would otherwise create, and each of those is a create toggle:
ROKSBNKCTL_CERT_MANAGER_CREATE=false (the common one — see
1D), ROKSBNKCTL_REGISTRY_COS_CREATE=false when the
mirror is external, and ROKSBNKCTL_CLUSTER_JUMPHOSTS_CREATE. Without these the CI path could
express “adopt this cluster” but not “adopt what is on it”, and topologies C and D would stop on
a collision no environment variable could clear.
Settings that define a topology — ROKSBNKCTL_CLUSTER_CREATE,
ROKSBNKCTL_CLUSTER_PUBLIC_GATEWAY, ROKSBNKCTL_LICENSE_MODE — are pinned in the workflow YAML
next to the steps they govern, not in the ConfigMap. That is why the connected and disconnected
workflows differ in one line.
Bring your own controller — versions and constraints
Argo Workflows, not Argo CD. These are argo submit pipelines: no git repository, no
Application, nothing to sync. If you already run Argo CD, it is not what drives this — you need
the Argo Workflows controller alongside it. (Argo CD can own the lifecycle of the
prerequisites below, since they are ordinary manifests.)
Versions tested are in the table above. The constraints your controller must satisfy — the first architectural, the rest each having cost a real run:
-
It must sit where it can reach the private addresses. For topologies B and D the controller pods pull the runner image from Harbor’s private IP and talk to the target cluster’s API. A hosted or SaaS Argo can do neither. Its nodes must be on, or routed to, the services VPC.
-
One workflow at a time per workspace. The PVC holds one terraform state, and
ReadWriteOncemeans one pod mounts it at a time anyway. Two runs against the same workspace fight over the state lock and the loser dies mid-apply. The sharedbnk-envConfigMap compounds it: it is re-rendered per run, andenvFromresolves per pod at creation, so an already-running workflow’s pending steps silently pick up the next run’s cluster name. -
One roksbnkctl workspace per cluster —
bnkfor the mirror,bnkconnfor the connected pair,bnkdiscofor the disconnected pair,flpfor the proxy. Terraform state describes one cluster, andregistry-mirror.jsonis inherited by anything sharing the workspace, which is exactly how a connected cluster ends up rendering every image at a mirror it cannot reach. -
ROKSBNKCTL_REGISTRY_TARGET=genericfor B and D — see the warning in S2. -
The namespace is
bnk-ciin the manifests, hardcoded in each Workflow’smetadata.namespace.
The shared substrate
Applied once (workflows/00-prereqs.yaml): the namespace, the PVC, the ServiceAccount, and the
executor RBAC.
apiVersion: v1
kind: Namespace
metadata: { name: bnk-ci }
---
apiVersion: v1
kind: PersistentVolumeClaim # the persistent workspace every Workflow shares
metadata: { name: bnk-work, namespace: bnk-ci }
spec:
accessModes: [ReadWriteOnce]
resources: { requests: { storage: 8Gi } }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: bnk-runner, namespace: bnk-ci }
---
apiVersion: rbac.authorization.k8s.io/v1 # Argo's emissary executor writes workflowtaskresults
kind: Role
metadata: { name: bnk-runner-executor, namespace: bnk-ci }
rules:
- { apiGroups: ["argoproj.io"], resources: ["workflowtaskresults"], verbs: ["create", "patch"] }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: bnk-runner-executor, namespace: bnk-ci }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: bnk-runner-executor }
subjects: [{ kind: ServiceAccount, name: bnk-runner, namespace: bnk-ci }]
The PVC must not be an
emptyDir. roksbnkctl keeps its config, terraform state and the mirror record under/work/.roksbnkctl, and every teardown reads that state. AnemptyDirloses it when the pod ends — orphaning the IAM trusted profile and pull secrets, and leaving terraform nothing to destroy from.
A second, deliberately separate file (workflows/01-flp-handoff-rbac.yaml) grants write access to
one named Secret, needed only if a workflow publishes the FLP handoff. It is separate so a
blanket secret-write grant is never applied by accident.
Every step reuses one template:
- name: rbk
inputs: { parameters: [{ name: cmd }] }
container:
image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.42.0
command: [sh, -ec]
args: ["roksbnkctl {{inputs.parameters.cmd}}"]
workingDir: /work
envFrom:
- configMapRef: { name: bnk-env }
- secretRef: { name: bnk-secrets }
env:
- { name: ROKSBNKCTL_HOME, value: "/work/.roksbnkctl" }
volumeMounts: [{ name: work, mountPath: /work }]
For B and D the image is pulled from the mirror instead (
<HARBOR_PRIVATE_IP>/bnk-mirror/roksbnkctl-tools-runner:v1.42.0), so the pipeline itself needs no public registry at run time. Pin by@sha256:digest in production.
2A — New VPC + connected cluster (Argo)
wf-new-cluster.yaml. Steps: init → cluster up → bnk up → bnk status.
env:
- { name: ROKSBNKCTL_CLUSTER_CREATE, value: "true" }
- { name: ROKSBNKCTL_CLUSTER_PUBLIC_GATEWAY, value: "true" } # ← CONNECTED
# Blanked ON PURPOSE — bnk-env carries a gateway name for the reuse workflows, and
# inheriting it here would ADOPT the shared gateway instead of creating its own.
# OverrideFromEnv skips empty values, so blank leaves transit_gateway at create:true.
- { name: ROKSBNKCTL_TRANSIT_GATEWAY_NAME, value: "" }
# Blanked for the same reason: bnk-env carries the mirror host for the disconnected
# workflows. Inherited, `init` would write registry.generic_host into this workspace
# and every image would render at a mirror this cluster cannot reach.
- { name: ROKSBNKCTL_GENERIC_HOST, value: "" }
Uses workspace -w bnkconn.
2B — New VPC + disconnected cluster (Argo)
wf-new-cluster-disconnected.yaml. Steps: init → cluster up → registry adopt →
bnk up → bnk status.
envFrom:
- configMapRef: { name: bnk-env }
- secretRef: { name: bnk-secrets }
- secretRef: { name: flp-handoff } # written by the FLP workflow; last envFrom wins
env:
- { name: ROKSBNKCTL_CLUSTER_CREATE, value: "true" }
- { name: ROKSBNKCTL_CLUSTER_PUBLIC_GATEWAY, value: "false" } # ← DISCONNECTED
- { name: ROKSBNKCTL_LICENSE_MODE, value: "f5licenseproxy" }
# ROKSBNKCTL_TRANSIT_GATEWAY_NAME comes from bnk-env and is REQUIRED here.
# ROKSBNKCTL_CLUSTER_VPC_CIDR likewise — this is the ONE workflow that creates a VPC
# AND joins a shared gateway, so a second run collides by construction without it.
Uses workspace -w bnkdisco. The FLP endpoint and CA arrive as a Secret (flp-handoff)
published by the FLP workflow rather than passing through a human.
2C — Existing connected cluster (Argo)
wf-existing-cluster.yaml. Steps: init → cluster register → kubeconfig --download →
bnk up (with the CWC guard sidecar) → bnk status.
The kubeconfig fetch is new relative to 2A: the adopt path needs it so the k verbs and the guard
can reach the cluster. Workspace -w bnkconn, with the same ROKSBNKCTL_GENERIC_* blanking.
- name: bnk-up
container:
image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.42.0
args: ["roksbnkctl -w bnkconn bnk up --auto"]
# …envFrom / env / volumeMounts as the rbk template…
sidecars:
- name: cwc-guard
image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.42.0
command: [sh, -ec]
args:
- |
# Patch the single-replica RWO CWC Deployment to Recreate so the rolling
# update cannot deadlock on Multi-Attach, then wait for the licence.
kubectl -n "$CWC_NAMESPACE" patch deploy "$CWC_DEPLOYMENT" \
-p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'
…
Why a sidecar and not a step. The deadlock it clears is what makes
bnk upfail, so a guard running afterwards could never run at all. It has to be alongside. Observed working:cwc: strategy=Recreate applied/cwc: licence Active — guard done, whilebnk upwas still running. The guard is on the reuse workflows only — a fresh install has no prior CWC and does not deadlock.
2D — Existing disconnected cluster (Argo)
wf-existing-disconnected.yaml — 2C plus the mirror. Steps: init → cluster register →
kubeconfig --download → registry adopt → bnk up (with the CWC guard sidecar) →
bnk status. Workspace -w bnkdisco, flp-handoff in envFrom.
The two prerequisite workflows
Topologies B and D also need Part 0 filled, and that is itself two workflows:
| Workflow | Steps |
|---|---|
wf-far-mirror.yaml | init → registry target → registry bom → registry replicate --target generic → registry verify |
wf-flp-vsi.yaml | init → flp up → (optional) flp status → publish the flp-handoff Secret |
flp status is gated behind a parameter that defaults to false, because the status service is
an optional add-on: asserting on a component that was never configured would fail a run whose real
output — the handoff Secret — is perfectly good.
Teardown (CI)
A pod that mirrors the workflow container exactly — same image, same PVC, same env carriers — because anything less has no terraform state to destroy from:
$ ./blueprint-workflows-ci-demo.sh teardown bnkdisco # bnk down → tgw disconnect → cluster down
$ ./blueprint-workflows-ci-demo.sh teardown bnkconn flp
The workspace filter matters: running all four topologies means two clusters with two different
prefixes, and teardown rebuilds each workspace’s config from the current bnk-env — so each
must come down under the environment it was built with, one at a time.
Timing — what to expect
| Phase | Typical |
|---|---|
| Services VPC + Harbor (S1, incl. cloud-init) | 10–20 min |
| Mirror FAR → Harbor (S2) | 5–10 min |
| FLP VSI (S3) | 3–5 min |
cluster up (a real ROKS cluster) | 45–55 min |
bnk up | 8–12 min |
bnk up on an already-installed cluster (2C/2D) | 1–3 min |
cluster down | 10–20 min |
The scripted walkthroughs
Everything above is automated in the repository, and the scripts are the executable form of this appendix:
| Script | Covers |
|---|---|
disconnected-cluster-cli-demo | Part 0 + topology D from the CLI |
disconnected-cluster-ci-demo | the same as a two-Workflow pipeline, plus the k3s + Argo VSI build |
artifactory-adopt-ci-demo | topology D with nothing created at all — an existing gateway, VPC and cluster, images from a JFrog Artifactory the customer already runs, and cert-manager adopted. No config.yaml exists: the workspace is built entirely from docker run -e. This is the shape most customers arrive in. |
blueprint-workflows-ci-demo | all four topologies as Argo Workflows, plus the mirror and FLP prerequisites |
B — Replicating FAR into a JFrog Artifactory
This appendix mirrors the F5 Artifact Repository into a JFrog Artifactory you already run. It does not cover standing one up; that is the standalone Artifactory demo’s job.
Nothing here needs a Kubernetes cluster. registry replicate copies
registry-to-registry from wherever roksbnkctl runs, so the mirror can be seeded
long before a cluster exists — which is usually the point, since the cluster is
air-gapped and the mirror is what makes it installable at all.
Everything below has been run end to end against a self-hosted Artifactory. Where
a behaviour is edition-specific it says so, because the free and licensed
editions differ in ways that are not obvious until something returns a 400.
What Artifactory has to provide
A local Docker repository. BNK’s artifacts are OCI images and OCI-packaged Helm charts, so the target must be a Docker repository — a local one. A remote repository is a read-through cache of an upstream and cannot be pushed to; a virtual repository accepts a push only if it has a local deployment target, so pointing at the local repository directly is simpler and fails more clearly when permissions are wrong.
Create it in the web console: Administration → Repositories → Add Repository →
Local Repository → Docker. Any name works; the examples here use bnk-mirror.
Creating repositories is a console job on the free edition. The REST API answers
400 This REST API is available only in Artifactory Pro, so a repository cannot be created by script on JFrog Container Registry. The web UI works fine — it is the automated paths that are restricted. On Pro or Enterprise the REST call works normally.
A credential that can deploy. An access token is the right shape: it is validated independently of the account password, so rotating the password does not invalidate it. That matters because the same credential is handed to a pipeline.
curl -u admin:'<password>' -X POST \
-d username=admin -d scope=applied-permissions/user -d expires_in=0 \
https://artifactory.example.com/artifactory/api/security/token
expires_in=0 issues a non-expiring token. Omitting scope fails with
Insufficient scope, which is not obvious from the error alone.
A scoped user needs a licensed edition. Users, groups and permission targets are Pro features; on JCR their REST APIs answer
400 … only in Artifactory Proand the corresponding screens are absent, so the mirror runs asadmin. That suits a single-purpose mirror host and does not suit a shared registry — on Pro, create a user, grant it read and write on the repository through a permission, and issue that user’s token instead.
Pointing roksbnkctl at it
roksbnkctl registry target generic
roksbnkctl registry target generic_host artifactory.example.com
roksbnkctl registry target generic_repo_prefix bnk-mirror
roksbnkctl registry target generic_username admin
echo "$ARTIFACTORY_TOKEN" | roksbnkctl registry target generic_password --password-stdin
--password-stdin keeps the token out of shell history, out of ps output, and
out of any terminal recording. The password field accepts either an access token
or the account password; a token is preferable for the reason above.
The repository name is the only thing that changes for a different
repository. roksbnkctl composes references as
<generic_host>/<generic_repo_prefix>/<image>, so the name is a path segment —
bom, diff, replicate, verify, list, prune and delete are identical
whatever it is called.
That composition is Artifactory’s repository-path method. If your instance is
configured for the subdomain method instead, put the full subdomain in
generic_host and leave generic_repo_prefix empty. Getting this wrong produces
a 404 that reads like a bad prefix and sends you looking in the wrong place.
If Artifactory presents a certificate from a private CA, record the CA from the file that issued it:
roksbnkctl registry target generic_ca /etc/pki/ca-trust/source/anchors/corp-root.crt
Recording it from the file rather than capturing it from the host matters: a CA
learned over the connection it is meant to authenticate proves nothing. With
generic_ca set, replicate never dials the registry to discover trust. A
publicly-trusted certificate needs none of this, and replicate will say so —
no private CA captured is expected, not a warning to act on.
Doing the replication
roksbnkctl registry bom # what BNK needs — offline, no registry required
roksbnkctl registry diff # what is missing from Artifactory right now
roksbnkctl registry replicate # copy it
roksbnkctl registry verify # confirm every artifact is present + digest-matched
A successful run ends on two lines worth putting on screen:
✓ mirrored 89 artifacts into artifactory.example.com/bnk-mirror
✓ all 89 BOM artifacts present + digest-matched in the mirror
Gate on verify, not on replicate or diff. replicate reports what it
pushed. verify independently re-reads the bill of materials and checks each
artifact against the digest it should have, so a partial copy or a tag that moved
underneath you is caught.
diffcan be wrong when the recorded mirror is stale. A workspace records what it last replicated, and that record can describe a registry that has since been rebuilt or replaced. In that statediffhas been observed reporting “mirror is in sync” against an empty registry, whileverifycorrectly reported every artifact missing. If a workspace has been re-pointed at a different registry, trustverify.
Configuration and environment equivalents
Every field is config.yaml state:
registry:
target: generic
generic_host: artifactory.example.com
generic_repo_prefix: bnk-mirror
generic_username: admin
generic_password_b64: <base64 of the access token>
generic_ca_b64: <base64 of the CA PEM> # only for a private CA
generic_password_b64 is obfuscation, not encryption — it exists so the
value survives YAML and does not trip the plaintext-secret check. Treat the file
as a secret: chmod 600, never commit it.
For CI, each has an environment override, so no config.yaml need be authored at
all:
| Variable | Field |
|---|---|
ROKSBNKCTL_GENERIC_HOST | registry.generic_host |
ROKSBNKCTL_GENERIC_REPO_PREFIX | registry.generic_repo_prefix |
ROKSBNKCTL_GENERIC_USERNAME | registry.generic_username |
ROKSBNKCTL_GENERIC_PASSWORD | registry.generic_password_b64 (raw; encoded for you) |
ROKSBNKCTL_GENERIC_CA_B64 | registry.generic_ca_b64 (already base64) |
ROKSBNKCTL_GENERIC_CA_SHA256 | registry.generic_ca_sha256 |
Three more are needed for an unattended run, and are easy to miss because
init --non-interactive refuses without them rather than defaulting:
ROKSBNKCTL_REGION, ROKSBNKCTL_RESOURCE_GROUP and ROKSBNKCTL_PREFIX.
The FAR pull credential is resolved from Cloud Object Storage, which needs
ROKSBNKCTL_COS_INSTANCE, ROKSBNKCTL_COS_BUCKET and ROKSBNKCTL_COS_REGION.
The bucket has no usable default — COS names are globally unique, so every
account’s is suffixed (bnk-artifacts-<hex>), and the built-in default belongs
to someone else. A wrong bucket fails with AccessDenied rather than
NotFound, which reads like a bad API key and sends you to check the wrong
credential.
The same mirror as an Argo workflow
Everything above is the interactive path: an operator with a workspace, testing a
mirror by hand. Unattended, the same binary runs the same verbs in a container
with no config.yaml anywhere — every setting arrives as an environment
variable, because a CI runner has no shell, no prompts and nowhere to stage a
file.
The credentials go in a Secret. Keep the token off the command line here too:
printf '%s' "$ARTIFACTORY_TOKEN" | kubectl -n bnk-ci create secret generic artifactory-mirror \
--from-literal=host=artifactory.example.com \
--from-literal=repo=bnk-mirror \
--from-literal=username=admin \
--from-file=password=/dev/stdin \
--from-literal=ibmcloud-api-key="$IBMCLOUD_API_KEY"
The workflow runs the five verbs as pipeline steps and gates on verify:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: bnk-artifactory-mirror-
namespace: bnk-ci
spec:
entrypoint: mirror
serviceAccountName: bnk-runner
arguments:
parameters:
- {name: runner-image, value: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.50.0}
- {name: bnk-version, value: 2.3.0-3.2598.3-0.0.170}
- {name: cos-instance, value: bnk-supply-chain}
- {name: cos-bucket, value: ""} # account-specific; no default
- {name: cos-region, value: us-south}
- {name: region, value: us-east}
- {name: resource-group, value: default}
- {name: prefix, value: art-mirror}
# No onExit teardown: a failed replicate should leave the workspace on the PVC
# so the next run RESUMES. replicate is idempotent — an artifact already
# present at the right digest is skipped — so a re-run costs only what did not
# land, not the whole mirror again.
volumes:
- name: work
persistentVolumeClaim: {claimName: bnk-work}
templates:
- name: mirror
steps:
- - {name: init, template: rbk, arguments: {parameters: [{name: cmd, value: "init -w art --non-interactive --override-from-env"}]}}
- - {name: show-target, template: rbk, arguments: {parameters: [{name: cmd, value: "-w art registry target"}]}}
- - {name: registry-bom, template: rbk, arguments: {parameters: [{name: cmd, value: "-w art registry bom"}]}}
- - {name: registry-replicate, template: rbk, arguments: {parameters: [{name: cmd, value: "-w art registry replicate --target generic"}]}}
- - {name: registry-verify, template: rbk, arguments: {parameters: [{name: cmd, value: "-w art registry verify"}]}}
- name: rbk
inputs:
parameters: [{name: cmd}]
container:
image: "{{workflow.parameters.runner-image}}"
command: [sh, -ec]
args: ["roksbnkctl {{inputs.parameters.cmd}}"]
workingDir: /work
env:
- {name: ROKSBNKCTL_REGION, value: "{{workflow.parameters.region}}"}
- {name: ROKSBNKCTL_RESOURCE_GROUP, value: "{{workflow.parameters.resource-group}}"}
- {name: ROKSBNKCTL_PREFIX, value: "{{workflow.parameters.prefix}}"}
- {name: ROKSBNKCTL_CLUSTER_CREATE, value: "false"}
- {name: ROKSBNKCTL_CLUSTER_NAME, value: "none"}
- {name: ROKSBNKCTL_REGISTRY_TARGET, value: "generic"}
- {name: ROKSBNKCTL_MANIFEST_VERSION, value: "{{workflow.parameters.bnk-version}}"}
- {name: ROKSBNKCTL_COS_INSTANCE, value: "{{workflow.parameters.cos-instance}}"}
- {name: ROKSBNKCTL_COS_BUCKET, value: "{{workflow.parameters.cos-bucket}}"}
- {name: ROKSBNKCTL_COS_REGION, value: "{{workflow.parameters.cos-region}}"}
- name: ROKSBNKCTL_GENERIC_HOST
valueFrom: {secretKeyRef: {name: artifactory-mirror, key: host}}
- name: ROKSBNKCTL_GENERIC_REPO_PREFIX
valueFrom: {secretKeyRef: {name: artifactory-mirror, key: repo}}
- name: ROKSBNKCTL_GENERIC_USERNAME
valueFrom: {secretKeyRef: {name: artifactory-mirror, key: username}}
# The RAW token; roksbnkctl base64s it into generic_password_b64 itself.
- name: ROKSBNKCTL_GENERIC_PASSWORD
valueFrom: {secretKeyRef: {name: artifactory-mirror, key: password}}
- name: IBMCLOUD_API_KEY
valueFrom: {secretKeyRef: {name: artifactory-mirror, key: ibmcloud-api-key, optional: true}}
# The workspace lives on the PVC so a later run — or registry delete —
# sees the same mirror record.
- {name: ROKSBNKCTL_HOME, value: "/work/.roksbnkctl"}
- {name: HOME, value: "/home/runner"}
volumeMounts:
- {name: work, mountPath: /work}
argo submit -n bnk-ci --wait wf-artifactory-mirror.yaml -p cos-bucket=bnk-artifacts-<hex>
It ends on the same two lines the CLI does, from a step that ran in a container that never saw a config file:
✓ mirrored 89 artifacts into artifactory.example.com/bnk-mirror
✓ all 89 BOM artifacts present + digest-matched in the mirror
Three variables are easy to miss and stop the run before any registry work
begins. init --non-interactive requires ROKSBNKCTL_REGION,
ROKSBNKCTL_RESOURCE_GROUP and ROKSBNKCTL_PREFIX, and refuses rather than
defaulting — the error names them, which is the fastest way to diagnose it. The
three ROKSBNKCTL_COS_* variables are needed for registry bom to resolve the
FAR credential, and the bucket has no usable default.
A workflow with the same shape ships as
scripts/demos/artifactory-mirror-demo/wf-artifactory-mirror.yaml.
When it goes wrong
UNAUTHORIZED on push, docker login works by hand. The account can read
but not deploy. Docker surfaces this at the first blob upload, well after the
login that is easy to mistake for proof of access.
404 or repository not found. Usually the repository-path/subdomain
mismatch above. Confirm which method the instance uses before assuming the prefix
is wrong.
x509: certificate signed by unknown authority. generic_ca is unset or
does not chain to the certificate actually being served — which, on a
load-balanced instance, may not be the one issued to the origin.
Storage. The BOM is 89 artifacts of container images and OCI-packaged charts.
An instance with a repository quota, or a filestore near capacity, fails
partway through replicate rather than cleanly. Re-run diff to see what
landed, then replicate again; it copies only what is missing.
no mirror recorded — nothing to delete. registry delete works from the
workspace’s recorded mirror, so it cannot clean a registry it has no record of —
after the record has been cleared, or from a different workspace. Remove the
artifacts through Artifactory instead.
C — Single-NIC sizing
Sizing figures here come from F5’s BIG-IP Next for Kubernetes 2.3.1 on Red Hat
OpenShift on IBM Cloud — Single-NIC Sizing Guide, which derives them from
reference testing of BNK 2.3.1/2.3.2 on ROKS. They are reproduced rather than
computed: roksbnkctl does not model throughput, and nothing here is our
measurement.
Only Small and Medium are ever needed
The CNEInstance’s deploymentSize accepts Tiny, Small, Medium, Large and
Max. The reference designs use only Small and Medium. That is the single
most useful thing to know about sizing, and it is easy to miss, because F5’s
guide names three cluster sizes — Small, Medium and Large — and those names do
not mean the same thing as deploymentSize.
The Large cluster runs the Medium BNK profile. It reaches ~31 Gbit/s not
by asking BNK for a bigger profile, but by putting nine TMM pods on nine larger
nodes, three per availability zone. Capacity comes from pod count and node size,
not from a larger deploymentSize. So if you find yourself reaching for
deploymentSize: Large, the guide’s answer is almost certainly more nodes
instead.
| Small | Medium | Large | |
|---|---|---|---|
| Worker nodes | 6 (2 per AZ) | 6 (2 per AZ) | 9 (3 per AZ) |
| Reference-tested flavour | cx3d.8x20 | cx2.16x32 | cx2.48x96 |
| Recommended flavour | bx2.8x32 | cx2.16x32 | cx2.48x96 |
deploymentSize | Small | Medium | Medium |
| TMM pods | 3 (1 per AZ) | 3 (1 per AZ) | 9 (3 per AZ) |
| CPU free for applications | 38.8 % | 60 % | 82.9 % |
| L4 ingress per AZ | ~2.6 Gbit/s | ~3.5 Gbit/s | ~10.4 Gbit/s |
| L4 ingress, cluster | ~7.8 Gbit/s | ~10.4 Gbit/s | ~31 Gbit/s |
Two footnotes on the flavours. For Small, the guide recommends bx2.8x32 over
the reference-tested cx3d.8x20: the tested flavour works but leaves only ~24
GiB cluster-wide for applications, and the balanced variant buys four times that
for a modest price step.
At the other end, the guide reports cx2.8x16 as leaving 0.1 % memory free
and not holding the platform. roksbnkctl does not refuse it — you can set
it, and the tool will build it. It is recorded here so the number is visible
before you choose, not to make the decision for you. A per-node DaemonSet floor
of ~2.9 vCPU (OpenShift/Calico, IBM observability, BNK node agents) consumes 36 %
of an 8-vCPU node before anything of yours is scheduled, which is what makes the
16 GiB variant tight and why the small flavours generally trade application CPU
for cost.
All of this is single-NIC. Multi-NIC is expected to change the picture, because per-pod throughput is bounded by TMM thread count rather than by the worker node’s NIC — so higher-performance nodes become worth adding in a way they are not here. Treat this appendix as describing the single-NIC case only, and do not extrapolate it.
Reference deployments
The figures above are F5’s. This section is ours: one clean deployment per sizing, on ROKS, recorded so a reader can tell what has actually been run from what has only been reproduced from a guide.
Every run below is a fresh install onto a fresh cluster, pinned to the manifest
the BNK 2.4 IBM install guide documents — 2.4.0-EA, pulled from
repo.f5.com — with containerPlatform: IBM and one namespace for every
component.
One trap worth naming: BNK 2.4 is Early Access, and the ordinary production FAR token cannot see it. Both keys live in the same production project, so this is a GA-versus-non-GA content split rather than a prod-versus-test one:
| FAR key | can pull |
|---|---|
cne-gar-pull (standard production) | 2.3.0 only — no 2.4 version resolves |
non-ga-prod-pull (non-GA production grant) | 2.4.0-EA |
2.4.0, 2.4.0-GA and 2.4.1 do not resolve with either key, so until 2.4
goes GA the non-GA production grant is required. The failure mode is
misleading: the wrong key authenticates happily and then reports
f5-bigip-k8s-manifest:2.4.0-EA: not found, which reads like a wrong version
rather than a wrong credential.
When 2.4 GAs this becomes a two-line change — bnk.manifest_version and the
key file. Nothing about the configuration shape depends on it. A run counts as verified only if all of the
following hold at once:
- exactly one
f5-*namespace exists, and nof5-utils - the
CNEManifestversion matches the reference build exactly - every component CR reports
Available=True—CNEInstance,F5Tmm,CSRC,Cwc,DSSM,Coremond,Observer,RabbitMQ - CSRC pods are Running and the
macvlan-internalNetworkAttachmentDefinition exists — CSRC creates it at runtime, so its absence is the visible symptom of a wrongcontainerPlatform - the licence reports
Active - no pod anywhere is outside
Running/Completed
scripts/sizing/sizing-matrix.sh builds and checks one sizing per invocation.
| Baseline | Small cluster | Medium cluster | Large cluster | |
|---|---|---|---|---|
| Worker nodes | 3 (1 per AZ) | 6 (2 per AZ) | 6 (2 per AZ) | 9 (3 per AZ) |
| Flavour | bx2.16x64 | bx2.8x32 | cx2.16x32 | cx2.48x96 |
deploymentSize | Tiny | Tiny | Tiny | Tiny |
tmmReplicas | 1 | 3 | 3 | 9 |
| Verified | yes | yes | yes | yes |
All four are verified: every component CR reporting Available=True, every TMM
pod 5/5 Running, macvlan-internal present, licence Active, and nothing
outside Running/Completed. On the Large cluster the nine TMM pods land on
nine distinct nodes, three per availability zone, which is the anti-affinity and
topology-spread arrangement F5’s reference configuration specifies.
Note the third row. deploymentSize is Tiny everywhere — the column
headings are cluster sizes, and the thing that changes between them is the
node flavour and tmmReplicas, not the BNK profile. The “Baseline” column is
the shape the BNK 2.4 IBM install guide itself describes (a 3-node 16-vCPU
cluster); the other three are F5’s sizing-guide cluster shapes.
The configuration for each
Common to all four. manifest_version must be set explicitly — leaving it out
installs the 2.3 line, and the first symptom is a CNEInstance validation
error minutes into the apply complaining that deploymentSize: Tiny is
unsupported, because Tiny does not exist on 2.3.
bnk:
manifest_version: 2.4.0-EA
far_auth_local_file: /path/to/non-ga-prod-pull-key.tgz
subscription_jwt_local_file: /path/to/subscription.jwt
flo_namespace: f5-bnk
flo_utils_namespace: f5-bnk # same name = one namespace for everything
cneinstance_size: Tiny # every size below; see the section above
far_auth_local_file and subscription_jwt_local_file are a pair — the no-COS
path is gated on both being set. With only one, the tool silently falls back
to the COS supply chain and the GA key, which fails much later with a 403 naming
a GCP project rather than a config field.
Then the part that differs. Only two settings change between sizes, plus the node flavour:
Baseline — 3 nodes, the shape the BNK 2.4 IBM install guide describes:
cluster: {workers_per_zone: 1, worker_flavor: bx2.16x64}
bnk: {tmm_replicas: 1}
Small cluster — 6 nodes, ~7.8 Gbit/s L4 ingress:
cluster: {workers_per_zone: 2, worker_flavor: bx2.8x32}
bnk: {tmm_replicas: 3}
Medium cluster — 6 nodes, ~10.4 Gbit/s:
cluster: {workers_per_zone: 2, worker_flavor: cx2.16x32}
bnk: {tmm_replicas: 3}
Large cluster — 9 nodes, ~31 Gbit/s:
cluster: {workers_per_zone: 3, worker_flavor: cx2.48x96}
bnk: {tmm_replicas: 9}
As environment overrides
Every field above has a ROKSBNKCTL_* override, so a blueprint or CI job can
select a sizing without editing config.yaml. init --non-interactive --override-from-env builds the workspace from these alone:
# common
export ROKSBNKCTL_MANIFEST_VERSION=2.4.0-EA
export ROKSBNKCTL_FAR_AUTH_LOCAL_FILE=/path/to/non-ga-prod-pull-key.tgz
export ROKSBNKCTL_SUBSCRIPTION_JWT_LOCAL_FILE=/path/to/subscription.jwt
export ROKSBNKCTL_FLO_NAMESPACE=f5-bnk
export ROKSBNKCTL_FLO_UTILS_NAMESPACE=f5-bnk
export ROKSBNKCTL_CNEINSTANCE_SIZE=Tiny
# per sizing — baseline / small / medium / large
export ROKSBNKCTL_WORKERS_PER_ZONE=1 ROKSBNKCTL_WORKER_FLAVOR=bx2.16x64 ROKSBNKCTL_TMM_REPLICAS=1
export ROKSBNKCTL_WORKERS_PER_ZONE=2 ROKSBNKCTL_WORKER_FLAVOR=bx2.8x32 ROKSBNKCTL_TMM_REPLICAS=3
export ROKSBNKCTL_WORKERS_PER_ZONE=2 ROKSBNKCTL_WORKER_FLAVOR=cx2.16x32 ROKSBNKCTL_TMM_REPLICAS=3
export ROKSBNKCTL_WORKERS_PER_ZONE=3 ROKSBNKCTL_WORKER_FLAVOR=cx2.48x96 ROKSBNKCTL_TMM_REPLICAS=9
ROKSBNKCTL_CNEINSTANCE_SIZE is listed for completeness rather than necessity:
leaving it unset reaches Tiny on the 2.4 line anyway, and unset is what lets
the line default apply. Set it when you want the choice recorded explicitly.
If the cluster shares a transit gateway, ROKSBNKCTL_CLUSTER_VPC_CIDR also
matters — two VPCs with overlapping prefixes cannot both be routed, and the
gateway blackholes one silently.
deploymentSize stays Tiny; capacity comes from replicas and node size
This is the single most important thing to know about running BNK 2.4 on ROKS,
and it is easy to get wrong because F5’s sizing guide names its cluster sizes
Small, Medium and Large — the same words deploymentSize uses.
They are not the same setting. The BNK 2.4 IBM install guide uses
deploymentSize: "Tiny" with tmmReplicas: 3, and that is what every cluster
size below runs. Capacity comes from how many TMM pods and how big the nodes
are, never from a larger deploymentSize — which is exactly the point this
appendix already makes about the Large cluster running the Medium profile.
deploymentSize above Tiny cannot run on ROKS at all, and the reason is worth
recording because the failure is silent. Every size above Tiny requests
hugepages, and the amount grows with the profile. Read from the controller-derived F5Tmm CR on a live 2.4 cluster by
changing only deploymentSize:
deploymentSize | TMM cpu | TMM memory | hugepages-2Mi |
|---|---|---|---|
Tiny | — | — | none |
Small | 1 | 1536Mi | 4 GiB |
Medium | 4 | 2Gi | 8 GiB |
The Large cluster runs the Medium profile, as the table above this section records, so it derives the same 8 GiB and is not a separate case.
TMM validates hugepages against the 2 MB page cgroup limit when it starts, and exits if that limit is zero:
<get_mem_info> ERROR: No memory available based on 2MB page cgroup limit
<init_memory> ERROR: Could not validate sufficient hugepages
<main> ERROR: Failed to create cmdline args
It exits code 0, so this does not look like a crash — the pod reports
4/5 Running and the Deployment simply never becomes Available.
A ROKS cluster cannot allocate hugepages. Six approaches were measured on live 4.21 clusters and every one fails:
| approach | result |
|---|---|
bnk.hugepages (Node Tuning Operator, [bootloader]) | needs the Machine Config Operator; ROKS has 0 MachineConfigPools |
Tuned with [sysctl] vm.nr_hugepages | ROKS deletes the CR — created object returned, next read NotFound |
privileged pod writing /proc/sys/vm/nr_hugepages | allocates (HugePages_Total: 2048) but kubelet still advertises allocatable: 0 |
advanced.tmm.resources with hugepages-2Mi: 0 | TMM schedules, then refuses to start (above) |
advanced.tmm.env.TMM_MEM | accepted, but hugepages are still required |
ibmcloud ks worker-pool | no kernel-argument or sysctl option exists |
Note that bnk.hugepages is therefore a no-op on ROKS, even though it is
what the scheduling failure recommends. Setting it applies a Tuned CR that
ROKS removes, which surfaces as terraform reporting a provider bug.
F5’s 2.3.1 sizing guide describes the reference-tested Small profile as “TMM: 1 thread, 1 vCPU / 1.5 GiB, hugepages disabled”, and its TMM resource table lists CPU and memory only. That configuration cannot be reproduced on the 2.4 build: Small derives a hugepages request, and TMM will not start without one. Whether the 2.3.1 profile still exists in 2.4 is an open question with F5 (issue #203).
Until it is answered, treat the Small, Medium and Large rows above as F5’s
published figures rather than as configurations this tool can stand up on ROKS.
Tiny is the only verified one, which is consistent with F5’s own reference
cluster running Tiny.
Tiny is not in F5’s sizing guide — it is the profile the engineering reference
cluster itself runs, recorded here as the smallest configuration known to work
end to end, not as a recommendation for production traffic.
Two things worth knowing before you reproduce these
Set bnk.manifest_version explicitly. A workspace that leaves it unset
installs the 2.3 line, and the first sign of it is a CNEInstance
validation error several minutes into the apply complaining that
deploymentSize: Tiny is unsupported — because Tiny does not exist on 2.3.
The message names the size, not the version, which sends you looking in the
wrong place entirely.
bnk down does not leave a clean cluster. It returns success while leaving
the namespace and all of BNK’s CRDs installed. That is harmless when
reinstalling the same release line, and actively misleading when changing
manifest versions, because the previous line’s CRD schema stays in force. To
reinstall from genuinely clean state, delete the BNK namespace and the
*.k8s.f5.com CRDs after bnk down and before the next bnk up.
What this looks like for a real deployment
The sizing above assumes you are building the cluster. Most deployments are not
like that. The common shape is that a platform team already has the VPC, the
transit gateway and the ROKS cluster, built to their own standards, and wants
roksbnkctl to install BNK onto it — cluster register rather than cluster up.
In that case the table is a specification to check your existing cluster
against, not a thing to provision. Worth confirming before installing: node
count and distribution across three AZs, the flavour’s memory variant against the
“free for applications” column, and that the cluster is not on cx2.8x16.
Those clusters are also usually disconnected: no route to repo.f5.com, so
BNK images come from a mirrored registry and licensing goes through an FLP rather
than direct to F5. That combination — adopted cluster, adopted transit gateway,
mirrored registry, FLP licensing — is the path most installations will actually
take, and it is the one worth exercising hardest.
Coverage note. The five-run validation behind the 2.4 support claim used a cluster this tool created, with connected licensing. The adopted-cluster, disconnected, mirrored-registry, FLP path is supported and has its own demos, but it has not had the same repeated-run treatment. Read the 2.4 support status with that distinction in mind rather than assuming it covers the shape you are most likely to deploy.