Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Preface

Foreword

Standing up F5 BIG-IP Next for Kubernetes (BNK) on IBM Cloud Red Hat OpenShift (ROKS) used to be a multi-step deployment that hit a different surface at every step. A terraform init/plan/apply against an HCL tree somebody handed you. A manual ibmcloud ks cluster config to pull a kubeconfig. A separate IBM Cloud CLI install with its own apt-source dance. A manual oc adm policy add-scc-to-user privileged to let iperf3 actually run. SSH, environment variables, kubeconfig — each a small thing, and together half a day before BNK was even on the cluster.

roksbnkctl collapses that into a single static binary plus four interchangeable execution backends (local, docker, k8s, ssh:<target>) plus an opt-in in-cluster ops pod. One command brings a workspace up; one command tears it down; the connectivity, DNS, and throughput tests run from whichever network vantage the question actually requires. The tool exists because the manual path has too many moving parts for somebody who just wants to evaluate BNK or run a customer demo.

This book is the user-facing documentation for roksbnkctl, and ships with the binary.

Who this book is for

Four audiences:

  • BNK evaluators kicking the tires on F5 BIG-IP Next for Kubernetes who want a low-friction path to a working trial deployment.
  • F5 sales engineers (SEs) who need a repeatable demo and proof-of-concept toolchain for customer engagements.
  • Customer engineers standing up BNK in their own IBM Cloud account, either for evaluation or as the foundation of a production rollout.
  • Contributors who want to extend roksbnkctl — add a backend, add a test suite, ship a new chapter. See Part IX — Contributing.

How to read this book

The book is organised so it can be read either way.

  • Linear: Parts I-VII walk from concepts through your first deployment, day-2 operations, and the built-in test suite. New users should read in order.
  • Reference: Part VIII (Command reference, Configuration reference, Terraform variable reference, Glossary) is exhaustive and indexed for lookups during day-to-day work. Search at the top of every page also reaches into reference material.

If you have 30 minutes and an IBM Cloud account, skip straight to Chapter 7 — Quick start. It’s the canonical “first cluster up” walkthrough and the rest of the book makes more sense after you’ve seen the happy path end-to-end.

Part IX is for contributors who want to build roksbnkctl from source or extend it.

Prerequisites

This book assumes:

  • Basic familiarity with IBM Cloud — you have an account, you know what an API key is, and you’ve used the IBM Cloud console at least once.
  • Basic familiarity with Kubernetes — you know what a pod, service, and namespace are; you’ve run kubectl (or oc) before.
  • A working terminal on Linux or macOS. Windows is supported for roksbnkctl itself, with documented limitations around interactive SSH (see Chapter 16).

You do not need prior experience with:

  • Terraform — roksbnkctl embeds a vetted HCL tree and drives terraform for you. You can ignore the underlying HCL until you want to customize it (Chapter 13).
  • OpenShift specifics — the tool treats ROKS as Kubernetes with a thin SCC + project overlay; the few OpenShift-specific gotchas are called out in Chapter 22 and Chapter 26.
  • F5 BIG-IP Next — BNK is the thing the book deploys; you don’t need to be a Big-IP engineer to evaluate it. Chapter 1 is the 5-minute “what is this product” primer.

Book conventions

  • Code blocks: shell commands use bash syntax highlighting; YAML snippets use yaml; HCL fragments use hcl; sample command output is shown in plain text blocks to distinguish output from input.
  • Cross-references: every chapter ends with a “Cross-references” section linking related chapters. Inline links use the form [Chapter 7 — Quick start](./07-quick-start.md) — a chapter number, an em-dash, the chapter title, and the relative path to the chapter source.
  • PRD links: design documents under docs/prd/ are linked as full GitHub URLs (e.g. https://github.com/jgruberf5/roksbnkctl/blob/main/docs/prd/03-EXECUTION-BACKENDS.md) so they resolve from the published book at GitHub Pages. The PRDs are the design surface; the book is the user surface — read PRDs only if you’re contributing or want the why behind a design call.
  • Forward references to post-v1.0 work: where a feature is explicitly queued for a v1.x release (e.g. terraform --backend k8s, multi-hop SSH ProxyJump), the prose flags it in future tense and points at docs/PLAN.md §“What’s deliberately deferred to post-v1.0” for the roadmap.

Welcome.

What is BIG-IP Next for Kubernetes (BNK)

F5 BIG-IP Next for Kubernetes (BNK) is F5’s containerised, Kubernetes-native re-imagining of the BIG-IP data plane. It runs the BIG-IP Traffic Management Microkernel (TMM) as pods inside a cluster, and exposes its configuration surface through Custom Resources rather than the classic TMSH / iControl REST APIs. The point of BNK is to give Kubernetes workloads the L4 and L7 traffic management features F5 customers already rely on — advanced load balancing, TLS termination, WAF policy, GSLB — without bolting an external appliance onto the cluster’s edge.

This chapter sets the context for the rest of the book. If you already deploy and operate BNK day-to-day you can skim it; if you arrived here knowing generic Kubernetes but new to F5’s product family, read it first.

Where BNK fits in F5’s product family

F5 has historically delivered traffic management as the BIG-IP appliance: a hardened Linux box (physical or virtual) running TMOS, with TMM as the data-plane kernel module. BIG-IP works extremely well at the cluster edge — north-south traffic — but it sits outside the cluster and is configured through its own control surface.

The next-generation lineage is BIG-IP Next: the same TMM data plane refactored to run as a regular Linux process, configurable through declarative APIs instead of imperative TMSH. BIG-IP Next ships in three deployment shapes:

  • BIG-IP Next for VMs / Bare Metal — same form factor as classic BIG-IP, modernised control plane.
  • BIG-IP Next Service Proxy for Kubernetes (SPK) — telco-focused, for 5G core workloads.
  • BIG-IP Next for Kubernetes (BNK) — general-purpose, runs inside any conformant Kubernetes cluster.

BNK is the focus of this book. It is the option you pick when your workloads already live in Kubernetes and you want F5 traffic management without standing up a separate appliance fleet.

What problems BNK solves

A standard Kubernetes cluster ships with a basic Service / Ingress story: kube-proxy iptables rules, a community ingress controller, maybe an external load balancer in front. That covers the common case but falls short when you need:

  • Real L7 traffic management for north-south traffic — fine-grained routing, header manipulation, TLS termination with custom cipher suites, mTLS enforcement, advanced HTTP/2 + HTTP/3 handling, WAF policy enforcement at the edge.
  • East-west service mesh-style features without a sidecar — connection pooling, circuit breaking, retries, observability for pod-to-pod traffic, applied at a per-namespace or per-workload granularity.
  • GSLB-style global traffic management — health-checked DNS responses that send a client to the nearest healthy cluster, integrated with the cluster’s own service health.
  • Compliance and regulated workloads — DDoS mitigation, behavioural anomaly detection, audit logging that an enterprise security team will accept.

BNK delivers all of the above as cluster-native primitives. You install it once, and from then on you express traffic management intent through CRDs (F5BigIpCtx, F5IngressTls, F5GslbPool, etc.) committed alongside your application manifests.

The components

BNK isn’t a single binary; it’s a set of cooperating components installed into a cluster. The pieces you’ll see most often:

  • TMM (Traffic Management Microkernel) — the data plane. Runs as DaemonSet pods on dedicated worker nodes. Every packet handled by BNK passes through TMM.
  • FLO (F5 Lifecycle Operator) — the control-plane operator. Watches BNK CRDs and reconciles them into TMM data-plane configuration. Owns the lifecycle of the TMM pods themselves: image pulls, version upgrades, rolling restarts.
  • CIS (Container Ingress Services) — Kubernetes-native ingress controller piece. Watches Ingress and BNK ingress CRDs, programmes TMM to terminate the corresponding traffic.
  • CNE Instance — Cloud-Native Edge configuration, the umbrella resource that ties a BNK install to its tenant context.
  • Cert-Manager — not strictly an F5 component, but a hard dependency. BNK uses cert-manager to mint and rotate the certificates TMM presents to clients.

Deeper chapters reference these names; you don’t need to memorise them now. The thing to take away is that BNK is an operator + DaemonSet pattern: the operator (FLO) reconciles your declarative intent into running data-plane pods (TMM).

Where BNK runs

BNK runs on a conformant Kubernetes cluster. F5 publishes a support matrix — read it for definitive answers — but in practice you’ll see BNK deployed on:

  • Managed Kubernetes: ROKS (IBM Cloud’s managed OpenShift), OpenShift Dedicated, EKS, AKS, GKE.
  • Self-managed OpenShift on bare metal or VMs.
  • Upstream Kubernetes in private clouds, with an LB provider that BNK can integrate with.

This book targets ROKS specifically. The next chapter explains why. The decisions and patterns documented here will translate to other Kubernetes flavours, but the bundled Terraform that roksbnkctl ships only knows how to provision ROKS.

North-south and east-west, in one install

It’s worth calling out explicitly: BNK is not “just an ingress controller” and it’s not “just a service mesh data plane”. It’s both, in the same install:

  • North-south (client outside the cluster talking to a workload inside) — BNK fronts a LoadBalancer-typed service, terminates TLS, applies WAF policy, routes to backend pods. Replaces the role a hardware BIG-IP or community ingress controller would play.
  • East-west (pod-to-pod or namespace-to-namespace inside the cluster) — BNK can be inserted into the path with no application sidecar, providing per-workload connection pooling, retries, and observability.

A single BNK install can handle both at once. Customer architectures often start with the north-south story (the obvious replacement for an existing BIG-IP appliance), then expand into east-west as the team gets comfortable with the operator-driven configuration model.

Pointer to F5’s official docs

Everything in this chapter is intentionally a sketch — enough to make the rest of this book legible. For definitive and up-to-date product information, including the full CRD reference, version compatibility matrix, sizing guidance, and license model, see F5’s official BNK documentation: https://clouddocs.f5.com/bigip-next/latest/.

The rest of this book focuses on deploying BNK with roksbnkctl and validating that the deployment works end-to-end. It does not duplicate F5’s product documentation; it complements it.

For an at-a-glance view of how roksbnkctl’s components fit together — the four execution backends, the cluster, the jumphost, the IBM Cloud control plane — see the architecture diagram at the top of Chapter 17 — Execution backends. For the happy-path lifecycle from one command to the next, see Chapter 7 — Quick start.

Why ROKS (Red Hat OpenShift on IBM Cloud)

This book and the roksbnkctl tool target ROKS — IBM Cloud’s managed Red Hat OpenShift offering — specifically. Other Kubernetes flavours can run BNK, and most of the patterns you’ll learn here translate, but the bundled Terraform that roksbnkctl ships only knows how to provision a ROKS cluster.

This chapter explains the rationale behind that choice. If you’re already using ROKS, you can skim this. If you’re evaluating whether ROKS is the right substrate for your BNK trial, read in full.

What ROKS is

ROKS is short for Red Hat OpenShift on IBM Cloud. It’s IBM Cloud’s managed-OpenShift service: you ask IBM for a cluster, IBM provisions the masters, etcd, the OpenShift control plane, and a pool of worker nodes; you get a kubeconfig and start deploying.

ROKS clusters are real OpenShift. They run the same Operator Lifecycle Manager (OLM), the same oc CLI, the same SecurityContextConstraints (SCC) model, the same routes-and-services machinery you’d find on any OpenShift install. The only thing IBM has done differently is take responsibility for keeping the control plane and the underlying infrastructure healthy.

What IBM manages, what you manage

The boundary between “IBM’s responsibility” and “your responsibility” is the principal value proposition of any managed Kubernetes service. For ROKS the line falls roughly here:

ConcernOwner
Master nodes (API server, scheduler, controllers)IBM
etcd (persistence + backups)IBM
OpenShift control plane (OLM, ingress operator, image registry)IBM
OpenShift version upgrades for the control planeIBM (you opt in to a major-version bump)
Worker node provisioning (VPC VSIs, subnets, security groups)IBM, on your behalf via the cluster API
Worker node OS patching and CVE remediationIBM
Worker pool sizing and lifecycle (workers create/delete)You
Pod workloads running on the clusterYou
Application-level RBAC, network policy, TLS, service accountsYou
BNK install + configurationYou — this is what roksbnkctl automates

The thing to internalise: with ROKS you do not rack hardware, install RHEL, run openshift-install, manage etcd backups, or chase CVE patches across a worker fleet. IBM does all of that. You start at “I have an OpenShift cluster” and go from there.

Why managed-OpenShift over self-managed for BNK evaluation

If you want to evaluate BNK quickly, the calculus is straightforward. Self-managed OpenShift is a multi-week lift before you have a cluster:

  • Provision the underlying VMs (OpenStack / vSphere / bare metal).
  • Run openshift-install and debug whatever doesn’t go right.
  • Configure DNS, load balancers, container registry mirrors.
  • Stand up monitoring + logging + cert-manager.
  • Now you can start thinking about BNK.

ROKS compresses that to one Terraform apply of ~50 minutes. You get back a kubeconfig that authenticates against a real OpenShift cluster, with cert-manager already installable via OLM, and a worker pool of the size and zone topology you specified. From there, the BNK install is the same set of CRDs and Helm charts it would be on any OpenShift cluster.

For a sales-engineering demo or a customer proof-of-concept, “I have a cluster in 50 minutes” beats “I have a cluster in 2 weeks” every time. That trade-off is the reason this book exists in this shape.

Why OpenShift (not just any Kubernetes) for BNK

BNK runs on conformant Kubernetes generally, but it integrates more cleanly with OpenShift specifically because:

  • Operator-driven install — BNK is shipped as a set of operators. OpenShift has Operator Lifecycle Manager (OLM) as a first-class citizen, so the install pattern is familiar to OpenShift admins.
  • SecurityContextConstraints (SCC) — TMM pods need elevated capabilities (notably NET_ADMIN, raw socket access, hugepages). OpenShift’s SCC model formalises that grant; on upstream Kubernetes you’d be configuring PodSecurityAdmission policies by hand.
  • Routes — OpenShift’s Route CRD predates and is more capable than Ingress. BNK can act as an alternate Route implementation, slotting into existing OpenShift application architectures without forcing teams to migrate.
  • Image streams + the internal registry — useful for the BNK supply chain (FAR images, license bundles) which can be mirrored once and consumed by many installs.

If you’re already an OpenShift shop, BNK fits naturally. If you’re not, BNK still works but you’ll need to translate this book’s OpenShift-specific examples (SCCs, oc adm policy, Route) to your platform’s equivalents.

What’s out of scope for this book

A short list of Kubernetes flavours this book does not cover:

  • EKS / AKS / GKE — BNK runs on these, but roksbnkctl up won’t provision them. You’d use cloud-specific tooling, then deploy BNK on top with the standard Helm charts F5 publishes.
  • Self-managed OpenShift on bare metal or VMs — same: no roksbnkctl up. You’d use openshift-install, then deploy BNK.
  • K3s, RKE2, microk8s — BNK’s not formally supported on these for production; useful for local dev work but outside this book’s scope.

The patterns from later chapters — workspaces, the --on flag, the connectivity / DNS / throughput tests — would still be useful on any of these, but the lifecycle commands (init, up, down, cluster register) assume ROKS.

What you need before continuing

To follow this book end-to-end you need:

  • An IBM Cloud account with billing enabled. The free tier won’t provision a worker pool; you’ll need a Pay-As-You-Go or Subscription account.
  • An IBM Cloud API key with permission to create ROKS clusters in the target account.
  • A resource group to scope cluster resources to. The default Default group works fine for a single-user evaluation; production deployments tend to use a dedicated group per environment.

The next chapters walk through installation and the quick-start path. By the end of Chapter 7 you’ll have a deployed BNK trial on a fresh ROKS cluster.

What roksbnkctl does (and doesn’t do)

roksbnkctl is a single-binary CLI for deploying and validating F5 BIG-IP Next for Kubernetes (BNK) onto IBM Cloud ROKS. It exists to compress a multi-step deployment — clone the right Terraform, configure it, run terraform, fetch a kubeconfig, install BNK, run smoke tests — into a four-command lifecycle.

This chapter is about scope. What roksbnkctl owns, what it deliberately does not own, and what’s coming in future releases. Read it before you reach for the tool to do something it isn’t trying to do.

The 4-command lifecycle

The everyday user-facing flow is four commands:

roksbnkctl init        # answer a few prompts about region, RG, cluster name
roksbnkctl up          # terraform plan + apply (~50 min for fresh ROKS + BNK)
roksbnkctl test        # connectivity + DNS + throughput against the deployment
roksbnkctl down        # tear it all back down when you're done evaluating

That’s it. From “I have an IBM Cloud API key” to “deployed BNK with a passing throughput test” with no manual terraform apply, no hand-editing kubeconfig paths, no chasing down BNK Helm charts — then a clean tear-down when you’re done so you stop paying for the cluster.

Chapter 7 walks through this end-to-end with sample output.

What roksbnkctl owns

roksbnkctl’s scope is everything between “you have an IBM Cloud API key” and “you have a working BNK install you can run tests against”. Concretely:

  • Workspace state — kubectl-style per-environment isolation under ~/.roksbnkctl/<workspace>/. Each workspace has its own config, terraform state, kubeconfig, scratch artefacts. Switch with roksbnkctl ws use <name> or override per-command with -w <name>.
  • Terraform-exec orchestration — wraps HashiCorp’s terraform-exec library to drive terraform init/plan/apply/destroy with the right state file, the right TF_DATA_DIR, the right tfvars layering. You don’t run terraform directly; roksbnkctl up does.
  • Kubeconfig fetch — after a successful up, fetches the admin kubeconfig from IBM Cloud’s container service API and writes it to ~/.kube/config at mode 0600. Retries on the 404s that happen during cluster propagation lag.
  • COS supply chain — the BNK install needs FAR images and JWT licenses staged in IBM Cloud Object Storage. roksbnkctl cos instance/bucket/object handles instance creation, bucket lifecycle, and streaming object I/O (multipart for large files) without making you pip install the IBM COS SDK separately.
  • Post-deploy validation — roksbnkctl test runs three suites: HTTP/HTTPS connectivity (built-in net/http, no external curl), DNS resolution (built-in net.Resolver, no external dig), and iperf3 throughput (deploys an iperf3 -s pod into the cluster, runs the client, parses JSON output, tears down).
  • Credentials handling — IBM Cloud API key resolution chain: env vars (IBMCLOUD_API_KEY etc.), OS keychain (macOS Keychain / libsecret / Windows Credential Manager via zalando/go-keyring), opt-in base64 in workspace config, interactive prompt as last resort. Plaintext keys in config.yaml are rejected.

If any of those words don’t make sense yet, don’t worry — later chapters cover each in depth.

What roksbnkctl does not try to do

Equally important: the explicit non-goals. roksbnkctl deliberately stays out of these spaces because well-established tools already cover them:

  • Not a generic IBM Cloud CLI. That’s ibmcloud. If you want to manage VPCs, IAM policies, classic infrastructure, Watson, or any of the hundred-plus other IBM Cloud services, use ibmcloud. roksbnkctl ibmcloud <args...> exists as a convenience passthrough that loads workspace credentials, but it doesn’t try to replace ibmcloud’s surface.
  • Not a generic Kubernetes CLI. That’s kubectl. roksbnkctl kubectl <args...> is again a passthrough that loads the workspace’s kubeconfig; it does not try to be a kubectl re-implementation. (Phase 2 internalises a small subset — roksbnkctl k get/apply/logs/exec/port-forward — so the happy path doesn’t require a host kubectl binary, but that’s targeted convenience, not replacement.)
  • Not an OpenShift admin tool. That’s oc. Same story: roksbnkctl oc <args...> passthrough, no attempt to re-implement.
  • Not a BNK runtime UI. Once BNK is deployed, you configure it through its CRDs (F5BigIpCtx, F5IngressTls, etc.). roksbnkctl doesn’t ship a TUI / web UI for editing those — it gets you to a deployed BNK and steps out of the way.
  • Not a Terraform authoring tool. The HCL lives in this repo’s terraform/ directory and is embedded into the binary at build time. roksbnkctl runs that HCL; it doesn’t help you write more of it. If you fork the HCL, point roksbnkctl at your fork via tf_source: github or tf_source: local.
  • Not an arbitrary workload deployer. BNK is the workload. The iperf3 / nginx fixtures used by roksbnkctl test exist only to validate BNK; they’re not a general-purpose deployment surface.

The principle is “do one thing well”. roksbnkctl does BNK-on-ROKS lifecycle and validation. Every other concern is delegated to the right purpose-built tool.

The relationship to bundled HCL

A core design decision worth surfacing: the Terraform that drives the deployment lives in this repo under terraform/, and is embedded into the roksbnkctl binary at build time via Go’s embed package.

This means:

  • One install gets you the CLI + a matched HCL pair. No “clone the right tag of the terraform repo separately” step.

  • Versioning is unified. A roksbnkctl v1.0 release ships with a specific snapshot of the HCL. Upgrading the binary upgrades the HCL atomically. There’s no skew between “binary version” and “Terraform version”.

  • Power users can override. The workspace config has a tf_source: block:

    tf_source:
      type: embedded     # default; uses HCL bundled into the binary
      # type: local
      # path: /path/to/your/terraform
      # type: github
      # repo: yourfork/roksbnkctl-terraform
      # ref: my-branch
    

    tf_source: local is the right setting if you’re iterating on the HCL itself. tf_source: github lets you point at a fork of the terraform repo if you’ve published one separately. The default — embedded — covers the everyday case.

Chapter 13 covers the tfvars layering rules; this is just the elevator pitch for “the HCL ships with the binary”.

What v1.0 ships and what’s queued for v1.x

This book ships with v1.0. The surface it documents:

  • kubectl internalisation — roksbnkctl k get/apply/logs/exec/port-forward is a first-class verb talking to the cluster directly via client-go. Host kubectl is informational only; the only required prereq on PATH is terraform.
  • Four execution backends — every external tool (ibmcloud, iperf3, terraform) selectable across local | docker | k8s | ssh via --backend. iperf3 runs entirely in-cluster by default; ibmcloud runs in a pinned-version Docker container if you don’t want to install it; any tool proxies through a jumphost via --backend ssh:<target>. Chapter 17 is the user-facing surface; PRD 03 is the design rationale.
  • GSLB-aware DNS testing — the DNS probe is miekg/dns-based with multi-vantage support, so you can verify that BNK’s GSLB is returning different answers from different network locations. Chapter 21 covers it.
  • Polished book — all 32 chapters, every code example verified, four Mermaid diagrams (architecture / lifecycle / GSLB cross-vantage / backend matrix), per-Part worked examples.

A handful of items are explicitly deferred to v1.x:

  • terraform over --backend k8s and --backend ssh (state-file portability design needed).
  • Multi-hop SSH ProxyJump for the --on and ssh:<target> paths.
  • Windows full TTY (interactive shell on Windows ships as line-buffered; full PTY is a v1.x item).
  • Typed OpenShift CRDs (today’s unstructured printer works; richer per-type output is queued).
  • Cross-driver cluster-sharing for e2e-test-full.sh (each driver brings up its own cluster today).

See docs/PLAN.md §“What’s deliberately deferred to post-v1.0” for the full roadmap.

Pointers to the next chapters

Installation

This chapter gets roksbnkctl onto your machine and verifies it works. Three install paths are covered: build-from-source (native Go), build-with-Docker (no host Go required), and the all-in-one container image (no install at all — run roksbnkctl and every tool it needs from one container).

Pre-built binaries are attached to every GitHub Release (Linux, macOS, Windows × amd64, arm64). The book also ships as an offline PDF (roksbnkctl-book-<tag>.pdf) on the same release page. A Homebrew tap is on the v1.x roadmap; until then macOS users grab the binary from the release page or build from source.

Quick install (one line)

The fastest path — downloads the latest release for your OS/arch and puts it on PATH:

Linux / macOS:

curl -fsSL https://raw.githubusercontent.com/jgruberf5/roksbnkctl/main/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/jgruberf5/roksbnkctl/main/install.ps1 | iex

Each script downloads the release archive, verifies its checksum, extracts the binary, runs roksbnkctl install to copy it into ~/.local/bin (Linux/macOS) or a PATH directory (Windows) — replacing any existing copy — then deletes the archive, leaving only the installed binary. Options:

  • Pin a version: set VERSION=v1.23.1 before the command (e.g. curl -fsSL … | VERSION=v1.23.1 sh, or $env:VERSION='v1.23.1' on Windows).
  • Change the install directory / pass install flags: set ROKSBNKCTL_INSTALL_ARGS="--dir $HOME/bin".

Manage the installed binary with the built-in verbs:

  • roksbnkctl version — prints the release (e.g. v1.23.1) for a release binary, or a git-derived build string for a local build.
  • roksbnkctl upgrade — on a terminal, lists the newer releases on GitHub and lets you pick one; --version vX.Y.Z pins a specific release; --yes takes the latest non-interactively.
  • roksbnkctl uninstall — removes the installed binary (the opposite of install).

The manual methods below (build-from-source, Docker build, runner container) remain available.

Choosing an install method

MethodYou getHost needsBest for
A — native build / release binarya roksbnkctl binaryGo 1.26+ (build) or nothing (release binary); the runtime tools you use (above)a developer laptop
B — Docker builda roksbnkctl binaryDocker only (for the build)a host without Go; CI that wants a binary artefact
C — runner containera container with roksbnkctl + every toolDocker/Podman + a volume for stateCI runners, fleet provisioning, air-gapped sites

Methods A and B put a binary on PATH and assume the runtime tools are installed too. Method C (Path C below) is self-contained: nothing on the host but a container runtime and a place to keep state.

Prerequisites

  • Linux, macOS, or native Windows. The full deploy path — cluster, BNK, and testing phases — runs on native Windows without WSL: roksbnkctl drives Terraform’s provisioning glue itself (no curl/tar/grep/bash on the host), and the whole up → shared-Transit-Gateway → down flow is validated on Windows PowerShell. A few interactive conveniences (the TTY-bound SSH shell, ssh-agent integration) are still not first-class on Windows; use --on/backends or a Linux/macOS host for those.
  • Git to clone the repository (only if building from source — not needed if you grab a pre-built binary).
  • Go 1.26 or newer if you want a native build. If you don’t have Go (or have an older version), use the Docker-based build or a pre-built release binary.
  • Terraform >= 1.5 on PATH at runtime — required for roksbnkctl up / plan / apply / down.
  • Helm 3 on PATH at runtime — doctor flags it as required during roksbnkctl up. The charts install via the helm_release resource (in-process via the hashicorp/helm provider — see Chapter 10 §“The terraform-native deployment model”), but FAR chart-version discovery still shells out to host helm: the data.external.versions lookup runs helm registry login + helm pull to read the FLO/CIS chart versions. So helm is genuinely required. Keep helm installed to stay doctor-green.

The remaining tools (ibmcloud, kubectl, oc, iperf3, docker) are optional and only needed for the corresponding passthrough or backend.

You do not need Docker installed to use roksbnkctl with the default local backend. Docker is required only if you opt in to --backend docker for terraform / ibmcloud. The k8s and ssh backends are alternatives that need neither host Docker nor host Go.

Installing prerequisites

Install paths per platform. terraform and helm are flagged required by doctor (helm is shelled out for FAR chart-version discovery during roksbnkctl up — keep it installed to stay green); the rest are optional, install only what you need.

macOS — Homebrew

brew install terraform               # required
brew install helm                    # flagged required by doctor — FAR chart-version discovery shells out to `helm`
brew install --cask ibmcloud-cli     # optional — only for `roksbnkctl ibmcloud …` passthrough
brew install kubectl                 # optional — only for `roksbnkctl kubectl …` passthrough (`roksbnkctl k *` is internalised)
brew install iperf3                  # optional — only for `--backend local`/`--backend ssh:<t>` throughput tests

# oc (Red Hat OpenShift CLI) — optional, only for `roksbnkctl oc …` passthrough.
# No brew formula; install via the Red Hat mirror tarball:
curl -sSL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-mac.tar.gz \
  | sudo tar -xz -C /usr/local/bin oc

If you installed ibmcloud-cli, add the plugins roksbnkctl uses:

ibmcloud plugin install kubernetes-service -f
ibmcloud plugin install cloud-object-storage -f

Linux — Ubuntu / Debian

# terraform — required
wget -qO- https://apt.releases.hashicorp.com/gpg \
  | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
  | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y terraform

# helm 3 — flagged required by doctor (FAR chart-version discovery shells out to `helm` to read the FLO/CIS chart versions)
curl https://baltocdn.com/helm/signing.asc \
  | sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] \
https://baltocdn.com/helm/stable/debian/ all main" \
  | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update && sudo apt-get install -y helm

# ibmcloud CLI + plugins — optional, for `roksbnkctl ibmcloud …` passthrough with --backend local
curl -fsSL https://clis.cloud.ibm.com/install/linux | sudo sh
ibmcloud plugin install kubernetes-service -f
ibmcloud plugin install cloud-object-storage -f

# kubectl — optional, only for `roksbnkctl kubectl <args>` passthrough (`roksbnkctl k *` is internalised and needs no host install)
sudo snap install kubectl --classic
# or via direct download:
# curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
# chmod +x kubectl && sudo mv kubectl /usr/local/bin/

# oc (Red Hat OpenShift CLI) — optional, only for `roksbnkctl oc <args>` passthrough.
# No apt package; install via the Red Hat mirror tarball:
curl -sSL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz \
  | sudo tar -xz -C /usr/local/bin oc

# iperf3 — optional, only for `--backend local` / `--backend ssh:<t>` throughput tests
sudo apt-get install -y iperf3

Instructions above target Ubuntu and Debian. For other Linux distributions (RHEL, Fedora, Arch, openSUSE, Alpine, …), a quick online search for “install terraform on <your distro>” — and the same pattern for ibmcloud, kubectl, and iperf3 — yields the equivalent commands. HashiCorp ships an RPM repo at https://rpm.releases.hashicorp.com covering RHEL/Fedora, and most distributions package kubectl and iperf3 in their official repos; the IBM Cloud CLI installer at https://clis.cloud.ibm.com/install/linux is a single curl-pipe-sh that works across distros.

Windows — Chocolatey

choco install terraform
choco install kubernetes-helm  # required — terraform local-exec provisioner shells out to `helm`
choco install ibmcloud-cli     # optional
choco install kubernetes-cli   # optional, provides kubectl
choco install openshift-cli    # optional, provides oc (Red Hat OpenShift CLI)
choco install iperf3           # optional

Or via Scoop:

scoop install terraform helm ibmcloud-cli kubernetes-cli openshift-cli iperf3

If choco/scoop don’t carry openshift-cli for your version, grab the Windows tarball from the Red Hat mirror directly:

Invoke-WebRequest -Uri https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-windows.zip -OutFile oc.zip
Expand-Archive oc.zip -DestinationPath "$env:USERPROFILE\bin\"
# then add %USERPROFILE%\bin to your PATH

After installing ibmcloud-cli, add the plugins:

ibmcloud plugin install kubernetes-service -f
ibmcloud plugin install cloud-object-storage -f

Windows TTY-bound SSH features (the roksbnkctl shell --on <target> interactive path) have known limitations on Windows; file-based SSH keys + non-interactive commands work, but ssh-agent named-pipe integration is a v1.x item. See docs/PLAN.md §“What’s deliberately deferred to post-v1.0”.

Path A — native build (requires Go 1.26+)

If go version reports 1.26 or newer, this is the simplest path:

git clone https://github.com/jgruberf5/roksbnkctl.git
cd roksbnkctl

go mod tidy                          # first time only — populates go.sum
make build                           # → bin/roksbnkctl

# Install via roksbnkctl itself (recommended — copies into ~/.local/bin):
./bin/roksbnkctl install

That’s the whole thing. The install subcommand is idempotent and copies the running binary into a directory on your PATH. Default destination is ~/.local/bin/roksbnkctl.

Make targets you’ll use most often:

make build      # go build -ldflags ... -o bin/roksbnkctl ./cmd/roksbnkctl
make test       # go test ./...
make vet        # go vet ./...
make tidy       # go mod tidy
make clean      # rm -rf bin/

If make build fails, the most likely cause is Go too old. The module declares go 1.26.0 in go.mod (forced by transitive deps from the SSH/integration test layers); older versions error out with go: module requires Go 1.26. Either upgrade Go or fall back to the Docker path below.

Path B — Docker-based build (no host Go required)

This path is ideal for sealed CI workstations, custom VM images, or anywhere installing Go on the host is awkward. The official golang:1.26-alpine image has everything needed (the minimum tracks whatever the transitive dependencies demand — currently the k8s.io/* v0.36 modules, which declare go 1.26); the build artefact lands in ./bin/ owned by your host user.

git clone https://github.com/jgruberf5/roksbnkctl.git
cd roksbnkctl

docker run --rm -v "$PWD:/work" -w /work \
  --user "$(id -u):$(id -g)" -e HOME=/tmp \
  golang:1.26-alpine sh -c 'go mod tidy && go build -o bin/roksbnkctl ./cmd/roksbnkctl'

./bin/roksbnkctl install

Anatomy of the docker invocation:

FlagWhy
-v "$PWD:/work"Bind-mount the repo into the container at /work.
-w /workContainer working directory matches the mount.
--user "$(id -u):$(id -g)"Output binary is owned by your host user, not root.
-e HOME=/tmpGo writes its module cache under $HOME; /tmp is writable by any user. Without this, go mod tidy fails on a writable-/root permission error.
golang:1.26-alpinePinned major version; matches go.mod’s minimum.

Cross-compile via Docker

Set GOOS / GOARCH env vars in the same docker run to produce binaries for other platforms:

# macOS arm64 (Apple Silicon)
docker run --rm -v "$PWD:/work" -w /work \
  --user "$(id -u):$(id -g)" -e HOME=/tmp \
  -e GOOS=darwin -e GOARCH=arm64 \
  golang:1.26-alpine sh -c 'go mod tidy && go build -o bin/roksbnkctl-darwin-arm64 ./cmd/roksbnkctl'

# Windows amd64 (compile-only; not tested at runtime)
docker run --rm -v "$PWD:/work" -w /work \
  --user "$(id -u):$(id -g)" -e HOME=/tmp \
  -e GOOS=windows -e GOARCH=amd64 \
  golang:1.26-alpine sh -c 'go mod tidy && go build -o bin/roksbnkctl.exe ./cmd/roksbnkctl'

Each binary is statically linked (Alpine + CGO_ENABLED=0 is the cross-compile default) so the produced file has no runtime library dependencies.

Path C — run from the all-in-one container image (no install)

If you’d rather not put a binary and its tools on the host at all — CI runners, fleet provisioning, air-gapped sites — use the runner image. It carries roksbnkctl plus every tool it dispatches (terraform, helm, kubectl, oc, ibmcloud, iperf3, h2load), so one docker run is a complete roksbnkctl with nothing on the host:

docker run --rm \
  -v bnk-state:/work \
  -e IBMCLOUD_API_KEY="$IBMCLOUD_API_KEY" \
  ghcr.io/jgruberf5/roksbnkctl-tools-runner:latest version

podman run works identically. The image’s ENTRYPOINT is roksbnkctl, so everything after the image name is roksbnkctl’s own arguments (version, up, test matrix --dry-run, …). Tags: :latest and a per-release :vX.Y.Z are published to GHCR on each release; :dev tracks main.

State lives in a volume, not the image

roksbnkctl keeps its workspaces — config.yaml, terraform state, kubeconfigs, keys — under ROKSBNKCTL_HOME, which the image sets to /work/.roksbnkctl. The image itself is stateless; mount durable storage at /work or every run starts empty:

  • A named volume (recommended) — -v bnk-state:/work. Docker seeds a fresh named volume from the image’s /work, which is owned by the image’s uid 1000, so it’s writable out of the box.
  • A host bind-mount — -v "$PWD/bnk-state:/work" — carries the host directory’s ownership, which may not be writable by uid 1000. Pre-create it writable (mkdir -p bnk-state && chmod 0777 bnk-state, or chown it to 1000), or add --user "$(id -u):$(id -g)". This is the usual docker bind-mount caveat, not a roksbnkctl quirk.

For a truly stateless runner — no volume at all — a COS/S3 remote state backend is the planned answer (PRD 16); until it lands, mount a volume.

Use --backend local inside the runner

Every tool roksbnkctl dispatches is already on PATH in the image, so the docker/k8s execution backends (whose whole job is to fetch a tool from elsewhere) aren’t needed here — runs are local, with no docker-in-docker. Chapter 18 — Choosing a backend covers when the other backends matter (the binary-on-host case).

What’s not in it

The runner is a use image, not a docs/dev-build image — the book toolchain (mdbook, pandoc, texlive) is deliberately excluded to keep it lean. Build the book with the separate roksbnkctl-tools-mdbook image (see Chapter 31 — Building from source).

Because it’s one self-contained artefact, the runner is also the natural thing to mirror behind a firewall for an air-gapped install — pair it with the registry mirror (Chapter 10a — Air-gapped install).

The install subcommand

roksbnkctl install [--dir PATH] [--force]

install copies the running binary into a directory on PATH. Defaults:

  • Destination: ~/.local/bin/roksbnkctl — this directory is on the default PATH for most modern Linux and macOS user environments and does not require sudo.
  • Mode: 0755.
  • Idempotent: if the running binary is already at the destination, no-op (no error).

Override the destination with --dir:

./bin/roksbnkctl install --dir ~/bin
sudo ./bin/roksbnkctl install --dir /usr/local/bin

--force overwrites an existing file at the destination. Without it, install refuses if the destination is a different binary.

If ~/.local/bin is not on your PATH, add it. On bash:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
exec $SHELL -l

On zsh, swap ~/.bashrc for ~/.zshrc.

Verifying the install

Two quick checks: version (proves the binary runs) and doctor (proves the runtime environment is set up for actual work).

roksbnkctl version

roksbnkctl version

Sample output:

roksbnkctl v1.0.0 (commit abc1234, built 2026-05-10T14:22:08Z)
Docs: https://jgruberf5.github.io/roksbnkctl/book/

The version string is populated via -ldflags at build time; make build VERSION=v1.0.0 injects an explicit tag. A bare make build produces something like dev (commit abc1234, built ...). The Docs: URL is a compile-time constant (internal/cli/meta.go::DocsURL) — every binary built from this tree points at the same book URL.

roksbnkctl doctor

roksbnkctl doctor

doctor runs the prereq + credentials report. Sample output on a healthy machine looks like this (yours will differ depending on which optional binaries you have installed and whether you’ve initialised a workspace):

✓  terraform         /usr/bin/terraform (Terraform v1.15.2)                                   (required for `roksbnkctl up`)
✓  helm              /usr/local/bin/helm (v3.20.2)                                            (required for `roksbnkctl up`; terraform `local-exec` provisioners shell out to helm)
⚠  iperf3            not on PATH                                                              (needed for `roksbnkctl test throughput`)
✓  kubectl           /usr/local/bin/kubectl (clientVersion:)                                  (optional; `roksbnkctl kubectl` passthrough)
✓  oc                /usr/local/bin/oc (Client Version: 4.21.10)                              (optional; `roksbnkctl oc` passthrough)
✓  ibmcloud          /usr/local/bin/ibmcloud (ibmcloud 2.43.0 ...)                            (optional; `roksbnkctl ibmcloud` passthrough)
✓  kubeconfig        /home/jgruber/.kube/config                                               (needed for cluster-side ops)
✓  workspace         default                                                                  (per-environment config + state)
✓  ibmcloud api key  resolved via OS keychain                                                 (auth for terraform + IBM SDK calls)
✓  ibm cloud auth    OK (account: Main F5 Account)                                            (verifies API key works against IBM IAM)

Each row is <status> <name> <detail> <why we care>. Failures are red ✗ and exit non-zero; warnings are yellow ⚠ and don’t fail the run. terraform and helm are the hard-required checks at v1.0 — the rest are either optional passthroughs or specific to test suites. Chapter 5 walks through what each check is verifying and how to fix common failures.

OS support matrix

OSNative buildDocker buildCross-compile targetRuntime status
Linux (amd64, arm64)yesyesyesfirst-class
macOS (amd64, arm64)yesyesyesfirst-class
Windows (amd64, arm64)yesyesyescompile-only; roksbnkctl shell --on and roksbnkctl exec --on jumphost PTY behaviour limited

“First-class” means the v1.0 acceptance criteria are validated on those platforms; “compile-only” means the binary builds and runs but interactive features (notably TTY-bound SSH) have known limitations and are not part of the v1.0 release gate.

The Windows limitations are tracked in PRD 01 (the SSH client design) and largely come down to golang.org/x/crypto/ssh’s incomplete PTY handling on Windows and the absence of an SSH agent named-pipe protocol. File-based SSH keys work; full PTY and ssh-agent integration on Windows are on the v1.x roadmap (see docs/PLAN.md §“What’s deliberately deferred to post-v1.0”).

Required prerequisites — terraform and helm at v1.0

The cluster lifecycle needs two binaries on PATH:

  • terraform (>= 1.5) — hard-required for any cluster lifecycle command (up, down, plan, apply).
  • helm (3.x) — flagged required by doctor during roksbnkctl up. The charts install via the helm_release resource — the hashicorp/helm provider speaks the Helm 3 protocol via an embedded Go runtime, so no host helm is shelled out for the chart installs (see Chapter 10). FAR chart-version discovery does shell out to host helm, though: the data.external.versions lookup runs helm registry login + helm pull to read the FLO/CIS chart versions, so helm is genuinely required. Keeping helm installed keeps doctor green.

Optional binaries — only needed for the corresponding passthrough or fallback path:

  • iperf3 — only needed for --backend local and --backend ssh:<target> throughput modes. The default --backend k8s runs iperf3 entirely in cluster (no host binary needed).
  • kubectl / oc — only needed for the roksbnkctl kubectl <args...> / roksbnkctl oc <args...> passthroughs. The everyday verbs (get, apply, describe, delete, logs, exec, port-forward) are internalised under roksbnkctl k and need no host binary.
  • ibmcloud — only needed for the roksbnkctl ibmcloud <args...> passthrough on --backend local. The cluster-lifecycle path uses IBM Go SDKs internally and does not shell out to ibmcloud. The docker, k8s, and ssh backends ship their own ibmcloud — no host install needed.
  • docker — only needed for --backend docker. Optional; the k8s and ssh backends are alternatives if docker isn’t available.

Run roksbnkctl doctor to see exactly what your environment is missing for the workflow you intend to run.

Updating

git pull && make build is the source-build update mechanism (or re-run the Docker build for the containerised path).

roksbnkctl self update upgrades from a tagged GitHub release. Use it once you’ve installed an initial v1.0 binary:

roksbnkctl self update
# Checks https://github.com/jgruberf5/roksbnkctl/releases/latest, downloads
# the matching asset for your OS+arch, verifies the checksum, swaps the
# binary atomically.

Next

With a working binary on PATH, Chapter 5 — Doctor explains what every doctor check is looking at, Chapter 6 — Workspaces explains the ~/.roksbnkctl/<workspace>/ layout, and Chapter 7 — Quick start walks the 4-command lifecycle end-to-end.

Doctor: checking your environment

roksbnkctl doctor is the prereq + credentials report. It runs in under five seconds, exits non-zero on any hard error, and prints a tabular report that maps one-to-one to the runtime dependencies the rest of the tool reaches for.

This chapter walks every check, explains what each row’s “why we care” blurb means, covers why kubectl and oc are informational rather than required, and describes the --target SSH probe.

What doctor checks

A bare roksbnkctl doctor runs the general checks: tooling on PATH, kubeconfig location, the resolved workspace, and the IBM Cloud authentication chain. Sample output on a healthy machine looks like this:

roksbnkctl doctor
✓  terraform         /usr/bin/terraform (Terraform v1.15.2)                                   (required for `roksbnkctl up`)
✓  helm              /usr/local/bin/helm (v3.20.2)                                            (required for `roksbnkctl up`; terraform `local-exec` shells out to helm)
⚠  iperf3            not on PATH                                                              (needed for `roksbnkctl test throughput`)
✓  kubectl           /usr/local/bin/kubectl (clientVersion:)                                  (internalised in roksbnkctl k *; passthrough still works if installed)
✓  oc                /usr/local/bin/oc (Client Version: 4.21.10)                              (internalised in roksbnkctl k *; passthrough still works if installed)
✓  ibmcloud          /usr/local/bin/ibmcloud (ibmcloud 2.43.0 ...)                            (optional; `roksbnkctl ibmcloud` passthrough)
✓  kubeconfig        /home/jgruber/.kube/config                                               (needed for cluster-side ops)
✓  workspace         default                                                                  (per-environment config + state)
✓  workspace permissions  owner-only                                                          (credentials in the workspace are protected by file mode alone)
✓  ibmcloud api key  resolved                                                                 (auth for terraform + IBM SDK calls)
✓  ibm cloud auth    OK (account: 1a2b3c..., user: you@example.com)                           (verifies API key works against IBM IAM)

Each row has the same shape:

<status> <name> <detail> <why we care>
  • status is one of ✓ (green / OK), ⚠ (yellow / warning), or ✗ (red / error). Skipped checks render as ⚠.
  • name is the dependency or capability being checked.
  • detail is the resolved value — usually a path, a version line, or an error message.
  • why we care is a parenthetical clause naming the roksbnkctl feature that depends on this row.

The BackendName column on the underlying Check struct (internal/doctor/check.go) is reserved for the per-backend probes that land in Sprint 4 (PRD 03). Until then it stays empty for every general check.

Each check explained

terraform — required

One of two hard-required binaries for the roksbnkctl up happy path. roksbnkctl shells out to terraform via terraform-exec for plan/apply/destroy; without it nothing in the cluster lifecycle works.

Pass condition: a binary on PATH, version 1.5 or newer.

Failure mode: not on PATH. Fix: install Terraform from terraform.io, or your distro’s package manager, then re-run doctor.

helm — required

The second hard-required binary (doctor flags it required). It is needed because FAR chart-version discovery — the data.external.versions lookup — runs helm registry login + helm pull to read the FLO/CIS chart versions during roksbnkctl up. Without helm on PATH, the apply fails partway through the cluster lifecycle with:

Error: local-exec provisioner error
Error running command 'helm upgrade --install cert-manager ...':
exit status 127. Output: /bin/sh: 1: helm: not found

Pass condition: a helm (v3.x) binary on PATH. Doctor parses helm version --short for the version detail.

Failure mode: not on PATH. Fix: install Helm 3 from helm.sh/docs/intro/install/, or via your distro’s package manager:

# Linux (Ubuntu/Debian — official Helm apt repo):
curl https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update && sudo apt-get install -y helm

# macOS:
brew install helm

# Windows:
choco install kubernetes-helm

The BNK charts (cert_manager / flo / cne_instance) install via the helm_release terraform resource, which uses the hashicorp/helm provider’s embedded Helm 3 runtime — no host helm is shelled out for the chart installs themselves. Doctor still flags helm required because FAR chart-version discovery shells out to host helm (helm registry login + helm pull) to read the FLO/CIS versions. See Chapter 10 §“The terraform-native deployment model”.

iperf3 — informational

Used only by roksbnkctl test throughput in its host-iperf3 modes. After Sprint 4 lands the k8s execution backend (PRD 03), iperf3 moves entirely in-cluster and this row goes away for the everyday user.

Failure mode: not on PATH. Fix: install iperf3 if you plan to use the throughput test today; otherwise ignore.

kubectl — informational

kubectl is informational. The everyday verbs (get, apply, describe, delete, logs, exec, port-forward) are native Go via client-go and live under roksbnkctl k. Missing host kubectl does not disable the happy path; it only disables the roksbnkctl kubectl <args...> passthrough.

If kubectl is on PATH, the row is still ✓ and shows the version line. If it’s missing, the row is informational, not a warning, and the detail explains where the equivalent functionality lives.

oc — informational

Same story as kubectl — the OpenShift-relevant verbs are internalised (Project, Route, and ImageStream are available via roksbnkctl k get). Host oc is preserved as an escape hatch; a missing oc is informational, not a warning.

ibmcloud — optional

Required only for the roksbnkctl ibmcloud <args...> passthrough. The cluster-lifecycle path uses IBM Go SDKs internally — roksbnkctl up does not shell out to ibmcloud — so you can skip this binary if you don’t need the passthrough.

kubeconfig

Resolves the kubeconfig path via $KUBECONFIG first, then ~/.kube/config. Cluster-side commands (status, logs, every k <verb>) need it.

roksbnkctl up writes the admin kubeconfig at ~/.kube/config (mode 0600) on a fresh apply.

If you already have a multi-cluster ~/.kube/config, do not point $KUBECONFIG at ~/.roksbnkctl/<workspace>/state/kubeconfig. That is a directory of per-phase kubeconfigs Terraform writes as a side effect of config_dir, each carrying a roughly one-hour IAM token that nothing refreshes. An hour after an apply they authenticate as nobody, and the only symptom is:

error: You must be logged in to the server (the server has asked for the client to provide credentials)

Use the workspace-scoped verbs instead — they select the kubeconfig that addresses this workspace’s cluster, and skip any whose credential has expired:

roksbnkctl -w <workspace> k get pods -A
roksbnkctl -w <workspace> kubectl get nodes

To refresh the admin kubeconfig at ~/.kube/config, run roksbnkctl kubeconfig --download. roksbnkctl doctor reports expired phase kubeconfigs under workspace phase kubeconfigs.

Failure mode: $KUBECONFIG and ~/.kube/config both missing. Fix: run roksbnkctl kubeconfig --download to fetch the admin kubeconfig from IBM Cloud.

workspace

Reports the resolved workspace name and whether its config.yaml exists.

  • ✓ default — the current workspace pointer at ~/.roksbnkctl/config.yaml resolves and the named workspace has a populated config.yaml.
  • ⚠ "default" not initialised — the directory may exist (created by roksbnkctl ws new) but config.yaml is empty. Run roksbnkctl init to populate.
  • ✗ no config context — the global config can’t be loaded at all.

The one-off -w / --workspace flag overrides which workspace doctor reports against. See Chapter 6 — Workspaces.

workspace permissions

Reports whether the workspace tree is owner-only, and tightens it if not.

config.yaml can hold ibmcloud.api_key_b64 and registry.generic_password_b64; state/terraform.tfstate holds whatever those resolve to. All of it is base64 at most, which is obfuscation and not encryption — the file mode is the only protection, so config.yaml is 0600 and the directories are 0700.

The check is read-only. Loading a workspace already tightens what it can, so anything still loose by the time doctor looks is something the repair could not fix — which is exactly what is worth reporting.

  • ✓ owner-only — nothing to do.
  • ✗ <path> readable by other users and could not be tightened (N path(s)) — the only case that needs your hands. A read-only mount, or a filesystem with no POSIX modes (a WSL DrvFs mount without metadata, an SMB share). Move the workspace onto a filesystem that can hold 0600, or use the keychain or env-var credential paths so nothing sensitive lands in the file at all.

The row is absent on Windows: Go’s Chmod there only toggles the read-only bit, so there is no owner-only mode to assert and a permanent unactionable warning would be worse than no row.

The scope is the workspace directory and its immediate children: config.yaml, registry-mirror.json, cluster-outputs.json, ssh/, and every state-* tree. Children are enumerated rather than matched against a list of known names, so a state tree added by a future phase is covered the day it appears. Owner-only on a state directory makes everything inside it unreachable regardless of those files’ own modes, which is why the check does not walk deeper.

The repair happens on load, not here, and announces itself once:

⚠ workspace "prod" was readable by other users on this host; tightened 3 path(s) to owner-only.
  config.yaml can hold your IBM Cloud API key and registry password. If this host is shared, rotate them.

Workspaces written by a release before v1.50.0 are 0644/0755 on disk and will produce that line the first time they are read. The exposure that already happened is not undone — on a shared host, rotate the API key.

ibmcloud api key

Resolves the API key via the chain documented in Chapter 14 — Credentials: env var → OS keychain → workspace config (base64) → TTY prompt.

Pass condition: the chain produces a non-empty key. The key value is never printed — only the source (“resolved”).

Failure mode: IBMCLOUD_API_KEY unset and no keychain entry for workspace "<name>". Fix: either export IBMCLOUD_API_KEY=... for the session, or re-run roksbnkctl init and accept the keychain-save prompt.

ibm cloud auth

Round-trips the resolved key against IBM IAM via the SDK (Verify() call). Confirms the key is not just present but actually authenticates.

Pass condition: IAM accepts the key; the row reports the resolved account and user identity.

Failure modes:

  • BXNIM0415E: Provided API key could not be found — the key is malformed or has been deleted in IBM Cloud.
  • network is unreachable / i/o timeout — your workstation can’t reach iam.cloud.ibm.com. Common in customer-firewall scenarios; route through a jumphost (Chapter 16) to confirm the key works from inside the customer network.

Common failures and how to fix them

The chapter readers most often land on. Each row maps a real-world symptom to its fix:

SymptomLikely causeFix
terraform not on PATHnot installedinstall Terraform >= 1.5; re-run doctor
kubeconfig: $KUBECONFIG and ~/.kube/config both missingnever ran up against this workspaceroksbnkctl kubeconfig --download or run roksbnkctl up
ibmcloud api key: ... no keychain entrynew shell, key not exportedexport IBMCLOUD_API_KEY=... or re-run roksbnkctl init
ibm cloud auth: BXNIM0415Ebad / rotated keyregenerate the key in the IBM Cloud console; update the keychain via roksbnkctl init
ibm cloud auth: i/o timeoutcorp-firewalled workstationuse --on jumphost to test from inside the customer network
workspace "foo" not initialisedws new was run but init was notrun roksbnkctl init -w foo
workspace: no config context~/.roksbnkctl/config.yaml corruptinspect the file; worst case delete it and re-run init

If a fix isn’t here, Chapter 26 — Troubleshooting covers the longer tail.

The --target <name> SSH check

The --on jumphost flag adds an optional second mode for doctor: probe an SSH target before you try to use it.

roksbnkctl doctor --target jumphost

This adds one row per resolved target:

✓  ssh:jumphost      ubuntu@169.45.91.177:22 (TOFU recorded)            (verifies the target is reachable)

The probe:

  1. Resolves the target’s host, user, port, and key source from ~/.roksbnkctl/<workspace>/config.yaml.
  2. Connects via the internal/remote SSH client.
  3. Validates the host key against ~/.roksbnkctl/known_hosts (TOFU prompt on first contact, unless --insecure-host-key).
  4. Runs a no-op command (true) to confirm the channel works end-to-end.

Failure modes specific to the SSH probe:

  • host key mismatch — the target was rebuilt; edit ~/.roksbnkctl/known_hosts to clear the entry, then re-probe.
  • unable to authenticate — the key source resolved but the remote rejected it. Check key_path / key_source in workspace config; if key_source: agent, verify ssh-add -l shows the right key.
  • dial tcp: i/o timeout — the host:port is unreachable. Verify with nc -vz <host> 22 from a known-good network.

Pass --target all to probe every target listed in the workspace’s targets: block. Useful in CI when you want a single command that asserts every entry is reachable.

Reading the exit code

doctor exits with:

  • 0 — all checks are green or warnings only. Warnings do not fail doctor. The everyday workflow can proceed.
  • non-zero — at least one row produced an ✗ error. The first error string is also written to stderr so wrapper scripts can grep it.

This is the contract scripts/e2e-test.sh and the Makefile rely on: a script that runs roksbnkctl doctor && roksbnkctl up --auto will only proceed past doctor if the environment is genuinely ready.

The “warnings don’t fail” rule is deliberate. An iperf3 not on PATH warning is informational — the everyday up / test connectivity flow doesn’t need it. Forcing exit-1 on every warning would be too aggressive for the common case.

If you want to gate scripts strictly (e.g. CI workflows that must have iperf3 installed because they run the throughput suite), parse the output rather than relying on the exit code:

if ! roksbnkctl doctor | grep -q '^✓  iperf3'; then
  echo "iperf3 missing — install it before running test throughput" >&2
  exit 1
fi

What doctor is not

A few deliberate non-features worth naming:

  • Not a fix-it tool. doctor reports; it never installs, never modifies workspace config, never calls IBM Cloud APIs that mutate state. The IAM verify call is read-only. If doctor could break things, users couldn’t run it freely — and “run doctor” needs to be a safe first move.
  • Not a backend probe. Per-backend availability checks (docker daemon reachable, k8s ops pod healthy, ssh target reachable) ship as separate BackendName-tagged rows via doctor --backend <name> (PRD 03). The --target probe was the early prototype of that pattern.
  • Not concurrent-safe. The CLI invokes doctor once per command; the side-channel for “why we care” blurbs in internal/doctor/doctor.go doesn’t synchronise. Don’t run two doctors against the same process.

Cross-references

Workspaces

A workspace is a per-environment bundle of config + state. The shape is modelled on kubectl contexts: you can have many of them, exactly one is “current” at a time, and a -w flag lets you address a specific one for a single command without flipping the pointer.

This chapter covers the on-disk layout, the everyday init / use / list flow, the full roksbnkctl workspaces command tree, the -w / --workspace override, and how creating or deleting a workspace moves the “current” pointer for you.

Workspace selection follows you automatically. Creating a workspace (init or ws new) makes it current; deleting the current workspace moves the pointer to another existing one, or clears it when none remain. There is no phantom default fallback — with nothing selected, commands say so instead of guessing.

The on-disk layout

Every workspace lives under ~/.roksbnkctl/<name>/:

~/.roksbnkctl/
  config.yaml                          # global; current_workspace pointer
  known_hosts                          # SSH host keys (shared across workspaces)
  default/                             # workspace "default" (mode 0700)
    config.yaml                        # this workspace's inputs (mode 0600)
    cluster-outputs.json               # post-apply cluster identity (when present)
    state/                             # BNK trial state (mode 0700)
      terraform.tfstate
      terraform.tfvars
      kubeconfig                       # admin kubeconfig (mode 0600)
      tf-source/                       # bundled HCL extracted to disk
      scratch/                         # docker bind-mounts, helm caches
    state-cluster/                     # cluster-phase state (separate tree)
      terraform.tfstate
      cluster-phase-override.tfvars
  prod/                                # workspace "prod"
    config.yaml
    state/
    ...

Three things are worth calling out:

  • ~/.roksbnkctl/config.yaml is global — non-secret user-wide preferences plus the current_workspace pointer. It is not a workspace config; the per-workspace files live one level deeper.
  • state/ and state-cluster/ are intentionally separate so roksbnkctl cluster up and roksbnkctl up don’t tangle their Terraform state. Most users won’t touch either directly.
  • cluster-outputs.json is the persisted identity of the workspace’s ROKS cluster — written by cluster up or cluster register, read by roksbnkctl up so BNK trials don’t have to re-state cluster identity in every tfvars. It also records the create-time settings the cluster was built with, and for those it outranks config.yaml — the record says what the cluster is, the config says what is wanted. See Chapter 8a §Which side wins.

Override the base directory with the ROKSBNKCTL_HOME env var. Test fixtures use this; everyday users shouldn’t need it.

terraform.applied.tfvars — what’s deployed right now

roksbnkctl keeps a per-phase snapshot of the effective Terraform var-file inputs that produced the workspace’s current state. After every successful terraform apply — roksbnkctl cluster up or roksbnkctl bnk up — roksbnkctl writes a canonical-HCL summary of “what var-files said” to the phase’s state directory. Re-create / audit / handoff workflows get a file-on-disk record of the inputs rather than reconstructing them from config.yaml (or memory).

Where it lives

Workspace shapePhasePath
ShapeSplit / ShapeClusterOnlyCluster phase~/.roksbnkctl/<workspace>/state-cluster/terraform.applied.tfvars
ShapeSplitTrial phase~/.roksbnkctl/<workspace>/state/terraform.applied.tfvars
ShapeLegacySingleboth phases (collapsed)~/.roksbnkctl/<workspace>/state/terraform.applied.tfvars

On ShapeLegacySingle, the file is a union of all sources (since the legacy shape doesn’t separate cluster and trial state) and the header comment records phase=legacy-single so the reader doesn’t mistake it for either a cluster-only or trial-only snapshot. See PRD 07 §“Design” for the format spec.

What it captures

A canonical HCL var-file: one assignment per line, variables sorted alphabetically within each source section. Each section is preceded by a comment line documenting which source contributed the values:

  • # === from config.yaml === — vars derived from the workspace’s config.yaml (written to terraform.tfvars on disk).
  • # === from terraform.tfvars.user === — the workspace-local user override file. If the file doesn’t exist, the section header is # === from terraform.tfvars.user (missing) === and the body is empty.
  • # === from cluster-phase override === — state-cluster/cluster-phase-override.tfvars (cluster-phase snapshots only).

Source-attribution comments matter because the same variable can appear in multiple sources; the “winner” — the value Terraform actually used — is the last section to mention it. The comments let the reader trace why a particular value ended up live.

Lifecycle

  • Written after every successful terraform apply. Plan flows don’t write the snapshot — the name terraform.applied.tfvars would mislead if a plan-time write existed.
  • Overwritten each apply. If you want history, copy the file aside before re-running up or wire restic / a git commit hook against ~/.roksbnkctl/<workspace>/.
  • Untouched by destroy. cluster down / bnk down leave the prior up’s snapshot in place; that’s what was last deployed. The file’s mtime + the absence of Terraform state is the “torn down on <date>” signal.
  • Never read by roksbnkctl itself. The snapshot is an output for the user — never an input the tool depends on. Making it an input would create a feedback loop where redacted values get written back as the literal string <redacted>.

Redaction

Every variable carrying a credential is redacted: ibmcloud_api_key, bigip_password, and registry_mirror_password. ibmcloud_api_key is the one whose value comes from the cred resolver rather than being authored by the user; the other two are authored in config.yaml (base64-obfuscated there) and would otherwise appear in this file in the clear. Since this page tells you the snapshot is suitable for committing, a rendered credential missing from that list is a credential in a repository — so adding one to the render without adding it to redactedVarNames now fails a test. See PRD 04 §“Cred tmpfile-bind-mount pattern” for why the API key isn’t in tfvars in the first place. The redacted line carries an inline comment:

ibmcloud_api_key = "<redacted>"  # source: cred resolver, not persisted

For team-handoff scenarios (a teammate receives this file out-of-band and wants to re-create the workspace): replace the <redacted> value with the teammate’s own API key, or simply remove the ibmcloud_api_key line so the cred resolver supplies it from the teammate’s own environment (keychain, shell env, ~/.bluemix/api_key, etc.) at apply time. Every other line round-trips verbatim.

The file mode is 0600 regardless. The non-redacted contents (workspace identifiers, region, resource group, cluster name, tunable values) aren’t credential-grade secrets, but aren’t world-readable-grade either. Tight permissions are the cheap default.

What it’s not

  • Not an input to subsequent applies. The -var-file chain on the next apply is unchanged: config.yaml-derived → terraform.tfvars.user → phase overrides.
  • Not a record of Terraform defaults. If variable "foo" { default = "bar" } and the user never set foo, the snapshot omits foo entirely. Capturing defaults would require running terraform output against the variables block — separate concern.
  • Not a state-derived value capture. Computed expressions, resource references, locals, and data-source values aren’t var-file inputs and don’t appear. terraform console against the live state dir is the right tool for those.
  • Not a TF_VAR_* env capture. roksbnkctl doesn’t set TF_VAR_* today — everything goes via -var-file — so the snapshot covers the complete input surface. A future cycle that starts using TF_VAR_* will need to extend this file.

Safe-to-commit guidance

The file is suitable for git commit alongside config.yaml after the user verifies the redaction matches their threat model. The standard reminder applies: the workspace dir may contain other semi-sensitive material — cluster-outputs.json records the cluster’s crn and admin identity hints; the state/ and state-cluster/ trees include terraform.tfstate (which contains resource IDs, IAM bindings, and any value Terraform’s provider exposed); the kubeconfig files are mode 0600 for a reason. Review the whole workspace dir with the same lens before committing.

roksbnkctl does not touch .gitignore. If you commit the workspace, you commit the workspace; if you don’t, you don’t. The tool stays out of that decision.

Worked example

For a ShapeSplit cluster phase apply, ~/.roksbnkctl/canada-roks/state-cluster/terraform.applied.tfvars looks like:

# Generated by roksbnkctl v1.4.0 at 2026-05-14T10:23:17Z after terraform apply on phase=cluster.
# Re-generated each apply. Do not edit by hand — your changes will be overwritten.

# === from config.yaml ===
cluster_name = "canada-roks"
ibmcloud_api_key = "<redacted>"  # source: cred resolver, not persisted
region = "ca-tor"
resource_group_name = "default"

# === from terraform.tfvars.user ===
worker_count = 4

# === from cluster-phase override ===
deploy_bnk = false

Re-applying from this snapshot alone reconstructs the inputs the user wrote; embedded Terraform module defaults are not captured (see §“What it’s not” above for the full list of what’s out of scope).

The header records the binary version and apply timestamp so the reader can correlate the snapshot to a specific roksbnkctl invocation. Alphabetic ordering within each section means re-running apply with identical inputs produces a byte-identical file (idempotency — handy for diffing snapshots across applies).

The everyday workspace routine

The minimum daily routine:

# Initialise (creates ~/.roksbnkctl/<name>/config.yaml and makes it current).
# With no -w and no current workspace, init asks for a name (defaults to "default").
roksbnkctl init

# Switch which workspace is "current"
roksbnkctl ws use prod

# See all workspaces and which one is current
roksbnkctl ws list

roksbnkctl init -w <name> is the one-shot path that creates the directory and populates config.yaml interactively — and selects it. Everything else (ws new, ws use, ws delete) is the deconstructed form for users who want finer-grained control. The init interview itself (which now lists your account’s regions and existing clusters) is covered in Chapter 7 — Quick start.

Skip the interview: init --config-file

config.yaml is the single declarative input, so if you already have one — from a prior workspace, a colleague’s hand-off, or roksbnkctl init example > config.yaml followed by an edit — seed the workspace from it and skip the prompts entirely:

roksbnkctl init example > config.yaml      # annotated template, every axis documented
$EDITOR config.yaml
roksbnkctl init -w myws --config-file ./config.yaml

This writes ~/.roksbnkctl/myws/config.yaml directly — unknown fields are rejected, not silently dropped — and is fully non-interactive when the file is complete (ibmcloud.region, ibmcloud.resource_group, prefix, tf_source.type). The API key is not required in the file; it resolves from the environment / keychain at run time (see Credentials). A local path or an http(s) URL is accepted, and --config-file composes with --override-from-env to inject secrets from the environment — the full unattended / CI patterns (including --non-interactive, which builds config.yaml from ROKSBNKCTL_* env alone) are in Chapter 7a — Unattended setup.

Raw terraform-variable overrides

For the rare case where you need to override a raw terraform variable the config.yaml schema doesn’t surface, drop a terraform.tfvars.user at the workspace root (~/.roksbnkctl/<ws>/terraform.tfvars.user, mode 0600) — the lifecycle auto-layers it on every up / plan / apply / down for both phases — or pass --var-file <path> on a phase command (cluster up, bnk up, …) for a one-shot. The precedence chain is: rendered terraform.tfvars → terraform.tfvars.user → a phase command’s --var-file flags → phase overrides. See Chapter 13 — Terraform variables. (init seeds the workspace from config.yaml — that’s the seed surface.)

The full command tree

roksbnkctl workspaces ...     # canonical name
roksbnkctl ws ...              # alias

ws new <name> — empty skeleton

Creates ~/.roksbnkctl/<name>/ with no config.yaml. Useful when you want the directory to exist (so ws use works) before you run init.

roksbnkctl ws new staging
# ✓ Created workspace "staging" and made it current (run `roksbnkctl init -w staging` to configure)

Creating the skeleton selects it (sets current_workspace), so the next bare command runs against it. Most users skip ws new and use roksbnkctl init -w staging directly, which creates, configures, and selects in one go.

ws use <name> — switch current

Sets the current_workspace pointer in ~/.roksbnkctl/config.yaml:

roksbnkctl ws use prod
# ✓ Current workspace: prod

roksbnkctl ws current
# prod

Refuses to point at a non-existent workspace. The pointer is the only thing that changes — workspace state stays put.

ws current — print the pointer

roksbnkctl ws current
# default

Prints the current workspace name on stdout. If no pointer is set, prints a hint like “no current workspace; run roksbnkctl ws use <name> or roksbnkctl init” to stderr and exits 0 with empty stdout — so WS=$(roksbnkctl ws current) produces an empty string in scripts rather than spurious output.

ws list — table view

roksbnkctl ws list
NAME      CURRENT  REGION    CLUSTER          TF SOURCE
default   *        us-south  bnk-quickstart   embedded@v1.0.0
prod               eu-de     bnk-prod         embedded@v1.0.0
staging            us-south  bnk-staging      local:./terraform

The * marker on CURRENT highlights the active workspace. Other columns reflect each workspace’s config.yaml. Rows where config.yaml is missing or unparseable still show the name, with the other columns blank — the list never errors out because of one corrupt workspace.

ws delete <name> [--force]

Removes the workspace directory and the OS-keychain entry for its API key.

You can delete the current workspace directly — there’s no “switch first” dance. If the workspace you delete was current, the pointer moves to another existing workspace (the alphabetically-first remaining one); if it was the last workspace, the pointer is cleared (there’s no fallback default — see §“The current-workspace pointer”).

One safety rail remains: delete refuses if Terraform state lists provisioned resources (unless --force). That catches the foot-gun where you forget to run roksbnkctl down first.

roksbnkctl ws delete staging
# Delete workspace "staging"? [y/N]: y
# ✓ Deleted workspace "staging"
# ✓ Current workspace is now "default"          # only printed when "staging" was current

# Refused — state still has resources
roksbnkctl ws delete prod
# Error: workspace "prod" has terraform-managed resources; pass --force to delete anyway

# I really mean it
roksbnkctl ws delete prod --force
# ✓ Deleted workspace "prod"

--force skips both the prompt and the state-non-empty check. Use it sparingly — there’s no “undo” for rm -rf ~/.roksbnkctl/<name>/.

If a down errored partway and left cloud resources stranded, the state guard will (correctly) block ws delete. Sweep the orphans with roksbnkctl cleanup first, then delete.

The current-workspace pointer

The pointer lives at ~/.roksbnkctl/config.yaml:

current_workspace: prod

Every command that doesn’t pass -w reads this pointer. The workspace verbs keep it in step with reality, so you rarely set it by hand:

  • init and ws new select the workspace they create — afterwards it’s current.
  • ws use rewrites it.
  • ws delete moves it off a deleted current workspace (to another existing one), or clears it when the last workspace is removed.

There is no fallback default. When the pointer is empty — you’ve deleted every workspace, or never created one — a command run without -w reports no workspace selected; create one with roksbnkctl init or pick one with roksbnkctl ws use <name> rather than silently operating on a phantom default. (roksbnkctl init with no -w and no current pointer asks for a workspace name, defaulting to default on the first run — so the bootstrap experience is unchanged.)

If the pointer references a workspace whose directory is gone (e.g. someone rm -rf’d it by hand), commands report workspace "prod" is not initialised; run roksbnkctl init first — repoint with ws use <other> or recreate with init.

-w / --workspace for one-off overrides

Every command accepts -w <name> to override the current pointer for a single invocation:

# Doctor against "prod" without flipping the global pointer
roksbnkctl -w prod doctor

# Run init for a new workspace called "staging"
roksbnkctl init -w staging

# Get pods from the "default" cluster while currently on "prod"
roksbnkctl -w default k get pods -A

Use this when:

  • You’re scripting against multiple workspaces in a single run (CI runner that exercises default + e2e-cleanup back-to-back).
  • You want to run a one-off command against a different environment without losing your current context.
  • You’re testing a fresh workspace before promoting it to current.

The flag only affects the running command — the pointer in ~/.roksbnkctl/config.yaml is unchanged. After the command exits, the next bare roksbnkctl reads the original pointer.

Deleting the workspace you’re in

ws delete removes the current workspace directly — there’s no “switch first” step. It moves the pointer to another existing workspace, or clears it when none remain.

# End-to-end test cleanup — just destroy and delete, in any order
roksbnkctl down --auto
roksbnkctl ws delete default --force
# ✓ Deleted workspace "default"
# ✓ Current workspace is now "e2e-cleanup"      # whatever remained, alphabetically first

# Delete everything — the last delete clears the pointer
roksbnkctl ws delete e2e-cleanup --force
# ✓ Deleted workspace "e2e-cleanup"
# ✓ No workspaces remain; current workspace cleared

With no workspaces left, the pointer is empty and the next bare command reports no workspace selected (no phantom default); roksbnkctl init starts fresh.

Using a workspace’s environment in your shell

roksbnkctl shell drops you into a subshell with KUBECONFIG, IBMCLOUD_API_KEY, IC_API_KEY, and IBMCLOUD_REGION pre-loaded from the current workspace. KUBECONFIG resolves the way kubectl resolves it — the first existing entry in an inherited $KUBECONFIG, else ~/.kube/config — not the workspace state directory:

roksbnkctl shell
# (now in a subshell)
echo $KUBECONFIG
# /home/you/.kube/config
exit
# (back to the parent shell)

Same for -w:

roksbnkctl -w prod shell

Useful when you want to run host kubectl / host oc / arbitrary tools with the workspace context loaded. The internalised verbs (roksbnkctl k get, etc.) read the same context automatically — you don’t need to be in a subshell to use them.

Common workspace patterns

A handful of patterns that come up in practice:

Use casePattern
Different IBM Cloud accountsdefault for personal, acct-foo for an account-specific key
Different regionsus-south, eu-de workspaces with distinct cluster.name values
Throwaway short-lived clustersbnk-trial-N workspaces; delete with --force after down (delete the current one directly — it auto-switches)
CI vs local devdev and ci workspaces; ci uses IBMCLOUD_API_KEY from env, dev uses keychain
Recover a half-torn-down envroksbnkctl cleanup sweeps stranded <prefix>-* cloud resources, then ws delete

Workspaces are cheap. If a flow benefits from isolation, make a new one rather than fighting with --var-file overrides on the existing one.

This chapter covers the workspace-as-a-unit: how to create, switch, list, delete. The schema of the per-workspace config.yaml itself — every field, default, valid range — is Chapter 12 — Workspace config.

Quick start: from API key to deployed BNK

This chapter walks the 4-command lifecycle (init → up → test → down) end-to-end. By the time you reach the bottom you’ll have a deployed BNK trial on a fresh ROKS cluster, a passing connectivity test, and a clean tear-down command ready when you’re done.

The lifecycle, at a glance:

sequenceDiagram
    autonumber
    actor User
    participant CLI as roksbnkctl
    participant TF as terraform-exec (embedded HCL)
    participant IBM as IBM Cloud API
    participant K8s as ROKS cluster + BNK
    User->>CLI: roksbnkctl init
    CLI->>CLI: write workspace config.yaml
    CLI-->>User: workspace ready
    User->>CLI: roksbnkctl up --auto
    CLI->>TF: terraform plan + apply
    TF->>IBM: provision cluster, VPC, jumphost
    IBM-->>TF: 77 resources created
    TF->>K8s: helm install cert-manager + flo + BNK
    K8s-->>CLI: cluster + BNK up
    CLI-->>User: kubeconfig saved, jumphost target auto-populated
    User->>CLI: roksbnkctl test
    CLI->>K8s: connectivity + DNS + (optional) throughput
    K8s-->>CLI: pass/fail per check
    CLI-->>User: green
    User->>CLI: roksbnkctl down --auto
    CLI->>TF: terraform destroy
    TF->>IBM: 77 resources destroyed
    CLI-->>User: workspace state retained

The walkthrough assumes:

  • You have a roksbnkctl binary on PATH (Chapter 4).
  • You have an IBM Cloud API key for an account with permission to create ROKS clusters.
  • terraform >= 1.10 is on PATH and roksbnkctl doctor looks healthy (Chapter 5). The floor is 1.10, not 1.5: roksbnkctl enforces it (requireTerraformVersion) because remote state locking uses terraform’s native S3 lockfile (Chapter 12a).

If roksbnkctl doctor is not green for terraform and IBMCLOUD_API_KEY resolves, fix those first — nothing below will work otherwise.

Note. The output blocks below are illustrative — version strings, cluster IDs, IPs, and timing all vary between runs. The shape of each step is what to look for.

Step 1 — set the API key

The cleanest way to make roksbnkctl see your API key is the IBMCLOUD_API_KEY environment variable. roksbnkctl init will offer to save it to your OS keychain afterwards, so you only paste it once.

export IBMCLOUD_API_KEY="ibmcloud-api-key-value-here"

If you’d rather not export it in your shell, roksbnkctl init will prompt for it on a TTY and offer the same keychain-save afterwards. See Chapter 14 for the full resolution chain.

Step 2 — roksbnkctl init

Initialises a workspace under ~/.roksbnkctl/default/ (or under <name>/ if you pass -w <name>). Verifies the API key against IBM IAM, resolves the resource group, and writes config.yaml.

Flag-value syntax. Short flags accept their value as -w value (one space) or -w=value (immediate equals); the stuck-together form -wvalue is rejected at parse time before any workspace dir is created. The same rule applies to every short flag (-w, -f, -n, -c, -l, -o). Long flags follow the same two shapes: --workspace value or --workspace=value. So if you typo the long-flag name as a short — e.g. roksbnkctl init -ws canada-roks --config-file ./config.yaml — the binary rejects it immediately with:

roksbnkctl: "-ws" is not a recognised flag (looks like a stuck-together short-flag-value, which this binary does not accept).
Use one of these forms instead:
  -w s              (short, space)
  -w=s              (short, equals)
  --workspace s     (long, space)
  --workspace=s     (long, equals)
Did you mean '--workspace s'?

Recovery is what the error names: re-run with --workspace canada-roks (or -w canada-roks).

roksbnkctl init

init interviews your account: after it verifies the API key it asks whether to create a new cluster, then lists the regions (or the existing clusters) your credentials can reach — you pick from a menu instead of typing free-text.

Sample interactive session (creating a new cluster):

roksbnkctl init
  Workspace name [default]:
→ Verifying IBM Cloud credentials ... ✓ you@example.com (account 1a2b3c...)
  Create a new ROKS cluster? [Y/n]: y
  Region for the new cluster:
       1) au-syd
       2) br-sao
     * 3) ca-tor
       ...
      10) us-south
  Choice [1-14] [3]: 10
  Workspace prefix [bnk-quickstart]:
  Workers per zone (min 1; cluster spans all 3 AZs) [1]: 2
  OpenShift version [4.20]:
  Create registry COS instance? [Y/n]: y
  Resource group [default]:
→ ✓ Resource group "default" (id ...)
  Create Transit Gateway? [Y/n]: y
  Install cert-manager? [Y/n]: y
  Deploy BIG-IP Next (BNK)? [Y/n]: y
  Add a testing client? [y/N]: y
  Region to install the test client in:
      ...
  Choice [1-14] [10]: 3
  Create a new client VPC for it? [Y/n]: y
  Create per-zone cluster jumphosts? [y/N]: n
  Save IBMCLOUD_API_KEY to OS keychain for this workspace? (y/N): y
✓ Current workspace: default
✓ Wrote ~/.roksbnkctl/default/config.yaml

What just happened:

  • A workspace called default now exists at ~/.roksbnkctl/default/ and is current — the next bare command runs against it (no separate ws use needed).
  • You picked the region from the account’s available regions; the cluster spans all three availability zones with workers-per-zone × 3 workers. One per zone (3 total) is the minimum — a ROKS cluster is always multi-AZ, so init floors the count there.
  • The optional testing client (TGW jumphost + client VPC) can live in a different region from the cluster — you’re prompted for its region separately.
  • config.yaml records the region, resource group, prefix, worker sizing, the per-resource resources: toggles, and BNK component defaults. The API key is saved to your OS keychain (macOS Keychain, libsecret on Linux, or Windows Credential Manager) under service roksbnkctl; subsequent runs resolve it from there without prompting.

You can re-run roksbnkctl init to update workspace settings; existing values become the prompt defaults.

Reusing an existing cluster

Answer n to Create a new ROKS cluster? and init lists the account’s running OpenShift clusters to pick one:

  Create a new ROKS cluster? [Y/n]: n
→ Listing running OpenShift clusters...
  Choose an existing cluster:
     * 1) bnk-demo        (us-south, 4.20.x_openshift)
       2) roks-prod-ca    (ca-tor, 4.19.x_openshift)
  Choice [1-2] [1]: 1

The chosen cluster’s region and name are recorded, and init populates cluster-outputs.json — the same identity file cluster register writes — so a subsequent roksbnkctl up deploys BNK straight onto it, with no cluster up and no separate register step.

Step 3 — roksbnkctl up --auto

The deployment. Runs terraform plan, runs terraform apply, fetches the admin kubeconfig from IBM Cloud, writes it to ~/.kube/config at mode 0600. The --auto flag skips the plan-and-confirm gate; without it up shows the plan and asks “apply? [y/N]” before continuing.

roksbnkctl up --auto

Sample output (heavily abridged — a real run is ~50 minutes and prints terraform’s full plan + apply log):

roksbnkctl up --auto
→ Resolving terraform source ... embedded (v1.0.0)
→ Extracting bundled HCL to ~/.roksbnkctl/default/state/tf-source/embedded-terraform/
→ Pre-creating kubeconfig + scratch directories
→ Rendering auto-tfvars from config.yaml ... ok
→ terraform init -reconfigure
  Initializing provider plugins... done.
→ terraform apply (auto-approved)
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Creating...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [10m elapsed]
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [20m elapsed]
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [30m elapsed]
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Creation complete after 38m12s
  module.cert_manager.helm_release.cert_manager: Creation complete after 2m11s
  module.flo.helm_release.flo: Creation complete after 4m02s
  module.cne_instance.kubernetes_manifest.cne_instance: Creation complete after 1m42s
  module.license.helm_release.license: Creation complete after 2m18s
  module.testing.tls_private_key.jumphost_shared_key: Creation complete after 0s
  module.testing.ibm_is_instance.tgw_jumphost: Creation complete after 1m48s

  Apply complete! Resources: 77 added, 0 changed, 0 destroyed.

→ Fetching admin kubeconfig for cluster "<cluster-id>"
✓ Wrote /home/you/.kube/config (chmod 0600)
✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`

What just happened:

  • 77 resources were created across ROKS, cert-manager, FLO, CNE Instance, BNK license, and a small testing footprint (the TGW jumphost).
  • An admin kubeconfig was fetched directly from IBM Cloud’s container service API (no ibmcloud ks cluster config shell-out) and written at mode 0600.
  • A jumphost target was auto-populated in your workspace config from terraform outputs. This makes Chapter 16’s --on jumphost flag work without any further configuration.

The actual elapsed time on a fresh run is dominated by ROKS cluster creation (~30-40 min) and cert-manager + FLO Helm install (~10 min). Re-runs are dramatically faster because terraform’s idempotence skips already-created resources.

Step 4 — roksbnkctl status

Quick sanity check: workspace pointer is right, cluster is reachable, BNK pods are healthy.

roksbnkctl status

Sample output:

Workspace: default
Region:    us-south
RG:        Default (id: ...)
Cluster:   bnk-quickstart (id: <cluster-id>) — Ready
TF source: embedded (v1.0.0)
Last apply: 2026-05-08T14:22:08Z
Nodes:     2/2 Ready
BNK pods:  flo (3/3), cis (1/1), cert-manager (3/3), cne-instance (1/1)

If anything is not green here, jump to Chapter 26 — Troubleshooting.

Step 5 — roksbnkctl test

Run the built-in validation suite. Bare test runs the connectivity + DNS checks (the throughput test takes a few minutes and is opt-in).

roksbnkctl test

Sample output:

roksbnkctl test
→ Suite: connectivity
  ✓ https://www.f5.com (200, 312ms)
  ✓ https://api.openshift.com (200, 88ms)
  ✓ https://us-south.containers.cloud.ibm.com (200, 142ms)
→ Suite: dns
  ✓ www.f5.com → 23.50.149.94 (A, 12ms)
  ✓ api.openshift.com → 35.190.27.231 (A, 18ms)

3 connectivity checks passed; 2 DNS checks passed; 0 failed.

For the throughput suite specifically:

roksbnkctl test throughput --mode east-west

Sample output:

→ Deploying iperf3 server pod into namespace "roksbnkctl-test"
✓ Pod ready (iperf3-server-...)
→ Exposing via ClusterIP service
✓ Service ready (cluster-ip: 172.21.45.108:5201)
→ Running iperf3 -c against the service from local
✓ throughput: 9.41 Gbits/sec (mean over 10s)
→ Tearing down iperf3 fixture
✓ pod and service deleted

The --mode east-west flag uses a ClusterIP service and runs the host iperf3 client through oc port-forward for in-cluster traffic; --mode north-south uses a LoadBalancer for outside-the-cluster traffic. See Chapter 22 for the full design.

The connectivity test uses Go’s built-in net/http — no external curl is shelled out — and similarly DNS uses Go’s net.Resolver. The --insecure flag on test connectivity skips TLS validation if you need to test against self-signed endpoints.

Step 6 — explore (optional)

A few useful follow-ups now that the cluster’s up:

# tail the F5 Lifecycle Operator logs
roksbnkctl logs flo -f

# drop into a shell with the workspace's KUBECONFIG + IBMCLOUD_API_KEY exported
roksbnkctl shell

# run a one-shot kubectl with the workspace context loaded
roksbnkctl kubectl get pods -A

# run ibmcloud through the auto-discovered jumphost
roksbnkctl ibmcloud --on jumphost ks cluster ls

The --on jumphost flag is covered in detail in Chapter 16. It lets you run any of the passthrough commands (exec, shell, kubectl, oc, ibmcloud) from inside the cluster’s network — useful when your workstation is behind a corporate firewall that can’t reach IBM Cloud directly.

Step 7 — roksbnkctl down --auto

Tear it all back down when you’re finished. The teardown is terraform destroy under the hood, with the same resilience to transient IBM API errors as up.

roksbnkctl down --auto

Sample output:

roksbnkctl down --auto
→ terraform destroy (auto-approved)
  module.testing.ibm_is_instance.tgw_jumphost: Destroying...
  ...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Destroying...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still destroying... [5m elapsed]
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Destruction complete after 8m16s

  Destroy complete! Resources: 77 destroyed.

✓ Workspace "default" state retained at ~/.roksbnkctl/default/
  (run `roksbnkctl ws delete default` to remove the workspace dir)

down retains the workspace dir and config so you can up again with the same settings. To remove the workspace entirely:

roksbnkctl ws delete default

This refuses if terraform state still lists resources (use --force to override) and cleans up the keychain entry.

What you just did

In effectively three commands you:

  1. Provisioned a fresh ROKS cluster on IBM Cloud.
  2. Installed cert-manager, F5 Lifecycle Operator, and a complete BNK trial on top of it.
  3. Validated the deployment with HTTP connectivity + DNS resolution + (optionally) throughput tests.
  4. Got an auto-discovered jumphost target ready for any --on jumphost follow-ups.

The same flow runs against multiple workspaces, multiple regions, and multiple resource groups — see Chapter 6 for the multi-environment patterns. From here, Chapter 16 covers the --on flag, Chapter 24 covers day-2 operations, and Chapter 26 covers what to do when one of the above steps doesn’t go right.

Unattended setup: seeding a workspace from a file, URL, or environment

The quick start drives init interactively. A CI pipeline or a fleet operator wants the opposite: hand init a finished config, fetch it from a URL, and inject secrets from the environment so a single committed template stands up many workspaces without ever baking an API key into version control.

Three init options make that work:

OptionWhat it does
--config-file <path|url>Seed the workspace config.yaml directly (no interview)
--override-from-envOverlay specific config.yaml fields from environment variables
--non-interactiveBuild config.yaml from environment variables alone — no file, no prompt (argv+env runners)

Getting a starting point: roksbnkctl config

Both forms of the input can be printed, so neither has to be written from memory. With no flags it prints an annotated template, ready to pipe:

roksbnkctl config yaml > config.yaml     # every setting, with the comment explaining it
roksbnkctl config env  > .env            # the same settings as ROKSBNKCTL_* overrides

The .env template has every variable commented out. That is deliberate: an empty assignment is not the same as leaving a setting alone. ROKSBNKCTL_X= sets X to empty, which overrides whatever default would otherwise apply — uncomment only what you mean to set.

Converting between the two

The same settings expressed either way, which is what you want when a pipeline needs as environment variables what a colleague handed you as a file:

roksbnkctl config env  --from-yaml config.yaml > .env
roksbnkctl config yaml --from-env  .env        > config.yaml

The converted output carries no comments — it is a populated file, not a template — and includes only what the input actually set, so it stays a statement of intent rather than a snapshot of every default. The handful of required fields are emitted even when empty, so a missing one is visible instead of silently absent.

Round-tripping is the quickest way to check a .env says what you think:

roksbnkctl config yaml --from-env .env   # read it back as config.yaml

The conversion runs the same override machinery --override-from-env uses, so a value lands where it would in a real run. A variable that this prints under bnk.tmm_replicas is the variable that sets bnk.tmm_replicas; there is no second parser that could disagree.

A worked example — the Appendix C Small cluster as environment variables:

cat > small.yaml <<'YAML'
ibmcloud: {region: us-east, resource_group: default}
prefix: small
cluster: {create: true, name: small, workers_per_zone: 2, worker_flavor: bx2.8x32}
bnk:  {manifest_version: 2.4.0-EA, cneinstance_size: Tiny, tmm_replicas: 3}
YAML

roksbnkctl config env --from-yaml small.yaml
# ROKSBNKCTL_CLUSTER_CREATE=true
# ROKSBNKCTL_CLUSTER_NAME=small
# ROKSBNKCTL_CNEINSTANCE_SIZE=Tiny
# ROKSBNKCTL_MANIFEST_VERSION=2.4.0-EA
# ROKSBNKCTL_PREFIX=small
# ROKSBNKCTL_REGION=us-east
# ROKSBNKCTL_RESOURCE_GROUP=default
# ROKSBNKCTL_TMM_REPLICAS=3
# ROKSBNKCTL_WORKERS_PER_ZONE=2
# ROKSBNKCTL_WORKER_FLAVOR=bx2.8x32

The pattern

flowchart LR
    T["config.yaml.tmpl<br/>(committed, no secrets)"] -->|--config-file| I["roksbnkctl init"]
    E["$IBMCLOUD_API_KEY<br/>$ROKSBNKCTL_*"] -->|--override-from-env| I
    I --> W["~/.roksbnkctl/&lt;ws&gt;/config.yaml<br/>(complete, secrets filled)"]

Commit a template with the secret fields blank, and let the pipeline supply the real values from its environment:

# config.yaml.tmpl — committed to git, no secrets
ibmcloud:
  region: ""              # ← ROKSBNKCTL_REGION
  resource_group: ""      # ← ROKSBNKCTL_RESOURCE_GROUP
  api_key_b64: ""         # ← IBMCLOUD_API_KEY (raw, base64-encoded)
prefix: ""                # ← ROKSBNKCTL_PREFIX
cluster: { create: true, name: "" }
tf_source: { type: embedded }
resources:
  bnk: { create: true }
  tgw_jumphost: { create: true }
  client_vpc: { create: true }
export IBMCLOUD_API_KEY="$(vault read -field=key secret/ibmcloud)"
export ROKSBNKCTL_REGION=eu-de
export ROKSBNKCTL_RESOURCE_GROUP=default
export ROKSBNKCTL_PREFIX=ci-run-42

roksbnkctl init -w ci-run-42 \
  --config-file https://raw.githubusercontent.com/acme/infra/main/config.yaml.tmpl \
  --override-from-env

No secret ever lands in git; the same template seeds every account.

Exit codes

Unattended callers branch on $?, so the codes are an interface:

codemeaning
0The command did what was asked.
1It went wrong. The default for anything without a more specific code.
2The invocation was rejected — a malformed flag, an argument the command cannot accept. Nothing was attempted.
126Permission denied: SSH authentication, a host-key mismatch, a credential the remote end refused.
127The target could not be reached at all.
130The operator interrupted it (Ctrl-C / SIGINT).
125upgrade / self update only. The upgrade removed the old binary and could not put anything back, so there is no roksbnkctl at the install path. The message names the .old sidecar and the command to rename it.
otherA wrapped tool’s own status, passed through unchanged — terraform, ibmcloud, and --on <target> remote commands all propagate theirs. That includes 125 from any command other than upgrade / self update, which spawn no child.

130 is worth wiring into CI cleanup: it distinguishes “someone stopped this” from “it broke”, which a job that retries on failure should treat differently.

125 matters for the same reason, in the opposite direction: an ordinary failed upgrade is safe to retry, and this one is not — there is nothing left to run. A wrapper that retries on any non-zero status will loop forever on a machine that cannot recover until someone renames the sidecar back. Before v1.50.0 only roksbnkctl init produced it — every other command turned the same Ctrl-C into 1.

2 is similarly worth separating. It means the command never ran, so retrying the same invocation will fail the same way; the fix is in the script, not the environment.

A command that propagates a wrapped tool’s status does not print its own error on top — terraform’s diagnostics have already gone to stderr, and a second roksbnkctl: ... line would only add noise. Check the stream, not just the code.

The interrupt outranks everything else

A failure that happens because of the Ctrl-C reports the interrupt, not the failure class it landed in: a connect aborted mid-handshake is 130, not 127; a test suite cut off mid-run is 130, not a red 1; a child killed by the signal propagates 130 (128+SIGINT, the shell convention), not 255. So $? -eq 130 is reliable — a job does not need to trap SIGINT itself to tell “someone stopped this” from “it broke”.

--config-file

--config-file parses the supplied YAML strictly — unknown fields are rejected, not silently dropped — into the workspace config.yaml, and skips the interview entirely. It is non-interactive when the config is complete. A config is complete when it has all of:

  • ibmcloud.region
  • ibmcloud.resource_group
  • prefix
  • tf_source.type

If any are missing (after --override-from-env runs), init exits with a clear message naming exactly which — supply them in the file, set them via the environment, or run init interactively. The API key is not required in the file; it resolves from the environment, keychain, or config at run time as usual (see Credentials).

URL inputs

--config-file accepts a local path or an http(s) URL. A URL is fetched (30 s timeout, 10 MB cap) and treated identically to a local file — a raw-git URL, a presigned IBM COS URL, or any reachable endpoint. There is no fetch authentication in this release; use a presigned or public URL.

--override-from-env

--override-from-env overlays config.yaml fields from environment variables that are set and non-empty, after the file/interview has produced the config. The environment wins over the seeded value — it is the explicit late-binding step. This is a fixed field map, not arbitrary templating.

This table is a selection, not the whole surface. The ones below are what most deployments set, chosen because their encoding or precedence needs explaining — a hand-written table of every override would be a second copy of the schema, and this book has been bitten by one of those before. The complete list is generated from the override machinery itself and cannot drift from it:

$ roksbnkctl config env > .env          # every variable, commented, with its field

or the config.yaml cheatsheet, which shows every field beside the variable that sets it. Prefer either of those when you are looking for a specific field; use this table when you want to know how a value is interpreted.

Environment variableconfig.yaml fieldEncoding
IBMCLOUD_API_KEYibmcloud.api_key_b64raw key, base64-encoded
ROKSBNKCTL_API_KEY_B64ibmcloud.api_key_b64verbatim (pre-encoded)
ROKSBNKCTL_PREFIXprefixverbatim
ROKSBNKCTL_REGIONibmcloud.regionverbatim
ROKSBNKCTL_RESOURCE_GROUPibmcloud.resource_groupverbatim
ROKSBNKCTL_CLUSTER_NAMEcluster.nameverbatim
ROKSBNKCTL_CLUSTER_CREATEcluster.createbool (true/false/1/0)
ROKSBNKCTL_OPENSHIFT_VERSIONcluster.openshift_versionverbatim
ROKSBNKCTL_WORKERS_PER_ZONEcluster.workers_per_zoneinteger
ROKSBNKCTL_CLUSTER_PUBLIC_GATEWAYcluster.public_gatewaybool — false builds a disconnected cluster whose workers have no Internet egress. Unset inherits the terraform default (true).
ROKSBNKCTL_TRUSTED_PROFILE_SAbnk.trusted_profile.service_accountKubernetes account allowed to assume the CNE controller’s IBM Cloud Trusted Profile. Must match the account the FLO chart creates.
ROKSBNKCTL_TRUSTED_PROFILE_ROLESbnk.trusted_profile.rolesIAM roles for that profile, scoped to the cluster’s own VPC. Comma-separated. An unparseable entry is dropped.
ROKSBNKCTL_VLAN_PREFIXLENbnk.network.vlan_prefixlenTMM self-IP mask. Independent of the zone CIDRs and never derived from them — a deliberate disagreement, plus static routes, is how a traffic pattern is forced. Out-of-range values are silently ignored and the terraform default stands.
ROKSBNKCTL_VLAN_PREFIXLEN_EXTERNALbnk.network.vlan_prefixlen_externalOverrides the shared mask for the external VLAN only. Unset = inherit. Silently ignored if out of range.
ROKSBNKCTL_VLAN_PREFIXLEN_INTERNALbnk.network.vlan_prefixlen_internalSame, internal VLAN. The two need not match.
ROKSBNKCTL_GENERIC_CA_B64registry.generic_ca_b64Mirror CA, verbatim — already base64, not re-encoded.
ROKSBNKCTL_GENERIC_CA_SHA256registry.generic_ca_sha256The out-of-band CA pin.
ROKSBNKCTL_REACHABILITY_RETRY_SECONDSbnk.preflight.reachability_retry_secondsPer-target retry window before the verdict is believed.
ROKSBNKCTL_REACHABILITY_TIMEOUT_SECONDSbnk.preflight.reachability_timeout_secondsTotal wait for every node to report.
ROKSBNKCTL_CLUSTER_NETWORK_MODEcluster.network_modeHow the workers are attached: single-nic (default) or multi-nic. Create-time only. Exists for exactly this chapter’s case — a runner that never writes a config.yaml would otherwise have no way to ask for it.
ROKSBNKCTL_CLUSTER_VPC_CIDRcluster.vpc_cidrCIDR (/18 or larger) for a new cluster VPC’s per-zone address prefixes. Unset = IBM auto, which is the SAME block for every VPC in the region — set a distinct one per cluster when two share a Transit Gateway, or the gateway silently blackholes one. See Chapter 9a.
ROKSBNKCTL_TRANSIT_GATEWAY_NAMEresources.transit_gateway (create:false + existing)a Transit Gateway name or id — cluster up/register attaches the cluster VPC to it (see Sharing a Transit Gateway)
ROKSBNKCTL_EXISTING_SUBNET_IDScluster.existing_subnet_idsPlace the cluster in subnets that already exist. Comma-separated, in zone order — each subnet’s zone is read from the subnet, so a reordered list places the cluster differently. Requires resources.cluster_vpc: {create: false, existing: <vpc-id>}.
ROKSBNKCTL_FLP_VSI_CREATE_VPCbnk.flp.vsi.create_vpcGive the FLP VSI its own VPC instead of placing it in one that exists. This is what lets the proxy be the first thing deployed in an air-gapped estate. Mutually exclusive with ROKSBNKCTL_FLP_VSI_VPC.
ROKSBNKCTL_FLP_VSI_VPC_NAMEbnk.flp.vsi.vpc_nameName for that VPC. Empty → flp-vsi-vpc.
ROKSBNKCTL_FLP_VSI_SUBNET_CIDRbnk.flp.vsi.subnet_cidrIts address prefix. Must not overlap anything the consuming clusters can already route to — a transit gateway silently blackholes one of two overlapping VPCs.
ROKSBNKCTL_CLUSTER_VPC_IDresources.cluster_vpc (create:false + existing)verbatim — adopt an existing cluster VPC by ID
ROKSBNKCTL_TESTING_VPC_NAMEresources.testing_client_vpc_nameverbatim (names the client VPC to create)
ROKSBNKCTL_TGW_JUMPHOST_CREATEresources.tgw_jumphost.createbool — the optional testing jumphost. Defaults off, as the interview does.
ROKSBNKCTL_CLIENT_VPC_CREATEresources.client_vpc.createbool — the testing client VPC. Defaults off; it consumes a Transit Gateway connection.
ROKSBNKCTL_CLIENT_VPC_NAMEresources.client_vpc.existingAdopt an existing client VPC instead of creating one — the env equivalent of the interview’s “Existing client VPC name”. Required if you enable the jumphost without creating a VPC.
ROKSBNKCTL_CERT_MANAGER_CREATEresources.cert_manager.createbool — set false to adopt the cert-manager the cluster already runs instead of installing one. An adopted customer cluster very often has one; without this, bnk up fails with namespaces "cert-manager" already exists, which names the collision but not the setting that resolves it.
ROKSBNKCTL_REGISTRY_COS_CREATEresources.registry_cos.createbool — set false when the mirror registry’s object storage already exists (or the registry is external, such as Artifactory or Harbor, and needs none).
ROKSBNKCTL_CLUSTER_JUMPHOSTS_CREATEresources.cluster_jumphosts.createbool — the per-cluster jumphosts. Defaults off.
ROKSBNKCTL_BIGIP_URLbnk.cis.bigip_urlverbatim
ROKSBNKCTL_BIGIP_USERNAMEbnk.cis.bigip_usernameverbatim
ROKSBNKCTL_BIGIP_PASSWORDbnk.cis.bigip_password_b64raw, base64-encoded
ROKSBNKCTL_ZONE<n>_EXT_VLAN_CIDR … _INTERNAL_SELFIPbnk.network.zones[n-1]per-zone VLAN/SNAT/VIP CIDRs + self-IPs (n = 1…3; all six fields required for a zone to apply)
ROKSBNKCTL_TESTING_SSH_KEY_NAMEresources.testing_ssh_key_nameverbatim
ROKSBNKCTL_BNKFORGE_CA_B64bnkforge.ca_b64PEM CA the BNK Forge server’s certificate must chain to, base64-encoded. Pins a self-signed lab cert so the session token is not sent over an unauthenticated connection. Supersedes bnkforge.insecure.
ROKSBNKCTL_TESTING_JUMPHOST_ALLOWED_CIDRSresources.testing_jumphost_allowed_cidrscomma-separated source CIDRs for SSH (:22) to the testing jumphosts. Unset → open (they carry a public floating IP; access is key-only). Set to your public /32 on a shared account.
ROKSBNKCTL_TESTING_CLIENT_VPC_INBOUND_CIDRSresources.testing_client_vpc_inbound_cidrscomma-separated; inbound to the testing client VPC’s default SG. Unset → the RFC-1918 ranges (in-fabric test traffic arrives over the Transit Gateway).
ROKSBNKCTL_CLUSTER_HTTP_ALLOWED_CIDRSresources.cluster_http_allowed_cidrscomma-separated; sources for :80 on the cluster SG. Unset → open (this is the ingress/ALB path).
ROKSBNKCTL_CLUSTER_VPC_DEFAULT_SG_INBOUND_CIDRSresources.cluster_vpc_default_sg_inbound_cidrscomma-separated; inbound (all protocols/ports) to the cluster VPC’s default SG. Unset → open, the historical behaviour.
ROKSBNKCTL_REGISTRY_TARGETregistry.targeticr | generic
ROKSBNKCTL_GENERIC_HOSTregistry.generic_hostverbatim, e.g. harbor.example.com
ROKSBNKCTL_GENERIC_REPO_PREFIXregistry.generic_repo_prefixverbatim — a Harbor project, an Artifactory repo key
ROKSBNKCTL_GENERIC_USERNAMEregistry.generic_usernameverbatim
ROKSBNKCTL_GENERIC_PASSWORDregistry.generic_password_b64raw, base64-encoded
ROKSBNKCTL_LICENSE_MODEbnk.license_modeconnected | disconnected | f5licenseproxy (see Chapter 10c)
ROKSBNKCTL_FLO_NAMESPACEbnk.flo_namespaceNamespace FLO installs into. Default f5-bnk.
ROKSBNKCTL_FLO_UTILS_NAMESPACEbnk.flo_utils_namespaceNamespace for the F5 utility components. Default f5-utils. Set both to the same value for one shared namespace — verified against BNK 2.3.
ROKSBNKCTL_GATEWAY_CLASS_NAMEgateway.class_nameGatewayClass name; blank → gateway-class. GatewayClass is cluster-scoped, so two BNK installs sharing a cluster must not share it.
ROKSBNKCTL_GATEWAY_CONTROLLER_NAMEgateway.controller_nameLeave blank. Blank derives f5.com/<flo_namespace>-f5-cne-controller — the value the CNE controller answers to — and follows ROKSBNKCTL_FLO_NAMESPACE automatically. Set it only to aim the GatewayClass at a controller this deployment did not install. A wrong value fails silently: the apply succeeds, the GatewayClass is never Accepted, and no traffic flows.
ROKSBNKCTL_GATEWAY_ROUTE_EXAMPLESgateway.route_examplesComma-separated extra route kinds to create working examples of. On BNK 2.3: GRPCRoute, L4Route. Blank leaves an existing deployment byte-identical. TCPRoute is not valid — 2.3 pins Gateway API 1.4.1 standard, which does not contain it; BNK uses its own L4Route for TCP.
ROKSBNKCTL_GATEWAY_L4_LISTENER_PORTgateway.l4_listener_portPort for the TCP listener L4Route needs. Blank → 8080.
ROKSBNKCTL_FLP_NAMESPACEbnk.flp.namespaceverbatim (FLP mode only)
ROKSBNKCTL_FLP_EXTERNAL_URLbnk.flp.external.urlverbatim — license against a proxy in another cluster
ROKSBNKCTL_FLP_ROOT_CA_B64bnk.flp.external.root_ca_b64verbatim; already base64 — re-encoding it hands the CWC a corrupt CA
ROKSBNKCTL_FLP_MODEbnk.flp.modehelm (default) | vsi — how the FLP phase deploys the proxy
ROKSBNKCTL_FLP_VSI_NAME_PREFIXbnk.flp.vsi.name_prefixPrefixes the FLP VSI’s resource names. Blank keeps the legacy unprefixed names — setting it replaces a running proxy. Needed to run more than one FLP in an account, and to make the proxy’s resources visible to cleanup’s <prefix>-* sweep.
ROKSBNKCTL_FLP_VSI_VPCbnk.flp.vsi.vpcan existing VPC id. With _MODE=vsi this arms the standalone, cluster-less appliance — as does _CREATE_VPC=true, which builds one instead. Mutually exclusive.
ROKSBNKCTL_FLP_VSI_ZONEbnk.flp.vsi.zonee.g. us-south-1; blank → the region’s first zone
ROKSBNKCTL_FLP_VSI_PROFILEbnk.flp.vsi.profileVSI instance profile; blank → bx2-4x16 (the FLP’s 4 vCPU / 8 GB floor)
ROKSBNKCTL_FLP_VSI_SSH_KEYbnk.flp.vsi.ssh_keyname of an existing IBM Cloud VPC SSH key
ROKSBNKCTL_FLP_VSI_BOOT_SIZE_GBbnk.flp.vsi.boot_size_gbinteger; blank → 100
ROKSBNKCTL_FLP_VSI_FLOATING_IPbnk.flp.vsi.floating_ipbool; an unparseable value leaves the module default (true) rather than pinning false
ROKSBNKCTL_FLP_VSI_MANAGEMENT_ALLOWED_CIDRSbnk.flp.vsi.management_allowed_cidrscomma-separated list — gates the :80 status UI
ROKSBNKCTL_FLP_VSI_LICENSING_ALLOWED_CIDRSbnk.flp.vsi.licensing_allowed_cidrscomma-separated list — gates the :8443 proxy + :22
ROKSBNKCTL_FLP_VSI_STATUS_IMAGEbnk.flp.vsi.status_imagethe flp-status web-UI image, e.g. <harbor>/bnk-status/flp-status:v1
ROKSBNKCTL_FLP_VSI_STATUS_REGISTRY_HOSTbnk.flp.vsi.status_registry_hostverbatim — the mirror the status image is pulled from
ROKSBNKCTL_FLP_VSI_STATUS_REGISTRY_CA_B64bnk.flp.vsi.status_registry_ca_b64verbatim; already base64 — the mirror’s CA, dropped into the VSI’s certs.d
ROKSBNKCTL_MANIFEST_VERSIONbnk.manifest_versionverbatim — the BNK manifest release
ROKSBNKCTL_FAR_AUTH_LOCAL_FILEbnk.far_auth_local_filepath to the FAR auth tarball in the run workspace
ROKSBNKCTL_SUBSCRIPTION_JWT_LOCAL_FILEbnk.subscription_jwt_local_filepath to the subscription JWT. Set both local-file vars and the COS download is skipped (use_cos_bucket = false)
ROKSBNKCTL_COS_INSTANCEcos.instanceverbatim; blank → bnk-supply-chain
ROKSBNKCTL_COS_BUCKETcos.bucketverbatim; blank → bnk-artifacts
ROKSBNKCTL_COS_REGIONcos.regionverbatim; blank → us-south
ROKSBNKCTL_FAR_AUTH_FILEbnk.far_auth_fileobject name in the bucket; blank → f5-far-auth-key.tgz
ROKSBNKCTL_SUBSCRIPTION_JWT_FILEbnk.subscription_jwt_fileobject name in the bucket; blank → subscription.jwt

The FLP-VSI block reproduces, from environment variables alone, the standalone licensing appliance the disconnected-cluster walkthrough builds from a hand-written config.yaml — for an argv-only runner (a CI container, BNK Forge’s container engine) that has no shell to write a YAML file with. bnk.flp.mode: vsi plus a network selects the cluster-less path — either bnk.flp.vsi.vpc (adopt an existing VPC) or bnk.flp.vsi.create_vpc: true (build one). The two are mutually exclusive; the mode alone leaves the cluster-required behaviour.

The supply-chain block matters because the COS fallbacks were renamed in v1.22.0 (bnk-orchestration → bnk-supply-chain, bnk-schematics-resources → bnk-artifacts, trial.jwt → subscription.jwt). Before these variables existed a runner that could not write a config.yaml was pinned to whatever the defaults happened to be; an account still holding the pre-v1.22 layout can now say so.

The FLP/registry handoff vars are what turn a CI pipeline into a workspace with no config.yaml to template. The registry four say where the mirror is; the FLP two are the cross-job handoff — the job that owns the proxy prints them with flp output flp_external_endpoint / flp_root_ca, and the job that installs BNK receives them as ordinary job outputs. See Flow C in CI.

Notes:

  • For the API key, IBMCLOUD_API_KEY holds the raw key and init base64-encodes it into api_key_b64 (matching how the rest of the tool reads IBMCLOUD_API_KEY). If you already have the base64 form, set ROKSBNKCTL_API_KEY_B64 instead — it takes precedence and is stored verbatim.
  • init logs how many overrides were applied and which fields — never the values, so secrets stay out of CI logs.
  • --override-from-env also applies on the interactive path, so you can answer the interview and still pull the API key from the environment.

--non-interactive — config from the environment alone

--override-from-env overlays env onto a file or interview. --non-interactive goes further: it builds config.yaml entirely from the environment — no --config-file, no prompts, no TTY. It is the path for an argv + env container runner (a CI job, or a BNK Forge container step) that can pass environment variables and arguments but cannot stage a seed file.

roksbnkctl -w forge init --non-interactive

It assembles the workspace from the same ROKSBNKCTL_* / IBMCLOUD_API_KEY variables in the table above, then:

  • defaults tf_source.type to embedded (the one required field with no env override — an empty type already means “embedded” at render time);
  • validates completeness and fails fast if a required field is missing (ibmcloud.region, ibmcloud.resource_group, prefix, tf_source.type) — it never falls back to a prompt;
  • logs which fields were applied (never the values).

A complete one-shot, file-free setup:

export IBMCLOUD_API_KEY=…              # raw key → ibmcloud.api_key_b64
export ROKSBNKCTL_REGION=eu-de
export ROKSBNKCTL_RESOURCE_GROUP=default
export ROKSBNKCTL_PREFIX=acme-eu
export ROKSBNKCTL_CLUSTER_NAME=acme-eu-roks
export ROKSBNKCTL_CLUSTER_CREATE=true

roksbnkctl -w forge init --non-interactive
roksbnkctl -w forge cluster up --auto
roksbnkctl -w forge bnk up --auto

This is exactly how the all-in-one runner image is meant to be driven from a pipeline: every step is roksbnkctl <args> with the config supplied through the environment, and state persisted on the mounted /work volume (ROKSBNKCTL_HOME=/work/.roksbnkctl).

GitHub Actions CI as an example

roksbnkctl is built to be driven from a pipeline: config is the input, the environment supplies the values, and one image carries every tool. This chapter is a worked GitHub Actions example — a single cluster, then a fleet — using the unattended / --non-interactive flow and the all-in-one runner image.

The whole model in one breath: GitHub secrets → ROKSBNKCTL_* env → init --non-interactive builds config.yaml → up reconciles it. No API key in the repo, no hand-edited config, no interactive prompts.

The runner image as a job container

Run the job in roksbnkctl-tools-runner — it has roksbnkctl plus terraform, helm, kubectl, oc, and the IBM Cloud CLI on PATH, so the workflow needs nothing installed:

jobs:
  deploy:
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:latest
    steps:
      - run: roksbnkctl version

Ephemeral runners need remote state

A GitHub-hosted runner is thrown away after the job, so a later teardown run can’t see the terraform state from the deploy run. Put the state in COS so it survives across runs (and across runners) — see Remote state:

state: { backend: s3 }   # in config.yaml; HMAC keys come from env (below)

Supply the bucket via config and the HMAC keys via env (ROKSBNKCTL_COS_HMAC_ACCESS_KEY / _SECRET_KEY).

A single-cluster workflow

name: Deploy BNK
on:
  workflow_dispatch:
    inputs:
      region:  { description: IBM Cloud region, default: eu-de }
      prefix:  { description: Resource prefix,  default: ci }

jobs:
  deploy:
    runs-on: ubuntu-latest
    container: { image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:latest }
    env:
      # secrets → the credential + cluster identity
      IBMCLOUD_API_KEY:               ${{ secrets.IBMCLOUD_API_KEY }}
      ROKSBNKCTL_REGION:              ${{ inputs.region }}
      ROKSBNKCTL_RESOURCE_GROUP:      default
      ROKSBNKCTL_PREFIX:              ${{ inputs.prefix }}
      ROKSBNKCTL_CLUSTER_NAME:        ${{ inputs.prefix }}-roks
      ROKSBNKCTL_CLUSTER_CREATE:      "true"
      # adopt shared infrastructure (reuse is config — no extra commands)
      ROKSBNKCTL_TRANSIT_GATEWAY_NAME: shared-corp-tgw
      # remote state so teardown works from a fresh runner
      ROKSBNKCTL_COS_HMAC_ACCESS_KEY: ${{ secrets.COS_HMAC_ACCESS_KEY }}
      ROKSBNKCTL_COS_HMAC_SECRET_KEY: ${{ secrets.COS_HMAC_SECRET_KEY }}
    steps:
      - name: Seed config.yaml from the environment
        run: roksbnkctl -w ci init --non-interactive

      - name: Deploy (cluster + BNK + testing, per config)
        run: roksbnkctl -w ci up --auto

      - name: Publish cluster outputs
        run: |
          roksbnkctl -w ci cluster output --json | tee -a "$GITHUB_STEP_SUMMARY"
          echo "roks_cluster_id=$(roksbnkctl -w ci cluster output roks_cluster_id)" >> "$GITHUB_OUTPUT"

init --non-interactive logs which fields it pulled from the environment (never the values), then up brings up the cluster (create or attach, per cluster.create) and BNK ∥ Testing — exactly the phases the config enables. Add roksbnkctl -w ci gateway up --auto if the deployment includes the gateway phase.

A fleet: deploy many clusters with a matrix

The thing that makes this a CI tool is that the same env-driven flow scales to a fleet — one workspace per cluster, varied by the matrix, all sharing one TGW:

name: Deploy fleet
on: { workflow_dispatch: {} }

jobs:
  deploy:
    runs-on: ubuntu-latest
    container: { image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:latest }
    strategy:
      fail-fast: false
      matrix:
        cluster:
          - { ws: eu,  region: eu-de,   prefix: acme-eu }
          - { ws: us,  region: us-south, prefix: acme-us }
          - { ws: jp,  region: jp-tok,   prefix: acme-jp }
    env:
      IBMCLOUD_API_KEY:                ${{ secrets.IBMCLOUD_API_KEY }}
      ROKSBNKCTL_RESOURCE_GROUP:       default
      ROKSBNKCTL_CLUSTER_CREATE:       "true"
      ROKSBNKCTL_TRANSIT_GATEWAY_NAME: shared-corp-tgw
      ROKSBNKCTL_COS_HMAC_ACCESS_KEY:  ${{ secrets.COS_HMAC_ACCESS_KEY }}
      ROKSBNKCTL_COS_HMAC_SECRET_KEY:  ${{ secrets.COS_HMAC_SECRET_KEY }}
    steps:
      - env:
          ROKSBNKCTL_REGION:       ${{ matrix.cluster.region }}
          ROKSBNKCTL_PREFIX:       ${{ matrix.cluster.prefix }}
          ROKSBNKCTL_CLUSTER_NAME: ${{ matrix.cluster.prefix }}-roks
        run: |
          roksbnkctl -w ${{ matrix.cluster.ws }} init --non-interactive
          roksbnkctl -w ${{ matrix.cluster.ws }} up --auto

Each matrix leg is one cluster; reuse-by-name (ROKSBNKCTL_TRANSIT_GATEWAY_NAME) lets them share one network fabric. To dictate per-AZ addressing or attach a BIG-IP for CIS, add the matching env vars (ROKSBNKCTL_ZONE<n>_*, ROKSBNKCTL_BIGIP_*) — the full set is in Unattended setup.

Teardown

A separate manually-triggered (or PR-close) workflow tears a workspace down. Because the state lives in COS, it runs cleanly on a fresh runner:

name: Teardown
on: { workflow_dispatch: { inputs: { ws: { required: true } } } }
jobs:
  destroy:
    runs-on: ubuntu-latest
    container: { image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:latest }
    env:
      IBMCLOUD_API_KEY:               ${{ secrets.IBMCLOUD_API_KEY }}
      ROKSBNKCTL_COS_HMAC_ACCESS_KEY: ${{ secrets.COS_HMAC_ACCESS_KEY }}
      ROKSBNKCTL_COS_HMAC_SECRET_KEY: ${{ secrets.COS_HMAC_SECRET_KEY }}
    steps:
      - run: roksbnkctl -w ${{ inputs.ws }} down --auto

(An attached existing cluster — ROKSBNKCTL_CLUSTER_CREATE=false — is not destroyed; down tears down only what roksbnkctl created.)

Notes

  • Pin the image by digest in production CI (...-runner@sha256:…), not :latest, so a run can’t silently change toolchains.
  • Secrets never reach the repo or the logs — they enter as job env from GitHub Secrets, and init redacts which fields it applied.
  • This is the same contract bnk-forge uses. The bnk-forge container step is a GitHub-Action-style invocation of this image with these env vars; this chapter is the do-it-yourself version of Registering with BNK Forge.
  • Every phase has status + output. roksbnkctl <phase> status [--json] and <phase> output [name] (scoped to that phase’s own attributes) are CI stage gates; roksbnkctl output merges them all — e.g. roksbnkctl testing output testing_tgw_jumphost_ip.
  • Where this is heading. up already reconciles cluster + BNK + Testing from config; folding gateway in (so a single up is the whole deployment) plus output -o json / validate are the next step — see PRD 17 — Declarative CLI.

The cluster phase (cluster up/down)

A workspace is three phases — Cluster / BNK / Testing. This chapter covers the Cluster phase specifically; the BNK trial and the testing jumphosts are separate sibling phases that deploy in parallel on top. The “trial phase” language in this chapter means the BNK phase, and the jumphosts live in their own Testing phase. See Chapter 8a — The three-phase lifecycle for the full model, parallel up, and roksbnkctl testing up/down.

A roksbnkctl workspace is two phases on top of each other: a durable cluster phase (the ROKS cluster + cluster-shared services that take 30+ minutes to provision) and a short-lived trial phase (the BNK trial that iterates on top in 5-10 minutes). The cluster phase is exposed as its own command pair, roksbnkctl cluster up / roksbnkctl cluster down, so the cluster survives across many BNK trial cycles.

This two-phase shape is the default for every new workspace. A fresh roksbnkctl up provisions the cluster phase first, then the trial phase, against separate state directories. Tearing down only the trial — the common iteration case — uses roksbnkctl bnk down and leaves the cluster intact. The unscoped up / down verbs are shape-aware composites that delegate to the right phase commands underneath.

This chapter covers what each phase deploys, why the two state directories are separate, the deploy_bnk=false override that makes “cluster only” work, the cluster-outputs.json artefact written on success, and a worked example. The companion BNK-trial chapter, Chapter 10, covers roksbnkctl bnk up / bnk down for the trial layer.

What’s deployed where

The bundled HCL has roughly two halves. The cluster phase owns the durable, cluster-scoped resources:

  • The ROKS cluster itself (VPC + subnets + worker pool)
  • A transit gateway (so the test jumphost can reach cluster internals)
  • The registry COS (Cloud Object Storage) instance — used by the BNK trial as its FAR image / license / schematic store
  • cert-manager (Helm release into the cluster)
  • The TGW jumphost VM (an Ubuntu VSI in the same VPC, used by --on jumphost)

The trial phase owns the BNK-specific resources:

  • F5 Lifecycle Operator (flo) Helm release
  • cne_instance Kubernetes manifest
  • BNK license + admin certs
  • Various cluster-side bits: ServiceAccounts, RoleBindings, Secrets

Two-phase split: cluster up provisions the first list; roksbnkctl up (the trial) provisions the second.

┌─────────────────────────────────────────────────────────┐
│  cluster phase (durable, reused across many trials)     │
│    ROKS cluster + VPC + transit gateway                 │
│    registry COS instance                                │
│    cert-manager (Helm)                                  │
│    TGW jumphost                                         │
├─────────────────────────────────────────────────────────┤
│  trial phase (one trial — destroyed by `roksbnkctl down`)│
│    flo (F5 Lifecycle Operator)                          │
│    cne_instance                                         │
│    license / admin cert / SCC bindings                  │
└─────────────────────────────────────────────────────────┘

The split exists because ROKS clusters take 30-50 minutes to provision and roughly $0.30/hour to run. Re-creating the cluster every time you want to re-test a BNK trial is wasteful; reusing one cluster for many trials cuts iteration time from “an hour” to “a few minutes”.

The two state directories

To keep cluster state and trial state from tangling, roksbnkctl uses separate Terraform state directories:

~/.roksbnkctl/<workspace>/
  state/                   # BNK trial state — written by `roksbnkctl up/down`
    terraform.tfstate
    terraform.tfvars
  state-cluster/           # cluster phase state — written by `roksbnkctl cluster up/down`
    terraform.tfstate
    cluster-phase-override.tfvars

Each phase’s commands read and write only their own state directory. Both phases use the same Terraform source (the bundled HCL) but with different effective tfvars — the trick is the deploy_bnk flag.

The deploy_bnk=false override

The bundled HCL has a top-level deploy_bnk boolean. When true, the BNK trial modules (flo, cne_instance, license) run; when false, they’re skipped and Terraform only provisions the cluster-phase resources.

roksbnkctl cluster up and roksbnkctl cluster down force deploy_bnk = false by writing a small auto-generated tfvars override into the cluster state directory:

# ~/.roksbnkctl/<workspace>/state-cluster/cluster-phase-override.tfvars
# Generated by roksbnkctl. Do not edit by hand.
# Cluster-phase override: BNK trial modules (flo / cne_instance /
# license) are skipped. cert-manager and the testing jumphost still run
# — they're cluster-shared singletons that belong with the cluster.
deploy_bnk = false

This file is layered onto the var-file chain after user-supplied --var-file flags so the override always wins. The user’s terraform.tfvars and --var-file <path> arguments still apply for everything else (region, RG, cluster name, worker count, …) — only deploy_bnk is forced.

roksbnkctl up doesn’t write this override file; its tfvars chain leaves deploy_bnk at the upstream default (true), so the trial modules run.

Reusing an existing VPC (multiple clusters in one VPC)

By default the cluster phase creates a prefix-named VPC for the cluster it builds. But you can point a new cluster at a VPC that already exists — so several clusters (each its own workspace) live in one shared VPC: the shape behind a shared licensing cluster or a common services network.

Decline “Create a new cluster VPC?” in init and it discovers the VPCs in the chosen region and lets you pick one by number (since v1.26.0):

$ roksbnkctl -w app-b init
...
Create a new cluster VPC? [Y/n] n
→ Discovering existing VPCs...
  Existing VPCs in us-south:
     1) shared-services-vpc       (available)
     2) app-a-cluster-vpc         (available)
Use an existing cluster VPC — pick a number (0 = none / create a new one): 1

Picking 0 (or a region with no VPCs) falls back to creating one — a cluster must have a VPC. In config that is (the value is the VPC id, unlike the transit-gateway adopt-by-name):

resources:
  cluster_vpc:
    create: false
    existing: r006-6fe0b20a-...   # the VPC id

which renders use_existing_cluster_vpc = true + existing_cluster_vpc_id. Two things make this safe for more than one cluster in a VPC: the cluster’s subnets are named per-cluster (<cluster>-subnet-zoneN) so they don’t collide, and per-zone public gateways are reused when the VPC already has them (IBM allows exactly one per zone per VPC). Make sure the VPC has free address-prefix space — each cluster adds three /24 subnets.

Combine this with an adopted Transit Gateway: clusters that share a VPC share that VPC’s single connection to the gateway, so N clusters can reach each other over one VPC and one gateway.

Teardown order matters in a shared VPC. The cluster that created the VPC owns its per-zone public gateways (IBM allows only one gateway per zone per VPC, so the adopters attach to the owner’s). The owner must be destroyed last — tear the adopters down first. roksbnkctl enforces this: cluster down / down refuses to destroy a workspace whose VPC still holds another cluster’s subnets, naming them, rather than failing mid-destroy with The VPC is in use. (Since v1.26.1; subnets also attach their gateway inline now, so an adopter’s teardown no longer errors even if the owner’s gateway is already gone.)

cluster-outputs.json — the cluster identity record

When roksbnkctl cluster up apply succeeds, it writes a workspace-scoped JSON file describing the cluster. Most of it comes from Terraform outputs (cluster name, ID, region, RG, VPC, registry COS); the create-time settings — network_mode, and vpc_cidr when this tool created the VPC — come from the workspace config instead, because that is the only place they are knowable and the moment they become settled:

~/.roksbnkctl/<workspace>/cluster-outputs.json

Sample contents:

{
  "cluster_name": "bnk-quickstart",
  "cluster_id": "cre6h4l20jjsg4kvt3a0",
  "region": "us-south",
  "resource_group_id": "abc123...",
  "vpc_id": "r006-...",
  "schema_version": 2,
  "network_mode": "single-nic",
  "vpc_cidr": "10.242.0.0/16",
  "registry_cos_crn": "crn:v1:bluemix:public:cloud-object-storage:global:a/...",
  "registry_cos_name": "bnk-quickstart-cos-instance",
  "master_url": "https://c106.us-south.containers.cloud.ibm.com:31415",
  "openshift_version": "4.14_openshift",
  "source": "cluster-up",
  "recorded_at": "2026-05-08T14:22:08Z"
}

The source field discriminates between cluster-up (we created it) and cluster-register (we discovered an existing cluster — see Chapter 9). Subsequent commands read this file to learn the workspace’s cluster identity without hitting IBM APIs.

Reading this file from your own tooling

The file is a versioned contract, and the rules for reading it matter more than the field list, because every rule exists to keep a file written by one version readable by another.

  • schema_version absent means 1. Every record written before the field existed describes a real cluster and must stay readable; treating a missing value as 0 would invalidate all of them.
  • network_mode absent means single-nic. That is what every cluster built before the field existed actually is.
  • vpc_cidr absent means unknown, not “no block was set.” It is recorded only when this tool created the VPC; on the adopt path the setting never applied, so recording the configured value would record something that was never true.
  • node_interfaces is declared but written by nothing today. Multi-NIC ROKS has not shipped, so no code path can populate it honestly yet. Empty means unknown — never “this cluster has no extra interfaces”. Both readings are true of every cluster that exists right now, which is exactly what makes the wrong one easy to adopt and expensive to unwind later.
  • Unknown fields are ignored, deliberately. New keys are only ever added, so an older reader keeps working against a newer file.

For the create-time settings, this file — not config.yaml — is the authority. See Chapter 8a for what happens when the two disagree.

roksbnkctl cluster down deletes the file as part of its post-destroy cleanup. roksbnkctl cluster show pretty-prints it for human readers:

roksbnkctl cluster show
workspace:        default
source:           cluster-up
recorded_at:      2026-05-08T14:22:08Z

cluster_name:     bnk-quickstart
cluster_id:       cre6h4l20jjsg4kvt3a0
region:           us-south
resource_group:   abc123...
openshift:        4.14_openshift
master_url:       https://c106.us-south.containers.cloud.ibm.com:31415

vpc_id:           r006-...
registry_cos:     bnk-quickstart-cos-instance
registry_cos_crn: crn:v1:bluemix:public:cloud-object-storage:global:a/...

Worked example: cluster up → kubectl get nodes → cluster down

The cluster-only flow, end to end:

Step 1 — roksbnkctl init

If you don’t have a workspace yet, initialise one. This is the same init flow as the trial path; the cluster commands reuse the workspace’s config.

roksbnkctl init

Step 2 — roksbnkctl cluster up --auto

Provisions the cluster phase only:

roksbnkctl cluster up --auto

Sample output (heavily abridged):

→ terraform plan (cluster phase: deploy_bnk=false forced)
→ Layering user tfvars from ~/.roksbnkctl/default/state-cluster/cluster-phase-override.tfvars (overrides config.yaml-derived values)
→ terraform init
→ terraform apply
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Creating...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [10m elapsed]
  ...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Creation complete after 38m12s
  module.cert_manager.helm_release.cert_manager: Creation complete after 2m11s
  module.testing.tls_private_key.jumphost_shared_key: Creation complete after 0s
  module.testing.ibm_is_instance.tgw_jumphost: Creation complete after 1m48s

  Apply complete! Resources: 36 added, 0 changed, 0 destroyed.

✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json
✓ Wrote /home/you/.kube/config (chmod 0600)
✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`

Roughly 36 resources land — the cluster phase is about half the size of a full BNK trial. Time-to-ready is dominated by the ROKS cluster itself; everything else after the cluster comes up is fast.

Step 3 — verify the cluster works

The post-apply admin kubeconfig is fetched automatically (unless --no-kubeconfig). kubectl get nodes confirms reachability:

kubectl get nodes
# NAME           STATUS   ROLES           AGE   VERSION
# 10.243.0.4     Ready    master,worker   3m    v1.28.6+5e1b9a1
# 10.243.64.4    Ready    master,worker   3m    v1.28.6+5e1b9a1

Or the same thing through the internalised verb:

roksbnkctl k get nodes

roksbnkctl status reports cluster identity + reachability:

roksbnkctl status
Workspace:    default
Region:       us-south
Cluster:      bnk-quickstart  (attach existing)
TF source:    embedded@v1.0.0
Last apply:   2026-05-08 14:22:08 UTC  (3m ago)
Kubeconfig:   /home/you/.kube/config
Cluster:      2/2 nodes ready

Step 4 — (optional) deploy a BNK trial on top

Now that the cluster is up, roksbnkctl up deploys a BNK trial onto it. It reads cluster-outputs.json and reuses the cluster:

roksbnkctl up --auto

See Chapter 10 — Deploying BNK trials for the trial-phase walkthrough. You can run up / down many times against the same cluster — each cycle is ~5 minutes rather than the ~50 minutes of a fresh-cluster run.

Step 5 — roksbnkctl cluster down --auto

Tear down the cluster phase. cluster down is strictly scoped: it refuses with a hard error on any workspace whose trial state is non-empty, so an out-of-order destroy can’t accidentally orphan BNK resources. Destroy the trial first with roksbnkctl bnk down (or roksbnkctl down for both at once); see Chapter 11 for the full refusal catalogue.

roksbnkctl cluster down --auto

Sample output:

→ terraform destroy (cluster phase)
  module.testing.ibm_is_instance.tgw_jumphost: Destroying...
  module.cert_manager.helm_release.cert_manager: Destroying...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Destroying...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still destroying... [5m elapsed]
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Destruction complete after 8m16s

  Destroy complete! Resources: 36 destroyed.

Post-destroy, cluster-outputs.json is deleted. The workspace directory and its config.yaml survive — re-running cluster up against the same workspace re-creates the cluster with the same name and region.

Why split cluster from trial?

Two-phase is the default because the cost of conflating them is concrete. ROKS clusters take 30-50 minutes to provision and bill at roughly $0.30/hour; a BNK trial on top takes 5-10 minutes. Iterating on the trial — different flo versions, different cne_instance shapes, license bundle revisions — happens far more often than iterating on the cluster underneath. Splitting state means a bnk down / bnk up cycle is a five-minute round-trip instead of an hour.

Three scenarios this shape unlocks:

  1. Many BNK trial iterations on one cluster. Run cluster up once, then loop bnk up / bnk down against the same cluster until you’ve covered all the trial permutations. Then cluster down once when you’re finished. This is the headline win of the phase split — see Chapter 10 §“Worked example — iterating on a BNK trial”.

  2. Pre-provisioning for a workshop or demo. You want the cluster ready and warm before the demo starts; you’ll deploy the BNK trial live in front of the audience. cluster up the night before; bnk up during the demo.

  3. Decoupling cluster lifecycle from trial lifecycle. A long-lived cluster used by multiple team members, where one person owns the cluster phase and others own the BNK trials. Cluster-phase outputs live in cluster-outputs.json; trials read it. Each trial can bnk up / bnk down without affecting the cluster.

For workspaces that just want “create a cluster, deploy BNK on it, test, tear it all down”, the unscoped roksbnkctl up / roksbnkctl down are the right verbs — they’re shape-aware composites that drive the cluster + trial steps in the right order without you having to think about it.

Cross-references

The three-phase lifecycle (Cluster / BNK / Testing)

Transcripts in this chapter are illustrative. They show the shape of the output, not byte-exact captures. Resource counts and timings vary with upstream chart releases and IBM Cloud region behaviour.

As of this release, a roksbnkctl workspace is three independent phases rather than two. The durable Cluster phase (the ROKS cluster + cluster VPC + transit gateway + registry COS) sits underneath two independent sibling phases that deploy on top of it: the BNK phase (cert-manager + F5 Lifecycle Operator + CNE Instance + License — everything that talks to the Kubernetes API) and the Testing phase (the jumphost rig — pure IBM VPC, no Kubernetes).

           ┌──────────── 1. CLUSTER ────────────┐  (create OR reuse)
           │  ROKS cluster + cluster VPC + TGW   │  → cluster-outputs.json
           │  + registry COS                     │
           └──────────────────┬──────────────────┘
                     ┌─────────┴─────────┐
                     ▼                   ▼
           ┌──────────────────┐   ┌──────────────────┐
           │ 2. BNK  (k8s)    │   │ 3. TESTING (VPC) │
           │ cert-manager,    │   │ jumphosts (pure  │
           │ flo, cne, license│   │ IBM VPC, no k8s) │
           │ needs: cluster   │   │ needs: cluster   │
           │   (k8s API)      │   │   VPC + TGW only  │
           └──────────────────┘   └──────────────────┘
                └── parallel; independent up/down ──┘

Why three? BNK depends fully on the cluster (its resources live in the cluster and need the Kubernetes API). Testing depends on the cluster only for the network — the cluster VPC it attaches a jumphost to, and the transit gateway it bridges across regions. The two workloads are otherwise unrelated. Splitting them means:

  • You can tear down the BNK trial and keep the jumphosts for the next iteration (roksbnkctl bnk down leaves Testing alone) — and the inverse (roksbnkctl testing down leaves BNK alone).
  • The two unrelated workloads deploy in parallel after the cluster is ready, shaving wall-clock off a fresh up.

testing is not test. This is the one naming collision worth internalising up front:

  • roksbnkctl testing up/down — a provisioning phase. It creates and destroys the jumphost infrastructure (the TGW jumphost, the per-AZ cluster jumphosts, the optional client VPC). Parallel to cluster and bnk. Writes state-testing/.
  • roksbnkctl test — runs validation probes (connectivity, DNS, throughput) against an already-deployed environment. Provisions nothing. See Chapter 20.
  • roksbnkctl test hosts — manages the list of hosts the test probes target. Still no provisioning.

Mnemonic: testing builds the rig; test runs the checks.

The three state directories

Each phase owns its own Terraform state directory so the phases never tangle:

~/.roksbnkctl/<workspace>/
  state-cluster/      # Cluster phase — `cluster up/down`
    terraform.tfstate
    cluster-phase-override.tfvars
  state/              # BNK phase — `bnk up/down` (and `up`/`down`/`plan`/`apply`)
    terraform.tfstate
    bnk-phase-override.tfvars
  state-testing/      # Testing phase — `testing up/down`
    terraform.tfstate
    testing-phase-override.tfvars
  cluster-outputs.json   # the cluster's identity — the handoff to BNK + Testing

The BNK phase keeps the original state/ directory it has always used (this is deliberate — see §“Migration” below); the jumphosts now live in the new state-testing/. Naming is slightly asymmetric (state/ for BNK vs the explicit state-cluster/ / state-testing/) for exactly one reason: keeping state/ as BNK means existing workspaces don’t have to migrate their BNK resources, only evict the jumphosts.

cluster-outputs.json — the handoff both siblings read

The Cluster phase writes cluster-outputs.json on success (or cluster register writes it for a pre-existing cluster). Both BNK and Testing read it to learn the cluster’s identity without re-specifying it:

  • BNK reads the cluster id/name (to attach to the existing cluster), the VPC id, and the cluster’s network mode and contract schema version — which it checks against the BNK release it is about to install, before it plans anything (see Chapter 28).
  • Testing reads the cluster VPC id (the jumphost attaches to it), the cluster id/name, and the transit gateway name (the Testing module looks the gateway up by name to bridge the client VPC to the cluster).

The transit-gateway name is recorded alongside the id specifically so the Testing phase can run standalone against a registered cluster.

Which side wins

For most settings config.yaml is the input and the record is just an echo of the result. For create-time settings the relationship inverts: the record describes what the cluster is, and the config is a request that has to agree with it. A cluster is never converted in place, so a contradiction is either a mistake or a request for a different cluster.

That gives three cases, and the middle one is the one worth knowing:

cluster.network_modevs the recordresult
set, and matchesagreesproceeds
unsetcannot disagreedefers to the record
set, and differscontradictsrefused, before planning

Silence is not an assertion. An unset network_mode means the config has no opinion, not that it insists on single-nic — so it defers to whatever the cluster actually is. This matters because a config.yaml is not always hand-written and durable: the BNK Forge modules regenerate one per step from a curated environment, so a mode set by the step that created the cluster is absent by the time the step that installs BNK runs. Reading that absence as a demand for single-nic would refuse a correct deployment at its second step, over a contradiction nobody expressed.

The refusal in the third row is what stops terraform from planning a replacement of a running cluster — which it will do without complaint, rendered as output that reads like an update.

Parallel up: Cluster first, then BNK || Testing

A fresh roksbnkctl up provisions the cluster serially first (both siblings depend on it), then launches BNK and Testing concurrently:

roksbnkctl up --auto
→ Cluster phase (serial; both BNK and Testing depend on it)
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Creating...
  ...
  Apply complete! Resources: 36 added.
✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json

→ BNK || Testing (parallel)
  [bnk]     → terraform apply
  [testing] → terraform apply
  [testing] module.testing.ibm_is_instance.tgw_jumphost: Creating...
  [bnk]     module.cert_manager.helm_release.cert_manager: Creating...
  [testing] module.testing.tls_private_key.jumphost_shared_key: Creation complete
  [bnk]     module.flo.helm_release.flo: Creating...
  [testing] Apply complete! Resources: 9 added.
  [bnk]     Apply complete! Resources: 32 added.
✓ Auto-registered target jumphost (169.45.91.177)

The two parallel phases’ output is line-prefixed ([bnk] / [testing]) so you can follow each independently. The Cluster phase, being serial, prints unprefixed.

Without --auto, up plans both phases first (sequentially, so each diff is cleanly attributed), then asks a separate confirmation for each — Apply BNK plan? [y/N] and Apply Testing plan? [y/N]. Two concurrent applies can’t each own an interactive prompt on one terminal, so the approval is taken up front; the approved phases then apply in parallel. Approving only one (e.g. y to BNK, N to Testing) brings up just that phase — handy when you want to redeploy the BNK trial without touching the jumphosts. A phase whose plan shows no changes is skipped without prompting.

To review a change out-of-band before applying — save the full plan to a file, get sign-off, then apply exactly that plan — use the dissociated roksbnkctl plan --out <file> / roksbnkctl apply --plan <file> flow (see Chapter 10 — Deploying BNK trials, §“Reviewing a plan before applying”).

Why Cluster-completes-first rather than starting Testing the moment the cluster VPC exists? The ROKS cluster create dominates wall-clock (~30-50 min); Testing is a handful of VSIs (minutes). Starting Testing against a half-built cluster invites data-source races for a few minutes’ savings. Cluster-first is simpler and correct.

Per-phase up / down

Each phase has its own command pair, all sharing the same flag model (--auto, --var-file, …):

PhaseUpDownState
Clusterroksbnkctl cluster uproksbnkctl cluster downstate-cluster/
BNKroksbnkctl bnk uproksbnkctl bnk downstate/
Testingroksbnkctl testing uproksbnkctl testing downstate-testing/

bnk up and testing up each bootstrap the Cluster phase first if the workspace has no cluster yet (with a confirmation prompt — the cluster take ~30 min). Against an existing cluster they go straight to their own apply.

bnk down leaves the jumphosts (and the inverse)

The headline capability of the split: the two siblings tear down independently.

# Iterate on BNK while keeping the jumphost rig warm:
roksbnkctl bnk down      # destroys cert-manager/flo/cne/license; Testing untouched
roksbnkctl bnk up        # redeploy BNK against the same cluster + same jumphosts
✓ BNK phase destroyed. Cluster phase state-cluster/ and Testing phase
  state-testing/ are intact. Run `roksbnkctl bnk up` to redeploy.

And the inverse — destroy the jumphost rig but keep BNK running:

roksbnkctl testing down   # destroys the jumphosts; cluster + BNK untouched

Because each phase owns a separate state directory, bnk down physically cannot touch state-testing/, and testing down cannot touch state/.

Jumphost sizing, and why it is worth checking once

testing up picks the jumphost instance profile itself unless resources.testing_jumphost_profile names one. The intent is the smallest profile meeting testing_min_vcpu_count (4) and testing_min_memory_gb (8) — these VMs run curl, iperf3 and a small echo server, nothing more.

Through v1.63.0 it picked the largest instead. The selection sorted profile names, and sort() is lexicographic, so among the eligible bx2 profiles

"bx2-128x512" < "bx2-16x64" < "bx2-2x8" < "bx2-32x128" < "bx2-4x16"

and the first element was a 128 vCPU / 512 GB machine — roughly $6/hour each, about $440/day for a three-zone cluster jumphost set. Fixed in v1.64.0.

If you already have oversized jumphosts

Check before upgrading:

ibmcloud is instances --output json | jq -r '.[] | select(.name|test("jumphost")) | "\(.name) \(.profile.name)"'

If they are bx2-128x512 (or anything larger than you expect), recreate them rather than resizing in place:

roksbnkctl testing down
roksbnkctl testing up

From v1.64.0 new jumphosts pin total_volume_bandwidth to 1000 Mbps (testing_jumphost_total_volume_bandwidth), which keeps a later downsize possible. Jumphosts created before that release do not carry the pin, and for them resizing in place is the tempting move and it does not work. total_volume_bandwidth is set by the API from the profile at creation and carried in state, so shrinking the profile is rejected —

instance's total volume bandwidth 20000Mbps (specified by total_volume_bandwidth)
must not be greater than max volume bandwidth 3500Mbps

— and the apply stops the VMs before it fails, leaving them stopped on the old profile. If you are already in that state, recover out of band and then re-run testing up, which will plan no further change:

ibmcloud is instance-update <vm> --profile bx2-2x8 --total-volume-bandwidth 1000
ibmcloud is instance-start <vm>

Reuse an existing cluster

If you already have a ROKS cluster (yours or a teammate’s), skip the Cluster phase entirely — register it, then deploy BNK and Testing on top:

roksbnkctl cluster register existing-bnk-cluster
roksbnkctl up --auto        # Cluster phase skipped; BNK || Testing deploy on the registered cluster
# or per-phase:
roksbnkctl bnk up --auto
roksbnkctl testing up --auto

cluster down is a no-op there — the cluster wasn’t yours to destroy.

Teardown ordering and the cluster down guard

Destroy in reverse of create: the siblings first (in parallel, they’re independent), then the cluster.

roksbnkctl down --auto
This will destroy the BNK, Testing, AND Cluster phases for workspace "default".
Continue? [y/N]: y

→ BNK || Testing (parallel destroy)
  [bnk]     Destroy complete! Resources: 32 destroyed.
  [testing] Destroy complete! Resources: 9 destroyed.
→ Cluster phase (destroy)
  Destroy complete! Resources: 36 destroyed.

The bare down takes one composite confirmation naming every present phase, then runs the parallel sibling-destroy followed by the cluster destroy — you can’t accidentally confirm one phase and miss another.

roksbnkctl cluster down refuses while BNK or Testing state still has resources — they reference the cluster VPC/TGW, so destroying the cluster out from under them would orphan resources:

$ roksbnkctl cluster down
BNK and Testing state exist in this workspace; run `roksbnkctl bnk down` and
`roksbnkctl testing down` first (or `roksbnkctl down` to tear down all phases)

This is a correctness guard, not a prompt — --auto does not bypass it. cluster-outputs.json is deleted only on cluster down (it’s the cluster’s identity); bnk down and testing down leave it in place.

Migration from the two-phase layout

Workspaces created before the three-phase split have their jumphosts living in the BNK state (state/, alongside the BNK modules — that’s how the old two-phase “trial” worked). The three-phase split needs those jumphosts in state-testing/ instead. Three cases:

  1. Fresh / empty workspaces — nothing to migrate. The first up lays out all three state directories cleanly.

  2. Two-phase split workspaces (the common case) — the jumphosts need to move out of state/ into state-testing/ without destroying the live jumphosts. The recommended path preserves them:

    roksbnkctl testing migrate    # terraform state mv: jumphosts → state-testing/
    

    This terraform state mvs every module.testing.* resource (including the shared SSH key) from state/ into a fresh state-testing/, with no cloud churn — the jumphosts keep their IPs and your known_hosts stays valid. A subsequent bnk up reconciles the now-jumphost-free BNK state (it plans no jumphost create), and testing up adopts the moved resources (in-place, not create).

    If a workspace is in an awkward partial-apply state, the bulletproof fallback is a full roksbnkctl down then roksbnkctl up, which re-creates everything in the three-state layout. This does churn the jumphosts (new IPs, known_hosts reset), so the state mv path is preferred whenever it’s available.

roksbnkctl nudges you when it detects jumphosts still living in the BNK state with an empty state-testing/ — it won’t silently rewrite your state.

The optional Gateway phase (gateway up / down)

The BNK phase brings up a licensed, running control plane and now also creates the cloud-network-mapping ConfigMap and the external/internal VLANs that TMM needs to program its data plane. The rest of the install guide’s “Configuration” section — the Gateway API objects, egress SnatPool/Egress, the per-zone static routes, and the cluster security-group VXLAN rule — is a separate, optional phase with its own state (state-gateway/):

roksbnkctl up                  # Cluster → BNK ∥ Testing  (Gateway NOT included)
# … verify BNK is healthy: TMM pods Ready, CNEInstance Available …
roksbnkctl gateway up          # apply the data-plane config
roksbnkctl gateway down        # remove it, leaving cluster/BNK/testing intact

It is never run by the composite up/down — the Gateway config needs a healthy BNK (its F5SPK* CRDs ship with the BNK manifest, and TMM must be up), so it is strictly opt-in. It reuses the existing cluster via cluster-outputs.json (like the Testing phase), forces every other phase’s creation off in its override, and manages only its own resources. Defaults all come from the BNK 2.3 install guide; override the HTTPRoute backend, client subnets, egress mode (snatpool/automap/both) and VXLAN port via the config.yaml gateway block. cluster down refuses while the Gateway phase has resources, so tear it down first (gateway down).

The VXLAN security-group rule, and what it opens

The gateway phase adds one inbound rule to the cluster’s own worker security group (kube-<cluster-id>), permitting the egress VXLAN port (UDP 6789 by default, gateway.vxlan_port).

This rule is required, not hardening. TMM answers a remote node’s VXLAN from its external-VLAN self-IP, and the workers’ security group does not otherwise admit that source — so without the rule, egress from any node that is not itself running a TMM fails.

Through v1.63.0 the rule permitted 0.0.0.0/0. From v1.64.0 it is scoped to the per-zone external-VLAN CIDRs (resources/bnk zone config ext_vlan_cidr), one rule per CIDR, because those are the only addresses a TMM self-IP can have. That matters because this rule sits on a security group shared by everything else running on those workers.

Upgrading replaces the rule rather than editing it. The resource moved from count to for_each, so terraform removes the single open rule and creates the scoped ones; ordering is not guaranteed, so run gateway up when a few seconds of egress disruption is acceptable.

If every zone’s ext_vlan_cidr is empty, gateway up now fails at plan time rather than silently creating no rule and leaving egress broken with nothing naming the cause.

Cross-references

Registering an existing cluster

Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the Workspace struct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.

roksbnkctl cluster register <name> wires roksbnkctl up to a ROKS cluster that already exists in your IBM Cloud account — one you didn’t provision via cluster up. After a successful register, the workspace behaves exactly as if you’d done cluster up: roksbnkctl up deploys BNK trials onto the registered cluster, roksbnkctl down tears those trials down, roksbnkctl status reports the cluster’s identity, and so on.

This chapter covers when registration is the right answer, what input is required vs auto-discovered, the COS naming convention, the cluster-outputs.json write, and a worked example.

roksbnkctl init can register for you. Answering no to Create a new ROKS cluster? in the init interview lists the account’s running OpenShift clusters and registers the one you pick — it performs the same cluster-outputs.json write described here, inline. Use cluster register <name> directly when you want to (re)register without re-running the full interview, or when scripting.

cluster up attaches when cluster.create=false. You don’t need a separate cluster register call: if the workspace’s cluster.create is false, roksbnkctl cluster up attaches to the existing cluster named in cluster.name (the same discovery + cluster-outputs.json write) instead of running the terraform create. So one command — and one toggle in an automation/CI form — covers both “make a new cluster” and “use the existing one”. Set it non-interactively with ROKSBNKCTL_CLUSTER_CREATE / ROKSBNKCTL_CLUSTER_NAME (see Chapter 7a).

When to use this

cluster register is the answer when all of these are true:

  • A ROKS cluster already exists in the IBM Cloud account.
  • You have IAM access to the cluster’s VPC + container service.
  • You want roksbnkctl to deploy BNK trials onto that cluster.
  • You don’t want roksbnkctl to own the cluster’s lifecycle (it shouldn’t be terraform destroy-able from your workstation).

Common scenarios:

  1. Your team operates the ROKS cluster centrally. A platform team provisioned the cluster via their own Terraform / Pulumi / IBM Cloud Schematics; you just want to deploy BNK trials onto it. Register it; deploy trials; tear them back down. The cluster itself stays under the platform team’s ownership.

  2. You’re attaching to an existing demo cluster. A workshop hosts a shared cluster that participants attach to. Each participant registers it in their own workspace and deploys their own trial — trials are isolated by namespace under the same cluster.

  3. You provisioned the cluster manually for testing. You created a one-off cluster via ibmcloud ks cluster create vpc-gen2 ... and want to move forward with roksbnkctl rather than re-creating it.

If none of those apply — i.e. you want roksbnkctl to own cluster lifecycle end-to-end — use cluster up instead. Register and cluster up are mutually exclusive per workspace; the second one wins.

Required input vs auto-discovery

cluster register takes one positional argument (the cluster name or ID) and one optional flag (--registry-cos-name).

roksbnkctl cluster register <cluster-name-or-id> [--registry-cos-name <cos-instance-name>]

Everything else is auto-discovered via the IBM SDK:

FieldSource
cluster_idibmcloud ks cluster get <name> (resolved by name → ID)
regionfrom the cluster lookup
resource_group_idfrom the cluster lookup
vpc_idfrom the cluster’s provider.vpcs[0].id
master_urlfrom the cluster lookup
openshift_versionfrom the cluster’s masterKubeVersion
registry_cos_crndiscovered via the registry COS instance lookup (see below)

The cluster lookup goes through the same container-service endpoint ibmcloud ks cluster get uses — no host ibmcloud install required. If the named cluster doesn’t exist in the account, the call returns a clear no cluster named <foo> error rather than a 404 stack trace.

A vpc-gen2 cluster is required. Classic infrastructure clusters return successfully but their vpc_id is empty, and cluster register refuses to write a record without one:

Error: cluster "old-classic" has no VPC — roksbnkctl only supports vpc-gen2 clusters

The COS naming convention

roksbnkctl up needs a Cloud Object Storage instance to act as the registry for FAR images, JWT licenses, and schematic state. cluster register verifies that this COS instance exists at registration time so a later up doesn’t fail mid-apply with a missing-instance error.

Default convention

The bundled HCL falls back to <cluster-name>-cos if the user’s tfvars don’t override roks_cos_instance_name. So cluster register defaults to looking up <cluster-name>-cos:

# Cluster name: "canada-roks" → expects COS instance "canada-roks-cos"
roksbnkctl cluster register canada-roks

Override with --registry-cos-name

If your team set roks_cos_instance_name to something else in their tfvars (or named the COS instance via the IBM Cloud console with a different convention), pass --registry-cos-name <name>:

roksbnkctl cluster register canada-roks \
  --registry-cos-name canada-roks-bnk-registry

The instance name is case-sensitive and must match exactly — Canada-ROKS-COS and canada-roks-cos are different instances.

What if the COS doesn’t exist yet?

cluster register errors out:

Error: registry COS instance "canada-roks-cos" not found in account: ...
  Either run `roksbnkctl cluster up` to create it, or pass --registry-cos-name <name>
  if your tfvars uses a different roks_cos_instance_name

You have two options:

  1. Create the COS instance in the IBM Cloud console with the conventional name (<cluster>-cos), then re-run register. The instance can be empty — roksbnkctl up will populate it with the bucket structure it needs on its first apply.

  2. Use a different name that already exists in the account, via --registry-cos-name <name>.

Either way, cluster register won’t write cluster-outputs.json until both the cluster and its registry COS instance exist.

The cluster-outputs.json write

On success, cluster register writes ~/.roksbnkctl/<workspace>/cluster-outputs.json — the same file cluster up writes. The contents look nearly identical; the differences are source, and what is deliberately left empty:

{
  "cluster_name": "canada-roks",
  "cluster_id": "cre6h4l20jjsg4kvt3a0",
  "region": "ca-tor",
  "resource_group_id": "abc123...",
  "vpc_id": "r038-...",
  "schema_version": 2,
  "network_mode": "single-nic",
  "registry_cos_crn": "crn:v1:bluemix:public:cloud-object-storage:global:a/...",
  "registry_cos_name": "canada-roks-cos",
  "master_url": "https://c106.ca-tor.containers.cloud.ibm.com:31415",
  "openshift_version": "4.14_openshift",
  "source": "cluster-register",
  "recorded_at": "2026-05-08T14:22:08Z"
}

The source field is cluster-register (vs cluster-up for self-provisioned clusters). Downstream commands that care about provenance — for example, a future roksbnkctl cluster down would refuse to destroy a cluster-register-sourced cluster — read this field. Subnet IDs (subnet_ids) and transit gateway ID (transit_gateway_id) are left blank for registered clusters; the bundled HCL doesn’t need them when roksbnkctl up runs against a pre-existing cluster.

vpc_cidr is also left empty here, and that is deliberate. It records the block this tool used to create a VPC. On the adopt path the setting never applied, so writing the configured value would record something that was never true of the VPC.

That has one consequence worth knowing: because the create-time warning for cluster.vpc_cidr fires only on a disagreement with a recorded value, a workspace built by cluster register will never emit it — there is nothing recorded to disagree with. Silence there means “not knowable”, not “your value is fine”. See Chapter 26.

network_mode, by contrast, is recorded — from the config, since an adopted cluster’s mode cannot yet be detected from the IBM API. On a single-NIC cluster (every cluster today) the default is correct and nothing needs saying. When multi-NIC ROKS ships, adopting one will mean setting cluster.network_mode to match what the cluster is, not what you want — the opposite of how the same field reads on cluster up.

Worked example: register canada-roks

The full flow for attaching to a hypothetical canada-roks cluster.

Step 1 — create or pick a workspace

roksbnkctl ws new canada
roksbnkctl init -w canada
# (interactive — fill in region as ca-tor; cluster.name = canada-roks)

You can also run cluster register against the current workspace; the -w is just for clarity.

Step 2 — cluster register

roksbnkctl -w canada cluster register canada-roks

Sample output:

→ Looking up cluster "canada-roks"
✓ Cluster canada-roks (cre6h4l20jjsg4kvt3a0) — state: normal, masters: 4.14_openshift
✓ VPC r038-... (resource group prod-rg)
→ Verifying registry COS instance "canada-roks-cos"
✓ COS instance canada-roks-cos (abc-123-def-...)
✓ Wrote ~/.roksbnkctl/canada/cluster-outputs.json

If the COS naming was non-conventional:

roksbnkctl -w canada cluster register canada-roks \
  --registry-cos-name canada-bnk-registry

Step 3 — verify with cluster show

roksbnkctl -w canada cluster show
workspace:        canada
source:           cluster-register
recorded_at:      2026-05-08T14:22:08Z

cluster_name:     canada-roks
cluster_id:       cre6h4l20jjsg4kvt3a0
region:           ca-tor
resource_group:   abc123...
openshift:        4.14_openshift
master_url:       https://c106.ca-tor.containers.cloud.ibm.com:31415

vpc_id:           r038-...
registry_cos:     canada-roks-cos
registry_cos_crn: crn:v1:bluemix:public:cloud-object-storage:global:a/...

Step 4 — fetch the kubeconfig

cluster register does not automatically download the kubeconfig — it’s a metadata-only operation. Grab it explicitly:

roksbnkctl -w canada kubeconfig --download
# → Fetching admin kubeconfig for "canada-roks"
# ✓ Wrote /home/you/.kube/config (12345 bytes)

Step 5 — use the cluster as if you’d done cluster up

From here, the workflow is identical to a self-provisioned cluster:

# Verify reachability
roksbnkctl -w canada k get nodes

# Deploy a BNK trial onto it
roksbnkctl -w canada up --auto

# Tear the trial back down (cluster survives)
roksbnkctl -w canada down --auto

roksbnkctl up reads cluster-outputs.json and uses the cluster identity directly — no need to re-state cluster name/region/RG in the trial’s tfvars.

When register isn’t enough

Some scenarios where cluster register won’t get you over the line:

  • The cluster is in a different IBM Cloud account. API keys are account-scoped; you’d need a key for the cluster’s account. cluster register doesn’t cross account boundaries.
  • The cluster is private (no public master endpoint). roksbnkctl up needs to apply Helm charts and Kubernetes manifests against the master. If the master is only reachable from inside a VPN, route the apply through --on jumphost or wait for the SSH execution backend in Sprint 4.
  • The cluster is a classic-infrastructure ROKS (not vpc-gen2). Registration refuses; classic clusters aren’t supported.
  • The cluster’s worker pool is too small. BNK trials need at least 2 workers with adequate CPU/memory. The upstream HCL provisions appropriately-sized workers; an existing cluster might not.

For the first three, the cluster isn’t a candidate. For the last one, the apply may run but flo / cne_instance will fail to schedule — scale the worker pool first.

Re-registering and unregistering

To re-register with new data (e.g. you renamed the COS instance, or the master URL changed), just run cluster register again — it overwrites cluster-outputs.json in place.

To unregister without destroying anything, delete the file directly:

rm ~/.roksbnkctl/canada/cluster-outputs.json

The workspace’s config.yaml and state/ survive; only the cluster identity record is removed. The next roksbnkctl up will fail with workspace has no cluster-outputs.json until you either re-register or run cluster up.

There’s deliberately no roksbnkctl cluster unregister command. Deleting the JSON is a single-file operation that doesn’t deserve its own subcommand, and the absence of one nudges users toward “destroy the trial first, then deal with the cluster identity” rather than “unregister without thinking about the consequences”.

Cross-references

Sharing a Transit Gateway across clusters

A Transit Gateway (TGW) routes between VPCs. By default roksbnkctl cluster up creates a prefix-named gateway for the cluster it builds. But you often want the opposite: one gateway that several clusters share, so they can reach each other — the shape behind a shared licensing cluster, a central services VPC, or cross-cluster testing.

roksbnkctl attaches a cluster’s VPC to an existing Transit Gateway — by name or by id, at create time or after — as its own small phase. Each workspace owns its own connection, so N clusters can point at the one gateway.

First: give each cluster VPC its own address block

Decide this before cluster up, because it cannot be changed afterwards.

A Transit Gateway routes on VPC address prefixes. Two attached VPCs claiming the same block make routing ambiguous, and the gateway resolves that by silently dropping traffic for one of them. Nothing logs an error. What you see instead is intermittent image-pull timeouts against a mirror while every security group and network ACL in the path plainly allows the traffic — which is why this is worth a minute up front.

It is easy to hit by accident. IBM’s default address prefix management is auto, which assigns every VPC in a region the same three per-zone prefixes:

10.241.0.0/18   10.241.64.0/18   10.241.128.0/18

So a second roksbnkctl-created cluster joining a shared gateway collides by construction, not by bad luck. Give each one a distinct block:

cluster:
  create: true
  name: acme-eu-roks
  vpc_cidr: 10.242.0.0/16     # this cluster's own block

or ROKSBNKCTL_CLUSTER_VPC_CIDR=10.242.0.0/16 for CI. The block is split into three per-zone prefixes (a /16 becomes three /18s), so /18 is the smallest usable value. Leaving it blank keeps IBM’s auto assignment, and the default 10.241.0.0/16 produces byte-identical prefixes to what auto gives today — so setting it explicitly on a first cluster changes no addresses.

A worked allocation for a shared gateway:

VPCvpc_cidrPer-zone prefixes
services (Harbor, FLP)existing, 10.243.0.0/24—
first cluster10.241.0.0/1610.241.0.0/18, .64.0/18, .128.0/18
second cluster10.242.0.0/1610.242.0.0/18, .64.0/18, .128.0/18

roksbnkctl checks this for you. cluster up refuses before terraform creates anything when the VPC it is about to build would overlap a VPC already on the gateway, and tgw connect refuses before attaching an existing VPC that would overlap — each naming the other VPC and the colliding prefixes. If the check cannot reach the API it warns and continues rather than blocking the build.

Already built two overlapping clusters? The prefixes of a live VPC can’t be edited — moving a subnet’s CIDR replaces the subnet, which destroys the cluster on it. Either keep only one of them attached at a time (tgw disconnect on the other; the cluster itself survives), or rebuild one with vpc_cidr set.

Attach when creating (the interview)

Decline to create a gateway, and init discovers the transit gateways already in your account and lets you pick one by number — name, location, and status — so you don’t have to remember a name or id (since v1.25.0):

$ roksbnkctl -w prod init
...
Create Transit Gateway? [Y/n] n
→ Discovering existing transit gateways...
  Existing transit gateways in this account:
     1) shared-tgw            (us-south, available)
     2) prod-tgw-eu           (eu-de, available)
Attach an existing Transit Gateway — pick a number (0 = none / attach later): 1

Picking 0 (or if the account has none) leaves the cluster unattached, to connect later with tgw connect; if the discovery call fails it falls back to a free-text name/ID prompt.

cluster up (or cluster register) then attaches the cluster’s VPC to shared-tgw automatically — no extra step. In config that is:

resources:
  transit_gateway:
    create: false
    existing: shared-tgw      # a NAME or an ID — either is resolved

or, for CI, the environment variable ROKSBNKCTL_TRANSIT_GATEWAY_NAME=shared-tgw (it too accepts a name or an id).

Attach (or detach) after the fact

tgw connect works on a cluster that already exists — one roksbnkctl built, or one you registered — because it reads the cluster’s VPC from cluster-outputs.json, not from any terraform state:

roksbnkctl -w prod tgw connect shared-tgw       # by name
roksbnkctl -w prod tgw connect 2f28a749-fcc0-…  # or by id

Run it in several workspaces with the same gateway and each cluster gets its own connection to it:

roksbnkctl -w app-a tgw connect shared-tgw
roksbnkctl -w app-b tgw connect shared-tgw      # same gateway, second cluster

Detach removes only this cluster’s connection — the gateway and every other cluster’s connection stay:

roksbnkctl -w prod tgw disconnect

Status — the connected state, id, and name

tgw status shows the gateway identity and the live connection state, queried from IBM (not just what was recorded at apply time):

$ roksbnkctl -w prod tgw status
transit_gateway_id:    2f28a749-fcc0-43d7-97dd-403a6b00b56f
transit_gateway_name:  shared-tgw
connection_name:       prod
connection_id:         a1b2c3d4-…
cluster_vpc:           r010-072ad88a-…
connection_state:      attached

connection_state is the live attachment: attached, pending, or detached (no connection for this VPC on the gateway). roksbnkctl -w prod cluster config shows the same block alongside the rest of the cluster identity.

How it works, and what to know

  • It’s its own phase (state-tgw/), separate from cluster / BNK / testing / gateway / FLP. So tgw connect/disconnect never touch anything else, and the connection survives a bnk down or a re-applied trial.
  • Name or id is resolved against your account’s gateway list, so an ambiguous name (two gateways sharing it) is an error, not an arbitrary pick — pass the id.
  • One VPC, one connection per gateway. Clusters that share a VPC (rare) can’t each hold a separate connection to the same gateway; clusters in distinct VPCs each get their own, which is the normal shared-gateway case.
  • A global gateway connects VPCs in any region; a local one only its own region. tgw connect attaches to whatever gateway you name — make sure it can reach the region your cluster VPC is in.
  • cluster down refuses while a connection exists, so the cluster isn’t destroyed out from under its own TGW attachment — run tgw disconnect first. roksbnkctl down (the composite) auto-detaches for you (since v1.26.0): it removes this cluster’s connection after the BNK/Testing teardown and before the cluster, because the connection pins the VPC’s CRN and the VPC delete would otherwise fail (VPC still has an attached transit gateway connection). Only this cluster’s connection is removed — the shared gateway and every other cluster’s connection stay.

Deploying BNK trials on top

roksbnkctl up deploys a BNK trial — F5’s Lifecycle Operator, the CNE Instance, license bundles, and the cluster-side glue that makes them work — onto a ROKS cluster that already exists. “Already exists” means either provisioned by cluster up or registered from a pre-existing cluster.

For workspaces where the cluster and the trial are managed as separate phases (the default — see Chapter 8), the trial layer also gets its own command pair: roksbnkctl bnk up / bnk down. bnk down tears down only the trial; the cluster keeps running, so the next iteration starts in 5-10 minutes instead of an hour. The bnk group is documented in §“The bnk up / bnk down command group” below.

This chapter is the deeper-than-quick-start view of up: what each module does, the ~77-resource shape of a clean apply, the token-rotation observation when you re-run up against an existing cluster, how to read the Terraform plan output, and how the bnk group + the shape-aware composite up / down fit together.

Chapter 7 — Quick start shows the happy path end-to-end with sample output. This chapter goes deeper.

What “deploying BNK” means

A BNK trial is a deliberately small set of Kubernetes resources that share state with a cluster-shared cert-manager and a cluster-scoped registry COS. The components that roksbnkctl up is responsible for landing:

ComponentWhat it isModule in the bundled HCL
floF5 Lifecycle Operator — the controller that watches CNE Instance CRs and reconciles them into running BIG-IP Next podsmodule.flo (helm_release)
cne_instanceThe CR that declares “I want a BIG-IP Next data plane here” — drives flo to provision the TMM podsmodule.cne_instance (kubectl_manifest + wait_for)
licenseThe License CR (JWT + operation mode) that gates BNK’s runtime — sourced from the registry COSmodule.license (kubectl_manifest + wait_for)
cluster-side bitsNamespaces, Secrets, ServiceAccounts, RoleBindings, SCC bindings, NADs, the cert-manager issuer chainkubernetes_namespace/kubernetes_secret + kubectl_manifest, across the modules above

up does not own the cluster, cert-manager’s chart, the registry COS, or the jumphost — those are cluster-phase resources. See Chapter 8 for the split.

The BNK phase is terraform-native. Chart installs are helm_release (wait = true); the custom resources are kubectl_manifest (from the alekc/kubectl provider) with wait_for blocks that watch the resource’s real .status; namespaces and Secrets are the kubernetes provider. The CRs are real Terraform state — plan diffs them, destroy deletes them (finalizer-aware), and drift is detected. See §“The terraform-native deployment model” below.

The 77-resource shape

A clean roksbnkctl up against a fresh cluster lands roughly 77 resources when the cluster phase is bundled in (i.e. cluster up and up were one combined run). Against a pre-existing cluster (cluster up then up), the trial-only count is smaller — roughly the difference, ~41 resources.

The number isn’t load-bearing; it shifts a few resources up or down between upstream chart releases as the charts add/remove Secrets and kubectl_manifest CRs. Treat “77” as a sanity-check tag, not a contract.

A representative breakdown:

Cluster phase (~36 resources, owned by `cluster up`)
  ROKS cluster + worker pools          ~5
  VPC + subnets + security groups       ~6
  Transit gateway + connections          ~4
  Registry COS instance + bucket          ~3
  cert-manager Helm release               ~2
  TGW jumphost VSI + cloud-init         ~16

Trial phase (~41 resources, owned by `roksbnkctl up`)
  flo + cis helm_release                 ~4
  cne_instance kubectl_manifest          ~1
  license kubectl_manifest               ~1
  namespaces + secrets (kubernetes)      ~5
  cert issuer chain (kubectl_manifest)   ~3
  NADs (kubectl_manifest)                ~2
  SCC / node-labeler kubectl_manifest   ~20
  IBM IAM trusted profile + COS reads   ~5

The kubectl_manifest resources are ordinary Terraform state: a re-run plan shows them as no-ops unless their spec changed — see §“Re-running up is a no-op”.

Apply timing

A clean up against a fresh cluster takes ~50 minutes:

  • ROKS cluster provisioning: 30-40 min (the bulk of the wait)
  • cert-manager + flo helm_release (wait = true): ~5 min
  • cne_instance reconcile: 1-2 min (gated on the CNEInstance’s real Available condition, not a sleep)
  • license verification: 1-2 min (gated on the License CR’s real status.state, not a sleep)
  • Cluster-side bits: applied concurrently with the above

Against a pre-existing cluster (already-up’d or registered), the trial-only run is 5-10 minutes. Most of that is helm_release waiting for flo to stabilise and the two CR wait_for blocks resolving.

Readiness is gated on real signals, not fixed sleeps. The terraform-native path uses helm_release wait = true and kubectl_manifest wait_for, which return the instant readiness is observed and no sooner — the CNEInstance’s Available condition and the License CR’s status.state are the actual signals it watches. On a warm cluster this real-readiness gating is the single largest contributor to the fast re-deploy loop.

Re-running up is a no-op

If you re-run roksbnkctl up against an already-deployed BNK trial with no config change, the plan reports 0 to add, 0 to change, 0 to destroy and up skips apply.

Each cluster object is a real Terraform resource:

  • A helm_release knows the installed chart version, so an unchanged version is a no-op.
  • A kubectl_manifest carries the rendered manifest as state, so an unchanged spec is a no-op (and a changed spec shows a precise ~ update in-place diff of exactly the changed fields).
  • A kubernetes_secret is keyed by its data, so an unchanged Secret is a no-op.

So the second up against an unchanged trial genuinely changes nothing. To bump a BNK version or tweak a CNEInstance parameter, edit config.yaml (or a --var-file) and re-run: terraform plan diffs only the changed CNEInstance spec and helm versions — Terraform is the delta engine. Use roksbnkctl plan to preview the diff without applying.

Reading the Terraform plan output

roksbnkctl up runs terraform plan first and prints its output. The plan summary at the end is the most useful part:

Plan: 77 to add, 0 to change, 0 to destroy.

Or, post-rotation:

Plan: 12 to add, 0 to change, 12 to destroy.

The body of the plan shows individual resource changes with one of three markers:

  • + create — a new resource. Lines are green in a TTY.
  • <= read — a data source the plan read but did not change. Common for data "ibm_resource_group" and similar lookups; effectively informational.
  • # destroy — an in-progress destroy of an existing resource. Followed by a + create if it’s being replaced (e.g. a kubectl_manifest whose spec changed in a way that forces replacement).
  • ~ update in-place — a resource whose attributes are being mutated without re-creation.

The <= data sources are the ones that look like:

data "ibm_resource_group" "default" {
  name = "Default"
  id   = "abc123..." (will be read)
}

These are read-only — Terraform is just resolving the resource group’s ID at plan time so downstream modules can reference it. They show up in every plan, including no-op plans.

# destroy lines without a corresponding + create — i.e. resources actually leaving — should make you stop and read carefully. On a re-run of up, this generally means an upstream HCL change removed a resource. It’s rare but not zero.

When up doesn’t apply (no-op runs)

If the plan reports zero changes, up skips apply and prints:

✓ no changes

But it still does two best-effort post-actions:

  1. Fetch the kubeconfig (unless --no-kubeconfig). Useful when the cluster exists but you’ve never grabbed the admin kubeconfig on this workstation.
  2. Auto-register the jumphost target. Reads testing_tgw_jumphost_ip and jumphost_shared_key from Terraform outputs and writes a targets:jumphost entry in workspace config. Re-runs are idempotent.

So roksbnkctl up against an unchanged cluster is a useful “re-establish my workstation’s view of this workspace” verb — it can’t hurt anything (no apply runs), and it freshens local artefacts.

The --auto, --no-kubeconfig, --var-file flags

roksbnkctl up [--auto] [--no-kubeconfig] [--var-file <path>]...
FlagEffect
--autoSkip the “Apply this plan? [y/N]” prompt. Required for non-interactive runs (CI, scripted pipelines).
--no-kubeconfigSkip the post-apply kubeconfig fetch. Useful when you’ve already got a kubeconfig and don’t want it overwritten.
--var-file <path>Layer extra Terraform var-files onto the chain (repeatable; later wins). Lets you parameterise without editing config.yaml.
--tf-source <ref>Override the pinned TF source for this run only. Skip the embedded HCL and use a path or URL instead. Mostly for dev.

--var-file is the canonical way to stage a non-default deploy. For example, deploying a BNK trial with a non-default cne_instance.replicas:

echo 'cne_replicas = 3' > ./more-replicas.tfvars
roksbnkctl up --auto --var-file ./more-replicas.tfvars

The var-file chain is, in order:

  1. The auto-generated terraform.tfvars (rendered from config.yaml).
  2. ~/.roksbnkctl/<workspace>/terraform.tfvars.user if present — sibling to config.yaml, serving both the trial and cluster phases (a manual raw-tfvars override you place at the workspace root — see Chapter 6 §“Raw terraform-variable overrides”).
  3. Each --var-file flag, left-to-right.

Later wins on conflict — same as Terraform’s own ordering. If you find yourself passing the same --var-file on every up / plan / apply / down, drop it once at ~/.roksbnkctl/<ws>/terraform.tfvars.user (sibling to config.yaml) — the lifecycle auto-layers that file on every subsequent call, so you can drop the flag.

Reviewing a plan before applying (plan --out / apply --plan)

plan and apply are separate commands, so you can review a change in full before it touches the cluster — useful for change-control sign-off, or simply when the diff is larger than your terminal scrollback. Save the plan to a file, review it, then apply exactly that plan:

# 1. Save the plan: a binary plan file + a human-readable .txt copy
roksbnkctl plan -w dev --out ./bnk.plan
#   ✓ Saved plan: /…/bnk.plan (reviewable copy: /…/bnk.plan.txt)
#     Review it, then apply EXACTLY this plan:
#       roksbnkctl apply -w dev --plan /…/bnk.plan

# 2. Review bnk.plan.txt (the complete terraform diff — no scrollback limit)

# 3. Apply exactly the reviewed plan — no re-plan
roksbnkctl apply -w dev --plan ./bnk.plan

apply --plan applies the saved plan verbatim: it does not re-plan, and it passes no var-files (the plan already captured every variable). If state or config has drifted since the plan was saved, Terraform refuses the stale plan rather than applying something you didn’t review — that is the change-control guarantee.

Without --plan, a bare roksbnkctl apply re-plans and applies fresh. That is fine for iteration, but it does not guarantee the applied change equals the one you reviewed with plan. Use --out / --plan whenever “what I reviewed is exactly what applied” matters.

The saved-plan flow requires the local execution backend; on a docker or remote backend, --out / --plan error clearly rather than being silently ignored.

Apply retries on transient errors

ROKS master endpoints take 1-5 minutes to fully propagate after the cluster reaches Ready. The helm, kubernetes, and alekc/kubectl providers all talk to the master directly; on a fresh cluster, they sometimes race propagation and fail with a connection error (Connection refused, i/o timeout, TLS handshake timeout).

roksbnkctl up has built-in retry: up to 3 apply attempts, with a 60-second sleep between attempts, on any of these heuristic patterns:

  • exit status 7 (curl couldn’t connect)
  • Connection refused / connection refused
  • i/o timeout
  • no route to host
  • network is unreachable
  • no such host
  • TLS handshake timeout
  • failed to dial
  • to download the config doesn't exist

If your apply hits one of these, you’ll see:

→ apply attempt 1 hit a transient-looking failure; waiting 60s and retrying...

Terraform’s idempotence means already-created resources are skipped on the retry; only the failed resources / data sources re-execute. After 3 attempts, up gives up:

✗ apply still failing after 3 attempts — giving up

At that point, fix the underlying cause (usually wait longer or re-run manually) and try again. The retry is for transient races, not persistent failures.

What happens on success

A successful up does five things in order:

  1. Apply complete. Apply complete! Resources: 77 added, 0 changed, 0 destroyed.
  2. Fetch the admin kubeconfig from IBM Cloud’s container service API. Written to $KUBECONFIG (or ~/.kube/config) at mode 0600.
  3. Auto-register the jumphost target in workspace config (so --on jumphost works without manual config — see Chapter 16).
  4. Stamp terraform.tfstate’s mtime. roksbnkctl status reads this as “last apply” timestamp.
  5. Exit 0.

The kubeconfig fetch and jumphost registration are best-effort: they log warnings on failure but don’t fail the parent command. up succeeded if Terraform succeeded; the post-apply niceties are conveniences.

The bnk up / bnk down command group

The roksbnkctl bnk group is the trial-only counterpart to roksbnkctl cluster — it operates on the trial state under state/ and leaves the cluster state under state-cluster/ untouched. Because it iterates on the trial without rebuilding the cluster, a bnk down / bnk up round-trip is the 5-10 minute trial-apply window, with the cluster running underneath.

roksbnkctl bnk up

Deploys the BNK trial against the workspace’s registered cluster.

  • If the workspace already has a cluster phase (either from cluster up or from cluster register), bnk up runs the trial apply directly — same plan, same ~41 resources, same 5-10 minute window as the trial half of a full up.
  • If the workspace is empty (no cluster registered yet), bnk up offers to bootstrap the cluster phase first with a confirmation prompt, then runs the trial apply. This keeps the new user’s quick-start path one command, even if they typed bnk up instead of up.

Sample output of the bootstrap-prompt path:

$ roksbnkctl bnk up
No cluster registered for this workspace.
→ Provisioning the cluster phase first (ROKS cluster + transit gateway +
  registry COS + cert-manager + jumphost; ~30 min) before the BNK trial.
Continue? [y/N]: y
→ terraform plan (cluster phase: deploy_bnk=false forced)
...
✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json
→ terraform plan (trial phase)
...
Apply complete! Resources: 41 added, 0 changed, 0 destroyed.

Three prompts fire in the empty-workspace case — one for “do you want to bootstrap the cluster phase,” one for “apply this terraform plan” inside the nested cluster up, and a third when the trial-phase apply prompts. (On a non-empty workspace where bnk up skips the cluster bootstrap, only the latter two fire — and a ShapeClusterOnly/ShapeSplit bnk up is the common iteration case.) For a 30-minute operation we kept the prompts explicit rather than collapsing them. --auto skips all three:

$ roksbnkctl bnk up --auto

roksbnkctl bnk down

Destroys the trial only. The cluster phase keeps running.

  • On a split workspace (cluster + trial both present), bnk down runs terraform destroy against the trial state — ~41 resources, the same as the trial half of a full down.
  • On an empty or cluster-only workspace, bnk down is a no-op success (exit 0): there’s no trial to destroy, and an orchestrated teardown runs every phase unconditionally, so failing for having nothing to do would fail a teardown that worked.

Sample output against a split workspace:

$ roksbnkctl bnk down --auto
→ terraform destroy (trial phase)
  module.license.kubectl_manifest.license: Destroying...
  module.cne_instance.kubectl_manifest.cneinstance: Destroying...
  module.flo.helm_release.flo: Destroying...
  ...
  Destroy complete! Resources: 41 destroyed.

✓ Trial phase destroyed. Cluster phase ~/.roksbnkctl/default/state-cluster/ is intact.
  Run `roksbnkctl bnk up` to deploy another trial against the same cluster.

The shape dispatch matrix

The unscoped roksbnkctl up / down verbs are now shape-aware composites — they detect the on-disk shape of the workspace and delegate to the right phase commands underneath. The full picture for all four shapes and all six commands:

CommandEmpty (nothing applied)ClusterOnly (cluster up ran)Split (cluster + trial both applied)
upcluster up → trial uptrial upcluster up (refresh) → trial up
downno-op (exit 0): nothing to destroycluster downtrial down → cluster down
bnk upconfirm + cluster up → trial uptrial uptrial up
bnk downno-op (exit 0): no trialno-op (exit 0): no trialtrial down
cluster upcluster upcluster up (refresh)cluster up (refresh)
cluster downno-op (exit 0): nothing to destroycluster downrefuse: trial exists

The user-facing simplification: the unscoped up / down “just work” against every shape. The phase-scoped commands (bnk, cluster) only operate when the shape allows isolation and refuse loudly with an actionable message otherwise. Refusals always point at the resolution — see Chapter 11 §“Refusal messages” for the full catalogue.

The engineering version of this table — with the implementation details, the ShapeUnknown edge cases, and the rationale — lives in PRD 06 §“Dispatch table”.

Worked example — iterating on a BNK trial

The headline workflow the phase split unlocks. You’re testing different cne_instance parameter combinations against a stable cluster.

# Step 1 — one-time cluster provision (~38 minutes)
roksbnkctl cluster up --auto
# → terraform apply (cluster phase: deploy_bnk=false forced)
#   ...
#   Apply complete! Resources: 36 added, 0 changed, 0 destroyed.
# ✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json

# Step 2 — first BNK trial (~7 minutes — trial only, cluster is reused)
roksbnkctl bnk up --auto
# → terraform plan (trial phase)
#   Plan: 41 to add, 0 to change, 0 to destroy.
#   ...
#   Apply complete! Resources: 41 added, 0 changed, 0 destroyed.

# Step 3 — poke at the trial, find something to tune
roksbnkctl k get pods -n f5-bnk
roksbnkctl test connectivity

# Step 4 — destroy just the trial (~3 minutes — cluster persists)
roksbnkctl bnk down --auto
# → terraform destroy (trial phase)
#   Destroy complete! Resources: 41 destroyed.
# ✓ Trial phase destroyed. Cluster phase ~/.roksbnkctl/default/state-cluster/ is intact.

# Step 5 — edit config.yaml (or a --var-file) to change cne_instance settings
$EDITOR ~/.roksbnkctl/default/config.yaml

# Step 6 — second BNK trial against the same cluster (~7 minutes; the 30-minute
#          cluster provision from step 1 does NOT repeat)
roksbnkctl bnk up --auto
# → terraform plan (trial phase)
#   ...
#   Apply complete! Resources: 41 added, 0 changed, 0 destroyed.

The win is in step 6: the cluster persists across the bnk down / bnk up boundary, so the second trial deploy is ~7 minutes instead of the ~50 minutes a full down → up cycle would cost. Across a day of iteration, that’s the difference between five trial permutations and one.

When you’re done with the whole session:

# Step 7 — tear down the cluster too
roksbnkctl cluster down --auto
# (or `roksbnkctl down` from any starting state — see the dispatch matrix above)

The terraform-native deployment model

The BNK phase has three layers, each owned by a purpose-built Terraform provider. Nothing shells out to curl, kubectl, or helm; nothing sleeps a fixed interval.

1. The install layer — helm_release

cert-manager, FLO, and CIS install as helm_release resources with wait = true. helm_release runs Helm in-process (via the hashicorp/helm provider), tracks the installed chart/version in Terraform state, and — with wait = true — blocks until the release’s workloads report ready. FAR version discovery stays Terraform-side: the manifest chart is still pulled to read the FLO and CIS chart versions, which feed the helm_release.version argument.

The chart installs have prerequisites that must exist first, or helm_release wait = true would hang on ImagePullBackOff:

  • Namespaces (f5-bnk, f5-utils) — kubernetes_namespace. One namespace if you collapse them: setting bnk.flo_namespace and bnk.flo_utils_namespace to the same value is supported, and the utils-side namespace (and its duplicate secrets) are then not created at all.
  • Secrets — the FAR image-pull secret far-secret (in both namespaces — or once, when they are collapsed to one) and the CIS f5-bigip-ctlr-login login secret — kubernetes_secret. These carry the registry credentials the charts pull with, so they are ordered before the charts via depends_on.

2. The CR layer — kubectl_manifest + wait_for

Every custom resource — the cert-manager issuer chain (ClusterIssuer, the CA Certificate, the CA ClusterIssuer), the two NetworkAttachmentDefinitions, the OpenShift SCC ClusterRoleBindings, the node-labeler Job, the CNEInstance, and the License — is a kubectl_manifest resource from the alekc/kubectl provider. Each manifest body is the CR spec rendered by yamlencode.

Two CRs gate on real readiness via a wait_for block that watches the live object’s .status:

# CNEInstance — wait for FLO to report the instance up
resource "kubectl_manifest" "cneinstance" {
  yaml_body         = yamlencode(local.cneinstance_manifest)
  server_side_apply = true
  field_manager     = "roksbnkctl"
  wait_for {
    condition {
      type   = "Available"
      status = "True"
    }
  }
  depends_on = [helm_release.flo]   # the FLO chart installs the CRD
}

# License — wait for the CPCL state machine to reach the licensed state
resource "kubectl_manifest" "license" {
  yaml_body         = yamlencode(local.license_manifest)
  server_side_apply = true
  field_manager     = "roksbnkctl"
  wait_for {
    field {
      key   = "status.state"
      value = "Verification Complete"
    }
  }
  depends_on = [helm_release.flo, kubectl_manifest.cneinstance]
}

wait_for polls the live object and returns the instant the matched condition/field becomes true. The CNEInstance signal is its Available condition (status.conditions[type=Available].status == True); the License signal is its status.state reaching the licensed value. (The "Verification Complete" literal comes from the architect spike’s CWC-REST evidence and the FAR-shipped License CRD; it is confirmed against a live licensed cluster by the integrator’s e2e-bnk-native.sh run before merge — if the live value differs, staff pins the real one or switches to the status.conditions[] fallback matcher, which is equally expressible.) Resources with no meaningful readiness status — NADs, SCC bindings, issuers — carry no wait_for and simply apply.

3. Namespaces + Secrets — the kubernetes provider

Covered above as the chart prerequisites. They use the hashicorp/kubernetes provider rather than kubectl_manifest because they have typed, first-class resource schemas (kubernetes_namespace, kubernetes_secret) and no plan-time-CRD concern.

Provider wiring and ordering

All three providers are fed from the same data "ibm_container_cluster_config" the modules already use — host, token, and the decoded cluster_ca_certificate. The create_roks_cluster ? 0 : 1 guard that keeps the provider config valid when the cluster doesn’t exist yet at plan time is unchanged.

The only genuinely serial spine is cert-manager helm_release → the CA issuer chain → flo helm_release → CNEInstance → License. Everything else — the NADs, the three Secrets, the node-labeler subtree, and the CNEInstance SCC bindings (19 — 9 FLO-side, 10 utils-side; 18 when the namespaces are collapsed and the duplicate default binding dedupes) — carries no depends_on edge back to the spine, so Terraform’s default parallelism applies them concurrently with the install spine. That parallelism, plus real-readiness gating instead of fixed sleeps, is where the speed comes from.

Why alekc/kubectl

A concept note on the deployment model, for readers who want the why rather than the how.

The plan-time-CRD problem. Applying a CR whose CRD is installed in the same apply (the FLO chart installs the CNEInstance and License CRDs, then we apply those CRs) is the classic Terraform trap. HashiCorp’s own kubernetes_manifest does a typed-schema lookup of the kind at plan time and fails with cannot select exact GVK ... no matches for kind because the CRD doesn’t exist yet. The alekc/kubectl provider’s kubectl_manifest carries the manifest as an opaque YAML string and resolves the GVK at apply time — so it applies a CR whose CRD is created in the same run, no plan-time schema lookup. It gives us real state + drift + finalizer-aware delete + wait_for, without the plan-time-CRD failure and without building a custom provider or a Go reconciler. Ordering (CRD before CR) is expressed with depends_on on the chart’s helm_release — wait_for removes the sleep, not the ordering.

What stays IBM-Cloud-native. The IBM IAM trusted-profile resources (for the CNE controller service account) and the COS reads (the FAR auth archive and the license JWT, fetched via the IBM IAM token + the COS S3 REST API) are IBM-Cloud-native Terraform resources/data sources, not Kubernetes mutations.

The air-gap caveat for alekc/kubectl. alekc/kubectl is a third-party provider on the public Terraform registry (registry.terraform.io/providers/alekc/kubectl). On connected runs it downloads at terraform init like any provider. For air-gapped / offline ROKS installs it must be pre-staged: build a terraform providers mirror bundle on a connected machine and point the offline runner at it via a provider_installation { filesystem_mirror { ... } } block in the Terraform CLI config (or an internal network_mirror). The provider is bounded (~> 2.4) and pinned in .terraform.lock.hcl, which now ships inside the roksbnkctl binary and seeds each workspace’s terraform directory — so a mirrored install resolves the same version, and the recorded checksums verify what the mirror serves.

Note — transcripts, resource counts, and timings are illustrative. The sample terraform plan / apply / bnk up / bnk down output, the ~41/~77-resource breakdowns, and the timing figures in this chapter are representative of the terraform-native path, not captured from a live run. The command surface (the resource shapes — helm_release / kubernetes_*_v1 / kubectl_manifest + wait_for) is reconciled against the built binary and the landed terraform and is accurate; treat the specific numbers and console lines as orientation, not contract.

Cross-references

Air-gapped install: mirroring BNK into a private registry

Some ROKS environments forbid pulling install resources from external repositories at deploy time. For these, roksbnkctl can replicate every artifact BNK needs — the F5 charts and images, plus the non-F5 dependencies — from the F5 Artifact Repository (FAR, repo.f5.com) into a private registry you control, and then install BNK entirely from there. With the mirror in place, the cluster needs no egress to repo.f5.com, quay.io, docker.io, or charts.jetstack.io to bring up BNK.

The mirror target is a registry you own — IBM Container Registry (ICR), the default, or a generic OCI-compliant registry (Artifactory, Harbor, Quay, registry:2). Choosing and configuring the target is its own chapter, Registry targets: ICR and Artifactory; this chapter is the air-gap flow itself.

The registry surface is CRUD-shaped, modeled on the IBM COS client:

CommandWhat it does
roksbnkctl registry targetShow or set the mirror target — icr / generic (see Registry targets)
roksbnkctl registry bomPrint the bill-of-materials — every chart + image BNK needs
roksbnkctl registry replicateMirror the BOM into the target registry
roksbnkctl registry adoptRecord a mirror that already exists, without re-copying (no source access)
roksbnkctl registry listWhat’s present in the target
roksbnkctl registry diffBOM vs. target — what’s missing or stale
roksbnkctl registry verifyEvery BOM artifact present + digest-matched
roksbnkctl registry pruneRemove mirrored artifacts no longer in the BOM
roksbnkctl registry deleteDelete all replicated artifacts from the target + clear the record

The bill-of-materials

The BOM comes from the f5-bigip-k8s-manifest that FAR publishes per BNK release — the same manifest roksbnkctl reads for version discovery. It enumerates every F5 chart and image (for BNK 2.3, that’s 25 charts + 56 images), tag-pinned. roksbnkctl unions in the two dependencies the F5 manifest does not cover — cert-manager (the Jetstack chart + its quay.io images) and the registry.k8s.io/kubectl node-labeler image — plus the manifest chart itself (release/f5-bigip-k8s-manifest), because the install needs to read it and a mirror that lacked it would send every install back to repo.f5.com. So the mirror is complete:

$ roksbnkctl registry bom
charts  repo.f5.com/charts/f5-lifecycle-operator   v2.21.13-0.0.28
charts  repo.f5.com/charts/f5-tmm                  15.430.5-0.2.157
...
images  repo.f5.com/images/tmm-img                 v10.159.3-0.1.5
images  quay.io/jetstack/cert-manager-controller   v1.17.3
images  registry.k8s.io/kubectl                   v1.36.0
...
81 F5 artifacts + 6 dependency artifacts

The walk-through

$ roksbnkctl cluster up               # the durable cluster
$ roksbnkctl registry target icr      # choose the target (icr is the default)
$ roksbnkctl registry replicate       # mirror the BOM (one-time per BNK version)
$ roksbnkctl registry verify          # confirm completeness
$ roksbnkctl bnk up                   # installs from the mirror

The walk-through runs registry replicate after the Cluster phase for convenience, but replicate needs no cluster. It is a purely host-side, registry-to-registry copy: it pulls each artifact by digest from the FAR source (repo.f5.com) and pushes to your target registry via go-containerregistry, from wherever roksbnkctl runs. So you can pre-seed the mirror as a standalone supply-chain step, before any cluster exists — all it requires is a configured registry: block, the FAR source credential, and network reachability to both repo.f5.com and your registry:

$ roksbnkctl -w mirror init --config-file mirror.yaml   # registry: block + bnk.far_auth_local_file
$ roksbnkctl -w mirror registry target generic          # or icr
$ roksbnkctl -w mirror registry replicate               # copies FAR → your registry, no cluster
$ roksbnkctl -w mirror registry verify

registry replicate is a deliberate, occasional step — it is not part of the composite roksbnkctl up (like gateway, you run it explicitly). With no --target, it uses the workspace’s registry.target (default icr). It copies each artifact by digest, idempotently — re-running only moves what changed (after a BNK version bump, registry diff shows exactly the changed artifacts).

How the install reads the mirror

registry replicate records what it copied — and where — in registry-mirror.json. The BNK phase reads that record and redirects the install to the target: every chart and image resolves from your private registry instead of FAR, so BNK comes up with no external pulls. The air-gap acceptance is exactly this, with egress to the external registries blocked.

The record is only used when it describes the mirror the workspace is currently configured for — same target kind, repository and host. A record outlives the mirror it describes: repoint registry.generic_repo_prefix at another repository, or switch registry.target, and the record on disk still names the old one. Nothing re-probes on read, so an unchecked record would redirect the whole install onto a registry nobody asked for, and terraform would apply it without complaint. bnk up refuses instead, naming both repositories:

the recorded registry mirror does not describe the configured mirror: it was
written for repository "bnk-mirror", the configured repository is "docker-local".
  Acting on it would use a mirror this workspace is not configured for.
  Re-record it:   roksbnkctl -w prod registry replicate   (needs the FAR source)
  Already filled: roksbnkctl -w prod registry adopt       (no source access needed)

The same check gates the node CA-trust install, which would otherwise push the old mirror’s CA onto every worker. When the configured mirror cannot be resolved from config at all — an unset generic_host, an ICR region with no known registry host — the record is left alone: knowing less than the record is not grounds for discarding it.

The target’s own pull credential is wired in for you:

  • ICR — the cluster authenticates to *.icr.io with iamapikey + the workspace IBM Cloud API key, for both the FLO chart pull and the image pulls.

  • Generic OCI — chart and image pulls authenticate with the same basic-auth credential replication used (registry target generic_username / generic_password), so a private registry needs no anonymous/public project. Concretely: chart pulls helm registry login with it, and the pods get a mirror-secret dockerconfig built from it, created in every namespace that pulls images — cert-manager, the FLO/BNK namespaces, and kube-system for the node-labeler — and referenced from the CNEInstance. You do not create any of it.

    This is the one place the two mirror kinds differ. An in-cluster/ICR mirror authorizes by RBAC and gets no pull secret; an external one (Harbor, Artifactory) gets mirror-secret. Dropping the pull secret for every mirror would force people to make their Harbor project world-readable — for a registry holding F5’s proprietary images, not an acceptable requirement.

Doing it all in CI, with nothing installed

Every command in this chapter runs unchanged inside the tools-runner container, so a pipeline can mirror the registry and install from it with no roksbnkctl, terraform, helm, kubectl or ibmcloud on the runner. The registry target itself comes from the environment — ROKSBNKCTL_REGISTRY_TARGET, ROKSBNKCTL_GENERIC_HOST, ROKSBNKCTL_GENERIC_REPO_PREFIX, ROKSBNKCTL_GENERIC_USERNAME, ROKSBNKCTL_GENERIC_PASSWORD — so nothing has to template a config.yaml:

docker run --rm -v "$PWD/state:/work" \
  -e IBMCLOUD_API_KEY -e ROKSBNKCTL_REGION=eu-gb \
  -e ROKSBNKCTL_PREFIX=ci -e ROKSBNKCTL_CLUSTER_NAME=my-cluster -e ROKSBNKCTL_CLUSTER_CREATE=false \
  -e ROKSBNKCTL_REGISTRY_TARGET=generic \
  -e ROKSBNKCTL_GENERIC_HOST=harbor.example.com \
  -e ROKSBNKCTL_GENERIC_REPO_PREFIX=bnk-mirror \
  -e ROKSBNKCTL_GENERIC_USERNAME=admin -e ROKSBNKCTL_GENERIC_PASSWORD \
  ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.19.0 -w ci \
  init --non-interactive --override-from-env

For the full disconnected picture — a private registry and licensing through a proxy in another cluster, as a two-job pipeline — see Flow C in CI.

Why the install never phones home

Two things have to be true for a mirrored install to be genuinely disconnected, and roksbnkctl handles both:

  1. The manifest is mirrored. f5-bigip-k8s-manifest is the BOM’s own source, so it is easy to overlook — but the install reads it to derive the FLO and CIS chart versions. It is a normal OCI chart, so it is replicated like any other artifact and pulled from the mirror.
  2. FLO reads the manifest from the cluster, not a registry. The F5 Lifecycle Operator resolves the BNK manifest by listing cluster-scoped CNEManifest CRs and matching spec.version; only when none matches does it fall back to pulling the manifest chart from the CNEInstance’s spec.registry.uri. roksbnkctl converts the manifest into a CNEManifest CR and applies it before the CNEInstance, so FLO finds it in the cluster and never fetches a manifest at all. (Without the CR, that fallback is what breaks a mirrored install: FLO reports “No CNEManifest exists which contains expected manifestVersion” and the CNEInstance never reconciles.)

One mirror, many clusters

Nothing ties a mirror to a single cluster. registry replicate is a supply-chain step, not a cluster step — the record it writes (registry-mirror.json) just tells the install where to pull from. So several workspaces can point at the same registry: replicate once, then every cluster installs from it.

The second workspace still needs a record of its own, though, because bnk up refuses to render against a mirror it has no record of. Use registry adopt:

$ roksbnkctl -w cluster-b registry adopt
  ✓ 10.241.0.4 holds 89 repositories under "bnk-mirror"
  ✓ recorded the mirror CA from 10.241.0.4 (nodes install it before pulling)
✓ adopted the mirror at 10.241.0.4/bnk-mirror — `bnk up` will render against it

adopt derives the record from the workspace’s own registry config and asks the mirror what it holds — it needs no access to repo.f5.com at all, which matters because an air-gapped operator usually has none. Re-running replicate would also work when the source is reachable, and copies nothing (every artifact is present and digest-matched), but it cannot run without the source. Add --verify-contents to adopt when you do have the source and want the mirror digest-checked before it is recorded.

That pairs naturally with a shared licensing cluster: one cluster holds the F5 License Proxy and reaches F5, one registry holds the artifacts, and any number of air-gapped clusters install from the registry and license through the proxy — reaching neither repo.f5.com nor F5’s licensing service themselves.

The per-target host, namespace/repository, and credential specifics — including the ICR namespace and a full Artifactory walkthrough — are in Registry targets.

TLS trust. Charts and images are pulled over the target’s HTTPS endpoint. ICR and a public Artifactory carry publicly-trusted certificates. For a custom or self-signed cert, give roksbnkctl the CA itself — registry target generic_ca <file> records it in the workspace (registry.generic_ca_b64), and bnk up installs it on every node before pulling, so you do not have to touch node trust by hand. roksbnkctl will not adopt a self-signed CA it merely discovered over the wire; see Configuration reference for the pin (generic_ca_sha256).

A truly disconnected cluster (no worker egress)

Mirroring removes external pulls at install time, but by default the ROKS cluster is still built with a public gateway on each subnet — its workers keep Internet egress. For a genuinely disconnected cluster with no egress at all, set:

cluster:
  create: true
  name: dc-roks
  public_gateway: false     # no public gateways → no worker Internet egress

init also prompts for this (“Attach public gateways for worker Internet egress?”); answer No for a private cluster. It renders the cluster_public_gateway = false terraform variable, so no ibm_is_public_gateway is created and no subnet attaches one. An omitted or true value is the current behavior (egress on).

This is an expert topology. A cluster with no egress can only pull images and reach IBM Cloud services over private paths that roksbnkctl does not build for you. Before setting public_gateway: false you must provide:

  • Private reachability to your mirror registry — e.g. the registry in a VPC the cluster reaches over a Transit Gateway, or a Virtual Private Endpoint (VPE). All BNK images/charts then come from the mirror (this chapter).
  • VPEs / private service endpoints for the IBM Cloud services the cluster needs (Container Registry if you mirror to ICR, COS for remote state, etc.).
  • Licensing without F5 egress — license through an in-cluster or VSI F5 License Proxy (bnk.license_mode: f5licenseproxy) that itself has the one controlled egress path to F5, or a disconnected license mode, so the cluster never calls F5 directly.

The public_gateway toggle only removes the egress path; wiring the private paths above is the operator’s responsibility. Note the cluster master keeps its public service endpoint regardless — this toggle governs worker/subnet egress, not the API endpoint.

bnk up checks the private paths before it installs

Because those private paths are yours to wire, bnk up verifies them rather than assuming them. Before it plans anything it probes the mirror — and the F5 License Proxy, when one is configured — from every node, in every availability zone, and fails if the mirror is unreachable from any of them.

The check has to run on a node. The host running roksbnkctl typically sits on the services VPC with egress, while the workers are air-gapped behind the gateway; a probe from there returns a confident green for a mirror the cluster cannot route to. This was not hypothetical during validation — registry adopt timed out reaching Harbor from the operator host while all three nodes reached it fine.

Without the gate, an unroutable mirror surfaced as ImagePullBackOff and then a helm context deadline exceeded about ten minutes later, naming neither the registry nor the node. A security group or subnet fixed in two zones out of three is a common shape, and one pod per node is what makes it visible.

A registry with no CA to install — already in the node trust bundle, or publicly signed — is still probed. Being trusted and being reachable are different things, and only the second fails silently.

Both of the gate’s timers are tunable under bnk.preflight; raise the retry budget when the Transit Gateway is attached in the same run as the install. The full output, rationale, and per-topology walkthroughs are in Appendix A.

Registry targets: ICR and a private Artifactory

The air-gapped install chapter mirrors BNK into a private registry you control. There are two targets roksbnkctl registry replicate can populate:

registry.targetBackendPush / pull host
icr (default)IBM Container Registry<region>.icr.io/<namespace>
genericAny OCI registry — Artifactory, Harbor, Quay, registry:2<host>/<repo-prefix>

Both are ordinary OCI registries: push, image-pull, and chart-pull all use the same host, with the FAR category nested under a namespace (<host>/<ns>/images/<name>).

Configuring the target — registry target

Configure the mirror with the registry target command — no config.yaml editing required. With no arguments it prints the current target and fields; otherwise the first argument is a backend kind (icr / generic) or a field name followed by its value. Everything it sets is written to the workspace config.yaml.

roksbnkctl registry target                       # show the current target + fields
roksbnkctl registry target icr                   # select a backend kind
roksbnkctl registry target icr_namespace my-bnk  # set a field
roksbnkctl registry replicate --target generic   # or override the backend for one run

Recognized fields: icr_host, icr_namespace, generic_host, generic_repo_prefix, generic_username, generic_password (the last reads from stdin with --password-stdin).

The default is ICR

An empty registry.target resolves to icr. Set generic explicitly to mirror into an OCI-compliant registry such as Artifactory.

The OpenShift internal registry was removed as a target. Earlier releases supported an openshift target that mirrored into the cluster’s own internal registry; it has been removed in favor of ICR and OCI-compliant registries. A workspace still carrying registry.target: openshift now errors — unsupported registry target "openshift" (expected icr or generic) — switch it with roksbnkctl registry target icr (or generic).

ICR configuration

roksbnkctl registry target icr
roksbnkctl registry target icr_namespace my-bnk   # optional — defaults to the workspace prefix
roksbnkctl registry target icr_host de.icr.io     # optional — derived from ibmcloud.region
  • Host comes from ibmcloud.region (eu-de → de.icr.io, eu-gb → uk.icr.io, us-south/us-east → us.icr.io, …). Override with icr_host for any region the map doesn’t cover.
  • Namespace is the ICR tenant unit; it defaults to the workspace prefix. Create it first — roksbnkctl does not provision the ICR namespace:
    ibmcloud cr namespace-add my-bnk
    
  • Auth is iamapikey + your workspace IBM Cloud API key (resolved the usual way). On the cluster, pods pull from <region>.icr.io using the ROKS global *.icr.io pull secret.

Walkthrough: replicate FAR into a private Artifactory

A generic target works against any standard OCI registry. Here is the full flow for a private JFrog Artifactory OCI repository.

1 — Provision an Artifactory OCI repository. In Artifactory, create a local Docker/OCI repository (e.g. bnk-mirror) and an access token with push + pull on it. Note the registry host (e.g. acme.jfrog.io) and the repository key.

2 — Configure the workspace. Set the generic target with registry target, feeding the token via stdin so it never lands in your shell history or argv:

roksbnkctl registry target generic -w prod
roksbnkctl registry target generic_host acme.jfrog.io -w prod
roksbnkctl registry target generic_repo_prefix bnk-mirror -w prod
roksbnkctl registry target generic_username ci-bot -w prod
echo "$ARTIFACTORY_TOKEN" | roksbnkctl registry target generic_password --password-stdin -w prod

roksbnkctl registry target -w prod   # confirm (the password shows as "(set)")

For a fully unattended (CI) workspace you can instead supply the token through ROKSBNKCTL_GENERIC_PASSWORD at init --override-from-env time — see Unattended setup.

3 — Replicate. Preview, then mirror the bill-of-materials into Artifactory:

roksbnkctl registry diff -w prod        # what replicate would copy
roksbnkctl registry replicate -w prod   # copy charts + images into acme.jfrog.io/bnk-mirror
roksbnkctl registry verify -w prod      # every BOM artifact present + digest-matched

Artifacts land at acme.jfrog.io/bnk-mirror/{images,charts,jetstack,...}/<name>.

4 — Install from the mirror. replicate records the target in registry-mirror.json, which redirects the BNK install off repo.f5.com onto the Artifactory mirror automatically:

roksbnkctl bnk up -w prod

The install pulls images and charts from acme.jfrog.io/bnk-mirror, using the same credential you replicated with — you do not wire the pull secret yourself. Chart pulls log in with generic_username / generic_password, and the pods get a mirror-secret dockerconfig built from that credential, created in every namespace that pulls images (cert-manager, the FLO/BNK namespaces, and kube-system for the node-labeler) and referenced from the CNEInstance.

So a private repository is fine — the registry does not need an anonymous or public project. The only requirement is network reachability from the cluster.

Licensing from the same mirror. The mirror’s BOM includes charts/f5-license-proxy, so the F5 License Proxy is replicated alongside BNK and can be installed from your registry too — including the shared-licensing topology, where one cluster runs the proxy and other clusters license through it while reaching nothing but this mirror. See Licensing BNK with the F5 License Proxy.

Removing a mirror

To wipe everything you replicated and revert the install to pulling from FAR:

roksbnkctl registry delete -w prod          # confirms first; --force to skip

registry delete removes every artifact recorded in registry-mirror.json from the target (by digest) and clears the record. Artifacts that fail to delete are kept in the record so a re-run retries exactly those. For icr the API key needs Manager (delete) rights on the namespace; for generic the registry must have deletes enabled. (To remove only artifacts that are no longer in the current BOM, use registry prune instead.)

Licensing BNK with the F5 License Proxy (FLP)

Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the Workspace struct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.

By default roksbnkctl licenses BIG-IP Next for Kubernetes (BNK) with a subscription JWT — the subscription.jwt (or your production token) staged in the orchestration COS bucket. In that mode the cluster’s controller reaches F5’s licensing service directly. Nothing in this chapter changes that default.

Some environments prefer (or require) an F5 License Proxy (FLP): an in-cluster service that brokers licensing on behalf of every BNK instance in the cluster, so the workloads themselves never talk to F5 directly. roksbnkctl can deploy the FLP and point BNK at it, as an opt-in capability — a new flp lifecycle phase plus a one-line config switch. Both licensing modes are first-class; if you don’t opt in, the JWT path is byte-for-byte unchanged.

FLP is optional. Everything here is gated behind bnk.license_mode: f5licenseproxy. Leave it unset (the default) and BNK licenses with the subscription JWT exactly as before — no flp phase, no extra resources.

What the FLP is

The FLP is a Helm-installed workload (the f5-license-proxy chart: the proxy itself plus a bundled Vault and PostgreSQL). It presents an in-cluster TLS endpoint that the BNK Cluster-Wide Controller (CWC) calls for licensing; the proxy forwards to F5’s licensing backend. Because the CWC must trust the proxy’s TLS, the proxy’s root CA is delivered to the CWC and the BNK License custom resource is set to operationMode: f5licenseproxy with the proxy’s service URL.

roksbnkctl automates all of that: roksbnkctl flp up generates the proxy’s CA and mTLS certs, creates its secrets, installs the chart (pulling from your registry mirror or FAR), and records the CA + endpoint. A later bnk up reads that handoff and wires the License CR — you never copy a certificate by hand.

The FLP’s charts and images are part of the standard BNK bill of materials, so registry replicate already mirrors them into a private registry — an air-gapped, FLP-licensed install pulls everything, including the proxy, from a registry you control.

Three topologies

The proxy is a cluster-wide broker, and it is the only component that needs to reach F5. That gives you three ways to arrange it:

Where the proxy runsUse it when
Flow Athe cluster roksbnkctl createsthe normal case
Flow Ba cluster you already own and registeryou did not provision the cluster
Flow Ca different cluster — one proxy serving manyonly one cluster may reach F5; the rest are air-gapped

Flows A and B are the same shape: proxy and BNK in one cluster. Flow C splits them.

The flp phase

Like testing and gateway, the FLP is an independent, opt-in phase with its own terraform state (state-flp/). The composite roksbnkctl up never runs it; you run it explicitly, between the cluster and BNK:

roksbnkctl flp up      # install the proxy into an existing cluster
roksbnkctl flp down    # remove it (leaves cluster / BNK / testing intact)
roksbnkctl flp output  # print its terraform outputs (endpoint, namespace)

flp up refuses if the workspace has no cluster yet, and records flp-outputs.json (the proxy’s root CA + service endpoint) on success. flp down is a no-op success when there is no FLP state, so it is safe in a reverse-order teardown of every phase.

The proxy installs into the f5-license-proxy namespace by default; override with bnk.flp.namespace (or ROKSBNKCTL_FLP_NAMESPACE).

Opting in

Set the license mode in the workspace config.yaml:

bnk:
  manifest_version: 2.3.0-3.2598.3-0.0.170
  license_mode: f5licenseproxy   # connected (default) | disconnected | f5licenseproxy
  flp:
    namespace: f5-license-proxy  # optional; this is the default

Or answer yes to License via an in-cluster F5 License Proxy (FLP)? in the roksbnkctl init interview, or set it non-interactively:

export ROKSBNKCTL_LICENSE_MODE=f5licenseproxy
export ROKSBNKCTL_FLP_NAMESPACE=f5-license-proxy   # optional

The subscription JWT is still required in FLP mode (the proxy presents it to F5); it is resolved from COS exactly as in the JWT path — no change to bnk.subscription_jwt_file or its credential.

Flow A — a cluster you are creating

The full sequence when roksbnkctl provisions the cluster:

# 1. Seed the workspace with FLP mode (interview, config.yaml, or env).
roksbnkctl -w prod init          # answer "yes" to the FLP prompt

# 2. Create the cluster.
roksbnkctl -w prod cluster up

# 3. (Optional) mirror BNK — including the FLP — into your private registry.
#    Skip this to pull from repo.f5.com directly.
roksbnkctl -w prod registry replicate --target generic

# 4. Install the F5 License Proxy.
roksbnkctl -w prod flp up

# 5. Install BNK — it licenses via the proxy automatically.
roksbnkctl -w prod bnk up

Step 5 reads flp-outputs.json and sets the License CR to operationMode: f5licenseproxy, its teem*Url at the proxy service, and delivers the proxy root CA to the CWC. bnk up errors clearly if the flp phase has not been run.

Flow B — an existing cluster registered in the workspace

You do not have to let roksbnkctl build the cluster. If you already have a ROKS cluster, register it into the workspace and install the proxy onto it. Registration looks the cluster up in your IBM Cloud account and writes the very same cluster-outputs.json that the cluster up path produces — and that is the only thing the FLP and BNK phases consume. So from step 3 on, the two flows are character-for-character identical:

# 1. Seed the workspace with FLP mode, and do NOT create a cluster.
roksbnkctl -w prod init          # answer "no" to "create a cluster?", "yes" to FLP
#    (equivalently, in config.yaml: cluster.create: false + bnk.license_mode: f5licenseproxy)

# 2. Register the cluster you already have.
roksbnkctl -w prod cluster register my-existing-roks

# 3. (Optional) mirror BNK — including the FLP — into your private registry.
roksbnkctl -w prod registry replicate --target generic

# 4. Install the F5 License Proxy into the registered cluster.
roksbnkctl -w prod flp up

# 5. Install BNK, licensed via the proxy.
roksbnkctl -w prod bnk up

roksbnkctl never takes ownership of a registered cluster. flp up adds only the proxy workload (its namespace, secrets and Helm release); flp down removes exactly that and nothing else. The cluster itself is never created, modified at the infrastructure level, or destroyed — cluster down will not touch it.

FLP behind a private registry

The FLP’s chart and its four images are part of the standard BNK bill of materials, so registry replicate mirrors them along with everything else. Combining the two gives a genuinely disconnected, FLP-licensed install: flp up and bnk up pull every chart and image from your registry, and the CWC brokers licensing through the in-cluster proxy rather than reaching F5 itself.

Nothing extra is required — point the workspace at the mirror and the FLP phase follows it automatically:

roksbnkctl -w prod registry target generic         # your Harbor/Artifactory/ICR
roksbnkctl -w prod registry replicate              # mirrors BNK + the FLP
roksbnkctl -w prod flp up                          # proxy, pulled from the mirror
roksbnkctl -w prod bnk up                          # BNK, pulled from the mirror

Flow C — a shared licensing cluster

Flows A and B both put the proxy and BNK in the same cluster. But the proxy is a cluster-wide licensing broker, and it is the only thing that needs to reach F5. So you can run it once, in a cluster that has egress, and have BNK installs in other clusters license through it — clusters that reach nothing but your mirror and the proxy.

   ┌──────────────── services cluster (has egress to F5) ─────────────┐
   │  F5 License Proxy  ──NodePort 30001──┐                           │
   └──────────────────────────────────────┼───────────────────────────┘
                                          │ same VPC, or a transit gateway
   ┌──────────────────────────────────────┼── air-gapped cluster ─────┐
   │  BNK + CWC  ─────────────────────────┘                           │
   │      └── charts + images ── your private registry (Harbor/…)     │
   └──────────────────────────────────────────────────────────────────┘

Neither cluster reaches repo.f5.com, and only the services cluster reaches F5’s licensing service.

1. The services cluster — expose the proxy

roksbnkctl -w services cluster up            # (or `cluster register` an existing one)

# point the workspace at your private registry, then mirror FAR into it
roksbnkctl -w services registry target generic
roksbnkctl -w services registry target generic_host        harbor.example.com
roksbnkctl -w services registry target generic_repo_prefix bnk-mirror
roksbnkctl -w services registry target generic_username    admin
printf '%s' "$HARBOR_PASSWORD" | \
  roksbnkctl -w services registry target generic_password --password-stdin
roksbnkctl -w services registry replicate    # BNK *and* the FLP chart — both are in the BOM

roksbnkctl -w services flp up \
    --add-node-port-access \
    --node-port-source-cidr 10.242.0.0/18,10.242.64.0/18,10.242.128.0/18

Set the registry target before replicating: with none set it defaults to icr, and you would mirror into IBM Container Registry rather than your own. The password is piped on stdin so it never lands in argv or your shell history. See Registry targets for the ICR and Artifactory variants.

The proxy itself is installed from that registry — charts/f5-license-proxy is part of the mirror BOM, so even the licensing component does not come from repo.f5.com.

The CIDR flag is a list, and it must name every zone. A multi-zone VPC carries one address prefix per zone:

$ ibmcloud is vpc-address-prefixes <vpc> --output json | jq -r '.[] | "\(.zone.name)  \(.cidr)"'
eu-gb-1  10.242.0.0/18
eu-gb-2  10.242.64.0/18
eu-gb-3  10.242.128.0/18

List only the first and the proxy will appear to work — a pod you exec into for a test may well be scheduled in that zone and get an HTTP 200 — while the consuming cluster’s CWC, scheduled in one of the other two, is dropped at the security group and reports connect: connection timed out. The symptom is a proxy that answers some pods and silently times out for others. Pass every prefix, or a supernet covering them.

No BNK is installed here — this cluster exists to run the proxy.

--add-node-port-access does three things you cannot do by hand afterwards:

  • Makes every worker answer. The chart’s Service is already type: NodePort (30001), but it hardcodes externalTrafficPolicy: Local and runs one replica — so only the node currently hosting the pod answers, and that node changes when the pod reschedules. This flips it to Cluster, so any worker IP is a valid endpoint.
  • Puts the worker IPs in the proxy’s certificate. A remote CWC dials https://<node-ip>:30001 — a literal address, which DNS SANs do not cover. Without IP SANs the TLS handshake fails with bad certificate.
  • Opens the port. ROKS workers sit in a kube-<cluster-id> security group that does not admit another cluster. --node-port-source-cidr opens only 30001, and only to the CIDRs you name — one security-group rule per prefix. Omit it if a path already exists.

It prints the address to hand over, and records it in flp-outputs.json:

→ FLP reachable from other clusters at https://10.242.0.9:30001

2. The consuming cluster — license against it

Read the endpoint and CA out of the owning workspace:

roksbnkctl -w services flp output       # → external_endpoint, root_ca_b64

and point the other workspace at them. It never runs flp up — it does not own a proxy:

# consuming workspace's config.yaml
bnk:
  license_mode: f5licenseproxy
  flp:
    external:
      url: https://10.242.0.9:30001     # from `flp output` → external_endpoint
      root_ca_b64: LS0tLS1CRUdJTi…      # from `flp output` → root_ca_b64

Point that workspace at the same private registry, and install. bnk up wires the License CR at the remote proxy and delivers its CA to the CWC, exactly as it would for a local one:

roksbnkctl -w app cluster register my-app-cluster   # or `cluster up` to build one

# the SAME registry as the services workspace
roksbnkctl -w app registry target generic
roksbnkctl -w app registry target generic_host        harbor.example.com
roksbnkctl -w app registry target generic_repo_prefix bnk-mirror
roksbnkctl -w app registry target generic_username    admin
printf '%s' "$HARBOR_PASSWORD" | \
  roksbnkctl -w app registry target generic_password --password-stdin

roksbnkctl -w app registry replicate   # copies nothing — the artifacts are already there
roksbnkctl -w app bnk up               # licenses via the proxy in the OTHER cluster

registry replicate in the second workspace has nothing left to copy. Run it anyway: its other job is to write registry-mirror.json, and that record is what redirects the install off repo.f5.com onto your registry. Without it, bnk up would go to FAR. (See One mirror, many clusters.)

Why this cluster is genuinely disconnected

Three things have to be true, and bnk up arranges all three — there is nothing to wire by hand:

  • Charts and images come from your registry. Chart pulls log in with the credential above; the pods get a mirror-secret dockerconfig built from it, created in every namespace that pulls (cert-manager, the FLO/BNK namespaces, kube-system for the node-labeler) and referenced from the CNEInstance. Your registry stays private — it needs no anonymous or public project.
  • The manifest is already in the cluster. The BNK manifest is mirrored too and pre-applied as a CNEManifest CR, so the lifecycle operator resolves it locally instead of fetching it from F5.
  • Licensing goes to the proxy next door, not to F5.

The result: this cluster talks to your registry and to https://<node-ip>:30001. Nothing else.

Flow C in CI — the runner container, no host install

Everything above is a roksbnkctl you installed. The same topology runs as a CI pipeline with nothing installed at all: every step is the tools-runner container, and the whole workspace is described by environment variables — the natural shape of CI, where the registry credential and the proxy handoff are already secrets and job outputs.

Two jobs. The first owns the proxy and emits its address and CA; the second installs BNK into a cluster that reaches nothing but the registry and that proxy.

# .github/workflows/bnk-disconnected.yml
env:
  RUNNER: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.19.0

jobs:
  licensing-cluster:
    runs-on: ubuntu-latest
    container: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.19.0
    outputs:
      flp_url: ${{ steps.flp.outputs.url }}      # ← the handoff
      flp_ca:  ${{ steps.flp.outputs.ca }}
    env:
      IBMCLOUD_API_KEY:              ${{ secrets.IBMCLOUD_API_KEY }}
      ROKSBNKCTL_REGION:             eu-gb
      ROKSBNKCTL_RESOURCE_GROUP:     default
      ROKSBNKCTL_PREFIX:             ci-svc
      ROKSBNKCTL_CLUSTER_NAME:       ${{ vars.SERVICES_CLUSTER }}
      ROKSBNKCTL_CLUSTER_CREATE:     "false"        # adopt it; do not build it
      ROKSBNKCTL_REGISTRY_TARGET:    generic
      ROKSBNKCTL_GENERIC_HOST:       ${{ vars.REGISTRY_HOST }}
      ROKSBNKCTL_GENERIC_REPO_PREFIX: bnk-mirror
      ROKSBNKCTL_GENERIC_USERNAME:   ${{ vars.REGISTRY_USER }}
      ROKSBNKCTL_GENERIC_PASSWORD:   ${{ secrets.REGISTRY_PASSWORD }}
    steps:
      - run: roksbnkctl -w ci-svc init --non-interactive --override-from-env
      - run: roksbnkctl -w ci-svc cluster register "$ROKSBNKCTL_CLUSTER_NAME"
      - run: roksbnkctl -w ci-svc registry replicate --target generic
      - run: roksbnkctl -w ci-svc registry verify
      - name: Deploy the proxy, reachable from the other cluster
        run: |
          roksbnkctl -w ci-svc flp up --auto \
            --add-node-port-access \
            --node-port-source-cidr "${{ vars.APP_CLUSTER_CIDRS }}"
      - id: flp
        run: |
          echo "url=$(roksbnkctl -w ci-svc flp output flp_external_endpoint)" >> "$GITHUB_OUTPUT"
          echo "ca=$(roksbnkctl -w ci-svc flp output flp_root_ca)"            >> "$GITHUB_OUTPUT"

  application-cluster:
    needs: licensing-cluster
    runs-on: ubuntu-latest
    container: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.19.0
    env:
      IBMCLOUD_API_KEY:              ${{ secrets.IBMCLOUD_API_KEY }}
      ROKSBNKCTL_REGION:             eu-gb
      ROKSBNKCTL_RESOURCE_GROUP:     default
      ROKSBNKCTL_PREFIX:             ci-app
      ROKSBNKCTL_CLUSTER_NAME:       ${{ vars.APP_CLUSTER }}
      ROKSBNKCTL_CLUSTER_CREATE:     "false"
      ROKSBNKCTL_REGISTRY_TARGET:    generic
      ROKSBNKCTL_GENERIC_HOST:       ${{ vars.REGISTRY_HOST }}
      ROKSBNKCTL_GENERIC_REPO_PREFIX: bnk-mirror
      ROKSBNKCTL_GENERIC_USERNAME:   ${{ vars.REGISTRY_USER }}
      ROKSBNKCTL_GENERIC_PASSWORD:   ${{ secrets.REGISTRY_PASSWORD }}
      # ── license through the proxy in the OTHER cluster ──
      ROKSBNKCTL_LICENSE_MODE:       f5licenseproxy
      ROKSBNKCTL_FLP_EXTERNAL_URL:   ${{ needs.licensing-cluster.outputs.flp_url }}
      ROKSBNKCTL_FLP_ROOT_CA_B64:    ${{ needs.licensing-cluster.outputs.flp_ca }}
    steps:
      - run: roksbnkctl -w ci-app init --non-interactive --override-from-env
      - run: roksbnkctl -w ci-app cluster register "$ROKSBNKCTL_CLUSTER_NAME"
      - run: roksbnkctl -w ci-app registry replicate --target generic   # records the mirror
      - run: roksbnkctl -w ci-app bnk up --auto
      - name: Gate the pipeline on the license
        run: roksbnkctl -w ci-app k get license -n f5-utils

The handoff is two environment variables. ROKSBNKCTL_FLP_EXTERNAL_URL and ROKSBNKCTL_FLP_ROOT_CA_B64 are exactly what flp output prints, so the proxy’s address and CA travel between jobs as ordinary job outputs. Nothing templates a config.yaml; init --override-from-env builds the whole workspace from the environment. (flp_root_ca is already base64, and the variable takes it verbatim — re-encoding it hands the CWC a corrupt CA.)

Neither job installs anything. No roksbnkctl, no terraform, no helm, no kubectl, no ibmcloud — the runner image carries them, and roksbnkctl runs the Kubernetes verbs in-process. That includes the FLP chart’s post-render fix-ups, which roksbnkctl performs itself (roksbnkctl flp postrender) rather than shelling out to an interpreter the image does not have.

State lives in the workspace directory, which in a container is ephemeral. For a pipeline that will run again — to upgrade, or to tear down — put the terraform state in COS/S3 (roksbnkctl state migrate); see Remote state and Ephemeral runners need remote state. The demo below mounts a volume instead, which is the same idea with a shorter lifetime.

What the last step proves. k get license prints the CR the CWC wrote:

NAME          STATE    MODE             ENTITLEMENT   ENVIRONMENT   EXPIRY
bnk-license   Active   f5licenseproxy   eval          test          2026-08-13T16:11:06Z

Active + f5licenseproxy, in a cluster with no proxy of its own and no route to repo.f5.com. Fail the job on anything else and the pipeline gates on the license.

What to watch out for

  • The two clusters must be able to reach each other — same VPC (simplest: give the second cluster resources.cluster_vpc.create: false + existing: <vpc-id>), or a transit gateway between their VPCs.
  • Worker IPs change when a worker is replaced. The certificate covers the IPs that existed at flp up time, so replacing a node in the services cluster means re-running flp up (which re-issues the cert) and updating the consuming workspace if you pinned the URL of the node that went away. flp output lists every worker URL — all are cert SANs — so you can point at another one. A stable VIP would avoid this entirely; NodePort is the trade-off you accept for not needing a load balancer.
  • The consuming workspace needs no flp phase at all — no state-flp/, nothing to tear down. bnk down there leaves the proxy untouched, serving its other clusters.

Running the FLP as a VSI (mode: vsi)

The flows above run the FLP as an in-cluster Helm deployment. It can instead run as a standalone Virtual Server Instance — a VM, no Kubernetes — which is the right shape when the proxy must be the one controlled egress point for an otherwise disconnected cluster. Select it with bnk.flp.mode: vsi:

bnk:
  license_mode: f5licenseproxy
  flp:
    mode: vsi
    vsi:
      profile: bx2-4x16          # >= 4 vCPU / 16 GB
      zone: us-south-1
      reach: floating            # "private" (VPC IP) or "floating" (public IP → egress to F5)
      allowed_cidrs: [10.242.0.0/18]   # sources allowed to reach the proxy (the cluster's workers)

roksbnkctl flp up then provisions the VSI; it terminates in the same flp-outputs.json (endpoint + root CA) the Helm path produces, so bnk up consumes it unchanged.

Standalone — a licensing appliance in another VPC (no cluster)

By default the FLP VSI joins the cluster’s own VPC (from cluster-outputs.json), so flp up requires a cluster. Give the proxy a network of its own — set bnk.flp.vsi.vpc to an existing VPC id, or bnk.flp.vsi.create_vpc: true to build one (mutually exclusive) — and flp up deploys the proxy into that VPC with no cluster at all — a standalone licensing appliance:

bnk:
  license_mode: f5licenseproxy
  flp:
    mode: vsi
    vsi:
      vpc: r006-…                # existing VPC id — deploy here, no cluster required
      zone: us-south-1
      reach: floating            # the appliance's controlled egress to F5
      allowed_cidrs: [10.0.0.0/8]   # allow the consuming cluster's workers (e.g. over a TGW)

This is the piece that makes a genuinely disconnected topology clean: put Harbor + the FLP in a services VPC that has egress, run the ROKS cluster in another VPC with cluster.public_gateway: false (no worker egress — see Chapter 10a §“A truly disconnected cluster”), join both with a Transit Gateway, and point the cluster’s workspace at the proxy with bnk.flp.external (its flp output endpoint + root CA). The cluster reaches Harbor and the FLP privately over the TGW; all egress is concentrated in the services VPC, and the cluster never phones home. A worked end-to-end script is in disconnected_deployment_demo.sh.

# in a services workspace (no cluster) — deploy the standalone proxy, then read its handoff
$ roksbnkctl -w svc-flp flp up
$ roksbnkctl -w svc-flp flp output flp_external_endpoint
$ roksbnkctl -w svc-flp flp output flp_root_ca            # → the disconnected cluster's bnk.flp.external

Tearing down

The FLP is an opt-in phase, so — exactly like gateway — the composite down does not cover it. Tear it down first, while the cluster is still up:

roksbnkctl -w prod flp down     # remove the proxy
roksbnkctl -w prod down         # then BNK + the cluster (composite)

Both roksbnkctl down and cluster down refuse while FLP state exists, and tell you to run flp down first. That guard is deliberate: the proxy’s Helm release and secrets live in the cluster, so destroying the cluster out from under them would strand state-flp/ pointing at resources that no longer exist, and a later flp down could never reconcile.

flp down is a no-op success when there is no FLP state, so it is safe to put unconditionally in a teardown script. Registered clusters are never destroyed — flp down removes only the proxy workload.

Reference

Config keyMeaning
bnk.license_modeconnected (default), disconnected, or f5licenseproxy
bnk.flp.namespacenamespace for the proxy (default f5-license-proxy)
bnk.flp.chart_versionoptional override. Left unset, the chart version is read from the BNK manifest (which lists charts/f5-license-proxy for the release), exactly like the FLO and CIS charts — you do not normally pin it.
bnk.flp.node_port_accessexpose the proxy outside its cluster (Flow C). Set by flp up --add-node-port-access; persisted so a later flp up does not tear the exposure down.
bnk.flp.node_port_source_cidrswith the above: the CIDRs allowed to reach the NodePort on the worker security group. A list — one address prefix per zone
bnk.flp.external.urllicense against a proxy in another cluster — its external_endpoint. This workspace never runs flp up.
bnk.flp.external.root_ca_b64that proxy’s root_ca_b64, so the CWC can verify its certificate
Flag (flp up)Meaning
--add-node-port-accessexpose the proxy for other clusters: externalTrafficPolicy: Cluster, worker IPs as cert SANs, external endpoint recorded
--node-port-source-cidropen the NodePort to these CIDRs on the worker security group. Repeatable / comma-separated — pass every zone’s prefix
Environment variableOverrides
ROKSBNKCTL_LICENSE_MODEbnk.license_mode
ROKSBNKCTL_FLP_MODEbnk.flp.mode ("" | helm | vsi)
ROKSBNKCTL_FLP_EXTERNAL_URLbnk.flp.external.url — license via a proxy in another cluster
ROKSBNKCTL_FLP_ROOT_CA_B64bnk.flp.external.root_ca_b64 (verbatim; already base64)
ROKSBNKCTL_FLP_VSI_CREATE_VPCbnk.flp.vsi.create_vpc — give the proxy its own VPC
ROKSBNKCTL_FLP_VSI_VPC_NAMEbnk.flp.vsi.vpc_name
ROKSBNKCTL_FLP_VSI_SUBNET_CIDRbnk.flp.vsi.subnet_cidr
ROKSBNKCTL_FLP_VSI_VPCbnk.flp.vsi.vpc — adopt an existing VPC (mutually exclusive with _CREATE_VPC)
ROKSBNKCTL_FLP_VSI_ZONE / _PROFILE / _SSH_KEY / _BOOT_SIZE_GB / _REACH / _FLOATING_IPthe rest of bnk.flp.vsi.*
ROKSBNKCTL_FLP_VSI_MANAGEMENT_ALLOWED_CIDRS / _LICENSING_ALLOWED_CIDRSthe two per-plane security-group lists (comma-separated)
ROKSBNKCTL_FLP_NAMESPACEbnk.flp.namespace
FileWritten byRead by
~/.roksbnkctl/<ws>/flp-outputs.jsonflp upbnk up (FLP mode) — or copied into another workspace’s bnk.flp.external

flp-outputs.json carries endpoint (in-cluster only) and, when exposed, external_endpoint plus external_endpoints — every worker URL the proxy answers on. All of them are IP SANs on its certificate, so any one is a valid bnk.flp.external.url.

BNK 2.4

roksbnkctl drives BNK 2.3 and 2.4 from one build. Which one you get is decided by bnk.manifest_version and nothing else — there is deliberately no bnk.line field and no ROKSBNKCTL_BNK_LINE override, because a second way to say which release this is, is a second thing that can disagree with the manifest being installed.

bnk:
  manifest_version: 2.3.0-3.2598.3-0.0.170   # -> the 2.3 line
  # manifest_version: 2.4.0-EA               # -> the 2.4 line

The derived value is rendered as a bnk_line terraform variable ("2.3" or "2.4"), and the modules gate per-release resources with count on it. There is no forked copy of the tree: the 2.4 changes are additive resources, which count expresses cleanly and two copies of a file do not.

2.4.0-EA carries no build suffix, where 2.3 versions have four dot/dash segments (2.3.0-3.2598.3-0.0.170). Anything you write that parses a manifest version should key on the major.minor prefix rather than the segment count.

What actually changes

2.4 replaces the data-plane configuration model. It runs the CNEInstance with USE_GATEWAY_SETTINGS=true, and under that flag the CNE controller ignores the cloud-network-mapping ConfigMap and the entire F5SPK* custom-resource family that a 2.3 install creates, reading two new CRs instead.

2.32.4
cloud-network-mapping ConfigMapInfra.spec.networks[] + networkAttachments[]
F5SPKVlan × 2Infra.spec.networks[type=vlan]
F5SPKStaticRoute × NInfra.spec.staticRoutes[]
F5SPKSnatpoolGatewaySettings.spec.sourceNATPools[]
F5SPKEgress × 2GatewaySettings.spec.egressConfigs[] + Infra.spec.egressDefaults
F5BnkGatewayGatewaySettings (the Gateway’s parametersRef target)
VIP address arithmeticIPAM allocation, read from Gateway.status.addresses

roksbnkctl emits both CRs for you. The addressing comes from the per-zone CIDRs you already configure — nothing new to fill in for a standard install:

your existing configthe 2.4 IPAM pool it becomes
ext_vlan_cidrexternal-vlan-ipam — TMM VLAN self-IPs
int_vip_cidrvip-listener-ipam — ingress VIPs
int_snat_cidregress-snat-ipam — egress SNAT addresses

gateway_egress_mode means the same thing on both lines: snatpool becomes a Pool egress config backed by egress-snat-ipam, automap becomes an Automap config, and both produces both.

Things 2.4 stops needing

Three of the four items in the 2.4 guide’s own “Changes From BNK Release-2.3” section are subtractions, which is easy to miss when reading a list of additions. On the 2.4 line roksbnkctl no longer creates:

  • a separate CIS Helm release — CIS is integrated into FLO, and externalBigip.enabled on the CNEInstance is what turns it on;
  • the node-labeler Job — app=f5-tmm labelling is not required;
  • the macvlan-conf NetworkAttachmentDefinition — the product creates its own internal NAD as macvlan-internal on dummy0, owned by the F5Tmm CR, and that name is reserved;
  • eighteen of nineteen privileged SCC bindings. FLO grants its components what they need, so the install requires one binding rather than nineteen.

The guide also asks for one privileged SCC binding per application namespace (oc adm policy add-scc-to-user privileged -n <ns> -z default). roksbnkctl does not create those: it knows the FLO and utils namespaces and nothing about where you will run workloads. That one stays an operator step.

The Trusted Profile

2.4’s CNE controller authenticates to IBM Cloud with a Trusted Profile and uses it to write into your VPC — adding the Infra IPAM CIDRs as VPC address prefixes, creating a routing table, and advertising TMM virtual-IP routes to the transit gateway.

roksbnkctl creates the profile, links it to the cluster’s service account, and passes its id to the CNEInstance as advanced.cneController.env.CLOUD_TRUSTED_PROFILE. Two access policies are attached: Viewer + Editor on the cluster’s VPC, so the controller can manage the network objects it creates, and Viewer on the cluster itself, so it can read the topology it next-hops routes against.

Because those IPAM CIDRs become real VPC address prefixes, they must not collide with what the cluster VPC already allocates — see Transit gateway sharing.

Moving between lines

You cannot flip a deployed workspace from 2.3 to 2.4 in place. bnk up and gateway up both refuse it, and the refusal names the reasons, because none of them is visible in a terraform plan:

  • GTM object naming differs. 2.3 writes server_<tmm_selfip>; 2.4 writes server_<digitalassetid>_<cluster-name>_<ns>_<ip>. Those objects live on an external BIG-IP — outside the cluster and outside terraform state — so nothing here removes them, and leaving both formats in place causes device IP conflicts.
  • The network model is replaced, not migrated. The F5SPK* objects a 2.3 install created are simply no longer read, and they stay on the cluster.
  • The CWC licence secrets survive a teardown and must be cleared before a reinstall licenses cleanly. bnk down now removes them for you.

To run 2.4, use a new workspace and a new install. Changing CLUSTER_IDENTIFIER on a deployed workspace needs the same manual GTM cleanup, for the same reason.

mTLS and the Gateway API bundle

A base 2.4 install works with whatever Gateway API bundle OpenShift ships: its crd-installer logs a graceful skip and leaves the platform’s CRDs alone.

mTLS is the exception. It needs Gateway API 1.5.0 standard, and installing that means deleting the OpenShift ingress-operator’s ValidatingAdmissionPolicy and its binding — which the platform recreates quickly, so it is a race. Opt in:

bnk:
  gateway_api_mtls: true      # ROKSBNKCTL_GATEWAY_API_MTLS=true

Off by default, because deleting a platform admission policy on a cluster that does not need the newer bundle is a change nobody asked for. Ignored on 2.3, where that sweep always runs — there the crd-installer does force its own CRDs and is blocked without it.

Where the bundle comes from

With gateway_api_mtls on, bnk up installs the upstream kubernetes-sigs/gateway-api standard-install bundle itself — eight CRDs, a ValidatingAdmissionPolicy of its own, and no container images — at the release named by bnk.gateway_api_version (default 1.5.0, the same number the CNE controller is given as GATEWAY_API_VERSION, so the two cannot disagree). It is applied server-side with --force-conflicts, inside the window where the admission sweep is running, and before terraform starts.

The bytes are always checked against a sha256 that ships in roksbnkctl, not in your configuration. A release with no pin in your build is refused rather than fetched unverified.

It is fetched from, in order:

  1. The registry mirror, whenever the workspace has one. The bundle is a normal BOM artifact, so registry bom lists it, registry replicate copies it and registry verify proves it arrived. This is the path a disconnected cluster needs: in the CI flow roksbnkctl runs as a pod inside that cluster, so its egress is the cluster’s and github.com is not reachable.

    The bundle enters the BOM only when gateway_api_mtls is on, so a mirror replicated before you set it does not carry the bundle — re-run registry replicate.

  2. bnk.gateway_api_bundle_url, if set — an internal proxy of the upstream release, for an estate that blocks github.com but has no mirror yet. It moves only where the bytes come from; the pin still applies.

  3. The upstream release asset, derived from the version.

Conforming to F5’s reference CNEInstance

The 2.4 spec this tool renders is checked against F5’s own reference install — the live capture in /mnt/d/roksbnkctl-gap-2-3-to-2-4. Every default below is F5’s value, not an invented one.

Pod placement replaced the node labeller

2.4 removed the node-labeler job, and spec.placement is the mechanism that took over. For a while this tool had neither: the labeller was gone and placement was never emitted, so TMM landing one-per-node and spread across zones was left to the scheduler’s discretion rather than required.

bnk:
  tmm_anti_affinity: true                                  # one TMM per node
  tmm_anti_affinity_topology_key: kubernetes.io/hostname   # the ROKS node label
  tmm_zone_spread: true                                    # spread across zones
  tmm_zone_topology_key: topology.kubernetes.io/zone       # the ROKS zone label
  tmm_zone_max_skew: 1
  tmm_zone_when_unsatisfiable: DoNotSchedule
  tmm_pod_label: f5-tmm

The topology keys are IBM ROKS node labels, and they are settings rather than constants on purpose: hard-coding them is the assumption the node-labeller baked in, and a cluster that labels its topology differently should still be configurable.

Turn both toggles off and no placement key is emitted at all — an operator who does not want the constraint gets no empty object for the CR to interpret.

The rest of the conformance set

settingdefaultwhat it is
tmm_replicas3data-plane replica count
watch_namespaces["All"]namespaces the controller watches
tmm_rolling_updatetruepins maxSurge 0 / maxUnavailable 1
external_bigipfalseexternal BIG-IP controller
external_bigip_login_secretf5-bigip-ctlr-loginits credentials secret
cluster_identifierderivedpassed to that controller
gateway_api_version1.5.0GATEWAY_API_VERSION on the controller
demo_modefalse on 2.4advanced.demoMode.enabled

Three of these are worth a sentence each.

tmm_rolling_update is the same shape as the cwc Multi-Attach deadlock: an unconstrained rolling update on a workload holding a single-attach resource can wedge, with the old pod holding the volume the new one is waiting for.

gateway_api_version was previously set nowhere, so the controller ran on whatever the operator defaulted to — v1.4.1 on the verified cluster, against F5’s 1.5.0. The 2.4 EA guide requires the 1.5 bundle for mTLS. Note that roksbnkctl still does not install the bundle; see the mTLS caveat above.

demo_mode was being set true on every install, on both lines. On 2.4 it now defaults to false, matching the reference. On 2.3 it stays true — that is what has shipped and been exercised there, and changing a working line’s behaviour without a run to back it would be a guess. Set it explicitly to pin it on either line.

Every one of these is a config.yaml field, a ROKSBNKCTL_* override, and a row in Chapter 28 with its line and default.

Hugepages, and why deploymentSize stays Tiny

The defaults aim at a working cluster with no overrides at all. On 2.4 that means deploymentSize: Tiny, OCP 4.21, and no hugepages — and on ROKS that is not merely a sensible default, it is the only configuration that runs.

deploymentSize decides how much TMM asks for, hugepages included. Measured on live 4.21 clusters by changing only that field:

deploymentSizeTMM cpuTMM memoryhugepages-2Mi
Tiny——none
Small11536Mi4 GiB
Medium42Gi8 GiB

A stock ROKS worker reports zero, and there is no supported way to change that.

bnk.hugepages does not work on ROKS

This is worth stating plainly, because the setting exists and reads like the answer. It is not. Six approaches were measured and all six fail:

approachresult
bnk.hugepages (Node Tuning Operator, [bootloader])needs the Machine Config Operator; ROKS has 0 MachineConfigPools
a Tuned CR with [sysctl] vm.nr_hugepagesROKS deletes the CR — the create returns a real object, the next read is NotFound
a privileged pod writing /proc/sys/vm/nr_hugepagesallocates (HugePages_Total: 2048), but kubelet only reads hugepages at startup so the node still advertises 0
advanced.tmm.resources with hugepages-2Mi: 0TMM schedules, then refuses to start
advanced.tmm.env.TMM_MEMaccepted, requirement remains
ibmcloud ks worker-poolno kernel-argument or sysctl option exists

IBM manages the worker OS on ROKS, which is what removes the first two. TMM validates hugepages against the 2 MB page cgroup limit when it starts and exits if that limit is zero — with exit code 0, so it never looks like a crash: the pod reports 4/5 Running and the Deployment simply never becomes Available.

Setting bnk.hugepages on ROKS now fails with an explanation naming this, rather than surfacing as a terraform “provider produced inconsistent result” error that blames the kubectl provider (issue #203).

So how do you get more capacity?

With more TMM pods on bigger nodes, which is what F5’s own sizing guide does. Appendix C’s Large cluster reaches ~31 Gbit/s at the Medium profile by running nine TMM pods on nine larger nodes — capacity comes from pod count and node size, never from a larger deploymentSize.

cluster:
  workers_per_zone: 2
  worker_flavor: cx2.16x32
bnk:
  cneinstance_size: Tiny     # or leave unset — Tiny is the 2.4 default
  tmm_replicas: 3            # one TMM per worker node
ROKSBNKCTL_WORKERS_PER_ZONE=2
ROKSBNKCTL_WORKER_FLAVOR=cx2.16x32
ROKSBNKCTL_TMM_REPLICAS=3

All four cluster shapes in Appendix C are verified this way, at 1, 3, 3 and 9 replicas.

cneinstance_size stays configurable rather than hardcoded, deliberately: if IBM ships a worker profile that supports hugepages, only the platform’s answer changes and nothing about config.yaml or the ROKSBNKCTL_* surface has to.

What happens if you ask for a size that cannot run

bnk up fails immediately, with the numbers and the alternatives, rather than waiting fifteen minutes to report Available=False. The scheduler’s verdict is already in the CNEInstance condition; the wait recognises that no amount of waiting creates node capacity and stops there.

Tuning the data-plane TCP profile

F5’s approved reference cluster carries a hand-applied sys-default-tcp F5BigTcpSetting in the BNK namespace. Despite the “sys-default” name it was applied by engineering rather than created by the product, which is why it is surfaced here.

bnk:
  tcp_settings:
    idleTimeout: "300"
    congestionControl: high-speed
    abc: "true"
ROKSBNKCTL_BNK_TCP_idleTimeout=300
ROKSBNKCTL_BNK_TCP_congestionControl=high-speed

A map rather than 54 config fields, because the CR has 54 of them across bool, int and string. Enumerating each would be 54 rows nobody reads, and it would go stale the moment F5 adds one.

Values are text — a config file and an environment variable cannot carry anything else — and are coerced on the way out by YAML’s own scalar rules: 300 becomes an integer, true a boolean, and anything else is quoted as a string. Field names are case-sensitive and used verbatim, because the CR is case-sensitive and guessing a mapping would silently write a field it ignores.

Setting nothing writes no CR at all. The product manages its own default profile, and emitting an empty one would fight it.

Per-component settings

Field specs for everything below live in Chapter 28 — Configuration reference, which is generated from the struct and test-checked. The line column there says which release line each field applies to.

The 2.4 CNEInstance takes an environment list for each of its ~26 components. Anything not surfaced as a first-class field is reachable through the passthrough, in YAML or from the environment:

bnk:
  advanced:
    tmm:
      env:
        TMM_DEFAULT_MTU: "9000"
    cneController:
      env:
        GSLB_DATACENTER_NAME: "datacenter-eu-gb"
ROKSBNKCTL_BNK_ADV_TMM_ENV_TMM_DEFAULT_MTU=9000
ROKSBNKCTL_BNK_ADV_CNECONTROLLER_ENV_GSLB_DATACENTER_NAME=datacenter-eu-gb

An entry here replaces the default of the same name rather than adding a second copy of it — the CNEInstance spec is read by the lifecycle operator, not by kubelet, so “the last duplicate wins” is not a rule to rely on. That is what makes this the way to change a default rather than only to add to it.

USE_GATEWAY_SETTINGS is deliberately not the example here, though earlier drafts of this page used it. roksbnkctl sets it on 2.4 itself, because on 2.4 there is no working configuration without it: without the flag the controller never reads the Infra and GatewaySettings CRs that gateway up applies, and they sit at Accepted=Unknown forever. You can still override it through this passthrough — that is what the replace-not-append rule buys you — but you should not have to.

The component name is matched case-insensitively, and one that roksbnkctl does not recognize is passed through rather than refused — the component set belongs to the product, and this tool should not be the reason a newly shipped one is unreachable.

Checking a 2.4 install

bnk status reports the CNEInstance’s aggregate Available condition plus any component condition that is not True, which is what turns a timeout into a diagnosis:

cneinstance:            Available=False (ComponentsNotReady)
cneinstance_not_ready:  F5TmmAvailable=False

That distinction matters on 2.4. CNEControllerAvailable=True can sit next to Available=False with TMM at 0/3 and nothing passing traffic.

The gate itself is CNEControllerAvailable on both lines, and deliberately so: waiting on the aggregate at the gate deadlocks 2.4, because the aggregate cannot go True until the License CR exists and the License CR is created after the gate. The aggregate is checked separately, after licensing, which is where it can actually be satisfied.

If a 2.4 install looks broken in a way that has nothing to do with what you changed, check the License CR first. The controller refuses to run any resource controller until it is Active, logging License is not enabled.. skip Resource controllers.

gateway status reports the Infra and GatewaySettings CRs alongside the Gateway, and reads the IPAM-allocated VIP from the Gateway’s status — on 2.4 there is no address to compute, only one to read.

Support status

2.4 is verified against 2.4.0-EA: bnk up reaches all eighteen CNEInstance conditions with TMM at 5/5 and the licence Active, on a single-NIC OCP 4.20 cluster with connected licensing.

What that run covered, and what it did not, because the difference matters:

exercisedbnk up on an adopted cluster — manifest 2.4.0-EA, FAR repo.f5.com with the non-GA service account, three workers, single-NIC, connected licensing
exercisedthe gateway phase — gateway up applies Infra and GatewaySettings, and both reach Accepted, ResolvedRefs and Programmed = True, with the CNEInstance still 18/18 and the licence still Active afterwards
exercisedfive consecutive clean cluster-lifecycle CLI demo runs on a cluster reproducing F5’s approved reference (6 × cx3d.8x20, OCP 4.21), each a cold start through all six phases
not exercisedthe CI demo — it is docker-driven and needs its own five runs
not exercisedtraffic — every check above reads CR conditions, and Accepted / ResolvedRefs / Programmed are all computed without calling IBM Cloud. #313 was a 2.4 install where the CNE controller could not assume its Trusted Profile, so no VPC address prefixes and no routes were created: all four CRs reported Programmed=True and nothing passed traffic in either direction. A condition-based verification cannot see that class of defect
not exercisedmTLS, multi-NIC, disconnected licensing, EgressGateway CRs, and the contract boundary

Every conformance setting was confirmed on the cluster, not inferred from a plan: deploymentSize: Tiny, tmmReplicas: 3, wholeCluster: false paired with watchNamespaces: ["All"], pod placement present, demoMode off, GATEWAY_API_VERSION: 1.5.0, and the three TMM environment settings.

The teardown namespace repair fired in all five runs and in every teardown between them. That is worth stating plainly: the 2.4 finalizer deadlock is not intermittent, so without the repair a 2.4 cluster cannot be reinstalled into at all.

Running the gateway phase is what found the three defects that made it impossible before it was run: the gateway plan failed outright on 2.4, nothing set USE_GATEWAY_SETTINGS so no controller ever reconciled the CRs, and the cneinstance_advanced_env override meant to let you set it by hand was read by no terraform at all. None of the three was visible in the source — the third least of all, since the variable’s name appears everywhere it should.

2.4 EA is single-NIC throughout — one NetworkAttachmentDefinition, one external-vlan network in the Infra CR, and no second data-plane interface anywhere. An earlier expectation that 2.4 would bring multi-NIC is withdrawn, and the verified run was single-NIC, so it is evidence for single-NIC and nothing else.

support_matrix.yaml records the same thing in the same terms. Its rule is worth repeating, because it is what keeps that file useful: a cell with no coverage is a claim nobody has checked.

Tearing down

Three-phase note. With the Cluster / BNK / Testing split there is now a fourth destroy verb, roksbnkctl testing down (destroys the jumphost rig, leaving the cluster and BNK intact), and the bare roksbnkctl down tears down BNK || Testing in parallel, then the cluster. cluster down refuses while either BNK or Testing state exists. The decision tree below is written for the earlier two-phase shape; see Chapter 8a §“Teardown ordering and the cluster down guard” for the three-phase ordering and guard.

roksbnkctl down, roksbnkctl bnk down, and roksbnkctl cluster down are the three destroy verbs — the inverses of up, bnk up, and cluster up respectively. This chapter covers what each one removes, the ordering constraint between them, the refusal messages you’ll hit if you ask for the wrong one, what survives a destroy, the --auto flag for non-interactive runs, and the workspace-cleanup story.

The phase-aware decision tree

Which verb do you want? The shape of your workspace and your intent both matter. Start here:

I want to keep the cluster and just tear down the BNK trial:
    → roksbnkctl bnk down

I want to tear down everything (cluster + trial):
    → roksbnkctl down

I want to tear down only the cluster (no trial currently deployed):
    → roksbnkctl cluster down

The big rule, stated up front: destroy in reverse of create. Trial first (bnk down), cluster second (cluster down). The unscoped roksbnkctl down does this ordering for you — it runs the trial destroy first and then the cluster destroy. You don’t have to think about ordering; down is the safe default.

The phase-scoped commands (bnk down, cluster down) are the precision tools — they let you keep one phase across many cycles of the other. They also refuse loudly if you ask them to do something that would orphan resources or that the shape doesn’t allow. The full refusal catalogue is in §“Refusal messages catalogue” below; the rule of thumb is that the error message always names the verb that would actually work.

The three destroys

There are three teardown verbs matching the three slices of state:

roksbnkctl down — shape-aware composite

The unscoped down is a shape-aware composite: it detects the on-disk shape of the workspace and dispatches to the right phase destroys in the right order.

roksbnkctl down
Workspace shapedown does
Split (cluster + trial)trial destroy → cluster destroy
ClusterOnly (only cluster applied)cluster destroy
Emptyno-op success: ✓ Nothing to destroy in this workspace — no phase has any state.

This is the safe default — down always does the right thing regardless of shape.

roksbnkctl bnk down — destroy the BNK trial only

Tears down everything the trial phase created — the flo / cis Helm releases, the cne_instance and license custom resources, and the cluster-side namespaces / Secrets / ServiceAccounts / RoleBindings / SCC bindings — and leaves the cluster running. These are all real terraform resources (helm_release, kubernetes_*, and alekc/kubectl kubectl_manifest), so bnk down is an ordinary terraform destroy that deletes the CRs finalizer-aware.

roksbnkctl bnk down

What survives:

  • The ROKS cluster itself
  • cert-manager
  • The registry COS instance and its bucket contents (FAR images, license artefacts)
  • The TGW jumphost
  • All cluster-phase Terraform state under state-cluster/
  • cluster-outputs.json (the cluster is still registered)
  • The workspace’s config.yaml

Roughly 41 resources destroyed on a clean trial-only bnk down. Time is dominated by Helm’s pre-delete hooks and the cne_instance finaliser unwind — usually 2-5 minutes total.

bnk down is a no-op success (exits 0, “nothing to do”) on Empty and ClusterOnly workspaces — there’s no trial state to destroy, so it leaves the cluster alone and returns cleanly. This lets a tool like bnk-forge call bnk down unconditionally in a reverse-order teardown without a missing-trial error blocking the cluster destroy. See §“Refusal messages catalogue” for the exact text.

roksbnkctl cluster down — destroy the cluster phase

Tears down the cluster + cluster-shared services: the ROKS cluster, transit gateway, registry COS instance, cert-manager Helm release, and the TGW jumphost.

roksbnkctl cluster down

What survives:

  • The workspace’s config.yaml
  • ~/.roksbnkctl/<workspace>/state/ (now empty of resources but the directory persists)
  • ~/.roksbnkctl/<workspace>/state-cluster/ Terraform state files (the cluster-side state itself is empty; the directory and terraform.tfstate persist)

Roughly 36 resources destroyed. The ROKS cluster destroy alone is 5-10 minutes; everything else is fast.

The post-destroy cleanup deletes cluster-outputs.json automatically — the workspace no longer has a registered cluster.

Order matters: trial first, then cluster

The upstream HCL’s resource graph requires this ordering. The trial-phase resources have implicit dependencies on cluster-phase resources (they live in the cluster, after all), and Terraform’s destroy graph traverses dependencies in reverse. If the cluster phase tries to destroy first, the trial phase’s resources are still there — finalisers block the destroy of the cluster’s namespaces, the cluster-side SCC bindings reference SCCs that are in the way, and so on.

roksbnkctl cluster down enforces this ordering with a hard refusal: if the trial state has any resources in it, cluster down errors out and points you at bnk down (or down) instead. Even --auto won’t bypass the guard, because correctness, not confirmation, is the issue. The full refusal text:

$ roksbnkctl cluster down
BNK trial state exists in this workspace; run `roksbnkctl bnk down` first
(or `roksbnkctl down` to tear down both phases)

So in practice, always destroy the trial before the cluster. The unscoped down does this ordering for you on a split workspace; the phase-scoped pair is bnk down then cluster down.

The clean teardown sequence — split workspace, explicit phase commands:

# 1. Destroy the BNK trial
roksbnkctl bnk down --auto

# 2. Now safe to destroy the cluster phase
roksbnkctl cluster down --auto

# 3. (Optional) Delete the workspace itself
roksbnkctl ws delete <name> --force

Or the one-shot equivalent:

# 1. Tear down both phases in order
roksbnkctl down --auto

# 2. (Optional) Delete the workspace itself
roksbnkctl ws delete <name> --force

If you roksbnkctl up against a registered cluster (one you didn’t cluster up yourself), step 2 doesn’t apply — the cluster wasn’t yours to destroy. Just bnk down the trial and stop there, then optionally unregister by deleting cluster-outputs.json.

SSH key cleanup. If init generated a testing SSH key and you accepted the prompt to copy it into ~/.ssh/, ws delete removes those copied files too, so a generated key doesn’t outlive its workspace. It deletes only the files init actually created (recorded in resources.copied_ssh_key_files) — a pre-existing ~/.ssh key with the same name is never touched. The confirmation prompt names the files it will remove.

roksbnkctl cleanup — recovering from a failed down

terraform destroy is not always clean. A transient IBM Cloud API error, a resource stuck behind a finaliser, or a down you Ctrl-C’d partway can leave orphaned cloud resources that Terraform no longer tracks — and a re-run of down may not finish the job. The classic symptom is a follow-up up that fails with … name is not unique because a half-deleted security group or VPC is still there.

roksbnkctl cleanup is the recovery tool. It sweeps your IBM Cloud account for every resource named after the workspace prefix (<prefix>-*) and deletes them in dependency order — regardless of what Terraform state says.

roksbnkctl cleanup            # scan, list, confirm, delete
roksbnkctl cleanup --dry-run  # scan + list only, delete nothing
roksbnkctl cleanup --auto     # skip the confirmation prompt

What it deletes

Everything whose name is the workspace prefix or a <prefix>- child, across:

  • VPC (per region): virtual server instances (jumphosts), floating IPs, public gateways, subnets, security groups, SSH keys, and VPCs.
  • Transit Gateway (and its connections).
  • The registry COS instance.
  • The ROKS cluster itself.
  • The BNK trusted profile (named after the cluster ID — found via cluster-outputs.json, so it’s still identifiable after the cluster is gone).

Deletion runs in reverse-dependency order (instances → floating IPs → public gateways → subnets → security groups → VPCs → Transit Gateway → registry COS → cluster → trusted profile) so children go before their parents. It is best-effort: a failure on one resource is reported and the sweep continues. Re-run cleanup for anything blocked on an async delete (e.g. a VPC that can’t go until its cluster finishes terminating).

What it will NOT delete: resources this workspace adopted

A name match cannot tell a resource roksbnkctl created from one the workspace adopted. They look identical, and the adopted name usually matches the prefix exactly — a workspace with prefix sm-cli adopting sm-cli-registry-cos is the motivating case.

So cleanup reads the workspace’s adopt decisions and refuses to delete what they name. Adopted resources are still listed, separately, with the config key that spared them:

2 resource(s) match the prefix but are ADOPTED by this workspace and will NOT be deleted:

KIND          NAME                 REGION    ADOPTED VIA
cos_instance  sm-cli-registry-cos            resources.registry_cos.existing
cluster       sm-cli               us-east   cluster.name (create: false)
  (roksbnkctl never created these; remove the config key if you want cleanup to own them)

The protected set is every path where roksbnkctl reads a resource it never creates:

Config keyKind
cluster.name with create: falsethe ROKS cluster
resources.transit_gateway.existingtransit gateway
resources.registry_cos.existingCOS instance
resources.client_vpc.existingVPC (by name)
resources.cluster_vpc.existingVPC (by ID)
resources.testing_ssh_key_nameSSH key

The SSH key has no create toggle because it is never created: the testing module reaches it only through a data source, so whenever it is named, it is adopted.

This is why --auto is safe to schedule. Protected resources are removed from the set before the confirmation step, so the delete loop never sees one. --auto skips the prompt, not the protection.

Note that terraform was never the danger here. Adopted resources are read through data sources, and data sources are never destroyed — so down was always safe. It is cleanup’s independent sweep that can reach them, precisely because it does not consult Terraform state at all. That independence is the feature (it catches what state lost) and was the defect (it caught what state never owned).

How the Transit Gateway is handled

A gateway cannot be deleted while anything is attached to it, so cleanup detaches its connections first, waits for them to actually clear, and only then deletes the gateway. The detach is asynchronous — IBM leaves the connection in deleting for a few seconds — and deleting the gateway during that window fails with:

412 Precondition Failed: Before you can delete this gateway,
you must delete all attached connections.

Which connections it may detach is a deliberate limit, not an oversight:

  • A connection to a VPC this same sweep is deleting is yours; cleanup removes it.
  • Anything else — a VPC under a different prefix, another account’s network, a Direct Link or GRE attachment — belongs to someone else. cleanup refuses the gateway and names what is attached, rather than quietly disconnecting a shared gateway’s other tenants.

Connections mid-transition are a third case, and the two directions are not the same:

statusmeaningwhat cleanup does
deleting / detachingdeparting — going awaywaited for, whoever started it. It will be gone either way, so it is never a reason to refuse the gateway
pendingarriving — still attachingownership decides: yours is waited for; someone else’s is refused straight away

IBM accepts a connection DELETE only from a settled state, so acting on either would fail with 409 invalid_state. The pending case matters more than it sounds: cleanup is most often reached for right after an interrupted up, which is exactly when a connection is still attaching. Sweeping then used to fail and advise a re-run that failed identically until IBM finished attaching — a wait the sweep now does itself.

Ownership is checked before that wait, because it cannot change as a connection settles. A gateway attached to a network outside the sweep is refused on the first listing rather than after the timeout.

If a connection never finishes attaching, the sweep says so rather than timing out twice on something it never had a chance to delete:

transit gateway f5orph-tgw still has a connection attaching after 5m0s:
conn-1 [vpc crn:…, pending] — IBM refuses a DELETE from `pending`, so this
has to finish before the gateway can go. Re-run once it reports `attached`

States that are settled — including failed and suspended — are detached normally. Waiting for attached specifically would mean a broken connection was never cleaned up, which is the opposite of what a sweep is for.

A refusal is not a transient, and an identical re-run will not clear it. Check the region first: cleanup scans only the workspace’s cluster and client regions by default, so the commonest reason a VPC looks foreign is that it is yours and simply was not scanned. Re-run with --all-regions (or --region <name>) and it comes into scope. If the network really is someone else’s, detach it yourself (ibmcloud tg connection-delete <gateway-id> <connection-id>) or delete it, then sweep again.

This matters most for the shared-gateway topology in Sharing a Transit Gateway, where one gateway deliberately outlives the clusters attached to it.

Which regions it scans

By default cleanup scans the workspace’s cluster region plus the testing-client region (from config.yaml’s resources.client_region and cluster-outputs.json). If resources landed in a region not recorded in config, add it explicitly or sweep everything:

roksbnkctl cleanup --region ca-tor   # add a specific region to the scan
roksbnkctl cleanup --all-regions     # sweep every IBM Cloud region (slower)

Caveats

  • It matches purely on the <prefix>- name convention. If unrelated resources happen to share the workspace prefix, they match too — always review the --dry-run (or the pre-delete confirmation) list first.
  • It needs a prefix. A workspace with no configured prefix has nothing to match; cleanup reports there’s nothing to sweep.
  • It does not touch Terraform state or config.yaml. After a successful sweep the state files still reference the now-gone resources; run roksbnkctl ws delete <name> --force to drop the workspace, or re-up to rebuild.

A typical recovery — a down errored, and up now refuses with “not unique”:

roksbnkctl cleanup --dry-run   # confirm what's stranded
roksbnkctl cleanup --auto      # sweep it
roksbnkctl up --auto           # rebuild clean

Refusal messages catalogue

The phase-scoped destroy verbs refuse loudly when the shape doesn’t allow what you’ve asked for. Every refusal names the verb that would actually work. If you hit one in the wild, grep your terminal output for the message text and you should land here:

Command + shapeRefusal textResolution
bnk down on Empty or ClusterOnly(not a refusal — exits 0) ✓ No BNK trial state to destroy in this workspace — nothing to do.No-op success — no trial is deployed, so the command returns cleanly and leaves the cluster alone. If you want to destroy the cluster, use roksbnkctl cluster down.
testing down with no jumphosts(not a refusal — exits 0) ✓ No testing jumphost state to destroy in this workspace — nothing to do.No-op success — the testing phase is optional, so down returns cleanly when nothing was provisioned.
gateway down with no gateway state(not a refusal — exits 0) ✓ No gateway phase state to destroy in this workspace — nothing to do.No-op success — the gateway phase is opt-in, so down returns cleanly when it was never applied.
cluster down on SplitBNK trial state exists in this workspace; run `roksbnkctl bnk down` first (or `roksbnkctl down` to tear down both phases)Run bnk down first to remove the trial, then cluster down for the cluster — or roksbnkctl down to do both in one shot.
cluster down on Empty(not a refusal — exits 0) ✓ No cluster state to destroy in this workspace — nothing to do.No-op success — the cluster hasn’t been provisioned.
down on Empty(not a refusal — exits 0) ✓ Nothing to destroy in this workspace — no phase has any state.No-op success — no phase has any state.
any down on an uninitialised workspaceworkspace "<name>" is not initialised; run `roksbnkctl init` firstA real error, unlike the empty case above. The two are indistinguishable to the state check but mean opposite things: a mistyped -w reporting a successful teardown is how someone concludes a cluster came down while it is still running.

What survives a destroy

The contract: roksbnkctl never destroys local state without explicit consent, and never destroys cloud resources outside its Terraform state.

After a successful down:

SurvivesWhere
Workspace config~/.roksbnkctl/<name>/config.yaml
Workspace directory + state files~/.roksbnkctl/<name>/ (empty state/; state-cluster/ untouched if cluster down not run)
OS keychain entry for the API keyper-workspace, named roksbnkctl/<name>/ibmcloud_api_key
~/.kube/configleft in place
The cluster (if only trial was destroyed)runs and bills as before
The registry COS bucket’s contentsFAR images, JWT licenses, schematic state — survive cluster destroy too if the bucket was created outside the bundled HCL
~/.roksbnkctl/known_hostsSSH host keys persist; deleting a workspace does not clear them

Re-running up against a down’d workspace re-creates everything from scratch. The workspace’s config.yaml is preserved precisely so this re-create can use the same inputs without re-prompting.

The COS bucket point is worth highlighting: the bundled HCL provisions the COS instance but generally does not provision the buckets inside it (those are written by post-apply provisioners or by the BNK runtime itself). When cluster down destroys the COS instance, the bucket goes with it — but if the COS instance was created out-of-band (e.g. by a registered cluster’s owner) and roksbnkctl is just attaching, then cluster down doesn’t apply and the COS survives.

--auto for non-interactive runs

All three destroy commands prompt for confirmation by default. On a Split workspace the composite down takes a single up-front confirmation that names both phases, then runs trial → cluster without re-prompting — so you can’t accidentally answer “yes” to the trial and miss the cluster gate:

$ roksbnkctl down
This will destroy BOTH the BNK trial AND the cluster phase for workspace "default" (ROKS + transit gateway + registry COS + cert-manager + jumphost).
Continue? [y/N]: 

On LegacySingle the monolithic prompt is unchanged (one state, one destroy):

$ roksbnkctl down                # LegacySingle shape
This will destroy workspace "default"'s resources.
Continue? [y/N]: 
$ roksbnkctl bnk down
This will destroy workspace "default"'s resources.
Continue? [y/N]: 
$ roksbnkctl cluster down
This will destroy the cluster phase for workspace "default" (ROKS + transit gateway + registry COS + cert-manager + jumphost).
Continue? [y/N]: 

--auto skips the prompt — required for CI / scripted pipelines:

roksbnkctl down --auto
roksbnkctl bnk down --auto
roksbnkctl cluster down --auto

--auto does not override the shape-based refusals (see §“Refusal messages catalogue” above) — those are correctness guards, not confirmation prompts. If trial state is present, cluster down --auto still refuses.

Like up, transient errors retry

down doesn’t share up’s explicit retry-on-transient-error logic, but Terraform’s destroy is naturally idempotent: re-running down after a partial destroy picks up where the previous run left off. If you see a transient network error during destroy, just re-run:

roksbnkctl down --auto
# (some resources destroyed, then transient error)

roksbnkctl down --auto
# (picks up where it left off, completes)

The same applies to cluster down. ROKS cluster destroy specifically can take longer than expected when the master is propagating its delete state — wait a few minutes and re-try if you see master-not-found errors.

Cleaning up workspaces

A successful down leaves the workspace directory in place. You usually want to clean that up too:

roksbnkctl ws delete <name> --force

One safety rail on ws delete: it refuses if Terraform state still lists resources (unless --force) — which catches the case where you forgot to run down first. You can delete the current workspace directly; the pointer moves to another existing workspace (or clears when none remain), so you don’t need to switch away first (see Chapter 6).

The --force flag overrides the state check — but if you ws delete --force a workspace that still has provisioned cloud resources, you’ll have stranded them. Recover with roksbnkctl cleanup: run it before deleting the workspace (it reads the prefix from config.yaml), or recreate the workspace’s config.yaml with the same prefix and run it after.

The full clean-as-you-go teardown (scripts/e2e-test.sh Phase D destroys; Phase H deletes):

# 1. Destroy the trial
roksbnkctl down --auto

# 2. Destroy the cluster phase
roksbnkctl cluster down --auto

# 3. Delete the workspace — even if it's the current one (the pointer auto-switches)
roksbnkctl ws delete default --force

Cost note: an undestroyed cluster keeps billing

ROKS clusters bill at roughly $0.30/hour per cluster + worker pool — call it $7/day for a 2-worker cluster, plus a few cents/day for the VPC / load balancers / COS / jumphost. A forgotten cluster can rack up real cost over a weekend.

To verify what’s still running in your account:

  1. IBM Cloud console → Kubernetes → Clusters — every cluster, billing or not.
  2. IBM Cloud console → VPC Infrastructure → VPCs — networks left over after a partial destroy.
  3. IBM Cloud console → Resource list — exhaustive view of everything in the account, filterable by RG.

If you find a leaked cluster from a past roksbnkctl run, the right move is to re-attach to it via roksbnkctl cluster register <name> and then cluster down --auto — roksbnkctl cleans up cleanly when it has the cluster in its state. Manually deleting via the console works too but leaves dangling VPCs and security groups that the bundled HCL would have cleaned up.

roksbnkctl status and roksbnkctl cluster show both report the cluster identity recorded in cluster-outputs.json, but they don’t probe for “are there other clusters in this account?” — that’s deliberately not their job. The IBM Cloud console is the canonical source of truth for what’s billing.

Workspace deletion ≠ destroy

A subtle but important distinction. roksbnkctl ws delete removes the local workspace directory and the OS-keychain API key entry. It does not destroy any cloud resources. If you ws delete --force without first running down / cluster down, the cloud resources keep running and you’ve lost the local Terraform state that roksbnkctl would use to destroy them.

In that scenario, recovery is:

  1. Find the leaked cluster in the IBM Cloud console.
  2. Recreate the workspace: roksbnkctl init -w recovery.
  3. Register the existing cluster: roksbnkctl cluster register <leaked-cluster-name>.
  4. Then run roksbnkctl cluster down --auto to destroy it cleanly.

The Terraform state is regenerated implicitly during register + plan; the resources roksbnkctl would otherwise have tracked get re-discovered through the IBM SDK lookups. It’s not seamless, but it’s recoverable.

The ws delete --force flag’s “still has resources” check exists exactly to prevent this scenario — don’t bypass it without thinking about the consequences.

Worked example: register an existing cluster, deploy BNK, tear down

End-to-end Part III scenario: somebody on your team already provisioned a ROKS cluster manually via the IBM Cloud console (or via a different terraform tree); you need to deploy BNK on top of it using roksbnkctl, validate, and tear the whole thing down cleanly. The flow exercises Chapter 9, Chapter 10, and this chapter end-to-end.

# 1. Workspace bootstrap — same as a fresh deploy
roksbnkctl init -w preexisting
# (answer prompts for region + resource group; pick the values matching
#  the existing cluster's location)

# 2. Register the already-running cluster into the workspace
roksbnkctl cluster register existing-bnk-cluster -w preexisting
# Expected:
#   → Discovering cluster "existing-bnk-cluster" via IBM Cloud API ...
#   ✓ Cluster ID: <crn>
#   ✓ Wrote ~/.roksbnkctl/preexisting/cluster-outputs.json
#   ✓ Fetched admin kubeconfig to ~/.kube/config (chmod 0600)

# 3. Verify roksbnkctl sees the cluster
roksbnkctl status -w preexisting
# Expected: cluster Ready, workers count, no BNK pods yet

# 4. Deploy BNK on top — `up` is idempotent over the existing cluster
roksbnkctl up --auto -w preexisting
# Expected: terraform applies the cert-manager + flo + cne_instance +
# license modules only; the roks_cluster module sees the cluster already
# exists and skips. ~10-15 min vs ~50 min for a from-scratch up.

# 5. Validate
roksbnkctl test -w preexisting
# Expected: green across connectivity + dns

# 6. Tear down — destroys the BNK overlay; the registered cluster survives
roksbnkctl down --auto -w preexisting
# Expected:
#   → terraform destroy (auto-approved)
#   Destroy complete! Resources: N destroyed.
#   ✓ Workspace "preexisting" state retained at ~/.roksbnkctl/preexisting/

The destroy count N is the BNK overlay + jumphost only — typically 30-40 resources, not the from-scratch ~77 count. cluster register is a discovery-only path: terraform state holds the overlay modules (cert_manager, flo, cne_instance, license) and the testing jumphost, but not the roks_cluster module, because the cluster pre-existed roksbnkctl. down destroys only what terraform knows about, so the registered cluster survives untouched.

If you also want to release the underlying cluster, you have to tear it down through whatever provisioned it originally (the IBM Cloud console, or the separate terraform tree your teammate used). roksbnkctl cluster down only works against clusters roksbnkctl cluster up created in the first place — see Chapter 8 for the cluster-phase boundary.

The full register → up → test → down loop above is what Phase E + Phase H of the e2e plan exercise; see Chapter 23 for the CI version.

Cross-references

Workspace config (config.yaml)

This is the teaching chapter. The field specs live in Chapter 28 — Configuration reference. Chapter 28 is generated from the Workspace struct and checked by a test, so it cannot drift: go there for a field’s name, type, default, and which BNK line it applies to. This chapter explains what each block is for, which combinations make sense, and the judgement calls a schema table cannot express.

This chapter is the guided tour of the per-workspace config.yaml. If you’ve read Chapter 6 — Workspaces you’ve seen the on-disk layout; this chapter zooms in on the YAML file that drives everything else (init, up, down, cluster up, the test suite, the SSH targets, the execution backends).

You don’t usually edit this file by hand. roksbnkctl init generates it interactively; later runs read it. But because every other knob in the tool reads from here, it’s worth knowing what each block means and what happens when you leave one out. Every section below links to the part of Chapter 28 that lists its fields; what you get here is the reasoning around them.

File location

Each workspace’s config lives at:

~/.roksbnkctl/<workspace>/config.yaml

Override the base directory with the ROKSBNKCTL_HOME env var (test fixtures use this; everyday users shouldn’t need it). The file is created mode 0600, inside a workspace directory created 0700 — owner-only, because this file can hold ibmcloud.api_key_b64 and registry.generic_password_b64, and base64 is obfuscation rather than encryption.

Workspaces created by a release before v1.50.0 are on disk at 0644/0755. They are tightened in place the first time roksbnkctl reads them, and it says so once:

⚠ workspace "prod" was readable by other users on this host; tightened 3 path(s) to owner-only.
  config.yaml can hold your IBM Cloud API key and registry password. If this host is shared, rotate them.

The repair fixes the mode from that point on; it does not undo the exposure that already happened. On a shared host, rotate the API key. roksbnkctl doctor reports the current state as the workspace permissions check, and is the only thing that reports a tree that could not be tightened (a read-only mount, or a filesystem with no POSIX modes). On Windows the check is skipped: Go’s Chmod there only toggles the read-only bit, so there is no owner-only mode to assert.

There’s also a global ~/.roksbnkctl/config.yaml at the top level — it holds the current_workspace pointer and other user-wide preferences. That’s a different file with a different schema; this chapter is about the per-workspace one.

When it gets written

ActionEffect on config.yaml
roksbnkctl initCreates the file from interactive prompts. Existing file? Asks before overwriting.
roksbnkctl init --upgrade-tfUpdates tf_source: only; leaves every other field alone.
roksbnkctl targets add <name> ...Adds an entry under targets:.
roksbnkctl targets remove <name>Removes the entry.
roksbnkctl up (post-apply)Auto-populates targets.jumphost if the upstream HCL emitted a TGW jumphost output.
Anything elseReads the file. Doesn’t write back.

Direct hand-editing is supported (the file is plain YAML) but discouraged for fields that have dedicated commands — adding an SSH target via roksbnkctl targets add keeps the schema validation in one place.

Top-level structure

prefix: acme-eu  # base for every account-scoped resource name

ibmcloud:        # IBM Cloud account + auth
  region: ca-tor
  resource_group: default
  api_key_source: keychain
  # api_key_b64: <base64-of-api-key>   # OPTIONAL fallback when keychain unavailable

cluster:         # ROKS cluster identity
  create: true
  name: acme-eu   # equals prefix when init derived the name
  openshift_version: "4.20"
  workers_per_zone: 2

resources:       # per-resource create/adopt toggles
  transit_gateway:   { create: true }
  registry_cos:      { create: true }
  cert_manager:      { create: true }
  bnk:               { create: true }
  tgw_jumphost:      { create: true }
  cluster_jumphosts: { create: false }
  client_vpc:        { create: false }

bnk:             # BNK trial knobs (optional; falls through to upstream HCL defaults)
  cneinstance_size: Small
  far_repo_url: repo.f5.com
  manifest_version: 2.3.0-3.2598.3-0.0.170

test:            # test-suite tuning (optional)
  throughput:
    duration: 30
    streams: 8
  connectivity:
    extra_hosts:
      - https://my.gslb.example.com

tf_source:       # where the Terraform HCL comes from
  type: embedded         # embedded | github | local

targets:         # SSH targets (see Chapter 15)
  jumphost:
    host: 169.45.91.177
    user: ubuntu
    key_source: tf-output:jumphost_shared_key

exec:            # per-tool execution backend defaults (see Chapter 17)
  ibmcloud:  { backend: local }
  iperf3:    { backend: k8s }
  terraform: { backend: local }

cos:             # optional COS supply-chain config
  instance: bnk-supply-chain
  bucket: bnk-artifacts

Every block except ibmcloud:, cluster:, and tf_source: is optional. Omit a block and the tool falls through to either a documented default or the upstream HCL’s own default for terraform variables. The prefix: field and resources: block are also optional — omit them and the workspace renders a sparse terraform.tfvars that falls through to the upstream module default names.

That sketch is not the whole schema. A workspace can also carry gateway: (the Gateway API resources installed after BNK), registry: (an air-gap mirror), state: (remote Terraform state — Chapter 12a), bnkforge: and agent:. Each has its own chapter, and every field of every block is listed in Chapter 28. The sections below walk the blocks you meet first.

prefix:

prefix: acme-eu

The field spec is Chapter 28 §Workspace.

The init interview asks for a workspace prefix and derives every account-scoped resource name from it — acme-eu becomes cluster acme-eu, VPC acme-eu-cluster-vpc, TGW acme-eu-tgw, COS acme-eu-registry-cos, jumphosts acme-eu-jh-tgw / acme-eu-jh-<zone>. This stops two workspaces in the same IBM Cloud account from colliding on the old shared tf-* default names.

A prefix must be a lowercase DNS-style label — start with a letter, then [a-z0-9-], no trailing hyphen — and 35 characters or fewer. The 35-char cap is the ROKS cluster-name limit, the tightest of all the resource types. roksbnkctl validates the prefix — and every name it derives — at init time and re-prompts on overflow; there is no silent truncation. An empty prefix keeps the legacy sparse render (no name variables emitted at all), so old configs are unaffected. The full derivation table, the per-resource length/charset limits, the source citations, and the override path live in Chapter 13 §“Resource naming & collision avoidance”.

ibmcloud:

ibmcloud:
  region: ca-tor
  resource_group: default
  api_key_source: keychain
  api_key_b64: ""

The field spec is Chapter 28 §IBMCloudCfg.

This block is the account context every phase runs against. region is the one field with no default worth having — it decides where the cluster, the VPCs and the COS instance are created, and there is no sensible guess, so init prompts for it and a programmatic load of a config without it fails rather than picking one.

api_key_source pins the credential resolver to a single source. Leaving it empty is the normal case: the resolver then walks its whole chain (env → keychain → the workspace’s own api_key_b64 → a hidden TTY prompt). Pin it when you want a failure rather than a fallback — a CI runner that must use the environment variable and must not silently prompt, say.

api_key_b64 is the last-resort inline copy of the key, for hosts with no usable OS keychain (WSL2 without libsecret is the common one). Base64 is obfuscation, not encryption: anyone holding the file can decode it instantly, so treat config.yaml as a plaintext credential — mode 0600, never committed, rotated if the host is shared. The plaintext field name api_key: is rejected at load time; roksbnkctl refuses to read a workspace config that contains it, and the encoded form is the only inline path. Full discussion in Chapter 14 — Credentials and the resolver chain.

cluster:

cluster:
  create: true
  name: tf-openshift-cluster
  openshift_version: "4.20"
  workers_per_zone: 2
  public_gateway: true          # optional; false = private/disconnected cluster (no egress)
  min_worker_vcpu_count: 16     # optional; worker-flavor auto-select floor
  min_worker_memory_gb: 64      # optional; worker-flavor auto-select floor
  vpc_cidr: 10.242.0.0/16       # optional; address block for a NEW cluster VPC
  network_mode: single-nic      # optional; single-nic (default) | multi-nic

The field spec is Chapter 28 §ClusterCfg. The rest of this section is the reasoning behind the fields that have some.

Create or adopt. create: true builds a ROKS cluster; create: false adopts one that already exists, looked up by name (a cluster name or ID) in ibmcloud.resource_group. Adopting is the common case: most estates build clusters to their own standards and ask roksbnkctl only for BNK. The cluster is deliberately not part of the resources: block below; it kept its own pair of fields because it predates that block and because a cluster is the one thing a workspace cannot do without. When a prefix is set, init fills cluster.name with the prefix itself — the cluster name carries no suffix, for the reason given in Chapter 13.

Quote the OpenShift version. openshift_version: 4.20 is a YAML float, and 4.20 as a float is 4.2. Write openshift_version: "4.20". Leaving it empty is legitimate and takes IBM’s current default, which moves over time — pin it when a run has to be reproducible.

workers_per_zone is per zone. ROKS spans three availability zones in an MZR region, so 2 is a six-node cluster.

public_gateway: false builds a disconnected cluster, and that is a bigger decision than one boolean. It removes the public gateway from every cluster subnet, so the workers have no Internet egress at all. Everything they used to pull now has to be reachable privately: a mirror registry they can route to, VPEs or private service endpoints for the IBM Cloud services the install touches, and disconnected (or F5 License Proxy) licensing instead of the connected JWT path. Get one of those wrong and the failure shows up much later, as an image pull that hangs. Chapter 10a §“A truly disconnected cluster” is the checklist. Note the scope: this governs worker egress only — the cluster master keeps its public API endpoint, so kubectl still works from your desk.

Worker sizing. Leave worker_flavor unset and the cluster module auto-selects the smallest profile meeting min_worker_vcpu_count and min_worker_memory_gb. The auto-select only considers the bx2 family, so any other profile is unreachable without naming it in worker_flavor — F5’s approved reference cluster runs cx3d.8x20, which no combination of minimums can produce.

vpc_cidr is what lets two clusters share a Transit Gateway. Left empty, the VPC uses IBM’s auto address-prefix management, which hands every VPC in a region the same three prefixes. Two roksbnkctl-built clusters in one region then overlap, and a Transit Gateway joining them cannot route to both — it blackholes one, and what you see is intermittent image-pull timeouts rather than a routing error (issue #46). Give each cluster its own block whenever they must share a gateway, which is the norm for disconnected installs, since the cluster reaches its private mirror across that gateway. The block is split into three per-zone prefixes of /n+2, and each cluster subnet is the first /n+8 of its zone — so a /16 gives 256-address subnets (today’s size), a /17 gives 128, and a /18 gives 64. /18 is therefore the smallest value Terraform accepts, and /16 is the one you want unless address space is tight. It is meaningful only for a created VPC; an adopted VPC keeps its own prefixes, and changing the value after the fact warns rather than silently re-planning.

existing_subnet_ids is the other half of “bring your own network”. Adopting the VPC alone is not enough: in an estate that allocates address space centrally, the subnets carry the ACLs and routing that make them acceptable, and a cluster dropped into freshly-created subnets sits outside all of it. Supply one subnet ID per zone, in zone order. Each subnet’s zone is read from the subnet itself, so a reordered list places the cluster differently rather than failing — check the order. It requires resources.cluster_vpc: {create: false, existing: <vpc-id>}, because a subnet cannot be adopted independently of the VPC containing it. ROKSBNKCTL_EXISTING_SUBNET_IDS sets it from the environment, comma-separated.

network_mode is create-time only, and enforced. Omitted means single-nic, which is what every cluster this tool builds is; the field exists to name that, not to change it. multi-nic requires BNK 2.4+. There is no in-place conversion between the two — Terraform would plan a replacement of a running cluster — so a workspace whose network_mode contradicts the cluster recorded in cluster-outputs.json is refused rather than planned. See Chapter 28 §ClusterCfg for which line each value belongs to.

The cluster: block translates to the terraform variables create_roks_cluster, openshift_cluster_name, roks_cluster_id_or_name, openshift_cluster_version, roks_workers_per_zone, cluster_public_gateway, cluster_vpc_cidr and cluster_network_mode — see Chapter 13 and Chapter 29 for the full mapping.

resources:

resources:
  transit_gateway:   { create: true }
  registry_cos:      { create: true }
  cert_manager:      { create: true }
  bnk:               { create: true }
  tgw_jumphost:      { create: true }
  cluster_jumphosts: { create: false }
  client_vpc:        { create: false, existing: shared-client-vpc }
  client_region:     ca-tor               # testing-client region (plain string)

The init interview’s create/adopt answers, one {create, existing} pair per resource. create: true provisions a new, prefix-named resource; create: false declines it, and when a still-enabled resource depends on the declined one, existing: names the pre-existing resource to consume instead.

The field spec — every toggle, plus the jumphost sizing and security-group keys that also live in this block — is Chapter 28 §ResourcesCfg and §ResourceToggle. What the table below adds is the mapping: which terraform variables each toggle drives, and when declining one makes init ask for an existing resource to adopt instead.

Sub-blockRenders intoAsks for existing when…
transit_gatewaycreate_roks_transit_gateway (+ roks_transit_gateway_name)declined and the TGW jumphost is enabled (it needs a TGW)
registry_coscreate_roks_registry_cos_instance (+ roks_cos_instance_name)declined
cert_managerinstall_cert_manager—
bnkdeploy_bnk—
tgw_jumphosttesting_create_tgw_jumphost (+ testing_tgw_jumphost_name)—
cluster_jumphoststesting_create_cluster_jumphosts (+ testing_cluster_jumphost_name_prefix)—
client_vpctesting_create_client_vpc (+ testing_client_vpc_name)TGW jumphost enabled but you decline creating a new client VPC for it
cluster_vpcuse_existing_cluster_vpc + existing_cluster_vpc_id, emitted only when adoptingdeclining it is the adoption — existing: takes the VPC ID, and it is the prerequisite for cluster.existing_subnet_ids

One extra key, client_region, is a plain string rather than a toggle: the region the testing client (TGW jumphost + client VPC) is installed in, set when init asks where to install the test client. It renders into testing_client_vpc_region (omitted → the terraform default applies) and seeds the regions roksbnkctl cleanup scans.

The block is optional — omit it and the sparse render (upstream module default names) applies; a fresh init writes it in full. For what each of those terraform variables defaults to when a toggle is absent, see Chapter 29.

bnk:

bnk:
  cneinstance_size: Small
  far_repo_url: repo.f5.com
  manifest_version: 2.3.0-3.2598.3-0.0.170
  flo_namespace: f5-bnk               # optional; FLO namespace
  flo_utils_namespace: f5-utils       # optional; = flo_namespace ⇒ ONE namespace
  gslb_datacenter_name: ""            # optional; CNEInstance GSLB datacenter
  cert_manager:                       # optional; cert-manager coordinates
    namespace: cert-manager
    version: v1.17.3
  network:                            # optional; data-plane subnets + TMM self-IPs
    vlan_prefixlen: 24                #   self-IP prefix length (F5SPKVlan)
    vlan_prefixlen_external: 23       #   optional; overrides the shared value
    vlan_prefixlen_internal: 26       #   optional; the two VLANs need not match
    tmm_k8s_routes: 172.17.0.0/18     #   pod CIDR TMM routes to
    zones:                            #   one entry per AZ (3 total)
      - ext_vlan_cidr: 10.155.15.0/24
        int_vlan_cidr: 10.254.99.0/24
        int_snat_cidr: 10.10.11.0/24
        int_vip_cidr: 10.135.15.0/24
        external_selfip: 10.155.15.101
        internal_selfip: 10.254.99.101
      # …zones 2 and 3
  flp:
    storage_class: ""                 # optional; FLP PVC StorageClass

Every field here is optional — leave the block out entirely and you get the upstream HCL’s defaults. For a one-page view of every field with its dotted path and the ROKSBNKCTL_* variable that sets it, see the config.yaml cheatsheet — generated from the same struct this chapter teaches, so the two cannot disagree. The field spec is Chapter 28 §BNKCfg, with the sub-blocks broken out as §BNKTrustedProfileCfg, §BNKCertManagerCfg, §BNKFLPCfg and §BNKPreflightCfg. Chapter 28’s line column matters here more than anywhere else in the config: a good part of bnk: is 2.4-only, and manifest_version is the only thing that selects the line.

manifest_version cannot be bumped on a live workspace — bnk up refuses it. The manifest version names the CNEManifest object (bnk-<version>, lowercased), so changing it renames that object. Terraform plans a rename as an in-place update and then aborts with Provider produced inconsistent final plan, and re-running does not converge — the stale value is in terraform state, not on disk.

The failure is not a clean stop. The apply gets partway, having already rolled every pod, and a retry rolls them again: a previous attempt did this three times, exhausted the registry’s pull quota (pull QPS exceeded), and left the cluster with f5-cne-controller, f5-tmm and f5-dssm-sentinel-2 in ImagePullBackOff and no CNEManifest at all (#309). So bnk up now compares the configured version against the one the workspace last applied and refuses before planning, rather than starting an apply that cannot finish.

To move to a new manifest:

roksbnkctl bnk down
roksbnkctl bnk up

That is the supported path, and the one the 2.4.0 GA upgrade was verified with (37 added, 0 changed, 0 destroyed). A cross-line change (2.3 → 2.4) is refused separately and for different reasons — see bnk_line above; that one needs a new workspace, not a reinstall.

cneinstance_size is passed through unvalidated, on purpose. The legal set of sizes is a property of the BNK manifest, not of this tool. Hardcoding a list would go stale the first time F5 adds one and would then refuse a size the product supports, so a size a given manifest does not define is rejected by the operator on the cluster, not here. Tiny is what the BNK 2.4 install guide uses, and on ROKS it is the only size that runs: everything above it requests hugepages (Small 4 GiB, Medium 8 GiB) that the platform has no supported way to allocate. Do not reach for bnk.hugepages — it is a no-op on ROKS and issue #203 explains why. Capacity comes from bnk.tmm_replicas and node size instead; see Appendix C.

gslb_datacenter_name is the whole feature. It is what the CNEInstance calls itself in GSLB, and it is emitted whenever set. There was also a gtm: block carrying the BIG-IP DNS url, username and password; it was removed in #227 because nothing read it — the f5ingress controller binary contains no occurrence of the environment variables it produced, on either BNK line. A config.yaml still carrying bnk.gtm needs no edit: an unrecognised key is ignored.

trusted_profile.service_account is a matcher, not a pointer — and the name differs by BNK line. The IBM Cloud Trusted Profile is the identity the CNE controller assumes to manage the VPC network attachments it creates for TMM, so it works without a stored API key. IBM’s trust relationship compares a pod’s service-account token against crn / namespace / name with EQUALS. Left empty, roksbnkctl derives the name the controller actually runs as, and that is not the same name on both lines:

LineDerived service account
2.3f5-cne-controller-<flo_namespace>-f5-cne-controller-serviceaccount
2.4f5-cne-controller

On 2.3 it is FLO’s helm construction, <release>-<chart>-serviceaccount with the namespace baked into the release name. On 2.4 the CNE controller chart names the account outright. Both names were read off live installs rather than inferred.

Write a name that matches neither and nothing can assume the profile. The install still reports itself healthy. The controller logs the token exchange failing (BXNIM0398E … no matching rule or link found), then falls back to Cloud environment is not IBM or cloud provider instance is nil, skipping cloud address prefix handling and skips VPC address-prefix and route programming altogether — so nothing reaches the listener addresses and egress SNAT replies have no route back. Meanwhile Infra, GatewaySettings, Gateway and EgressGateway all report Programmed=True, because those conditions are computed without ever calling the cloud, and the profile looks correct in the IBM Cloud console the whole time. A status check cannot detect this; only traffic can (roksbnkctl testing exercises exactly that path).

So set it only if you can also make FLO name the account differently — and roksbnkctl cannot, because FLO creates that account when it reconciles the CNEInstance and the CNEInstance spec has no service-account field. In practice this field exists for estates driving FLO by some other means. The derived name is safe to share across clusters: uniqueness comes from the profile name (which carries the cluster name) and from the link’s cluster CRN, not from the account name.

cert_manager: overrides coordinates, not the decision. bnk.cert_manager.namespace / .version say where and which chart; whether cert-manager is installed at all stays on resources.cert_manager.create. The two are easy to confuse because they share a name.

license_mode: f5licenseproxy additionally requires the FLP phase to be up (roksbnkctl flp up) before bnk up runs, so the install has a proxy to point at. The proxy’s endpoint and root CA are not config — they are produced by flp up and read back from flp-outputs.json.

bnk.network: — data-plane subnets + TMM self-IPs

BNK’s data plane (TMM) needs per-availability-zone VLAN subnets, SNAT/VIP ranges, and self-IPs. Leave bnk.network out entirely and the BNK install-guide defaults apply. Set it to match your cluster’s fabric — roksbnkctl init prompts for every field (opt in at “Customize BNK networking?”, seeded with the values shown below), or hand-write the block.

The field spec is Chapter 28 §BNKNetworkCfg and, for one zone’s six CIDRs, §BNKZoneCfg.

Provide all three zones when you set zones — supplying zones replaces the install-guide defaults entirely (they render cneinstance_network_zones, driving the cloud-network-mapping ConfigMap and the external/internal F5SPKVlan CRs). The YAML above shows zone 1’s defaults; zones 2 and 3 follow the same shape one octet up (10.156.16.0/24 and 10.157.17.0/24 externally), and the full default set is the cneinstance_network_zones default in terraform/modules/cne_instance/modules/cneinstance/variables.tf. Zone names are derived from the region and aren’t configurable.

vlan_prefixlen and tmm_k8s_routes are network-wide, shared across all zones. vlan_prefixlen is independent of your VLAN CIDRs and is never derived from them — nothing validates the two against each other. Usually you want them to agree; a deliberate disagreement, paired with static routes, is how a smaller or larger directly-connected block is forced and the remainder steered. The mask can also differ between the two VLANs via vlan_prefixlen_external / vlan_prefixlen_internal: TMM can front a /23 externally while the internal side is a /26, which one shared scalar could not express. Unset either override to inherit the shared value; ROKSBNKCTL_VLAN_PREFIXLEN_EXTERNAL and ROKSBNKCTL_VLAN_PREFIXLEN_INTERNAL set them from the environment.

tmm_k8s_routes is the pod CIDR TMM installs a route toward (TMM_K8S_ROUTES) so it can reach the backend pods on the internal data path. The default is the ROKS default pod subnet — set it if your cluster’s isn’t.

What each zone CIDR becomes on 2.4. The same values drive a different set of objects on each line, which is why they are marked “2.3 + 2.4” rather than being split: on 2.3 they build the cloud-network-mapping ConfigMap and the F5SPKVlan pair, and on 2.4 they become the Infra CR’s IPAM pools — ext_vlan_cidr is the external-vlan-ipam pool and the next hop of every generated route, int_snat_cidr is egress-snat-ipam, and int_vip_cidr is vip-listener-ipam. The exceptions are the ones chapter 28 marks 2.3: int_vlan_cidr (2.4 has no internal VLAN), both self-IPs (2.4 allocates TMM’s VLAN addresses from external-vlan-ipam), and the three vlan_prefixlen* fields (a 2.4 IPAM pool is a CIDR, and a CIDR states its own mask).

test:

test:
  throughput:
    image: networkstatic/iperf3:latest
    duration: 30
    streams: 8
    default_mode: north-south
  connectivity:
    extra_hosts:
      - https://my.gslb.example.com
      - https://internal.example.test

The field spec is Chapter 28 §TestCfg, broken out as §ThroughputCfg, §ConnectivityCfg and §DNSCfg. duration and streams are iperf3’s -t and -P.

throughput.image is the in-cluster iperf3 server fixture, not the client. roksbnkctl test throughput deploys that image as a pod in the cluster and then runs an iperf3 client against it — and the client’s image is a separate thing, chosen by the execution backend (the docker and k8s backends run F5’s bundled ghcr.io/jgruberf5/roksbnkctl-tools-* tool images; see Chapter 17). Changing throughput.image moves the server end only.

The default server image, networkstatic/iperf3:latest, runs as root. OpenShift’s SCC admission overrides the pod’s user with a project-allocated UID, so it works on ROKS; plain Kubernetes with the restricted Pod Security standard does not, and the pod is refused with a RunAsNonRoot admission error. On a non-OpenShift cluster, point throughput.image at the bundled image instead — ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<version>, which is built to run as UID 1000.

connectivity.extra_hosts adds targets to the reachability probe alongside the canonical IBM/F5 endpoints. It exists mainly for disconnected estates: a probe against a public host there reports a failure that says nothing useful, so give it something the cluster can actually reach.

tf_source:

tf_source:
  type: embedded
typeOther fieldsUse case
embedded (default)noneUse the HCL bundled into the roksbnkctl binary via go:embed. The recommended path for users — install one binary, get matched CLI + Terraform together.
githubrepo: "owner/name", ref: "v0.6.1"Pull a tarball from a GitHub release. Useful for testing forks or pinning to a specific upstream tag.
localpath: "/abs/path/to/tf-source"Point Terraform at an on-disk directory. For active development on the HCL itself.

An empty type is treated as embedded (legacy / forgot-to-set). The field spec for repo / ref / path is Chapter 28 §TFSourceCfg; the table above is about which type to pick, not what the fields are.

roksbnkctl init --upgrade-tf is the helper for bumping the source between versions without retyping the rest of the config — see “Editing by hand vs helpers” below.

targets: — SSH targets

targets:
  jumphost:
    host: 169.45.91.177
    user: ubuntu
    key_source: tf-output:jumphost_shared_key
  bastion:
    host: ops.example.com
    user: jgruber
    key_path: ~/.ssh/id_ed25519

Each entry has host, user, an optional port, and exactly one of key_path (a PEM file) or key_source (agent, or tf-output:<name> to pull the key from a Terraform output). The field spec is Chapter 28 §TargetCfg.

The deep reference is Chapter 15 — SSH targets, and the user-facing prose is Chapter 16 — The –on flag and SSH jumphosts. This chapter just notes the block’s place in the overall config.

You don’t typically edit this block by hand. roksbnkctl up auto-populates jumphost post-apply, and roksbnkctl targets add ... populates the rest.

exec: — execution-backend defaults

exec:
  ibmcloud:  { backend: local }
  iperf3:    { backend: k8s }
  terraform: { backend: local }

Per-tool defaults for the --backend system. Each entry is keyed by the tool name (ibmcloud, iperf3, terraform, and others as the matrix grows) and its one field, backend, is specified in Chapter 28 §ExecToolCfg. The table below is the enum of values that field accepts:

BackendNotes
localos/exec against the host binary. The default for terraform and ibmcloud.
dockerRuns inside a vendored container image. Frozen toolchain version, no host install.
k8sRuns inside the cluster (long-lived ops pod or one-shot Job). Default for iperf3.
sshRuns on a registered SSH target. Format: ssh:<target-name>.

A --backend <value> flag on the command line overrides the workspace config for that single invocation. The flag wins; the config sets the default.

The iperf3 default is k8s because measuring throughput from a laptop’s internet uplink isn’t useful — you want the test to run from a network location adjacent to or inside the cluster. The local default is wrong for that tool, so the workspace config flips it.

Chapter 17 — Execution backends covers the full backend matrix; Chapter 18 — Choosing a backend per tool is the decision tree.

cos: — COS supply-chain (optional)

cos:
  instance: bnk-supply-chain
  bucket: bnk-artifacts
  region: us-south
  upload:
    - source: ./local/f5-far-auth-key.tgz
      key: f5-far-auth-key.tgz
    - source: ./local/subscription.jwt
      key: subscription.jwt

The field spec is Chapter 28 §COSCfg and §COSUpload.

This is the orchestration COS — the bucket holding the FAR auth tarball and the subscription JWT, which is a different thing from the registry COS instance the resources: block can create. instance / bucket / region are honoured by both the terraform render (ibmcloud_cos_instance_name / ibmcloud_resources_cos_bucket / ibmcloud_cos_bucket_region) and the registry FAR-file resolver, so a customer-owned bucket is used consistently across both.

The bucket init provisions is not the bare default. COS bucket names share one global namespace, like S3, so a generic base such as bnk-artifacts frequently collides with a bucket some other account already owns. When init provisions the supply chain it therefore appends a short token derived from your account ID — bnk-artifacts-<first 12 of the account id> — and writes that name into cos.bucket. Keying the suffix off the account (rather than the COS instance) makes the name both globally unique and stable and discoverable: a second workspace created from the same account’s API key derives the same name and reuses the bucket the first one provisioned, instead of making a second copy of the same artefacts.

upload performs pre-flight uploads from local files into that bucket before the phases that read them run — useful in CI, where the supply-chain artefacts are produced by the pipeline rather than staged by hand.

The block is optional — if you’ve already populated COS by hand or via the upstream HCL’s roks_cos_instance_name variable, you don’t need it. Chapter 25 — COS supply chain management covers the full workflow.

Behaviour when fields are missing

roksbnkctl falls through three layers in order: workspace config → upstream HCL default → fail.

Missing fieldBehaviour
prefixEmpty ⇒ legacy sparse terraform.tfvars (no name variables; upstream tf-* module defaults). init prompts for one on a fresh interactive run.
resourcesEmpty ⇒ no per-resource toggles rendered; goes with the empty-prefix legacy path. A fresh prefix-driven init writes the full block.
ibmcloud.regionroksbnkctl init prompts; programmatic loads error with “region is empty”.
ibmcloud.api_key_sourceResolver walks the full chain (env → keychain → config → prompt).
ibmcloud.api_key_b64Skipped in the resolver chain.
cluster.nameinit prompts; programmatic loads error.
cluster.openshift_versionEmpty string passed to upstream HCL; the module picks the current default.
cluster.workers_per_zoneFalls through to 1 (upstream default).
bnk.*Field is omitted from the generated terraform.tfvars and the upstream HCL default applies.
tf_sourceTreated as type: embedded (legacy default).
targets.*Block absent ⇒ roksbnkctl --on jumphost errors with “no target named jumphost”; auto-populated by up.
exec.*Per-tool defaults: ibmcloud→local, terraform→local, iperf3→k8s, DNS probe→local. Override per-tool via this block, or per-invocation via --backend.
cos.*No pre-flight uploads, and both the terraform render and the registry FAR resolver fall back to the built-in orchestration-COS defaults (bnk-supply-chain / bnk-artifacts / us-south) — which are also the upstream HCL’s.

The general rule: if you don’t write it in config.yaml, roksbnkctl doesn’t write it into terraform.tfvars, and the upstream HCL’s default = ... clause takes over. The full upstream defaults are listed in Chapter 29.

How --var-file interacts with config.yaml

Both roksbnkctl up and roksbnkctl plan/apply/destroy accept the same --var-file flag terraform itself accepts (repeatable, later files win). The layering rule is:

1. config.yaml-derived terraform.tfvars        (written first by roksbnkctl)
2. ~/.roksbnkctl/<ws>/terraform.tfvars.user  (optional manual override)
3. --var-file <path>                           (CLI; repeatable)

Later layers override earlier. Concretely: config.yaml’s cluster.workers_per_zone: 2 writes roks_workers_per_zone = 2 into the generated tfvars. If you then pass --var-file ./bigger.tfvars containing roks_workers_per_zone = 5, terraform sees 5. The config.yaml value didn’t get re-applied; --var-file wins.

The terraform.tfvars.user middle layer is for when you want a workspace-local override that survives across runs without modifying config.yaml — it’s typically used for fields the YAML schema doesn’t model (rare; the schema covers the common knobs). Chapter 13 goes deep on this.

If you want a workspace to always apply a raw terraform.tfvars without re-passing --var-file on every phase command, drop it verbatim at ~/.roksbnkctl/<ws>/terraform.tfvars.user (mode 0600, sibling to config.yaml); subsequent up / plan / apply / down against bare -w <ws> pick it up automatically for both phases. To seed the workspace’s config.yaml itself from a file, use roksbnkctl init -w <ws> --config-file ./config.yaml — see Chapter 6 §“Skip the interview: init --config-file”.

The IBMCLOUD_API_KEY is the one exception that never goes through tfvars on disk. It’s passed as a TF_VAR_ibmcloud_api_key env var on the terraform invocation. --var-file cannot supply the API key — the resolver chain in Chapter 14 is the only path.

Editing by hand vs helpers

Several commands manage subsets of config.yaml so you don’t have to:

SubsetHelper
Whole file (interactive)roksbnkctl init
tf_source: onlyroksbnkctl init --upgrade-tf
targets: blockroksbnkctl targets add/remove
registry: blockroksbnkctl registry target …
test.connectivity.extra_hostsroksbnkctl test hosts add/remove/clear
state: block (local ↔ COS/S3)roksbnkctl state local / state s3 … / state show
exec: block (per-tool backend)roksbnkctl backend set/unset/show
bnkforge: blockroksbnkctl bnkforge enable/disable/status
ibmcloud.api_key_b64roksbnkctl init (after entering the key, it offers to save)

When you do edit by hand, the load-time validators run on next roksbnkctl invocation:

  • The plaintext-secret heuristic rejects an api_key: field (it must be api_key_b64: to be tolerated).
  • Workspace name validation runs on directory access (workspace names must match [A-Za-z0-9][A-Za-z0-9_.-]{0,63}).
  • YAML parse errors surface a line number.

If a hand edit breaks the file, every command that reads the workspace fails fast with the parse error path, so you’ll know within one invocation.

Worked example: bootstrap a workspace from scratch

End-to-end Part IV scenario: brand-new laptop, no roksbnkctl workspaces yet, an IBM Cloud API key in your password manager. Goal: a usable workspace with the key in the OS keychain, the right region + resource group resolved, and terraform.tfvars ready to drive the HCL.

The transcript below is captured against the shipped binary. Prompts are written to stderr, indented two spaces, with the label left-padded; the [default] (or [Y/n] / [y/N]) in brackets is what you get on a bare Enter. Here the operator types acme-eu for the prefix, keeps the cluster + COS + cert-manager + BNK + TGW-jumphost defaults, declines the Transit Gateway (adopting shared-corp-tgw), declines a new client VPC for the jumphost (adopting shared-client-vpc), and declines per-zone cluster jumphosts.

$ roksbnkctl init -w dev
Setting up workspace "dev"

  Region                         [ca-tor]:

→ Verifying IBM Cloud credentials...
✓ IBM Cloud user you@example.com (account 1a2b3c…)

  Resource group                 [default]:
✓ Resource group "default" (id 0d1e2f…)

  Workspace prefix (≤ 35 chars)  [dev]: acme-eu
  Create new ROKS cluster?       [Y/n]: y
  OpenShift version              [4.20]:
  Workers per zone               [1]: 2
  Create registry COS instance?  [Y/n]: y
  Create Transit Gateway?        [Y/n]: n
  Install cert-manager?          [Y/n]: y
  Deploy BIG-IP Next (BNK)?      [Y/n]: y
  Create TGW test jumphost?      [Y/n]: y
  Create a new client VPC for it? [y/N]: n
  Existing client VPC name       : shared-client-vpc
  Existing Transit Gateway name  : shared-corp-tgw
  Create per-zone cluster jumphosts? [y/N]: n

Resolved resource names for prefix "acme-eu":
  cluster                acme-eu
  cluster VPC            acme-eu-cluster-vpc
  registry COS instance  acme-eu-registry-cos
  transit gateway        shared-corp-tgw  (existing)
  TGW jumphost           acme-eu-jh-tgw
  client VPC             shared-client-vpc  (existing)

✓ Wrote /home/you/.roksbnkctl/dev/config.yaml

(The IBM Cloud API key is resolved before the prompts above via the credentials resolver chain — env, then keychain, then the workspace api_key_b64, then a hidden TTY prompt that offers to save the key. When the key is already in your environment or keychain, no key prompt appears, which is the common re-init case shown here.)

Two ordering details worth noting against the transcript: the OpenShift version and Workers per zone prompts appear only when you answer yes to “Create new ROKS cluster?” (a declined cluster prompts instead for an existing cluster name/ID), and the Existing Transit Gateway name prompt fires after the client-VPC questions — it is only asked when the TGW was declined and the TGW jumphost is enabled (the jumphost rides the gateway, so it needs the existing one’s name).

Three things to note in that flow:

  • The prefix prompt validates and re-prompts. Enter a prefix that’s too long, starts with a digit, or has a trailing hyphen, and init rejects it with the offending resource, its computed length, its limit, and the maximum allowable prefix length — then asks again. In a non-TTY context (CI), an invalid default is a hard error rather than a silent truncation. The cap is 35 characters (the ROKS cluster-name limit). See Chapter 13 §“The length / charset limits”.
  • Declining a create toggle triggers existing-resource discovery — but only when something still needs it. Here the Transit Gateway was declined, and because the TGW jumphost is enabled (it rides the TGW), init asks for an existing TGW name (the Existing Transit Gateway name prompt). The same pattern drives the Existing client VPC name follow-up. Declining a resource that nothing else depends on (e.g. cluster jumphosts) just turns it off with no follow-up.
  • The resolved name plan is printed before the workspace is saved, so you see exactly what roksbnkctl will ask IBM Cloud to create (and which names it will adopt rather than create) before committing.

The resulting ~/.roksbnkctl/dev/config.yaml:

prefix: acme-eu
ibmcloud:
  region: ca-tor
  resource_group: default
  api_key_source: keychain
cluster:
  create: true
  name: acme-eu
  workers_per_zone: 2
resources:
  transit_gateway:   { create: false, existing: shared-corp-tgw }
  registry_cos:      { create: true }
  cert_manager:      { create: true }
  bnk:               { create: true }
  tgw_jumphost:      { create: true }
  cluster_jumphosts: { create: false }
  client_vpc:        { create: false, existing: shared-client-vpc }
tf_source:
  type: embedded

That’s the minimum a prefix-driven workspace writes. Everything else (bnk:, test:, targets:, exec:, cos:) is empty and falls through to defaults. A workspace created without a prefix (e.g. an old config) omits both prefix: and resources: and renders the legacy sparse terraform.tfvars. The API key can also be supplied non-interactively from your password manager’s CLI by setting IBMCLOUD_API_KEY in the environment of the init invocation:

op here is the 1Password CLI; the op://... URI is its secret-reference scheme. Any password-manager CLI that prints a secret to stdout works the same way — Bitwarden (bw), gopass, aws secretsmanager get-secret-value, Doppler, etc. — the only thing roksbnkctl cares about is that IBMCLOUD_API_KEY is set in the environment when init runs.

# Alternative: pre-set IBMCLOUD_API_KEY so init resolves it from env rather than prompting
IBMCLOUD_API_KEY=$(op read 'op://Private/IBM Cloud/api-key') roksbnkctl init -w dev

Chapter 14 §“The IBMCLOUD_API_KEY resolver chain” covers the full env → keychain → workspace api_key_b64 → TTY-prompt order; this env-var path is the first link in that chain, so anything init resolves at bootstrap time follows the same precedence later invocations use. Once init has saved the key to the OS keychain (the default sink), no further prompting is needed. init still prompts interactively for the remaining workspace metadata (region, resource group, workspace prefix, and the per-resource create toggles — the cluster name is derived from the prefix when you create a cluster, not prompted) — a fully non-interactive bootstrap is on the v1.x roadmap.

You don’t render terraform.tfvars by hand. roksbnkctl derives it from config.yaml automatically on every up / plan / apply, writing it into the phase’s state directory (~/.roksbnkctl/dev/state/terraform.tfvars). With the prefix: acme-eu set above, that generated file names every account-scoped resource from the prefix:

# ~/.roksbnkctl/dev/state/terraform.tfvars (generated; do not hand-edit)
openshift_cluster_name    = "acme-eu"
roks_cluster_vpc_name     = "acme-eu-cluster-vpc"
roks_cos_instance_name    = "acme-eu-registry-cos"
create_roks_transit_gateway = false
roks_transit_gateway_name = "shared-corp-tgw"
testing_tgw_jumphost_name = "acme-eu-jh-tgw"
testing_client_vpc_name   = "shared-client-vpc"
# ...

(The separate roksbnkctl tfvars command is unrelated — it writes a copy of the upstream terraform.tfvars.example starter template to edit by hand, not this config-derived render. See Chapter 13 §“roksbnkctl tfvars — bootstrap a starter”.)

Chapter 13 covers the precedence rules between the generated terraform.tfvars, terraform.tfvars.user (the hand-edit overlay), and --var-file.

Finally, verify the workspace is healthy before the first real up:

# 3. Sanity-check
$ roksbnkctl doctor -w dev
✓ terraform     1.6.2  on PATH
✓ IBMCLOUD_API_KEY resolves via keychain
✓ region "ca-tor" accepts the key (IAM round-trip OK)
✓ resource group "default" exists (id: ...)
✓ workspace dev healthy

From here, roksbnkctl up --auto -w dev is the next step (see Chapter 7 — Quick start). You can layer on bnk:, test:, targets:, exec:, cos: blocks by hand-editing config.yaml whenever you need them — init only writes the minimum to keep first-run friction low.

Cross-references

Remote state (COS / S3)

By default, roksbnkctl keeps terraform state in local files under your workspace — ~/.roksbnkctl/<workspace>/<phase>/terraform.tfstate, one per phase. That’s the right thing on a laptop, and it’s fine for a single containerized runner with a mounted volume. It breaks down in two cases:

  • Stateless / ephemeral runners — a CI container with no persistent volume loses its state between runs.
  • Fleet / parallel runs — local state has no locking, so two runs against the same state can corrupt it.

The fix is an optional remote state backend on IBM Cloud Object Storage (which exposes an S3-compatible API): each phase’s state lives in a bucket, with a lock, so state is decoupled from the container and concurrent runs are safe. This chapter is how to turn it on, and how to move an existing workspace over.

The default is unchanged — omit the state: block and everything works exactly as before. Nothing here is required unless you want remote state.

Turning it on

Turn it on from the CLI — you never hand-edit config.yaml:

roksbnkctl -w acme state s3 \
  --endpoint https://s3.us-south.cloud-object-storage.appdomain.cloud \
  --bucket   acme-bnk-tfstate \
  --region   us-south \
  --key-prefix roksbnkctl

--bucket you create ahead of time; --key-prefix is optional (defaults to the workspace name). Two more optional flags — --access-key-source / --secret-key-source — name the env vars the HMAC keys come from (see Credentials below). Switch back, or inspect, with:

roksbnkctl -w acme state local    # back to per-phase local tfstate
roksbnkctl -w acme state show     # backend + s3 endpoint/bucket/region/key sources

That writes the state: block into the workspace config.yaml for you (the same marshaller every other command uses — no hand-edit):

state:
  backend: s3
  s3:
    endpoint: "https://s3.us-south.cloud-object-storage.appdomain.cloud"
    bucket:   "acme-bnk-tfstate"       # you create this ahead of time
    region:   "us-south"
    key_prefix: "roksbnkctl"           # optional; defaults to the workspace name
    # access_key_source / secret_key_source: env var names (see Credentials)

Three things are your responsibility to provision first; roksbnkctl consumes them and errors clearly if they’re missing:

  1. The bucket — create it in COS (it is your state store; roksbnkctl never creates or deletes it).
  2. HMAC credentials — COS’s S3 API authenticates with HMAC access/secret keys (a Service Credential with “Include HMAC Credential” enabled), not the IAM API key roksbnkctl uses elsewhere.
  3. terraform ≥ 1.10 — required for the native S3 lockfile (below). Selecting s3 on an older terraform fails a preflight check with a clear message; the runner image bundles a new-enough terraform.

Credentials

The HMAC keys are never written into config.yaml, the rendered HCL, or the state object. roksbnkctl reads them from the environment and passes them to terraform as AWS_* env at runtime:

export ROKSBNKCTL_COS_HMAC_ACCESS_KEY="…"
export ROKSBNKCTL_COS_HMAC_SECRET_KEY="…"

access_key_source / secret_key_source name the env vars to read; left unset they default to the two above, falling back to the standard AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY. In CI you just export two secrets.

How state is laid out

All four phases share the one bucket, each at its own key:

<key_prefix>/<workspace>/cluster/terraform.tfstate
<key_prefix>/<workspace>/bnk/terraform.tfstate
<key_prefix>/<workspace>/testing/terraform.tfstate
<key_prefix>/<workspace>/gateway/terraform.tfstate

So the cluster / BNK / testing / gateway states never collide, and the parallel up (BNK ∥ Testing) operates on distinct keys.

Locking — and the single-writer guarantee

COS has no DynamoDB equivalent, so the lock is terraform’s native S3 lockfile (use_lockfile, added in terraform 1.10 — hence the version floor): a small <key>.tflock object alongside each state object. It guarantees one writer per phase key — a second apply against the same phase blocks/fails on the lock rather than corrupting state.

What it does not do: serialize different phases or different workspaces — those are different keys with different locks, by design. So a parallel up is safe, and two operators on two different workspaces don’t contend.

Migrating an existing workspace

A new workspace with state: backend: s3 just works — the first up creates state straight in the bucket. To move an existing local-state workspace over, run roksbnkctl state s3 … (above), export the HMAC keys, then:

roksbnkctl state migrate

It copies each deployed phase’s local state into the bucket (via terraform init -migrate-state), one phase at a time:

→ migrating cluster → s3://acme-bnk-tfstate/roksbnkctl/prod/cluster/terraform.tfstate
✓ cluster migrated
→ migrating bnk → s3://acme-bnk-tfstate/roksbnkctl/prod/bnk/terraform.tfstate
✓ bnk migrated
· testing: remote key … already holds state — skipping (use --force to overwrite)

state migrate: 2 migrated, 1 skipped, 0 failed
Local state files left in place — verify the remote read-back, then remove them when satisfied.

Two safety properties:

  • It won’t clobber. Before copying, it HEADs the target key; if the key already holds state, that phase is skipped (pass --force to overwrite deliberately).
  • It leaves your local files in place. Migration is a copy, not a move — verify a roksbnkctl status / plan reads cleanly from the bucket, then delete the local terraform.tfstate files when you’re satisfied.

Notes and limits

  • Default is local. An absent state: block (or backend: local) renders byte-identical to pre-remote-state behavior — no migration, no surprises.
  • COS is the supported target. Other S3-compatible stores (AWS S3, MinIO) will likely work via the same block, but COS is what’s tested.
  • roksbnkctl doesn’t provision the bucket or keys. That’s deliberate — the state store is yours to own and lifecycle.

See also

Terraform variables (terraform.tfvars)

Variable specs live in Chapter 29 — Terraform variable reference. That chapter is generated from terraform/variables.tf and checked by a test.

roksbnkctl is a thin orchestration layer over a Terraform HCL bundle. The HCL has its own variables — well over 60 of them — declared in terraform/variables.tf. The workspace’s config.yaml covers the common knobs; for the rest, you reach into terraform.tfvars directly.

This chapter is the surface for that lower layer: where the example file lives, how roksbnkctl tfvars bootstraps a starter, what --var-file does, the layering rule between config.yaml-derived tfvars and your overrides, and the one variable that never goes on disk (ibmcloud_api_key).

Where the bundled HCL lives

The Terraform HCL is bundled into the roksbnkctl binary via go:embed. On first use of a workspace, it gets extracted to:

~/.roksbnkctl/<workspace>/state/tf-source/embedded-terraform/
├── main.tf
├── variables.tf
├── outputs.tf
├── providers.tf
├── versions.tf
├── terraform.tfvars.example
└── modules/

The extracted tree is the base plus a per-release overlay

Extraction writes the base tree above, then layers terraform/lines/<line>/ on top of it, where <line> is the BNK release derived from bnk.manifest_version (2.3.0-3.2598.3-0.0.170 → 2.3). Same relative path replaces, new path is added, nothing is removed — so an overlay can only add to or replace the base, never quietly delete a resource it declares.

No supported release needs one today, so lines/ ships empty and the extracted tree is byte-identical to the base. It is documented here anyway, because the consequence is not obvious and will not announce itself: the extracted tree is no longer a function of the binary alone, but of the binary and the workspace’s manifest version. If you are ever comparing your extracted main.tf against the repo’s and they differ, that is the first thing to check.

Why this rather than a branch per BNK release — which was tried and withdrawn — is in terraform/lines/README.md. The short version: a branch forks the whole tool to express a difference that lives in a handful of .tf files.

That terraform.tfvars.example file is the canonical reference for what’s tunable — every variable with a sensible starter value, grouped by module (ROKS cluster, cert-manager, FLO, CNEInstance, License, testing). terraform/variables.tf (linked at the GitHub canonical URL) is the formal declaration with types, descriptions, and defaults.

You don’t edit the example file in place. Copy or generate from it instead.

roksbnkctl tfvars — bootstrap a starter

The roksbnkctl tfvars subcommand prints a starter terraform.tfvars to stdout, populated from the current workspace state:

$ roksbnkctl tfvars > ~/.roksbnkctl/dev/terraform.tfvars.user

What gets pre-filled:

  • Every field from config.yaml that maps to a tfvar (cluster name, region, workers, BNK fields, COS fields)
  • cluster_network_mode — rendered unconditionally, unlike the settings around it. Most are omitted when unset so the HCL default stands; this one decides how a cluster is built, and a value that means two different things depending on which side you read it from is worth avoiding. Unset renders "single-nic", which is today’s behaviour exactly.
  • The cluster’s identity from cluster-outputs.json if cluster up has already run
  • A commented-out section for the variables you might want to tune next (jumphost profile, GSLB datacenter, license mode)

What’s deliberately excluded:

  • ibmcloud_api_key — never on disk (see “The IBMCLOUD_API_KEY exception” below)
  • Sensitive outputs (BIG-IP passwords, COS HMAC secrets) — left as upstream defaults

The starter is meant to be copied into ~/.roksbnkctl/<ws>/terraform.tfvars.user (the workspace-local override file) or into a --var-file path you keep alongside the workspace.

What you typically edit

The variables that matter for day-to-day BNK trial work, ordered by likely-to-touch:

VariableDefaultWhat it controls
openshift_cluster_nametf-openshift-clusterCluster name. Mirrors config.yaml’s cluster.name.
roks_workers_per_zone1Worker nodes per AZ. 2 ⇒ 6 workers in a 3-AZ MZR region.
create_roks_clustertrueSet false to adopt an existing cluster. Pair with roks_cluster_id_or_name.
cluster_public_gatewaytruefalse = private/disconnected cluster (no public gateways, no worker egress). config.yaml’s cluster.public_gateway. Expert — needs private connectivity you provide.
openshift_cluster_version"4.20"OpenShift minor. Quote it — YAML/HCL parses 4.20 as float otherwise.
cneinstance_deployment_sizeSmallSmall/Medium/Large. CNEInstance sizing.
f5_bigip_k8s_manifest_versionupstream pinPin a specific BNK manifest chart version.
far_repo_urlrepo.f5.comFAR Docker/Helm registry. Override for a non-production FAR (an EA repo) or a staging mirror. Reaches every consumer including the standalone FLP VSI, whose chart pulls used to spell the host as a literal and silently ignored this.
flo_namespacef5-bnkWhere the F5 Lifecycle Operator runs.
testing_create_tgw_jumphosttrueCreate the testing jumphost in a client VPC over Transit Gateway.
testing_ssh_key_name"" (must set)Existing IBM Cloud SSH key name for jumphost provisioning.
flo_trusted_profile_sa_name / flo_trusted_profile_rolesthe CNE controller’s IBM Cloud identity — modelled as bnk.trusted_profile.*
cneinstance_vlan_prefixlen_external / _internalper-VLAN self-IP masks — modelled as bnk.network.vlan_prefixlen_{external,internal}
cneinstance_gslb_datacenter_name""Set when wiring BNK into an F5 BIG-IP GSLB datacenter.
cneinstance_network_zones[] (install-guide defaults)Per-zone VLAN/SNAT/VIP subnets + TMM self-IPs. Mirrors config.yaml’s bnk.network.zones. Supply all three zones or none.
cneinstance_vlan_prefixlen24TMM self-IP prefix length (F5SPKVlan spec.prefixlen_v4). bnk.network.vlan_prefixlen. Match your VLAN CIDRs.
cneinstance_tmm_k8s_routes172.17.0.0/18Pod CIDR TMM routes to (TMM_K8S_ROUTES). bnk.network.tmm_k8s_routes. Set to your cluster’s pod subnet if non-default.
license_modeconnectedconnected | disconnected.

For the full list with types and per-field descriptions, see terraform/variables.tf directly — link here — or the auto-generated Chapter 29 — Terraform variable reference.

Resource naming & collision avoidance

Every account-scoped IBM Cloud resource name is derived from a single workspace prefix. Without per-workspace naming, two workspaces creating infrastructure in the same IBM Cloud account would collide at the account level — the second up would hit Provided Name … is not unique / gateway with the same name already exists. The prefix avoids this: roksbnkctl derives the full name set from it, validates each name against its resource type’s length/charset limit at init time, and renders a complete terraform.tfvars that names every resource explicitly.

Prefix → name derivation

Give a workspace the prefix acme-eu, and roksbnkctl generates this name set:

Resourcetfvars variableDerived name (prefix = acme-eu)Suffix
ROKS/OpenShift clusteropenshift_cluster_nameacme-eu(none — cluster name is the prefix)
Cluster VPCroks_cluster_vpc_nameacme-eu-cluster-vpc-cluster-vpc
Registry COS instanceroks_cos_instance_nameacme-eu-registry-cos-registry-cos
Transit Gatewayroks_transit_gateway_nameacme-eu-tgw-tgw
Client VPC (TGW jumphost)testing_client_vpc_nameacme-eu-client-vpc-client-vpc
TGW jumphosttesting_tgw_jumphost_nameacme-eu-jh-tgw-jh-tgw
Per-zone cluster jumphoststesting_cluster_jumphost_name_prefixacme-eu-jh (module appends -<zone>, e.g. acme-eu-jh-us-south-1)-jh

The names are deterministic from the prefix, so roksbnkctl re-derives them on every up / plan / apply — regeneration is always safe, and the rendered terraform.tfvars is a faithful record of exactly what the tool asked IBM Cloud to create.

Why the cluster name takes no suffix

The cluster name deliberately equals the prefix with no suffix appended. This is load-bearing, not an oversight:

  • The ROKS/OpenShift cluster name has the tightest limit of any resource here — 35 characters (see the table below). Every other name is an IS resource capped at 63.
  • By making the cluster name be the prefix, the prefix’s own length limit becomes that same 35-character cluster limit. A prefix that’s short enough to be a valid cluster name is automatically short enough that every suffixed name fits inside 63 — the worst case is the zone-appended cluster jumphost (<prefix>-jh-us-south-1), which at a 35-char prefix reaches only 49 of its 63-char budget.

Do not “tidy” the cluster name into <prefix>-cluster. Adding a suffix to the cluster name would shrink the usable prefix length (the binding 35-char limit would now have to absorb -cluster too) without buying anything. The no-suffix cluster name is what lets a single prefix-length check guarantee the whole name set is valid.

The length / charset limits

roksbnkctl validates the prefix — and every name it derives — against these per-resource-type constraints. They are pinned from IBM Cloud’s own validators so the tool rejects a bad prefix at init time rather than letting IBM Cloud reject it minutes into an apply:

Resource kindMax lengthCharset ruleSource
ROKS / OpenShift cluster name35Start with a letter; letters, digits, and hyphen -; ≤ 35 charsIBM Cloud provider docs, ibm_container_cluster / ibm_container_vpc_cluster name argument (“must start with a letter, can contain letters, numbers, and hyphen (-), and must be 35 characters or fewer”).
IS resource — VPC, VSI, Transit Gateway, SSH key63Start with a lowercase letter; lowercase letters, digits, and hyphen -; must end alphanumeric; ≤ 63 chars (regex ^[a-z][-a-z0-9]*$ + ends [a-z0-9])terraform-provider-ibm ValidateISName (ibm/validate/validators.go).
COS / Resource Controller instance180Permissive server-side; roksbnkctl reuses the lowercase-alphanumeric IS subset since the name derives from an already-validated lowercase prefixIBM Cloud Resource Controller create-instance name field (“must be 180 characters or less”).

Because the prefix flows into all three, roksbnkctl enforces the strictest applicable rule on the prefix label itself (lowercase, start with a letter, [a-z0-9-], no trailing hyphen) and the tightest length (35, from the cluster). An over-long prefix is rejected with an actionable message naming the offending resource, its computed length, its limit, and the maximum allowable prefix length — and init re-prompts (TTY) or hard-errors (non-TTY). There is no silent truncation or hashing.

Namespaces are NOT prefixed

Kubernetes namespaces — cert_manager_namespace (cert-manager), flo_namespace (f5-bnk), flo_utils_namespace (f5-utils) — default to their conventional values (and are settable via bnk.flo_namespace / bnk.flo_utils_namespace — including to the same value, for one namespace instead of two) and are never prefixed. They are cluster-internal: two workspaces only ever collide on namespaces if they target the same cluster, in which case a shared namespace is usually what you want (and a prefix would actively break the convention FLO and the BNK charts expect). Only account-scoped IBM Cloud infrastructure names — cluster, VPCs, TGW, COS, jumphosts — are prefixed, because those are what collide at the account level.

Overriding a generated name

Every generated name is just a value in the rendered terraform.tfvars, so the normal layering rule overrides it. To pin a specific name (e.g. to adopt a VPC that doesn’t follow the prefix convention), set the matching variable in ~/.roksbnkctl/<ws>/terraform.tfvars.user or pass it via --var-file:

# Override just the Transit Gateway name for this workspace, permanently
echo 'roks_transit_gateway_name = "shared-corp-tgw"' \
  >> ~/.roksbnkctl/<ws>/terraform.tfvars.user

The .user file (or --var-file) layers after the generated terraform.tfvars, so the override wins. This is the same override path documented in §“The layering rule” — the prefix machinery doesn’t change it. Declining a resource at init time and supplying an existing one is the cleaner path for adoption; see Chapter 12 §“Worked example”.

The layering rule

When roksbnkctl up (or plan/apply/destroy) invokes Terraform, it composes three layers of tfvars in this order:

1. terraform.tfvars              (rendered by roksbnkctl from config.yaml)
2. terraform.tfvars.user         (workspace-local override, optional)
3. --var-file <path> ...         (CLI flag, repeatable, later file wins)

Later layers override earlier ones — same rule Terraform itself uses for -var-file chaining.

Concretely:

# config.yaml says cluster.workers_per_zone: 2
# ~/.roksbnkctl/dev/terraform.tfvars.user contains:
#   roks_workers_per_zone = 4
# Run with no flag:
roksbnkctl up
# → terraform sees 4 (.user wins over generated .tfvars)

# Pass a CLI override:
roksbnkctl up --var-file ./perf-test.tfvars
# perf-test.tfvars contains: roks_workers_per_zone = 8
# → terraform sees 8 (.var-file wins over .user)

# Multiple --var-files; later wins:
roksbnkctl up \
  --var-file ./base.tfvars \
  --var-file ./override.tfvars
# → values in override.tfvars win over base.tfvars,
#   which both win over .user, which wins over .tfvars

The --var-file flag matches Terraform’s own --var-file exactly — repeatable, paths interpreted relative to the working directory at invocation time.

The IBMCLOUD_API_KEY exception

The upstream HCL declares ibmcloud_api_key as a sensitive variable. Every other tfvar can land in a file on disk; this one never does.

Instead, the API key flows through the resolver chain (env → keychain → config-b64 → prompt — see Chapter 14), and roksbnkctl exports it as TF_VAR_ibmcloud_api_key in the environment of the terraform-exec child process. Terraform reads the env var and injects it as if it had been declared in tfvars, but no plaintext key ever touches the filesystem.

If you put ibmcloud_api_key = "..." in a hand-edited tfvars and run terraform directly (not via roksbnkctl), it works — Terraform itself is happy. But this is not how roksbnkctl runs Terraform, and putting the key in a .tfvars.user or --var-file is strongly discouraged: the file persists on disk, gets backed up, gets committed to git by accident, and gets read by other processes. The env-var path eliminates the on-disk window entirely.

Other secrets in scope:

  • bigip_password — upstream HCL declares it as a regular string (not sensitive). If you set it in tfvars, the value lands on disk. Treat that file like a credential.
  • COS HMAC keys — auto-generated by the roks_cluster module via the COS service-credentials resource; they live in terraform.tfstate (which is itself sensitive — chmod 0600, never commit, treat the workspace as a secret store).

Worked example: bigger cluster for a perf test

Default workspace, default cluster. You want to bump worker count for one perf-test run, then go back.

# 1. Confirm the current value comes from config.yaml
$ grep workers ~/.roksbnkctl/dev/config.yaml
  workers_per_zone: 2

# 2. Drop a one-off override into a file
$ cat > ~/perf-cluster.tfvars <<'EOF'
roks_workers_per_zone = 6
roks_min_worker_vcpu_count = 32
roks_min_worker_memory_gb = 128
EOF

# 3. Plan against it (note: --var-file passes through to terraform plan)
$ roksbnkctl plan --var-file ~/perf-cluster.tfvars

# 4. Apply
$ roksbnkctl apply --var-file ~/perf-cluster.tfvars

# 5. Run the throughput test
$ roksbnkctl test throughput

# 6. Roll back: re-apply WITHOUT the var-file
$ roksbnkctl apply
# → terraform sees workers_per_zone=2 again from config.yaml-derived tfvars

Notice step 6 — dropping the --var-file flag is the rollback. Terraform compares its current state to the new desired state (from config.yaml) and scales the worker pool back down. No special “undo” command needed.

For a more permanent override (you want this workspace to always run with bigger nodes), put the contents of perf-cluster.tfvars into ~/.roksbnkctl/dev/terraform.tfvars.user instead. Then every roksbnkctl up/apply picks it up automatically without a CLI flag.

When to edit config.yaml vs .tfvars.user vs --var-file

A rough decision matrix:

You want to change…Edit…
Cluster identity, region, OpenShift version, worker countconfig.yaml (via roksbnkctl init or by hand)
BNK chart version, CNEInstance size, FAR repoconfig.yaml (the bnk: block)
A variable not modelled in config.yamlterraform.tfvars.user (workspace-local, persistent)
You have a complete ./terraform.tfvars you want this workspace to always useCopy it to ~/.roksbnkctl/<ws>/terraform.tfvars.user (sibling to config.yaml, mode 0600) so bare -w <ws> commands Just Work for both phases. See Chapter 6 §“Raw terraform-variable overrides”.
A one-off override for a single run (perf test, capacity bump)--var-file ./oneoff.tfvars (CLI)
A CI-pipeline variable bundle that’s checked into git--var-file ./ci-overrides.tfvars (CLI; the file lives in your CI repo, not the workspace)

The schema in config.yaml covers about a third of the upstream HCL variables — the ones that nearly every workspace needs to set. The other two-thirds (jumphost details, every BNK module’s full surface, the testing module’s full surface) are reachable through the lower layers.

Cross-references

Credentials and the resolver chain

Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the Workspace struct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.

roksbnkctl handles four kinds of secrets: an IBM Cloud API key, a kubeconfig, an SSH private key, and the Terraform state file. Each has a different threat model, a different lookup chain, and a different rule for “what’s safe to commit to a workspace”.

This chapter is the user-facing distillation of PRD 04 — credential propagation. PRD 04 is the design surface for developers extending the credential system; this chapter is the operational surface for users who need to know “where does my key live, and how does the tool find it”.

The four secrets in scope

CredentialUsed byResolved from
IBMCLOUD_API_KEYibmcloud CLI, terraform’s IBM provider, IBM SDK callsEnv → OS keychain → workspace api_key_b64 → prompt
kubeconfigkubectl/oc passthroughs, roksbnkctl k get/apply/..., terraform’s k8s + helm providersKUBECONFIG env → ~/.kube/config (kubectl-style)
SSH private keyThe SSH client backing --on and the ssh:<target> execution backendPer-target: file path, ssh-agent, or tf-output:<name>
Terraform stateThe terraform-exec calls inside roksbnkctl up/apply/destroyWorkspace state/terraform.tfstate (filesystem only)

Each has its own discovery rules. Walk them in turn.

The IBMCLOUD_API_KEY resolver chain

The single most-used credential. Resolved by internal/cred/resolver.go. The resolver walks four sources in order:

1. Environment variables (process-scoped, never persisted)
2. OS keychain (per-user, system-managed)
3. Workspace config api_key_b64 (per-workspace, base64 obfuscation)
4. Interactive prompt (TTY-only)

The first source that yields a non-empty value wins. The chain stops there — the key isn’t re-fetched from a “more authoritative” source on subsequent calls.

Source 1 — Environment

The resolver checks these env vars in order, returning the first non-empty value:

IBMCLOUD_API_KEY            # canonical
IC_API_KEY                  # short alias
TF_VAR_ibmcloud_api_key     # terraform passthrough form
TF_VAR_IBMCLOUD_API_KEY     # uppercase variant some pipelines use
TF_VAR_IC_API_KEY           # uppercase variant of the short alias

Env vars are first because they’re the most explicit path — if you’ve gone to the trouble of setting one, you’ve made a deliberate choice. Pre-existing CI pipelines, automation scripts, and direnv setups all live here. The resolver respects that ordering even when a keychain entry also exists.

Source 2 — OS keychain

roksbnkctl stores per-workspace API keys in the OS-native keychain via github.com/zalando/go-keyring:

OSBackend
macOSKeychain (security framework)
Linux (with libsecret)GNOME Keyring / KWallet via Secret Service API
WindowsCredential Manager
Linux (no libsecret)Falls back to source 3 (config base64)

Entries are namespaced under service roksbnkctl, with user <workspace>/ibmcloud_api_key:

# What `roksbnkctl init` writes (no-op shown for clarity)
$ keyring set roksbnkctl dev/ibmcloud_api_key

This is the recommended secure default. The OS handles process isolation; roksbnkctl only sees the value during the brief window between fetch and use.

Source 3 — Workspace api_key_b64

A base64-encoded blob in ~/.roksbnkctl/<workspace>/config.yaml:

ibmcloud:
  api_key_b64: ZW5jb2RlZC1hcGkta2V5LXZhbHVl

Important framing: base64 is obfuscation, not encryption. Anyone with read access to the file can decode it instantly:

echo -n "ZW5jb2RlZC1hcGkta2V5LXZhbHVl" | base64 -d
# → encoded-api-key-value

The encoding exists for two reasons:

  1. Visual. A glancing cat config.yaml doesn’t surface the literal API key.
  2. Format. API keys can contain = and other YAML-special characters that complicate inline storage. Base64 normalises them.

api_key_b64 is the fallback when the OS keychain isn’t available — most commonly WSL2 without libsecret, headless Linux servers, and CI runners where bringing up a keychain daemon is more friction than it’s worth. Treat the file like a plaintext credential: never commit it, never share it.

File-mode note: config.yaml is written mode 0600 inside a 0700 workspace directory, and roksbnkctl tightens both on read for workspaces written by an earlier release — you no longer need to chmod it yourself or re-chmod after init. Verify with roksbnkctl doctor (the workspace permissions check).

Releases before v1.50.0 wrote 0644 inside 0755, so on any host you share with another account, rotate an API key that has been sitting in config.yaml. Tightening the mode now does not un-expose what was readable before. The keychain and env-var paths avoid the question entirely: nothing sensitive lands in config.yaml at all.

The plaintext field name api_key: is rejected at workspace-load time:

$ roksbnkctl up
error: ~/.roksbnkctl/dev/config.yaml: plaintext secret detected (offset 47):
       workspace config.yaml must not contain credentials — use IBMCLOUD_API_KEY
       env var or the OS keychain (see `roksbnkctl init`)

The regex catches api_key, apikey, ibmcloud_api_key, password, token, secret_access_key, hmac_secret. The _b64 suffix is the documented escape — it’s the only inline form the loader tolerates.

Source 4 — Interactive prompt

When sources 1-3 all come up empty AND stdin is a TTY, the resolver prompts:

Enter IBM Cloud API key for workspace "dev": ********
Save the key for future runs? [Y/n]: y
  ✓ saved to OS keychain

The key is read with echo disabled (via golang.org/x/term). The prompt offers to persist — by default it tries the OS keychain first, falls back to api_key_b64 in config.yaml if the keychain is unavailable.

If stdin isn’t a TTY (CI runner, piped input, daemon process), the resolver errors instead of hanging:

error: no IBM Cloud API key available and stdin is not a TTY (cannot prompt;
       set IBMCLOUD_API_KEY or run `roksbnkctl init`)

Pinning a single source

The chain is the default. To force one specific source, set ibmcloud.api_key_source in config.yaml:

ibmcloud:
  api_key_source: keychain    # env | keychain | config | prompt

This is useful in two scenarios:

  • CI: api_key_source: env makes a missing env var a hard error rather than falling through to a (locked / non-existent) keychain.
  • Auditable single-source-of-truth: pinning to keychain documents that this workspace’s key lives in the OS keychain and nowhere else; reading the key from a different source becomes an error rather than a silent fallback.

kubeconfig discovery

Different chain, different rules. roksbnkctl discovers the kubeconfig the same way kubectl does — two sources, in this order:

1. KUBECONFIG environment variable (first existing path in a colon-separated list)
2. ~/.kube/config

This is the kubectl-standard discovery chain, implemented in internal/k8s/client.go::DefaultKubeconfigPath(). Whatever you’ve already taught kubectl to read, roksbnkctl reads too.

cluster up’s post-apply step writes the admin kubeconfig to ~/.kube/config (mode 0600) by default — so the second source in the chain is also the destination of the tool’s own output, and the same KUBECONFIG-overrides-everything rule applies. If KUBECONFIG is set when cluster up runs, the download lands at that path instead.

Note: there is also a ~/.roksbnkctl/<workspace>/state/kubeconfig/ directory under the workspace state dir. It’s a Terraform input (kubeconfig_dir tfvar) that the upstream HCL writes per-component sub-files into (cert_manager, cne_instance, flo, license); it is not a kubeconfig file the resolver reads. Don’t confuse the two.

When the file is missing

If neither source yields a kubeconfig, commands that need one error with:

error: no kubeconfig: KUBECONFIG env not set, ~/.kube/config not present.
       Run `roksbnkctl cluster up`, `roksbnkctl cluster register <name>`,
       or set KUBECONFIG.

The remediation message tells you which path to take. cluster register <name> is the path for an existing cluster you want to adopt without re-creating it (see Chapter 9).

File permissions

cluster up writes ~/.kube/config chmod 0600 (owner read/write only). It contains the cluster admin token; treat it like a credential. Don’t commit it, don’t email it, don’t cat it in screen-shared sessions.

SSH private keys

Per-target, not per-workspace. Each entry under targets: in config.yaml declares exactly one of:

SourceFormNotes
Filekey_path: ~/.ssh/id_ed25519Standard OpenSSH key formats. Tilde expansion honoured.
Agentkey_source: agentTalks to ssh-agent over $SSH_AUTH_SOCK. Linux/macOS only at v1.0; Windows ssh-agent named-pipe support is on the v1.x roadmap.
TF outputkey_source: tf-output:jumphost_shared_keyReads from terraform state at connect time; never written to disk separately.

The tf-output: form is the auto-discovered jumphost path — the upstream HCL provisions a tls_private_key resource per cluster create, marks it sensitive, and surfaces it as a terraform output. roksbnkctl reads the output via terraform output -raw at SSH-connect time, never persists it, and the key only exists in TF state plus in memory during a connect.

Chapter 15 — SSH targets is the deep reference for the targets: block; this chapter just notes the credential-side framing.

Terraform state

~/.roksbnkctl/<workspace>/state/terraform.tfstate is the workspace’s terraform state file. It contains:

  • IBM Cloud admin tokens (cluster admin, COS HMAC credentials)
  • Generated TLS private keys (the jumphost shared key referenced above)
  • Sensitive outputs (FAR auth bundles, license JWTs)
  • Every resource attribute terraform tracks

It is plaintext-credential-equivalent. The file mode is 0600; the parent directory is 0700. Backup the workspace dir intact, never commit it to git, treat compromise of the state file as compromise of every secret it contains.

There is no separate “TF state credential” — the file’s filesystem ACL is the only access control. PRD 04 covers the cross-backend story for moving state into a Docker bind-mount, a Kubernetes Secret, or an SCP’d remote temp directory; at v1.0 the local file is the only path (terraform --backend k8s / ssh are deferred to v1.x; see docs/PLAN.md §“What’s deliberately deferred to post-v1.0”).

What’s safe to commit vs not

A short rule:

SAFE TO COMMIT:    nothing in ~/.roksbnkctl/<workspace>/
NOT SAFE:          everything in ~/.roksbnkctl/<workspace>/

The longer version, by file:

PathCommit?Why
~/.roksbnkctl/<ws>/config.yamlNoEven without api_key_b64, this file documents your cluster identity, region, COS bucket — useful inventory for an attacker.
~/.roksbnkctl/<ws>/config.yaml (with api_key_b64)Hard noThe base64 value is plaintext-equivalent. Committing it = leaking the key.
~/.roksbnkctl/<ws>/state/kubeconfigNoCluster admin token.
~/.roksbnkctl/<ws>/state/terraform.tfstateNoEvery secret terraform manages, in plaintext.
~/.roksbnkctl/<ws>/state/terraform.tfvarsNoGenerated; references no secrets directly but documents resource layout.
~/.roksbnkctl/<ws>/terraform.tfvars.userMaybeIf you’ve kept secrets out (no bigip_password, no ibmcloud_api_key), it’s just config. Audit before committing.
~/.roksbnkctl/<ws>/cluster-outputs.jsonNoCluster identity + COS instance name. Not directly a secret but tied to the workspace.
~/.roksbnkctl/known_hostsYes (if you want)Host-key fingerprints; not a secret. Same threat model as OpenSSH’s ~/.ssh/known_hosts.

The simplest policy: a .gitignore that excludes the entire ~/.roksbnkctl/ tree. If you really want to share a workspace skeleton with a colleague, send the config.yaml minus api_key_b64 and let them re-run roksbnkctl init against their own account.

How roksbnkctl init writes the API key

Walk through the writeable side of the resolver:

$ roksbnkctl init
Workspace name [default]: dev
IBM Cloud region [ca-tor]:
Enter IBM Cloud API key (input hidden): ********
Save the key for future runs? [Y/n]: y
  ✓ saved to OS keychain

What just happened:

  1. init prompted for the key. Input echo was off; the key never appeared on screen.
  2. The user said “save”.
  3. SaveAPIKeyForWorkspace tried SaveAPIKeyToKeychain first.
  4. The OS keychain accepted the entry (Linux + libsecret in this case). The success path returned "OS keychain" and init printed the confirmation.
  5. The key was not written to terraform.tfvars (that’s the resolver’s job at terraform-invoke time, via the TF_VAR_ibmcloud_api_key env var).

If step 4 had failed (no keychain, WSL2 without libsecret), SaveAPIKeyForWorkspace would have fallen through to saveAPIKeyToConfig — base64-encoded the key, written it into config.yaml’s api_key_b64 field, returned "config.yaml (base64)". init would have printed:

  ✓ saved to config.yaml (base64)
  warning: base64 is obfuscation, not encryption — chmod 0600 the file

Both destinations work. The keychain path is the recommended default; the config-b64 path is the documented fallback.

The cred-tmpfile and trusted-profile paths

roksbnkctl --backend docker does not leak IBMCLOUD_API_KEY in docker inspect, and roksbnkctl --backend k8s ops install auto-provisions an IBM Cloud trusted profile so the ops pod never sees a static API key (see PRD 04 §“Open questions”). Both have fallbacks for environments where the primary path doesn’t apply.

The tmpfile-bind-mount pattern (docker backend)

The docker backend writes the resolved IBMCLOUD_API_KEY to a 0600 tempfile on the host, bind-mounts that single file read-only at /run/secrets/ibmcloud_api_key inside the container, and points the container at the file via IBMCLOUD_API_KEY_FILE=/run/secrets/ibmcloud_api_key. The value never appears in the container’s stored env metadata — docker inspect <id> shows the path, not the key. The tempfile is owned by the calling user and is removed on backend exit (and on context cancellation, so an interrupted run still cleans up).

You don’t have to do anything to opt in — the pattern is the default for --backend docker. The engineering shape (lifecycle, the inline sh -c shim that re-exports the value into the legacy IBMCLOUD_API_KEY env name for tools that read from env, the why-not-just-use---secret discussion) lives in PRD 04 → “Cred tmpfile-bind-mount pattern (docker backend)”. For most users the takeaway is one line: --backend docker is docker inspect-clean.

The --trusted-profile flag (k8s backend)

New flag on roksbnkctl ops install that controls how the ops pod gets its IBM Cloud credential. Three values:

ValueWhat it doesWhen to use
auto (default)Try to provision an IBM Cloud trusted profile (roksbnkctl-ops-<workspace>) linked to the ops pod’s ServiceAccount. The pod assumes the profile via its projected SA token and the static API key never lands in any Secret. If your workspace API key doesn’t have IAM iam-identity permissions, automatically fall back to the static-key Secret with a stderr warning that names the missing perm and how to silence it (--trusted-profile=off).Default for new installs. Production users get the secure path automatically; restricted-IAM users still complete ops install successfully.
onTry to provision; fail loudly with a non-zero exit if perms don’t allow. No fallback.CI / hardened environments where the static-key path is unacceptable and a perm-missing case should block, not warn.
offSkip the trusted-profile path entirely; provision the static-key Secret.Compatibility / debugging — and the documented path for clusters whose IAM admin doesn’t grant iam-identity perms and isn’t expected to.

Chapter 19 — The in-cluster ops pod walks through the --trusted-profile=auto install flow, the verification commands (oc get serviceaccount roksbnkctl-ops -o yaml showing the trusted-profile annotation), the fallback warning shape, and how ops uninstall cleans up a provisioned profile.

Backend-specific cred propagation

The credential-propagation rules differ per backend. All four backends ship at v1.0:

BackendWhere creds liveMechanism
localThe user’s environmentos/exec inherits parent env
dockerCaller’s env, propagated by referencedocker run --env IBMCLOUD_API_KEY (no =value) — value inherits, never appears in docker inspect
k8sKubernetes Secret in the roksbnkctl-ops namespaceMounted into the ops pod via envFrom: secretRef; or IAM trusted profile (preferred)
sshRemote env or wrapper scriptssh -o SetEnv=IBMCLOUD_API_KEY=... first; falls back to a 0700 wrapper script with trap rm EXIT

Each backend’s “where creds live” surface is summarised in Chapter 17 — Execution backends; the design rationale is in PRD 04.

The user-facing invariant across all four: you put the key into one of the resolver chain’s sources, and roksbnkctl figures out the rest. You don’t have to learn four different credential APIs to use four different backends.

The redactor

roksbnkctl writes a fair amount to its own logs (stdout, stderr) — terraform plan output, ibmcloud CLI output, error traces. Anywhere we can plausibly print the IBM API key (because a downstream tool printed it, because an error message included it, because a debug trace dumped a struct), the redactor masks it before the bytes leave the binary.

What gets redacted:

  • The IBM Cloud API key value, anywhere it appears in Stdout or Stderr of an exec backend’s RunOpts. Replaced with [REDACTED].
  • The same value in roksbnkctl’s own log output (the lifecycle commands that wrap terraform-exec).

What does not get redacted:

  • Output captured by callers via -o yaml/-o json for resources that legitimately contain the key (e.g., a Secret returned from roksbnkctl k get). The redactor doesn’t know about Kubernetes resource semantics; if you k get secret -o yaml, you’ll see the key. (The same is true of kubectl.)
  • Output from a tool you ran outside roksbnkctl (e.g., piping to tee after invoking terraform directly). The redactor only sees bytes that pass through the exec backend’s Stdout/Stderr writers.
  • The terraform state file. State is on-disk; the redactor is an in-memory stream filter.

The implementation is internal/exec/redact.go — a wrapping io.Writer with byte-comparison redaction and cross-write prefix buffering (so a secret split across two Write calls still gets masked). The matcher uses the resolved API key value verbatim (a known string at run-time) rather than a generic “looks like an IBM API key” pattern, to avoid false positives on legitimate output.

PRD 04’s acceptance criteria require that the API key never appears in docker inspect, ps -ef, kubectl get pods/events -o yaml, or kubectl describe pod. The redactor is the defence-in-depth layer; the per-backend cred-propagation rules are the primary control.

Cross-references

SSH targets

Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the Workspace struct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.

This chapter is the technical reference for the targets: system. Its companion is Chapter 16 — The –on flag and SSH jumphosts, which is the user-facing prose for “how do I run a command on the jumphost”. Chapter 16 introduces targets briefly; this chapter goes deeper into the schema, the key sources, the host-key trust model, the auto-discovery pipeline, and what the ssh execution backend layers on top of the lightweight --on flag.

If you arrived here from Chapter 16 looking for “where do I learn the full surface”, you’re in the right place.

The targets: block schema

Targets live under targets: in ~/.roksbnkctl/<workspace>/config.yaml:

targets:
  jumphost:
    host: 169.45.91.177
    port: 22
    user: ubuntu
    key_source: tf-output:jumphost_shared_key

  bastion:
    host: ops.example.com
    user: jgruber
    key_path: ~/.ssh/id_ed25519

  prod-jump:
    host: 10.0.0.5
    user: ec2-user
    key_source: agent

The Go struct backing it is internal/config.TargetCfg:

type TargetCfg struct {
    Host      string `yaml:"host"`
    Port      int    `yaml:"port,omitempty"`        // default 22
    User      string `yaml:"user"`
    KeyPath   string `yaml:"key_path,omitempty"`    // file path
    KeySource string `yaml:"key_source,omitempty"`  // "agent" | "tf-output:<name>"
}
FieldTypeRequiredNotes
hoststringyesIP or hostname. Resolved via the standard Go resolver chain (no special DNS handling).
portintnoDefaults to 22. Only set when the remote sshd listens elsewhere.
userstringyesRemote login username.
key_pathstringone-ofFile path to a PEM-encoded private key. Tilde expansion honoured.
key_sourcestringone-ofagent or tf-output:<output-name>.

Validation rules at load time:

  • Exactly one of key_path or key_source must be set. Setting neither, or both, fails the load with a clear error.
  • The target name (the YAML map key) must be non-empty and stable across YAML round-trips — roksbnkctl targets show <name> and roksbnkctl targets remove <name> look up by this name.

The TargetCfg type lives in internal/config rather than internal/remote to avoid an import cycle: the YAML (de)serialiser needs the wire shape, and the SSH client (internal/remote) needs to consume it. Keeping the shape in config and the runtime Target (parsed key, dialer config, etc.) in remote keeps the dependency direction one-way.

Key sources

The three options for telling roksbnkctl how to find the SSH private key for a target.

key_path: <file>

A PEM-encoded private key on disk:

bastion:
  host: ops.example.com
  user: jgruber
  key_path: ~/.ssh/id_ed25519

Standard OpenSSH formats are accepted: id_rsa, id_ed25519, id_ecdsa, id_dsa (deprecated but supported). Tilde expansion uses os.UserHomeDir() semantics — ~/ → user home, ~user/ is not supported (use an absolute path).

The file is read at SSH-connect time, not at config-load time. A missing or unreadable file fails the connect, not the workspace load. This matters for ergonomics: you can edit a target into config.yaml referencing a key path that doesn’t exist yet, then create the key separately, without roksbnkctl init/use failing in between.

Encrypted keys (passphrase-protected) are not currently supported in the SSH client — the agent path is the recommended workflow for keys that need a passphrase.

key_source: agent

Talks to ssh-agent over $SSH_AUTH_SOCK:

prod-jump:
  host: 10.0.0.5
  user: ec2-user
  key_source: agent

The agent presents whichever keys it currently holds; roksbnkctl tries each in turn against the target’s authorized_keys (via SSH’s standard publickey-authentication exchange). The first key the server accepts is the one that gets used.

This is the right setting when:

  • Your team manages keys via 1Password / hardware tokens / gpg-agent and you don’t want a key file on disk.
  • You’re on a shared workstation where putting the key file in ~/.ssh/ is undesirable.
  • You’re already using ssh-agent for everything else and want consistent behaviour.

Platform note: ssh-agent integration is Linux/macOS-only. Windows users should use key_path to a file. The restriction is structural to the Go SSH library, which doesn’t wrap the Windows ssh-agent named-pipe protocol — see golang.org/x/crypto/ssh/agent and upstream tracking issues for status; full Windows support is a v2 item.

key_source: tf-output:<output-name>

Reads the key from the workspace’s terraform state output of that name:

jumphost:
  host: 169.45.91.177
  user: ubuntu
  key_source: tf-output:jumphost_shared_key

This is the auto-discovered jumphost path. The upstream HCL provisions a tls_private_key resource per cluster create, marks it sensitive, and surfaces it as a terraform output named jumphost_shared_key. roksbnkctl reads it via the equivalent of terraform output -raw <name> at SSH-connect time.

What this gets you that key_path doesn’t:

  • No on-disk key file separate from terraform state. The key only exists in terraform.tfstate (which is already a sensitive workspace artefact) and in memory during the SSH handshake.
  • Auto-rotation on cluster re-create. Destroy and re-create the cluster, terraform generates a new tls_private_key, and the next --on jumphost invocation picks up the new key without any manual rewriting of the workspace config.
  • Single source of truth. The key value is in terraform state — the same place every other cluster-generated secret lives.

The terraform output must be a string-typed PEM-encoded private key. terraform output -raw <name> returns the value regardless of the sensitive flag (the flag just suppresses display; the data is still readable to anyone with state access).

Host-key TOFU and ~/.roksbnkctl/known_hosts

roksbnkctl keeps its own known_hosts file at ~/.roksbnkctl/known_hosts. It does not read or write ~/.ssh/known_hosts. The two files are independent.

Why a separate file

Three reasons:

  1. Isolation. roksbnkctl’s SSH client is a different program from ssh(1); mixing host-key state between the two creates surprising behaviour (deleting a key from ~/.ssh/known_hosts doesn’t clear it from roksbnkctl’s view, or vice versa).
  2. Audit. A roksbnkctl-managed file lets the tool’s behaviour be reasoned about without inspecting the user’s broader SSH state.
  3. Cleanup. roksbnkctl ws delete <name> could theoretically scrub host-key entries on workspace destroy; mixing into ~/.ssh/known_hosts would mean editing a file the tool didn’t own.

The format matches OpenSSH’s ~/.ssh/known_hosts exactly (so future cross-pollination is technically possible), but the filenames are deliberately separate.

TOFU on first connect

The first time you connect to a target, roksbnkctl shows the host key fingerprint and asks whether to trust it:

$ roksbnkctl exec --on jumphost -- whoami
Add 169.45.91.177:22's key (SHA256:abc123def456ghi789jkl0mnopqrstuvwxyz/+=) to ~/.roksbnkctl/known_hosts? [y/N]: y
ubuntu

Answer y and the key is appended. Subsequent connects to the same host:port with the same server key trust silently.

Answer n and the connect aborts with exit code 126.

Mismatch behaviour

If the host key changes — re-provisioned VM, MITM attack, configuration drift — roksbnkctl refuses to connect:

error: host key mismatch: 169.45.91.177:22 known with SHA256:abc123... but
       server presented SHA256:zyx987...; if the host was rebuilt, edit
       ~/.roksbnkctl/known_hosts

Same model OpenSSH uses. The fix is the same: edit (or ssh-keygen -R) the file to remove the stale entry, then re-connect to re-trigger the TOFU prompt.

The default ssh-keygen binary works against ~/.roksbnkctl/known_hosts — pass -f:

ssh-keygen -R 169.45.91.177 -f ~/.roksbnkctl/known_hosts

--insecure-host-key for CI

Automation contexts can’t answer a TOFU prompt. The --insecure-host-key flag skips host-key verification entirely:

roksbnkctl exec --on jumphost --insecure-host-key -- whoami

This is insecure — anyone in the network path can MITM the connection — and is intended only for short-lived CI runs against ephemeral test infrastructure. Don’t use it where session content is sensitive.

The flag is per-invocation, not per-target. There’s deliberately no targets.<name>.insecure_host_key: true config knob: forcing the choice into the call site keeps the security implications visible at every invocation.

When to use it:

  • E2E tests against a freshly-provisioned cluster jumphost where the host key is just-generated and changes per run.
  • Pipeline runs against ephemeral test VMs that get torn down within minutes.
  • Recovery scenarios where the known-hosts file is corrupt and you need to bootstrap.

When not to use it:

  • Production jumphosts with stable identity.
  • Customer environments where session integrity matters.
  • Anything where the SSH session carries secrets you can’t afford to leak to a passive attacker.

roksbnkctl targets — full reference

Four subcommands. Chapter 16 introduces them with worked examples; here’s the complete flag surface.

roksbnkctl targets list

NAME       HOST                USER     KEY
jumphost   169.45.91.177:22    ubuntu   tf-output:jumphost_shared_key
bastion    ops.example.com:22  jgruber  file:~/.ssh/id_ed25519
prod-jump  10.0.0.5:22         ec2-user agent

Flags:

  • --verbose / -v: also prints whether the target has a known-hosts entry recorded.
  • -o json: machine-readable form. Schema: {"targets": [{"name": ..., "host": ..., "port": ..., "user": ..., "key_source": ...}]}.

The KEY column shows the source descriptor — never the key material. File-backed sources are prefixed file: to distinguish them visually from tf-output: and agent.

roksbnkctl targets show <name>

name:        jumphost
host:        169.45.91.177
port:        22
user:        ubuntu
key_source:  tf-output:jumphost_shared_key

Same restriction: key material itself is never printed.

-o json is supported for scripted callers.

roksbnkctl targets add <name> ...

Required flags: --host, --user, and exactly one of --key-path / --key-source.

# File-backed key
roksbnkctl targets add bastion \
  --host ops.example.com \
  --user jgruber \
  --key-path ~/.ssh/id_ed25519

# ssh-agent
roksbnkctl targets add prod-jump \
  --host 10.0.0.5 \
  --user ec2-user \
  --key-source agent

# Non-default port
roksbnkctl targets add custom \
  --host 10.0.0.5 \
  --user root \
  --key-path ~/.ssh/custom \
  --port 2222

# tf-output (rare; usually auto-populated by `up`)
roksbnkctl targets add backup-jump \
  --host 10.0.0.6 \
  --user ubuntu \
  --key-source tf-output:backup_jumphost_key

Refuses to add a target whose name collides with an existing entry — use targets remove <name> first, or pick a different name.

roksbnkctl targets remove <name>

roksbnkctl targets remove bastion

Removes the entry from config.yaml. Does not remove the corresponding host-key line from ~/.roksbnkctl/known_hosts — re-adding the same target later doesn’t re-trigger TOFU. This is deliberate; if you want to wipe the host key too, edit the known-hosts file by hand.

Auto-discovery from terraform outputs

The single most-used target — jumphost — is auto-populated post-roksbnkctl up. The flow:

  1. roksbnkctl up runs terraform apply against the workspace’s HCL.

  2. After successful apply, roksbnkctl reads three outputs: testing_tgw_jumphost_ip, testing_tgw_jumphost_user, jumphost_shared_key.

  3. If testing_tgw_jumphost_ip is non-empty AND not the literal sentinel string "TGW jumphost not created" (which the upstream HCL emits when the testing module is disabled), roksbnkctl writes a jumphost target into config.yaml:

    targets:
      jumphost:
        host: <testing_tgw_jumphost_ip>
        user: <testing_tgw_jumphost_user || "ubuntu">
        key_source: tf-output:jumphost_shared_key
    
  4. A confirmation line is printed:

    ✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`
    

The auto-population is idempotent — re-running up against an already-jumphost-populated workspace re-writes the same fields. If you’ve manually customised the entry (changed the user, swapped to key_path), the auto-population overwrites your changes. There’s no merge logic; the latest up wins.

If testing_create_tgw_jumphost = false in tfvars, the upstream HCL skips creating the jumphost VM and emits the sentinel output. Auto-population is then a no-op, and you’re free to create your own jumphost (or differently-named) entry via targets add.

Per-AZ cluster jumphosts (jumphost-<zone>)

When testing_create_cluster_jumphosts = true, the deploy builds one cluster jumphost per cluster-VPC availability zone in addition to the single TGW jumphost — each on its own floating IP, all sharing the same key as the TGW jumphost. The same post-up hook that seeds the singular jumphost also auto-registers one target per AZ:

  1. After a successful up, roksbnkctl reads the testing_cluster_jumphost_ips terraform output — a map { zone => floating-IP }.

  2. For each zone => fip, it upserts a target named jumphost-<zone>, reusing the same shared key the singular jumphost uses:

    targets:
      jumphost-ca-tor-1:
        host: <ca-tor-1-fip>
        user: ubuntu
        key_source: tf-output:jumphost_shared_key
      # …one per AZ…
    
  3. A summary line is printed:

    ✓ Auto-registered 3 per-AZ cluster jumphost targets (jumphost-ca-tor-1, jumphost-ca-tor-2, jumphost-ca-tor-3); use `roksbnkctl --on jumphost-<zone> ...`
    

Verify with roksbnkctl targets list — you should see jumphost plus one jumphost-<zone> per AZ. Each is a first-class --on target (full kubectl/oc/ibmcloud/shell passthrough, no SSH hop): roksbnkctl --on jumphost-ca-tor-2 kubectl get pods.

Preinstalled load generators. Every jumphost — the TGW jumphost and each per-AZ jumphost-<zone> — comes with iperf3 and h2load (the nghttp2-client package) already installed by the Testing-phase user_data. So both test throughput (iperf3) and the performance matrix (iperf3 + h2load) run over --backend ssh:<target> with no --bootstrap — the SSH backend’s apt-bootstrap path (Chapter 17 §“SSH backend”) is needed only for hosts you registered by hand. Like the singular jumphost, registration is best-effort and idempotent — a parse/write failure logs a single warning: and does not fail up, and re-running up after a floating-IP rotation refreshes each jumphost-<zone> host in place. When testing_create_cluster_jumphosts = false (or the output is absent/empty), this is a silent no-op — only jumphost is seeded, with no warning noise.

Orphaned-target caveat (option (a) upsert-only). Auto-registration upserts but never prunes. If a later apply removes a zone, or testing_create_cluster_jumphosts is flipped to false, the now-orphaned jumphost-<oldzone> target points at a destroyed host and lingers in your config until you remove it by hand:

roksbnkctl targets remove jumphost-ca-tor-3

A host re-created on a recycled floating IP will also trip the host-key mismatch refusal — see §“Host-key TOFU and ~/.roksbnkctl/known_hosts” and clear the stale known_hosts line with ssh-keygen -R <fip> -f ~/.roksbnkctl/known_hosts. An automatic-prune (reconcile) mode that removes orphans on the next up is a deliberate post-v1.5.0 follow-up (it needs unambiguous “this target is auto-managed” ownership semantics so a hand-named jumphost-mybox is never deleted). See PRD 09.

What is not auto-discovered

The auto-discovery flow registers the TGW jumphost (always) and the per-AZ jumphost-<zone> targets when testing_create_cluster_jumphosts = true. It does not register:

  • Per-AZ jumphosts by private IP. There is no top-level testing_cluster_jumphost_private_ips terraform output; the private-IP hop pattern in Chapter 16 §“Per-AZ cluster jumphosts” is a documented zero-setup technique, not an auto-registered target.
  • Any non-jumphost host. Bastions, ops boxes, and the like are always targets add by hand.

Inspecting what the post-up flow saw

When the auto-population doesn’t happen and you expected it to, check:

roksbnkctl tf output testing_tgw_jumphost_ip
roksbnkctl tf output testing_tgw_jumphost_user
roksbnkctl tf output -json jumphost_shared_key | head -c 50

(The third one returns a JSON-encoded string for a sensitive output; truncate to confirm it’s non-empty without dumping the key.)

If all three are populated and the auto-write didn’t fire, that’s a bug — file an issue with the output values redacted.

What the SSH execution backend adds on top of --on

The --on <target> flag is the lightweight remote-exec path — one SSH session, one command, no remote state. The ssh:<target> execution backend layers more on top, reusing the same internal/remote.Client under internal/exec/ssh.go:

CapabilityWhat it gives you
File materialisationRunOpts.Files map gets written to /tmp/roksbnkctl.<rand>/<basename> on the remote, available as the working directory for the command. Cleanup via trap 'rm -rf' EXIT in a wrapper.
Env passing with fallbackFirst tries ssh -o SetEnv=KEY=VALUE (requires remote sshd AcceptEnv). On failure, writes a 0700 wrapper script that exports the env and execs the command, with trap 'rm -f $0' EXIT to scrub.
Apt bootstrapIf the remote target doesn’t have a tool (iperf3, ibmcloud) installed, the backend can sudo apt-get install it on demand (Ubuntu only).
SCP-and-cleanup for kubeconfigThe backend’s recommended path for shipping a kubeconfig to the remote: SCP to a tempdir, run, trap 'rm -rf' EXIT to scrub.
Wrapper-script credential propagationDetailed in PRD 04 § SSH. Brief on-disk window with strict cleanup.

The targets: schema and the roksbnkctl targets commands are the same surface for both — the backend just uses each target in a heavier-weight way. Anything you set up for --on keeps working under --backend ssh:<target>.

The split between the lightweight --on path and the full ssh backend is deliberate: --on stays simple — one SSH session, one command, no remote state. The backend handles the heavier lifting (file materialisation, package installation, multi-step orchestration).

Cross-references

The –on flag and SSH jumphosts

The --on <target> flag (most commonly --on jumphost) re-runs a roksbnkctl passthrough command (exec, shell, kubectl, oc, ibmcloud) on a remote SSH host instead of locally. After a successful roksbnkctl up, a jumphost target is auto-populated from the upstream HCL’s terraform outputs, so --on jumphost works with no manual configuration in the common case.

This chapter covers when to reach for --on, the targets: workspace config block, the auto-population behaviour, the roksbnkctl targets command tree for managing your own targets, and how host-key trust is established.

The full design rationale for this feature lives in PRD 01. This chapter is the user-facing distillation.

Why this exists

There are a handful of scenarios where running a command from your laptop is the wrong answer:

  • Customer-firewall scenarios. Your customer’s network policy lets the corporate jumphost reach *.cloud.ibm.com but blocks your laptop’s egress to anything except web traffic. ibmcloud iam oauth-tokens works from the jumphost; from your laptop it times out.
  • Air-gapped environments. The cluster lives in a VPC with no public ingress, accessible only via a bastion VM. The cluster API server isn’t reachable from your laptop at all; you need to be inside the network to talk to it.
  • Pre-cluster operations. You want to run ibmcloud commands against the IBM Cloud API but your workstation doesn’t have ibmcloud installed and you’d rather not install it. The jumphost has it; route through there.

--on makes those scenarios one flag rather than “ssh to the jumphost, install your tools there, copy your kubeconfig over manually”. The SSH client is built into roksbnkctl (using golang.org/x/crypto/ssh); no host ssh binary is required.

The targets: workspace config block

Targets are stored in your workspace config at ~/.roksbnkctl/<workspace>/config.yaml under a targets: key:

targets:
  jumphost:                                # auto-populated after `roksbnkctl up`
    host: 169.45.91.177
    user: ubuntu
    key_source: tf-output:jumphost_shared_key
    port: 22                               # default; can be omitted

  bastion:                                 # user-defined
    host: ops.example.com
    user: jgruber
    key_path: ~/.ssh/id_ed25519

  prod-jump:
    host: 10.0.0.5
    user: ec2-user
    key_source: agent                      # use ssh-agent

Each entry has at minimum host and user. Port defaults to 22. Key resolution is determined by exactly one of key_path or key_source — see “Key sources” below.

You don’t typically edit this file by hand. The auto-discovery flow populates jumphost for you, and roksbnkctl targets add ... populates other entries.

Auto-discovery from roksbnkctl up

The upstream HCL provisions a small testing jumphost as part of every cluster apply. Two terraform outputs surface it:

  • testing_tgw_jumphost_ip — the public IP of the jumphost VM.
  • jumphost_shared_key — the private key (PEM) the jumphost was provisioned with, marked sensitive in the HCL.

After a successful roksbnkctl up, roksbnkctl reads both outputs and writes a jumphost target into your workspace config:

✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`

The auto-registered target uses user: ubuntu (the upstream HCL provisions an Ubuntu cloud image whose default user is ubuntu).

The key_source: tf-output:jumphost_shared_key form means the private key is read from terraform state at SSH-connect time rather than being copied into the workspace config or written to disk separately. The key only ever exists in terraform state and in memory during a connect; destroying and re-creating the cluster generates a new key, and roksbnkctl picks up the new one without any manual intervention.

If your cluster apply produced a testing_tgw_jumphost_ip output of "TGW jumphost not created" (the upstream HCL emits this string when the testing module is disabled) the auto-population is skipped. You can still add a jumphost target manually with roksbnkctl targets add if you have a different bastion in mind.

Key sources

Three ways to tell roksbnkctl how to find the SSH private key:

  1. key_path: <path> — a file on disk. Standard OpenSSH key formats are accepted (~/.ssh/id_ed25519, ~/.ssh/id_rsa, etc.). Tilde expansion is honoured.

  2. key_source: agent — talk to the user’s ssh-agent over the socket pointed at by $SSH_AUTH_SOCK. The agent presents whichever keys it currently holds; roksbnkctl tries each in turn against the target’s authorized_keys. This is the right setting if your team already manages keys via 1Password / hardware tokens / gpg-agent and you don’t want a key file on disk. Note: ssh-agent integration is Linux/macOS-only; Windows users should use key_path instead. Windows ssh-agent named-pipe support is on the v1.x roadmap.

  3. key_source: tf-output:<output-name> — read the key from the workspace’s terraform state output of that name. Used by the auto-discovered jumphost target. The terraform output must be a string-typed PEM-encoded private key; sensitive outputs work fine because terraform output -raw <name> returns the value regardless of the sensitive flag.

Exactly one of key_path or key_source must be set per target. roksbnkctl targets show <name> will tell you which is in use without printing the key material.

Host-key TOFU on first connect

The first time you connect to a target, roksbnkctl shows the host key fingerprint and asks whether to trust it. The prompt is a single line:

$ roksbnkctl exec --on jumphost -- whoami
Add 169.45.91.177:22's key (SHA256:abc123def456ghi789jkl0mnopqrstuvwxyz/+=) to ~/.roksbnkctl/known_hosts? [y/N]: y
ubuntu

Answer y and the key is appended to ~/.roksbnkctl/known_hosts (the same format as OpenSSH’s ~/.ssh/known_hosts). Subsequent connects trust silently.

Answer n and the connect fails with a clear “host key not trusted” error.

If the host key changes between runs — which would happen on a re-provisioned VM, or could happen as a man-in-the-middle attack — roksbnkctl refuses to connect:

error: host key mismatch: 169.45.91.177:22 known with SHA256:abc123... but server presented SHA256:zyx987...; if the host was rebuilt, edit ~/.roksbnkctl/known_hosts

This is “trust on first use” (TOFU) — the same model OpenSSH uses for new hosts. Exit code is 126 on host-key rejections.

--insecure-host-key for CI

In automation contexts where a TOFU prompt would block forever, pass --insecure-host-key to skip host-key verification entirely:

roksbnkctl exec --on jumphost --insecure-host-key -- whoami

This is insecure — anyone in the network path can MITM the connection — and is intended only for short-lived CI runs against ephemeral test infrastructure. Don’t use it in any context where the SSH session matters for security.

The roksbnkctl targets command tree

Four subcommands for managing target entries:

roksbnkctl targets list
roksbnkctl targets show <name>
roksbnkctl targets add <name> --host ... --user ... --key-path ...
roksbnkctl targets remove <name>

targets list

roksbnkctl targets list
NAME       HOST                USER     KEY
jumphost   169.45.91.177:22    ubuntu   tf-output:jumphost_shared_key
bastion    ops.example.com:22  jgruber  file:~/.ssh/id_ed25519

Prints every target in the current workspace’s config. The KEY column shows the key source — never the key material itself. File-backed keys are prefixed with file: so they’re visually distinct from tf-output: and agent sources.

targets show <name>

roksbnkctl targets show jumphost
name:        jumphost
host:        169.45.91.177
port:        22
user:        ubuntu
key_source:  tf-output:jumphost_shared_key

Prints the full record. Note that the key material itself is never printed — only the source descriptor (file path, ssh-agent, or terraform-output name).

targets add <name> ...

roksbnkctl targets add bastion \
  --host ops.example.com \
  --user jgruber \
  --key-path ~/.ssh/id_ed25519

# or with ssh-agent:
roksbnkctl targets add prod-jump \
  --host 10.0.0.5 \
  --user ec2-user \
  --key-source agent

# or with a non-default port:
roksbnkctl targets add custom \
  --host 10.0.0.5 \
  --user root \
  --key-path ~/.ssh/custom \
  --port 2222

Writes the new target into ~/.roksbnkctl/<workspace>/config.yaml. Refuses if a target of that name already exists (use targets remove first).

targets remove <name>

roksbnkctl targets remove bastion

Removes the entry from config.yaml. Does not remove the corresponding line from ~/.roksbnkctl/known_hosts — the host key stays recorded so re-adding the same target later doesn’t re-trigger a TOFU prompt.

Working examples

The everyday verbs:

# Run an arbitrary command on the jumphost
roksbnkctl exec --on jumphost -- whoami
# → ubuntu

roksbnkctl exec --on jumphost -- uname -a
# → Linux jumphost-vm 5.15.0-... #... SMP ... x86_64 GNU/Linux

# Interactive PTY shell
roksbnkctl shell --on jumphost
# → drops you into the jumphost's default shell as the configured user
# → exit returns you to your local prompt

# ibmcloud passthrough — runs `ibmcloud ks cluster ls` on the jumphost
# (handy when your laptop's network can't reach IBM Cloud APIs)
roksbnkctl ibmcloud --on jumphost ks cluster ls

# kubectl passthrough — same pattern
roksbnkctl kubectl --on jumphost get pods -A

# oc passthrough
roksbnkctl oc --on jumphost projects

Per-AZ cluster jumphosts

When your deploy sets testing_create_cluster_jumphosts = true, the upstream HCL builds one cluster jumphost per cluster-VPC availability zone in addition to the single TGW jumphost. These are auto-registered by roksbnkctl up as jumphost-<zone> targets (alongside the singular jumphost) — see Chapter 15 §“Auto-discovery from terraform outputs”. They are first-class --on targets: full passthrough, no hop.

# verify what `up` registered:
roksbnkctl targets list
# → jumphost, jumphost-ca-tor-1, jumphost-ca-tor-2, jumphost-ca-tor-3

# run directly on a specific per-AZ cluster jumphost (no hop, full passthrough):
roksbnkctl kubectl --on jumphost-ca-tor-2 get nodes
roksbnkctl shell --on jumphost-ca-tor-1

Preinstalled perf tools. Every jumphost ships with iperf3 and h2load (the nghttp2-client package) preinstalled by the Testing-phase user_data, so the performance matrix and test throughput run over ssh:<target> with no --bootstrap. The matrix’s locality axis (same-zone / different-zone / different-VPC) is just which jumphost-<zone> (or the TGW jumphost) a cell names as its client — see Chapter 22a.

Hopping to a cluster jumphost via the registered jumphost (zero-setup, no roksbnkctl state)

The shared key is installed on every jumphost and the private key file is present on each box at /home/ubuntu/.ssh/id_rsa. So from the auto-registered TGW jumphost you can hop to any cluster jumphost by its private IP (the TGW jumphost reaches the cluster VPC over the Transit Gateway) with no key copying and nothing added to roksbnkctl state — handy for a one-off against a host you don’t want to register:

# the per-AZ private IPs are not a top-level terraform output; read them
# from inside the TGW jumphost (it sits in the routed network):
roksbnkctl exec --on jumphost -- \
  curl -s http://169.254.169.254/...   # or your own inventory source

# run a command on a cluster jumphost, hopping through the TGW jumphost,
# using the shared key already on the box (no scp):
roksbnkctl exec --on jumphost -- \
  ssh -i /home/ubuntu/.ssh/id_rsa -o StrictHostKeyChecking=accept-new \
  ubuntu@<cluster-jumphost-private-ip> kubectl get nodes

The inner ssh uses the on-box /home/ubuntu/.ssh/id_rsa (the same shared key the auto-registered targets use — one key for all jumphosts), so no key material is copied. StrictHostKeyChecking=accept-new on the inner hop avoids an interactive prompt the outer non-PTY session can’t answer. This is the zero-setup path: nothing is written to ~/.roksbnkctl/. For first-class repeated access prefer the auto-registered jumphost-<zone> targets above.

Orphaned-target caveat. The auto-registered jumphost-<zone> targets use option-(a) upsert-only registration: if a destroy removes a zone (or testing_create_cluster_jumphosts is flipped to false), the stale jumphost-<oldzone> entry lingers in your config until you roksbnkctl targets remove jumphost-<oldzone> by hand. A re-created host on a recycled floating IP will also trip the host-key TOFU mismatch — see Chapter 15 §“Host-key TOFU and ~/.roksbnkctl/known_hosts”. An automatic-prune (reconcile) mode is a tracked post-v1.5.0 follow-up.

Behaviour details worth knowing:

  • Streaming I/O. stdout, stderr, stdin all stream in real time — the same as running the command locally. Long-running commands (oc adm top nodes, ibmcloud ks cluster get on a slow API call) work normally.
  • Exit code propagation. The remote command’s exit code is the local exit code. A failing remote command produces a non-zero roksbnkctl exit; a succeeding remote command produces 0. CI scripts can rely on this.
  • TTY auto-detection. roksbnkctl shell --on auto-allocates a PTY. Other verbs (exec, kubectl, oc, ibmcloud) run without a PTY; if you need a PTY for top or another isatty()-sensitive command, fall back to roksbnkctl shell --on jumphost and run the command from the interactive shell.
  • Environment passthrough. Only machine-portable values cross the SSH boundary: IBMCLOUD_API_KEY, IC_API_KEY, IBMCLOUD_REGION, and IBMCLOUD_VERSION_CHECK are propagated to the remote session via SSH SetEnv, so ibmcloud iam oauth-tokens on the jumphost authenticates with the same key your local workspace uses. The remote sshd must be configured to accept AcceptEnv IBMCLOUD_* etc. for this to work; the upstream HCL’s jumphost is already configured for it. KUBECONFIG is not forwarded — it is a local filesystem path, meaningless on the target, and forwarding it would shadow the jumphost’s own cloud-init-provisioned /home/ubuntu/.kube/config. Passthrough kubectl/oc on the target therefore use the target’s own kubeconfig, which is the correct behaviour. Local roksbnkctl kubectl without --on still resolves KUBECONFIG via the local chain, unchanged.

What --on doesn’t do (yet)

A few things deliberately deferred to later phases:

  • Lifecycle commands (up, down, plan, apply) reject --on with a clear error. Running terraform on a remote host has different state-handling considerations and is the job of the SSH execution backend (Chapter 17) — terraform over --backend ssh:<target> is itself deferred to v1.x (state-file portability).
  • ProxyJump / multi-hop SSH. If your jumphost itself is reached through another bastion, that’s not directly supported. The upstream HCL’s jumphost design lets the TGW jumphost reach cluster-internal VMs natively, so you usually don’t need multi-hop in practice. ProxyJump support is on the v1.x roadmap. (The per-AZ cluster jumphosts themselves are auto-registered as first-class jumphost-<zone> targets — see §“Per-AZ cluster jumphosts” above; the manual ssh -i … via the registered jumphost hop there is the zero-setup alternative for one-offs, not a roksbnkctl-native multi-hop.)
  • ~/.ssh/config parsing. Targets must be defined explicitly in workspace config; roksbnkctl does not read your existing ~/.ssh/config.
  • Password auth. Keys + agent only. Passwords are not supported and won’t be.
  • SCP / SFTP. File transfer is the SSH execution backend’s job (handled via RunOpts.Files materialisation; see Chapter 17 §“SSH backend”). --on does one-shot remote exec only.
  • Windows ssh-agent. The key_source: agent path is Linux/macOS only; Windows users must use key_path to a file. Already noted in Key sources above; called out here so a Windows reader who skipped to this section doesn’t miss it.

Cross-reference

Chapter 17 — Execution backends extends the SSH client used here into a full execution backend with file materialisation, env-file fallback for sshd configurations that can’t AcceptEnv, and apt-bootstrap of missing tools on Ubuntu jumphosts. The --on flag stays as the lightweight one-shot path; --backend ssh is the deeper integration. The two share the same internal/remote.Client so what you learn here translates directly.

For the design rationale, edge cases, and open questions, read PRD 01 — this chapter is the user-facing surface; PRD 01 is the developer-facing surface.

Execution backends: local, docker, k8s, ssh

Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the Workspace struct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.

roksbnkctl runs a handful of external tools as part of its job — ibmcloud, terraform, iperf3, eventually dig-equivalents and others. By default each tool runs as a child process on your laptop. That’s fine for some tools and wrong for others: iperf3 from your laptop measures your laptop’s internet uplink, not the cluster’s bandwidth. Likewise, terraform via docker (the --backend docker mode covered below) lets you pin a frozen tool version for CI reproducibility without installing it on the host.

The execution-backend system lets you pick where each tool runs without changing the surface command. The same roksbnkctl ibmcloud ks cluster ls invocation can run as a local process, inside a vendored container, inside the cluster, or on a remote SSH host — selected by a flag or a per-tool default in your workspace config.

This chapter is the user-facing reference for all four backends. After the introduction, each backend gets its own deep-dive section covering the mechanics, the credential-propagation rules, the failure modes, and a short “when to use it” callout. Chapter 18 is the decision-tree companion that picks one for a given (tool, scenario) pair.

Architecture at a glance

The four backends sit between roksbnkctl (the binary on your laptop) and the external tools each backend runs. Every backend produces the same observable behaviour from the user’s point of view — the same roksbnkctl ibmcloud ks cluster ls invocation — but routes the actual tool execution to a different network vantage.

graph LR
    User[laptop<br/>roksbnkctl binary]
    subgraph local[local backend]
        L_tf[terraform]
        L_ibm[ibmcloud]
        L_iperf[iperf3]
        L_dns[dns probe]
    end
    subgraph docker[docker backend]
        D_ibm[ibmcloud<br/>frozen image]
        D_tf[terraform<br/>frozen image]
    end
    subgraph k8s[k8s backend]
        K_ops[ops pod<br/>ibmcloud]
        K_job[one-shot Job<br/>iperf3 / dns]
    end
    subgraph ssh[ssh:target backend]
        S_ibm[ibmcloud<br/>on jumphost]
        S_iperf[iperf3<br/>on jumphost]
    end
    Cluster[ROKS cluster<br/>cert-manager + flo + BNK]
    Jump[SSH jumphost<br/>auto-discovered from terraform]
    IBMAPI[IBM Cloud API<br/>+ IAM]

    User --> local
    User --> docker
    User --> k8s
    User --> ssh
    local --> IBMAPI
    docker --> IBMAPI
    k8s --> Cluster
    Cluster --> IBMAPI
    ssh --> Jump
    Jump --> IBMAPI
    Jump -.->|optional<br/>private path| Cluster

    classDef bk fill:#f4f4f4,stroke:#666,color:#000;
    class local,docker,k8s,ssh bk;

Chapter 18 is the decision-tree companion that picks a backend for a given scenario; the rest of this chapter is the per-backend mechanics.

The four backends at a glance

BackendWhat it does
localos/exec — spawns the tool as a child process, inheriting your env and PATH
dockerdocker run against a vendored image (ghcr.io/jgruberf5/roksbnkctl-tools-<tool>:<v>); frozen toolchain version
k8sRuns inside the cluster, either in a long-lived ops pod or as a one-shot Job; auth via the pod’s ServiceAccount token
sshRuns on a registered SSH target via the built-in SSH client; opt-in apt-bootstrap of missing tools on Ubuntu

Each backend solves a different problem:

  • local: fastest startup, simplest mental model, requires the host tool to exist on PATH.
  • docker: reproducible across dev machines, no host install needed, frozen at a known-good tool version.
  • k8s: network-correct (private IPs reachable, cluster-internal services accessible), zero host install, in-cluster identity via ServiceAccount.
  • ssh: pre-cluster ops from a known-IP bastion, customer-firewall workflows, air-gapped environments where the laptop can’t reach IBM Cloud APIs but the jumphost can.

All four implementations conform to the same Go interface (internal/exec.Backend) so callers don’t branch on backend type — they just call backend.Run(ctx, argv, opts) and let the implementation handle the mechanics. That uniformity is what lets the same roksbnkctl ibmcloud ks cluster ls work across all four with no surface-level change.

The --backend CLI flag

Override the per-tool default for a single invocation:

# Local (the implicit default for ibmcloud + terraform)
roksbnkctl ibmcloud ks cluster ls

# Same command, in a vendored docker image
roksbnkctl ibmcloud --backend docker ks cluster ls

# Same command, in the cluster (requires `roksbnkctl ops install` first)
roksbnkctl ibmcloud --backend k8s ks cluster ls

# Same command, on a remote SSH host
roksbnkctl ibmcloud --backend ssh:jumphost ks cluster ls

Format:

--backend local|docker|k8s|ssh:<target>

The ssh:<target> form pins the SSH backend to a specific named target from roksbnkctl targets list (registered via roksbnkctl targets add; see Chapter 16).

The flag is persistent at the root — it works for any command that runs an external tool. Commands that don’t run external tools (like roksbnkctl ws list) ignore it.

The flag wins over the workspace-config default. If config.yaml says iperf3: { backend: k8s } and you pass --backend local, the local backend runs.

Backend-failure semantics

Each backend has a different failure surface. The convention is:

  • Backend startup failure (Docker daemon unreachable, k8s API unreachable, SSH connect refused, binary not on PATH for local) ⇒ exit code 127, with a message naming the cause. No silent fallback to local. Silent fallback hides intent and produces confusing test results.
  • Backend mid-run failure (the container started but couldn’t pull a sub-resource; the pod was OOMKilled before the wrapped tool ran; the SSH session died after apt-get install but before the tool exec) ⇒ exit code 126, distinct from 127 so CI can tell “we never got going” from “we got going then broke”.
  • Tool exit code (the actual ibmcloud / terraform / iperf3 exit code, anything in 0-125 or 128-255) ⇒ propagated 1:1, including non-zero codes.
  • Context cancellation / timeout ⇒ exit code 137 (the conventional SIGKILL-on-signal code).

This way, your CI script can tell “the tool said X failed” (typical exit codes) from “we never reached the tool” (127) from “we reached the tool, then the backend died mid-flight” (126) from “we ran out of time” (137).

One code outside the backend surface: 125

roksbnkctl upgrade and roksbnkctl self update exit 125 when an upgrade removed the old binary and could not put anything back — so there is no roksbnkctl at the install path and the only copy is the .old sidecar beside it. The error names the file and the command to rename it.

This is separate from 1 because the two need opposite responses. An ordinary failed upgrade is safe to retry; this one cannot be retried at all, because there is nothing left to run. A wrapper that treats them alike either loops forever on a machine that will never recover, or reports a bricked install as routine.

Note the overlap with the passthrough range above. It is deliberate and bounded, but it is an overlap: a wrapped tool that exits 125 still propagates 125, exactly as the table says. What cannot happen is a single invocation meaning both — upgrade and self update spawn no child process at all (extraction, checksum verification and the install are in-process), so a 125 from those two commands has exactly one meaning, and a 125 from anything else is a child’s status. This is the same bargain already made for 2, which is both “bad invocation” and whatever a child chose to exit with.

In practice this fires on Windows, where a running .exe is locked and the upgrade works by renaming the old binary aside before moving the new one in. A second process holding a handle at that moment — an antivirus scanner opening the freshly written file is the usual one — can make the second rename fail; if the rollback then fails too, this is the result.

Per-tool defaults from exec:

Workspace config carries the per-tool default backend in the exec: block:

# ~/.roksbnkctl/<workspace>/config.yaml
exec:
  ibmcloud:  { backend: local }
  iperf3:    { backend: k8s }
  terraform: { backend: local }

The defaults shipped today:

ToolDefault backendSupported backendsWhy
terraformlocallocal, dockerThe terraform-exec local path is the established workflow. State handling is simplest here. The docker backend runs frozen hashicorp/terraform:1.5.7 with a bind-mounted state dir — see § terraform via docker. k8s and ssh are deferred to v1.x.
ibmcloudlocallocal, docker, k8s, ssh:<target>Most users have it on PATH or are happy installing it. Compliance/firewall scenarios opt in via --backend ssh:jumphost or docker.
iperf3k8slocal, k8s, ssh:<target>Throughput from a laptop’s uplink isn’t the cluster’s bandwidth. The k8s default runs the iperf3 client adjacent to (or inside) the cluster so the number reflects cluster fabric, not your office Wi-Fi.
dnslocallocal, k8s, ssh:<target>Single-vantage by default; --gslb-compare fans out across configured vantages for GSLB validation. See Chapter 21.

Chapter 12 — Workspace config covers the exec: block schema in detail; this chapter just notes its place in the backend system.

Chapter 18 — Choosing a backend per tool is the decision tree for “which backend should I pick for this tool in this scenario”.

Per-backend deep dives

local backend

The default for ibmcloud and terraform. os/exec.CommandContext(ctx, argv[0], argv[1:]...), inheriting the parent process’s environment, PATH, and working directory. Mechanically the simplest of the four — no container, no cluster, no network handshake.

os/exec shape

internal/exec/local.go resolves argv[0] via exec.LookPath, then builds a *exec.Cmd:

bin, err := exec.LookPath(argv[0])
// fall through to argv[0] verbatim if it's an absolute path that LookPath rejects
cmd := exec.CommandContext(ctx, bin, argv[1:]...)
cmd.Env    = effectiveEnv     // os.Environ() + opts.Env + Credentials.EnvVars()
cmd.Dir    = opts.WorkDir     // empty → inherit caller's CWD
cmd.Stdin  = opts.Stdin
cmd.Stdout = redactor(opts.Stdout, creds)
cmd.Stderr = redactor(opts.Stderr, creds)

The redactor wrap is defense-in-depth — see Chapter 14 §“The redactor”. If a wrapped tool ever prints IBMCLOUD_API_KEY value to stdout (a debug trace, an error message), the redactor replaces it with [REDACTED] before the bytes leave the binary.

Env propagation

Three sources, in order:

  1. The host process’s environment (os.Environ()) — your shell’s PATH, HOME, KUBECONFIG, etc.
  2. RunOpts.Env — caller-supplied KEY=VALUE strings (e.g., IBMCLOUD_REGION=ca-tor from the workspace config).
  3. Credentials.EnvVars() — IBMCLOUD_API_KEY=… plus the legacy IC_API_KEY=… alias older ibmcloud versions accept.

os/exec documents that for duplicate keys the last entry wins. So caller-supplied vars override host env, and credential vars override caller-supplied — meaning a workspace’s API key always wins over a stale IBMCLOUD_API_KEY in your shell.

The local backend does not scrub the host env. If you have an unrelated AWS_ACCESS_KEY_ID in your shell, the wrapped tool sees it. That’s by design — local is the “trust the user’s shell” path; if you want a hermetic env, switch to docker.

Working directory

RunOpts.WorkDir becomes cmd.Dir. Empty → inherit the caller’s CWD (Cobra’s RootCmd.Run runs from wherever the user invoked roksbnkctl).

When RunOpts.Files is non-empty and WorkDir is empty, the local backend creates a tempdir under os.TempDir(), writes each Files entry as a 0600 file inside, and uses the tempdir as WorkDir. The tempdir is removed via defer after Run returns. This is mostly there for symmetry with the docker / k8s / ssh backends; today’s ibmcloud passthrough never uses it.

Signal handling

exec.CommandContext wires ctx cancellation to the child: when the ctx ticks past its deadline (or the user hits Ctrl-C and the root cobra command cancels), Go sends SIGKILL (the default Cmd.Cancel) to the child. The child has no opportunity to clean up; this is intentional — we’d rather kill a stuck terraform than wait on an indefinite hang.

The kill is process-only, not process-group. If terraform has spawned grandchildren (the IBM provider’s helpers, an SSH key generator, etc.) those grandchildren may outlive the ctx-cancel by a few seconds. We haven’t seen this matter in practice; if it does, a pgid kill is a small follow-up.

Exit-code mapping

OutcomeExit codeSource
Child exits 00child
Child exits non-zero (e.g., terraform plan saw drift)child’s exit code, 1-125 or 128-255child
argv[0] not on PATH and not an absolute path127local backend (POSIX shell convention)
Child binary couldn’t be exec’d despite being present (e.g., not executable)126local backend (mid-run failure: we found the binary but couldn’t spawn it)
Ctx cancelled mid-run, child SIGKILL’d137128 + SIGKILL

Note the 126 vs 127 split: 127 means “we never reached the tool” (binary missing, daemon unreachable, SSH refused); 126 means “we reached the tool but the backend itself broke after that point” (couldn’t fork, container created but crashed, pod scheduled but evicted before exec). The local + docker backends split 126 vs 127 per PRD 03 §“Backend interface”. CI scripts that distinguish “test infra broken” from “real test failure” can now key on the difference.

upgrade / self update add one code of their own, listed separately because they wrap no child:

OutcomeExit codeSource
Upgrade left no binary at the install path (rollback also failed)125exitcode.SelfUpdateStranded

When to use it

  • You have the tool installed and on PATH already.
  • You want the fastest startup — no container daemon, no SSH handshake, no cluster API call.
  • You’re running terraform against the workspace’s local state (the established workflow).
  • You’re debugging and want the simplest mental model for “where did that output come from”.

Chapter 18 §“Decision tree” expands these into a per-(tool, scenario) walkthrough.

docker backend

Runs the tool inside a vendored container image, talking to the local docker daemon over its socket. docker on PATH is not required — roksbnkctl uses the official Docker Go SDK (github.com/moby/moby/client) and dials the socket directly.

roksbnkctl ibmcloud --backend docker ks cluster ls

Container shape

Mechanically (the ibmcloud passthrough; iperf3 client is similar with a different image and ports):

docker run --rm \
  -v <tempdir>/kubeconfig:/root/.kube/config:ro \  # if Credentials.KubeconfigBytes set
  -e IBMCLOUD_API_KEY \                            # bare name; value inherits
  -e IC_API_KEY \                                  # legacy alias
  ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:<v> \
  ks cluster ls

internal/exec/docker.go doesn’t shell out to docker run; it builds a container.Config + container.HostConfig and calls cli.ContainerCreate → ContainerStart → ContainerLogs(stream=true). The bash-style above is the conceptual equivalent.

There’s no workspace-wide bind-mount. Per-invocation mounts come from three sources only:

  1. Credentials.KubeconfigBytes — written to <tempdir>/kubeconfig (mode 0600) on the host, bind-mounted as a single file at /root/.kube/config read-only. Single-file mount per PRD 04 §“Anti-patterns” — bind-mounting ~/.kube/ exposes other clusters’ configs.
  2. RunOpts.Files — each name → bytes entry written to <tempdir>/<basename> and bind-mounted at /work/<basename>. The container’s WorkingDir is set to /work so callers can reference files by relative path. (ibmcloud passthrough doesn’t use this; it lands when the iperf3 client backend wants to ship iperf3.json to the pod, or when a future tool wants a config file.)
  3. RunOpts.WorkDir — overrides WorkingDir if explicitly set.

The tempdir is removed via defer after Run returns, regardless of exit code or panic.

Credential propagation specifics

Three things matter, all enforced by internal/exec/creds.go::Credentials.DockerArgs(...):

  1. --env IBMCLOUD_API_KEY (bare name, no =value). The docker daemon looks up the value from the daemon’s environment at container-create time, not from argv. So the literal API key string never appears in docker inspect, docker ps -a --format, or the daemon’s container metadata. PRD 04 §“Anti-patterns” calls out the --env IBMCLOUD_API_KEY=$KEY form as a leak vector — we don’t use it. See Chapter 14 — Credentials.DockerArgs() for the full call shape.
  2. Single-file kubeconfig mount, read-only. Not the parent dir. The container can read exactly the kubeconfig you handed it — nothing else under ~/.kube/.
  3. Stdout/stderr through the redactor. Same defense-in-depth as the local backend: if the wrapped tool prints the API key value (rare but possible), the redactor masks it before the bytes leave roksbnkctl’s process.

:dev tag resolution

The vendored images live at:

ToolImage
ibmcloudghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:<tag> (vendored from icr.io/ibm-cloud/ibmcloud-cli upstream)
iperf3ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<tag> (Alpine + iperf3)
h2loadghcr.io/jgruberf5/roksbnkctl-tools-h2load:<tag> (Alpine + nghttp2; the L7 load generator for test matrix)
terraformhashicorp/terraform:<v> (official upstream)

The h2load image is the L7 generator the performance matrix runs for family: l7 cells. It’s built from tools/docker/h2load/Dockerfile (Alpine’s nghttp2 package, built against OpenSSL so https / TLS-terminate-at-TMM targets work) and published by the tools-images workflow alongside the other per-tool images. Like the iperf3 image it declares USER 1000 so it satisfies the OpenShift restricted-v2 SCC (h2load is a pure client and binds no privileged port). The docker / k8s backends pull this image; the ssh backend uses the jumphost’s apt-installed nghttp2-client instead (preinstalled by the Testing phase — see Chapter 15 §“Per-AZ cluster jumphosts”).

The <tag> for the vendored per-tool images (ibmcloud, iperf3) is resolved at runtime by internal/exec/docker.go::toolImageTag(). It reads the binary’s internal/version.Version (set via ldflags at build time): a release-built binary like v0.10.0 pulls :v0.10.0; a dev build (Version == "dev") pulls :dev. The tag is version-pinned, so a go install of a tagged release pulls a matching tagged image rather than a :dev that may not exist for the published binary. The terraform row is the exception — it points at the upstream hashicorp/terraform image and stays pinned to a specific version (currently 1.5.7) regardless of roksbnkctl’s own version.

The :dev tag is still the local-development idiom: cd tools/docker && make build-all builds and tags every tools image as :dev locally; a dev-build roksbnkctl finds them via the local docker cache without a ghcr.io round-trip.

If you’re cutting a custom tools image and want roksbnkctl to pick it up, the simplest path is docker tag your-image ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:dev locally — the docker backend pulls the local-cached version first.

Auto-remove and ctx-cancel-kill

Two cleanup mechanisms work together:

  • AutoRemove: true in HostConfig. The docker daemon removes the container as soon as it exits, regardless of exit code. No docker ps -a clutter, no manual docker rm ever required.
  • Ctx-cancel triggers ContainerKill. When ctx.Done() fires, the docker backend issues cli.ContainerKill(ctx, id, "SIGKILL") and waits a few seconds for the daemon to confirm. The --rm then takes care of removal. Net effect: hitting Ctrl-C during a stuck ibmcloud login doesn’t leave a zombie container behind.

Combined with the daemon’s own watchdog on the container, the worst case is a few seconds of “container is dying” between Ctrl-C and the container disappearing. We haven’t seen leaked containers in dev or CI.

Image build pipeline

Image versions are tagged in lock-step with roksbnkctl releases; the GitHub Actions workflow that builds + pushes them runs on every release tag. See Chapter 31 — Building from source for the build pipeline details.

terraform via docker

terraform is the second tool routed through the docker backend (alongside ibmcloud). The shape is similar to ibmcloud (docker run against a vendored image, single-file mounts for sensitive data, no creds in argv) but with two terraform-specific concerns: state persistence across runs, and host-user UID alignment so state files written inside the container stay readable on the host.

State persistence via bind-mount

Terraform’s local state file lives at terraform.tfstate in the working directory. For the docker backend the working directory has to be a host-side path bind-mounted into the container, not a container-internal path that disappears on --rm. The docker backend bind-mounts the workspace’s state directory into the container:

docker run --rm \
  -v ~/.roksbnkctl/<workspace>/state:/state \
  --workdir /state/tf-source/embedded-terraform \
  --user $(id -u):$(id -g) \
  hashicorp/terraform:1.5.7 \
  apply -auto-approve

Concretely:

  • Host source: ~/.roksbnkctl/<workspace>/state/ — the same directory the local terraform backend writes state to today, so switching between --backend local and --backend docker against the same workspace doesn’t fork state.
  • Container target: /state — the bind-mount root inside the container.
  • Container working directory: /state/tf-source/<source>/ (e.g., /state/tf-source/embedded-terraform/ for the default embedded source) — the same path the local backend resolves to, so terraform sees the same main.tf either way.
  • The HCL is bind-mounted alongside state. The embedded HCL is materialised at run time into ~/.roksbnkctl/<workspace>/state/tf-source/<source>/ (chapter 31 covers the embedded-source layout); since state/ is the bind-mount root, both terraform.tfstate and the HCL tree land inside the container together. There’s no separate HCL projection.

The bind-mount is read-write — terraform needs to write terraform.tfstate, rotate terraform.tfstate.backup, and populate the .terraform/ cache. Combined with --rm, the file lifecycle is: container creates state, container exits, --rm removes the container, state files persist on the host. Subsequent runs (re-mounted at the same host path) pick up where the prior run left off.

Image: hashicorp/terraform:1.5.7

The image is the official upstream hashicorp/terraform published by HashiCorp on Docker Hub, pinned to a literal version in internal/exec/docker.go’s toolImages map (currently 1.5.7). The pin is intentional — the embedded HCL has been validated against this terraform version, and the docker backend’s whole point is reproducibility. Bumping the pin is a deliberate change to the binary and lands as a release.

The vendored per-tool images (ibmcloud, iperf3) get their tag from the roksbnkctl binary’s own version (see § :dev tag resolution above). Terraform is the exception — the binary’s version doesn’t follow upstream terraform’s release cadence, so the pin stays literal.

The UID/GID alignment gotcha

Linux Docker containers run as root by default. With a root-owned container writing into a bind-mount, the resulting host files end up owned by root — and any subsequent local-backend terraform apply (or even a cat ~/.roksbnkctl/<ws>/state/terraform.tfstate) hits permission errors. The docker backend works around this by passing --user $(id -u):$(id -g) explicitly:

docker run --rm \
  --user 1000:1000 \                                     # host's caller-uid:caller-gid
  -v ~/.roksbnkctl/dev-tor/state:/state \
  --workdir /state/tf-source/embedded-terraform \
  hashicorp/terraform:1.5.7 \
  apply -auto-approve

The container process runs as the host user, so files written into the bind-mount are owned by the host user — same as a local-backend terraform apply would have produced. Switching backends mid-debug doesn’t strand state files behind a permission wall.

The UID/GID values are read from the host process at run time (Go’s os.Getuid() / os.Getgid()). On macOS this is mostly cosmetic — Docker Desktop’s VM normalises ownership on the host bind-mount automatically — but it’s required for clean Linux behaviour, so the backend always passes the flag.

Supported commands

The terraform docker backend honours --backend docker for the four lifecycle commands:

roksbnkctl up    --backend docker  [--var-file <path>] [--auto]
roksbnkctl plan  --backend docker  [--var-file <path>]
roksbnkctl apply --backend docker  [--var-file <path>] [--auto]
roksbnkctl down  --backend docker  [--var-file <path>] [--auto]

Flags that the local terraform backend honours (--var-file, --auto, plus the -w/--workspace selector) plumb through to the docker backend identically — the backend’s job is to spawn hashicorp/terraform:1.5.7 with the right argv; it doesn’t filter or rewrite the lifecycle commands’ flags. (--auto is roksbnkctl’s shorthand for terraform’s -auto-approve; the wrapper renames it for terseness and consistency across up/apply/down.)

roksbnkctl up --backend docker is the apply-with-auto-approve shorthand the existing local lifecycle uses; --backend docker switches the spawn target without changing the command shape.

Deferred: k8s and ssh terraform backends

--backend k8s and --backend ssh:<target> for terraform are not in v1.0. The blocker is state-handling: the local backend keeps state on the host filesystem, the docker backend bind-mounts the same path, but k8s (run terraform in a one-shot Job) and ssh:<target> (run terraform on a remote host) need a story for shipping state between the run vantage and the canonical workspace state dir. Designs under consideration include a versioned ConfigMap/Secret pair for k8s and an scp-pre-and-post atomic move for ssh; both are deferred to v1.x once the trade-offs have settled (see docs/PLAN.md §“What’s deliberately deferred to post-v1.0”).

PRD 03 §“State concerns” is the design spec; trying --backend k8s against terraform errors at parse time:

$ roksbnkctl up --backend k8s
error: terraform doesn't support backend `k8s` at v1.0 (state-handling design
       open; tracked in PRD 03 § State concerns); supported: local, docker

When to use it

  • You’re on a clean dev machine without ibmcloud installed and don’t want to install it.
  • You need a frozen tool version for CI reproducibility.
  • You’re debugging a “works on my machine” issue and want to factor out the host install variable.

When docker is the wrong call:

  • The tool needs network access that your laptop has but the container doesn’t (rare; default bridge networking usually preserves laptop’s egress).
  • You’re running iperf3 and want a network-locality benefit — docker doesn’t give you that vs local. Use k8s instead.
  • You’re running a DNS probe and want a different network vantage — same network identity as the host, no value-add. The DNS subcommand rejects --backend docker by design.
  • You’re on Windows. Linux/macOS docker daemons are in scope; Windows Docker Desktop coverage is deferred to a future round.

k8s backend

Runs the wrapped tool inside the cluster. Two distinct execution patterns share the same Backend.Run interface:

PatternUsed forLives inLifetime
Long-lived ops podad-hoc ibmcloud commands, future interactive shellsroksbnkctl-ops namespacemanually managed via roksbnkctl ops install/uninstall
One-shot Jobiperf3 client runs, future terraform runs, future DNS probesroksbnkctl-test namespaceper-invocation; auto-deleted after ttlSecondsAfterFinished: 60

The split mirrors the two latency budgets. Long-lived pods amortise the pod-startup cost across many invocations — perfect for ibmcloud iam oauth-tokens which you might run twenty times in a debugging session. One-shot Jobs are clean (no leftover state, no concurrency questions) — perfect for iperf3 -c <server> which runs once, emits its JSON, and exits.

Long-lived ops pod pattern

The pod is named roksbnkctl-ops in the roksbnkctl-ops namespace. roksbnkctl ops install deploys it (see Chapter 19 for the full lifecycle). The image bundles ibmcloud CLI plus kubectl as backup; future iterations may add oc, terraform, etc. The container inside the pod is named tools.

Backend.Run(ctx, argv, opts) for the ops-pod path is essentially:

exec, _ := remotecommand.NewSPDYExecutor(restConfig, "POST",
    clientset.CoreV1().RESTClient().Post().
        Resource("pods").Namespace("roksbnkctl-ops").Name("roksbnkctl-ops").
        SubResource("exec").
        VersionedParams(&corev1.PodExecOptions{
            Container: "tools",
            Command:   argv,
            Stdin:     opts.Stdin != nil,
            Stdout:    true,
            Stderr:    true,
            TTY:       opts.TTY,
        }, scheme.ParameterCodec).URL())
exec.StreamWithContext(ctx, remotecommand.StreamOptions{
    Stdin: opts.Stdin, Stdout: redactor(opts.Stdout, creds), Stderr: redactor(opts.Stderr, creds), Tty: opts.TTY,
})

The exit code comes back via the SPDY channel’s metav1.Status — the executor surfaces it as a exec.CodeExitError. We propagate that as the backend’s exit code, same as local propagates exec.ExitError.ExitCode().

opts.WorkDir is ignored for the ops pod path. The pod’s WorkingDir is fixed at container-spec time (/work); per-exec working-dir changes would require recreating the pod. Callers that need a specific cwd should cd <dir> && it into argv (the local backend’s symmetric escape hatch).

One-shot Job pattern

For each invocation, the backend builds a batchv1.Job spec, applies it, streams logs from the Job’s pod, waits for completion, reads the exit code from the pod’s container status, and lets ttlSecondsAfterFinished clean up.

Skeleton:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: roksbnkctl-iperf3-client-     # randomized; multiple runs don't collide
  namespace: roksbnkctl-test
spec:
  ttlSecondsAfterFinished: 60                  # auto-delete the Job + its Pod 60s after completion
  backoffLimit: 0                              # no retries; the test reports failure once and stops
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: iperf3-client
        image: ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<v>
        command: ["iperf3", "-c", "<server-svc>", "-J"]
        envFrom:
        - secretRef:
            name: roksbnkctl-job-creds-<random>   # projected per invocation
        volumeMounts:
        - name: files
          mountPath: /work
      volumes:
      - name: files
        projected:
          sources:
          - secret:
              name: roksbnkctl-job-files-<random>  # one Secret per invocation, holds RunOpts.Files

Three details to call out:

  1. Projected Secret for cred propagation. Credentials.IBMCloudAPIKey (when set) becomes a one-shot Secret, mounted via envFrom: secretRef. Per PRD 04 §“In-cluster pod” this beats argv (which would show in kubectl describe pod) and beats inline env: blocks (which surface in kubectl get pod -o yaml). The Secret carries the same ttlSecondsAfterFinished-equivalent lifecycle: when the Job’s ttlSecondsAfterFinished deletes the Job, the owning controller’s GC sweeps the Secret too via ownerReferences.
  2. Log streaming via client-go. Once the Job’s pod is in Running state, clientset.CoreV1().Pods(ns).GetLogs(name, &corev1.PodLogOptions{Follow: true}).Stream(ctx) returns an io.ReadCloser that we copy through the redactor into opts.Stdout. The stream stays open until the pod terminates or ctx cancels.
  3. Exit-code extraction. When the pod transitions to Succeeded or Failed, we read pod.Status.ContainerStatuses[0].State.Terminated.ExitCode and return that as the backend’s exit code. A Failed pod with ExitCode: 0 (rare; usually OOMKilled or evicted) maps to backend exit code 126 — backend mid-run failure rather than tool failure.

The roksbnkctl-test namespace is a fresh namespace dedicated to one-shot test workloads. It’s separate from roksbnkctl-ops (the long-lived pod’s home) so RBAC can be scoped tighter — see Chapter 19 §“RBAC”.

iperf3 server side

Worth calling out because it’s the asymmetric piece. The iperf3 test deploys a server bare Pod + Service into roksbnkctl-test, then runs the client as the one-shot Job described above:

SideResourceLifetime
Serverroksbnkctl-iperf3 bare Pod + Service (LoadBalancer for --mode north-south; ClusterIP for --mode east-west)torn down after the client Job completes
Clientone-shot JobttlSecondsAfterFinished: 60

The bare-Pod (rather than Deployment) shape is intentional — the iperf3 server is single-shot, scoped to one test, torn down on completion; the controller-managed replica machinery a Deployment provides is unused and would only confuse the cleanup story. Service type is driven by --mode: north-south measures laptop-to-cluster bandwidth and needs a publicly reachable endpoint (LoadBalancer); east-west measures node-to-pod and stays in-cluster (ClusterIP). See Chapter 22 — Throughput testing for the user-facing flag surface.

The client Job’s argv is iperf3 -c <server-cluster-ip-or-lb> -J. The -J JSON flows back via log streaming, parsed in internal/test/throughput.go, surfaced as roksbnkctl test throughput JSON output.

The server pod’s securityContext is set to satisfy OpenShift’s restricted-v2 SCC: runAsNonRoot: true, allowPrivilegeEscalation: false, seccompProfile.type: RuntimeDefault, capabilities.drop: [ALL]. iperf3 listens on port 5201 (unprivileged) so no root is needed. The manifest the k8s backend emits sets all four fields to satisfy the SCC.

When to use it

  • You’re running iperf3 and want a number that reflects cluster fabric, not your office Wi-Fi.
  • You’re running ibmcloud from a network that can reach the cluster but not *.cloud.ibm.com directly. The ops pod has both lines of sight; your laptop has only one.
  • You want a cluster-side ad-hoc shell for debugging — roksbnkctl exec --backend k8s -- bash (when implemented) drops into the ops pod.

When k8s is the wrong call:

  • The cluster doesn’t exist yet (roksbnkctl ops install requires a working kubeconfig). Use local or ssh for pre-cluster ops.
  • You haven’t run roksbnkctl ops install. Run it first; it’s a one-time setup per cluster.
  • You’re running terraform — --backend k8s for terraform is deferred to a future release pending a state-handling design (see PRD 03 §“State concerns”).

Chapter 19 is the full reference for the cluster-side mechanics: namespace, ServiceAccount, ClusterRole, Secret, lifecycle.

ssh backend

Runs the wrapped tool on a registered SSH target. Builds on the internal/remote.Client (the same SSH client backing the --on flag); this section assumes you’ve read Chapter 16 for the target-config and host-key TOFU framing.

roksbnkctl ibmcloud --backend ssh:jumphost ks cluster ls
roksbnkctl ibmcloud --backend ssh:bastion --bootstrap iam oauth-tokens

Per-tool apt-bootstrap and the --bootstrap flag

Before exec’ing the wrapped tool, the SSH backend probes whether it’s installed:

ssh <target> 'command -v <tool>'

Exit 0 → tool present, proceed. Non-zero → tool missing. What happens next depends on --bootstrap:

  • Without --bootstrap (the default). The backend errors with exit 127 and a clear message:

    error: tool `iperf3` not found on ssh target jumphost; re-run with --bootstrap to install via apt-get,
           or pre-install on the target manually
    

    No sudo apt-get ever runs. The backend won’t surprise the user with package-manager invocations or sudo password prompts on a remote they didn’t expect mutation on.

  • With --bootstrap. The backend runs the per-tool bootstrap recipe. For Ubuntu (the only OS supported this round), the recipe is roughly:

    # ibmcloud needs IBM's apt repo + GPG key first
    curl -fsSL https://download.clis.cloud.ibm.com/Linux/Ubuntu/repo.gpg | sudo apt-key add -
    echo 'deb https://download.clis.cloud.ibm.com/Linux/Ubuntu jammy main' \
      | sudo tee /etc/apt/sources.list.d/ibmcloud.list
    sudo -n apt-get update -y
    sudo -n apt-get install -y ibmcloud-cli
    

    iperf3 is simpler — no repo addition, just sudo -n apt-get install -y iperf3.

The opt-in default reflects PRD 03 open question §“--bootstrap opt-in for SSH”: silent sudo apt-get on a remote host is the kind of surprise that erodes operator trust, especially when the remote is shared between teams. Make the user say “yes, install for me”.

Bootstrap failure modes split between the two backend-failure exit codes per §“Backend-failure semantics”: 127 when we never got going (couldn’t reach the repo, no apt mapping, tool missing without --bootstrap); 126 when we got partway in and then something broke (sudo / OS-detect / install).

FailureExitWhat you see
--bootstrap not set and tool missing127“tool <name> not found on ssh target <target>; re-run with –bootstrap to install via apt-get, or pre-install on the target manually”
sudo requires a password (NOPASSWD not configured)126sudo: a password is required → “the SSH user needs passwordless sudo for apt-get install. Configure <user> ALL=(ALL) NOPASSWD: /usr/bin/apt-get in /etc/sudoers, or pre-install <pkg> manually.”
Non-Ubuntu OS (lsb_release -is doesn’t return Ubuntu)126“auto-install only supports Ubuntu. Pre-install <pkg> on the target (RHEL: yum install <pkg>).”
Network unreachable from target (apt-get can’t reach the repo)127“target can’t reach the package repo. Check the target’s egress policy or pre-install <pkg> manually.”
No apt mapping for the requested tool126“no bootstrap recipe known for tool <name>; the SSH backend only auto-installs ibmcloud + iperf3 today”

File materialisation

RunOpts.Files entries are written to a per-invocation tempdir on the remote. The tempdir is /tmp/roksbnkctl.<random>/ where <random> is a fresh 16-byte hex string per Run:

# pseudo-flow
ssh <target> 'mkdir -m 0700 /tmp/roksbnkctl.<rand>'
scp <local-temp>/<basename> <target>:/tmp/roksbnkctl.<rand>/<basename>
ssh <target> '
  trap "rm -rf /tmp/roksbnkctl.<rand>" EXIT
  cd /tmp/roksbnkctl.<rand>
  <argv...>
'

The trap … EXIT is shell-builtin; it fires on normal exit, on set -e failure, on SIGINT (Ctrl-C), on SIGTERM. So even if the user kills their roksbnkctl invocation mid-run, the remote tempdir is cleaned up by the wrapper script’s own trap before the SSH session terminates.

The 0700 mode on the tempdir ensures only the SSH user can read it during the brief on-disk window. On shared bastions (multi-user jumphosts) this matters — and it’s why we materialise to /tmp (which the user owns) rather than /var/tmp or some shared scratch path.

Kubeconfig follows the same pattern: Credentials.KubeconfigBytes becomes <tempdir>/kubeconfig, the wrapper exports KUBECONFIG=<tempdir>/kubeconfig, the trap removes the file on exit. PRD 04 §“Kubeconfig options for SSH backend” calls this “Option A” — scp-and-cleanup. We picked it over the in-memory <() process-substitution alternative because it’s robust across remote shells and sshd configs.

Env propagation: SetEnv vs wrapper script

OpenSSH supports two ways to pass an env var to a remote command:

  1. ssh -o SetEnv=KEY=VALUE target … — client tells the server “please add this to the env”. Works only if the server’s sshd_config has AcceptEnv KEY matching. Most stock sshd configs don’t enable AcceptEnv for arbitrary keys.
  2. Wrapper script with export KEY=VALUE — the script writes the env var into its own process before exec "$@". Works regardless of sshd config, but the value lives briefly in a 0700 file on the remote.

The SSH backend tries SetEnv first. On the first connect to a new target, it sends a sentinel env var (ROKSBNKCTL_SETENV_TEST=ok) and runs echo "$ROKSBNKCTL_SETENV_TEST". If the output is ok, SetEnv works on this target — the result is cached in workspace state, and subsequent runs use SetEnv directly.

If the sentinel doesn’t surface, sshd silently dropped it (it logs refused setenv request on the server side, but clients don’t see that). The backend falls back to a wrapper script:

#!/bin/sh
# /tmp/roksbnkctl.<rand>/wrap.sh, mode 0700, owner-readable only
trap 'rm -f "$0"' EXIT
set +o history
export IBMCLOUD_API_KEY='<value>'
exec "$@"

Then: ssh <target> /tmp/roksbnkctl.<rand>/wrap.sh ibmcloud iam oauth-tokens.

The wrapper-script path is the validator Issue 4 carry-over — the same shape --on uses for env passing today. Risks (file content includes the secret) are mitigated by:

  • Mode 0700 so only the SSH user can read.
  • set +o history so the value doesn’t leak into shell history.
  • trap 'rm -f "$0"' EXIT deletes the wrapper as soon as it exits — including on Ctrl-C, since the trap covers SIGINT/SIGTERM by virtue of being in the script’s main process.
  • The key is never in argv, so ps -ef on the remote doesn’t show it.

roksbnkctl targets show <name> reports which mechanism the target uses (e.g., env propagation: SetEnv (AcceptEnv ok) or env propagation: wrapper script (sshd refused SetEnv)) so users can choose to enable AcceptEnv server-side if they prefer.

Bootstrap failure modes (consolidated)

SymptomCauseRemediation
sudo: a password is requiredNOPASSWD sudo not configuredAdd <ssh-user> ALL=(ALL) NOPASSWD: /usr/bin/apt-get to /etc/sudoers.d/roksbnkctl on the target
auto-install only supports Ubuntu/etc/os-release ID is not ubuntuPre-install the tool manually; RHEL: sudo yum install <pkg>; Alpine: sudo apk add <pkg>
target can't reach the package repoTarget’s egress policy blocks download.clis.cloud.ibm.com (or upstream Ubuntu mirrors)Pre-install or open egress; doctor’s --backend ssh:<target> flags this
tool not found on ssh target …; re-run with --bootstrap--bootstrap not passed and tool missingRe-run with --bootstrap, or pre-install on the target

When to use it

  • You’re running ibmcloud from a customer-firewalled office where the corporate jumphost can reach IBM Cloud APIs but your laptop can’t.
  • You’re working in an air-gapped environment where roksbnkctl runs on your laptop but the IBM Cloud API conversations have to happen from a specific bastion’s IP.
  • You want a low-overhead remote-exec path that doesn’t require a cluster (the k8s backend’s prereq).

When ssh is the wrong call:

  • The target lacks the tool and you don’t want to mutate it. Skip --bootstrap; the backend errors clearly without installing anything.
  • The target isn’t Ubuntu and you don’t want to pre-install. Bootstrap won’t work; pre-install or use local/docker/k8s.
  • You’re running iperf3 to measure cluster bandwidth. SSH puts the client somewhere on the network path to the cluster but not necessarily adjacent to it — k8s is the right answer for that case.

Chapter 16 covers the lighter-weight --on jumphost predecessor that uses the same targets: config block. The SSH backend is the heavier-duty form: file materialisation, env propagation hardening, opt-in bootstrap. Chapter 18 is the decision tree.

The Backend interface

For the curious, the Go interface every backend conforms to:

package exec

type Backend interface {
    Run(ctx context.Context, argv []string, opts RunOpts) (int, error)
    Name() string
}

type RunOpts struct {
    Stdin           io.Reader
    Stdout, Stderr  io.Writer
    Env             []string         // KEY=VALUE pairs
    WorkDir         string           // best-effort; some backends ignore (k8s)
    TTY             bool             // request PTY where supported
    Files           map[string][]byte // files materialized at exec time
    Credentials     *Credentials     // routed via PRD 04's per-backend mechanism
}

type Credentials struct {
    KubeconfigBytes []byte
    IBMCloudAPIKey  string
}

All four implementations satisfy this interface. Call sites in cli/cluster.go, cli/test.go, etc., get a Backend from the registry and call Run(...) — no branching on backend type. The uniformity is what makes the system extensible without rewriting callers each time a backend lands.

The Credentials struct is the bridge between the resolver chain (env → keychain → config-b64 → prompt) covered in Chapter 14 and the per-backend propagation rules in PRD 04. Each backend translates the struct into the mechanism appropriate to where it runs: env vars for local, --env KEY (no =value) for docker, secretKeyRef for k8s, SetEnv or wrapper script for ssh.

Cross-references

Choosing a backend per tool

Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the Workspace struct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.

Chapter 17 covered the mechanics of each backend. This chapter is the decision tree: given a tool and a scenario, which of local / docker / k8s / ssh is the right call.

If you’re searching for “which backend should I use”, you’ve landed on the right page.

The four backends in one line each

BackendOne-line summaryDeep dive
localos/exec on your laptop; fastest, requires the tool on PATH§ Local backend
dockerdocker run against a vendored image; frozen tool version, no host install§ Docker backend
k8sinside the cluster (long-lived ops pod or one-shot Job); cluster-correct network identity§ K8s backend
ssh:<target>on a registered SSH target; opt-in apt-bootstrap on Ubuntu§ SSH backend

If you’re skimming, the cheat-sheet is:

  • local when you have the tool installed and the host’s network identity is correct for the call.
  • docker when you don’t have the tool and don’t want to install it, or you need a frozen version for CI.
  • k8s when the call’s network position matters and the cluster is the right vantage point.
  • ssh:<target> when the call needs to originate from a specific external host (a customer bastion, an air-gapped bridge).

Inside the all-in-one runner image, use local. When you run roksbnkctl from the runner container, every tool it dispatches is already on PATH in the image — so local is the right call and the docker/k8s backends (which fetch a tool from elsewhere) are redundant there (and docker would mean docker-in-docker). The decision tree below is for the binary-on-host case, where those backends earn their keep.

The rest of this chapter is the longer version.

Per-tool default backends

Every tool has a default backend baked into roksbnkctl. Workspace config (exec: block) can override the default per workspace; --backend overrides for a single invocation.

ToolDefaultResolved by
ibmcloudlocalinternal/cli/cluster.go::resolveBackendSpecWith("ibmcloud", flagOverride)
iperf3k8sinternal/cli/test.go::resolveBackendSpecWith("iperf3", flagOverride)
terraformlocalinternal/cli/lifecycle.go::resolveBackendSpecWith("terraform", flagOverride)

The defaults reflect “what’s the right answer for the most common scenario”:

  • ibmcloud defaults to local because most users have it on PATH or are happy installing it. The compliance + firewall scenarios where ssh or docker are better are the minority of calls.
  • iperf3 defaults to k8s because throughput from a laptop’s uplink isn’t the cluster’s bandwidth. The k8s backend places the iperf3 client in (or adjacent to) the cluster so the number reflects fabric, not Wi-Fi. Laptop-uplink-to-cluster is a real measurement too, but it’s the special case — opt in via --backend local.
  • terraform defaults to local because the terraform-exec local path is the established workflow. State handling is simplest there. Frozen-version CI runs use --backend docker; non-local network-locality use cases (cluster-side, SSH-bastion-side) are deferred to a future release pending a state-handling design — see PRD 03 §“State concerns”.

To change a default per workspace, use the backend command — no config.yaml hand-edit:

roksbnkctl -w acme backend set ibmcloud ssh:bastion
roksbnkctl -w acme backend set terraform docker
roksbnkctl -w acme backend show              # effective backend per tool + where it came from
roksbnkctl -w acme backend unset terraform   # revert to the built-in default

That writes the exec: block for you:

exec:
  ibmcloud:  { backend: ssh:bastion }
  terraform: { backend: docker }

Chapter 12 §“exec:” covers the schema. The --backend CLI flag overrides whatever is in exec: for a single invocation.

Per-tool supported-backend matrix

Not every tool supports every backend. The authoritative matrix at v1.0:

Toollocaldockerk8sssh:<target>
ibmcloudyes (default)yes (frozen image)yes (long-lived ops pod)yes
iperf3yes (opt-in: laptop vantage)not supported (same network identity as local)yes (default)yes
terraformyes (default)yes (frozen image)deferred to v1.x (state-file handling)deferred to v1.x (state-file handling)
DNS probeyes (default for laptop vantage)not supported (same network identity as local)yes (cluster vantage)yes (remote vantage)
kubectl / ocinternalised — runs via the Go client, not via a host binaryn/an/an/a
diginternalised — DNS probe replaces dig for in-tree workn/an/an/a

Legend:

  • yes — supported; same surface command works on this backend.
  • yes (default) — this backend is the per-tool default; pass --backend other to override.
  • not supported — rejected at CLI parse time with a clear error pointing at the right alternative.
  • deferred to v1.x — a real design constraint, not a gap; see the cell text and PRD 03 §“State concerns”.
  • internalised — roksbnkctl performs the operation via its own embedded library, not by shelling out; no backend selection applies.

The “no” entries are intentional design decisions, not gaps:

  • iperf3 over docker is rejected because a Docker container running locally has the same network identity as the host — same NAT egress, same uplink, same observed bandwidth as --backend local. The user’s mental model would be “I picked docker, so the iperf3 must be hermetic now” but the throughput number wouldn’t actually differ. Better to refuse and force the user to pick local (deliberate laptop measurement) or k8s (cluster measurement).
  • DNS probe over docker is rejected for the same reason. DNS resolution from a Docker container with default bridge networking goes through the same resolver as the host. There’s no GSLB-relevant network-locality difference. The probe subcommand errors with “DNS probe doesn’t benefit from docker; use local instead” when --backend docker is passed.
  • terraform over k8s and ssh is deferred to v1.x. The state file is sensitive (admin tokens, generated TLS keys, license bundles); moving it into a Kubernetes Secret or scp’ing it pre/post-run requires a state-handling design that hasn’t shipped yet. PRD 03 §“State concerns” lays out the considerations; the roadmap entry lives in docs/PLAN.md §“What’s deliberately deferred to post-v1.0”.

Passing an unsupported (tool, backend) pair errors at the CLI layer before the backend is invoked:

$ roksbnkctl test throughput --backend docker
error: iperf3 doesn't support backend `docker` (same network identity as `local`,
       no value-add); supported: local, k8s, ssh:<target>

Decision tree

Pick the question that matches your scenario.

“I want to measure cluster bandwidth”

Use --backend k8s. The default for iperf3 is already k8s — the explicit flag is redundant unless you’ve overridden the default in workspace config:

roksbnkctl test throughput
# equivalent to:
roksbnkctl test throughput --backend k8s

The k8s backend deploys a server-side Deployment + LoadBalancer Service in roksbnkctl-test, runs the iperf3 client as a one-shot Job in the same namespace, collects the JSON output from the client pod’s logs, and tears down both. The bandwidth number reflects the cluster fabric.

If you instead want to measure your laptop’s uplink to the cluster:

roksbnkctl test throughput --backend local --endpoint <cluster-LB-ip>:5201

That’s a deliberately different measurement — useful when you suspect office Wi-Fi, not cluster fabric, is the bottleneck.

“I’m doing GSLB DNS validation”

Use both local and k8s. F5 BIG-IP Next’s GSLB returns different answers depending on the requesting resolver’s IP — geographic affinity, datacenter routing, health-check state. To validate that the GSLB is actually doing this, query from multiple network vantage points and compare.

The multi-vantage probe ships at v1.0 via roksbnkctl test dns --gslb-compare:

roksbnkctl test dns \
  --target www.example.com \
  --type A \
  --server gslb-vip.f5.example.com \
  --gslb-compare

--gslb-compare fans out to every configured vantage (local for your office IP, k8s for the cluster’s egress IP, ssh:<region-bastion> for a bastion in another region) in parallel and emits a single comparison JSON with a gslb_divergence boolean. Different answers across vantages are expected in a healthy GSLB; identical answers might mean the GSLB rules aren’t taking effect for the resolver positions you queried from.

Chapter 21 — DNS testing for GSLB is the full reference.

“I need to run ibmcloud from a customer-firewalled office”

Use --backend ssh:<bastion>. Your customer’s network policy lets the corporate jumphost reach *.cloud.ibm.com but blocks your laptop. The SSH backend ships your kubeconfig to the bastion (single file, mode 0600, removed via trap on session exit), runs ibmcloud there, streams the output back:

roksbnkctl ibmcloud --backend ssh:bastion ks cluster ls

If ibmcloud isn’t installed on the bastion, you’ll get a clear error:

error: tool `ibmcloud` not found on ssh target bastion; re-run with --bootstrap to install
       via apt-get, or pre-install on the target manually

Re-run with --bootstrap if you want roksbnkctl to sudo apt-get install -y ibmcloud-cli on the bastion. The opt-in default reflects “we don’t surprise users with sudo apt-get on a remote they didn’t expect mutation on” — see Chapter 17 §“SSH backend” for the bootstrap mechanics.

“I’m in CI and want a frozen toolchain version”

Use --backend docker. The vendored images are tagged in lock-step with roksbnkctl releases — ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:v1.0.0 is the exact same ibmcloud binary every CI run sees, regardless of when the runner image was built or what apt-get happens to ship that day:

roksbnkctl ibmcloud --backend docker iam oauth-tokens
roksbnkctl up --backend docker     # terraform inside hashicorp/terraform:<v>

For CI specifically, also pin ibmcloud.api_key_source: env in workspace config so the API key resolution is unambiguous (no keychain fallback to confuse a non-interactive runner) — see Chapter 14 §“Pinning a single source”.

“I’m on a clean dev machine without ibmcloud installed”

Use --backend docker. No apt-get install ibmcloud-cli, no IBM repo + GPG key dance, no upstream-package-version mismatch — docker pull ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:dev is the only setup, and roksbnkctl does that for you on first invocation.

Alternatively, if your laptop is the dev machine and you’ll run ibmcloud more than once, just install it. The local backend has lower per-invocation startup latency than docker (no container create/start/log-attach), so once you’ve paid the install cost the local path is faster for the rest of the session.

“I want a cluster-side ad-hoc shell”

Use --backend k8s with the long-lived ops pod. Once roksbnkctl ops install has run, --backend k8s for ibmcloud (or any future tool) routes through kubectl exec -n roksbnkctl-ops ops -- <argv>. The pod stays alive between invocations, so the second and subsequent commands skip pod-startup latency.

roksbnkctl ops install
roksbnkctl ibmcloud --backend k8s iam oauth-tokens
roksbnkctl ibmcloud --backend k8s ks cluster ls
roksbnkctl ibmcloud --backend k8s account list

Chapter 19 is the full reference for the ops pod lifecycle.

“I’m pre-cluster — there’s no cluster yet”

Use local or ssh:<target>. The k8s backend prereq is a working kubeconfig pointing at a running cluster; before roksbnkctl up has succeeded, that doesn’t exist. For pre-cluster ibmcloud + terraform calls (account inspection, IAM tinkering, the cluster-create itself), local and ssh:bastion are the only two paths.

When not to use a backend

Common foot-guns, in rough order of how often they come up:

--backend k8s without roksbnkctl ops install

The ops pod must exist before the k8s backend can route ibmcloud calls through it. First-time use:

roksbnkctl ops install         # one-time setup per cluster
roksbnkctl ibmcloud --backend k8s ks cluster ls

If you skip the install, the backend errors with a clear remediation:

error: ops pod not found in roksbnkctl-ops namespace; run `roksbnkctl ops install` first

Chapter 19 covers the install/show/uninstall lifecycle.

--backend docker for a network-locality test

iperf3 and the DNS probe both reject --backend docker because a local Docker container has the same network identity as the host (default bridge networking). The probe wouldn’t measure anything different. The CLI errors at parse time:

$ roksbnkctl test throughput --backend docker
error: iperf3 doesn't support backend `docker` (same network identity as `local`,
       no value-add); supported: local, k8s, ssh:<target>

If you actually want a hermetic-tools throughput test, --backend k8s is the right answer.

--backend ssh:host without --bootstrap on a fresh target

If ibmcloud (or iperf3) isn’t installed on the target, the SSH backend won’t silently sudo apt-get for you — --bootstrap is opt-in. The first call on a fresh target tells you exactly what’s needed:

error: tool `ibmcloud` not found on ssh target bastion; re-run with --bootstrap to install
       via apt-get, or pre-install on the target manually

Re-run with --bootstrap if mutation is OK; otherwise pre-install via your config-management of choice (Ansible, Salt, baked-in-image).

--backend ssh:host to a non-Ubuntu target with --bootstrap

The apt-bootstrap recipe is Ubuntu-only this round. RHEL / CentOS / Alpine targets need pre-installation via yum / dnf / apk — --bootstrap errors out cleanly:

error: auto-install only supports Ubuntu. Pre-install `ibmcloud-cli` on the target
       (RHEL: `yum install ibmcloud-cli`)

Once the tool is installed, --backend ssh:host works without --bootstrap.

--backend k8s for terraform

Deferred to v1.x. The terraform tool’s k8s + ssh backends require a state-handling design that hasn’t shipped — moving the state file into a Kubernetes Secret or scp’ing it pre/post-run is fiddly enough to be a feature in its own right (PRD 03 §“State concerns”; roadmap in docs/PLAN.md §“What’s deliberately deferred to post-v1.0”). For now, terraform supports local and docker only. If the network-locality use case (running terraform from a customer VPC for IP-egress reasons) is blocking, file an issue.

Mixing --on and --backend ssh:<target>

--on <target> is the Chapter 16 lightweight remote-exec — it runs the passthrough shape (exec, shell, kubectl, oc, ibmcloud) on the target by literally re-running the command via SSH. --backend ssh:<target> is the heavier-duty form — it routes through the Backend interface, which means file materialisation, env propagation hardening, opt-in apt-bootstrap, and the redactor are all wired in.

You generally want one or the other, not both. The supported precedence is “--backend ssh:<target> wins”; passing both flags on the same invocation surfaces a warning. If you’re calling roksbnkctl ibmcloud …, prefer --backend ssh:<target> for the same target — you get the better cred-handling story automatically.

Workspace config + --backend flag interaction

Recap of Chapter 12 §“exec:”:

The flag wins. If ~/.roksbnkctl/<ws>/config.yaml says:

exec:
  iperf3: { backend: k8s }

…and you run roksbnkctl test throughput --backend local, the local backend runs. The flag is the per-invocation override; the workspace config is the per-workspace default.

If neither is set, the per-tool default from the previous section applies (iperf3 → k8s, ibmcloud → local, terraform → local). The resolution order is exact:

  1. --backend flag
  2. exec.<tool>.backend in workspace config
  3. Per-tool baked-in default

There’s no fallback chain inside this resolution — if you pass --backend k8s and the cluster is unreachable, the backend errors with “cluster API unreachable” (exit 127). It does not fall through to local. Silent fallback hides intent and produces confusing CI results; the failure-mode discipline in Chapter 17 §“Backend-failure semantics” applies here too.

Summary table

The decision-tree contents collapsed into one table:

If you want to…BackendNotes
Measure cluster bandwidthk8siperf3 client + server in cluster (the default)
Measure laptop-uplink-to-cluster bandwidthlocaldeliberate; not the iperf3 default
GSLB DNS cross-vantage comparelocal + k8s (--gslb-compare)multiple vantages in parallel
ibmcloud from a customer-firewalled officessh:bastionwith --bootstrap if first call on fresh Ubuntu
Frozen-version CI for any tooldockerimage tag matches roksbnkctl release
Cluster-side ad-hoc ibmcloud debuggingk8srequires roksbnkctl ops install first
Pre-cluster ibmcloud / terraformlocal or sshk8s requires a working cluster
terraform up on a clean dev machinelocal (default) or dockerk8s + ssh deferred
Air-gapped: laptop can’t reach IBM Cloud, bastion canssh:bastionwith kubeconfig propagation
Just learning the toollocalsimplest mental model

Worked example: bare-metal + jumphost office workflow

End-to-end Part V scenario: you’re an F5 SE running a customer POC from a corporate-firewalled office. The laptop can’t reach *.cloud.ibm.com directly (the office proxy blocks it) but a customer-provisioned Ubuntu jumphost at 10.20.30.40 can. The jumphost was already auto-discovered by an earlier roksbnkctl up against this customer’s account, so targets list shows it. You need to: install the in-cluster ops pod, run ibmcloud from the bastion, and run a throughput test from inside the cluster — all without installing tools locally.

# 1. Verify jumphost is registered + reachable
$ roksbnkctl targets list -w customer
NAME       HOST          KEY_SOURCE         STATUS
jumphost   10.20.30.40   workspace/state    reachable

# 2. Run ibmcloud from the jumphost (Sprint 1 --on flag, lightweight)
$ roksbnkctl ibmcloud --on jumphost ks cluster ls
OK
Name              ID                                     State    Created     ...
customer-cluster  c4abc123def456                         normal   3 days ago  ...

# 3. For the same call routed through the Backend interface (cred-handling
# hardened, redactor wired, opt-in apt-bootstrap available), use --backend
$ roksbnkctl ibmcloud --backend ssh:jumphost ks cluster ls
# Same output; different code path. Prefer --backend ssh:<target> for
# everything except quick interactive shells where --on is faster to type.

# 4. Install the in-cluster ops pod (one-time per cluster)
$ roksbnkctl ops install
✓ Namespace roksbnkctl-ops created
✓ ServiceAccount + Role + RoleBinding applied
✓ Secret roksbnkctl-ibm-creds applied (envFrom secretRef)
✓ Pod roksbnkctl-ops Running (2.3s)

# 5. Same ibmcloud call routed through the ops pod (k8s backend)
$ roksbnkctl ibmcloud --backend k8s iam oauth-tokens
IAM token: Bearer eyJ...
# (The token comes from inside the cluster — different egress IP from the
# jumphost's, useful when IAM policy is IP-conditional.)

# 6. Throughput test using the cluster vantage (default for iperf3)
$ roksbnkctl test throughput
→ Deploying iperf3 server pod into namespace "roksbnkctl-test"
✓ Pod ready (iperf3-server-...)
→ Deploying iperf3 client Job in the same namespace
✓ Client Job complete
✓ throughput: 8.92 Gbits/sec (mean over 10s)
→ Tearing down iperf3 fixture
✓ pod, service, and Job deleted

# 7. Throughput test from the jumphost into the cluster (north-south, real
# customer-network bandwidth — not laptop wifi)
$ roksbnkctl test throughput --backend ssh:jumphost --mode north-south
✓ throughput: 936 Mbits/sec  (jumphost → cluster LB; customer's WAN)

# 8. Persist the per-tool routing in workspace config (one-time)
$ cat >> ~/.roksbnkctl/customer/config.yaml <<'YAML'
exec:
  ibmcloud:  { backend: ssh:jumphost }
  iperf3:    { backend: k8s }
  terraform: { backend: local }
YAML
# Subsequent runs skip the --backend flag — every ibmcloud call routes via
# the jumphost automatically; every iperf3 runs in-cluster.

The point of this walkthrough: with no tools installed locally beyond roksbnkctl itself, you’ve reached the IBM Cloud control plane from the customer’s bastion (compliance-correct egress), exercised the cluster fabric for throughput, and persisted the routing per workspace. The same laptop with the same roksbnkctl binary handles a different customer’s POC by pointing at a different workspace; nothing on the laptop is workspace-specific.

Chapter 19 covers the ops-pod lifecycle in detail; Chapter 22 covers the north-south vs east-west modes.

Cross-references

The in-cluster ops pod

The k8s execution backend has two execution patterns: a long-lived ops pod for ad-hoc commands, and one-shot Jobs for throughput tests, DNS probes, and other per-invocation workloads. Chapter 17 §“K8s backend” covered the interface mechanics — how Backend.Run dispatches into either pattern.

This chapter is the reference for the pod itself: what roksbnkctl ops install deploys, what RBAC it grants, where credentials live, how to rotate them, and how to debug when something goes wrong.

If you’ve never run roksbnkctl ops install, you can read this chapter front-to-back; otherwise the § Operability section near the end is the troubleshooting jump-off point.

What the ops pod is

A long-lived pod in the roksbnkctl-ops namespace, running an image bundled with the tools roksbnkctl may want to invoke cluster-side: ibmcloud CLI plus kubectl as a fallback, with oc and terraform reserved for future iterations.

The pod sits idle waiting for kubectl exec calls. Each roksbnkctl ibmcloud --backend k8s … invocation routes through client-go’s SPDY executor, runs the wrapped tool inside the existing pod, streams stdout/stderr back, and returns the exit code. No pod create/start latency between invocations — a session of twenty ibmcloud commands pays the startup cost once.

Compared to the one-shot Job pattern (used for iperf3 and the DNS probe), the ops pod trades a bit of resource-usage idle-state for substantially lower per-call latency. It’s the right shape when you want to debug interactively or run many small commands.

roksbnkctl ops install

Idempotent setup. Run once per cluster; re-run any time you want to refresh the image, rotate the API key Secret, or recover from a partial uninstall.

roksbnkctl ops install

What it does, step by step:

1. Create the namespace

apiVersion: v1
kind: Namespace
metadata:
  name: roksbnkctl-ops
  labels:
    app.kubernetes.io/name: roksbnkctl
    app.kubernetes.io/component: ops-pod

The roksbnkctl-ops namespace is dedicated to the long-lived pod. Separate from roksbnkctl-test (where one-shot Jobs run) so RBAC can be scoped per namespace — see § RBAC below.

2. Create the ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: roksbnkctl-ops
  namespace: roksbnkctl-ops

The pod runs as this SA. Its projected token is auto-mounted at /var/run/secrets/kubernetes.io/serviceaccount/, which is what the bundled kubectl uses for in-cluster authentication. The IBM Cloud API key (a separate credential) reaches the pod through a Kubernetes Secret — see § Credential propagation below.

3. Create the ClusterRole + ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: roksbnkctl-ops
rules:
- apiGroups: [""]
  resources: ["pods", "pods/exec", "pods/log"]
  verbs:     ["get", "list", "watch", "create", "delete"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs:     ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs:     ["get", "list"]
  resourceNames: ["roksbnkctl-ibm-creds"]
- apiGroups: [""]
  resources: ["services"]
  verbs:     ["get", "list", "create", "delete"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs:     ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs:     ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: roksbnkctl-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: roksbnkctl-ops
subjects:
- kind: ServiceAccount
  name: roksbnkctl-ops
  namespace: roksbnkctl-ops

The full manifest lives at internal/exec/k8s_install.yaml (embedded into the binary). § RBAC walks through what each rule is for.

4. Create or update the credential Secret

Note. This step describes the static-key Secret applied under --trusted-profile=off and under the auto-fallback when IAM perms don’t allow the trusted-profile path. Under --trusted-profile=auto success the Secret is still applied with empty data fields (placeholder so envFrom: secretRef always resolves); the cred propagation happens via the trusted-profile annotation on the SA + the IAM_PROFILE_ID env var instead. See §“Trusted-profile flow” below.

apiVersion: v1
kind: Secret
metadata:
  name: roksbnkctl-ibm-creds
  namespace: roksbnkctl-ops
  annotations:
    helm.sh/resource-policy: keep            # don't sweep on accidental destroy
type: Opaque
stringData:
  IBMCLOUD_API_KEY: <resolved-key-value>

The key value comes from the workspace’s resolver chain (env → keychain → config-b64 → prompt) — see Chapter 14 for the resolution order. The Secret carries two keys (IBMCLOUD_API_KEY and the legacy alias IC_API_KEY) both populated from the same resolved value, so older ibmcloud CLI versions that look for the IC_ name find it.

If the Secret already exists (re-running ops install after a key rotation), roksbnkctl does a client-side Get + Update: the Secret’s data is overwritten with the freshly resolved value, the roksbnkctl.io/rotated-at annotation is stamped with the current timestamp, and the rest of the Secret’s metadata is left untouched. roksbnkctl ops show surfaces last cred rotation: <timestamp> by reading that annotation.

5. Create the Pod

apiVersion: v1
kind: Pod
metadata:
  name: roksbnkctl-ops
  namespace: roksbnkctl-ops
  labels:
    app: roksbnkctl-ops
    roksbnkctl.io/managed: "true"
spec:
  serviceAccountName: roksbnkctl-ops
  restartPolicy: Always
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: tools
    image: ${OPS_IMAGE}                  # resolved from roksbnkctl's version at install time
    imagePullPolicy: IfNotPresent
    command: ["sleep", "infinity"]
    envFrom:
    - secretRef:
        name: roksbnkctl-ibm-creds
    securityContext:
      allowPrivilegeEscalation: false
      runAsNonRoot: true
      capabilities:
        drop: ["ALL"]
    resources:
      requests: { cpu: 50m,  memory: 64Mi }
      limits:   { cpu: 500m, memory: 256Mi }

Three details to call out:

  • command: ["sleep", "infinity"] — the pod’s own command. Each Backend.Run invocation issues a kubectl exec against this idle process, which means the pod’s main process never exits as long as the pod is healthy.
  • securityContext is set explicitly for OpenShift’s restricted-v2 SCC. Pod-level runAsNonRoot + seccompProfile.type: RuntimeDefault; container-level allowPrivilegeEscalation: false + capabilities.drop: [ALL] + runAsNonRoot — the same fields the iperf3 server pod sets, for the same reason.
  • envFrom: secretRef — the API key reaches the pod’s env without ever touching the pod manifest’s argv or env: block. kubectl describe pod roksbnkctl-ops shows the secret reference name but not the value, per PRD 04 §“In-cluster pod”.

6. Wait for readiness

roksbnkctl ops install waits for Pod.Status.Phase == Running and the container’s Ready condition before returning. Default timeout is 60 seconds; longer for clusters with slow image pulls (the ghcr.io image is ~80 MiB). Failures surface a kubectl describe pod roksbnkctl-ops excerpt for context.

Trusted-profile flow

The static-key Secret described above is the fallback. The default ops install invocation auto-provisions an IBM Cloud IAM trusted profile linked to the ops pod’s ServiceAccount, so the static API key no longer needs to land in any Kubernetes Secret. PRD 04 → “Trusted-profile auto-provisioning (k8s backend)” is the design reference; this section is the operational walkthrough.

ops install --trusted-profile=auto provisions the profile, and the in-pod ibmcloud login wrap detects the SA’s trusted-profile annotation and authenticates via the projected SA token at runtime.

roksbnkctl ops install --trusted-profile=auto

--trusted-profile=auto is the default — running roksbnkctl ops install with no flag picks the auto path. Naming the flag explicitly is useful in scripts that pin behaviour or in docs that want to read unambiguously:

$ roksbnkctl ops install --trusted-profile=auto
✓ Provisioned IAM trusted profile roksbnkctl-ops-canada-roks (iam-Profile-9f2…)
✓ created namespace roksbnkctl-ops
✓ created sa roksbnkctl-ops/roksbnkctl-ops
✓ created secret roksbnkctl-ops/roksbnkctl-ibm-creds
✓ created clusterrole roksbnkctl-ops
✓ created crb roksbnkctl-ops
✓ created pod roksbnkctl-ops/roksbnkctl-ops
→ Waiting for ops pod to be Ready (60s timeout)
✓ Ops pod is Ready (trusted profile roksbnkctl-ops-canada-roks)

Re-runs against an existing install emit updated <kind> … / <kind> … exists instead of created for each resource that already matches the desired state. The trusted-profile provisioning line above is the single line internal/cli/ops.go emits for the whole IBM IAM-side flow (perm probe + profile create + compute-resource link + SA annotation) — the work happens silently inside resolveTrustedProfileForInstall; the one line you see is the receipt.

What just happened, in order (the binary doesn’t narrate these steps but they’re what’s actually going on):

  1. IAM perm probe. ops install calls IBM IAM Identity to confirm the resolved API key has iam-identity perms. On 403, the flag value drives the next step: auto falls back to the static-key Secret with a warning (see §“--trusted-profile=auto falling back” below); on errors out with a non-zero exit.
  2. Profile creation. Names the profile roksbnkctl-ops-<workspace> so multiple workspaces against the same IBM Cloud account don’t race for a single shared name. The compute-resource link binds the profile to your cluster’s OIDC issuer URL + the roksbnkctl-ops/roksbnkctl-ops ServiceAccount specifically — other SAs on the same cluster can’t assume the profile.
  3. Policy attachment. No default policies are attached — the profile inherits whatever IAM policies your account has set up for trusted profiles in general (typically nothing, until you grant). A future cycle will surface ibmcloud.trusted_profile.policies as a workspace-config block; tracked under v1.x deferred. If you need the profile to actually authorise specific actions (Container Registry pulls, Cloud Object Storage reads), grant the policies via IBM Cloud Console or ibmcloud iam trusted-profile-policy-create after ops install returns.
  4. SA annotation. The ServiceAccount gets iam.cloud.ibm.com/trusted-profile: roksbnkctl-ops-<workspace> plus the roksbnkctl.io/trusted-profile-managed: "true" marker that signals ops uninstall to delete the profile during cleanup.
  5. Pod creation. The pod’s container always has envFrom: secretRef: roksbnkctl-ibm-creds; what changes between modes is the Secret’s contents. Under --trusted-profile=auto success the Secret is created with empty data — IBMCLOUD_API_KEY is the empty string — plus an extra IAM_PROFILE_ID env var pointing at the provisioned profile’s ID, and a projected ServiceAccount-token volume mounted at /var/run/secrets/tokens/token (audience iam) so the pod has a cluster-issued JWT the IBM IAM endpoint will accept. The in-pod ibmcloud login wrap detects the SA’s trusted-profile annotation and runs ibmcloud login -a https://cloud.ibm.com --cr-token @/var/run/secrets/tokens/token --profile "$IAM_PROFILE_ID" -r "${IBMCLOUD_REGION:-us-south}" --quiet — the --cr-token @<path> form reads the projected SA token from disk; IBM IAM validates that JWT against the trusted profile’s ROKS_SA compute-resource link (the link internal/ibm/trusted_profile.go::ensureLink provisions). The static API key never transits the pod env. Under --trusted-profile=off (or the auto-fallback) the Secret carries the resolved API key, no projected token volume is mounted, and the wrap runs ibmcloud login --apikey "$IBMCLOUD_API_KEY".

Verifying the profile is in use

The ServiceAccount carries the truth-of-record annotation:

$ oc get serviceaccount roksbnkctl-ops -n roksbnkctl-ops -o yaml
# or, kubectl-equivalent via the bundled passthrough:
$ roksbnkctl k get sa roksbnkctl-ops -n roksbnkctl-ops -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.cloud.ibm.com/trusted-profile: Profile-ccba11f2-3b1f-4b1a-b8a4-aeed2b7b3320  # ← the IBM IAM Profile ID
    roksbnkctl.io/trusted-profile-managed: "true"                                     # ← ops uninstall will delete it
    roksbnkctl.io/provisioned-at: "2026-05-13T14:08:33Z"
  name: roksbnkctl-ops
  namespace: roksbnkctl-ops

End-to-end smoke test of the runtime cred flow:

$ roksbnkctl --backend k8s ibmcloud iam oauth-tokens
IAM token:  Bearer eyJ…

The token is fresh-each-call: the in-pod wrap detects the SA’s iam.cloud.ibm.com/trusted-profile annotation, trades the pod’s projected SA token for an IAM token against the trusted profile, and returns it to the caller. No static API key transits the pod env.

The first invocation may take 30–60 seconds after ops install returns because IBM IAM needs to pick up the cluster’s OIDC issuer URL before it will accept the projected SA token as proof for the profile. The wrap absorbs this with a 3-attempt × 20s-backoff retry — up to ~40s of waiting inside the wrap before it surfaces the failure. On triple-fail the wrap prints trusted-profile login failed after 3 attempts: <captured-stderr> to your terminal (the captured stderr will include the underlying ibmcloud login diagnostic — typically the “Unable to authenticate” / FAILED banner shape). If your first smoke test produces that line, give IAM a few more seconds and re-run. After the first successful call the wrap’s auth state is cached for the pod’s lifetime, so subsequent roksbnkctl --backend k8s ibmcloud <subcommand> invocations don’t re-pay the propagation window.

--trusted-profile=auto falling back

auto falls back to the static-key Secret when any of three pre-conditions for trusted-profile provisioning aren’t met. The warning prints first (the fallback decision is made before any cluster-side resource is applied), then the rest of the install proceeds with the static-key shape:

$ roksbnkctl ops install
warning: IAM perm 'iam-identity' missing; using static-key Secret. Pass `--trusted-profile=off` to silence.
✓ created namespace roksbnkctl-ops
✓ created sa roksbnkctl-ops/roksbnkctl-ops
✓ created secret roksbnkctl-ops/roksbnkctl-ibm-creds
✓ created clusterrole roksbnkctl-ops
✓ created crb roksbnkctl-ops
✓ created pod roksbnkctl-ops/roksbnkctl-ops
→ Waiting for ops pod to be Ready (60s timeout)
✓ Ops pod is Ready (static-key Secret)

The three warning shapes (in source order — internal/cli/ops.go resolveTrustedProfileForInstall):

TriggerWarning
Workspace has no registered cluster yet (cluster-outputs.json missing — run cluster up or cluster register first)warning: trusted-profile mode 'auto' needs a registered cluster (<err>); falling back to static-key Secret. Pass –trusted-profile=off to silence.
Registered cluster lookup against the IBM Cloud API failed (network, key auth, cluster deleted out-of-band)warning: trusted-profile mode 'auto' couldn't look up cluster (<err>); falling back to static-key Secret. Pass –trusted-profile=off to silence.
API key lacks IAM iam-identity permission (the most common fallback)warning: IAM perm 'iam-identity' missing; using static-key Secret. Pass –trusted-profile=off to silence.

All three are non-fatal; the install completes and the pod works normally. The warnings are terse on purpose — the actionable detail belongs in this chapter, not in every stderr line. Three ways to clear them permanently:

  • Run cluster up or cluster register first if the warning names the missing cluster registration. ops install re-run after the registration completes will detect the cluster and switch to the trusted-profile path.
  • Ask your IAM admin to grant iam-identity Operator role on the API key (or use a different key that already has it) if the warning names the missing IAM perm. Re-run ops install — the install detects the changed perm posture on re-run and replaces the static-key Secret with a trusted-profile binding.
  • Opt out via --trusted-profile=off (next subsection) if you don’t want the warning every install.

--trusted-profile=off

Explicit opt-out. Skips the IAM perm check entirely and provisions the static-key Secret:

$ roksbnkctl ops install --trusted-profile=off
✓ created namespace roksbnkctl-ops
✓ created sa roksbnkctl-ops/roksbnkctl-ops
✓ created secret roksbnkctl-ops/roksbnkctl-ibm-creds
✓ created clusterrole roksbnkctl-ops
✓ created crb roksbnkctl-ops
✓ created pod roksbnkctl-ops/roksbnkctl-ops
→ Waiting for ops pod to be Ready (60s timeout)
✓ Ops pod is Ready (static-key Secret)

Use cases:

  • Byte-for-byte parity tests — for scripts whose assertions match a static-key ops show output verbatim, or parity tests against an existing deployment.
  • Air-gapped clusters that can’t reach the IBM IAM API at runtime — without that connectivity, the pod can’t trade its projected SA token for an IAM token, so the trusted-profile path is non-functional regardless of perms.
  • Cred rotation runbooks that already automate the static-key path and aren’t yet ready to switch to the projected-token model.

The third value, --trusted-profile=on, is the inverse — it forces the trusted-profile path and refuses to fall back on perm-missing, returning a non-zero exit with the same warning text. Use it in CI to surface IAM-perm regressions explicitly.

Cleanup on ops uninstall

roksbnkctl ops uninstall --confirm honors the roksbnkctl.io/trusted-profile-managed: "true" annotation on the SA and deletes the IBM Cloud trusted profile alongside the cluster-side objects:

$ roksbnkctl ops uninstall --confirm
✓ deleted trusted profile roksbnkctl-ops-canada-roks
✓ deleted pod roksbnkctl-ops
✓ deleted secret roksbnkctl-ibm-creds
✓ deleted serviceaccount roksbnkctl-ops
✓ deleted clusterrolebinding roksbnkctl-ops
✓ deleted clusterrole roksbnkctl-ops
✓ deleted namespace roksbnkctl-ops
✓ deleted namespace roksbnkctl-test

The trusted profile is deleted first, before the cluster-side objects, so even if the cluster API becomes unreachable mid-uninstall the IBM Cloud-side state isn’t left orphaned. The Secret is always deleted regardless of mode (it’s always rendered by ops install, just with empty data under --trusted-profile=auto success).

The trusted-profile delete is best-effort — if the calling user’s API key has lost iam-identity perms in the meantime (or the key itself has rotated and the new key doesn’t have those perms), the cluster-side objects still delete and a warning line is printed instructing the user to delete the profile manually via the IBM Cloud console. The annotation remains correct documentation of what was provisioned; roksbnkctl ops install on a fresh cluster will pick a fresh profile name unconditionally so an orphaned profile from a prior install doesn’t collide.

--trusted-profile=off installs leave no trusted profile to clean up — ops uninstall --confirm just deletes the cluster-side objects + the static-key Secret.

roksbnkctl ops show

Reports current state without making any changes:

$ roksbnkctl ops show
namespace:    roksbnkctl-ops
pod:          roksbnkctl-ops
phase:        Running
ready:        true
image:        ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:v0.9.0
rbac subject: system:serviceaccount:roksbnkctl-ops:roksbnkctl-ops
trusted-profile: Profile-ccba11f2-3b1f-4b1a-b8a4-aeed2b7b3320
secret:       roksbnkctl-ibm-creds (rotated 2026-05-10T11:03:17Z)

What each line surfaces:

  1. Pod phase + readiness — Running + true is green; anything else means the pod is unhealthy and Backend.Run calls will fail. The container count is exactly one (tools); ready: true is a single bool, not a 2/2-style ratio.
  2. Image — the :v… tag matches the roksbnkctl release the image was published with (resolved at install time from the binary’s version; see Chapter 17 §:dev tag resolution). Mismatched against your roksbnkctl --version means re-running ops install will pull the matching image.
  3. RBAC subject — the SA the pod runs as. kubectl describe clusterrole roksbnkctl-ops prints the full ruleset (the ClusterRoleBinding is named the same as the role).
  4. Trusted-profile line — reads the SA’s iam.cloud.ibm.com/trusted-profile annotation. The value is the IBM IAM Profile ID (Profile-<uuid>, the canonical IAM identifier — runOpsInstall annotates with tp.ID rather than the friendly roksbnkctl-ops-<workspace> name so the value is grep-friendly against IBM Cloud IAM audit logs; the parenthetical form on the ✓ Provisioned IAM trusted profile … install line cross-references both). Present + non-empty means ops install ran with --trusted-profile=auto or =on and the provisioning succeeded; the runtime cred flow is going via the projected SA token. Under --trusted-profile=off (or auto-fallback when IAM perms were missing) the line reads trusted-profile: (none — static-key Secret path).
  5. Secret line — the cred Secret’s name + the roksbnkctl.io/rotated-at annotation that ops install stamps each time the Secret is applied. Always emitted (the Secret manifest is always rendered — empty data under trusted-profile success, populated under the static-key paths). If the Secret resource is missing entirely on the cluster, the line reads secret: (missing: …).

The current output is a fixed seven-line key/value block; a structured --output json mode is on the v1.x roadmap once ops show grows additional fields (image-id hash, env-hash reconciliation against the live pod, etc.). See docs/PLAN.md §“What’s deliberately deferred to post-v1.0”.

roksbnkctl ops uninstall

Full removal. Run when decommissioning the cluster, or when you want a clean re-install. The command is a destructive-action gate: by default it prints a preview of what would be deleted and exits successfully; the actual deletion only runs with --confirm.

$ roksbnkctl ops uninstall
Would delete (re-run with --confirm to proceed):
  - Pod        roksbnkctl-ops/roksbnkctl-ops
  - Secret     roksbnkctl-ops/roksbnkctl-ibm-creds
  - ServiceAccount roksbnkctl-ops/roksbnkctl-ops
  - ClusterRole/ClusterRoleBinding roksbnkctl-ops
  - Namespace  roksbnkctl-ops
  - Namespace  roksbnkctl-test

$ roksbnkctl ops uninstall --confirm
✓ deleted pod roksbnkctl-ops
✓ deleted secret roksbnkctl-ibm-creds
✓ deleted serviceaccount roksbnkctl-ops
✓ deleted clusterrolebinding roksbnkctl-ops
✓ deleted clusterrole roksbnkctl-ops
✓ deleted namespace roksbnkctl-ops
✓ deleted namespace roksbnkctl-test

Note that the cluster-scoped objects (ClusterRole, ClusterRoleBinding) get cleaned too — they’re not garbage-collected by namespace deletion since they live above the namespace. roksbnkctl ops uninstall --confirm makes this explicit so a stale roksbnkctl-ops ClusterRole can’t outlive a namespace removed via kubectl delete ns.

Both managed namespaces (roksbnkctl-ops and roksbnkctl-test) are deleted. The roksbnkctl-test namespace is where one-shot Job pods (iperf3 client, future probes) land — by the time you’re running uninstall you’ve already concluded those test workloads are finished, so removing the namespace alongside the ops-pod surface keeps the cluster clean.

When to run uninstall:

  • Cluster decommission — the cluster is going away, clean up cluster-scoped objects before destroying it.
  • Cred rotation when paranoid — the rotation story (next section) doesn’t require uninstall, but if you’re worried about old secrets persisting in etcd snapshots, an uninstall + re-install regenerates the Secret cleanly.
  • Image upgrade with a major manifest change — if the embedded k8s_install.yaml evolves (new RBAC rule, security-context tweak), uninstall + install is the cleanest way to apply.

RBAC: the ClusterRole rules

The full ClusterRole rule set (transcribed from internal/exec/k8s_install.yaml):

API groupResourcesVerbsWhy
batchjobsget, list, watch, create, deleteOne-shot Job lifecycle (iperf3 client, future probes). The backend creates the Job, watches it, reads logs, deletes it.
"" (core)podsget, list, watchThe backend lists/watches pods to find Job-spawned pods + to wait for the ops pod’s Ready state. No create/delete — pods are owned by their Jobs (or by ops install’s user-side privilege), not by the pod’s SA.
"" (core)pods/logget, listLog streaming from one-shot Job pods (the bytes the wrapped tool wrote to stdout/stderr).
"" (core)pods/execcreate, getkubectl exec is a create against the pods/exec subresource — the SPDY-channel verb the long-lived ops-pod path uses.
"" (core)secrets (named roksbnkctl-ibm-creds)getThe pod reads the cred Secret directly only if a future workflow opts to (kubelet’s projection of envFrom: secretRef runs as kubelet, not as this SA). The resourceNames filter keeps the SA from reading any other Secret in the namespace — least-privilege per PRD 04 §“In-cluster pod”.

Notably not granted:

  • pods create / delete — the SA can list and watch pods but can’t create or delete them. Pod lifecycle is mediated by Jobs (which the SA does manage) and by the ops pod itself (which ops install creates with the user’s privilege, not the SA’s).
  • secrets create / update / delete / list — the pod never writes Secrets, and can’t even list to discover which Secrets exist in the namespace. The install-time Secret creation is done by the user invoking ops install (whose kubeconfig has cluster-admin or comparable), not by the pod’s SA. Combined with the resourceNames filter on get, this is the tightest practical surface that still lets the pod consume its own cred.
  • services, deployments, namespaces — the SA can’t touch these at all. The iperf3 server fixture (when the throughput test runs) is provisioned by roksbnkctl test throughput running on the caller’s side using the user’s kubeconfig, not by anything inside the ops pod.
  • clusterroles, clusterrolebindings — the pod never modifies its own RBAC.
  • * cluster-admin — explicitly avoided. The pod has exactly the verbs it needs and nothing else.

This matches PRD 04 §“Least privilege per backend” and PRD 03 §“K8s”: the ops pod is a powerful tool but its blast radius is bounded.

To audit the rules on a running cluster:

kubectl describe clusterrole roksbnkctl-ops
kubectl auth can-i --as=system:serviceaccount:roksbnkctl-ops:roksbnkctl-ops \
  '*' '*' --all-namespaces       # should print mostly "no"

Credential propagation

Note. What follows is the static-key propagation path — the fallback rather than the default. --trusted-profile=auto installs assume an IBM Cloud trusted profile via the pod’s projected SA token and the static API key never lands in a Kubernetes Secret. See §“Trusted-profile flow” above for that path. The hop-by-hop description below describes what happens under --trusted-profile=off (and under the auto-fallback when IAM perms don’t allow the trusted-profile path).

The IBMCLOUD_API_KEY reaches the wrapped tool in three hops:

resolver chain (env → keychain → config-b64 → prompt)
       ↓                        on the laptop, at `roksbnkctl ops install` time
  Kubernetes Secret roksbnkctl-ibm-creds                in roksbnkctl-ops namespace
       ↓                        applied by `ops install` via kubectl-equivalent
  Pod env (IBMCLOUD_API_KEY=…)                          via `envFrom: secretRef`
       ↓                        kubelet reads Secret, sets env on container start
  Wrapped tool (`ibmcloud iam oauth-tokens`)            reads from os.Getenv

Three properties this gives you:

  1. The key never appears in argv. kubectl describe pod roksbnkctl-ops shows envFrom: secretRef: name: roksbnkctl-ibm-creds, not the value. kubectl get pod roksbnkctl-ops -o yaml shows the same.
  2. The key never appears in the pod’s own logs. The wrapped tool uses the env var; the env var name (not value) is what the pod’s startup logs print.
  3. The redactor is the defense-in-depth backstop. If the wrapped tool ever prints the value (e.g., ibmcloud --debug), the SPDY stream from the pod is wrapped through internal/exec/redact.go before reaching the caller’s stdout — same as the local + docker backends.

The Secret carries two keys today — IBMCLOUD_API_KEY and the legacy IC_API_KEY alias older ibmcloud versions accept — both populated from the same resolved value. Names are stable; embedded in internal/exec/k8s_install.yaml. Future cluster-side credentials (an AWS access key, a GCP service-account JSON) will add new keys to the same Secret rather than spinning up new Secrets, simplifying RBAC.

Rotation: rotating the API key

Note. Under --trusted-profile=auto / =on (default), there’s nothing to rotate — the ops pod’s IAM tokens are short-lived and the IBM IAM endpoint refreshes them transparently each time the SDK trades the projected SA token. Key rotation only matters when the install ran with --trusted-profile=off or fell back to the static-key Secret because the resolved key lacked iam-identity perms. The procedure below covers that static-key case.

When the IBM Cloud API key changes (key rotation, account takeover, key compromise), you need to update the cluster-side Secret. The flow:

# 1. Update the local resolver chain — pick whichever source you populated
#    initially (the chain order is: env > keychain > config-b64; see chapter 14):
export IBMCLOUD_API_KEY=<new-key>             # env (one-shot)
# or update the keychain entry directly: `keyring` / `secret-tool` / Keychain.app
# or edit ~/.roksbnkctl/<workspace>/config.yaml's api_key_b64 field

# 2. Re-run ops install — this re-resolves the key, updates the cluster
#    Secret, and rolls the pod
roksbnkctl ops install

What ops install does on re-run:

  • The Secret roksbnkctl-ibm-creds is updated with the new value via a client-side Get + Update (the existing Secret’s data is overwritten, the roksbnkctl.io/rotated-at annotation is refreshed, the rest of the metadata is left alone).
  • The pod’s env, however, is set at container-start time — kubelet reads the Secret value when the pod is created, not on every Secret update. So an updated Secret doesn’t propagate to the running pod’s env until the pod is recreated.
  • ops install therefore deletes and recreates the bare ops pod after the Secret update. New pod → kubelet reads the updated Secret → env contains the new value. (Re-creation takes a few seconds for the image cache hit; up to ~30 seconds on a cold cluster.)

The ops pod is a bare Pod, not a Deployment or DaemonSet, so kubectl rollout restart won’t work on it (rollout restart only operates on controller resources). The canonical way to force a fresh pod is roksbnkctl ops install (idempotent — it’ll handle the delete-and-recreate). If you really want to do it by hand:

kubectl delete pod roksbnkctl-ops -n roksbnkctl-ops
# then re-run `roksbnkctl ops install` to create the replacement;
# the bare pod has no controller, so nothing else will recreate it.

roksbnkctl ops show will report phase: Pending briefly during recreation, then Running + ready: true once kubelet finishes projecting the updated Secret.

Operability

Things to know when something’s wrong.

Where pod logs go

roksbnkctl k logs -n roksbnkctl-ops roksbnkctl-ops
# or
kubectl logs -n roksbnkctl-ops roksbnkctl-ops

The pod’s main process is sleep infinity, so the log is mostly empty. Each kubectl exec invocation runs in its own ephemeral process — those processes’ stdout/stderr go back through the SPDY channel to the caller, not into the pod’s log. So kubectl logs is helpful for debugging pod startup (image pull failures, SCC denials, OOMKills) but not for “what did ibmcloud iam oauth-tokens actually print” — that’s just the caller’s stdout.

For a paper trail of recent invocations, capture roksbnkctl ibmcloud --backend k8s … 2>&1 | tee /tmp/ibmcloud.log on the calling side.

Debugging a stuck ops install

ops install waits up to 60 seconds for the pod to become Ready. If it times out:

roksbnkctl k describe -n roksbnkctl-ops pod/roksbnkctl-ops
roksbnkctl k get -n roksbnkctl-ops events --sort-by=.lastTimestamp | tail -20

Common causes:

SymptomCauseFix
ImagePullBackOffghcr.io rate limit, or image tag doesn’t existcheck roksbnkctl --version, ensure ghcr.io is reachable from the cluster
CreateContainerConfigError referencing the SecretSecret was deleted between Secret apply and Pod create (race)re-run roksbnkctl ops install (idempotent)
RunContainerError with SCC denialthe cluster’s PodSecurity admission rejected the manifestkubectl get events will name the missing field; usually means an OpenShift cluster expects the restricted-v2 profile and a manifest field is wrong — file an issue with the event message
Pod stuck in Pending with no Eventscluster is at capacity / out of CPUscale the cluster or trim resources; the pod requests 50m CPU + 128Mi mem, very small

Cluster API outage during ops install

If the kube-apiserver becomes unreachable mid-install (transient cloud-provider issue, kubeconfig expired, network partition), ops install fails fast at whichever step hit the apiserver:

✓ applied namespace roksbnkctl-ops
applying secret roksbnkctl-ops/roksbnkctl-ibm-creds       ... ERROR: Get "https://...": dial tcp: i/o timeout

The install is partial at that point — earlier steps succeeded, later steps didn’t. ops install is idempotent, so just re-run once the apiserver is back; the steps that already completed are no-ops the second time, the steps that didn’t will run.

If the apiserver is permanently gone (cluster destroyed): ops uninstall will fail the same way, since it also needs the apiserver. In that case the cluster-scoped objects (ClusterRole, ClusterRoleBinding) become orphans you can clean up manually if you ever rebuild the cluster, or ignore if you’re done with this cluster’s identity entirely.

Verifying the install end-to-end

A one-liner sanity check:

roksbnkctl ibmcloud --backend k8s iam oauth-tokens

If the SA/Secret/RBAC/Pod chain is healthy, this prints a fresh OAuth token. If it errors, the error message names which link in the chain broke (pod not found, Secret missing, exec denied, ibmcloud CLI exit non-zero).

Chapter 26 — Troubleshooting covers the broader “ops pod is unhappy” failure modes alongside other end-user troubleshooting.

Cross-references

Connectivity testing

roksbnkctl test connectivity answers one question: can my workspace reach the HTTP/HTTPS endpoints I care about right now?

It’s the simplest of the three test suites — no cluster fixtures, no remote vantage, no JSON parsing harness. Each configured URL gets one HTTP GET, the suite reports pass/fail, and the runner exits 0 if every probe passed.

Use it as the first sanity check after roksbnkctl up, as a CI smoke step against a known-good fixture set, or as the “is it me or is it the network” baseline before reaching for curl -v or openssl s_client.

What the connectivity suite does

For each configured URL the runner:

  1. Adds an https:// scheme if you didn’t write one.
  2. Issues a single GET with a 10-second timeout and the user-agent roksbnkctl/test.
  3. Records the HTTP status code, the wall-clock duration, and (for HTTPS) the negotiated TLS version.
  4. Marks the probe pass if the status code is in [200, 400) (any 2xx or 3xx); fail for anything else, any TLS error, any DNS error, any timeout.
  5. Aggregates the per-URL results into a suite result; the suite passes only when every URL passed.

That’s it. No retries, no expected-body matching, no configurable status assertions, no L4 reachability — those are deliberate non-goals (see § When connectivity is the wrong tool below).

Configuring extra_hosts

The list of URLs to probe lives in your workspace config under test.connectivity.extra_hosts:

# ~/.roksbnkctl/<workspace>/config.yaml
test:
  connectivity:
    extra_hosts:
      - https://my-bnk-cis-controller.example.com
      - https://bigip-next-admin.example.com:8443
      - https://gslb.example.com
      - my-bare-host.example.com    # scheme defaults to https://

The schema is intentionally minimal — extra_hosts is a []string of URLs (or bare hostnames; https:// is added when no scheme is present). One entry per line. The order in the file is the order the runner probes.

You don’t have to hand-edit the YAML to populate this list — roksbnkctl test hosts {list,add,remove,clear} reads and writes the same slice. See § Managing test hosts via the CLI below.

There’s no per-host method, no per-host expected-status, and no per-host TLS-trust override today. If you need to assert something more specific than “does HTTP work” — a particular status code, a custom header, a body match — curl is the right tool, not roksbnkctl test connectivity. A richer per-host schema is queued for v1.x; the v1.0 surface holds the YAML simple on purpose.

Chapter 12 — Workspace config covers the full test: block; this chapter expands the connectivity slice.

What extra_hosts typically holds

Three classes of URL show up most often in a real workspace:

  • The BNK CIS controller — confirms the data-plane front-end is reachable and is returning a sane status code.
  • The F5 BIG-IP Next admin endpoint — confirms the management plane is reachable from your seat (often :8443 rather than :443).
  • The GSLB VIP that fronts the application — confirms the routed name actually serves a 2xx; pair with roksbnkctl test dns for the GSLB-aware DNS-side validation.

What doesn’t belong in extra_hosts: anything you only care about on a specific TLS error, anything that needs a request body, anything where pass/fail is more nuanced than “got a 2xx or 3xx”. Those are curl jobs, not connectivity-suite jobs.

Managing test hosts via the CLI

The YAML in § Configuring extra_hosts is the persistent backing store, but you don’t have to hand-edit config.yaml to populate it. roksbnkctl test hosts mirrors the roksbnkctl targets ergonomic (see Chapter 15 — SSH targets) and gives you four subcommands that read and write the test.connectivity.extra_hosts slice directly:

roksbnkctl test hosts list                       # print current extra_hosts, one per line
roksbnkctl test hosts add <url> [<url>...]       # append; idempotent (no-op if already present)
roksbnkctl test hosts remove <url> [<url>...]    # remove; tolerant of absent (no error)
roksbnkctl test hosts clear                      # remove ALL (confirmation prompt, --auto skip)

All four route through the same workspace marshaller targets add uses, so a test hosts add write lands in ~/.roksbnkctl/<workspace>/config.yaml under test.connectivity.extra_hosts — operators who already know the YAML field see the connection. Run cat ~/.roksbnkctl/<workspace>/config.yaml after any mutation to confirm.

Argv discipline matches the rest of the v1.7.x surface: list and clear take cobra.NoArgs (a stray positional is rejected at parse time, not silently consumed); add and remove take cobra.MinimumNArgs(1). A typo like roksbnkctl test hosts list extra fails before any RunE runs.

Worked example

Start from an empty workspace, add a URL, list, then run the connectivity probe — the exact sequence an operator hits the first time the no hosts configured to probe; add via roksbnkctl test hosts add <url> error fires from roksbnkctl test connectivity or no-flag roksbnkctl test dns:

$ roksbnkctl test hosts list
$ echo $?
0

$ roksbnkctl test hosts add https://docs.f5.com
added https://docs.f5.com

$ roksbnkctl test hosts add https://docs.f5.com
already present: https://docs.f5.com

$ roksbnkctl test hosts list
https://docs.f5.com

$ roksbnkctl test connectivity
running connectivity ...
  PASS  https://docs.f5.com  200 OK in 187ms
connectivity PASS (1/1 passed)

Note: the byte-for-byte output above is illustrative — it reflects the surface the Sprint 24 CLI ships and is what tech-writer’s drift sweep will re-capture against the built binary before the v1.7.1 cut. The shape is stable (list empty → zero bytes + exit 0; add against already-present → “already present:” log + exit 0); minor wording deltas in the log lines may surface in the GREEN-verdict re-capture.

--output json is supported on list for scripted callers — empty list emits [], populated list emits a JSON array preserving insertion order:

$ roksbnkctl test hosts list -o json
["https://docs.f5.com","https://bigip-next-admin.example.com:8443"]

clear

clear requires confirmation (defaults to No) because it wipes the entire slice — the same pattern roksbnkctl down and roksbnkctl cluster down use, not the per-target targets remove pattern:

$ roksbnkctl test hosts clear
This will remove ALL configured test hosts (2 entries). Continue? [y/N]: y
cleared 2 entries

$ roksbnkctl test hosts clear --auto
cleared 0 entries

--auto skips the prompt for automation contexts. Mirrors the --auto semantics elsewhere in the binary.

Scope of the CLI

The CLI manages only test.connectivity.extra_hosts — the slice shared by roksbnkctl test connectivity and no-flag roksbnkctl test dns. The test.dns.{default_target,resolvers} and test.throughput.* fields have working flag-driven equivalents (--target, --server, --duration, …) and aren’t covered by test hosts; populate those by hand-editing config.yaml for now. A broader test config surface is a candidate for a future sprint.

YAML-comment preservation through the marshaller is best-effort and depends on the workspace round-trip — if you keep hand-written comments in config.yaml, prefer the YAML path for additions that need to land alongside specific comments; use the CLI for the routine add/remove/list/clear flow.

The --insecure flag

Self-signed certs are common in pre-production BNK deployments — the F5 BIG-IP Next admin endpoint, the CIS controller, an internal GSLB VIP that hasn’t yet been re-fronted with a public CA cert. By default Go’s TLS stack rejects them and the probe fails with x509: certificate signed by unknown authority.

Pass --insecure to skip certificate verification for the run:

roksbnkctl test connectivity --insecure

What --insecure does:

  • Sets tls.Config.InsecureSkipVerify = true on the HTTP client used by the connectivity suite.
  • Applies for the duration of one invocation only.
  • Affects every URL probed in that run.

What --insecure does not do:

  • It does not change L4 / DNS behaviour. A name that won’t resolve still fails; a host that drops TCP still fails.
  • It is not per-host — there’s no --insecure-only=foo.example.com. Once set, the run skips verification for everything in extra_hosts.
  • It is not persisted. Setting it in one invocation does not affect the next.
  • It is not the same as a config-level insecure_tls: true per host. The schema doesn’t have that knob; the only way to skip cert verification today is the session-wide flag.

If you need different TLS-trust posture per endpoint (one URL strict, another lenient), run two invocations with two different extra_hosts lists in two workspaces — that’s the workaround until per-host trust lands.

Reading the output

Default output is human-readable on stderr; pass -o json for machine-readable on stdout.

Human-readable

$ roksbnkctl test connectivity
running connectivity ...
  PASS  https://my-bnk-cis-controller.example.com  200 OK in 142ms
  PASS  https://bigip-next-admin.example.com:8443  302 Found in 88ms
  FAIL  https://gslb.example.com                   Get "...": dial tcp: i/o timeout
connectivity FAIL (2/3 passed)
$ echo $?
1

A 3xx redirect counts as pass — the runner doesn’t follow redirects, but the redirect itself is a successful HTTP response, which is what the suite measures. If you specifically need the final 200 after a redirect chain, curl -L is the tool.

JSON

$ roksbnkctl test connectivity -o json
{
  "schema": "roksbnkctl.v1",
  "command": "test",
  "suite": "connectivity",
  "timestamp": "2026-05-10T14:32:01.123Z",
  "duration_ms": 235,
  "overall": "fail",
  "results": [
    {
      "suite": "connectivity",
      "name": "https://my-bnk-cis-controller.example.com",
      "status": "pass",
      "detail": "200 OK in 142ms",
      "duration_ms": 142,
      "extra": { "status_code": 200, "tls_version": "TLS 1.3" }
    },
    {
      "suite": "connectivity",
      "name": "https://gslb.example.com",
      "status": "fail",
      "detail": "Get \"https://gslb.example.com\": dial tcp: i/o timeout",
      "duration_ms": 10003
    }
  ]
}

Exit code follows the same rules as the human-readable form: 0 on overall: pass, 1 on overall: fail. CI runners can branch on the exit code; richer assertions (e.g., “I tolerate one fail out of five”) need to consume the JSON.

Running connectivity inside roksbnkctl test all

Connectivity is one of the suites the bare roksbnkctl test (or roksbnkctl test all) command dispatches. The runner walks every configured suite, prints per-suite summaries on stderr, and exits non-zero if any suite failed:

$ roksbnkctl test
running connectivity ...
  PASS  https://bnk-cis.dev-tor.example.com  200 OK in 174ms
running dns ...
  PASS  bnk-cis.dev-tor.example.com  resolved 1 address(es)
connectivity PASS (1/1 passed)
dns          PASS (1/1 passed)

PASS overall (2/2 suites passed)

In -o json mode, roksbnkctl test all emits an all-shape envelope with one suites[] entry per suite. CI assertions can pin to either the suite-level overall or to a specific probe’s status:

roksbnkctl test all -o json | jq -e '.suites[] | select(.suite=="connectivity") | .overall == "pass"'

The bare roksbnkctl test defaults to the all suite. To run connectivity in isolation:

roksbnkctl test connectivity            # explicit suite
roksbnkctl test connectivity --insecure # session-wide TLS skip

Exit codes and CI integration

exit 0  →  every probe passed (every URL returned 2xx or 3xx)
exit 1  →  any probe failed (non-2xx/3xx, TLS error, DNS error, timeout)

There’s no third “infra error” exit code from the connectivity suite specifically — the suite is straight Go HTTP, no external tooling, no backend dispatch. If roksbnkctl test connectivity exits non-zero, the cause is in the response from one of your configured URLs.

For a CI step that tolerates a known-flaky endpoint while still failing on the others, consume the JSON instead of relying on the exit code:

roksbnkctl test connectivity -o json \
  | jq -e '[.results[] | select(.name | test("flaky-staging.example") | not) | .status] | all(. == "pass")'

Worked example: probing a BNK deployment

A typical post-up config for a BNK trial — cover the data-plane VIP, the admin endpoint, and the GSLB front:

# ~/.roksbnkctl/dev-tor/config.yaml
test:
  connectivity:
    extra_hosts:
      - https://bnk-cis.dev-tor.bnkfun.example.com         # BNK CIS controller (data plane)
      - https://bigip-next-admin.dev-tor.bnkfun.example.com:8443  # F5 BIG-IP Next admin
      - https://gslb-vip.dev-tor.bnkfun.example.com        # the GSLB front

Then:

$ roksbnkctl test connectivity --insecure
running connectivity ...
  PASS  https://bnk-cis.dev-tor.bnkfun.example.com               200 OK in 174ms
  PASS  https://bigip-next-admin.dev-tor.bnkfun.example.com:8443 302 Found in 91ms
  PASS  https://gslb-vip.dev-tor.bnkfun.example.com              200 OK in 211ms
connectivity PASS (3/3 passed)

--insecure is needed here because BNK’s admin endpoint and the GSLB VIP are fronted by a self-signed cert in dev. Once the trial moves to a production cert chain, drop the flag — the strict path is what you want for staging and prod.

When connectivity is the wrong tool

roksbnkctl test connectivity is “does HTTP work”. For anything finer-grained, reach for the right tool:

ScenarioUse this instead
You want to see the full TLS handshake, the cert chain, the SNI resolution, the negotiated cipheropenssl s_client -connect host:port -servername host
You want headers, redirect-following, body matching, a specific status assertioncurl -v -L --fail-with-body <url>
You want to confirm L4 reachability on a specific port, no HTTP layernc -vz host port (or bash -c 'echo > /dev/tcp/host/port')
You want to confirm DNS resolution from a specific resolver, especially across vantages for GSLBroksbnkctl test dns
You want to see what answer a name returns from inside the cluster vs from your laptoproksbnkctl test dns --gslb-compare
You want to measure bandwidth between two endpointsroksbnkctl test throughput

The connectivity suite is intentionally a thin probe. When the answer to “is it broken” is “yes” and you need to know why, the suite has done its job — it’s flagged the URL — and the next step is one of the tools above.

Cross-references

DNS testing for GSLB

Field specs live in Chapter 28 — Configuration reference. That chapter is generated from the Workspace struct and checked by a test, so it cannot drift. Tables here are for teaching and may be shortened over time; when they disagree with Chapter 28, Chapter 28 is right.

roksbnkctl test dns is the diagnostic surface for DNS-driven traffic management — the kind of behaviour an F5 BIG-IP Next GSLB deployment depends on, where the answer a name returns isn’t a single global truth but a function of who’s asking from where.

This is the longest chapter in the testing section because the question it answers is the most subtle. Connectivity testing tells you “the URL works”; throughput testing tells you “the path is fast”. Both assume the name resolved. When the GSLB is the thing under test, “the name resolved” is itself the question — and the answer changes depending on the network vantage of whoever’s asking.

The flag surface, the JSON output, and the multi-vantage workflow on this page are all what v1.0 ships. The design rationale lives in PRD 03 §“DNS probe (GSLB-aware)”; read that for the why, this chapter for the how.

Pointing the deployment at a BIG-IP DNS. This chapter covers testing GSLB behaviour. The deployment side is one setting: bnk.gslb_datacenter_name, which names the datacenter the CNEInstance registers as.

There was also a bnk.gtm.{url,username,password_b64} block for the BIG-IP DNS connection. It was removed in #227: the environment variables it produced appear nowhere in the f5ingress controller binary on either BNK line, so it configured nothing. The GTM side of the relationship is configured on the GTM, not from here.

Three vantages, one comparison

--gslb-compare is the flagship workflow: a single roksbnkctl test dns invocation fans out across local, k8s, and (optionally) ssh:<target> vantages in parallel, asks each one to resolve the same name, and reports whether the answers diverged.

graph TB
    subgraph runner[roksbnkctl test dns --gslb-compare]
        cmd[fan-out parallel<br/>per-vantage probe]
        cmp[divergence detector<br/>+ JSON aggregate]
    end
    subgraph vantage_local[local vantage]
        L[laptop resolver<br/>public DNS / GSLB outside-the-cluster answer]
    end
    subgraph vantage_k8s[k8s vantage]
        K[ops-pod resolver<br/>cluster CoreDNS / cluster-routed GSLB answer]
    end
    subgraph vantage_ssh[ssh:jumphost vantage]
        S[remote-host resolver<br/>third network path's GSLB answer]
    end
    GSLB[F5 BIG-IP Next GSLB<br/>per-resolver dispatch rule]

    cmd --> L
    cmd --> K
    cmd --> S
    L --> GSLB
    K --> GSLB
    S --> GSLB
    GSLB -. region A IP .-> L
    GSLB -. region B IP .-> K
    GSLB -. region C IP .-> S
    L --> cmp
    K --> cmp
    S --> cmp
    cmp -->|gslb_divergence: true/false| out[stdout JSON<br/>roksbnkctl.dns.v1]

The point of the diagram: a single roksbnkctl invocation probes from network positions a dig from your laptop can’t reach. The cluster vantage answers from the cluster’s egress IP; the SSH vantage answers from a third network path. Comparing the three is exactly the assertion “is the GSLB rule taking effect” needs.

The design rationale for this shape lives in PRD 03 §“DNS probe (GSLB-aware)”. The rest of this chapter is the user-facing surface.

The GSLB problem

F5 BIG-IP Next’s GSLB (Global Server Load Balancing) returns different DNS answers depending on the requesting resolver’s IP address. The discrimination is a feature, not a bug — that’s the whole point of GSLB:

  • Geographic affinity: a user in the US gets a US datacenter IP; a user in the EU gets an EU datacenter IP. The dispatch rule is per-region.
  • Datacenter routing: a request from a known partner CIDR gets a private VIP; a request from the public internet gets a public VIP.
  • Health-check state: when the primary pool member goes unhealthy, GSLB starts handing out the secondary pool member’s IP — but only after the resolver’s TTL on the prior answer expires.
  • Anycast vs unicast: a name fronted by an anycast resolver fleet may return the same answer everywhere; the same name fronted by GSLB returns a per-region answer.

Validating GSLB means validating that the right answer comes back from the right vantage. From your laptop in Toronto, you should see the US datacenter IP. From a workload running in the EU, you should see the EU datacenter IP. From an east-Asia bastion, you should see whatever your GSLB rule says east-Asia gets.

The standard dig www.example.com from your laptop only ever tells you what your laptop’s resolver, talking to its configured upstream, gets back. That’s one vantage. To validate GSLB you need several, and you need to be able to compare the answers.

Why per-vantage probing matters

A concrete worked example. Suppose your GSLB rule says:

“Users in the US get the IP for dc1.example.com (169.45.91.10). Users in the EU get the IP for dc2.example.com (52.123.45.67).”

You run from your laptop in the US:

$ dig +short www.example.com
169.45.91.10

That confirms the US rule. But it tells you nothing about the EU rule. To verify the EU rule you’d have to actually be in the EU — or, more practically, run the query from a network vantage that the GSLB will treat as EU.

roksbnkctl test dns makes that vantage selection a flag:

# From the laptop (your home / office / coffee-shop IP)
roksbnkctl test dns --target www.example.com --type A --backend local

# From inside the cluster (the cluster's egress IP — often a different region)
roksbnkctl test dns --target www.example.com --type A --backend k8s

# From a registered SSH target in the EU
roksbnkctl test dns --target www.example.com --type A --backend ssh:eu-bastion

The --gslb-compare flag fans out across the configured backends in parallel and emits a single JSON report that calls out whether the answers diverged across vantages — exactly the assertion you need for “is the GSLB rule taking effect”.

The roksbnkctl test dns flag surface

roksbnkctl test dns \
  [--target <name>] \
  [--type <record-type>] \
  [--server <server-spec>] \
  [--iterations <N>] \
  [--backend <local|k8s|ssh:<target>>] \
  [--gslb-compare] \
  [-o json]
FlagDefaultNotes
--targetthe workspace’s test.dns.default_target if set, otherwise requiredThe DNS name to query. FQDN preferred (the trailing dot is added if missing).
--typeAAny record type the underlying miekg/dns library accepts via dns.StringToType. Common picks: A, AAAA, CNAME, MX, NS, TXT, SRV, SOA, PTR, CAA, DS, DNSKEY, ANY. The full table also includes HTTPS, SVCB, TLSA, SSHFP, URI, NAPTR, RRSIG, NSEC/NSEC3, LOC, etc.
--serversystemWhere to send the query. Literal IP, host:port, the keyword system (use the host’s /etc/resolv.conf), the keyword cluster (use the cluster’s CoreDNS — --backend k8s only), or a name from the workspace’s test.dns.resolvers map.
--iterations1How many queries to send to the same server. The runner reports per-query RTT plus p50/p95/p99 across the run.
--backendper-tool default (see § Backend selection)local, k8s, or ssh:<target>. Docker is rejected — see § Why --backend docker is rejected.
--gslb-compareoffFan out across all configured vantages and emit a comparison JSON. See § The --gslb-compare workflow.
-o jsontextSwitch from human-readable text on stderr to JSON on stdout. Two schemas: roksbnkctl.dns.v1.vantage for single-vantage runs (a flat document), roksbnkctl.dns.v1 for --gslb-compare (wraps one or more vantages plus a gslb_divergence boolean).

The probe library is github.com/miekg/dns — the same DNS implementation CoreDNS uses. Replacing the standard library’s net.Resolver got us three things we couldn’t get otherwise:

  • Full record-type surface: net.Resolver only exposes a fixed subset; GSLB validation often needs CAA (cert provisioning), DS/DNSKEY (DNSSEC), or SOA (authority chain) which net.Resolver doesn’t.
  • Per-query server selection: standard library hides the upstream resolver behind whatever /etc/resolv.conf says; we need to be able to point at a specific GSLB VIP for the query.
  • Per-query RTT measurement: miekg/dns’s Exchange() returns time.Duration directly. No timing-overhead fudging from running queries serially through the resolver chain.

When --backend k8s is selected, the probe self-execs as a one-shot Job in the roksbnkctl-test namespace — no separate image. roksbnkctl is its own probe runner; the Job’s pod runs roksbnkctl test dns ... with --backend local from the cluster’s network vantage. See Chapter 17 §“K8s backend” for the Job mechanics.

Server resolution

--server accepts five forms:

Literal IP or host:port

roksbnkctl test dns --target www.example.com --type A --server 8.8.8.8
roksbnkctl test dns --target www.example.com --type A --server 8.8.8.8:53
roksbnkctl test dns --target www.example.com --type A --server gslb-vip.example.com:53

Bare hosts default to port 53. IPv6 literals must be bracketed: [2001:4860:4860::8888]:53.

system

roksbnkctl test dns --target www.example.com --type A --server system

Reads /etc/resolv.conf from the host running the probe (so for --backend local that’s your laptop; for --backend k8s that’s the Pod’s /etc/resolv.conf, which CoreDNS owns; for --backend ssh:<target> that’s the target’s /etc/resolv.conf). This is the default if --server is omitted entirely.

cluster

roksbnkctl test dns --target www.example.com --type A --server cluster --backend k8s

Identical to system when running with --backend k8s (CoreDNS is what the Pod’s /etc/resolv.conf points at). Allowed only with --backend k8s — using --server cluster from a local or ssh vantage errors at parse time, since “cluster CoreDNS” isn’t a meaningful concept from outside.

Named resolver from workspace config

roksbnkctl test dns --target www.example.com --type A --server gslb-vip

Looks up gslb-vip in test.dns.resolvers (see next section). Useful for checking the same name against several different upstream resolvers without remembering each IP.

Workspace config: test.dns

Two new keys land this sprint under the existing test: block:

# ~/.roksbnkctl/<workspace>/config.yaml
test:
  dns:
    default_target: www.example.com
    resolvers:
      google:     "8.8.8.8:53"
      cloudflare: "1.1.1.1:53"
      gslb-vip:   "169.45.91.5:53"
FieldTypeDefaultNotes
dns.default_targetstringemptyThe name roksbnkctl test dns queries when --target isn’t passed. Lets you keep the per-workspace canonical name out of every CLI invocation.
dns.resolversmap[string]stringemptyNamed resolvers usable as --server <name>. Values are <host>:<port> (port required; mirrors the --server literal-IP form).

Both are optional. With neither set, --target is required on every invocation and --server only accepts literal IPs / system / cluster.

Chapter 12 §“test:” is the full workspace-config reference; this is the GSLB-relevant subset.

Backend selection for the probe

The probe runs from one network vantage at a time per --backend:

  • --backend local: in-process. Runs in the roksbnkctl binary itself, so the network vantage is your laptop’s. No cluster prereq, no SSH prereq.
  • --backend k8s: a one-shot Job in roksbnkctl-test. The Job’s pod runs the bundled tools image ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:<tag> — the same image the in-cluster ops pod uses, which carries both ibmcloud and roksbnkctl on PATH. (If a Job fails to pull, kubectl describe pod will name the roksbnkctl-tools-ibmcloud image — there is no separate roksbnkctl-cli image to look for.) The Job’s command is roksbnkctl test dns --target ... --type ... --server ... --backend local -o json; the stdout is collected via the k8s backend’s log-stream path. Vantage is the cluster’s egress IP.
  • --backend ssh:<target>: scps the roksbnkctl binary onto the target if it’s missing (or skips if it’s already there, marker-file gated), then runs the same roksbnkctl test dns ... --backend local -o json over SSH. The vantage is the target’s IP.
  • --backend docker: rejected — see below.

The default backend per roksbnkctl invocation (when --backend is omitted and there’s no exec.dns.backend in workspace config) is local. To run GSLB cross-vantage you generally pass --gslb-compare, which fans out instead of picking a single vantage.

Chapter 17 §“K8s backend” has the one-shot-Job mechanics; Chapter 17 §“SSH backend” has the file-materialisation and bootstrap story. Both apply to the DNS probe verbatim.

The --gslb-compare workflow

Pass --gslb-compare to fan out across all configured vantages in parallel and emit a single comparison JSON:

roksbnkctl test dns \
  --target www.example.com \
  --type A \
  --server gslb-vip.example.com \
  --gslb-compare \
  -o json

What happens:

  1. The runner enumerates configured vantages: local always; k8s when a kubeconfig is reachable on the host (the probe runs as a one-shot Job in roksbnkctl-test — the long-lived ops pod isn’t required); plus every entry in the workspace’s targets: block, each as ssh:<name>.
  2. Each vantage runs the probe in sequence (one at a time; the run completes when the slowest vantage returns). The query (target, type, server) is identical; only the backend differs. Worst-case wall time with three vantages and the default 2-second per-query timeout is ~6 seconds.
  3. Per-vantage results are collected with their full RTT distribution and answer set.
  4. The runner compares the answer sets across vantages. If they differ, gslb_divergence is set to true in the output and a human-readable summary names the diverging vantages.
  5. The output is a single roksbnkctl.dns.v1 JSON document wrapping one vantages[] entry per backend.

gslb_divergence: true is not a failure signal — for a healthy GSLB it’s the expected outcome. The exit code is 0 whenever every per-vantage probe succeeded (got an answer, even if the answers differ). The exit code is 1 when any per-vantage probe failed (NXDOMAIN, SERVFAIL, timeout).

JSON output schema

There are two distinct JSON shapes depending on whether --gslb-compare was passed. Both are versioned, both pin against PRD 03 §“DNS probe”, and both can be consumed by CI:

  • roksbnkctl.dns.v1.vantage — single-vantage probe. A flat document describing one vantage’s result.
  • roksbnkctl.dns.v1 — multi-vantage --gslb-compare. Wraps an array of per-vantage entries plus a gslb_divergence boolean.

Single-vantage output (roksbnkctl.dns.v1.vantage)

roksbnkctl test dns --target www.cloudflare.com --type A --server 8.8.8.8 \
  --iterations 10 --backend local -o json
{
  "schema": "roksbnkctl.dns.v1.vantage",
  "backend": "local",
  "server": "8.8.8.8:53",
  "iterations": 10,
  "rtt_ms": { "p50": 12.4, "p95": 18.1, "p99": 22.7 },
  "answers": [
    { "name": "www.cloudflare.com.", "type": "A", "ttl": 60, "rdata": "104.16.132.229" },
    { "name": "www.cloudflare.com.", "type": "A", "ttl": 60, "rdata": "104.16.133.229" }
  ],
  "rcode": "NOERROR",
  "authoritative": false,
  "truncated": false
}

This is the per-vantage shape — no target / type wrapper at the top level (the caller already knows what they queried), no vantages[] array, no gslb_divergence field.

Multi-vantage output (roksbnkctl.dns.v1, divergence detected)

roksbnkctl test dns --target www.example.com --type A --server gslb-vip.example.com \
  --gslb-compare -o json
{
  "schema": "roksbnkctl.dns.v1",
  "target": "www.example.com",
  "type": "A",
  "vantages": [
    {
      "schema": "roksbnkctl.dns.v1.vantage",
      "backend": "local",
      "server": "169.45.91.5:53",
      "iterations": 1,
      "rtt_ms": { "p50": 14.2, "p95": 14.2, "p99": 14.2 },
      "answers": [
        { "name": "www.example.com.", "type": "A", "ttl": 30, "rdata": "169.45.91.10" }
      ],
      "rcode": "NOERROR",
      "authoritative": true,
      "truncated": false
    },
    {
      "schema": "roksbnkctl.dns.v1.vantage",
      "backend": "k8s",
      "server": "169.45.91.5:53",
      "iterations": 1,
      "rtt_ms": { "p50": 8.7, "p95": 8.7, "p99": 8.7 },
      "answers": [
        { "name": "www.example.com.", "type": "A", "ttl": 30, "rdata": "10.20.30.40" }
      ],
      "rcode": "NOERROR",
      "authoritative": true,
      "truncated": false
    }
  ],
  "gslb_divergence": true,
  "gslb_divergence_summary": "answers differ between local (169.45.91.10) and k8s (10.20.30.40) — GSLB returning location-specific records as expected"
}

The comparison document embeds the per-vantage shape unchanged inside vantages[] — each entry still carries "schema": "roksbnkctl.dns.v1.vantage" so a downstream parser can validate per-vantage entries against the same schema independent of whether they came in standalone or as part of a comparison.

Schema field reference

Per-vantage shape (roksbnkctl.dns.v1.vantage):

PathTypeMeaning
schemastringAlways roksbnkctl.dns.v1.vantage.
backendstringlocal, k8s, or ssh:<target>.
serverstringThe resolver address actually used (literal, system-resolvconf result, or named-resolver lookup).
iterationsintHow many queries went to that vantage. Mirrors --iterations.
rtt_msobject{ p50, p95, p99 } across the iterations. Single-iteration runs report the same number for all three.
answers[]arrayThe RRs returned. name is the FQDN, type is the record type, ttl is from the response, rdata is the RR’s data (IP for A/AAAA, target for CNAME, etc.).
rcodestringThe DNS response code: NOERROR, NXDOMAIN, SERVFAIL, REFUSED, TIMEOUT.
authoritativeboolWhether the AA flag was set in the response.
truncatedboolWhether the TC flag was set. (The probe automatically retries truncated UDP responses over TCP; the field reflects the final response.)
errorstringPresent only when the probe could not get a usable response. Carries the underlying Go error string.

Comparison shape (roksbnkctl.dns.v1, emitted by --gslb-compare):

PathTypeMeaning
schemastringAlways roksbnkctl.dns.v1.
targetstringThe queried name, normalised to FQDN (trailing dot included).
typestringThe record type queried.
vantages[]arrayOne per-vantage entry (each conforms to roksbnkctl.dns.v1.vantage).
gslb_divergenceboolTrue when the answer sets across vantages[] differ.
gslb_divergence_summarystringPresent only when gslb_divergence: true. Human-readable explanation naming the diverging vantages and answers.

Both schemas are stable at v1.0 — additive changes (new optional fields) are allowed within v1; field renames or removals would bump to .v2. v1.0 includes the optional edns_client_subnet object on each per-vantage entry, emitted when the resolver echoes an EDNS Client Subnet option (RFC 7871) in its response — most GSLB-aware authoritative servers do; vanilla recursive resolvers don’t, in which case the field is omitted from the JSON via omitempty. Sub-fields are family (1 = IPv4, 2 = IPv6), source_netmask, scope_netmask, and address. Useful for confirming the GSLB actually saw your client’s geographic scope rather than the resolver’s IP.

RTT measurement and --iterations

Per-query RTT is captured directly from miekg/dns’s Exchange(), which returns the round-trip duration without the runner having to wrap a stopwatch around the call. That keeps the timing honest — no scheduling jitter from a goroutine yield between time.Now() and the actual UDP send.

--iterations N runs the same query against the same server N times serially and reports p50/p95/p99 across the samples. Use cases:

  • Detecting health-check flapping: if a GSLB pool member is on the edge of its health threshold, the answer can flip back and forth across iterations. A single query catches one snapshot; ten iterations show the ratio.
  • Detecting anycast routing changes: when an anycast resolver changes which BGP path serves your AS, RTT can jump 30-100ms. p99 catches the worst case; p50 stays steady.
  • Establishing a baseline before a change: run with --iterations 30 before a GSLB rule change, again after, and compare distributions.

For --backend k8s and --backend ssh:<target>, RTT is measured inside the remote vantage. The number reflects the resolver-to-resolver path from that vantage, not the laptop-to-cluster (or laptop-to-jumphost) transit. That’s the correct measurement: when you ask “how slow is the GSLB from inside the cluster”, you mean cluster-side latency, not how long it took your laptop to wait for the SSH-tunnelled answer.

Sample F5 BIG-IP Next GSLB scenarios

Three concrete scenarios you’ll hit when validating a real BNK deployment.

Scenario 1: Geographic affinity working as expected

You’ve configured a GSLB rule that returns dc1 (169.45.91.10) for US queries and dc2 (52.123.45.67) for EU queries. You’re in the US, your jumphost is in eu-de (Frankfurt).

roksbnkctl test dns \
  --target www.example.com \
  --type A \
  --server gslb-vip.example.com \
  --gslb-compare \
  -o json

Expected output:

{
  "schema": "roksbnkctl.dns.v1",
  "target": "www.example.com",
  "type": "A",
  "vantages": [
    { "backend": "local",            "answers": [{ "rdata": "169.45.91.10" }], "rcode": "NOERROR" },
    { "backend": "ssh:eu-jumphost",  "answers": [{ "rdata": "52.123.45.67" }], "rcode": "NOERROR" }
  ],
  "gslb_divergence": true,
  "gslb_divergence_summary": "answers differ between local (169.45.91.10) and ssh:eu-jumphost (52.123.45.67) — GSLB returning location-specific records as expected"
}

gslb_divergence: true is the assertion you wanted. There are two ways to key CI on it:

# Option A: parse the JSON yourself
roksbnkctl test dns ... --gslb-compare -o json | jq -e '.gslb_divergence == true'

# Option B: built-in --require-divergence flag (the binary returns
# non-zero exit when --gslb-compare finds NO divergence)
roksbnkctl test dns ... --gslb-compare --require-divergence

Both forms produce a non-zero exit when gslb_divergence is false — the --require-divergence flag is the same assertion baked into the binary so CI scripts don’t need a jq dependency. Pick whichever fits your pipeline.

If gslb_divergence flips to false, something has changed — the GSLB rule was disabled, the geographic dispatch broke, or the jumphost moved out of the EU range. Surface that as a CI failure.

Scenario 2: Health-check-driven failover

A BNK pool member backing www.example.com has an active health check. You want to verify that GSLB stops returning that member’s IP when the health check fails.

Steps:

# 1. Baseline: while the member is healthy, both vantages return its IP
roksbnkctl test dns --target www.example.com --type A --server gslb-vip.example.com \
  --gslb-compare -o json > /tmp/before.json

# 2. Take the member offline (BNK admin UI / API; outside roksbnkctl's surface)
# 3. Wait for the GSLB's TTL to expire (in this example, 30 seconds)
sleep 35

# 4. Probe again
roksbnkctl test dns --target www.example.com --type A --server gslb-vip.example.com \
  --gslb-compare -o json > /tmp/after.json

# 5. Diff the answer sets
diff <(jq '.vantages[].answers' /tmp/before.json) <(jq '.vantages[].answers' /tmp/after.json)

You expect the IP in vantages[].answers[].rdata to change from the failed member’s IP to the secondary’s IP across both vantages — and to do so on roughly the same TTL boundary across vantages. If one vantage flips and the other doesn’t, the GSLB’s health-check propagation is asymmetric — useful diagnostic data.

Scenario 3: Anycast vs unicast detection

You suspect a name is fronted by a public anycast resolver fleet (Cloudflare’s 1.1.1.1, Google’s 8.8.8.8) rather than a real GSLB. Anycast returns the same answer everywhere; GSLB returns per-region answers.

roksbnkctl test dns --target www.cloudflare.com --type A --server 8.8.8.8 \
  --gslb-compare -o json
{
  "schema": "roksbnkctl.dns.v1",
  "target": "www.cloudflare.com",
  "vantages": [
    { "backend": "local", "answers": [{ "rdata": "104.16.132.229" }] },
    { "backend": "k8s",   "answers": [{ "rdata": "104.16.132.229" }] }
  ],
  "gslb_divergence": false
}

gslb_divergence: false despite probing from two vantages → the answer is anycast, not GSLB-dispatched. Useful when handing off to a customer who’s claiming “the GSLB isn’t routing me right” — you can prove the name they’re querying isn’t actually under GSLB control.

Why --backend docker is rejected

A Docker container running locally on the user’s laptop has the same network identity as the host (default bridge networking). The container’s egress NATs out via the host’s interface, so a DNS query from inside the container reaches the upstream resolver from the host’s IP.

For GSLB validation, that means --backend docker would give an answer identical to --backend local. No new vantage. The CLI rejects the combination at parse time:

$ roksbnkctl test dns --backend docker --target www.example.com --type A
error: DNS probe doesn't benefit from --backend docker (same network identity
       as --backend local, no GSLB-relevant vantage difference). Use --backend
       local, --backend k8s, or --backend ssh:<target> instead.

This is by design and called out in PRD 03 §“DNS probe”. If you want frozen-toolchain DNS testing for CI reproducibility, the roksbnkctl binary itself is already a single static binary — pinning it to a specific version pins the probe.

Integration with extra_hosts

If you’ve configured connectivity.extra_hosts for your workspace and want a quick “does each of those names resolve” check:

roksbnkctl test dns

With no --target and no --gslb-compare, the runner falls back to today’s behaviour: probe each host in connectivity.extra_hosts, single-vantage, single-iteration, with the host’s system resolver. This is the same shape as the connectivity suite’s reachability probe — no GSLB awareness, no per-server selection — and is the right tool for “did I typo a hostname in my config” rather than “is the GSLB doing what I configured”.

For real GSLB validation you almost always want --gslb-compare plus an explicit --target and --server. The extra_hosts fallback is the carry-over from earlier roksbnkctl releases, kept for compatibility.

If the no-flag invocation exits with no hosts configured to probe; add via roksbnkctl test hosts add <url>, the slice is empty for this workspace. Populate it with roksbnkctl test hosts add <url> (see Chapter 20 §“Managing test hosts via the CLI”) instead of hand-editing config.yaml.

Chapter 20 — Connectivity testing covers connectivity.extra_hosts in full.

Worked example: GSLB divergence troubleshooting

End-to-end Part VI scenario: a customer’s BNK GSLB rule says “users in the US get IP A, users in the EU get IP B”. They claim the rule isn’t working — EU users keep landing on the US datacenter. You have roksbnkctl configured with a local laptop (your office), the customer’s cluster (ops pod installed), and a customer-provided EU bastion as an SSH target named eu-bastion. Goal: prove or disprove the divergence, point at where to look next.

# 1. Baseline — three-vantage probe of the GSLB-fronted name
$ roksbnkctl test dns \
    --target www.example.com \
    --type A \
    --server gslb-vip.example.com \
    --gslb-compare \
    -o json | tee /tmp/gslb-baseline.json | jq .
{
  "schema": "roksbnkctl.dns.v1",
  "target": "www.example.com",
  "type": "A",
  "server": "gslb-vip.example.com",
  "gslb_divergence": false,
  "vantages": [
    {
      "backend": "local",
      "answers": [{"rdata": "192.0.2.10"}],
      "rtt_ms": {"p50": 18.2, "p95": 22.1}
    },
    {
      "backend": "k8s",
      "answers": [{"rdata": "192.0.2.10"}],
      "rtt_ms": {"p50": 6.4, "p95": 9.8}
    },
    {
      "backend": "ssh:eu-bastion",
      "answers": [{"rdata": "192.0.2.10"}],
      "rtt_ms": {"p50": 24.7, "p95": 31.4}
    }
  ]
}

# Note: `gslb_divergence: false` means ALL THREE vantages got 192.0.2.10
# back. That's the smoking gun — the customer is right. The GSLB is
# returning the same answer regardless of where the resolver is.

# 2. Confirm via `dig` from the EU bastion directly (sanity check that
# roksbnkctl isn't somehow masking the answer)
$ roksbnkctl exec --on eu-bastion -- dig +short @gslb-vip.example.com www.example.com
192.0.2.10
# Same answer. The probe isn't lying.

# 3. Check the GSLB rule's resolver-IP detection
$ roksbnkctl ibmcloud --backend ssh:eu-bastion ks cluster get \
    --cluster customer-cluster --json | jq '.serviceEndpoints'
# (Inspect the cluster's egress IP that GSLB sees for k8s vantage queries)

# 4. Repeat the probe with a known-good test name (one with a working
# anycast resolver, to rule out probe-side bugs)
$ roksbnkctl test dns \
    --target www.cloudflare.com \
    --type A \
    --server 8.8.8.8 \
    --gslb-compare
✓ local         : 104.16.124.96 (RTT 12ms)
✓ k8s           : 104.16.124.96 (RTT 6ms)
✓ ssh:eu-bastion: 104.16.124.96 (RTT 28ms)
  gslb_divergence: false  (anycast — expected)

# 5. Hand off the artefact to the GSLB owner with three pieces of evidence:
#    - /tmp/gslb-baseline.json (the divergence-false JSON)
#    - The cluster's resolver-IP-as-seen-by-GSLB from step 3
#    - Confirmation that the probe machinery itself works (step 4)

What this walkthrough lets you say with confidence: “the GSLB is returning 192.0.2.10 to all three vantages including the EU bastion. The rule isn’t dispatching. Check the rule’s resolver-IP-CIDR-match clause against the EU bastion’s egress IP <X>.” That’s a falsifiable claim a GSLB engineer can act on.

Common follow-up failure modes (covered in Chapter 26):

  • Probe says diverged but customer says it’s not — likely the customer’s testing point is behind a NAT that masks their network position; their resolver looks like a different region to the GSLB.
  • Probe says NOT diverged but customer says it IS — likely the customer is querying through a CDN-fronted resolver that anycasts the request. Re-probe with --server <gslb-vip-directly> to skip the resolver chain.
  • All vantages return SERVFAIL — the GSLB-VIP isn’t reachable from any of them. Check connectivity (Chapter 20) before re-running.

Cross-references

Throughput testing

roksbnkctl test throughput measures TCP bandwidth between an iperf3 client and an iperf3 server, with at least one side running adjacent to (or inside) the cluster so the number reflects something useful — cluster fabric, the inbound path through a LoadBalancer (the iperf3 north-south mode, default), the outbound path from a jumphost, or pod-to-pod (east-west).

The heavy lifting (server pod lifecycle, OpenShift SCC compliance, in-cluster client Job, log streaming) lives in Chapter 17 §“K8s backend”. This chapter is the user-facing flag surface, the mode selection, and the output-interpretation guide.

What the suite measures

Plain TCP throughput, plus jitter and retransmits, between two endpoints both running iperf3:

  • The server runs in the cluster — a single bare Pod plus a Service, deployed in the roksbnkctl-test namespace. Service type is ClusterIP for east-west, LoadBalancer for north-south. See Chapter 17 §“iperf3 server side” for the manifest details.
  • The client runs wherever you point the backend — by default in the cluster as a one-shot Job, alternatively on your laptop or on a registered SSH target.
  • Output is iperf3’s native -J JSON, parsed and surfaced as roksbnkctl test throughput JSON.

The suite is appropriate for “is the cluster fabric healthy”, “is the BNK data path delivering the bandwidth I expect from outside”, and “is this jumphost the bottleneck between my office and the cluster”. It is not a precision benchmark — TCP throughput is sensitive to MTU, NIC offloads, kernel tunables, and the iperf3 server’s own resource limits, none of which the suite tries to control.

The two modes

Mode is selected by --mode. The default is north-south.

roksbnkctl test throughput --mode north-south   # default
roksbnkctl test throughput --mode east-west

--mode north-south

Measures the inbound path from outside the cluster to a Pod inside it. The server’s Service is a LoadBalancer, so the cluster provisions an external endpoint (an IBM Cloud LB on ROKS, an external IP / hostname on bare-metal k8s). The client connects to that endpoint.

Use cases:

  • “Is the BNK ingress path delivering the bandwidth I expect”
  • “Is my office Wi-Fi or my home connection the bottleneck”
  • “Is the cluster’s egress capacity what the cloud provider promised”

Combine with --backend local (run the client on your laptop) when you specifically want to measure the laptop-to-cluster path. Combine with --backend ssh:<jumphost> when you want a known-stable measurement vantage from a jumphost in a known IP block — useful when laptop Wi-Fi is suspect.

--mode east-west

Measures the intra-cluster fabric — Pod-to-Pod or host-to-Pod. The server’s Service is ClusterIP, reachable only from inside the cluster. The default --backend k8s runs the client adjacent to the server (a one-shot Job in the same namespace), so the number reflects the CNI’s pod-to-pod throughput.

Use cases:

  • “Is the cluster’s network plugin healthy”
  • “Are the worker nodes hitting the link rate the underlying fabric promises”
  • “Has the BNK CIS deployment regressed cluster-internal throughput”

Today’s east-west still allows --backend local (the client runs on the host and reaches the ClusterIP via NodePort-equivalent access if the kubeconfig is the same one a kubectl port-forward would use), but the number is a host-to-cluster-via-NodePort hybrid in that case rather than a true pod-to-pod measurement. True pod-to-pod east-west — both sides scheduled to specific pods, optionally pinned to different nodes via --cross-node — is the v1.x refinement; today the in-cluster Job client gets you most of the way there.

Per-tool default backend

The default backend for iperf3 is k8s. From the per-tool defaults table in Chapter 18 §“Per-tool default backends”:

ToolDefault backendWhy
iperf3k8sThroughput from a laptop’s uplink isn’t the cluster’s bandwidth. Default to running adjacent to the cluster so the number reflects fabric, not Wi-Fi.

The default holds whether or not you’ve set exec.iperf3.backend in workspace config. To override per-invocation:

roksbnkctl test throughput --backend local                  # client on laptop
roksbnkctl test throughput --backend ssh:jumphost           # client on jumphost
roksbnkctl test throughput --backend k8s                    # default; explicit

--backend docker is rejected by the throughput suite. A Docker container running locally has the same network identity as the host (default bridge networking), so the client’s view of the network is identical to --backend local. The CLI errors at parse time:

$ roksbnkctl test throughput --backend docker
error: --backend docker isn't supported for iperf3 — docker shares the host
       network namespace by default and gives no network-locality benefit over
       local. Use --backend local or --backend k8s instead

Chapter 18 §“Throughput testing” is the decision-tree row that walks through the (mode, backend) matrix.

When local or ssh:<target> makes sense:

  • local: you’re deliberately measuring laptop-to-cluster bandwidth (north-south from your seat). Typical use is debugging “the dashboard feels slow from my desk” — you want to confirm the office uplink, not the cluster fabric, is the bottleneck.
  • ssh:<target>: you have a registered SSH target with a known IP (often a customer jumphost in a specific datacenter) and want a bandwidth measurement from that vantage. The SSH backend ensures iperf3 is on the target (auto-installs via apt with --bootstrap on Ubuntu; see Chapter 17 §“SSH backend”).

The bundled image and the runAsNonRoot constraint

The iperf3 server pod’s securityContext is set to satisfy OpenShift’s restricted-v2 SCC:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  seccompProfile:
    type: RuntimeDefault
containers:
- name: iperf3
  securityContext:
    allowPrivilegeEscalation: false
    runAsNonRoot: true
    capabilities:
      drop: ["ALL"]

iperf3 listens on port 5201 (unprivileged) so root isn’t needed. The bundled image at ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<v> declares USER 1000 in its Dockerfile, matching the pod’s runAsUser: 1000.

Two things follow:

  1. Stock images that run as root will fail admission. The default in workspace config is networkstatic/iperf3:latest, which runs as root. On OpenShift / on any cluster with restricted-v2 PodSecurity admission, that image will fail with forbidden: violates PodSecurity "restricted:v1.x". The fix is to switch to the bundled image:

    # ~/.roksbnkctl/<workspace>/config.yaml
    test:
      throughput:
        image: ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:v0.9.0
    

    The bundled image is the --backend k8s default for the client Job regardless; the workspace override only affects the server pod’s image. Keep them in sync to avoid version skew during a debug session.

  2. A custom workspace-overridden image must respect runAsNonRoot. If you point test.throughput.image at your own iperf3 image, that image must not require root to start. iperf3 itself doesn’t need privilege; if your image does, drop the USER root line and rebuild.

Chapter 17 §“iperf3 server side” goes deeper on the SCC story — what the four securityContext fields do, why each is required, and how to debug an admission failure.

OpenShift SCC failure mode

If your throughput pod fails to start with one of:

  • Forbidden: violates PodSecurity "restricted:v1.x"
  • unable to validate against any security context constraint: ... restricted-v2
  • runAsNonRoot is required

…then either the configured image runs as root (use the bundled ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<v> image instead — set test.throughput.image in workspace config) or the cluster’s PodSecurity admission is stricter than restricted-v2 (the manifest the k8s backend builds satisfies restricted-v2 but not privileged; if your cluster requires privileged for the test namespace, that’s a cluster policy question outside the suite’s control).

Chapter 17 §“iperf3 server side” is the canonical source for the manifest’s securityContext. If you’re hand-rolling an iperf3 image for the suite, it’s the spec to match.

Reading the output

Default output is human-readable on stderr; -o json switches to JSON on stdout.

Human-readable

$ roksbnkctl test throughput
→ Deploying iperf3 fixture
→ Waiting for iperf3 server pod ready
→ Waiting for LoadBalancer endpoint (can take 30–90s on IBM Cloud)
✓ iperf3 endpoint: 169.45.91.10:5201
running throughput ...
  PASS  iperf3 north-south → 169.45.91.10:5201 (k8s)  3.41 Gbps received, 0% retransmits in 30s
throughput PASS (1/1 passed)
✓ iperf3 fixture removed

JSON

iperf3’s -J output is rich (sender, receiver, per-stream stats, CPU usage). The roksbnkctl wrapper preserves the iperf3 JSON in the probe’s detail field so all of iperf3’s data survives, while the suite-level shell follows the roksbnkctl.v1 schema:

roksbnkctl test throughput -o json
{
  "schema": "roksbnkctl.v1",
  "command": "test",
  "suite": "throughput",
  "timestamp": "2026-05-10T14:32:01Z",
  "duration_ms": 31420,
  "overall": "pass",
  "results": [
    {
      "suite": "throughput",
      "name": "iperf3 north-south → 169.45.91.10:5201 (k8s)",
      "status": "pass",
      "duration_ms": 30015,
      "detail": "{ ...full iperf3 -J JSON, including sum_received, sum_sent... }"
    }
  ]
}

The fields you’ll most often want from the embedded iperf3 JSON, in order of usefulness:

FieldWhat it tells you
end.sum_received.bits_per_secondThe throughput number you should report. iperf3 measures both sender and receiver and the receiver number is the right one to quote — it accounts for retransmits and any path losses.
end.sum_sent.bits_per_secondSender-side throughput. If sent ≫ received, packets were dropped on the path. If sent ≈ received, the path is healthy.
end.sum_sent.retransmitsTCP retransmits over the run. A handful is normal; double-digit-percent of streams indicates congestion or a bad NIC.
end.streams[].sender.jitter_msPer-stream jitter. Useful for diagnosing variable-latency paths.
end.cpu_utilization_percent.host_totalWhether the client CPU was the bottleneck. >80% suggests the iperf3 client maxed out CPU before the network did — increase the iperf3 server’s stream count (a server-pod knob, not a roksbnkctl flag) to spread load, or run on a beefier client.

Example interpretation:

sum_received: 3.41 Gbps     → headline number
sum_sent:     3.42 Gbps     → very close, healthy path
retransmits:  127           → normal-low

vs

sum_received: 1.21 Gbps     → headline number
sum_sent:     2.95 Gbps     → ≫ received; >50% of bytes lost or retransmitted
retransmits:  18743         → heavy

The second shape is what a saturated link or a flaky NIC looks like. The first is a healthy gigabit-class path.

Tuning knobs in workspace config

# ~/.roksbnkctl/<workspace>/config.yaml
test:
  throughput:
    image: ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:v0.9.0
    duration: 30        # iperf3 -t flag, seconds
    streams: 8          # iperf3 -P flag, parallel streams
    default_mode: north-south

The defaults (30s, 8 streams, north-south) are a reasonable starting point for “is the BNK data path healthy”. For deeper diagnosis:

  • Bump duration to 60-90s if the path is variable and you want a stable average.
  • Bump streams to 16 or 32 if the path’s bandwidth-delay product is high (long-haul links benefit from more parallelism).
  • Drop streams to 1 if you’re specifically testing single-flow throughput (e.g., reproducing a customer’s “single-stream upload feels slow” complaint).

Chapter 12 — Workspace config lists the full schema.

Cleanup and --keep

By default the suite tears down the iperf3 server pod and Service after the client run completes. If a test fails and you want to poke at the fixture (kubectl exec into the server, hand-run iperf3 -c from a third location, etc.), pass --keep:

roksbnkctl test throughput --keep
# ... fixture stays up; debug to your heart's content ...
kubectl delete -n roksbnkctl-test pod/roksbnkctl-iperf3 svc/roksbnkctl-iperf3

The fixture is in the roksbnkctl-test namespace (same namespace the k8s backend uses for one-shot Jobs). It’s a bare Pod plus a Service; nothing else lingers when you delete the two resources.

Cross-references

The performance matrix

Throughput testing runs a single measurement: one iperf3 client against one iperf3 server. roksbnkctl test matrix runs a whole set of them at once. You write a matrix.yaml listing the measurements you want — each one a row — and the command runs them all and prints the results in one report.

Reach for test throughput to check a single path; reach for test matrix to characterise a deployment — throughput, request rates, and latency from several network locations — in one repeatable command instead of a dozen steps by hand. (Don’t confuse it with roksbnkctl testing up, which builds the jumphosts. test only runs tests.)

The matrix runs against an environment that is already deployed: a complete Cluster + BNK deploy with the Testing phase jumphosts up, and — for the HTTP tests — the Gateway phase in place.

The example numbers in this chapter are illustrative. The --dry-run plan is exercised in CI and its transcript is real, but a full live run has not yet been measured against a production ROKS cluster.

The two families

Each row (a “cell”) picks a family:

  • iperf3 — raw TCP throughput (Layer 4).
  • l7 — HTTP load, driven by h2load (Layer 7).

The runner turns each cell into a tool command and runs it from wherever you point it — locally, or on a jumphost over SSH.

iperf3 — TCP throughput

iperf3 measures plain TCP throughput against a host:port target (an L4Route VIP — BNK’s own L4 CRD, gateway.k8s.f5net.com/v1; the Gateway API standard channel BNK installs has no TCPRoute). The length field sets the TCP block size, which stands in for message size: "128" for tiny messages, "512K" for bulk transfer. The other knobs map straight to iperf3 flags:

Cell fieldiperf3 flagMeaning
length-lblock size ("128", "512K")
bytes-nsend a fixed number of bytes instead of running for a time
duration-trun length in seconds
streams-Pparallel TCP streams

A cell becomes, for example, iperf3 -c 10.240.0.10 -J -p 5201 -t 30 -P 8 -l 512K. The result reports throughput_gbps and retransmits (the same parser as Chapter 22).

l7 — HTTP load

h2load drives HTTP traffic at a URL. The scheme picks the path: an http:// URL hits the plain HTTP route; an https:// URL hits the TLS route, where TMM terminates TLS (h2load accepts the self-signed certificate, so there’s nothing extra to configure). The response size is whatever the URL returns — the test’s backend serves /128, /5k, and /512k, so you point a cell at the size you want.

h2load reports requests per second, a transfer rate, and request latency as min / max / mean (it does not report percentiles).

The l7.mode field is a shortcut that fills in sensible h2load flags; anything you set yourself overrides it:

ModeWhat it stressesDefaults (when you leave the knobs unset)
cpsnew connections — short connections, roughly one request each--h1, -m 1, -c 50, -n = clients×200
tpssustained requests — reuse connections, many requests each-c 50, -m 100 (or 1 with --h1), -n = clients×2000
throughputpayload — a few connections pulling a large body-c 8, -m 1, -n = clients×200

A cps cell becomes h2load -c 50 -m 1 --h1 -n 10000 http://10.240.0.10/128; a tps cell written as l7: { mode: tps, duration: 30 } becomes h2load -c 50 -m 100 -D 30 …. Set any of clients, streams, threads, requests, duration, http1 to override the defaults. The result reports req_per_sec, throughput_mbps, request counts (requests_total / succeeded / failed), status-code buckets, and req_time_{min,max,mean}_ms.

The locality axis = jumphost placement

There is no “location” setting. Where the traffic comes from is simply which jumphost a cell names as its client. Each vsi endpoint’s target is an SSH target name the Testing phase registered: jumphost (in the client VPC — a different VPC from the cluster), or jumphost-<zone> (in the cluster VPC, in the zone you pick).

To measure traffic from…Name this jumphost as the cell’s client
same VPC, same zone as TMMjumphost-<tmm-zone>
same VPC, a different zonejumphost-<other-zone>
a different VPCjumphost

The runner runs the tool on that jumphost over SSH (see Chapter 15 and Chapter 16). Both tools — iperf3 and h2load — are already installed on every jumphost, so there’s nothing to set up first. A cell with no client runs the tool locally.

The matrix.yaml schema

This is a workspace-sibling file — under ~/.roksbnkctl/<workspace>/, next to config.yaml but separate from it, so you can keep and diff a campaign’s grid in git on its own. The runner looks for --file, then <workspace>/matrix.yaml, then ./matrix.yaml. It has four top-level keys, shown here against the example grid (internal/test/testdata/matrix.example.yaml):

# gateway — names your already-deployed BNK gateway so the test's HTTP/TCP
# routes can attach to it. Only needed when fixtures.routes is true.
gateway:
  app_namespace: bnk-apps     # namespace the test's objects are created in
  name: bnk-gateway           # the existing Gateway the routes attach to
  http_section: http          # a listener on that Gateway, by section name
  https_section: https        # the TLS listener
  tcp_section: tcp            # the TCP listener (for the iperf3 route)

# fixtures — the temporary objects the test creates and then removes
# (kept in place if you pass --keep).
fixtures:
  iperf3_server: true         # the TCP server for the iperf3 cells
  http_backend: true          # an nginx server returning /128 /5k /512k
  routes: true                # the HTTP/TCP routes wiring the gateway to them

# endpoints — named (where, what) anchors the cells reference.
endpoints:
  vsi-same-zone: { kind: vsi, target: jumphost-jp-osa-1 }       # client in TMM's zone
  vsi-diff-zone: { kind: vsi, target: jumphost-jp-osa-3 }       # another zone, same VPC
  vsi-diff-vpc:  { kind: vsi, target: jumphost }                # the client VPC
  tmm-tcp:       { kind: address, host: 10.240.0.10, port: 5201 }  # iperf3 target
  tmm-http-128:  { kind: url, url: "http://10.240.0.10/128"  }
  tmm-https-512k:{ kind: url, url: "https://10.240.0.10/512k" }

# cells — the grid. Each is one row in the report.
cells:
  - { name: "L4 512K VSI->TMM diff-zone", family: iperf3, client: vsi-diff-zone, server: tmm-tcp, length: "512K", duration: 30, streams: 8 }
  - { name: "L7 http CPS 128B",  family: l7, client: vsi-diff-vpc, server: tmm-http-128,  l7: { mode: cps } }
  - { name: "L7 https THRU 512K", family: l7, client: vsi-diff-vpc, server: tmm-https-512k, l7: { mode: throughput, duration: 30 } }

gateway: — the existing-gateway identity

Names your already-deployed gateway so the test’s routes can attach to it. Only read when fixtures.routes is true.

FieldNotes
app_namespacenamespace the test’s objects are created in (required for routes)
namethe existing Gateway the routes attach to (required for routes)
http_section / https_section / tcp_sectionwhich listeners on that Gateway the http / https / tcp routes attach to, by section name. They must already exist. Leave one empty to skip that route.
class_name / controller_name / bnkgateway_name / flo_namespaceoptional descriptive fields; not needed to attach routes

endpoints: — named anchors

kindFieldsIs
vsitargeta jumphost the test sends traffic from (the client)
addresshost, port (default 5201)an iperf3 target — a host:port (an iperf3 cell’s server)
urlurlan HTTP target — a full http(s):// URL (an l7 cell’s server)

cells: — the grid

FieldApplies toNotes
namebothrequired; the report row label and the --only glob target
familybothiperf3 | l7
clientbotha vsi endpoint; empty → run locally
serverbothan address endpoint for iperf3, a url endpoint for l7
length / bytes / duration / streamsiperf3iperf3 -l / -n / -t / -P
l7.model7cps | tps | throughput (required for l7)
l7.{clients,streams,threads,requests,duration,http1}l7h2load -c / -m / -t / -n / -D / --h1; override the mode defaults

The file is checked before anything runs: a client must be a vsi with a target; an iperf3 server must be an address with a host; an l7 server must be a url and the cell must set a mode; and routes needs gateway.name, app_namespace, and at least one listener section.

What the test sets up (and cleans up)

For a real run the test creates a few temporary objects, runs the measurements, then deletes them again (unless you pass --keep). Three toggles under fixtures — all skipped under --dry-run:

  • iperf3_server — the TCP server the iperf3 cells measure against (the same one test throughput uses).
  • http_backend — an nginx server that returns fixed-size bodies at /128, /5k, and /512k for the HTTP cells.
  • routes — the HTTP and TCP routes (plus a TLS route and a self-signed certificate, when an https section is named) that wire your Gateway to those backends.

Everything the test creates is labelled app.kubernetes.io/managed-by: roksbnkctl-matrix, so cleanup is a single delete by label. The TLS certificate is self-signed and generated fresh each run — fine for a load test.

One prerequisite for the HTTP/TCP routes: your Gateway must already have the listener sections the routes attach to (http / https / tcp). The test adds the routes and backends, not the listeners — those come from the Gateway phase. If a named section doesn’t exist, the route has nowhere to attach and that cell’s traffic has no path. Run with --keep and check the route status if a cell isn’t reaching its backend.

The workflow

--dry-run — see the plan first

--dry-run expands the grid and prints, for each cell, the resolved client, server, and exact tool command — plus the objects it would create — and makes no cluster calls. Use it to confirm placement and commands before a long run:

$ roksbnkctl test matrix --dry-run --file matrix.example.yaml
→ matrix file: matrix.example.yaml
matrix plan — 10 cell(s)

[1] L4 128B  VSI->TMM same-zone
    family : iperf3
    client : ssh:jumphost-jp-osa-1
    server : 10.240.0.10:5201
    run    : iperf3 -c 10.240.0.10 -J -p 5201 -t 30 -l 128

[2] L4 512K  VSI->TMM same-zone
    family : iperf3
    client : ssh:jumphost-jp-osa-1
    server : 10.240.0.10:5201
    run    : iperf3 -c 10.240.0.10 -J -p 5201 -t 30 -P 8 -l 512K

[5] L7 http CPS  128B VSI->TMM
    family : l7
    client : ssh:jumphost
    server : http://10.240.0.10/128
    run    : h2load -c 50 -m 1 --h1 -n 10000 http://10.240.0.10/128

[8] L7 https CPS  128B VSI->TMM
    family : l7
    client : ssh:jumphost
    server : https://10.240.0.10/128
    run    : h2load -c 50 -m 1 --h1 -n 10000 https://10.240.0.10/128

--- fixtures (would apply) ---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: matrix-http-backend
  namespace: bnk-apps
  labels:
    app.kubernetes.io/managed-by: roksbnkctl-matrix
...
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: matrix-httproute
  namespace: bnk-apps
spec:
  parentRefs:
  - name: bnk-gateway
    namespace: bnk-apps
    sectionName: http
  rules:
  - backendRefs:
    - name: matrix-http-backend
      port: 80
...

(Abridged; the full plan prints all ten cells of the example grid and the complete set of objects.)

--only '<glob>' narrows the plan (or a real run) to cells whose name matches a glob — --only 'L7*' for just the HTTP rows, handy when iterating on one family.

The real run and the report

Drop --dry-run against a complete deployment. The runner creates the fixtures, runs each cell from its jumphost, removes the fixtures (unless --keep), and prints the report. -o selects the shape:

  • -o text (default) — the grid on stderr.

  • -o md — the grid on stdout. iperf3 rows show Gbit/s and retransmits; HTTP rows show req/s, Mbit/s, and mean latency:

    ## matrix (pass, 184230ms)
    
    | Cell | Family | Status | Throughput | req/s | mean ms | notes |
    |---|---|---|---|---|---|---|
    | L4 512K VSI->TMM diff-zone | iperf3 | pass | 9.21 Gbit/s | — | — | 312 retransmits |
    | L7 http CPS 128B VSI->TMM  | l7     | pass | 18.4 Mbit/s | 21340 | 2.34 | — |
    

    (Numbers illustrative.)

  • -o json — the roksbnkctl.v1 schema: one result per cell, each with its family-specific fields (plus the client / server it used). Exit code is non-zero if any cell failed, so it drops into CI:

    {
      "schema": "roksbnkctl.v1",
      "command": "test",
      "suite": "matrix",
      "cells": [
        {
          "suite": "l7",
          "name": "L7 http CPS 128B VSI->TMM",
          "status": "pass",
          "extra": {
            "req_per_sec": 21340.0,
            "throughput_mbps": 18.4,
            "req_time_mean_ms": 2.34,
            "client": "ssh:jumphost",
            "server": "http://10.240.0.10/128"
          }
        }
      ],
      "overall": "pass"
    }
    

Flag summary

FlagEffect
--file <path>the matrix.yaml (default: <workspace>/matrix.yaml, then ./matrix.yaml)
--only <glob>run only cells whose name matches the glob
--dry-runprint the plan and the objects it would create; no cluster calls
--keepleave the test’s objects in place after the run
-o json|text|mdreport shape

Cross-references

The E2E test plan

roksbnkctl ships a layered end-to-end test suite that exercises the full surface — install, lifecycle, four execution backends, internalised kubectl, the DNS probe, the cred-leak audit, and a mixed-mode lifecycle — against a live IBM Cloud account. This chapter is the user-facing guide: what the suite is, how to run it locally, what each phase validates, what it costs, and how it’s re-run when (not if) part of it flakes.

The design rationale lives in PRD 05; read that for the why. This chapter is the how and what.

What the E2E suite is

The suite is 14 automated phases organised into two tiers, plus Phase J as a manual integrator step (kubectl internalisation requires sudo mv of the host kubectl/oc binaries; that mutation is too disruptive to automate, so PRD 05 §J leaves it as a release-checklist item):

TierPhasesWhat it coversDriver script
BaselineA, B, C, D, E, F, G, Hinstall, init, plan, up, post-apply checks, test suites, downscripts/e2e-test.sh
Backends + extrasI, K, L, L-DNS, M, NSSH backend, docker backend, k8s backend + ops pod, DNS probe with GSLB compare, cred-leak audit, mixed-mode lifecyclescripts/e2e-test-backends.sh
ManualJkubectl internalisation (PATH-stripped, integrator-driven)per-release checklist

A combined driver, scripts/e2e-test-full.sh, runs both automated tiers in sequence: A-H first to bring up + exercise + tear down the baseline cluster, then I-N which provisions a fresh cluster via Phase N’s mixed-mode-lifecycle step. The two drivers stay decoupled — each can be run standalone — at the cost of an extra cluster apply (~70min wall-time, ~5-7h combined). Cluster-sharing across the two drivers (the PRD-envisioned design) is queued for v1.x; see PRD 05 §“Test infrastructure”.

Phase coverage at a glance

PhaseTierValidatesPRD
Abaselinedoctor + init—
Bbaselineplan (read-only)—
Cbaselinetargets list, registration01
Dbaselineup lifecycle — provision + deploy BNK—
Ebaselinepost-apply checks (status, k get, logs)02
Fbaselinetest connectivity (HTTP probes)—
Gbaselinetest throughput (iperf3)—
Hbaselinedown — destroy + cleanup—
IbackendsSSH backend / --on jumphost, host-key TOFU01
Jmanualkubectl internalisation (PATH-stripped — requires sudo mv)02
Kbackendsdocker backend (ibmcloud + iperf3 client)03 § Docker
Lbackendsk8s backend (ops pod + ibmcloud + iperf3)03 § K8s
L-DNSbackendsDNS probe + GSLB cross-vantage compare03 § DNS
Mbackendscred-leak audit (docker inspect, k8s events, ssh tempfiles)04
Nbackendsmixed-mode lifecycle (each tool on a different backend)all of the above

Phase J is an integrator-driven manual step; the per-release checklist in docs/E2E_TEST.md covers its procedure (PATH-strip kubectl + oc, then re-run the baseline driver’s Phase E to confirm roksbnkctl k get/apply/describe/... still works against the cluster).

How to run it locally

The three driver scripts all live under scripts/:

# Baseline only (A-H) — ~90 minutes
./scripts/e2e-test.sh

# Backends + extras only (I-N + L-DNS) — requires a live cluster (run after
# Phase D of e2e-test.sh, or provision via `roksbnkctl up` in a separate
# workspace first)
./scripts/e2e-test-backends.sh

# Combined — A-H baseline, then I-N + L-DNS against a fresh cluster the
# backends driver brings up via Phase N's mixed-mode-lifecycle step,
# ~5-7 hours total (two separate cluster applies)
./scripts/e2e-test-full.sh

Pre-requisites

Pre-reqRequired forNotes
IBM Cloud account with API keyevery phaseIBMCLOUD_API_KEY env var, or roksbnkctl init writes a keychain entry
terraform binary on PATHphases B-Hthe only strictly-required host tool for the baseline
Docker daemonphase Kdockerd or colima or Rancher Desktop — anything that publishes a docker socket
kind binaryphase L on CIthe in-CI k8s backend uses a kind cluster; on a real run it uses the ROKS cluster from D
An SSH bastion or jumphostphase I, Nprovisioned automatically by phase D’s terraform when testing_create_tgw_jumphost = true (the default)
Adequate disk for terraform plan outputphases B-H~200 MB for the embedded module’s state

Everything else (kubectl, oc, dig, iperf3) is internalised by the binary — phase J explicitly verifies the suite passes with kubectl and oc moved out of PATH.

Resuming a partial run

Every phase is re-runnable. The driver scripts respect a PHASE_FROM= env var:

# Restart from phase G (skipping A-F, which already ran)
PHASE_FROM=G ./scripts/e2e-test.sh

# Same for the backend driver
PHASE_FROM=L ./scripts/e2e-test-backends.sh

The phase pointer is read at startup and the script fast-forwards past every step before it. Assertion phases that hit external APIs (DNS resolvers, IBM Cloud control plane) include jitter and retry on the typical transient failure shapes — short DNS timeouts, IAM 5xx blips, etc. See PRD 05 §“Risks” for the retry policy.

Run logs

Each driver writes a single combined log per run:

  • scripts/e2e-test.sh → /tmp/roksbnkctl-e2e/run-<timestamp>.log
  • scripts/e2e-test-backends.sh → /tmp/roksbnkctl-e2e-backends/run-<timestamp>.log
  • scripts/e2e-test-full.sh → both of the above (the combined runner re-uses each child driver’s log directory)

Logs are preserved on both success and failure; clean them up manually when disk pressure warrants. On a CI machine that runs the suite nightly, the logs are the only forensics you get — keep them for at least 7 days. Per-phase log splitting is a v1.x consideration.

Dry-run

DRY_RUN=1 short-circuits every roksbnkctl invocation to a no-op that prints the command it would have run. Useful for re-validating the script wiring after edits without paying the 30-minute cluster-apply tax. The validator agent’s “is the test plan still well-formed” check runs in this mode.

What each phase validates

Phase A — init

roksbnkctl init prompts for region, resource group, cluster name, and BNK version, then writes ~/.roksbnkctl/<workspace>/config.yaml. The phase asserts the file exists, contains no plaintext API key (the rejection regex in internal/config/workspace.go catches that), and that roksbnkctl doctor reports green for terraform and informational for kubectl/oc.

Phase B — plan

roksbnkctl plan runs terraform init (downloads providers, ~30s) and terraform plan (computes the resource diff, ~30-60s on a clean workspace). No infrastructure is provisioned. The phase asserts the plan reports ~77 resources to add (the exact count is the upstream HCL’s full set of cluster + cert-manager + flo + cne_instance + license + testing resources).

Phase C — targets and registration

roksbnkctl targets list against a fresh workspace returns empty. roksbnkctl cluster register <existing-cluster> (optional, skipped if the workspace is provisioning new infra) ties an existing ROKS cluster’s COS instance + bucket discovery into the workspace.

Phase D — up lifecycle

The dominant cost phase. roksbnkctl up --auto runs terraform apply against the embedded HCL — provisioning ~77 resources: VPC, subnets, transit gateway, ROKS cluster, cert-manager, FLO, CNEInstance, License, jumphost. Expect 30-50 minutes on a clean apply, 5-15 minutes longer when IBM Cloud’s control plane is slow. The phase asserts terraform exits zero and the admin kubeconfig was fetched and written to $KUBECONFIG.

Post-apply, the phase auto-registers the jumphost target (per Chapter 16 §“Auto-discovery from roksbnkctl up”) so subsequent phases can --on jumphost without manual config.

Phase E — post-apply checks

roksbnkctl status shows the deployed BNK components. roksbnkctl k get nodes lists 3 worker nodes Ready. roksbnkctl logs flo (the F5 Lifecycle Operator) prints recent log lines. The phase asserts each command exits zero and that the cluster’s BNK install (FLO + CNE Instance + License) is in a healthy state.

Phase F — test connectivity

roksbnkctl test connectivity walks the workspace’s test.connectivity.extra_hosts list and probes each URL. Pass criteria: every URL returned a 2xx (or the expected status, when Chapter 20’s richer assertion shape lands).

Phase G — test throughput

roksbnkctl test throughput deploys the iperf3 server-pod fixture, runs a 30-second client measurement, and tears the fixture down. Pass criteria: bandwidth > 100 Mbps (a conservative floor — actual numbers on IBM Cloud are typically 1-5 Gbps), retransmits < 5% of streams, fixture removed afterwards.

Phase H — down

roksbnkctl down --auto runs terraform destroy. Pass criteria: all ~77 resources destroyed (terraform reports Destroy complete!), no orphan IBM Cloud resources detectable via roksbnkctl ibmcloud resource search.

Phase I — SSH backend / --on jumphost

roksbnkctl exec --on jumphost -- whoami returns root (the jumphost auto-provisioned with cloud-init’s root user). roksbnkctl ibmcloud --on jumphost iam oauth-tokens validates IBMCLOUD_API_KEY propagation over SSH. A negative test mutates ~/.roksbnkctl/known_hosts to a wrong fingerprint and asserts the next call exits 126 with a clear “host key mismatch” error. Phase I is the user-facing acceptance test for PRD 01.

Phase J — kubectl internalisation

The phase strips kubectl and oc out of PATH (via env-var sanitisation, not filesystem moves — see PRD 05 §“Open questions”) and verifies that roksbnkctl k get nodes, roksbnkctl k apply -f, roksbnkctl k describe, roksbnkctl k exec, roksbnkctl k port-forward, and roksbnkctl k delete all work against the cluster. A supplementary byte-equivalence step (run separately, not gated on PATH stripping) diffs kubectl get nodes -o yaml against roksbnkctl k get nodes -o yaml and asserts the diff (excluding managedFields, resourceVersion, creationTimestamp) is empty.

Phase K — docker backend

roksbnkctl ibmcloud --backend docker iam oauth-tokens pulls the roksbnkctl-tools-ibmcloud image on first call and runs the ibmcloud CLI inside it. The phase asserts the API key is not baked into the image (via docker history inspection) and not exposed in the running container’s env (via docker inspect). Chapter 17 §“docker backend” covers the credential-passing mechanism.

Phase L — k8s backend and ops pod

roksbnkctl ops install creates the roksbnkctl-ops namespace, deploys the long-lived ops pod, projects the IBM API key as a Kubernetes Secret, and binds the pod’s ServiceAccount to a least-privilege ClusterRole. Subsequent steps run roksbnkctl ibmcloud --backend k8s iam oauth-tokens (executes inside the ops pod) and roksbnkctl test throughput --backend k8s (the iperf3 server pod and client Job both run in-cluster). RBAC assertions confirm the SA can create Jobs in roksbnkctl-test but cannot delete Pods in default — least-privilege is enforced. Chapter 19 is the ops-pod reference.

Phase L-DNS — DNS probe and GSLB compare

The DNS phase exercises the miekg/dns-backed probe:

  • Single-vantage A and AAAA lookups against 8.8.8.8
  • NXDOMAIN negative test (asserts rcode=NXDOMAIN)
  • Iterated probe (10 queries to the same server, RTT p50/p95/p99 reported)
  • K8s-backend probe (runs as a Job in roksbnkctl-test, the binary self-execs in-cluster, RTT reflects in-cluster network path)
  • --server cluster (uses the pod’s /etc/resolv.conf, validates CoreDNS visibility)
  • --gslb-compare happy path (fans out local + k8s, asserts the answer schema)
  • --gslb-compare divergence (target a geo-resolved name where laptop and cluster IPs hit different DCs, asserts gslb_divergence: true)
  • Docker rejection negative (asserts the parse-time rejection error for --backend docker --target ...)

LD9 (SSH vantage) is exercised only when a jumphost is configured; LD5-LD8 are the must-pass set.

Phase M — cred-leak audit

Cross-cutting check that runs after I-L — confirms no credential value leaked during any prior phase. Concrete assertions:

  • docker history <ibmcloud-tool-image> — no IBMCLOUD_API_KEY=... ENV layer
  • docker inspect <last-container> — no API key value in env
  • kubectl get events -n roksbnkctl-ops -o yaml — grepping for the API key value returns nothing
  • kubectl logs <ops-pod> — grepping for the API key value returns nothing (the redactor masks any tool output that prints it)
  • ssh jumphost ls /tmp/roksbnkctl.* — empty (the SSH backend’s trap cleans up tempfiles on exit)
  • sshd auth.log — Accepted publickey lines present; the SetEnv var name (IBMCLOUD_API_KEY) is logged but not the value
  • ~/.roksbnkctl/*/state/*.log host-side logs — no API key value

The audit is the single most important gate on the v1.0 release. A leak in any of M1-M7 is a stop-ship. See PRD 04 for the threat model.

Phase N — mixed-mode lifecycle

A realistic scenario: workspace config routes each tool to its preferred backend, then a full up + test + down cycle runs end-to-end. Concretely, exec.terraform=local, exec.ibmcloud=ssh:jumphost, exec.iperf3=k8s — three different backends in one lifecycle. The phase asserts state is preserved across the per-tool dispatch (the workspace’s terraform state file is touched only by the local-backend terraform; the API key projected into the k8s ops pod is the same one resolved for the SSH dispatch) and that down cleanly destroys everything.

How CI runs it

.github/workflows/ci.yml runs unit + integration on every PR — go test ./... plus the testcontainers-go-backed integration tests. The full e2e suite is too expensive (4-6 hours, $5-10 of IBM Cloud spend per run) to gate on every PR.

The full e2e suite is run manually, out of band — on demand against main — rather than gated on every PR. It provisions real IBM Cloud infrastructure, so it is launched deliberately (with IBMCLOUD_API_KEY set) and its log artefacts archived against the run.

The release-cut policy is: don’t tag vX.Y.Z until the most recent manual-trigger run on main is green for three consecutive nights. (This project is trunk-based — there are no release branches. A per-BNK-release branch model was tried and withdrawn; what replaced it is the per-line terraform overlay in Chapter 13.) This catches the flakes that don’t reproduce locally — most of which are IBM Cloud control-plane blips rather than real regressions.

Cost and time

A full scripts/e2e-test-full.sh run currently costs:

ResourceApproximate cost (USD)
ROKS cluster (3 workers, ~5 hours uptime)$3-6
1-2 LoadBalancer Service objects (for north-south throughput)$0.50-1
COS instance + objects for the supply chain$0.10-0.20
Egress bandwidth (throughput tests, image pulls)$0.20-0.50
Total per run$5-10

Per-phase time estimates:

PhaseWall time
A (init)<1 minute
B (plan)1-2 minutes
C (targets / register)<1 minute
D (up)30-50 minutes (the dominant cost)
E (post-apply checks)1-2 minutes
F (connectivity)1-2 minutes
G (throughput)1-3 minutes
H (down)15-25 minutes
I (SSH backend)2-3 minutes
J (kubectl internal)3-5 minutes
K (docker backend)3-5 minutes (first call pulls the image, +1-2 minutes)
L (k8s backend + ops pod)3-5 minutes
L-DNS (DNS probe + GSLB compare)2-4 minutes
M (cred audit)<1 minute
N (mixed-mode lifecycle)30-50 minutes (full up + down again)
Total~4-6 hours

Phase N is the second-dominant cost — it runs a complete up/down cycle on top of D’s. Contributors who want a shorter test loop should skip N (PHASE_FROM= past it) and rely on D + I-M coverage; full N is a release-gate concern, not a per-PR concern.

Re-runnability

Every phase is re-runnable via PHASE_FROM=. The driver scripts are idempotent in two senses:

  1. Phase ordering: phases later than PHASE_FROM=<X> run unconditionally; phases at or before X are skipped. The script doesn’t try to remember whether earlier phases succeeded — that’s the user’s job (the per-phase log files are the evidence).
  2. Per-phase actions: each phase’s individual step calls are themselves idempotent where possible. roksbnkctl up on an already-applied workspace is a no-op (terraform plan reports zero changes). roksbnkctl ops install on an already-installed cluster is a no-op (the namespace + RBAC exist). The redactor + cred-resolver short-circuit cleanly on repeated invocations.

External-API steps include jitter+retry per PRD 05 §“Risks”: DNS resolvers occasionally return SERVFAIL on first query and succeed on the second, IBM IAM occasionally 5xxs during high-load periods, and the in-cluster ops pod can take a few seconds to be Running after ops install returns. Each of these is retried with a short exponential-backoff jitter rather than failing the phase.

The intended workflow on a flake is:

# Phase L flaked on "ops pod not yet Running"
# Re-run from L; everything before is preserved
PHASE_FROM=L ./scripts/e2e-test-backends.sh

If the same phase flakes on consecutive PHASE_FROM= runs, it’s a real bug — open an issue with the per-phase log attached.

Cross-references

Day-2 ops: status, logs, k get/apply/exec

This is the chapter to read after the cluster is up and BNK is deployed and you’re now living with the result. It opens with roksbnkctl status — the workspace-level read of what’s deployed — then covers the per-resource verbs: read pod state, tail logs, apply a manifest, port-forward to a service, exec into a pod. These per-resource verbs are native Go via client-go, so you don’t need kubectl on PATH for the everyday workflow.

The full design rationale lives in PRD 02. This chapter is the user-facing surface — the canonical “what’s the kubectl-equivalent in roksbnkctl?” reference.

Why internalise

Three reasons, in order of weight:

  1. Single binary. roksbnkctl is meant to be the one thing you install. The only required external prerequisite for the happy path is terraform. Everything else — kubectl, oc, iperf3, dig — is either built-in or an optional escape hatch.
  2. No version skew. The vendored client-go matches the kube API the bundled HCL targets. You can’t accidentally use kubectl 1.20 against a 1.28 cluster and have its print column heuristics go sideways.
  3. First-class output formats. cli-runtime gives byte-identical -o yaml/-o json/-o jsonpath output to kubectl. The validator agent’s golden-file tests in internal/k8s/golden_test.go assert this for representative resources.

roksbnkctl status

The one-shot read of a workspace’s posture. Always best-effort — every section reports its own missing pieces so a partial state still produces useful output rather than a hard error.

$ roksbnkctl status
Workspace:        canada-roks
Region:           ca-tor
Resource group:   default
Cluster:          canada-roks  (attach existing)
TF source:        jgruberf5/ibmcloud_terraform_bigip_next_for_kubernetes_2_3@v1.3.0
Cluster phase:    deployed (last apply 2026-05-13 14:08:33 MST)
BNK trial:        deployed (last apply 2026-05-13 14:15:01 MST)
Kubeconfig:       /home/you/.kube/config
Cluster:          2/2 nodes ready

The header rows (workspace, region, resource group, cluster identity, TF source pin, kubeconfig path, cluster reachability) are the same across every workspace shape. The per-phase deployment lines below them are shape-dependent: roksbnkctl status reads each phase’s terraform.tfstate mtime independently and emits one line per phase. The mapping from workspace shape to per-phase output:

The two Cluster: lines are by design: the first (in the header block) reports cluster identity — which cluster you’re targeting and whether the workspace creates it or attaches to an existing one. The second (the trailer line) reports cluster reachability — node count and ready count from a live API call. The label is reused because both pieces of information are about “the cluster”; the column to the right disambiguates.

TF source: reflects the workspace’s tf_source.type: github renders as <Repo>@<Ref> (the canonical happy-path shape — e.g., jgruberf5/ibmcloud_terraform_bigip_next_for_kubernetes_2_3@v1.3.0); local renders as local:<Path>; embedded or unset renders as (unset). The samples below use the github shape since it’s what most readers will see.

ShapeEmpty — fresh workspace, neither phase deployed

$ roksbnkctl status
Workspace:        dev
Region:           us-south
Resource group:   default
Cluster:          (unset)  (attach existing)
TF source:        jgruberf5/ibmcloud_terraform_bigip_next_for_kubernetes_2_3@v1.3.0
Cluster phase:    not deployed
BNK trial:        not deployed
Kubeconfig:       (none — run `roksbnkctl kubeconfig --download`)

Both phases report not deployed — the state directories either don’t exist or hold zero-resource state files. Running roksbnkctl cluster up (or roksbnkctl up for the monolithic path) advances the workspace to ShapeClusterOnly.

ShapeClusterOnly — cluster phase deployed, no BNK trial yet

$ roksbnkctl status
Workspace:        canada-roks
Region:           ca-tor
Resource group:   default
Cluster:          canada-roks  (attach existing)
TF source:        jgruberf5/ibmcloud_terraform_bigip_next_for_kubernetes_2_3@v1.3.0
Cluster phase:    deployed (last apply 2026-05-13 14:08:33 MST)
BNK trial:        not deployed
Kubeconfig:       /home/you/.kube/config
Cluster:          2/2 nodes ready

The Cluster phase line reads the mtime of <state-cluster-dir>/terraform.tfstate; the BNK trial line reads <state-dir>/terraform.tfstate and falls back to not deployed when the trial state is empty or missing. Running roksbnkctl bnk up advances the workspace to ShapeSplit.

ShapeSplit — both phases deployed (the steady state)

$ roksbnkctl status
Workspace:        canada-roks
Region:           ca-tor
Resource group:   default
Cluster:          canada-roks  (attach existing)
TF source:        jgruberf5/ibmcloud_terraform_bigip_next_for_kubernetes_2_3@v1.3.0
Cluster phase:    deployed (last apply 2026-05-13 14:08:33 MST)
BNK trial:        deployed (last apply 2026-05-13 14:15:01 MST)
Kubeconfig:       /home/you/.kube/config
Cluster:          2/2 nodes ready

Each phase has its own mtime; the timestamps move independently. Re-running roksbnkctl bnk down then roksbnkctl bnk up updates the BNK trial line without touching the Cluster phase line — useful for confirming which phase you most recently exercised.

The shape detection logic lives in internal/config/tfstate.go::DetectShape; the per-phase emission in runStatus is in internal/cli/inspect.go. See PRD 06 §“status command integration” for the design rationale.

The k command tree

All internalised verbs live under roksbnkctl k:

roksbnkctl k get          # fetch resources
roksbnkctl k describe     # human-readable detail
roksbnkctl k apply        # server-side apply from file/dir
roksbnkctl k delete       # delete with cascade options
roksbnkctl k logs         # pod or component logs
roksbnkctl k exec         # exec into a pod (SPDY)
roksbnkctl k port-forward # forward local ports to a pod (SPDY)

Two of those have top-level shortcuts for muscle-memory convenience — the verbs you’d type a hundred times a day:

roksbnkctl get  ↔  roksbnkctl k get
roksbnkctl logs ↔  roksbnkctl k logs

apply, exec, delete, describe, and port-forward only work under the k prefix.

Two verbs are deliberately not aliased to avoid shadowing existing top-level commands:

  • roksbnkctl apply is the existing top-level lifecycle verb that runs terraform apply against the workspace. Adding a second apply would shadow it and break roksbnkctl up / roksbnkctl apply muscle memory. Use roksbnkctl k apply -f ... explicitly for the Kubernetes-side server-side apply.
  • roksbnkctl exec runs a command on the host with the workspace’s env loaded (the host-exec verb — see Chapter 16, specifically the “Working examples” section). roksbnkctl k exec runs in a pod. The split keeps both meanings unambiguous without surprising name-collision behaviour.

kubectl/oc passthroughs stay as escape hatches

The roksbnkctl kubectl <args...> and roksbnkctl oc <args...> passthroughs are preserved alongside the internalised verbs. They shell out to the host binary (with the workspace’s KUBECONFIG and credentials loaded) for anything outside the internalised subset.

-w selects the cluster here too. The passthroughs resolve the same workspace-scoped kubeconfig and context the k verbs do, so -w acme-eu kubectl get nodes lists acme-eu’s nodes. This was not always true: before v1.42.0 they took the ambient KUBECONFIG and then fell back to ~/.roksbnkctl/forge/kubeconfig.yaml — a single file every workspace shares — so two workspaces on different clusters silently retargeted each other (#55). The output looked entirely believable, which is what made it dangerous for delete.

Two deliberate limits. An explicit --context or --kubeconfig in your arguments always wins — being told which cluster you mean is not something the tool overrides. And a workspace with no known cluster keeps the historical ambient behaviour rather than failing. If the workspace does know its cluster but no kubeconfig on disk can authenticate to it, you get the same error the k verbs give, telling you to run kubeconfig --download — rather than a silent fall-back to whatever the shared file points at.

roksbnkctl shell and roksbnkctl exec pin the same kubeconfig. Neither can pin a context, though — they hand you an arbitrary command line, and kubectl reads no environment variable for the context — so when the pinned file’s current-context names a different cluster they say so on stderr instead of letting the next command address it.

Across the --on SSH boundary nothing is pinned: a kubeconfig path and a context name are local facts that mean nothing on the remote host, which is exactly why that boundary scrubs them (Chapter 16).

When to reach for the passthrough:

Use caseWhy passthrough
kubectl rollout (status/history/undo/restart)Out of scope for v1.0; PRD 02 explicitly defers
kubectl scale / kubectl autoscaleOut of scope; passthrough is fine
kubectl edit / kubectl patchLow frequency for BNK ops; out of scope for v1.0
kubectl auth can-i / RBAC introspectionOut of scope; passthrough is fine
kubectl drain / cordon / taintCluster admin operations; not roksbnkctl’s role
kubectl run / kubectl createImperative resource creation; use k apply -f instead
oc adm / oc image / OpenShift admin verbsNiche enough to defer; passthrough
Niche flag combosAnything not in the internalised verb’s flag set

If kubectl is missing from PATH, the passthrough errors with:

Error: kubectl not on PATH; use `roksbnkctl k get/apply/...` for the in-process path,
       or install kubectl

Same for oc. The doctor check treats both as informational rather than warnings — see Chapter 5 — Doctor.

Worked examples

The verbs in everyday order. Every example below assumes roksbnkctl k and accepts the top-level alias where one exists.

roksbnkctl k get

The most-used verb. Resource type + optional name + optional flags:

# All pods in the default namespace
roksbnkctl k get pods

# Pods in a specific namespace
roksbnkctl k get pods -n f5-bnk

# Pods across all namespaces
roksbnkctl k get pods -A

# A specific pod by name
roksbnkctl k get pod flo-controller-abc123 -n f5-bnk

# Label selector
roksbnkctl k get pods -A -l app.kubernetes.io/name=f5-lifecycle-operator

# Cluster-scoped resources (no namespace)
roksbnkctl k get nodes
roksbnkctl k get storageclasses

Output formats — these match kubectl byte-for-byte:

roksbnkctl k get pods -n f5-bnk -o yaml
roksbnkctl k get pods -n f5-bnk -o json
roksbnkctl k get pods -n f5-bnk -o wide
roksbnkctl k get pods -n f5-bnk -o name
roksbnkctl k get pods -n f5-bnk -o jsonpath='{.items[*].metadata.name}'
roksbnkctl k get pods -n f5-bnk -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'

Plural / singular / shortname handling comes from the cluster’s RESTMapper via the discovery client, so pod, pods, po all work and pick up CRDs without a hardcoded list. roksbnkctl k get cneinstances (a BNK CRD) works as soon as the CRD is registered with the API server — no rebuild required.

Using the top-level alias:

roksbnkctl get pods -A

roksbnkctl k describe

Delegates to k8s.io/kubectl/pkg/describe — the same library kubectl uses internally. Output is identical to kubectl describe:

roksbnkctl k describe pod flo-controller-abc123 -n f5-bnk
roksbnkctl k describe node 10.243.0.4
roksbnkctl k describe service flo-webhook -n f5-bnk
roksbnkctl k describe cneinstance my-instance -n f5-bnk

The describe output’s “Events” section is especially useful for debugging stuck resources — pod scheduling failures, image pull errors, finaliser hangs all surface here.

roksbnkctl k apply

Server-side apply (SSA) with field-manager roksbnkctl. Inputs:

# Single file
roksbnkctl k apply -f pod.yaml

# Directory of YAMLs (recurses *.yaml)
roksbnkctl k apply -f manifests/

# Kustomize base (auto-detected if kustomization.yaml is present)
roksbnkctl k apply -f my-kustomize-base/

# stdin
cat pod.yaml | roksbnkctl k apply -f -

# Apply into a specific namespace (overrides metadata.namespace if absent)
roksbnkctl k apply -f manifests/ -n f5-bnk

# Force conflicts (SSA force-conflicts=true)
roksbnkctl k apply -f manifests/ --force

There is no top-level roksbnkctl apply alias for this verb — roksbnkctl apply is the lifecycle command that runs terraform apply. Always use roksbnkctl k apply for the Kubernetes-side apply.

Differences from kubectl apply:

  • Always SSA. Field manager is roksbnkctl. Client-side apply is not supported.
  • Kustomize auto-detect. A directory containing kustomization.yaml is built via sigs.k8s.io/kustomize/api before applying — no -k flag needed.
  • --force maps to SSA’s force-conflicts=true. Without it, conflicts with another field manager produce a clean error rather than silently winning.

For a vanilla kubectl apply -f workflow, the behaviour is functionally identical. For workflows that depend on client-side three-way merge or specific --server-side flag combinations, fall back to the passthrough.

roksbnkctl k delete

Cascade-aware deletion via the dynamic client:

# Delete by name
roksbnkctl k delete pod flo-controller-abc123 -n f5-bnk

# Cascade: orphan, background (default), foreground
roksbnkctl k delete deployment flo -n f5-bnk --cascade=foreground

# Force (bypass graceful deletion; immediate)
roksbnkctl k delete pod stuck-pod -n f5-bnk --force

# Custom grace period (seconds)
roksbnkctl k delete pod my-pod -n f5-bnk --grace-period=5

Use --cascade=foreground when you want to wait for owned resources (Pods owned by a Deployment, etc.) to be deleted before the parent disappears — useful for tearing down BNK trial CRs cleanly so finalisers run in order.

roksbnkctl k logs and roksbnkctl logs

Two paths, one verb. The component-aware path is a BNK-specific label-selector shortcut; the raw pod-name path takes a pod name directly.

Component-aware (by label selector):

roksbnkctl logs flo                # F5 Lifecycle Operator (label selector under the hood)
roksbnkctl logs cis                # F5 BNK CIS controller
roksbnkctl logs cert-manager       # cert-manager
roksbnkctl logs cneinstance        # BIG-IP TMM data plane pods

Raw pod-name (by pod name):

roksbnkctl k logs flo-controller-abc123 -n f5-bnk

Common flags (both paths):

-f, --follow              # stream live (kubectl logs -f)
-c, --container <name>    # specific container in a multi-container pod
--previous                # logs from the previous instance (after a crash)
--since=10m               # only logs in the last 10 minutes
--tail=100                # last N lines only

Top-level alias:

roksbnkctl logs flo -f --since=5m

If the named first arg matches one of the well-known BNK components (flo, cis, cert-manager, cneinstance), the component-aware path is used; otherwise it’s treated as a pod name. The component map lives in internal/cli/inspect.go and is keyed off the upstream chart’s default labels.

roksbnkctl k exec

SPDY exec into a pod. Same semantics as kubectl exec:

# One-shot command
roksbnkctl k exec flo-controller-abc123 -n f5-bnk -- ls -la /

# stdin attached
roksbnkctl k exec flo-controller-abc123 -n f5-bnk -i -- cat /etc/hostname

# Interactive PTY (the bash-style use)
roksbnkctl k exec flo-controller-abc123 -n f5-bnk -i -t -- bash

# Specific container in a multi-container pod
roksbnkctl k exec flo-controller-abc123 -n f5-bnk -c sidecar -- env

The -i and -t flags map directly to kubectl exec’s -i (stdin) and -t (PTY). For top / bash / interactive Python sessions, pass both.

There is no roksbnkctl exec (top-level) alias — roksbnkctl exec runs on the host. See “Disambiguating roksbnkctl exec” in PRD 02.

roksbnkctl k port-forward

SPDY port-forward to a pod:

# Forward localhost:8080 → pod's :80
roksbnkctl k port-forward flo-controller-abc123 -n f5-bnk 8080:80

# Multiple ports
roksbnkctl k port-forward flo-controller-abc123 -n f5-bnk 8080:80 8443:443

# Random local port (let the kernel pick)
roksbnkctl k port-forward flo-controller-abc123 -n f5-bnk :80

Ctrl+C closes the tunnel cleanly — no orphaned local listeners. The forward survives idle (reads/writes are bidirectional); it’s torn down only on signal or pod restart.

For a Service rather than a Pod, port-forward via the Service’s underlying pod or use kubectl port-forward svc/<name> through the passthrough — Service-targeted port-forwarding is currently passthrough-only.

Output format compatibility

The biggest user-visible promise: -o yaml / -o json / -o wide / -o jsonpath produce the same bytes as kubectl, modulo a small set of timestamp-and-resourceVersion fields that change between calls.

Concretely, the validator agent’s golden-file tests at internal/k8s/golden_test.go capture kubectl get <resource> -o yaml and roksbnkctl k get <resource> -o yaml against a live ROKS cluster and diff them, ignoring:

  • metadata.managedFields (ordering varies between callers; not user-visible)
  • metadata.resourceVersion (monotonic counter; changes on every read)
  • metadata.creationTimestamp (set server-side; not under our control)

Anything else differing is a test failure. The covered resources are Node, Pod, Service, ConfigMap — representative both of cluster-scoped (Node) and namespace-scoped (Pod, Service, ConfigMap), and of the typed-client (Node, Pod, Service) and dynamic-client (anything via cli-runtime’s resource.Builder) paths.

Run them locally with:

make test-live

…against a KUBECONFIG that points at a real ROKS cluster. They’re not part of the unit-test CI run because they need a live cluster; the integrator runs them before tagging a release. Documented in CONTRIBUTING.md.

OpenShift extensions

Beyond the core kubectl-equivalent verbs, ROKS clusters surface OpenShift-specific resource types — Project, Route, ImageStream, BuildConfig. roksbnkctl k get discovers these natively today via the dynamic client + RESTMapper path (the cluster advertises them through the API discovery doc; the deferred-discovery mapper picks them up):

roksbnkctl k get projects                    # OpenShift projects (vs Kubernetes namespaces)
roksbnkctl k get routes -n f5-bnk            # OpenShift Routes (vs Ingress)
roksbnkctl k get imagestreams -n f5-bnk      # OpenShift ImageStreams
roksbnkctl k get buildconfigs                # BuildConfigs (mostly empty in BNK trials)

Same verb shape (get / describe / delete); the dynamic-client + RESTMapper combination handles type discovery without needing a per-type Go-side scheme registration.

Phase 2.1 of PRD 02 adds typed clients via github.com/openshift/client-go for nicer printing and describe integration of these resources. This is on the v1.x roadmap (see docs/PLAN.md §“What’s deliberately deferred to post-v1.0”). Until typed clients land, roksbnkctl k get/describe still works against OpenShift CRDs — just with the generic unstructured printer. If you want richer per-type output today, fall back to the oc passthrough:

roksbnkctl oc get projects                   # typed-client output today
roksbnkctl oc describe route f5-bnk-svc      # typed Route fields

How doctor treats kubectl / oc

How roksbnkctl doctor reports the host binaries (covered in Chapter 5):

  • kubectl — informational (no warning when missing).
  • oc — also informational (no warning when missing).

A fresh dev box without kubectl / oc installed should run roksbnkctl doctor and see green-or-informational across the board for the everyday workflow. The host binaries aren’t required for the everyday workflow; they’re nice-to-have for the passthroughs.

kubectl muscle-memory cheat sheet

A reader migrating from kubectl should be able to use this section as a Rosetta Stone:

kubectl ...roksbnkctl ...
kubectl get podsroksbnkctl get pods (or roksbnkctl k get pods)
kubectl get pods -Aroksbnkctl get pods -A
kubectl get pods -o yamlroksbnkctl get pods -o yaml
kubectl describe pod <name>roksbnkctl k describe pod <name>
kubectl apply -f manifests/roksbnkctl k apply -f manifests/
kubectl apply -k overlay/roksbnkctl k apply -f overlay/ (auto-detects kustomize)
kubectl delete pod <name>roksbnkctl k delete pod <name>
kubectl logs <pod> -froksbnkctl logs <pod> -f (or roksbnkctl k logs <pod> -f)
kubectl exec -it <pod> -- bashroksbnkctl k exec <pod> -i -t -- bash
kubectl port-forward <pod> 8080:80roksbnkctl k port-forward <pod> 8080:80
kubectl rollout status deploy/fooroksbnkctl kubectl rollout status deploy/foo (passthrough)
kubectl edit deployment fooroksbnkctl kubectl edit deployment foo (passthrough)
kubectl scale deployment foo --replicas=3roksbnkctl kubectl scale deployment foo --replicas=3 (passthrough)
oc projectsroksbnkctl k get projects (works today via dynamic-client) or roksbnkctl oc projects for typed-client output

The general pattern: if it’s get / describe / apply / delete / logs / exec / port-forward against a typed or unstructured Kubernetes resource, the internalised verb is the right answer. Anything else, fall back to the passthrough.

Cross-references

Registering the cluster with BNK Forge

BNK Forge is a separate platform — a FastAPI/React app — for operating BIG-IP Next for Kubernetes across a fleet of clusters. When you run BNK Forge co-located with your ROKS deployments, roksbnkctl can hand each cluster it provisions straight to BNK Forge, so the cluster shows up in the Forge fleet the moment cluster up finishes.

This is opt-in and best-effort: it never blocks or fails a deploy. If you don’t enable it (the default), nothing changes.

Quick start

Three commands cover everything. None of them edits config.yaml by hand — enable writes the opt-in for you, exactly like registry target or targets add elsewhere in the tool:

roksbnkctl -w acme-eu bnkforge enable     # auto-register on every `cluster up`
roksbnkctl -w acme-eu bnkforge register   # register the current cluster right now
roksbnkctl -w acme-eu bnkforge status     # show config + readiness (url, session, cluster id)

enable turns on the post-cluster up hook; register is the on-demand path (no re-up needed) for a cluster that already exists. Both talk to BNK Forge v3’s REST API directly — there is no external CLI to install; see Prerequisites.

Why credential-backed

The naive way to register a cluster with an external platform is to hand it a kubeconfig. ROKS admin kubeconfigs are short-lived — they carry an embedded token that expires — so a stored kubeconfig goes stale and the platform loses access until someone re-uploads a fresh one.

roksbnkctl registers credential-backed instead. It points BNK Forge at an IBM Cloud credential template (your IBM Cloud API key, stored once in Forge) plus the cluster’s id and region. BNK Forge then re-derives the kubeconfig on demand from that credential whenever it needs cluster access. Nothing perishable is stored, so the registration doesn’t go stale.

roksbnkctl                      BNK Forge
-----------                     ---------
cluster id + region   ───────▶  credential template (IBM Cloud API key)
                                        │
                                        ▼
                                derive kubeconfig on demand (never stale)

Prerequisites

  • A co-located BNK Forge install that supports IBM ROKS credential-backed registration — i.e. a Forge that can re-derive a ROKS kubeconfig from an IBM Cloud credential template. (This backend support is being added on a BNK Forge branch; an older Forge that only accepts a static kubeconfig won’t work with this flow.)

  • The Forge URL and credentials. BNK_FORGE_URL, BNK_FORGE_USER and BNK_FORGE_PASSWORD (or bnkforge.url / --url / --username). The session token that results is cached in the OS keychain and reused until it expires, so the password is normally used once. roksbnkctl bnkforge status shows what is resolved.

    The password can also be stored as bnkforge.password_b64, base64-encoded, the same shape as every other machine credential this tool takes (ibmcloud.api_key_b64, bnk.cis.bigip_password_b64, registry.generic_password_b64). Precedence is BNK_FORGE_PASSWORD, then --password, then the field, then the prompt — so a stored value never prevents a runner overriding it.

    base64 is obfuscation, not encryption. Anyone holding the file holds the password. Storing it trades a long-lived secret on disk for not being asked, which is the right trade for an unattended runner and the wrong one for a laptop. Use scripts/generate_b64_password.sh to produce the value without it reaching your shell history:

    $ ./scripts/generate_b64_password.sh bnkforge.password_b64
    Password:
    Re-enter:
    
      # config.yaml — under the bnkforge block
      password_b64: aHVudGVyMg==
    

    On a terminal, bnkforge register prompts for anything missing. The automatic post-up hook runs non-interactively: with no cached token and no env credentials it declines with a one-line note rather than blocking a deploy on a prompt.

  • An IBM Cloud API key resolvable for the workspace (the usual env / keychain / config chain — see Chapter 14). It becomes the credential template Forge derives kubeconfigs from.

No external CLI. Nothing is looked up on PATH, and there is no ~/.bnk-forge/config.json. Earlier roksbnkctl versions shelled out to a v1 bnk-forge binary; v3 has no such CLI, so roksbnkctl speaks to the REST API itself.

The bnkforge commands

CommandWhat it does
bnkforge enable [--url U] [--project P]Persist register: true (+ optional overrides) so every cluster up registers the cluster.
bnkforge disablePersist register: false — turn the auto-hook off. Local only: it does not unregister anything.
bnkforge statusShow the effective config (register / url / project), whether a Forge session token is cached, and whether a cluster id is recorded.
bnkforge register [--url U] [--project P] [--force]Register the current workspace’s cluster now, regardless of the opt-in. Surfaces errors (the auto-hook swallows them). --force takes over a cluster held by another Forge project — see Registration is non-destructive.
bnkforge unregister [--url U] [--project P]Remove the cluster from its Forge project — the inverse of register. Every “not there” case exits 0, so it is safe on a teardown path.
bnkforge ssh-credential --host H --key KGive Forge the SSH private key for an appliance this workspace built (the FLP), and turn on the project’s infrastructure access. See Giving Forge an appliance’s SSH key.

enable, disable, and register all accept --url (override the Forge URL) and --project (target Forge project). On enable those are persisted; on register they’re one-off overrides that aren’t written to config.yaml.

What enable writes

enable writes a bnkforge: block into the workspace config.yaml for you (the same marshaller every other roksbnkctl command uses — no hand-edit):

bnkforge:
  register: true
  url: https://forge.example.com   # only if you passed --url
  project: "42"                    # only if you passed --project
FieldTypeDefaultSet by
registerboolfalsebnkforge enable / disable. false / omitted ⇒ no-op.
urlstring(BNK_FORGE_URL)bnkforge enable --url. Falls back to the BNK_FORGE_URL env var.
projectstring(the workspace name)bnkforge enable --project. Empty ⇒ a Forge project named after the workspace, created if absent.

The block is additive — an absent bnkforge: block (any existing config.yaml) loads unchanged.

What happens on cluster up

Forge kubeconfig for module-based registration. Independently of the credential-backed flow below, every cluster up also writes a portable, self-contained cert-based kubeconfig to $ROKSBNKCTL_HOME/forge/kubeconfig.yaml (one cluster entry with the public server + CA-if-any, and the cluster’s admin client certificate/key). ROKS is OpenShift: its API server authenticates via client certs or OAuth tokens and rejects raw IBM IAM bearer tokens (401), so the forge kubeconfig carries the admin client cert rather than a token. A BNK Forge that registers from a declared kubeconfig file can point at this path; roksbnkctl keeps it current automatically by re-fetching the admin kubeconfig as the certs near expiry (see kubeconfig --refresh). This is separate from, and complementary to, the credential-backed bnkforge register flow documented below.

When register: true, roksbnkctl runs a post-apply hook right after it fetches the admin kubeconfig (the same spot whether the apply made changes or was a no-op). The hook is exactly what bnkforge register runs, with errors swallowed. It:

  1. Checks the opt-in. No bnkforge.register: true ⇒ it does nothing.
  2. Reads the cluster identity from cluster-outputs.json (cluster id, region, name). No recorded cluster id yet ⇒ skipped with a note.
  3. Resolves the IBM Cloud API key non-interactively (env → keychain → config). This is what gets stored as the credential template, so Forge can re-derive the kubeconfig on demand instead of holding a perishable one.
  4. Authenticates to Forge over its REST API. A cached session token from the OS keychain is reused when still valid; otherwise it logs in with BNK_FORGE_USER / BNK_FORGE_PASSWORD — or bnkforge.password_b64 — and caches the new token. The auto-hook runs non-interactively, so with no valid token, no env credentials and no stored password it declines rather than blocking a deploy on a prompt.
  5. Ensures the credential template and the project, then registers the cluster by id + region + template.

Sample output on a successful run:

→ Registering cluster "acme-eu-roks" with BNK Forge (https://forge.example.com)…
✓ Registered cluster "acme-eu-roks" with BNK Forge (project "acme-eu", forge cluster id 17).

Registration is non-destructive

Re-registering used to DELETE any same-named cluster and re-POST it. Within one project that was called idempotent, and very nearly is — except the cluster id changed, so anything referencing it broke and any scan history attached to the old id was discarded. Across projects it was worse: the lookup was project-scoped, so a cluster held by another project was invisible, and the POST either moved it silently or failed with a bare exit 1 that named nothing.

Since v1.42.0 (#54):

The cluster name is held byWhat happens
this projectUpdated in place — the cluster id is preserved
another projectRefused, naming the owning project and both ids. --force takes it over
nobodyCreated
$ roksbnkctl -w acme-eu bnkforge register
Error: cluster is registered to another BNK Forge project: "acme-eu-roks" is held by
project "platform-team" (id 93, cluster id 35). Re-registering would move it, changing
its cluster id and removing it from that project's view. Pass --force to take it over
deliberately, or unregister it there first

--force is a real move — the cluster is removed from the owning project and re-created in yours, so its id changes. That is inherent to moving it, and is what --force opts into.

The automatic post-up hook never forces. An unattended step silently taking another project’s cluster is precisely the harm the refusal exists to prevent. Only an explicit roksbnkctl bnkforge register --force can do it.

Two behaviours worth knowing when Forge is older or your account is narrowly scoped:

  • An older Forge build with no PUT for the cluster resource falls back to the historical delete-and-recreate, so registration still works; the id changes, which is the cost of the older server. The fallback fires only on 404/405 — a transient 500 is reported as an error rather than escalated into a destructive retry.
  • A project you cannot read produces a warning, not a failure. The cross-project scan is a second opinion on top of a direct query of your project; refusing to register because some other project was unreadable would block a least-privilege Forge account from its own work.

Registering on demand (no re-up)

roksbnkctl bnkforge register registers the current workspace’s cluster right now, without enabling the auto-hook or re-running cluster up. Use it to register a cluster that was provisioned before you enabled the feature, or one you registered with an existing-cluster record:

roksbnkctl -w acme-eu bnkforge register
roksbnkctl -w acme-eu bnkforge register --project 42   # one-off project override

It reads the cluster id + region from cluster-outputs.json, resolves your IBM Cloud API key the usual way, and runs exactly what the post-up hook runs — the only difference is that errors are surfaced rather than swallowed.

Unregistering on teardown

bnkforge disable reads like the inverse of register, but it is not: it clears the local register flag and never contacts the server. A cluster already registered stays on Forge’s Kubernetes page, in a project that outlives everything it was created alongside, pointing at a cluster that may no longer have BNK on it — or may no longer exist.

roksbnkctl bnkforge unregister is the actual inverse:

roksbnkctl -w acme-eu bnkforge unregister
✓ unregistered cluster "acme-eu" (id 13) from BNK Forge project "acme-eu"

Absence is success. A destroy runs when things are already half dismantled, and may run twice, so each of these reports and exits 0 rather than failing:

SituationOutput
No project of that name✓ no BNK Forge project "acme-eu" — nothing to unregister
Project exists, cluster does not✓ cluster "acme-eu" is not registered in project "acme-eu" — nothing to do
Cluster already deleted (404)treated as done

Only a genuine server failure surfaces as an error. Treating absence as failure would make the command unusable in exactly the situation it exists for.

It also never creates anything. register ensures the project exists; a teardown asking “is this still here?” must not bring it into being, or it would leave behind the very thing being removed.

Registration is not removed automatically on cluster down. The post-up hook registers, but there is no matching hook on the way down — run bnkforge unregister as part of your teardown (this is what the BNK Forge blueprints do in their destroy step).

Doing it by hand against the REST API

There is no CLI to drive, but the flow is four ordinary REST calls, so it is reproducible with curl when you need to see exactly what Forge receives. Look up the cluster’s identity first:

roksbnkctl -w acme-eu cluster show     # cluster_id, region, cluster_name
StepCall
1. Log inPOST /api/auth/login → {"token": …}
2. Credential templateGET/POST /api/credential-templates — provider must be lowercase ibm, your API key, plus region and ibm_cos_instance_name
3. ProjectGET/POST /api/projects, then PUT /api/projects/{id} to set the ROKS/IBM platform
4. RegisterPOST /api/projects/{id}/k8s/clusters, or PUT /api/k8s/clusters/{id} to update in place

Step 3’s PUT matters: without it Forge shows the project’s platform as Unknown. Step 4 is the one that changed in v1.42.0 — roksbnkctl prefers the in-place PUT so the cluster id survives, and only falls back to DELETE + POST against a Forge build that has no PUT route.

Giving Forge an appliance’s SSH key

Registering the cluster is not the same as giving Forge a way into the appliances the workspace builds. Without the second step a perfectly healthy F5 License Proxy reports itself unreachable:

infrastructure_private_key_available: false
infrastructure_access_status:        recovery_required
infrastructure_access_message:       Infrastructure SSH private key is unavailable.
                                     Run state recovery/repair to rematerialize
                                     access metadata.

Nothing can be recovered there. The credential was never created.

The two “SSH keys” are different things

This is the part that makes the state above confusing:

  • bnk.flp.vsi.ssh_key names an existing IBM Cloud VPC key, which puts a public key on the VSI. That is your access to the appliance, and it works.
  • Forge separately needs the private half, so it can reach the appliance itself. Nothing supplies that as part of building the VSI.

Registering it

roksbnkctl -w acme-eu bnkforge ssh-credential \
  --host 52.116.1.2 \
  --ssh-username ubuntu \
  --key ~/.ssh/id_rsa
FlagMeaning
--hostThe address Forge reaches the appliance on. For an FLP that is the floating IP.
--ssh-usernameThe SSH user on the appliance (default ubuntu). Note the prefix: --username is the Forge login, as on every other bnkforge subcommand.
--keyThe private key file. Unencrypted — Forge stores the key and cannot prompt for a passphrase.
--expect-fingerprintOptional SHA256 fingerprint the key must match before anything is sent (e.g. the VPC key’s).
--nameCredential name in Forge. Default <project>-ssh.
--portSSH port. Default 22.

Use the floating IP, not the endpoint

flp status reports something like https://10.243.1.4:8443. That is a services-VPC address, and Forge sits outside that VPC — a credential built from it can never connect. Passing a URL is refused rather than accepted:

--host "https://10.243.1.4:8443" looks like an endpoint URL, not a host
  Pass the bare floating IP or hostname; the port is separate (--port)

Why the key is checked before it is stored

A credential that cannot log in is worse than no credential: Forge then reports infrastructure access as configured, and every later failure points somewhere other than the key. Two checks stand in the way:

  • --expect-fingerprint compares the private key’s SHA256 fingerprint — the same value ssh-keygen -l -E sha256 and IBM Cloud print — locally, before anything leaves the machine.
  • Forge itself tests the SSH connection before storing, and answers 400 when it fails. That is the stronger of the two: the fingerprint proves the key is the one the VPC knows about, the connection test proves it opens the box.

What it writes

StepCall
1. The credentialGET/POST/PUT /api/ssh-credentials — name, host, port, user, and the private key
2. Infrastructure accessPOST /api/cloud-auth/ssh/configure — sets infra_enabled, infra_host, infra_ssh_username, infra_auth_type
3. The project linkPUT /api/projects/{id} — sets ssh_credential_id

Step 2 is the one that is easy to miss. PUT /api/projects/{id} accepts ssh_credential_id and ignores the infra_* fields — they are not that route’s to set. With only the credential linked, infra_enabled stays false and the appliance is still unreachable.

The command reads the project back afterwards and names anything that did not stick, rather than reporting success on a write it did not verify.

TLS: pin the CA, don’t disable verification

Forge installs commonly carry a self-signed certificate, and --insecure makes that connect. What it costs is easy to miss: the session token is sent on every request, so disabling verification leaves the connection encrypted but unauthenticated. Anyone positioned on the path can present a certificate for the Forge host, terminate TLS, and read the token. Encryption without authentication protects the traffic from a passive observer and not from an active one.

That would matter less if the setting were transient, but bnkforge.insecure: true persists in config.yaml — it is typically set once for a lab and then forgotten, including when the same workspace is later pointed at a production Forge. So every request made with it prints a warning naming the host:

⚠ BNK Forge TLS verification is DISABLED for forge.lab.example.com:8443 — the API token
  is sent over a connection that is encrypted but NOT authenticated.
  Anyone able to intercept it can present any certificate for that host and read the token.
  Pin the Forge CA instead — `roksbnkctl bnkforge enable --forge-ca <file>` — then drop --insecure.

Pin the CA instead. For a self-signed Forge you generated the CA, so you already hold it — and pinning it authenticates the connection rather than abandoning authentication:

roksbnkctl bnkforge enable --url https://forge.lab.example.com:8443 --forge-ca ./forge-ca.pem

That stores the PEM as bnkforge.ca_b64 (base64 only for single-line YAML safety — a certificate is public data) and clears insecure. --forge-ca also works as a one-shot on bnkforge register and unregister. When both are set the pin wins and the certificate is verified.

Non-interactively, set ROKSBNKCTL_BNKFORGE_CA_B64 (base64 -w0 forge-ca.pem).

roksbnkctl bnkforge status reports which of the three is in force:

tls:         verified (pinned CA from bnkforge.ca_b64)
tls:         verified (system roots)
tls:         NOT VERIFIED (insecure: true) — the API token is sent over an unauthenticated connection; pin a CA with `bnkforge enable --forge-ca <file>`

If the pinned CA is wrong, the connection fails at the handshake with an x509 error rather than silently falling back — which is the point.

Behaviour notes

  • Best-effort — it never blocks or fails the deploy. Whatever happens with Forge, cluster up succeeds if the cluster came up. Worst case you get a one-line note and register later with roksbnkctl bnkforge register.
  • Re-registering preserves the cluster id. Running cluster up again on an unchanged cluster re-runs the hook harmlessly: the cluster is updated in place, so references to its Forge id and any scan history attached to it survive. Before v1.42.0 the id changed on every re-register (#54).
  • The auto-hook never forces. A cluster held by another Forge project is left alone and the note says so; only an explicit bnkforge register --force moves it.
  • Registering later. If it was skipped (no session, no credentials, no TTY), fix the prerequisite and run roksbnkctl bnkforge register — no re-up needed.

Troubleshooting

  • no BNK Forge URL (set bnkforge.url, BNK_FORGE_URL, or --url) — nothing told roksbnkctl where Forge lives. roksbnkctl bnkforge status shows what resolved.

  • no BNK Forge username / no BNK Forge password — the cached session token is absent or expired and there is nothing to log in with. Set BNK_FORGE_USER and BNK_FORGE_PASSWORD, store bnkforge.password_b64, or run roksbnkctl bnkforge register at a terminal to be prompted. On a CI runner the env var and the stored field are the only paths — there is no prompt to fall back to.

  • cluster is registered to another BNK Forge project — the name is held by a different project; the message names it and both ids. Either unregister it there, or pass --force to move it deliberately (its cluster id changes, because a move re-creates it). This refusal is the point: silently taking another project’s cluster is the harm.

  • Forge shows the credential template but nothing uses it — check its provider. Forge matches provider == "ibm" case-sensitively in several places and lowercases none of them, so a template created with "IBM" is accepted by the API and then matches nothing: credentials are never injected into a deployment, blueprint inputs sourced from credential_template never resolve, and the “IBM templates must carry an API key” validation never fires. Nothing errors. roksbnkctl wrote "IBM" before v1.56.0; re-running bnkforge register with a current build repairs an existing template rather than only fixing new ones.

  • HTTP 403 / lacks permission to manage credential templates — your BNK Forge session’s role can’t create or list credential templates. Log in as an operator/admin, or have one pre-create the template.

    A 403 on a project’s cluster list is different and is not fatal: the cross-project ownership scan warns and continues, so a least-privilege account can still register into its own project.

  • no cluster id recorded in cluster-outputs.json yet — the cluster identity wasn’t written. Run roksbnkctl cluster register <name> (or re-run cluster up) so cluster-outputs.json is populated, then register.

Cross-references

Two behaviours that exist because of how Forge drives this tool

The Forge modules run one container per roksbnkctl phase, argv-only, each step re-running init --override-from-env --non-interactive from a curated environment over a deployment-scoped shared /work, against a digest-pinned runner image. Two of roksbnkctl’s checks are shaped by that, and both would otherwise break blueprints:

An unknown BNK line warns instead of refusing. manifest_version is a free-text input on the bnk-install module, the support matrix ships inside the binary, and the image is pinned by digest — so a user selecting a release newer than the pinned build would hit a hard refusal on a combination never known to be wrong, with no action available to them. It warns and proceeds instead.

An unset network_mode defers to the recorded cluster. Each step’s environment carries only what that step needs, so a mode set by cluster-create is absent when bnk-install regenerates config.yaml. Reading that absence as a demand for single-nic would refuse a correct multi-NIC deployment at its second module. Only an explicit contradiction is refused.

The practical rule for blueprint authors: only the module that CREATES a cluster needs to know the network mode. Everything downstream reads the record.

COS supply chain management

BIG-IP Next for Kubernetes (BNK) pulls its runtime artefacts — the F5 Application Runtime (FAR) container images, the JWT licence used at install + renewal time, the f5-bigip-k8s-manifest Helm chart, and the schematic JSON the deployer renders — from IBM Cloud Object Storage (COS). The COS bucket is the supply chain: it’s how artefacts produced upstream (F5 build pipeline, licence-issuing service, schematic generator) reach the cluster.

roksbnkctl cos is the management surface for that supply chain. Three command levels — cos instance, cos bucket, cos object — cover the full CRUD on COS resources without touching the ibmcloud CLI; everything (most visibly cos object put for uploads and cos object get for downloads) goes through the IBM Cloud Go SDKs (go-sdk-core, platform-services-go-sdk, ibm-cos-sdk-go).

What COS is in this stack

COS is IBM’s S3-compatible object store. Two layers matter here:

  • Instance: a service instance under Resource Controller. Instances are global; they don’t pin to a region. A workspace typically uses one COS instance per environment (dev, staging, prod) and shares it across multiple buckets.
  • Bucket: an S3-style bucket with regional affinity, hosted on a COS instance. Buckets carry the storage class (standard, vault, cold, smart) and the access policy (HMAC keys, service-instance creds, public ACLs — the latter are off-limits for BNK supply-chain use).

The BNK supply chain reads from one COS bucket per cluster’s BNK install. The bucket holds:

ObjectWhat it isConsumed by
f5-far-auth-key.tgzFAR repository pull credentials — the F5-internal artefact key that lets FLO download FAR container imagesflo module at install time
subscription.jwt (or production equivalent)BNK subscription JWT — the licence the CNE Instance presentsflo and license modules
schematic-<v>.jsonThe deployer’s schematic JSON for the deployed BNK versioninformational, not directly mounted into the cluster
(optional) FAR image tarballsPre-pulled FAR images for air-gapped installsflo when running in disconnected mode

The bucket structure is defined by the upstream HCL — concretely by the ibmcloud_resources_cos_bucket variable. When you leave cos.bucket unset, init provisions (or discovers and reuses) an account-scoped bucket named bnk-artifacts-<first-12-of-account-id>: COS bucket names share one global namespace (like S3), so the account suffix keeps the name unique and lets a second workspace run from the same account find the bucket the first one created — no duplicate, no re-upload. The instance defaults to bnk-supply-chain. (The bare bnk-artifacts used in the manual cos examples below is just an illustrative name; supply your own.)

Local files instead of COS (no bucket needed)

COS is not mandatory for the two artefacts the BNK phase actually needs at apply time — the FAR auth tarball and the subscription JWT. You can point the workspace at local files and skip COS entirely:

bnk:
  far_auth_local_file: /path/to/f5-far-auth-key.tgz
  subscription_jwt_local_file: /path/to/subscription.jwt

When both are set, roksbnkctl reads them at render time — extracting the FAR _json_key_base64 service account from the tarball in Go — and injects the content directly into the flo and license modules (use_cos_bucket = false), so no COS instance, bucket, or S3 endpoint is contacted. This is useful when the account has no orchestration COS, or when the COS S3 endpoint isn’t reachable from where you run roksbnkctl.

roksbnkctl init wires this up automatically: it checks COS first, and on any COS error (including the dial tcp … no such host you get when a cluster region such as eu-fr2 has no COS S3 endpoint) it falls back to prompting for these local files and records them on the workspace. You can also set them by hand. When they’re unset, the COS path below is used unchanged.

The three command levels

roksbnkctl cos instance {create|delete|list}
roksbnkctl cos bucket   {create|delete|list|get} --instance <name-or-CRN>
roksbnkctl cos object   {put|get|delete|list} --instance <name-or-CRN>

All three layers resolve credentials through the standard credential resolver chain — env var, OS keychain, workspace api_key_b64, prompt. There’s no separate “COS credential”; the IBM API key authenticates against Resource Controller (instance ops) and IAM-signed S3 requests (bucket and object ops).

cos instance

Manages COS service instances at the account level via Resource Controller.

# Create a Standard-plan instance under the workspace's resource group
roksbnkctl cos instance create bnk-supply-chain --plan standard

# Override the plan by catalog UUID when roksbnkctl hasn't mapped the tier
roksbnkctl cos instance create bnk-supply-chain --plan-id <uuid>

# List instances in the account
roksbnkctl cos instance list

# Delete an instance (default: recursive — removes bound HMAC keys, service creds)
roksbnkctl cos instance delete bnk-supply-chain
roksbnkctl cos instance delete bnk-supply-chain --no-recursive --auto
FlagDefaultNotes
--planstandardFriendly name (standard, lite); maps to a Resource Controller plan UUID internally.
--plan-id—Catalog UUID — bypasses the friendly-name mapping. Use when IBM ships a plan tier roksbnkctl hasn’t seen yet.
--targetglobalCOS instances are global; this is left as a flag for forward compatibility.
--no-recursive(off)On delete, do NOT remove bound HMAC keys and service credentials. Hardly ever what you want.
--auto(off)On delete, skip the y/N confirmation.

The resource group is read from the workspace’s ibmcloud.resource_group field (defaulting to default when unset).

cos bucket

Manages buckets within a named instance. The --instance flag is required for every bucket and object call — buckets aren’t globally unique, only unique within an instance.

# Create a standard-class bucket
roksbnkctl cos bucket create bnk-artifacts \
  --instance bnk-supply-chain \
  --class standard

# List buckets on the instance
roksbnkctl cos bucket list --instance bnk-supply-chain

# Delete (the bucket must be empty first; cos object delete --recursive isn't implemented yet)
roksbnkctl cos bucket delete bnk-artifacts --instance bnk-supply-chain

# Snapshot the entire bucket to a local directory (recursive, streaming, byte-for-byte).
# Useful before rotating supply-chain artefacts — keeps a known-good set you can roll back to.
roksbnkctl cos bucket get bnk-artifacts ./snapshot \
  --instance bnk-supply-chain
FlagDefaultNotes
--instance(required)Instance name or CRN — the CRN starts with crn:v1: and is used as-is; a bare name is looked up via Resource Controller.
--regionworkspace regionThe IBM Cloud region the bucket is pinned to. Override only when you’re crossing regions deliberately.
--classstandardStorage class: standard (frequently accessed), vault (infrequent), cold (archive), smart (auto-tiered). The BNK supply chain uses standard because FLO reads at install + every restart.
--no-clobber(off)On cos bucket get: skip any object whose local path already exists. Default overwrites. Files-only flag; ignored by other cos bucket verbs.

cos object

Manages objects (files) within a bucket. The key syntax is <bucket>/<key/with/slashes> — the parser splits on the first slash, so bucket/dir/file.tgz parses as bucket bucket, key dir/file.tgz.

# Upload (streaming; multipart auto-engages for large files)
roksbnkctl cos object put bnk-artifacts/f5-far-auth-key.tgz \
  ./local/f5-far-auth-key.tgz \
  --instance bnk-supply-chain

# Download (streaming)
roksbnkctl cos object get bnk-artifacts/f5-far-auth-key.tgz \
  ./downloaded.tgz \
  --instance bnk-supply-chain

# Delete
roksbnkctl cos object delete bnk-artifacts/old-subscription.jwt \
  --instance bnk-supply-chain

# List (with an optional key prefix)
roksbnkctl cos object list bnk-artifacts \
  --instance bnk-supply-chain

roksbnkctl cos object list bnk-artifacts/schematics/ \
  --instance bnk-supply-chain

The list output is a tab-separated KEY SIZE MODIFIED table — pipe through column -t for readability or cut -f1 to extract just the keys.

The BNK supply chain shape

A typical bnk-artifacts bucket after a clean install looks like:

$ roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain
KEY                                     SIZE        MODIFIED
f5-far-auth-key.tgz                     2412        2026-05-08T14:12:33Z
subscription.jwt                               1857        2026-05-08T14:12:34Z
schematic-2.3.0-3.2598.3-0.0.170.json   18432       2026-05-08T14:13:01Z

Three pieces of metadata in the upstream HCL (terraform/variables.tf) pin the bucket layout:

HCL variableDefaultObject
f5_cne_far_auth_filef5-far-auth-key.tgzFAR pull credentials
f5_cne_subscription_jwt_filesubscription.jwtSubscription JWT
f5_bigip_k8s_manifest_version2.3.0-3.2598.3-0.0.170Schematic filename inferred from this

Changing any of these in terraform.tfvars (or the workspace bnk: block, which renders into tfvars) changes which COS keys FLO will look for. The HCL doesn’t auto-discover key names — they’re literal.

For air-gapped installs where the cluster can’t reach repo.f5.com, additional pre-pulled FAR image tarballs go in the same bucket and the far_repo_url variable points at a COS-backed proxy. That topology is out of scope for v1.0; the supply chain shape described here is the connected-mode happy path.

Multipart upload and streaming download

FAR image tarballs run 1-5 GB. cos object put streams the input file into the bucket in 5 MB parts using S3-style multipart uploads:

  • For files under 5 MB, a single-part PutObject is used (the SDK’s default).
  • For files over 5 MB, the SDK auto-engages multipart upload — the file is split into 5 MB parts, each uploaded in parallel (up to 4 concurrent parts), and finalised with a CompleteMultipartUpload call.

The split is transparent — there’s no --multipart flag to set. The SDK handles it under the hood. If you want to verify multipart is happening for a specific file, watch with roksbnkctl --verbose cos object put … and the SDK’s debug logging surfaces the part count.

cos object get is similarly streaming: the SDK pipes the body straight to the destination file without buffering it in memory. Multi-gigabyte downloads on a memory-constrained jumphost are safe.

If a multipart upload is interrupted (network drop, ^C), the partial-upload state lingers on COS until cleaned up. Today roksbnkctl doesn’t expose a “list and abort orphan multipart uploads” command — that’s a v1.x addition. The workaround is to use ibmcloud cos list-multipart-uploads directly via Chapter 17’s docker backend or the IBM Cloud console.

Workspace config integration

The workspace cos: block is optional — if the bucket is already populated (manually, or by an external CI pipeline), the block can be omitted entirely. When set, it triggers an auto-upload at roksbnkctl up time so the FAR pull and licence land before FLO needs them.

# ~/.roksbnkctl/<workspace>/config.yaml
cos:
  instance: bnk-supply-chain
  bucket: bnk-artifacts
  upload:
    - source: ./local/f5-far-auth-key.tgz
      key: f5-far-auth-key.tgz
    - source: ./local/subscription.jwt
      key: subscription.jwt

The block maps directly to internal/config/workspace.go::COSCfg:

FieldTypePurpose
instancestringCOS instance name or CRN. Looked up via Resource Controller at runtime.
bucketstringBucket name within the instance.
uploadlist of {source, key}Optional pre-flight uploads. source is a host filesystem path (relative or absolute); key is the destination object key in the bucket.

Pre-flight uploads run before terraform apply, so FLO sees the artefacts when it pulls. Idempotent: re-running up re-uploads, which COS treats as overwrite — safe.

When the supply chain matters

Three lifecycle moments where the COS bucket is in play:

Install time

roksbnkctl up provisions FLO, which queries the bucket for f5-far-auth-key.tgz and subscription.jwt. Missing either object → FLO fails to start → terraform apply retries (per internal/cli/lifecycle.go::applyWithRetry) for ~3 attempts before erroring. The fix is always “put the missing object in the bucket and re-run up”; the lifecycle retry hides transient bucket-policy propagation lag but won’t paper over a genuinely-empty bucket.

Upgrade time

When bnk.manifest_version (or the f5_bigip_k8s_manifest_version HCL variable) bumps, FLO pulls a new FAR image tarball and re-renders the CNE Instance. If the new manifest version references a FAR image that isn’t already in repo.f5.com’s public registry (rare, but happens for pre-release builds), the bucket holds the air-gapped fallback. Standard upgrades — the connected-mode case — don’t touch the bucket; they just pull from repo.f5.com using the credentials in f5-far-auth-key.tgz.

Licence rotation

When the trial expires or a production licence arrives, swap subscription.jwt for the new file:

roksbnkctl cos object put bnk-artifacts/subscription.jwt \
  ./new-license.jwt \
  --instance bnk-supply-chain

# Force FLO to re-read the licence (delete the CNE Instance's License resource;
# FLO's reconciler re-creates it from the updated JWT)
roksbnkctl k delete license -n f5-bnk --all

FLO picks up the new JWT within 60-90 seconds. No roksbnkctl up re-run required for licence rotation alone.

Worked example: rotating COS supply-chain assets

End-to-end Part VII scenario: the FAR auth key on file is about to expire, a new one arrived from the F5 distribution side, and you need to rotate it without taking BNK down. The same flow handles licence-JWT rotation (swap subscription.jwt for the production JWT) and FAR-image-tarball uploads for air-gapped clusters. Cross-link to Chapter 14 for the API-key half of the rotation story; this walkthrough focuses on the COS object half.

# 1. Sanity-check the current state
roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain

# 2. Upload the new auth key (overwrites the existing file)
roksbnkctl cos object put bnk-artifacts/f5-far-auth-key.tgz \
  ./new-far-auth-key.tgz \
  --instance bnk-supply-chain

# 3. Verify the upload
roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain
# Expected: the f5-far-auth-key.tgz row's MODIFIED timestamp is now

# 4. (optional, air-gapped only) Upload the FAR image tarball
roksbnkctl cos object put bnk-artifacts/far-2.3.0-images.tgz \
  ./far-2.3.0-images.tgz \
  --instance bnk-supply-chain

# 5. Force FLO to re-read the supply chain
roksbnkctl k delete pod -n f5-bnk -l app=flo
# (FLO's controller restarts; the new pod re-pulls f5-far-auth-key.tgz on first reconcile)

# 6. Verify FLO is healthy with the new key
roksbnkctl logs flo
# Expected: no "failed to pull FAR image: unauthorized" lines

The third step is the verification gate. If FLO’s logs still show auth failures after the pod restart, the new auth key was rejected by repo.f5.com — re-issue the key on the F5 side, not in the bucket.

Cross-references

Troubleshooting

Common failure modes you’ll hit when running roksbnkctl against real IBM Cloud accounts, organised as symptom → root cause → fix. The entries here are mined from the issue logs accumulated over Sprints 0-5 plus the failure shapes documented in PRD 05 §“Risks”.

Use the page as a lookup table. If your symptom isn’t here, Chapter 23 — The E2E test plan lists what every phase asserts; reverse-engineering from the assertions can narrow your diagnosis. For deeper-than-here debugging, the per-phase log files under /tmp/roksbnkctl-e2e-backends/ are the first stop.

Install and init

Symptom: roksbnkctl init errors with plaintext secret detected

Root cause: an existing ~/.roksbnkctl/<workspace>/config.yaml has a credential value sitting in a field whose name matches the rejection regex (api_key, password, token, secret_access_key, hmac_secret). The rejection is a deliberate safety net — see Chapter 14 §“What’s safe to commit vs not”.

Fix: move the credential into IBMCLOUD_API_KEY (env var) or the OS keychain (roksbnkctl init writes it via zalando/go-keyring). For a single-user dev box, the supported plaintext-on-disk channel is ibmcloud.api_key_b64 — base64-encoded, which doesn’t trip the regex.

Symptom: roksbnkctl init interactive prompts loop forever asking for the API key

Root cause: you’re running under CI / a non-TTY shell and roksbnkctl can’t read stdin. The interactive prompt fallback is the last step in the credential resolver chain and it doesn’t gracefully skip when stdin is closed.

Fix: set IBMCLOUD_API_KEY in the env, or pre-populate the keychain entry. For batch / CI runs, the documented invocation is:

IBMCLOUD_API_KEY=$(cat /path/to/secret) roksbnkctl init -w my-workspace

Pre-setting IBMCLOUD_API_KEY skips the API-key prompt (it’s the first link in the resolver chain). init still prompts for the remaining workspace metadata (region, resource group, cluster name) on TTY-bound stdin — a fully non-interactive bootstrap is on the v1.x roadmap.

Symptom: doctor reports terraform: not found on a fresh dev box

Root cause: terraform is the only strictly-required host tool for v1.0 (everything else is internalised). Doctor checks PATH; if your shell session hasn’t sourced the install location it’ll miss.

Fix: install terraform via your package manager (brew install terraform, apt-get install terraform, etc.) and re-source the shell, or set the TERRAFORM_BIN env var pointing at the binary explicitly.

Symptom (Windows): roksbnkctl install succeeds but roksbnkctl version says the command is not recognized

Root cause: roksbnkctl version (like any command) only resolves if the binary sits in a directory on the Windows %PATH%. If the install landed the binary somewhere off %PATH% (for example ~/.local/bin, a Unix convention that is never on the Windows %PATH%), the copy succeeds but the command doesn’t resolve.

Fix: roksbnkctl install (and the install.ps1 one-liner) install into a directory already on %PATH% — preferring %LOCALAPPDATA%\Microsoft\WindowsApps (on the per-user PATH by default, no admin) — so the binary resolves immediately in the same session. Remove any stray copy left in a non-PATH directory with Remove-Item "$env:USERPROFILE\.local\bin\roksbnkctl.exe". If a run ever lands in a non-PATH directory, it prints the PowerShell one-liner ([Environment]::SetEnvironmentVariable('Path', ...,'User')) to add it.

roksbnkctl up lifecycle

Symptom: bnk up refuses with cluster "…" already has BNK installed … but workspace "…" has no terraform state for it

Root cause: exactly what it says — the cluster is serving a BNK install that this workspace’s state knows nothing about. This is the normal shape when a cluster moves between deployments: BNK Forge gives each project its own deployment-scoped volume, so the second project to adopt a cluster legitimately has no state for the first project’s install.

Before v1.42.0 there was no refusal. An empty state planned a full install — resources_to_add=64 over a cluster that already had all 64 — and ground on for about thirteen minutes before exiting 1 without naming the cause or the cluster, three times over with retries (#53). The refusal replaces that with a fast, named failure.

Fix: pick one —

  • run from the workspace that installed it, whose state can converge or remove the install;
  • remove BNK from the cluster first, then re-run here;
  • if the install is genuinely abandoned, clear it from the cluster deliberately rather than letting an apply collide with it.

roksbnkctl -w <ws> bnk status shows what is actually on the cluster.

The check is narrow by construction: it fires only when this workspace’s BNK state holds no managed resources and the cluster is actually serving an install. A workspace that owns an install converges exactly as before and never reaches it. If the cluster cannot be reached, or what the state holds cannot be determined — including an s3 remote backend whose state cannot be listed — the guard stays silent rather than inventing a refusal.

There is deliberately no bnk adopt yet. Deriving and validating a complete install record from a live cluster is a larger change than a guard, and wants its own testing against real installs.

Symptom: terraform apply errors timeout while waiting for state to become 'normal'

Root cause: IBM Cloud’s control plane is occasionally 5-15 minutes slow propagating cluster state — a known transient. The cluster was created; the API just hasn’t caught up to reporting it as Ready.

Fix: roksbnkctl up retries the apply automatically up to 3 attempts with a 60-second sleep between (see applyWithRetry in internal/cli/lifecycle.go). If all three retries fail, just re-run roksbnkctl up manually — terraform’s state is durable, and the second attempt skips every resource that’s already provisioned.

Symptom: roksbnkctl up returns success but roksbnkctl k get nodes says No resources found

Root cause: the ROKS cluster’s worker nodes take 5-10 minutes to provision after the cluster’s master endpoint returns Ready. Terraform considers the cluster “applied” as soon as the master is up; the workers come up asynchronously.

Fix: wait 5-10 minutes and re-run. If you want a deterministic gate, watch the IBM Cloud console’s cluster page until the worker count matches workers_per_zone × zones, then proceed. There’s no roksbnkctl wait command.

Symptom: roksbnkctl up post-apply hook fails fetching the admin kubeconfig with a 404

Root cause: the IBM Cloud kubeconfig API (/global/v2/applications/kubeconfig) returns 404 for ~30-60 seconds after the cluster create call returns. The cluster exists but the kubeconfig endpoint hasn’t materialised.

Fix: the binary retries with exponential backoff and usually succeeds within a minute. If it still 404s after the retry budget, run roksbnkctl kubeconfig --download -w <workspace> to retry just the fetch without re-applying.

Symptom: Error: Inappropriate value for attribute "kubeconfig_dir": directory does not exist

Root cause: the upstream HCL’s IBM provider doesn’t MkdirAll for the kubeconfig output directory; it expects the parent dir to exist already. The variable’s default (/work/.bnk/scratch/kubeconfig) is the in-container path; on a direct-on-host run it’s a path that doesn’t exist.

Fix: roksbnkctl writes a workspace-scoped override (kubeconfig_dir = ~/.roksbnkctl/<ws>/state/kubeconfig) and creates the dir at apply time. If you’re hand-rolling terraform without roksbnkctl up, mkdir -p ~/.roksbnkctl/<ws>/state/kubeconfig first.

Symptom: cert-manager (or another chart) fails with could not download chart: ... open <HOME>/.cache/helm/repository/<hash>-index.yaml: no such file or directory

Root cause: a helm_release whose repository is a bare chart-repo URL (cert-manager off https://charts.jetstack.io) resolves through Helm’s anonymous-repo path, which downloads the repo index into helmpath.CachePath("repository") — i.e. $HELM_CACHE_HOME/repository, defaulting to $HOME/.cache/helm/repository. Crucially, that path is governed by HELM_CACHE_HOME, not HELM_REPOSITORY_CACHE (which only covers the named-repo / helm repo CLI path). In a runner whose $HOME resolves to an empty or non-writable path — how the bnk-forge tools-runner image launches the phases — that directory can’t be created, so the download fails even though charts.jetstack.io is perfectly reachable (the index downloads on the fly once the dir is writable — no pre-helm repo add is needed). The sibling symptom is a non-fatal warning: creating <HOME>/.kube: mkdir <HOME>: permission denied from the post-apply admin-kubeconfig fetch.

Fix: roksbnkctl exports writable, $ROKSBNKCTL_HOME-relative Helm homes before invoking terraform/helm — HELM_CACHE_HOME, HELM_CONFIG_HOME, HELM_DATA_HOME (plus HELM_REPOSITORY_CACHE / HELM_REPOSITORY_CONFIG / HELM_REGISTRY_CONFIG) under $ROKSBNKCTL_HOME/.helm/, and KUBECONFIG at $ROKSBNKCTL_HOME/.kube/config when $HOME/.kube isn’t writable — and pre-creates those dirs (see prepareToolEnv in internal/tf/terraform.go). The terraform helm provider runs as a plugin subprocess that inherits this env, so the redirect reaches the chart download. $ROKSBNKCTL_HOME is the persisted workspace tree (/work/.roksbnkctl under bnk-forge), so the cache is writable and survives across the phased invocations. Each var is set only when unset, so you can still point the cache at an air-gap mirror by exporting your own value. Set ROKSBNKCTL_DEBUG=1 to log the resolved HELM_CACHE_HOME / KUBECONFIG before each phase’s terraform exec. No action needed on a normal host run — $HOME there is already writable.

Symptom: terraform destroy leaves orphan IBM Cloud resources (LBs, security groups, VPEs)

Root cause: ROKS occasionally leaves dangling cluster-owned resources after the cluster itself is destroyed — the destroy returns success but the IBM Cloud account still shows a load balancer or a Virtual Private Endpoint Gateway tagged with the deleted cluster’s ID.

Fix: run roksbnkctl cleanup. It sweeps the account for every resource named after the workspace prefix (<prefix>-*) — VPC instances/subnets/security-groups/floating-IPs/public-gateways/VPCs, the Transit Gateway, registry COS, the cluster, and the BNK trusted profile — and deletes them in dependency order. Preview first with roksbnkctl cleanup --dry-run; add --all-regions if resources landed in a region not recorded in config.yaml. (For one-off resource types not covered by the prefix sweep, roksbnkctl ibmcloud is load-balancers | grep <cluster-name> + ibmcloud is load-balancer-delete <id> still works as a manual fallback.)

Teardown

bnk down sits for four minutes, then the namespace hangs in Terminating

⚠ 7 BNK custom resource(s) in f5-bnk did not finalize within 4m0s.
⚠ namespace "f5-bnk" was stuck Terminating; cleared F5 finalizers on 2 object(s).
Error: context deadline exceeded

Fixed in v1.59.0. If you see this, you are on v1.58.0 or earlier.

The teardown drains BNK’s custom resources before destroying the namespace, so that the F5 Lifecycle Operator is still running to finalize them — a finalizer that outlives its controller blocks the namespace forever. Three things made that drain unable to finish:

  • It deleted the components FLO manages. cnecontrollers, f5tmms, dssms, afms, downloaders and cnemanifests are declared by the CNEInstance, and FLO recreates any of them within about five seconds for as long as the CNEInstance exists. Deleting the CNEInstance is the uninstall; the components go with it.
  • Two product defaults cannot be drained at all. f5-big-tcp-settings/sys-default-tcp is refused by BNK’s own admission webhook (Default TCP parameters CR cannot be deleted!) and f5-big-dns-caches/security-fqdn-dns-resolver is recreated on sight. Neither carries a finalizer, so the namespace delete removes them; waiting for them achieves nothing.
  • f5validate-<ns> blocks its own removal. It has failurePolicy: Fail and is served from the namespace being destroyed, so once the controller stops answering the API server refuses every DELETE of a k8s.f5net.com object. Removing it does not hold — FLO restores it in under a second.

If you are stuck on an older version, the manual escape is to delete the CNEInstance and let FLO remove the rest, rather than deleting the components individually:

kubectl -n f5-bnk delete cneinstance --all
kubectl get validatingwebhookconfiguration | grep f5validate   # delete if the namespace will not terminate

The teardown says a custom resource “did not finalize”

The message names the object. Two cases, and they need opposite responses:

  • cneinstances/... — the CNEInstance finalizer tears down the whole product, so this is legitimately slow work rather than a stall. Let it run.
  • anything else — check whether it is being recreated: kubectl -n f5-bnk get <kind> <name> -o jsonpath='{.metadata.uid}' before and after a delete. A new UID means a controller replaced it, and no amount of waiting will remove it while its controller is running.

Cluster-scoped objects remain after a successful bnk down

Expected. A BNK 2.4 install leaves 95 CustomResourceDefinitions, 10 APIServices and a ValidatingAdmissionPolicy behind, none of which a namespace delete touches. They are shared with any other BNK install on the cluster, so removing them is a deliberate operator decision rather than a side effect of one workspace’s teardown. Reinstalling over them works — verified across consecutive install/uninstall cycles at the same manifest version.

scripts/testing/cluster-residue-check.sh reports what is left, and excludes cert-manager: it is shared infrastructure, and removing its CRDs would delete every Certificate and Issuer on the cluster including ones BNK never created.

What must not remain is a ValidatingWebhookConfiguration served from the BNK namespace. It is named for one namespace and points at a Service that no longer exists, and because it matches by namespaceSelector, recreating a namespace of the same name re-arms a webhook whose backend will never answer.

Workspaces

Symptom: after roksbnkctl ws delete <name>, which workspace is current?

Behaviour (since v1.9.0): you can delete the current workspace directly — ws delete moves the current_workspace pointer to another existing workspace (alphabetically first), or clears it when you delete the last one. There’s no “switch first” requirement and no phantom default fallback.

If a command then reports no workspace selected: you deleted the last workspace, so nothing is current. Create one with roksbnkctl init, or point at an existing one with roksbnkctl ws use <name>. (If the message is instead workspace "<name>" is not initialised, the current_workspace pointer references a directory that was removed by hand — ws use <other> repoints it.) Documented in Chapter 6 — Workspaces.

Symptom: workspace "<name>" has terraform-managed resources; pass --force to delete anyway

Root cause: the workspace’s terraform.tfstate is non-empty — live infrastructure exists. roksbnkctl ws delete refuses to orphan the resources by removing the state file out from under them.

Fix: run roksbnkctl down -w <name> --auto first to destroy the resources, then roksbnkctl ws delete <name> (no --force needed once state is empty). If you genuinely want to abandon the infra and clean up by hand later, roksbnkctl ws delete --force skips the check.

Backends

Symptom: --backend docker errors with Cannot connect to the Docker daemon

Root cause: dockerd isn’t running, or your user isn’t in the docker group, or you’re on a system that needs a separate rootless-docker socket path.

Fix:

  • Linux with system docker: sudo systemctl start docker; add yourself to the docker group (sudo usermod -aG docker $USER) and log out + back in.
  • Linux with rootless docker: systemctl --user start docker; set DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock.
  • macOS / Windows: launch Docker Desktop / Rancher Desktop.

Verify with docker info | head -1 — if that fails, roksbnkctl --backend docker will too.

Symptom: --backend k8s errors with ops pod not found in roksbnkctl-ops namespace

Root cause: you haven’t run roksbnkctl ops install against the target cluster. The k8s backend dispatches into a long-lived ops pod that has to be provisioned first.

Fix:

roksbnkctl ops install

Verify with roksbnkctl k get pod -n roksbnkctl-ops — the pod should be Running. See Chapter 19 for the install model.

Symptom: --backend ssh:<target> errors with tool not found: iperf3 — run with --bootstrap to apt-install

Root cause: the SSH target doesn’t have the tool installed, and roksbnkctl doesn’t auto-install without explicit opt-in (because apt-installing on a production jumphost without consent is rude).

Fix: pass --bootstrap once per fresh target:

roksbnkctl --backend ssh:jumphost --bootstrap test throughput

The bootstrap step runs apt-get install -y <tool> (or the equivalent for ibmcloud — adding the IBM apt repo first). Subsequent calls skip the install check and run normally. See Chapter 17 §“SSH backend” for the bootstrap mechanism.

Symptom: --backend ssh:jumphost errors host key mismatch for jumphost (got SHA256:..., known_hosts has SHA256:...)

Root cause: the jumphost was re-provisioned (terraform destroy + apply) and now has a fresh host key, but ~/.roksbnkctl/known_hosts still has the old fingerprint. TOFU refuses to silently accept the change — that’s the threat model the prompt exists to defend against.

Fix: if you know the re-provision is legitimate, delete the stale entry:

ssh-keygen -R '<jumphost-ip>' -f ~/.roksbnkctl/known_hosts
# Or for the whole roksbnkctl known_hosts:
rm ~/.roksbnkctl/known_hosts

The next roksbnkctl --on jumphost call will TOFU-prompt with the new fingerprint. For CI use the --insecure-host-key flag, which records the key on first contact without prompting.

OpenShift and PodSecurity

Symptom: throughput test pod fails admission: violates PodSecurity "restricted:v1.x": runAsNonRoot != true

Root cause: the throughput suite’s default iperf3 image is networkstatic/iperf3:latest which runs as root. OpenShift’s restricted-v2 SCC rejects root pods.

Fix: set the workspace config to use the bundled image, which is built USER 1000:

# ~/.roksbnkctl/<workspace>/config.yaml
test:
  throughput:
    image: ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:v0.9.0

Chapter 22 §“The bundled image and the runAsNonRoot constraint” is the full backstory. The same chapter’s §“OpenShift SCC failure mode” lists the three error-message variants OpenShift produces.

Symptom: roksbnkctl ops install errors ServiceAccount "roksbnkctl-ops" forbidden: violates PodSecurityPolicy

Root cause: rare — the cluster is running PodSecurityPolicy (the deprecated predecessor to PodSecurity admission) and the ops pod’s ServiceAccount doesn’t have the SCC binding it needs.

Fix: the ops manifest assumes restricted-v2 is acceptable. If your cluster forces privileged, that’s a cluster-policy question outside roksbnkctl’s control — talk to your cluster admin about granting restricted-v2 to the roksbnkctl-ops namespace.

Symptom: ImagePullBackOff on the ops pod or throughput pod

Root cause: most commonly, the cluster can’t reach the image registry. Three sub-causes:

  1. The cluster’s egress NAT doesn’t route to ghcr.io (the image host for roksbnkctl-tools-*).
  2. The image tag doesn’t exist for the version you’re running (e.g., you built roksbnkctl from main at a commit between releases, and :dev isn’t published).
  3. ghcr.io itself is rate-limiting unauthenticated pulls (rare; usually only an issue for shared CI hosts hitting ghcr.io en masse).

Fix:

  • Check egress with roksbnkctl k exec <ops-pod> -- curl -sI https://ghcr.io — if that hangs, you have a network path issue, not a roksbnkctl issue.
  • Check the tag with docker manifest inspect ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<version> — if 404, pin to a tagged release version in workspace config rather than running from main head.
  • For rate-limit issues, pre-pull images to a local registry mirror and override the workspace test.throughput.image to point there.

DNS

Symptom: roksbnkctl test dns returns NXDOMAIN against an internal GSLB record that you know exists

Root cause: your laptop’s resolver chain doesn’t have a route to the internal GSLB VIP. The default --server system uses your /etc/resolv.conf, which resolves against your office or ISP resolver — neither of which knows about the cluster-private GSLB.

Fix: query the GSLB VIP explicitly, or query from inside the cluster:

# Query the GSLB VIP directly
roksbnkctl test dns --target www.example.com --type A --server 169.45.91.5

# Or run the probe from inside the cluster (the cluster's resolvers reach the GSLB)
roksbnkctl test dns --target www.example.com --type A --backend k8s --server cluster

Chapter 21 §“Server resolution” is the full --server reference.

Symptom: --gslb-compare always reports gslb_divergence: false against a target you expect to diverge

Root cause: the chosen target’s GSLB rule isn’t differentiating your local vantage (laptop) from your k8s vantage (cluster). Two common shapes:

  1. The name is fronted by an anycast resolver fleet (Cloudflare, Google Public DNS) — same answer everywhere by design.
  2. Your laptop and your cluster are both in the same geographic region from GSLB’s perspective (both in North America hitting the same datacenter).

Fix: pick a target known to be geo-resolved (www.google.com is the canonical “different IPs from different regions” example), or add an SSH-based vantage (--backend ssh:eu-bastion) to bring in a third region. Chapter 21 §“GSLB cross-vantage compare” covers the multi-vantage workflow.

Symptom: roksbnkctl test dns --backend docker errors DNS probe doesn't benefit from docker

Root cause: design choice. Docker containers share the host’s network namespace by default, so a docker-backend probe has the same network identity as a --backend local probe — no GSLB-relevant vantage difference.

Fix: use --backend local, --backend k8s, or --backend ssh:<target> instead.

Cluster registration

Symptom: roksbnkctl cluster register <name> errors cluster not found

Root cause: the cluster name doesn’t exist in the workspace’s resource group, or the API key doesn’t have visibility into the resource group.

Fix: verify the name with roksbnkctl ibmcloud ks cluster ls --output json | jq '.[].name', and verify the resource-group scope in workspace config matches where the cluster lives. If the cluster is in a different resource group, set ibmcloud.resource_group in the workspace config to that group.

Symptom: register succeeds but roksbnkctl k get nodes immediately errors Unauthorized

Root cause: the kubeconfig was fetched but the auth token has already expired, or the IAM-based token that the kubeconfig embeds doesn’t match the API key that’s currently in env. Common after a 1-hour idle window.

Fix:

roksbnkctl kubeconfig --download --cluster <name>

The token refresh is automatic on every up/apply, but register against a cluster you didn’t just provision sometimes lands you with a stale token in the kubeconfig.

COS supply chain

Symptom: FLO fails to start with failed to pull FAR image: 403 Forbidden

Root cause: the f5-far-auth-key.tgz object in the bucket has stale credentials (the F5-side pull key was rotated, but the bucket still has the old one).

Fix: re-issue the key on the F5 side and upload to COS:

roksbnkctl cos object put bnk-artifacts/f5-far-auth-key.tgz \
  ./new-f5-far-auth-key.tgz \
  --instance bnk-supply-chain

# Restart FLO so it re-reads
roksbnkctl k delete pod -n f5-bnk -l app=flo

See Chapter 25 §“Worked example” for the full flow.

Symptom: cos object put for a 3 GB file errors midway with RequestTimeout

Root cause: the multipart upload SDK encountered a transient COS HTTP timeout on one of the part uploads. Multipart uploads aren’t currently resumed from the failure point — they restart from zero.

Fix: re-run the cos object put. If it fails reproducibly on the same part, the underlying network is the problem (your egress link is saturated, or COS is having a regional outage — check the IBM Cloud status page).

Symptom: cos bucket delete errors Bucket not empty

Root cause: COS requires buckets to be empty before delete; there’s no --recursive flag on bucket delete today.

Fix: list and delete each object, then delete the bucket:

roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain | \
  awk 'NR>1 {print $1}' | \
  xargs -I{} roksbnkctl cos object delete "bnk-artifacts/{}" --instance bnk-supply-chain

roksbnkctl cos bucket delete bnk-artifacts --instance bnk-supply-chain

Don’t forget to abort any pending multipart uploads first — they don’t appear in the standard object list but they do prevent bucket deletion. The workaround for now is ibmcloud cos list-multipart-uploads followed by ibmcloud cos abort-multipart-upload until v1.x lands a native command.

Shared VPC, Transit Gateway, and quotas

Symptom: roksbnkctl up fails at apply with a VPC or Transit Gateway quota error (Quota: 20, or cannot add more than 10 gateways)

Root cause: IBM Cloud caps VPCs at 20 per region and Transit Gateways at 10 per account (defaults). A full region/account fails the create — often ~40 minutes into a cluster build.

Fix: run roksbnkctl doctor first — the ibm cloud quota check reports VPCs-in-region and account-wide Transit Gateways vs the limits and warns when you’re at the wall. The init interview shows the same count at the create-VPC / create-TGW prompts; answer no and adopt an existing VPC/gateway (the picker lists them), or request a quota increase from IBM. See Reusing an existing VPC and Sharing a Transit Gateway.

Symptom: image pulls from the mirror time out intermittently, but every security group and ACL allows the traffic

Root cause: two VPCs on the same Transit Gateway have overlapping address prefixes. A gateway routes on VPC address prefixes; when two attached VPCs claim the same block the route is ambiguous and the gateway silently drops traffic for one of them. Nothing logs an error, which is what sends people into hours of firewall archaeology. The classic tell is that connectivity is partial or flaps — some pulls land, some hang — rather than failing cleanly.

It happens by accident because IBM’s default address-prefix management is auto, which gives every VPC in a region the identical three per-zone prefixes (10.241.0.0/18, 10.241.64.0/18, 10.241.128.0/18). A second roksbnkctl-created cluster joining a shared gateway therefore collides by construction.

Confirm it — list the prefixes of each VPC on the gateway and look for repeats:

ibmcloud tg connections <gateway-id> --output json | jq -r '.[]|select(.network_type=="vpc")|.network_id'
ibmcloud is vpc-address-prefixes <vpc-id>     # per VPC from that list

Fix: the prefixes of a live VPC cannot be edited — moving a subnet’s CIDR replaces the subnet, destroying the cluster on it. Either detach one (roksbnkctl -w <other-workspace> tgw disconnect --auto; the cluster survives), or rebuild one cluster on its own block, set before cluster up:

cluster:
  vpc_cidr: 10.242.0.0/16     # ROKSBNKCTL_CLUSTER_VPC_CIDR for CI

Set it before cluster up, not after. Editing vpc_cidr in a workspace whose cluster already exists does not re-address anything — an existing VPC’s prefixes cannot be changed without replacing its subnets. It now says so:

! cluster.vpc_cidr is a CREATE-time setting and it has changed.
  Cluster "x" was created from 10.242.0.0/16; this workspace now says 10.99.0.0/16.
  The new value is ignored — ...
  A future release will refuse this rather than ignore it; ...

The warning fires only on a real disagreement with the block the cluster was created from, so a workspace that set it once and left it alone stays silent. A workspace built by cluster register also stays silent, because the adopt path records no block to disagree with — see Chapter 9.

Prevention (v1.39.0+): cluster up refuses before terraform creates anything when the VPC it is about to build would overlap one already on the gateway, and tgw connect refuses before attaching — each naming the other VPC and the colliding prefixes. See Give each cluster VPC its own address block.

Symptom: cluster ... was created as a single-nic cluster, but this workspace now asks for multi-nic

Root cause: cluster.network_mode is a create-time setting and it changed after the cluster was built. There is no in-place conversion between modes.

Why it refuses rather than warns: terraform would happily plan this — as a destroy and create of the running cluster, rendered as ~60 lines that read like an update. The refusal happens before planning, so you never get the chance to approve it by mistake.

Fix: set cluster.network_mode back to what the cluster is, or build the other mode as a new cluster in a new workspace. What the cluster actually is comes from cluster-outputs.json, not from config.yaml.

Note that removing the setting is not the same as changing it: an unset network_mode defers to the record and is never refused. See Chapter 8a §Which side wins.

Symptom: BNK 2.3 does not support multi-nic clusters (it supports single-nic)

Root cause: the BNK release — derived from bnk.manifest_version — cannot drive a cluster of that network mode. 2.3 does not express the multi-NIC network attachments or CNEInstance options.

Fix: use a cluster built in a mode this line supports. There is currently no line to move to for multi-nic — 2.4 was expected to add it and, on the evidence of its EA install guide, does not, so its row was corrected to single-nic. See Chapter 28. The pairing is checked at plan time precisely so this is a message rather than a failed apply against real infrastructure.

Symptom: ! unknown BNK release line: BNK 2.5 is not in this build's support matrix

This is a warning, and the run continues. It is expected, not a defect.

Root cause: the support matrix ships inside the binary, so a release newer than your build is unknown to it. That is missing information, not a known incompatibility — refusing on it would make every build refuse every BNK release that shipped after it, which is unhelpful when the binary is pinned (the BNK Forge modules pin the runner image by digest, so “use a newer build” is not an action available to whoever is choosing the release).

What it means: the combination is unverified here, not blessed. Upgrade roksbnkctl if you want it actually checked.

Symptom: down fails with The VPC is in use and cannot be deleted. Subnets [...]

Root cause: this workspace created the VPC, and another workspace that adopted it still has a cluster (its subnets) inside. The VPC owner must be torn down last — deleting it first can’t remove a non-empty VPC, and also deletes the shared per-zone public gateways the other cluster still depends on.

Fix (v1.26.1+): down refuses this up front, naming the foreign subnets. Tear down the workspace(s) sharing the VPC first, then re-run down on the owner. See the teardown-order note in Reusing an existing VPC.

Symptom: down fails with UnsetSubnetPublicGateway ... the specified subnet has no public gateway

Root cause: an adopter cluster’s teardown tried to detach its subnets from a shared public gateway that the VPC-owning cluster had already deleted (the owner was torn down first — see above).

Fix: v1.26.1+ clusters can’t hit this — subnets attach their gateway inline, so deleting the subnet detaches implicitly. For an older cluster stuck mid-teardown, drop the dead attachment from state and retry: cd ~/.roksbnkctl/<ws>/state-cluster && terraform state rm 'module.roks_cluster.module.cluster.ibm_is_subnet_public_gateway_attachment.cluster_subnet_gateway_zone1[0]' (repeat for zone2/zone3), then roksbnkctl -w <ws> cluster down.

Symptom: roksbnkctl bnk up hangs on licensing / the License CR never reaches Active

Root cause: BNK licensing completes only when the License CR reports status.state: Active (or the LicenseActive condition), which roksbnkctl polls for deterministically (~15-minute bound). A hang usually means the FAR supply chain is wrong — a missing/expired JWT, or the JWT uploaded under the wrong object key — or, in f5licenseproxy mode, the F5 License Proxy isn’t reachable.

Fix: confirm the JWT is present under the exact object key subscription.jwt (roksbnkctl cos object list <bucket> --instance bnk-supply-chain) and not expired; for FLP mode check roksbnkctl flp status before bnk up. See Licensing BNK with the FLP.

Since v1.41.0 the reachability gate catches the network half of this before the install starts, so a licensing hang that is really an unreachable proxy now surfaces as a named per-node failure instead — see the next entry.

Symptom: bnk up fails the reachability gate on a cluster whose network you believe is correct

✗ kube-…-0000013a -> registry (10.243.0.4:443): dns=skipped-ip tcp=FAILED
    (tcp connect failed -- refused, filtered, or no route)

Root cause: most often the gate is right and something (a security group, an ACL, a missing Transit Gateway attachment, or overlapping VPC address prefixes) genuinely blocks the path from that node.

But there is one case where the network is correct and the probe is early. A Transit Gateway attachment is asynchronous: IBM programs the routes some time after the connection reports attached. A workspace that creates the VPC, joins the gateway, and installs in a single run can probe a path that is correct but not yet live. A probe 73 seconds after attach has failed on a path that was healthy minutes later, while a sibling cluster on the same gateway and the same blueprint passed simply by landing on the other side of route programming (#57).

Fix: read the attempt count in the detail first — it is what tells the two apart:

  • (still failing after 19 attempts over 180s) — it kept trying for the whole budget. That is a real break; go look at the security groups, the TGW attachment, and the VPC prefixes.

  • No attempt clause, or a low count — it gave up early. Raise the budget:

    bnk:
      preflight:
        reachability_retry_seconds: 600
    

    or ROKSBNKCTL_REACHABILITY_RETRY_SECONDS=600 on a CI runner. See Chapter 28 §bnk.preflight.

Check dns before tcp: dns=FAILED is a resolver or hostname problem and no retry budget will fix it.

Re-running gives you a fresh answer. Since v1.42.0 the gate rolls its DaemonSet on every run, so the verdict always belongs to the run you just started. Before that, a re-run after fixing the routing re-read the original pod’s log and showed the same failure with nothing to indicate it was a recording.

Networking

Symptom: roksbnkctl test connectivity reports Get "https://...": dial tcp: i/o timeout for an internal-only URL

Root cause: connectivity probes run from --backend local by default. From your laptop, internal-only URLs (cluster-private VIPs, internal GSLB names) aren’t reachable.

Fix: route the probe through the cluster’s network — either via --backend k8s (when it lands for the connectivity suite — currently k8s-backend is iperf3 + DNS only; connectivity stays local for v1.0) or via an SSH target inside the cluster’s VPC (--backend ssh:cluster-jumphost).

Symptom: roksbnkctl test connectivity fails with x509: certificate signed by unknown authority against a self-signed internal endpoint

Root cause: the URL’s TLS cert isn’t in the host’s trust store.

Fix: pass --insecure (session-wide; skips TLS validation for every probe in the run). The flag is deliberately session-wide rather than per-host — see Chapter 20 §“Mixed TLS-trust posture”. For mixed trust posture across multiple internal endpoints, run two separate test connectivity invocations, one per trust group.

CI-specific

Symptom: nightly e2e run fails on phase D with Error: Provider configuration is missing

Root cause: a terraform init cache invalidation under ~/.roksbnkctl/<ws>/state/.terraform/ left a partial provider download. Happens after a CI worker is recycled mid-init.

Fix: rm -rf ~/.roksbnkctl/<ws>/state/.terraform/ then re-run roksbnkctl up. Terraform-init re-downloads the providers cleanly. For CI workers that get recycled often, add a pre-step that purges .terraform/ before each run.

Symptom: cred audit (phase M) reports IBMCLOUD_API_KEY found in docker inspect output

Root cause: real stop-ship — credentials leaked into a docker container’s runtime env. Check internal/exec/docker.go::buildEnvArgv for any code path that passes the credential by value (-e IBMCLOUD_API_KEY=<value>) rather than by reference (-e IBMCLOUD_API_KEY — let docker pull from the caller’s env).

Fix: file an issue immediately, do not tag a release until this is green. Phase M is the v1.0 release gate; a leak here means the redactor or the cred-passing logic regressed. See PRD 04 for the threat model.

Getting more help

When the symptom isn’t on this page:

  1. Re-run with --verbose (-v) — the verbose output usually surfaces the root cause directly.
  2. Check /tmp/roksbnkctl-e2e-backends/<phase>-<ts>.log for the per-phase trail.
  3. Cross-reference Chapter 23 — The E2E test plan — the phase-by-phase pass criteria usually narrow down where the breakage lives.
  4. File an issue on github.com/jgruberf5/roksbnkctl with the verbose output, the roksbnkctl --version stamp, and the per-phase log if there is one.

Command reference

Auto-generated by go run ./tools/refgen/cobra-md > book/src/27-command-reference.md. Re-run on every CLI surface change.

roksbnkctl deploys F5 BIG-IP Next for Kubernetes (BNK) onto IBM Cloud ROKS, manages the COS supply chain BNK depends on, and runs built-in connectivity, DNS, and throughput tests against the deployed environment.

The 4-command lifecycle: roksbnkctl init Interactive setup; writes the workspace config roksbnkctl up Provision (or attach) and deploy BNK roksbnkctl test Run connectivity, DNS, and throughput tests roksbnkctl down Tear down BNK (and the cluster if cluster up provisioned it)

See https://jgruberf5.github.io/roksbnkctl/book/ for the canonical user guide.

Global flags

These flags apply to every command. They are declared on the root command and inherited by every subcommand.

FlagTypeDefaultDescription
--backendstring—execution backend: local | docker | k8s | ssh:<target> (default: per-tool from workspace exec: block, else local)
--bootstrapboolfalsefor –backend ssh:<target>: auto-install missing tools on Ubuntu via apt-get (requires passwordless sudo on the target)
--insecure-host-keyboolfalseskip the host-key TOFU prompt; record on first contact (CI use)
--no-colorboolfalsedisable colored output
--onstring—run on the named SSH target instead of locally (roksbnkctl targets list to see options)
--output / -ostringtextoutput format: text | json
--quiet / -qboolfalsesuppress all but errors
--verbose / -vboolfalseverbose output
--workspace / -wstring—workspace name (default: current; first run creates ‘default’)

roksbnkctl agent

Drive this workspace with an agentic CLI (personas + AGENTS.md)

roksbnkctl agent [claude|gemini|aider|openai|pi|opencode|agy] [flags]

Agentic mode. roksbnkctl embeds no LLM — bring your own coding-agent CLI.

roksbnkctl agent init Scaffold AGENTS.md + personas/ + journal/ into the workspace roksbnkctl agent List supported CLIs + this workspace’s default roksbnkctl agent <cli> Launch <cli> against the workspace\n roksbnkctl agent <cli> –show Print the invocation instead of running it

Personas (act as exactly one at a time): solution-architect (customer interface, owns scope), cloud-operator (runs the lifecycle), test-engineer (validation probes), doc-specialist (the report). See personas/ after init.

Flags

FlagTypeDefaultDescription
--showboolfalseprint the invocation instead of running the agent

roksbnkctl agent init

Scaffold the agentic-mode files into the workspace

Copies AGENTS.md, CLAUDE.md, the persona role contracts, and a decisions.md seed into the workspace dir, and creates an empty journal/. Safe to re-run: existing files are left untouched (your edits survive).

← back to roksbnkctl agent

roksbnkctl apikey

Print the workspace’s resolved IBM Cloud API key to stdout (a secret)

Resolve the workspace’s IBM Cloud API key via the standard chain and print it to stdout — the scripting seam for driving the ibmcloud CLI off a workspace:

ibmcloud login –apikey “$(roksbnkctl -w <ws> apikey)”

Resolution order (the same roksbnkctl itself uses):

  1. environment — IBMCLOUD_API_KEY / IC_API_KEY / TF_VAR_* (including any loaded from $PWD/.env, which roksbnkctl reads at startup)
  2. OS keychain (service “roksbnkctl”, account “<workspace>/ibmcloud_api_key”)
  3. workspace config.yaml — ibmcloud.api_key_b64 (base64-decoded)

Non-interactive: it never prompts — if no key is found it exits non-zero with an actionable error.

WARNING: this writes the credential to stdout. Pipe or redirect it; never echo it into logs or CI output.

roksbnkctl apply

Apply Terraform without re-prompting (assumes config.yaml exists)

roksbnkctl apply [flags]

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the confirmation prompt
--no-kubeconfigboolfalseskip the post-apply admin kubeconfig fetch
--planstring—apply exactly this saved plan file (from plan --out) instead of re-planning
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

roksbnkctl backend

Configure the per-tool execution backend default (writes config.yaml for you)

Set or inspect the per-tool execution-backend default — the exec: block — from the CLI, instead of hand-editing config.yaml.

show effective backend for each tool, and where it came from set <tool> <be> persist exec.<tool>.backend (be: local | docker | k8s | ssh:<target>) unset <tool> remove the override; the tool reverts to its built-in default

The per-invocation --backend flag still wins over whatever is persisted here.

roksbnkctl backend set

Persist a per-tool execution-backend default

roksbnkctl backend set <tool> <backend>

← back to roksbnkctl backend

roksbnkctl backend show

Show the effective execution backend for each tool

← back to roksbnkctl backend

roksbnkctl backend unset

Remove a per-tool backend override (revert to the built-in default)

roksbnkctl backend unset <tool>

← back to roksbnkctl backend

roksbnkctl bnk

BNK trial lifecycle (sits on top of a cluster)

Manage the BNK trial resources (flo, cne_instance, license) as a short-lived layer on top of a shared, durable cluster.

Commands: roksbnkctl bnk up Deploy the BNK trial (auto-provisions the cluster if missing) roksbnkctl bnk down Destroy the BNK trial, leaving the cluster in place

Use roksbnkctl cluster ... to manage the cluster phase itself.

roksbnkctl bnk down

Destroy the BNK trial, leaving the cluster in place

roksbnkctl bnk down [flags]

Destroys only the BNK trial resources, leaving the cluster phase intact for the next bnk up to attach to. The common iteration loop:

roksbnkctl bnk down && roksbnkctl bnk up # ~5 min trial reset roksbnkctl down # full teardown, ~30 min

Exits 0 (“nothing to do”) when there’s no trial state (cluster-only or empty), so it’s safe in a reverse-order teardown of every phase. Refuses on legacy single-state workspaces (use roksbnkctl down there).

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the destroy confirmation
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl bnk

roksbnkctl bnk output

Print the BNK phase’s own terraform outputs (text or –json; [name] = one raw value)

roksbnkctl bnk output [name] [flags]

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)
--show-sensitiveboolfalsereveal sensitive output values (default redacted)

← back to roksbnkctl bnk

roksbnkctl bnk status

Live runtime status of the BNK trial phase (namespaces + component readiness)

roksbnkctl bnk status [flags]

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)

← back to roksbnkctl bnk

roksbnkctl bnk up

Deploy the BNK trial; provisions the cluster first if missing

roksbnkctl bnk up [flags]

Provisions the BNK trial against the existing cluster phase. If the workspace has no cluster registered yet, bnk up bootstraps the cluster phase first (with a confirmation prompt — the cluster provision takes ~30 min) before the trial apply.

Flags

FlagTypeDefaultDescription
--autoboolfalseskip confirmation prompts (cluster-bootstrap + apply)
--no-kubeconfigboolfalseskip the post-apply admin kubeconfig fetch
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl bnk

roksbnkctl bnkforge

Configure + drive BNK Forge cluster registration

Configure and drive registration of this workspace’s cluster with a co-located BNK Forge (v3) install — without hand-editing config.yaml.

enable turn on auto-registration on cluster up (writes config.yaml) disable turn it back off status show the effective config + readiness register register this workspace’s cluster with BNK Forge right now

Registration uses BNK Forge v3’s REST API (it has no CLI). It is credential- backed: an IBM Cloud credential template is stored in Forge so it re-derives the kubeconfig on demand — nothing perishable is stored. The password is read from BNK_FORGE_PASSWORD or an interactive prompt (never persisted); the resulting session token is cached in the OS keychain.

roksbnkctl bnkforge disable

Turn off BNK Forge auto-registration (LOCAL only — does not unregister)

Clears the workspace’s auto-register flag, so cluster up stops registering with BNK Forge.

This is a LOCAL setting and never contacts the server. A cluster already registered stays on Forge’s Kubernetes page — to remove it, use roksbnkctl bnkforge unregister.

← back to roksbnkctl bnkforge

roksbnkctl bnkforge enable

Enable BNK Forge auto-registration on cluster up (writes config.yaml for you)

roksbnkctl bnkforge enable [flags]

Flags

FlagTypeDefaultDescription
--forge-castring—PEM file holding the CA the Forge server’s certificate must chain to (pins a self-signed lab cert; supersedes –insecure)
--insecureboolfalseDISABLE TLS verification — the API token is then sent over an unauthenticated connection; prefer –forge-ca
--projectstring—target BNK Forge project name (ensured-or-created)
--urlstring—BNK Forge server URL (e.g. https://forge.example.com)
--usernamestring—BNK Forge login username

← back to roksbnkctl bnkforge

roksbnkctl bnkforge register

Register this workspace’s cluster with BNK Forge now (no re-up needed)

roksbnkctl bnkforge register [flags]

Flags

FlagTypeDefaultDescription
--forceboolfalsetake over a cluster registered to ANOTHER Forge project (moves it, and its cluster id changes)
--forge-castring—PEM file holding the CA the Forge server’s certificate must chain to (pins a self-signed lab cert; supersedes –insecure)
--insecureboolfalseDISABLE TLS verification — the API token is then sent over an unauthenticated connection; prefer –forge-ca
--passwordstring—BNK Forge password (prefer BNK_FORGE_PASSWORD env or the prompt)
--projectstring—target BNK Forge project name (overrides config)
--urlstring—BNK Forge server URL (overrides config)
--usernamestring—BNK Forge login username (overrides config)

← back to roksbnkctl bnkforge

roksbnkctl bnkforge ssh-credential

Give BNK Forge the SSH private key for an appliance this workspace built

roksbnkctl bnkforge ssh-credential [flags]

Stores an appliance’s SSH private key in BNK Forge and attaches it to the workspace’s project, so Forge can reach the appliance itself.

bnk.flp.vsi.ssh_key names an IBM Cloud VPC key, which puts the PUBLIC half on the VSI — that is operator access, and it already works. Forge separately needs the PRIVATE half, and nothing supplied it, so a healthy FLP reports:

infrastructure_private_key_available: false
infrastructure_access_status:         recovery_required

Nothing can be recovered there — the credential was never created.

–host must be the address FORGE can reach, which for an FLP is the FLOATING IP. flp status reports a services-VPC endpoint that Forge sits outside of, so a credential built from it can never connect.

Flags

FlagTypeDefaultDescription
--expect-fingerprintstring—SHA256 fingerprint the key must match (e.g. the VPC key’s) — a credential that cannot log in is worse than none
--forge-castring—PEM file holding the CA the Forge server’s certificate must chain to (pins a self-signed lab cert; supersedes –insecure)
--hoststring—address FORGE reaches the appliance on — the FLOATING IP, not the endpoint flp status prints
--insecureboolfalseDISABLE TLS verification — the API token is then sent over an unauthenticated connection; prefer –forge-ca
--keystring—PRIVATE key file Forge will use (unencrypted; Forge stores it and cannot prompt)
--namestring—credential name in Forge (default: <project>-ssh)
--passwordstring—BNK Forge password (prefer BNK_FORGE_PASSWORD env or the prompt)
--portint22SSH port on the appliance
--projectstring—BNK Forge project to attach the credential to
--ssh-usernamestringubuntuSSH user on the appliance
--urlstring—BNK Forge server URL (overrides config)
--usernamestring—BNK Forge login username (overrides config)

← back to roksbnkctl bnkforge

roksbnkctl bnkforge status

Show this workspace’s BNK Forge registration config + readiness

← back to roksbnkctl bnkforge

roksbnkctl bnkforge unregister

Remove this workspace’s cluster from its BNK Forge project

roksbnkctl bnkforge unregister [flags]

Removes the cluster this workspace registered, so a teardown can undo what bnkforge register did.

Without it a workspace can create a registration it has no way to remove: bnkforge disable only clears the local auto-register flag and never contacts the server, so the cluster stays on Forge’s Kubernetes page pointing at something that may no longer exist.

Absence is success. No project, no cluster of that name, or a cluster already gone all report and exit 0, so this is safe to run from a destroy path that may run twice or run late.

Flags

FlagTypeDefaultDescription
--forge-castring—PEM file holding the CA the Forge server’s certificate must chain to (pins a self-signed lab cert; supersedes –insecure)
--insecureboolfalseDISABLE TLS verification — the API token is then sent over an unauthenticated connection; prefer –forge-ca
--passwordstring—BNK Forge password (prefer BNK_FORGE_PASSWORD)
--projectstring—BNK Forge project holding the cluster
--urlstring—BNK Forge base URL
--usernamestring—BNK Forge username

← back to roksbnkctl bnkforge

roksbnkctl cleanup

Delete orphaned IBM Cloud resources left by a failed down

roksbnkctl cleanup [flags]

cleanup sweeps the current workspace’s IBM Cloud account for resources named after the workspace prefix (<prefix>-*) and deletes them in dependency order.

Use it to recover from a ‘down’ that errored partway and stranded resources (jumphosts, security groups, floating IPs, subnets, VPCs, the Transit Gateway, the registry COS instance, the ROKS cluster, and the BNK trusted profile).

This is a DESTRUCTIVE, best-effort sweep keyed purely on the <prefix>- name convention. It always lists what it found and asks before deleting (unless –auto); –dry-run lists without deleting. Re-run if some deletes fail (e.g. a VPC waiting on an async cluster delete).

A Transit Gateway is deleted only once its connections are detached, and cleanup detaches only connections to VPCs it is also deleting. A gateway still attached to anything else — a VPC under another prefix, a Direct Link, a GRE tunnel — is REFUSED rather than silently disconnected from its other tenants. Re-running will not clear that: detach it yourself, or delete those networks.

Flags

FlagTypeDefaultDescription
--all-regionsboolfalsesweep every IBM Cloud region (slower; catches resources in regions not recorded in config)
--auto / -yboolfalseskip the confirmation prompt
--dry-runboolfalselist orphaned resources without deleting
--regionstringSlice[]additional region(s) to sweep (repeatable); defaults to the cluster + client regions

roksbnkctl cluster

ROKS cluster lifecycle (separate from BNK trials)

Manage the ROKS cluster as a durable, reusable resource that sits underneath your BNK trials.

Commands: roksbnkctl cluster up Create the ROKS cluster (+ transit gateway, registry COS) roksbnkctl cluster down Destroy the cluster and everything cluster-scoped roksbnkctl cluster register Discover an already-existing cluster and persist its identity roksbnkctl cluster config Print the recorded cluster identity from cluster-outputs.json roksbnkctl cluster status Live runtime status (endpoints + node readiness)

Each roksbnkctl up against this workspace will reuse the registered cluster (reading cluster-outputs.json) so multiple BNK trials can share one cluster.

roksbnkctl cluster config

Print the recorded cluster identity (cluster-outputs.json)

Aliases: show

roksbnkctl cluster config [flags]

Prints the cluster identity recorded at cluster up / register time (~/.roksbnkctl/<workspace>/cluster-outputs.json): cluster ID, endpoints, VPC, transit gateway, registry COS, and the create-time settings the cluster was built with (network mode, and the VPC address block when this tool created the VPC). This is the RECORDED config, and for the create-time settings it is AUTHORITATIVE — config.yaml describes what is wanted, this describes what exists. For live runtime state (node readiness) use roksbnkctl cluster status.

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)

← back to roksbnkctl cluster

roksbnkctl cluster down

Destroy the cluster phase (ROKS + cluster-shared services)

roksbnkctl cluster down [flags]

Tears down everything roksbnkctl cluster up created. Refuses to run if any BNK trial state exists for this workspace — destroy those first with roksbnkctl down to avoid orphaned BNK resources.

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the destroy confirmation
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl cluster

roksbnkctl cluster output

Print the Cluster phase’s own terraform outputs (text or –json; [name] = one raw value)

roksbnkctl cluster output [name] [flags]

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)
--show-sensitiveboolfalsereveal sensitive output values (default redacted)

← back to roksbnkctl cluster

roksbnkctl cluster register

Discover an existing ROKS cluster and persist its identity

roksbnkctl cluster register [cluster-name-or-id] [flags]

Looks up an existing ROKS cluster in your IBM Cloud account, verifies its registry COS instance exists, and writes the cluster’s identity to ~/.roksbnkctl/<workspace>/cluster-outputs.json.

Subsequent roksbnkctl up runs in this workspace will pick up the registered cluster automatically — no need to repeat its identity in trial tfvars.

The registry COS instance is found by probing the names roksbnkctl and the upstream HCL actually use: “<prefix>-registry-cos” (what cluster up creates), “<cluster>-registry-cos”, then the upstream fallback “<cluster>-cos”. Pass –registry-cos-name to override (e.g. if your tfvars sets roks_cos_instance_name to something else).

Flags

FlagTypeDefaultDescription
--promptboolfalseprompt for the cluster name even if one is given as an argument
--registry-cos-namestring—registry COS instance name (default: probe “<prefix>-registry-cos”, “<cluster>-registry-cos”, “<cluster>-cos”)

← back to roksbnkctl cluster

roksbnkctl cluster status

Live runtime status of the Cluster phase (endpoints + node readiness)

roksbnkctl cluster status [flags]

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)

← back to roksbnkctl cluster

roksbnkctl cluster up

Provision the ROKS cluster (and cluster-shared services) only

roksbnkctl cluster up [flags]

Creates the durable cluster-shared infrastructure only — the ROKS cluster, transit gateway, and registry COS. cert-manager and the BNK trial modules (flo, cne_instance, license) deploy in the BNK phase; the jumphosts deploy in the testing phase. On success, writes the cluster’s identity to ~/.roksbnkctl/<workspace>/cluster-outputs.json so subsequent roksbnkctl up runs can deploy the BNK and testing phases onto this cluster.

Uses a separate state directory (~/.roksbnkctl/<workspace>/state-cluster/) so it doesn’t tangle with BNK or testing state.

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the confirmation prompt before apply
--no-kubeconfigboolfalseskip the post-apply admin kubeconfig fetch
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl cluster

roksbnkctl config

Print an example config.yaml or .env, or convert one into the other

Prints the workspace input in either form, for piping to a file.

With no –from flag it prints an ANNOTATED template — every setting with the comment explaining it, which is what you want when starting from nothing:

roksbnkctl config yaml > config.yaml roksbnkctl config env > .env

With –from-yaml or –from-env it prints the POPULATED equivalent, without comments — the same settings expressed in the other form:

roksbnkctl config env –from-yaml config.yaml > .env roksbnkctl config yaml –from-env .env > config.yaml

The conversion carries what the input actually sets. A setting left at its default is omitted rather than written out, so the result stays a statement of intent instead of a snapshot of every default at the moment it ran – except the handful of REQUIRED fields, which are emitted empty so a missing one is visible rather than silently absent.

roksbnkctl config env

Print an annotated .env of ROKSBNKCTL_* overrides, or the .env equivalent of a config.yaml

roksbnkctl config env [flags]

Flags

FlagTypeDefaultDescription
--from-envstring—read settings from a .env file and print the populated equivalent (no comments)
--from-yamlstring—read settings from a config.yaml and print the populated equivalent (no comments)

← back to roksbnkctl config

roksbnkctl config yaml

Print an annotated config.yaml, or the config.yaml equivalent of a .env

roksbnkctl config yaml [flags]

Flags

FlagTypeDefaultDescription
--from-envstring—read settings from a .env file and print the populated equivalent (no comments)
--from-yamlstring—read settings from a config.yaml and print the populated equivalent (no comments)

← back to roksbnkctl config

roksbnkctl cos

Manage IBM Cloud Object Storage (instances, buckets, objects)

roksbnkctl cos provides full CRUD on the COS supply chain BNK depends on: COS instances (via Resource Controller), buckets, and keyed objects (FAR pull keys, JWT licenses, etc.). All calls go through the IBM Go SDKs — no ibmcloud CLI dependency.

roksbnkctl cos bucket

Manage COS buckets

Flags

FlagTypeDefaultDescription
--instancestring—COS instance name or CRN (required)

← back to roksbnkctl cos

roksbnkctl cos bucket create

Create a bucket on the named instance

roksbnkctl cos bucket create <bucket> [flags]

Flags

FlagTypeDefaultDescription
--classstringstandardstorage class (standard, vault, cold, smart)
--regionstring—bucket region (default: workspace region)

← back to roksbnkctl cos bucket

roksbnkctl cos bucket delete

Delete a bucket (must be empty)

roksbnkctl cos bucket delete <bucket>

← back to roksbnkctl cos bucket

roksbnkctl cos bucket get

Recursively download every object in a bucket to a local directory

roksbnkctl cos bucket get <bucket> <local-dir> [flags]

Recursively download every object in <bucket> to <local-dir>.

<local-dir> is created if it does not exist (mkdir -p semantics, mode 0755). Object keys map to nested subdirectories — a key foo/bar/baz.json lands at <local-dir>/foo/bar/baz.json. Streaming download per object (no whole-object in-memory buffering); text and binary are both copied through verbatim.

Default behavior is overwrite (the operator just asked to download); pass –no-clobber to skip objects whose local target already exists, matching cp -n.

With –output json, emits one JSON object per file completed: {“key”,“local_path”,“size”,“outcome”}. The final counts line goes to stderr in text mode; the JSON stream concludes with one summary record of shape {“counts”:{…}}.

Flags

FlagTypeDefaultDescription
--no-clobberboolfalseskip objects whose local target already exists (cp -n semantics)

← back to roksbnkctl cos bucket

roksbnkctl cos bucket list

List buckets on the named instance

← back to roksbnkctl cos bucket

roksbnkctl cos instance

Manage COS instances (service instances under Resource Controller)

← back to roksbnkctl cos

roksbnkctl cos instance create

Create a COS instance

roksbnkctl cos instance create <name> [flags]

Create a COS service instance under the workspace’s resource group.

–plan accepts a friendly name (standard | lite); –plan-id takes a catalog UUID directly when IBM ships a tier roksbnkctl hasn’t mapped yet. –target defaults to “global” (COS instances are global; buckets carry the regional affinity).

Flags

FlagTypeDefaultDescription
--planstringstandardservice plan name (standard | lite)
--plan-idstring—service plan UUID (overrides –plan; for plans roksbnkctl hasn’t mapped)
--targetstringglobaltarget region (default: global; COS instances are global)

← back to roksbnkctl cos instance

roksbnkctl cos instance delete

Delete a COS instance (and its bound resources unless –no-recursive)

roksbnkctl cos instance delete <name> [flags]

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the confirmation prompt
--no-recursiveboolfalsedo NOT delete bound resources (HMAC keys, service credentials)

← back to roksbnkctl cos instance

roksbnkctl cos instance list

List COS instances in the current account

← back to roksbnkctl cos instance

roksbnkctl cos object

Manage objects in COS buckets

Flags

FlagTypeDefaultDescription
--instancestring—COS instance name or CRN (required)

← back to roksbnkctl cos

roksbnkctl cos object delete

Delete an object

roksbnkctl cos object delete <bucket>/<key>

← back to roksbnkctl cos object

roksbnkctl cos object get

Download an object (streaming)

roksbnkctl cos object get <bucket>/<key> <local-file>

← back to roksbnkctl cos object

roksbnkctl cos object list

List objects (optionally under a prefix)

roksbnkctl cos object list <bucket>[/<prefix>]

← back to roksbnkctl cos object

roksbnkctl cos object put

Upload an object (multipart for large files, streaming)

roksbnkctl cos object put <bucket>/<key> <local-file>

← back to roksbnkctl cos object

roksbnkctl doctor

Check prerequisites and report missing pieces

roksbnkctl doctor [flags]

Verifies the host has what roksbnkctl needs.

Required (hard fail on missing):

  • terraform on PATH (the local backend’s workhorse for roksbnkctl up)

Informational (the binary internalises each surface; missing → no warning):

  • kubectl / oc — internalised via client-go (roksbnkctl k *)
  • ibmcloud — bundled image, run via –backend docker / –backend ssh:<target>
  • iperf3 — bundled image, run via –backend k8s
  • dig — DNS probe internalised via miekg/dns

A stock dev box with only terraform installed should produce exit 0 and zero warnings.

Pass –target <name> to additionally probe an SSH target (runs whoami). Pass –backend k8s | ssh:<target> for per-backend prereq checks.

Exits non-zero only when a required check fails (warnings don’t block).

Flags

FlagTypeDefaultDescription
--backendstring—additionally run per-backend checks: k8s | ssh:<target>
--targetstring—additionally probe the named SSH target with whoami

roksbnkctl down

Destroy everything in the workspace — terraform destroy

roksbnkctl down [flags]

Tears down the workspace’s phases in reverse-dependency order: BNK and Testing in parallel, then the cluster. Two behaviours worth knowing:

  • If the cluster was attached to an EXISTING (shared) Transit Gateway, down auto-detaches that connection first — removing only this cluster’s connection; the shared gateway and every other cluster’s connection are left intact.
  • If this workspace CREATED a VPC that another cluster’s subnets still live in (the shared-VPC topology), down refuses: the VPC owner must be torn down LAST, so tear the workspaces sharing the VPC down first.

The Gateway and FLP phases are separate and optional; if present, tear them down first (down reports which command to run).

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the destroy confirmation
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

roksbnkctl exec

Run a single command with cluster context loaded

roksbnkctl exec [command...]

roksbnkctl flp

F5 License Proxy phase — optional in-cluster licensing proxy

Manage the F5 License Proxy (FLP) as an optional, independent phase on an existing cluster: it deploys the f5-license-proxy chart (vault + postgresql + proxy) pulling from the configured registry (Harbor mirror or FAR), and records its root CA + service endpoint so a subsequent BNK install can license in FLP mode. Runs in its own state (state-flp/), separate from cluster/BNK/testing/gateway.

Commands: roksbnkctl flp up Install the F5 License Proxy (needs a cluster) roksbnkctl flp down Remove the F5 License Proxy, leaving the other phases intact

Typical flow (FLP-licensed BNK): roksbnkctl cluster up # or cluster register for an existing cluster roksbnkctl registry replicate # mirror FAR into your registry (optional) roksbnkctl flp up # install the proxy roksbnkctl bnk up # with bnk.license_mode: f5licenseproxy

FLP is opt-in: without it, BNK licenses with a subscription JWT as before.

roksbnkctl flp down

Remove the F5 License Proxy, leaving the other phases

roksbnkctl flp down [flags]

Destroys only the FLP-phase resources (state-flp/), leaving cluster, BNK, testing and gateway intact, and clears flp-outputs.json.

Exits 0 (“nothing to do”) when there’s no FLP state, so it’s safe in a reverse-order teardown of every phase.

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the destroy confirmation
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl flp

roksbnkctl flp output

Print the FLP phase’s own terraform outputs (text or –json; [name] = one raw value)

roksbnkctl flp output [name] [flags]

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)
--show-sensitiveboolfalsereveal sensitive output values (default redacted)

← back to roksbnkctl flp

roksbnkctl flp status

Show the F5 License Proxy’s live status (services, listener, F5/TEEM, CNE fields)

roksbnkctl flp status [flags]

Fetches /api/status from the FLP’s status service (the flp-status container, which runs alongside the proxy on the VSI pod or in the cluster) and renders it.

The service URL is derived from the workspace’s flp-outputs.json (the FLP endpoint host, port 80) unless –url is given. Output honors -o json.

Flags

FlagTypeDefaultDescription
--urlstring—flp-status base URL (default: derived from the FLP endpoint, port 80)

← back to roksbnkctl flp

roksbnkctl flp up

Install the F5 License Proxy (a cluster must exist)

roksbnkctl flp up [flags]

Deploys the F5 License Proxy into the existing cluster (read from cluster-outputs.json) in its own state (state-flp/), and writes flp-outputs.json (root CA + endpoint) for a later bnk up in f5licenseproxy mode.

Refuses when no cluster exists yet.

With –add-node-port-access the proxy is also reachable from OUTSIDE its own cluster, so a BNK install in a DIFFERENT cluster (same VPC, or across a transit gateway) can license through it — a “shared licensing cluster”, where only the cluster running the proxy needs egress to F5. That additionally puts the worker node IPs in the proxy’s server certificate (without them the remote cluster’s controller rejects the TLS handshake) and records an external endpoint in flp-outputs.json. Feed that endpoint + CA to the other workspace via its bnk.flp.external block.

Flags

FlagTypeDefaultDescription
--add-node-port-accessboolfalseexpose the proxy outside its cluster (NodePort + node-IP cert SANs) so a BNK install in another cluster can license through it
--autoboolfalseskip the confirmation prompt before apply
--node-port-source-cidrstringSlice[]with –add-node-port-access: open the NodePort on the worker security group to this CIDR. REPEATABLE — a multi-zone VPC has one address prefix PER ZONE, and a consuming pod scheduled in an unlisted zone is silently dropped. Pass every zone’s CIDR (or a supernet covering them).
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl flp

roksbnkctl gateway

Gateway-phase (data-plane config) lifecycle — optional ingress/egress setup

Manage the BNK data-plane configuration as an optional, independent phase on top of a healthy BNK deployment: the Gateway API objects (GatewayClass, F5BnkGateway, Gateway, HTTPRoute), the egress SnatPool + Egress CRs, the per-zone static routes, and the cluster security-group VXLAN rule. Runs in its own state (state-gateway/), separate from cluster/BNK/testing.

Commands: roksbnkctl gateway up Apply the data-plane config (needs a healthy BNK) roksbnkctl gateway down Remove the data-plane config, leaving cluster/BNK/testing intact

Run this only after roksbnkctl up has brought up the cluster + BNK and the BNK is healthy (TMM pods Ready). The composite up/down never runs this phase — it is opt-in.

roksbnkctl gateway down

Remove the BNK data-plane config, leaving cluster/BNK/testing

roksbnkctl gateway down [flags]

Destroys only the Gateway-phase resources (state-gateway/), leaving the cluster, BNK and testing phases intact.

Exits 0 (“nothing to do”) when there’s no gateway state, so it’s safe in a reverse-order teardown of every phase.

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the destroy confirmation
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl gateway

roksbnkctl gateway output

Print the Gateway phase’s own terraform outputs (text or –json; [name] = one raw value)

roksbnkctl gateway output [name] [flags]

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)
--show-sensitiveboolfalsereveal sensitive output values (default redacted)

← back to roksbnkctl gateway

roksbnkctl gateway status

Runtime status of the Gateway phase (CR readiness + assigned address)

roksbnkctl gateway status [flags]

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)

← back to roksbnkctl gateway

roksbnkctl gateway up

Apply the BNK data-plane config (BNK must be healthy)

roksbnkctl gateway up [flags]

Applies the Gateway API + SnatPool + Egress + static-route CRs and the VXLAN security-group rule against the existing cluster + BNK (read from cluster-outputs.json). Runs in its own state (state-gateway/).

Refuses when no cluster/BNK exists yet.

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the confirmation prompt before apply
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl gateway

roksbnkctl get

Get one or more resources (pods, nodes, services, CRDs, …)

roksbnkctl get <resource> [name] [-n <ns> | -A] [-l <selector>] [-o <fmt>] [flags]

Fetches Kubernetes resources via client-go, no host kubectl required.

The resource argument accepts plurals, singulars, and short names from RESTMapper (pods/pod/po, services/svc, deployments/deploy). Multiple types can be comma-separated:

roksbnkctl k get pods,services -n f5-bnk roksbnkctl k get nodes -o yaml roksbnkctl k get pods -A -l app.kubernetes.io/name=f5-lifecycle-operator roksbnkctl k get pod my-pod -n default -o jsonpath=‘{.status.phase}’

CRDs work via dynamic discovery without a hardcoded list — roksbnkctl k get cneinstances resolves the same way kubectl does.

Flags

FlagTypeDefaultDescription
--all-namespaces / -Aboolfalselist across all namespaces
--namespace / -nstring—namespace scope (default: current-context’s namespace)
--output / -ostring—output format: yaml | json | wide | name | jsonpath=… | go-template=…
--selector / -lstring—label selector (e.g. ‘app=foo,tier!=cache’)

roksbnkctl ibmcloud

Passthrough to local ibmcloud with workspace API key + region loaded

roksbnkctl ibmcloud [args...]

roksbnkctl init

Interactive setup; writes the workspace config.yaml

roksbnkctl init [flags]

roksbnkctl init walks through the prompts (region, resource group, cluster, BNK version) and writes ~/.roksbnkctl/<workspace>/config.yaml.

On first run with no -w flag, creates and uses the ‘default’ workspace. Re-run with –upgrade-tf to bump the pinned Terraform source to its latest release.

Flags

FlagTypeDefaultDescription
--config-filestring—path or http(s) URL to a workspace config.yaml to seed (non-interactive when complete; see init example)
--non-interactiveboolfalsebuild config.yaml from environment variables ALONE — no prompts, no –config-file (for argv+env runners; pair with the ROKSBNKCTL_* env vars)
--override-from-envboolfalseafter seeding, overlay config.yaml fields from environment variables (e.g. IBMCLOUD_API_KEY → ibmcloud.api_key_b64)
--tf-sourcestring—override TF source (path or URL); relative local paths are resolved to absolute before being pinned into config.yaml
--upgrade-tfboolfalseresolve and pin the latest TF release into config.yaml

roksbnkctl init example

Print an annotated example config.yaml to stdout (a template, or for piping)

Writes an annotated config.yaml to stdout — the canonical declarative input, with the required fields filled and every optional axis documented (cluster create-or-attach, BYO infrastructure reuse, BNK install, gateway, registry mirror, remote state). config.yaml is the single input to a workspace; the embedded terraform is internal.

Create a starting config or inspect the knobs with ordinary pipes:

roksbnkctl init example > config.yaml roksbnkctl -w demo init –config-file config.yaml roksbnkctl init example | grep -E ‘manifest_version|cluster_vpc’

← back to roksbnkctl init

roksbnkctl install

Copy the running roksbnkctl binary into a directory on PATH

roksbnkctl install [flags]

Install the roksbnkctl binary you’re currently running into a directory on $PATH so you can invoke it as roksbnkctl from any working directory.

Default destination: Linux/macOS, in order of preference: $HOME/.local/bin (preferred — typically writable without sudo) $HOME/bin (older convention; still on PATH for some setups) /usr/local/bin (system-wide; usually needs sudo) Windows: a writable directory already on %PATH% — preferring %LOCALAPPDATA%\Microsoft\WindowsApps (on the per-user PATH by default, no admin) — so the binary resolves immediately. Falls back to %LOCALAPPDATA%\Programs\roksbnkctl (with a PATH hint) if none is usable.

Override the destination with –dir.

Idempotent: if the running binary already lives at the destination, prints a message and exits 0. Use –force to overwrite (useful right after a local rebuild that landed at the install path).

Examples: roksbnkctl install # default — ~/.local/bin roksbnkctl install –dir ~/bin # specific user dir sudo roksbnkctl install –dir /usr/local/bin # system-wide

Note: this is distinct from roksbnkctl self update, which pulls the latest GitHub release tarball over the network.

Flags

FlagTypeDefaultDescription
--dirstring—destination directory (default: a PATH dir — ~/.local/bin on Unix, %LOCALAPPDATA%\Microsoft\WindowsApps on Windows)
--forceboolfalseoverwrite even if destination resolves to the running binary

roksbnkctl journal

Append-only timeline + report for agentic-mode handoffs

Manage the workspace journal (<workspace>/journal/), the append-only timeline the agentic-mode personas use to hand off to one another.

roksbnkctl journal add “<note>” Append a note to today’s journal roksbnkctl journal list List entries chronologically with summaries roksbnkctl journal report Assemble report.md from decisions + journal

roksbnkctl journal add

Append a note to today’s journal entry

roksbnkctl journal add <message>

← back to roksbnkctl journal

roksbnkctl journal list

List journal entries (chronological) with one-line summaries

← back to roksbnkctl journal

roksbnkctl journal report

Assemble report.md from decisions.md + the journal timeline

← back to roksbnkctl journal

roksbnkctl k

Kubernetes verbs (kubectl-internalised; no host kubectl required)

roksbnkctl k <verb> runs the BNK-relevant kubectl/oc verb subset natively in-process via client-go, with no host kubectl/oc binary required. Output formatting matches kubectl byte-for-byte for -o yaml/json/wide/name/jsonpath/go-template.

Verbs: k get list/show resources k describe human-friendly resource detail (delegates to kubectl/pkg/describe) k apply server-side apply for files, dirs, kustomize bases, or stdin k delete delete with cascade + grace period control k logs pod or component logs (extends roksbnkctl logs) k exec exec into a pod via SPDY k port-forward forward a local port to a pod via SPDY

The existing roksbnkctl kubectl / roksbnkctl oc passthroughs remain as escape hatches for verbs not internalised here (edit, patch, rollout, scale, etc.) — they require kubectl/oc on PATH.

roksbnkctl k apply

Server-side apply YAML/JSON manifests, directories, or kustomize bases

roksbnkctl k apply -f <file-or-dir> [-n <ns>] [--force] [flags]

Server-side apply with field-manager ‘roksbnkctl’.

-f <file> single YAML/JSON file (multi-doc YAML supported) -f <dir> directory: kustomization.yaml-detected → krusty build; otherwise recursive *.yaml / *.yml -f - stdin (multi-doc YAML)

–force passes through to SSA’s force-conflicts flag, identical to kubectl apply –server-side –force-conflicts.

Examples:

roksbnkctl k apply -f deploy.yaml -n f5-bnk roksbnkctl k apply -f manifests/ cat deploy.yaml | roksbnkctl k apply -f -

Flags

FlagTypeDefaultDescription
--filename / -fstring—file, directory, or ‘-’ for stdin
--forceboolfalseforce-conflicts on server-side apply (kubectl apply –force-conflicts)
--namespace / -nstring—namespace for namespaced resources without an explicit namespace field

← back to roksbnkctl k

roksbnkctl k delete

Delete resources by name or label selector

roksbnkctl k delete <resource> [name] [-n <ns> | -A] [-l <selector>] [--force] [--grace-period N] [--cascade orphan|background|foreground] [flags]

Deletes resources via the dynamic client. Cascade options match kubectl’s:

–cascade=background delete the object; controller cleans dependents async (default) –cascade=foreground block until dependents are gone –cascade=orphan delete only the object, leave dependents

Examples:

roksbnkctl k delete pod my-pod -n f5-bnk roksbnkctl k delete pods -l app=stale –force –grace-period=0 roksbnkctl k delete deployment foo –cascade=foreground

Flags

FlagTypeDefaultDescription
--all-namespaces / -Aboolfalsedelete across all namespaces
--cascadestringbackgroundcascade: orphan|background|foreground
--forceboolfalseforce-delete: implies –grace-period=0 unless overridden
--grace-periodint-1graceful termination period (seconds); -1 = use resource default
--namespace / -nstring—namespace scope
--selector / -lstring—label selector

← back to roksbnkctl k

roksbnkctl k describe

Show detailed human-readable resource info (events, conditions, related objects)

roksbnkctl k describe <resource> [name] [-n <ns> | -A] [-l <selector>] [flags]

Delegates to k8s.io/kubectl/pkg/describe — the same library kubectl/oc use internally, so output is byte-equivalent.

Examples:

roksbnkctl k describe pod my-pod -n f5-bnk roksbnkctl k describe nodes roksbnkctl k describe deployment f5-lifecycle-operator -n f5-bnk

Flags

FlagTypeDefaultDescription
--all-namespaces / -Aboolfalsedescribe across all namespaces
--namespace / -nstring—namespace scope
--selector / -lstring—label selector
--show-eventsbooltrueinclude the Events block (kubectl default: true)

← back to roksbnkctl k

roksbnkctl k exec

Exec into a pod via SPDY (kubectl-equivalent in-process)

roksbnkctl k exec <pod> [-n <ns>] [-c <container>] [-i] [-t] -- <cmd> [args...] [flags]

Opens an exec stream against the named pod over SPDY. The semantics mirror kubectl exec:

-i / –stdin attach stdin to the remote process -t / –tty allocate a PTY (use for top, bash-style interactive work) -c / –container pick a container in a multi-container pod

Examples:

roksbnkctl k exec my-pod – ls /tmp roksbnkctl k exec my-pod -it – bash roksbnkctl k exec my-pod -c sidecar – cat /etc/hostname

Note: this is the cluster-side exec. The host-side equivalent is ‘roksbnkctl exec <cmd>’ — distinct on purpose (PRD 02 §“Disambiguating roksbnkctl exec”, Option B).

Flags

FlagTypeDefaultDescription
--container / -cstring—container name in a multi-container pod
--namespace / -nstring—namespace scope (default: default)
--stdin / -iboolfalseattach stdin
--tty / -tboolfalseallocate a PTY

← back to roksbnkctl k

roksbnkctl k get

Get one or more resources (pods, nodes, services, CRDs, …)

roksbnkctl k get <resource> [name] [-n <ns> | -A] [-l <selector>] [-o <fmt>] [flags]

Fetches Kubernetes resources via client-go, no host kubectl required.

The resource argument accepts plurals, singulars, and short names from RESTMapper (pods/pod/po, services/svc, deployments/deploy). Multiple types can be comma-separated:

roksbnkctl k get pods,services -n f5-bnk roksbnkctl k get nodes -o yaml roksbnkctl k get pods -A -l app.kubernetes.io/name=f5-lifecycle-operator roksbnkctl k get pod my-pod -n default -o jsonpath=‘{.status.phase}’

CRDs work via dynamic discovery without a hardcoded list — roksbnkctl k get cneinstances resolves the same way kubectl does.

Flags

FlagTypeDefaultDescription
--all-namespaces / -Aboolfalselist across all namespaces
--namespace / -nstring—namespace scope (default: current-context’s namespace)
--output / -ostring—output format: yaml | json | wide | name | jsonpath=… | go-template=…
--selector / -lstring—label selector (e.g. ‘app=foo,tier!=cache’)

← back to roksbnkctl k

roksbnkctl k logs

Stream pod logs (kubectl-equivalent direct path)

roksbnkctl k logs <pod-name> [-n <ns>] [-c <container>] [-f] [--previous] [--since 5m] [--tail N] [flags]

Streams logs for a named pod. Differs from the top-level ‘roksbnkctl logs <component>’ in that this takes a literal pod name — matching kubectl’s surface — while the component variant maps a known BNK component name to a label selector.

Both forms honour -n, -c, -f, –previous, –since, –tail.

Flags

FlagTypeDefaultDescription
--container / -cstring—container name in a multi-container pod
--follow / -fboolfalsefollow log output
--namespace / -nstring—namespace scope (default: default)
--previousboolfalsefetch logs from the previous container instance
--sincestring—only return logs newer than this duration (e.g. 5s, 2m, 1h)
--tailint64-1tail the last N lines (-1 = full log)

← back to roksbnkctl k

roksbnkctl k port-forward

Forward local port(s) to a pod via SPDY

Aliases: port_forward

roksbnkctl k port-forward <pod> [-n <ns>] <local-port>[:<remote-port>] [...] [flags]

Forwards one or more local TCP ports to ports on the named pod. Equivalent to kubectl port-forward; signal handling closes the tunnel cleanly on Ctrl+C.

Port spec:

8080 local 8080 → pod 8080 8080:80 local 8080 → pod 80
80 ephemeral local port → pod 80

Examples:

roksbnkctl k port-forward my-pod 8080:80 roksbnkctl k port-forward my-pod -n f5-bnk 9090:9090 8080:80

Flags

FlagTypeDefaultDescription
--namespace / -nstring—namespace scope (default: default)

← back to roksbnkctl k

roksbnkctl kubeconfig

Print the kubeconfig path (or contents with –export)

roksbnkctl kubeconfig [flags]

Flags

FlagTypeDefaultDescription
--clusterstring—cluster name or ID for –download (default: workspace cluster.name)
--downloadboolfalsefetch admin kubeconfig from IBM Cloud and save to ~/.kube/config
--exportboolfalseprint kubeconfig contents instead of path
--refreshboolfalseforce-refresh the forge kubeconfig now (re-fetch the admin client certs)

roksbnkctl kubectl

Passthrough to local kubectl with workspace KUBECONFIG loaded

roksbnkctl kubectl [args...]

roksbnkctl logs

Tail logs for a BNK component (flo, cis, cert-manager, cneinstance)

roksbnkctl logs <component> [flags]

Looks up the named BNK component, finds its pod(s) by label, and streams logs to stdout. With –follow, streams live. With multiple matching pods, tails the first and prints a hint about using roksbnkctl kubectl for per-pod selection.

The component → namespace/selector map is hardcoded for v1 against the upstream TF chart’s default labels; if your install renamed namespaces or relabelled, fall back to:

roksbnkctl kubectl logs -n <ns> <pod>

Flags

FlagTypeDefaultDescription
--container / -cstring—container name in a multi-container pod
--follow / -fboolfalsefollow log output
--namespace / -nstring—override the component’s default namespace
--previousboolfalsefetch logs from the previous container instance
--sincestring—only return logs newer than this duration (e.g. 5s, 2m, 1h)
--tailint64-1tail the last N lines (-1 = full log)

roksbnkctl oc

Passthrough to local oc with workspace KUBECONFIG loaded

roksbnkctl oc [args...]

roksbnkctl ops

Manage the in-cluster ops pod (k8s execution backend)

roksbnkctl ops manages the long-lived ops pod the k8s execution backend exec’s tools into. The pod runs in the roksbnkctl-ops namespace with a least-privilege ServiceAccount + ClusterRole, and gets its IBM Cloud API key from a Secret apply-time-templated from the workspace credential.

Subcommands: install apply the embedded manifests (idempotent) show print pod + Secret + RBAC status uninstall delete every roksbnkctl.io/managed object created by install

roksbnkctl ops install

Apply (or update) the in-cluster ops fixtures

roksbnkctl ops install [flags]

Applies the embedded namespaces, ServiceAccount, Secret, ClusterRole, ClusterRoleBinding, and ops Pod. Idempotent: re-running with a new API key updates the Secret and rolls the Pod.

Credential mode is selected via –trusted-profile (auto|on|off):

auto (default) — provision an IBM Cloud IAM trusted profile linked to the ops pod’s ServiceAccount when the resolved API key has ‘iam-identity’ perms; otherwise fall back to the static-key Secret with a stderr warning. on — require the trusted-profile path; fail loudly if perms don’t allow. off — skip the trusted-profile path; install the v1.0.x static-key Secret.

Flags

FlagTypeDefaultDescription
--trusted-profilestringautoIBM IAM trusted profile mode: auto (default; provision when perms allow, fall back to static-key Secret), on (require trusted profile), off (static-key Secret only)

← back to roksbnkctl ops

roksbnkctl ops show

Print the ops pod’s status, image, RBAC subject, and Secret rotation timestamp

← back to roksbnkctl ops

roksbnkctl ops uninstall

Delete the ops fixtures (namespaces, RBAC, Pod, Secret)

roksbnkctl ops uninstall [flags]

Flags

FlagTypeDefaultDescription
--confirmboolfalseactually perform the uninstall (otherwise prints what would be deleted)

← back to roksbnkctl ops

roksbnkctl output

Print the merged outputs across all phases (cluster + bnk + testing + gateway + flp)

roksbnkctl output [name] [flags]

Print the union of every phase’s own outputs — each read from its owning phase’s state, so values are the populated ones and never conflict. This is the “everything” view; the per-phase <phase> output commands scope to just that phase’s managed attributes.

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)
--show-sensitiveboolfalsereveal sensitive output values (default redacted)

roksbnkctl plan

Read-only; show what roksbnkctl up would change

roksbnkctl plan [flags]

Flags

FlagTypeDefaultDescription
--outstring—save the plan to <file> (binary plan + a readable <file>.txt) for later apply --plan
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

roksbnkctl registry

Air-gap registry mirror — replicate BNK artifacts into a private registry (PRD 11)

Manage the air-gap registry mirror: replicate every chart + image a BNK install needs (the f5-bigip-k8s-manifest set plus the non-F5 dependencies) from the F5 Artifact Repository (repo.f5.com) into a private target — IBM Container Registry (ICR) or a generic OCI registry (Artifactory / Harbor / Quay) — so an air-gapped cluster installs BNK from a registry it controls.

Commands: roksbnkctl registry target Show or set the mirror target (icr|generic) roksbnkctl registry bom Build + print the bill-of-materials roksbnkctl registry list List the artifacts the last replicate recorded roksbnkctl registry diff Show what replicate would copy (BOM vs. mirror) roksbnkctl registry replicate Copy the BOM into the mirror (registry-to-registry; no cluster) roksbnkctl registry verify Confirm every BOM artifact is present + digest-matched roksbnkctl registry prune Remove mirrored artifacts no longer in the BOM roksbnkctl registry delete Delete ALL replicated artifacts from the target

registry bom works entirely offline against the FAR manifest. The other verbs need a configured registry: block and network reachability to the target registry — replicate also needs the FAR source (repo.f5.com). NONE require a Kubernetes cluster: replicate copies registry-to-registry (via go-containerregistry) from wherever roksbnkctl runs, so you can pre-seed the mirror as a standalone supply-chain step before any cluster exists.

roksbnkctl registry adopt

Record a mirror this workspace did not populate, so bnk up can use it

roksbnkctl registry adopt [flags]

Writes registry-mirror.json for a mirror that already exists.

bnk up refuses to render against a mirror the workspace has no record of — otherwise BNK would be pointed at far_repo_url, which an air-gapped cluster cannot reach. Until now only registry replicate wrote that record, which means a workspace could only use a mirror it had populated itself.

That is the wrong constraint for how mirrors are actually used. The registry is filled once, as a supply-chain step, and then many installs pull from it — often from a different workspace, a different host, or a different team. Those installs were forced to re-run replicate purely to re-derive a record, which needs the FAR source reachable at install time. An air-gapped operator frequently does not have that, and it is the whole point of having mirrored in the first place.

adopt derives the record from the configured registry target: the chart and image hosts, the repo namespace, the manifest version, and the mirror CA all come from the workspace config, so no source access is needed. It then asks the MIRROR what it holds under the configured prefix — a sanity check that catches a typo in the prefix or an empty registry, without pretending to prove the contents are correct.

Pass –verify-contents when the source IS reachable and you want proof rather than assertion: it builds the BOM and digest-checks every artifact before recording, and the record then carries the full artifact inventory.

Flags

FlagTypeDefaultDescription
--forceboolfalserecord the mirror even when it holds nothing under the configured prefix
--verify-contentsboolfalsedigest-check every BOM artifact before recording (needs the FAR source)

← back to roksbnkctl registry

roksbnkctl registry bom

Build + print the BNK bill-of-materials from the FAR manifest

roksbnkctl registry bom [flags]

Pulls the f5-bigip-k8s-manifest for the configured BNK release, parses it into the full artifact set (charts + images), unions the non-F5 dependencies, and prints the resulting BOM. Needs no target — it only reads from FAR.

Flags

FlagTypeDefaultDescription
--far-repo-urlstring—FAR registry host (default: workspace bnk.far_repo_url, else repo.f5.com)
--include-depsboolfalseforce-include the non-F5 dependency artifacts (cert-manager, node-labeler)
--jsonboolfalseemit the BOM as JSON (overrides –output)
--manifest-versionstring—BNK manifest version (default: workspace bnk.manifest_version)
--no-include-depsboolfalseexclude the non-F5 dependency artifacts
--source-sa-b64string—FAR _json_key_base64 service account (default: workspace registry.source_service_account_b64)

← back to roksbnkctl registry

roksbnkctl registry delete

Delete ALL replicated artifacts from the target registry

roksbnkctl registry delete [flags]

Removes every artifact roksbnkctl replicated (recorded in registry-mirror.json) from the configured target, then clears the mirror record so the BNK install reverts to pulling from FAR. Destructive — pass –force to skip the confirmation.

Deletion is by digest where recorded (the reliable form for a registry manifest DELETE). Artifacts that fail to delete are kept in the record so a re-run retries them. For target=icr the API key needs Manager (delete) rights on the namespace; for target=generic the registry must have deletes enabled.

Flags

FlagTypeDefaultDescription
--forceboolfalseskip the confirmation prompt
--targetstring—mirror target backend: icr|generic (default: workspace registry.target, else “icr”)

← back to roksbnkctl registry

roksbnkctl registry diff

Show BOM vs. mirror drift (what replicate would copy)

roksbnkctl registry diff [flags]

Flags

FlagTypeDefaultDescription
--far-repo-urlstring—FAR registry host (default: workspace bnk.far_repo_url, else repo.f5.com)
--include-depsboolfalseforce-include the non-F5 dependency artifacts (cert-manager, node-labeler)
--manifest-versionstring—BNK manifest version (default: workspace bnk.manifest_version)
--no-include-depsboolfalseexclude the non-F5 dependency artifacts
--source-sa-b64string—FAR _json_key_base64 service account (default: workspace registry.source_service_account_b64)

← back to roksbnkctl registry

roksbnkctl registry list

List artifacts currently recorded in the mirror

Lists the artifacts the last replicate recorded in registry-mirror.json.

← back to roksbnkctl registry

roksbnkctl registry prune

Remove mirrored artifacts no longer in the BOM

roksbnkctl registry prune [flags]

Flags

FlagTypeDefaultDescription
--far-repo-urlstring—FAR registry host (default: workspace bnk.far_repo_url, else repo.f5.com)
--include-depsboolfalseforce-include the non-F5 dependency artifacts (cert-manager, node-labeler)
--manifest-versionstring—BNK manifest version (default: workspace bnk.manifest_version)
--no-include-depsboolfalseexclude the non-F5 dependency artifacts
--source-sa-b64string—FAR _json_key_base64 service account (default: workspace registry.source_service_account_b64)
--targetstring—mirror target backend: icr|generic (default: workspace registry.target, else “icr”)

← back to roksbnkctl registry

roksbnkctl registry replicate

Copy the BOM into the mirror (registry-to-registry; no cluster needed)

roksbnkctl registry replicate [flags]

Prepares the target registry (auth + repository namespace), then copies every BOM artifact into it, idempotently. Records the result in registry-mirror.json so the BNK install can be redirected to the mirror.

Runs entirely host-side: it pulls each artifact by digest from the FAR source (repo.f5.com) and pushes to the target registry via go-containerregistry — no Kubernetes cluster is involved. So you can pre-seed the mirror as a standalone supply-chain step, before creating any cluster, from any host that can reach both the FAR source and the target registry. Requires a configured registry: block and the FAR source credential (registry.source_service_account_b64, the workspace FAR auth, or –source-sa-b64).

Flags

FlagTypeDefaultDescription
--concurrencyint0parallel copy workers (default: 4)
--far-repo-urlstring—FAR registry host (default: workspace bnk.far_repo_url, else repo.f5.com)
--include-depsboolfalseforce-include the non-F5 dependency artifacts (cert-manager, node-labeler)
--insecure-capture-caboolfalseadopt a self-signed mirror CA over an UNAUTHENTICATED connection (trust-on-first-use); prefer –registry-ca or –registry-ca-fingerprint
--manifest-versionstring—BNK manifest version (default: workspace bnk.manifest_version)
--no-include-depsboolfalseexclude the non-F5 dependency artifacts
--registry-castring—PEM CA the mirror serves TLS with, for air-gap node trust (preferred: the file you generated; else registry.generic_ca_b64)
--registry-ca-fingerprintstring—expected SHA-256 of the mirror CA (“sha256:ab:cd…” or bare hex), authenticating a captured CA out of band
--source-sa-b64string—FAR _json_key_base64 service account (default: workspace registry.source_service_account_b64)
--targetstring—mirror target backend: icr|generic (default: workspace registry.target, else “icr”)

← back to roksbnkctl registry

roksbnkctl registry target

Show or set the registry mirror target and its fields

roksbnkctl registry target [icr|generic | <field> <value>] [flags]

Configure the registry replication target without hand-editing config.yaml.

With no arguments, prints the current target + configured fields. Otherwise the first argument is either a backend KIND (sets registry.target) or a FIELD name (set with a following value):

Kinds: icr | generic Fields: icr_host icr_namespace generic_host generic_repo_prefix generic_username generic_password generic_ca (a PEM file) generic_ca_sha256

Examples: roksbnkctl registry target # show current config roksbnkctl registry target icr # use IBM Container Registry roksbnkctl registry target icr_namespace bnk-test roksbnkctl registry target generic # use a generic OCI registry roksbnkctl registry target generic_host art.example.com roksbnkctl registry target generic_repo_prefix bnk roksbnkctl registry target generic_username ci-bot echo “$TOKEN” | roksbnkctl registry target generic_password –password-stdin roksbnkctl registry target generic_ca /opt/harbor/certs/harbor.crt

Flags

FlagTypeDefaultDescription
--password-stdinboolfalseread the generic registry password from stdin (for registry target generic_password)

← back to roksbnkctl registry

roksbnkctl registry verify

Confirm every BOM artifact is present + digest-matched in the mirror

roksbnkctl registry verify [flags]

Flags

FlagTypeDefaultDescription
--far-repo-urlstring—FAR registry host (default: workspace bnk.far_repo_url, else repo.f5.com)
--include-depsboolfalseforce-include the non-F5 dependency artifacts (cert-manager, node-labeler)
--manifest-versionstring—BNK manifest version (default: workspace bnk.manifest_version)
--no-include-depsboolfalseexclude the non-F5 dependency artifacts
--source-sa-b64string—FAR _json_key_base64 service account (default: workspace registry.source_service_account_b64)
--targetstring—mirror target backend: icr|generic (default: workspace registry.target, else “icr”)

← back to roksbnkctl registry

roksbnkctl self

Manage the roksbnkctl binary itself

roksbnkctl self update

Pull the latest roksbnkctl release matching the host arch

Downloads the latest GitHub release for this platform, verifies its SHA256 against the release’s checksums.txt, and replaces the running binary in place. Works on Linux, macOS, and Windows.

This is the latest-only, interactive form; use roksbnkctl upgrade to pin a specific --version or to skip the prompt with --yes.

Requires write permission on the binary’s directory (typical install under /usr/local/bin needs sudo; brew/scoop should use their own upgrade verb).

← back to roksbnkctl self

roksbnkctl shell

Interactive bash with KUBECONFIG, IBMCLOUD_API_KEY, and region pre-loaded

roksbnkctl shell drops into a $SHELL subshell with the workspace’s KUBECONFIG, IBMCLOUD_API_KEY, IC_API_KEY, and IBMCLOUD_REGION exported so locally-installed kubectl / oc / ibmcloud commands work without further setup. Exits when the subshell does.

roksbnkctl state

Manage where terraform state lives (local vs COS/S3 remote backend)

roksbnkctl state local

Use local per-phase terraform state (the default) — writes config.yaml for you

← back to roksbnkctl state

roksbnkctl state migrate

Copy each phase’s local terraform state into the configured COS/S3 backend

roksbnkctl state migrate [flags]

Migrates a workspace’s terraform state from local files into the COS/S3 remote backend declared in config.yaml’s state: block (PRD 16).

Pre-requisite: set state.backend = “s3” (+ state.s3.{endpoint,bucket,region}) and export the COS HMAC keys (ROKSBNKCTL_COS_HMAC_ACCESS_KEY / ROKSBNKCTL_COS_HMAC_SECRET_KEY, or AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) before running.

For each deployed phase (cluster / bnk / testing / gateway), this runs terraform init -migrate-state to copy its local state to the per-phase key in the bucket. It refuses to overwrite a key that already holds state (pass –force to override). The local state files are left in place; verify the remote read-back before deleting them.

Flags

FlagTypeDefaultDescription
--forceboolfalsemigrate even if the remote key already holds state (overwrites it)

← back to roksbnkctl state

roksbnkctl state s3

Use the COS/S3 remote terraform-state backend — writes the state: block for you

roksbnkctl state s3 [flags]

Switch this workspace to the COS/S3 remote state backend (PRD 16) by writing its state: block to config.yaml — no hand-edit.

You still provision the bucket + HMAC credentials yourself: the HMAC access/ secret keys are NEVER written to config.yaml — only the names of the env vars they come from are (–access-key-source / –secret-key-source, defaulting to ROKSBNKCTL_COS_HMAC_ACCESS_KEY / ROKSBNKCTL_COS_HMAC_SECRET_KEY). After this, export those env vars and run roksbnkctl state migrate to move existing state.

Flags

FlagTypeDefaultDescription
--access-key-sourcestring—env var holding the HMAC access key (default ROKSBNKCTL_COS_HMAC_ACCESS_KEY)
--bucketstring—pre-provisioned bucket name (required)
--endpointstring—COS S3 endpoint URL (required)
--key-prefixstring—state key prefix (default: the workspace name)
--regionstring—COS location / region (required)
--secret-key-sourcestring—env var holding the HMAC secret key (default ROKSBNKCTL_COS_HMAC_SECRET_KEY)

← back to roksbnkctl state

roksbnkctl state show

Show the workspace’s terraform-state backend config

← back to roksbnkctl state

roksbnkctl status

Summary of the workspace: cluster, components, last apply

roksbnkctl status [flags]

roksbnkctl status reports a quick read of the workspace:

  • workspace name + region
  • configured cluster name
  • pinned Terraform source
  • per-phase deployment status (Cluster, BNK trial, Testing, Gateway)
  • kubeconfig path (if any)
  • cluster reachability (node count + ready count)

v1.x will add per-BNK-component readiness (flo, cis, cert-manager, cneinstance) once the component-discovery shape is finalised.

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)

roksbnkctl targets

Manage SSH targets used by –on

Targets are named SSH endpoints stored under the workspace’s targets: block. They become reachable via the persistent –on flag on commands like roksbnkctl exec, roksbnkctl shell, roksbnkctl kubectl, etc.

A jumphost target is auto-populated after a successful roksbnkctl up when the upstream HCL provisions one (testing_tgw_jumphost outputs).

roksbnkctl targets add

Add or update a target

roksbnkctl targets add <name> --host H --user U [--port P] [--key-path P | --key-source S] [flags]

Flags

FlagTypeDefaultDescription
--hoststring—host or IP
--key-pathstring—path to a PEM private key
--key-sourcestring—key source — “agent” or “tf-output:<name>”
--portint0ssh port (default 22)
--userstring—remote user

← back to roksbnkctl targets

roksbnkctl targets list

List all targets in the current workspace

← back to roksbnkctl targets

roksbnkctl targets remove

Remove a target

roksbnkctl targets remove <name>

← back to roksbnkctl targets

roksbnkctl targets show

Show detail for one target

roksbnkctl targets show <name>

← back to roksbnkctl targets

roksbnkctl terraform

Read-only passthrough to terraform against the workspace’s managed state

Aliases: tf

roksbnkctl terraform [subcommand] [args...]

roksbnkctl terraform runs a READ-ONLY terraform subcommand against the workspace’s managed state, with terraform’s working directory and TF_DATA_DIR resolved for you (no need to cd into ~/.roksbnkctl/… or export TF_DATA_DIR).

This command is read-only by allowlist. Permitted subcommands: output, show, state list, state show, state pull, providers, version, graph, validate, fmt -check

Everything else — including apply, destroy, init, plan, import, taint, untaint, state rm/mv/replace-provider, and any -auto-approve — is rejected before terraform runs. Mutations go exclusively through roksbnkctl up / plan / apply / down (or cluster/bnk up/down); running terraform apply/destroy outside the orchestration skips the rendered tfvars, the apply-retry wrapper, the post-apply kubeconfig fetch, the applied-tfvars snapshot, and the auto-jumphost seeding, and desyncs the managed state.

Use –phase cluster to target the cluster-phase state (/.roksbnkctl/<ws>/state-cluster/); the default is the trial/single state (/.roksbnkctl/<ws>/state/). –on is not supported: the managed terraform state lives on this workstation, not on a jumphost.

Examples: roksbnkctl terraform output testing_cluster_jumphost_ssh_commands roksbnkctl terraform state list roksbnkctl –phase cluster terraform show

roksbnkctl test

Run deployment validation tests (default: all)

roksbnkctl test [suite] [flags]

roksbnkctl test runs deployment validation against the current workspace.

Suites: connectivity HTTP/HTTPS reachability of deployed BNK services dns DNS resolution of ingress and service hostnames throughput iperf3 measurements (north-south by default; v1.x) all run all of the above (default if no suite is specified)

Honors -o json with the roksbnkctl.v1 schema. Exit code 0 on all-pass, non-zero on any-fail — CI-friendly.

Flags

FlagTypeDefaultDescription
--insecureboolfalseskip TLS certificate validation (connectivity only)

roksbnkctl test connectivity

HTTP/HTTPS reachability against configured hosts

← back to roksbnkctl test

roksbnkctl test dns

DNS resolution probe (single-vantage, GSLB-compare, or workspace-driven)

roksbnkctl test dns [flags]

roksbnkctl test dns runs DNS probes against configured resolvers.

Two modes:

Workspace-driven (no flags) — resolves each host listed under test.connectivity.extra_hosts via the std-lib resolver. Same as Sprint 0–4 behaviour; preserves CI invocations using the legacy roksbnkctl.v1 schema.

Flag-driven (any of –target/–type/–server/–gslb-compare set) — uses the embedded miekg/dns probe (no external dig install needed). Single-vantage emits roksbnkctl.dns.v1.vantage; –gslb-compare emits roksbnkctl.dns.v1 with a gslb_divergence boolean across all configured backends (local + k8s + ssh:<targets>).

Use –backend local|k8s|ssh:<target> to pick a single vantage point; –gslb-compare fans out across all available vantages. PRD 03 §“DNS probe (GSLB-aware)”.

Flags

FlagTypeDefaultDescription
--gslb-compareboolfalsefan out the probe across all configured backends (local + k8s + ssh:<targets>) and emit a comparison JSON with gslb_divergence
--iterationsint1number of repeated queries; >1 enables RTT distribution
--require-divergenceboolfalsewith –gslb-compare: exit non-zero if NO divergence is observed (CI assertion that GSLB is doing something)
--serverstring—resolver: <ip>[:<port>] | system | cluster | <named-from-workspace> (default: system)
--targetstring—DNS name to query (overrides workspace test.dns.default_target)
--timeoutduration2sper-query timeout
--typestringArecord type: A | AAAA | CNAME | MX | NS | TXT | SRV | SOA | PTR | CAA | DS | DNSKEY | ANY

← back to roksbnkctl test

roksbnkctl test hosts

Manage test.connectivity.extra_hosts in the workspace config

Test hosts are URLs probed by roksbnkctl test connectivity and (in workspace-driven mode) roksbnkctl test dns. They are stored under test.connectivity.extra_hosts in the workspace’s config.yaml.

This command group is the first-class CLI for managing that slice — without it, the only path is hand-editing ~/.roksbnkctl/<workspace>/config.yaml. Mirrors roksbnkctl targets’ ergonomics: idempotent add/remove, -o json on list.

← back to roksbnkctl test

roksbnkctl test hosts add

Append URLs to test.connectivity.extra_hosts (idempotent)

roksbnkctl test hosts add <url> [<url> ...]

Appends each <url> to test.connectivity.extra_hosts. Idempotent — adding an already-present URL is a no-op (logs to stderr; exit 0).

Each <url> is validated via the std-lib url.Parse; non-URLs are rejected with an actionable error naming the offending arg. Insertion order is preserved.

← back to roksbnkctl test hosts

roksbnkctl test hosts clear

Remove ALL entries from test.connectivity.extra_hosts

roksbnkctl test hosts clear [flags]

Clears test.connectivity.extra_hosts. Confirmation prompt defaults to No; pass --auto to skip the prompt (matches the roksbnkctl down / cluster down confirmation pattern).

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the confirmation prompt

← back to roksbnkctl test hosts

roksbnkctl test hosts list

List configured test hosts (one per line; -o json for array)

Prints the workspace’s test.connectivity.extra_hosts one per line on stdout. Empty list emits zero bytes + exit 0 (NOT an error; distinguishes “nothing configured” from “command failed” by exit code).

With -o json, emits the slice as a JSON array ([] when empty).

← back to roksbnkctl test hosts

roksbnkctl test hosts remove

Remove URLs from test.connectivity.extra_hosts (idempotent)

roksbnkctl test hosts remove <url> [<url> ...]

Removes each <url> from test.connectivity.extra_hosts. Idempotent — removing an absent URL is a no-op (logs to stderr; exit 0). Preserves the order of remaining entries.

← back to roksbnkctl test hosts

roksbnkctl test list

List available test suites

← back to roksbnkctl test

roksbnkctl test matrix

Run the declarative BNK-on-ROKS performance grid (iperf3 L4 + h2load L7)

roksbnkctl test matrix [flags]

roksbnkctl test matrix runs a declarative performance grid against an already-deployed cluster + BNK. It reads a matrix.yaml describing cells (endpoint-pair × test-family) and runs each, emitting a report shaped like the BNK-on-ROKS perf plan.

Two families: iperf3 — L4 TCP throughput over an L4Route VIP, with content-size knobs (length: “128” vs “512K”) — the L4 analog of the plan’s 128 B / 512 KB payload axis. l7 — h2load against an HTTPRoute, http and https (TLS terminate at TMM); cps / tps / throughput modes.

The locality axis (same-zone / different-zone / different-VPC) is implicit in which jumphost (“vsi”) endpoint a cell names as its client, so the per-AZ jumphost targets the Testing phase auto-registers (jumphost, jumphost-<zone>) are the traffic-source fleet. The Testing phase preinstalls both generators (iperf3 + h2load/nghttp2-client) on every jumphost, so the ssh runs need no –bootstrap.

This command provisions only ephemeral, runner-owned fixtures (an iperf3 server, an HTTP file backend, and optional L4Route/HTTPRoute/TLS objects that attach to the EXISTING Gateway by name) and tears them down after (unless –keep). It never touches Terraform or the gateway-phase objects.

roksbnkctl test matrix –dry-run expand + print the plan, no cluster calls roksbnkctl test matrix –only ‘L7*’ run a subset (glob on cell name) roksbnkctl test matrix -o md Markdown grid report

Honors -o json|text|md with the roksbnkctl.v1 schema. Exit code is non-zero if any cell fails.

Flags

FlagTypeDefaultDescription
--dry-runboolfalseexpand the grid and print the plan + fixtures; no cluster calls
--filestring—path to matrix.yaml (default: <workspace>/matrix.yaml, then ./matrix.yaml)
--keepboolfalseleave fixtures running after the run
--onlystring—glob on cell name; run only matching cells

← back to roksbnkctl test

roksbnkctl test throughput

iperf3 throughput; deploys server pod automatically (v1.x)

roksbnkctl test throughput [flags]

Deploys an iperf3 server in the test namespace and runs the client either from the roksbnkctl host (–mode north-south, default) or from a second in-cluster pod (–mode east-west).

Not yet implemented — landing in v1.x once the internal/k8s client-go fixture lifecycle is wired.

Flags

FlagTypeDefaultDescription
--cross-nodeboolfalseforce east-west client and server onto different nodes
--keepboolfalseleave the iperf3 server pod running after the test
--modestringnorth-souththroughput mode: north-south | east-west

← back to roksbnkctl test

roksbnkctl testing

Testing-phase (jumphost) lifecycle — provisions the test rig

Manage the testing jumphost infrastructure (TGW jumphost, per-AZ cluster jumphosts, client VPC) as an independent phase that sits beside the BNK phase on top of a shared cluster. Pure IBM VPC — no Kubernetes.

Commands: roksbnkctl testing up Provision the jumphosts (needs a cluster: VPC + transit gateway) roksbnkctl testing down Destroy the jumphosts, leaving the cluster and BNK intact roksbnkctl testing status Live jumphost IPs / SSH commands + reachability

This is the provisioning phase. To RUN connectivity/DNS/throughput probes against a deployed environment, use roksbnkctl test (and roksbnkctl test hosts to manage the probe target list) — those provision nothing.

roksbnkctl testing down

Destroy the testing jumphosts, leaving the cluster and BNK

roksbnkctl testing down [flags]

Destroys only the testing jumphost resources (state-testing/), leaving the cluster phase and the BNK phase intact. The inverse of bnk down: each phase tears down independently. Exits 0 (“nothing to do”) when there’s no testing state, so it’s safe in a reverse-order teardown of every phase.

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the destroy confirmation
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl testing

roksbnkctl testing output

Print the Testing phase’s own terraform outputs (text or –json; [name] = one raw value)

roksbnkctl testing output [name] [flags]

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)
--show-sensitiveboolfalsereveal sensitive output values (default redacted)

← back to roksbnkctl testing

roksbnkctl testing status

Live runtime status of the Testing phase (jumphost IPs + SSH reachability)

roksbnkctl testing status [flags]

Flags

FlagTypeDefaultDescription
--jsonboolfalseoutput JSON (CI-friendly)

← back to roksbnkctl testing

roksbnkctl testing up

Provision the testing jumphosts (cluster must exist)

roksbnkctl testing up [flags]

Provisions the testing jumphosts against the existing cluster’s VPC + transit gateway (read from cluster-outputs.json). Runs in its own state (state-testing/), independent of the BNK phase — bnk down leaves the jumphosts and testing down leaves BNK.

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the confirmation prompt before apply
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl testing

roksbnkctl tfvars

Emit the upstream TF’s terraform.tfvars.example for editing

roksbnkctl tfvars [flags]

Resolves the workspace’s pinned TF source (downloading the tarball if not yet cached) and writes its terraform.tfvars.example as a starting point you can edit and pass to roksbnkctl up.

Default writes to ./terraform.tfvars in the current directory. Pass -o <path> to write elsewhere, or -o - to print to stdout.

Refuses to overwrite an existing destination unless –force is set.

Workflow: roksbnkctl init # pins a TF source roksbnkctl tfvars # writes ./terraform.tfvars from the upstream example $EDITOR ./terraform.tfvars roksbnkctl up –var-file ./terraform.tfvars

Flags

FlagTypeDefaultDescription
--forceboolfalseoverwrite the destination if it already exists
--output / -ostring./terraform.tfvarsdestination file (or - for stdout)

roksbnkctl tgw

Attach the cluster to an existing Transit Gateway (share one across clusters)

Connect this workspace’s cluster VPC to an EXISTING IBM Transit Gateway, by name or id — at create time or after. Multiple workspaces can point at the same gateway; each owns its own connection.

Commands: roksbnkctl tgw connect <name-or-id> Attach the cluster VPC to the gateway roksbnkctl tgw disconnect Detach (leaves the gateway + other clusters) roksbnkctl tgw status Gateway id/name + live connection state

Runs in its own state (state-tgw/), separate from cluster/BNK/testing/gateway/FLP, and works against a created OR a registered cluster.

roksbnkctl tgw connect

Attach the cluster VPC to an existing Transit Gateway

roksbnkctl tgw connect [transit-gateway-name-or-id] [flags]

Attaches this workspace’s cluster VPC to an existing Transit Gateway.

The gateway may be given by NAME or by ID — either is resolved against your account. Pass it as the argument (persisted to config for later re-applies), or set resources.transit_gateway (create: false, existing: <name-or-id>) in config.yaml / the init interview.

Idempotent and shareable: run it in several workspaces with the same gateway and each cluster gets its own connection to the one gateway.

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the confirmation prompt before apply
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl tgw

roksbnkctl tgw disconnect

Detach the cluster VPC from its Transit Gateway

roksbnkctl tgw disconnect [flags]

Removes ONLY this cluster’s connection to the Transit Gateway. The gateway itself and every other cluster’s connection to it are left intact.

Exits 0 (“nothing to do”) when there’s no connection.

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the destroy confirmation
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

← back to roksbnkctl tgw

roksbnkctl tgw status

Show the Transit Gateway id/name and the live connection state

← back to roksbnkctl tgw

roksbnkctl uninstall

Remove the installed roksbnkctl binary from ~/.local/bin (opposite of install)

roksbnkctl uninstall [flags]

Delete the roksbnkctl binary that roksbnkctl install copied onto $PATH.

By default it removes <install-dir>/roksbnkctl, where <install-dir> is the same directory install uses (~/.local/bin, then ~/bin). Override with –dir.

It refuses to delete the binary you are currently running on Windows (a running .exe cannot be removed there) — delete it manually or run uninstall from a different binary. On Linux/macOS removing the running (installed) binary is fine.

Examples: roksbnkctl uninstall # remove ~/.local/bin/roksbnkctl roksbnkctl uninstall –dir ~/bin sudo roksbnkctl uninstall –dir /usr/local/bin

Flags

FlagTypeDefaultDescription
--dirstring—directory to remove roksbnkctl from (default: ~/.local/bin or ~/bin)
--yes / -yboolfalseskip the confirmation prompt

roksbnkctl up

Provision (or attach) and deploy BNK — terraform plan + apply

roksbnkctl up [flags]

roksbnkctl up validates credentials, resolves the pinned Terraform source, runs plan, and (after confirmation, unless –auto) applies. Idempotent and resumable: a partial failure is recovered by re-running ‘roksbnkctl up’.

Flags

FlagTypeDefaultDescription
--autoboolfalseskip the confirmation prompt before apply
--no-kubeconfigboolfalseskip the post-apply admin kubeconfig fetch
--tf-sourcestring—override TF source for this run only (path or URL; relative local paths resolved against the invocation CWD)
--var-filestringArray[]extra TF var-file (repeatable; later files override earlier)

roksbnkctl upgrade

Upgrade (or pin) the roksbnkctl binary to a GitHub release

roksbnkctl upgrade [flags]

Downloads a roksbnkctl release for this OS/arch from GitHub, verifies its SHA256 against the release’s checksums.txt, and replaces the running binary in place. With no –version it upgrades to the latest release; –version pins a specific release (and may downgrade or reinstall).

Works on Linux, macOS, and Windows. On Windows the running .exe cannot be overwritten, so it is moved aside to <binary>.old and the new binary takes its place; the .old file is removed automatically on the next run.

Requires write permission on the binary’s directory (a /usr/local/bin install needs sudo; Homebrew/Scoop installs should use their own upgrade verb).

Note: release binaries are not yet code-signed, so on a host with an application-allowlist policy (e.g. Windows Device Guard/WDAC) the freshly downloaded binary may be blocked until its hash is trusted.

Flags

FlagTypeDefaultDescription
--versionstring—release to install, e.g. v1.20.1 (default: latest)
--yes / -yboolfalseskip the confirmation prompt

roksbnkctl version

Print version, commit, and build date

roksbnkctl workspaces

Manage roksbnkctl workspaces (per-environment config + state bundles)

Aliases: ws

Each workspace lives under ~/.roksbnkctl/<name>/ with its own config.yaml and state. The current_workspace pointer in ~/.roksbnkctl/config.yaml decides which one commands run against; -w/–workspace overrides for one invocation.

roksbnkctl workspaces current

Print the current workspace name

← back to roksbnkctl workspaces

roksbnkctl workspaces delete

Delete a workspace (refuses if state is non-empty unless –force)

roksbnkctl workspaces delete <name> [flags]

Flags

FlagTypeDefaultDescription
--forceboolfalsedelete even if Terraform state lists provisioned resources

← back to roksbnkctl workspaces

roksbnkctl workspaces list

List workspaces and their states

← back to roksbnkctl workspaces

roksbnkctl workspaces new

Create a new (empty) workspace skeleton — run roksbnkctl init -w <name> to populate

roksbnkctl workspaces new <name>

← back to roksbnkctl workspaces

roksbnkctl workspaces use

Set the current workspace pointer

roksbnkctl workspaces use <name>

← back to roksbnkctl workspaces

Configuration reference

Field-by-field schema for the workspace config.yaml, generated from the Workspace struct in internal/config/workspace.go. This chapter is the single source of truth for what a field is called, what type it takes, and which BNK line it applies to.

For a one-page version carrying the full dotted paths and each field’s ROKSBNKCTL_* override, see the config.yaml cheatsheet — generated from the same struct, so the two cannot disagree.

Chapter 12 — Workspace config is the teaching chapter: use it to learn the shape. Use this one to look up a specific field. Every other chapter links here rather than restating fields, because a field restated in four places is a field that will disagree with itself by the next release.

The line column says which BNK release line a field applies to. Most apply to both. A field marked 2.4 has no effect on a 2.3 install and vice versa — the line itself is selected by bnk.manifest_version and nothing else.

Required means the field has no omitempty: it is always rendered, and its zero value is meaningful. It does not mean you must type it into your config.yaml.

Default is carried on the field as a default:"..." struct tag. A dash means no default is declared — either the zero value applies, or the default is computed at run time and belongs in the prose rather than a table cell.

Workspace

Workspace is ~/.roksbnkctl/<name>/config.yaml. Mirrors the per-workspace example in docs/PRD.md. Note that there is no api_key field — secrets live in env vars or the OS keychain, never in this struct. Plaintext keys in the YAML are rejected at load time by rejectPlaintextSecrets.

keytypelinedefaultrequireddescription
ibmcloudIBMCloudCfg2.3 + 2.4—yesIBMCloud is the account context every phase runs against: region, resource group, and the API key (base64-obfuscated, never plaintext).
clusterClusterCfg2.3 + 2.4—yesCluster describes the ROKS cluster — either one to CREATE, or one that already exists and is being adopted (create: false plus its name).
bnkBNKCfg2.3 + 2.4—noBNK configures the BIG-IP Next for Kubernetes install itself: the manifest version (which also decides whether the 2.3 or 2.4 model is rendered), licensing, the per-zone network mapping, and the CNEInstance settings that place the TMM pods.
gatewayGatewayCfg2.3 + 2.4—noGateway configures the Gateway API resources installed after BNK.
registry*RegistryCfg2.3 + 2.4—noRegistry selects where images and charts are pulled from: F5’s Artifact Repository directly, or a mirror that a disconnected cluster can reach — IBM Container Registry, or any OCI registry (generic) such as Artifactory or Harbor.
testTestCfg2.3 + 2.4—noTest configures the connectivity, DNS and throughput probes roksbnkctl test runs, and the jumphost they run FROM.
tf_sourceTFSourceCfg2.3 + 2.4—yesTFSource selects which Terraform the binary applies: the embedded tree shipped inside this binary (the default, and the only version matched to it), a GitHub release, or a local path for testing a fork.
cos*COSCfg2.3 + 2.4—noCOS is the IBM Cloud Object Storage bucket the FAR service-account credential is read from, for estates that stage it there rather than passing a local file.
targetsmap[string]TargetCfg2.3 + 2.4—noTargets are named remote hosts (ssh:<target>) that Exec backends can run tools on — a jumphost inside the VPC, say, when the operator’s workstation cannot reach the cluster directly.
stateStateCfg2.3 + 2.4—noState controls where Terraform state is kept and how it is locked.
bnkforge*BNKForgeCfg2.3 + 2.4—noBNKForge points at a BNK Forge server, which takes over a durable cluster once roksbnkctl has built it.
agent*AgentCfg2.3 + 2.4—noAgent configures roksbnkctl agent, which hands the workspace to a coding agent.
prefixstring2.3 + 2.4—noPrefix is the workspace’s account-scoped resource-name base.
resources*ResourcesCfg2.3 + 2.4—noResources carries the per-resource create toggles (and the existing-resource name/ID for any declined-but-still-depended-on resource).
execmap[string]ExecToolCfg2.3 + 2.4—noExec is the per-tool execution-backend config block introduced in Sprint 3 (PRD 03).

AdvancedComponentCfg

AdvancedComponentCfg is one component’s advanced settings. Only env is surfaced today; the CR’s advanced block carries more, and this is the shape that lets those arrive without moving what already works.

keytypelinedefaultrequireddescription
envmap[string]string2.3 + 2.4—noEnv sets environment variables on that component’s containers, name → value.

AgentCfg

AgentCfg configures roksbnkctl’s agentic mode (the agent command). It is purely advisory metadata for launching an external coding-agent CLI against the workspace’s scaffolded AGENTS.md + personas/ — roksbnkctl embeds no LLM. nil/absent = agent defaults to claude and the CLI’s own endpoint config.

keytypelinedefaultrequireddescription
defaultstring2.3 + 2.4—noDefault is the agentic CLI roksbnkctl agent (no arg) reports as this workspace’s default — claude | gemini | aider | openai | pi | opencode.
llm_endpointstring2.3 + 2.4—noLLMEndpoint is an optional OpenAI-/Anthropic-compatible base URL woven into the printed invocation (cloud vendor, local vLLM, etc.).

BNKCISCfg

BNKCISCfg configures the BNK CIS controller’s BIG-IP target. All optional. BigIPPasswordB64 stores the password base64-encoded (obfuscation, NOT encryption — like ibmcloud.api_key_b64); the raw value is rendered to terraform.tfvars as bigip_password at apply time.

keytypelinedefaultrequireddescription
bigip_urlstring2.3 + 2.4—noBigIPURL is the management address of the classic BIG-IP that the Container Ingress Services controller configures, e.g. “https://10.1.1.5”.
bigip_usernamestring2.3 + 2.4—noBigIPUsername is the account CIS authenticates to that BIG-IP as.
bigip_password_b64string2.3 + 2.4—noBigIPPasswordB64 is that account’s password, base64-encoded — obfuscation, not encryption.

BNKCertManagerCfg

BNKCertManagerCfg overrides cert-manager’s install coordinates. All optional; the create/skip decision stays on resources.cert_manager.create.

keytypelinedefaultrequireddescription
namespacestring2.3 + 2.4—noNamespace cert-manager installs into (rendered as cert_manager_namespace).
versionstring2.3 + 2.4—noVersion pins the cert-manager Helm chart (rendered as cert_manager_version).

BNKCfg

keytypelinedefaultrequireddescription
cneinstance_sizestring2.3 + 2.4—noCNEInstanceSize is the CNEInstance deploymentSize: Tiny, Small, Medium, Large or Max.
far_repo_urlstring2.3 + 2.4repo.f5.comnoFARRepoURL is the F5 Artifact Repository charts are pulled from.
manifest_versionstring2.3 + 2.4—noManifestVersion pins the BNK release, e.g. “2.4.0-EA”.
far_auth_filestring2.3 + 2.4—noFarAuthFile is the FAR auth tarball’s filename in the orchestration COS bucket; rendered as the f5_cne_far_auth_file tfvar + used by registry to resolve the FAR _json_key_base64 service account.
subscription_jwt_filestring2.3 + 2.4—noSubscriptionJWTFile is the subscription/license JWT’s filename in the orchestration COS bucket; rendered as the f5_cne_subscription_jwt_file tfvar.
far_auth_local_filestring2.3 + 2.4—noFarAuthLocalFile / SubscriptionJWTLocalFile point at LOCAL files instead of COS objects.
subscription_jwt_local_filestring2.3 + 2.4—noSubscriptionJWTLocalFile is the F5 subscription JWT as a LOCAL file, the companion to FarAuthLocalFile above.
trusted_profile*BNKTrustedProfileCfg2.3 + 2.4—noTrustedProfile tunes the IBM Cloud Trusted Profile the CNE controller assumes to manage its own cluster’s VPC.
flo_namespacestring2.3 + 2.4f5-bnknoFLONamespace / FLOUtilsNamespace override the namespaces the F5 Lifecycle Operator and its utility components install into (rendered as flo_namespace / flo_utils_namespace).
flo_utils_namespacestring2.3 + 2.4f5-utilsnoFLOUtilsNamespace is where FLO’s SHARED utility components install — coremond, the observer, the licence — as opposed to FLONamespace, which holds the BNK instance itself.
storage_class_namestring2.3 + 2.4—noStorageClassName is the StorageClass the CNEInstance’s persistent volumes use, TMM’s included.
gateway_api_mtlsbool2.4—noGatewayAPIMTLS opts into the Gateway API bundle BNK 2.4 needs for mTLS (#170).
tmm_replicasint2.43noTMMReplicas is the number of f5-tmm data-plane replicas.
watch_namespaces[]string2.4AllnoWatchNamespaces are the namespaces the CNE controller watches.
tmm_anti_affinity*bool2.4truenoTMMAntiAffinity requires f5-tmm pods onto different nodes.
tmm_anti_affinity_topology_keystring2.4kubernetes.io/hostnamenoTMMAntiAffinityTopologyKey is the node label the anti-affinity rule spreads across — the IBM ROKS per-node label.
tmm_zone_spread*bool2.4truenoTMMZoneSpread spreads f5-tmm pods across zones.
tmm_zone_topology_keystring2.4topology.kubernetes.io/zonenoTMMZoneTopologyKey is the IBM ROKS zone label the spread constraint uses.
tmm_zone_max_skewint2.41noTMMZoneMaxSkew is maxSkew for the zone topology-spread constraint.
tmm_zone_when_unsatisfiablestring2.4DoNotSchedulenoTMMZoneWhenUnsatisfiable is DoNotSchedule or ScheduleAnyway.
tmm_pod_labelstring2.4f5-tmmnoTMMPodLabel is the app label value the placement rules select TMM by.
tmm_rolling_update*bool2.4truenoTMMRollingUpdate pins TMM’s rolling update to maxSurge 0 / maxUnavailable 1 — the same shape as the cwc Multi-Attach deadlock, where an unconstrained rolling update on a single-attach resource wedges.
external_bigip*bool2.4falsenoExternalBigIP enables the external BIG-IP controller.
external_bigip_login_secretstring2.4f5-bigip-ctlr-loginnoExternalBigIPLoginSecret holds the external BIG-IP credentials.
cluster_identifierstring2.4—noClusterIdentifier is passed to the external BIG-IP controller.
gateway_api_versionstring2.41.5.0noGatewayAPIVersion is GATEWAY_API_VERSION for the CNE controller.
gateway_api_bundle_urlstring2.4—noGatewayAPIBundleURL is where the Gateway API standard-install bundle is FETCHED from when gateway_api_mtls is on (#185).
demo_mode*bool2.3 + 2.4false on 2.4, true on 2.3noDemoMode sets advanced.demoMode.enabled.
whole_cluster*bool2.3 + 2.4false on 2.4, true on 2.3noTCPSettings overrides fields on the data-plane F5BigTcpSetting CR.
hugepages*HugepagesCfg2.3 + 2.4—noHugepages optionally allocates hugepages on the worker pool via the OpenShift Node Tuning Operator.
tcp_settingsmap[string]string2.3 + 2.4—noTCPSettings overrides individual fields on the data-plane F5BigTcpSetting CR by name, e.g. {“congestionControl”: “bbr”, “delayedAcks”: “true”} — the key is the CR’s own spec field.
tcp_settings_namestring2.3 + 2.4sys-default-tcpnoTCPSettingsName is the F5BigTcpSetting to write.
advancedmap[string]AdvancedComponentCfg2.3 + 2.4—noAdvanced carries per-component environment passthrough for the 2.4 CNEInstance’s advanced.<component>.env[] lists (#175).
tmm_resourcesmap[string]map[string]string2.3 + 2.4—noTMMResources overrides the TMM pod’s resource requests/limits, rendered as the CNEInstance’s advanced.tmm.resources (#203).
gslb_datacenter_namestring2.3 + 2.4—noGSLBDatacenterName sets the optional CNEInstance GSLB datacenter name (rendered as cneinstance_gslb_datacenter_name).
cert_manager*BNKCertManagerCfg2.3 + 2.4—noCertManager overrides cert-manager’s namespace + chart version.
network*BNKNetworkCfg2.3 + 2.4—noNetwork holds the optional per-zone subnet CIDRs + TMM self-IPs for the cloud-network-mapping ConfigMap and the external/internal F5SPKVlan CRs (BNK install-guide “Configuration”).
cis*BNKCISCfg2.3 + 2.4—noCIS holds the BIG-IP management endpoint + credentials the BNK CIS controller (k8s-bigip-ctlr) uses.
license_modestring2.3 + 2.4connectednoLicenseMode selects the License CR operationMode: “connected” (default when empty), “disconnected”, or “f5licenseproxy”.
flp*BNKFLPCfg2.3 + 2.4—noFLP holds settings for the optional F5 License Proxy phase.
preflight*BNKPreflightCfg2.3 + 2.4—noPreflight tunes the pre-install checks bnk up runs before it plans anything.

BNKFLPCfg

BNKFLPCfg configures the F5 License Proxy (FLP) phase deployment. All optional; nil block means FLP is off. It never carries secrets — the FLP generates its own certs, and its subscription JWT is the same one resolved from COS.

keytypelinedefaultrequireddescription
modestring2.3 + 2.4—noMode selects HOW the FLP phase deploys the proxy: “” | “helm” → the f5-license-proxy Helm chart into the ROKS cluster (default).
vsi*BNKFLPVSICfg2.3 + 2.4—noVSI configures the mode: vsi deployment backend.
namespacestring2.3 + 2.4—noNamespace the FLP is installed into (helm mode).
chart_versionstring2.3 + 2.4—noChartVersion pins the f5-license-proxy chart.
storage_classstring2.3 + 2.4—noStorageClass is the dynamic StorageClass for the FLP’s PVCs (rendered as flp_storage_class).
node_port_accessbool2.3 + 2.4—noNodePortAccess exposes the proxy OUTSIDE its own cluster, so a BNK install in a DIFFERENT cluster (same VPC, or across a transit gateway) can license through it — the “shared licensing cluster” topology, where only the cluster running the proxy needs egress to F5.
node_port_source_cidrs[]string2.3 + 2.4—noNodePortSourceCIDRs, when set with NodePortAccess, opens the proxy’s NodePort on the cluster’s worker security group to these CIDRs — the consuming cluster’s subnets.
external*BNKFLPExternalCfg2.3 + 2.4—noExternal points a workspace at a FOREIGN proxy — one deployed by a DIFFERENT workspace/cluster.

BNKFLPExternalCfg

BNKFLPExternalCfg addresses an F5 License Proxy that this workspace does not own. Both fields come from the owning workspace’s roksbnkctl flp output — the URL is its externally-reachable endpoint and the CA is its (base64) root CA, which the CWC must trust to complete the TLS handshake to the proxy.

keytypelinedefaultrequireddescription
urlstring2.3 + 2.4—noURL of the proxy, e.g. https://10.240.64.5:30001 — reachable from THIS cluster’s pods.
root_ca_b64string2.3 + 2.4—noRootCAB64 is the proxy’s root CA, base64-encoded (as flp output emits it).

BNKFLPForwardProxyCfg

BNKFLPForwardProxyCfg describes an egress forward proxy for the FLP VSI’s calls to F5’s licensing backend (product-s.apis.f5.com).

keytypelinedefaultrequireddescription
hoststring2.3 + 2.4—noHost is the forward proxy’s address.
portint2.3 + 2.4—noPort is the forward proxy’s port.
protocolstring2.3 + 2.4—nohttp (default) | https.

BNKFLPVSICfg

BNKFLPVSICfg configures the mode: vsi FLP backend — a standalone VSI running the f5-license-proxy stack as a podman pod. All fields optional; sensible defaults apply.

keytypelinedefaultrequireddescription
name_prefixstring2.3 + 2.4—noNamePrefix prefixes the FLP VSI’s IBM Cloud resource names — instance, subnet, security group, floating IP, public gateway, boot volume, and the VPC when this module creates one.
vpcstring2.3 + 2.4—noVPC is an existing VPC id to deploy the standalone FLP VSI into, WITHOUT any ROKS cluster.
create_vpcbool2.3 + 2.4—noCreateVPC builds the proxy its OWN VPC, address prefix and public gateway rather than placing it in one that already exists (#60).
vpc_namestring2.3 + 2.4—noVPCName names the VPC created when CreateVPC is set.
subnet_cidrstring2.3 + 2.4—noSubnetCIDR is the address prefix for that VPC.
profilestring2.3 + 2.4—noProfile is the IBM Cloud VSI instance profile.
zonestring2.3 + 2.4—noZone the VSI lands in (e.g. us-south-1).
boot_size_gbint2.3 + 2.4—noBootSizeGB is the boot volume size.
management_allowed_cidrs[]string2.3 + 2.4—noManagementAllowedCIDRs are the source CIDRs permitted to reach the :80 flp-status web UI (read-only status).
licensing_allowed_cidrs[]string2.3 + 2.4—noLicensingAllowedCIDRs are the source CIDRs permitted to reach the :8443 licensing proxy (and :22 SSH).
allowed_cidrs[]string2.3 + 2.4—noAllowedCIDRs is DEPRECATED — a legacy single list.
ssh_keystring2.3 + 2.4—noSSHKey is the name of an existing IBM Cloud VPC SSH key (RSA) to attach to the FLP VSI, so an operator can SSH in to inspect/recover the licensing appliance (podman pod, Vault, logs).
floating_ip*bool2.3 + 2.4—noFloatingIP attaches an operator floating IP to the FLP VSI for remote management — running roksbnkctl flp status and reaching the :80 web UI + the :8443 proxy from a machine OUTSIDE the VPC.
status_imagestring2.3 + 2.4—noStatusImage, when set, runs the flp-status web UI as a container in the FLP pod (mobile-friendly status page + /api/status + live logs on :80, no auth — a read-only private status endpoint).
status_registry_hoststring2.3 + 2.4—noStatusRegistryHost + StatusRegistryCAB64 make the VSI trust a self-signed mirror so it can pull StatusImage: cloud-init drops the (base64) CA into /etc/containers/certs.d/<host>/ca.crt before the pod comes up.
status_registry_ca_b64string2.3 + 2.4—noStatusRegistryCAB64 is that registry’s CA, base64-encoded, written to /etc/containers/certs.d/<host>/ca.crt on the VSI.
forward_proxy*BNKFLPForwardProxyCfg2.3 + 2.4—noForwardProxy optionally routes the VSI’s egress to F5 licensing through an HTTP forward proxy (air-gapped/egress-controlled networks).

BNKForgeCfg

BNKForgeCfg is the optional integration with a co-located BNK Forge (v3) install. When Register is true, cluster up registers the just-provisioned ROKS cluster with BNK Forge via its REST API — credential-backed, so BNK Forge derives the kubeconfig on demand from an IBM Cloud credential template rather than storing a perishable one. Best-effort: registration never blocks or fails the deploy. nil/absent (legacy config) = no-op.

keytypelinedefaultrequireddescription
registerbool2.3 + 2.4—noRegister opts the workspace in.
urlstring2.3 + 2.4—noURL is the BNK Forge server base URL (e.g. https://forge.example.com).
projectstring2.3 + 2.4—noProject is the target BNK Forge project NAME.
usernamestring2.3 + 2.4—noUsername is the BNK Forge login user.
password_b64string2.3 + 2.4—noPasswordB64 is that user’s password, base64-encoded — obfuscation, NOT encryption, exactly as bnk.cis.bigip_password_b64 and registry.generic_password_b64 are.
insecurebool2.3 + 2.4—noInsecure skips TLS verification against the Forge server.
ca_b64string2.3 + 2.4—noCAB64 pins the CA the Forge server’s certificate must chain to, PEM, base64-encoded.

BNKNetworkCfg

BNKNetworkCfg is the optional cloud-network-mapping / VLAN zone data.

keytypelinedefaultrequireddescription
zones[]BNKZoneCfg2.3 + 2.4—noZones is the per-availability-zone network mapping, in zone order — one entry per zone the cluster spans.
vlan_prefixlen*int2.324noVLANPrefixLen is the self-IP prefix length (spec.prefixlen_v4) TMM applies to its external and internal self-IPs on the F5SPKVlan CRs — the size of the L2 subnet TMM treats as directly connected on each VLAN.
vlan_prefixlen_external*int2.3—noVLANPrefixLenExternal / VLANPrefixLenInternal override VLANPrefixLen for one VLAN.
vlan_prefixlen_internal*int2.3—noVLANPrefixLenInternal is the same override for the INTERNAL VLAN.
tmm_k8s_routesstring2.3 + 2.4172.17.0.0/18noTMMK8SRoutes is the Kubernetes pod CIDR TMM installs a route toward (advanced.tmm.env TMM_K8S_ROUTES), so TMM can reach backend pods on the internal data path.

BNKPreflightCfg

BNKPreflightCfg tunes the per-node reachability gate. These are exposed because the right values are a property of the ENVIRONMENT, not of roksbnkctl. A Transit Gateway attachment is asynchronous — IBM programs the routes some time after the connection reports attached — and how long that takes varies by account, region and gateway. A probe run ~73s after attach saw both targets unreachable and refused an install whose path was healthy minutes later (issue #57), while a sibling cluster on the same gateway passed simply by landing on the other side of route programming. A site that consistently sees slower propagation should raise the budget rather than rediscover the race; a site with a static, long-established gateway can lower it to fail faster.

keytypelinedefaultrequireddescription
reachability_retry_seconds*int2.3 + 2.4—noReachabilityRetrySeconds is how long a target may keep failing before the verdict is believed.
reachability_timeout_seconds*int2.3 + 2.4—noReachabilityTimeoutSeconds is how long to wait for the probe DaemonSet to report from every node.

BNKTrustedProfileCfg

BNKTrustedProfileCfg is the IBM Cloud Trusted Profile the CNE controller assumes at runtime — the identity that lets it manage the VPC network attachments it creates for TMM, without a stored API key. Both fields default in the HCL rather than here, so an absent block renders nothing and the shipped terraform decides — the same rule every other optional BNK field follows.

keytypelinedefaultrequireddescription
service_accountstring2.3 + 2.4—noServiceAccount is the Kubernetes service account the profile is LINKED to: which account may assume it.
roles[]string2.3 + 2.4[Viewer, Editor]noRoles granted to the profile, scoped to the cluster’s OWN VPC.

BNKZoneCfg

BNKZoneCfg is one availability zone’s subnet CIDRs + TMM self-IPs. Field order/names match the terraform cneinstance_network_zones object.

keytypelinedefaultrequireddescription
ext_vlan_cidrstring2.3 + 2.4—yesExtVLANCIDR is the zone’s EXTERNAL VLAN — the client side of the data plane, where traffic arrives.
int_vlan_cidrstring2.3—yesIntVLANCIDR is the zone’s INTERNAL VLAN — the pod side, where BNK reaches the workloads it fronts.
int_snat_cidrstring2.3 + 2.4—yesIntSNATCIDR is the pool BNK source-NATs to when it talks to pods, so the return traffic comes back through TMM rather than routing around it.
int_vip_cidrstring2.3 + 2.4—yesIntVIPCIDR is the range virtual servers are allocated from on the internal side.
external_selfipstring2.3—yesExternalSelfIP is TMM’s own address on the external VLAN.
internal_selfipstring2.3—yesInternalSelfIP is TMM’s own address on the internal VLAN.

COSCfg

COSCfg points roksbnkctl at the IBM Cloud Object Storage that holds the FAR auth key + subscription JWT (the “orchestration” COS). Empty fields fall back to the built-in defaults (bnk-supply-chain / bnk-artifacts / us-south). These are honoured BOTH by the terraform render (ibmcloud_cos_instance_name / ibmcloud_resources_cos_bucket / ibmcloud_cos_bucket_region) AND by the registry FAR-file resolver, so a customer-owned COS bucket is used consistently across both.

keytypelinedefaultrequireddescription
instancestring2.3 + 2.4bnk-supply-chainnoInstance is the IBM Cloud Object Storage service instance holding the bucket.
bucketstring2.3 + 2.4bnk-artifactsnoBucket is the bucket the FAR service-account credential is read from, for estates that stage it centrally rather than passing a local file.
regionstring2.3 + 2.4us-southnoRegion is the bucket’s region, which need not match ibmcloud.region.
resource_groupstring2.3 + 2.4—noResourceGroup is the IBM Cloud resource group holding Instance.
upload[]COSUpload2.3 + 2.4—noUpload lists local files to place into that bucket before the phases that read them run.

COSUpload

keytypelinedefaultrequireddescription
sourcestring2.3 + 2.4—yesSource is the local file to upload.
keystring2.3 + 2.4—yesKey is the object name it is stored under in the bucket.

ClusterCfg

keytypelinedefaultrequireddescription
createbool2.3 + 2.4—yesCreate decides whether this workspace BUILDS the ROKS cluster or adopts one that already exists.
namestring2.3 + 2.4—yesName is the ROKS cluster’s name — the one to create, or the existing one to adopt when Create is false.
openshift_versionstring2.3 + 2.4—noOpenShiftVersion pins the OpenShift version for a CREATED cluster (e.g. “4.20”).
workers_per_zoneint2.3 + 2.41noWorkersPerZone is the worker count PER ZONE, not in total: ROKS spans three availability zones, so 2 here is a six-node cluster.
public_gateway*bool2.3 + 2.4truenoPublicGateway controls whether the cluster subnets attach a public gateway for worker Internet egress.
vpc_cidrstring2.3 + 2.4—noVPCCIDR is the block the cluster VPC’s per-zone address prefixes are carved from — “10.241.0.0/16” becomes 10.241.0.0/18, 10.241.64.0/18, 10.241.128.0/18.
network_modestring2.3 + 2.4single-nicnoNetworkMode selects how the cluster’s worker nodes are attached: “single-nic” (the default, and today’s only behaviour) or “multi-nic”.
existing_subnet_ids[]string2.3 + 2.4—noExistingSubnetIDs places the cluster in subnets that ALREADY EXIST, one per zone in zone order, instead of creating them (#61).
worker_flavorstring2.3 + 2.4—noWorkerFlavor names the worker profile EXACTLY, e.g. “cx3d.8x20”.
min_worker_vcpu_countint2.3 + 2.416noMinWorkerVCPUCount is the vCPU floor for the worker-flavor auto-select: the cluster module picks the smallest bx2 profile meeting this and MinWorkerMemoryGB.
min_worker_memory_gbint2.3 + 2.464noMinWorkerMemoryGB is the memory floor for the same auto-select.

ConnectivityCfg

keytypelinedefaultrequireddescription
extra_hosts[]string2.3 + 2.4—noExtraHosts are additional targets the connectivity probe tries, beyond the defaults.

DNSCfg

DNSCfg drives the Sprint 5 flag-driven DNS probe (PRD 03 §“DNS probe (GSLB-aware)” §“Server resolution”). The map’s keys are the names users pass to --server <name> and the values are concrete “<ip>[:<port>]” strings the miekg/dns client dials. DefaultTarget is used when –target isn’t passed on the command line. Example: test: dns: resolvers: google: “8.8.8.8:53” cloudflare: “1.1.1.1:53” gslb-vip: “169.45.91.5:53” default_target: “www.example.com”

keytypelinedefaultrequireddescription
resolversmap[string]string2.3 + 2.4—noResolvers names the DNS servers to query, as name → “<ip>[:<port>]”.
default_targetstring2.3 + 2.4—noDefaultTarget is the resolver queried when a probe names none.

ExecToolCfg

ExecToolCfg is one entry under workspace.Exec — the chosen backend for a given tool.

keytypelinedefaultrequireddescription
backendstring2.3 + 2.4—yesBackend is the execution-backend spec: “local” | “docker” | “k8s” | “ssh:<target>”.

GatewayCfg

keytypelinedefaultrequireddescription
app_namespacestring2.3 + 2.4f5-appnoAppNamespace is the namespace the example application and its Gateway resources are created in.
class_namestring2.3 + 2.4gateway-classnoClassName is the GatewayClass name.
controller_namestring2.3 + 2.4—noControllerName is the GatewayClass controllerName.
backend_servicestring2.3 + 2.4nginx-servicenoBackendService is the Kubernetes Service the example HTTPRoute forwards to.
backend_portint2.3 + 2.480noBackendPort is the port on that Service.
egress_modestring2.3 + 2.4snatpoolnoEgressMode selects how return traffic is source-addressed: “snatpool”, “automap”, or “both”.
client_subnet_local[]string2.3 + 2.4—noClientSubnetLocal lists client CIDRs reachable on the same VPC as the cluster — routed directly rather than over the transit gateway.
client_subnet_remote[]string2.3 + 2.4—noClientSubnetRemote lists client CIDRs reached ACROSS the transit gateway, e.g. the testing client VPC.
vxlan_portint2.3 + 2.46789noVXLANPort is the UDP port for the VXLAN overlay between TMM and the nodes.
route_examples[]string2.3 + 2.4—noRouteExamples names extra route kinds to create WORKING examples of, alongside the HTTPRoute the gateway phase already creates.
l4_listener_portint2.3 + 2.48080noL4ListenerPort is the port for that TCP listener.

HugepagesCfg

HugepagesCfg allocates hugepages on the worker pool through the OpenShift Node Tuning Operator. BNK’s deploymentSize decides how much TMM asks for: Tiny requests none, Small requests 4Gi of hugepages-2Mi. A stock ROKS worker reports zero — including F5’s approved reference cluster, which runs Tiny for exactly this reason. So any size above Tiny needs this, or nodes prepared some other way. APPLYING THIS REBOOTS WORKERS. The profile sets a bootloader kernel argument, and the Machine Config Operator rolls the pool to apply it — draining and restarting each node in turn. On a live cluster that is a maintenance event, not a configuration change.

keytypelinedefaultrequireddescription
enabledbool2.3 + 2.4—yesEnabled allocates hugepages.
sizestring2.3 + 2.42MnoSize is the hugepage size, e.g. “2M” or “1G”.
countint2.3 + 2.42048noCount is the number of pages PER NODE.
node_rolestring2.3 + 2.4workernoNodeRole is the machineconfiguration.openshift.io/role the profile applies to.
profile_namestring2.3 + 2.4bnk-hugepagesnoProfileName names the Tuned profile and CR.

IBMCloudCfg

keytypelinedefaultrequireddescription
regionstring2.3 + 2.4—yesRegion is the IBM Cloud region everything in this workspace is created in or adopted from, e.g. “us-east”.
resource_groupstring2.3 + 2.4defaultyesResourceGroup is the IBM Cloud resource group resources are placed in, and the one an adopted cluster is looked up in.
api_key_sourcestring2.3 + 2.4—noenv | keychain | config | prompt — see secrets.go.
api_key_b64string2.3 + 2.4—noAPIKeyB64 stores the API key base64-encoded inline in the workspace config.

RegistryCfg

RegistryCfg configures the Sprint 29 air-gap registry mirror (PRD 11): which target the roksbnkctl registry replicate populates and which namespace it uses, plus the optional source/target credentials. All fields are optional — an absent block (nil) means the mirror is not configured and the BNK install pulls directly from FAR (far_repo_url). Additive + omitempty, so existing config.yaml files load unchanged.

keytypelinedefaultrequireddescription
targetstring2.3 + 2.4—noTarget selects the mirror backend: “icr” (IBM Container Registry — the DEFAULT when unset) or “generic” (any OCI-compliant registry — Artifactory / Harbor / Quay / registry:2).
icr_hoststring2.3 + 2.4—noICRHost overrides the IBM Container Registry host for target=icr (e.g. “de.icr.io”).
icr_namespacestring2.3 + 2.4—noICRNamespace is the ICR namespace artifacts nest under for target=icr.
generic_hoststring2.3 + 2.4—noGenericHost is the OCI registry host for target=generic (e.g. “artifactory.example.com”).
generic_repo_prefixstring2.3 + 2.4—noGenericRepoPrefix is the repository path artifacts nest under for target=generic (e.g. an Artifactory repo key).
generic_usernamestring2.3 + 2.4—noGenericUsername / GenericPasswordB64 are the static basic-auth credential for target=generic (an Artifactory user + access token).
generic_password_b64string2.3 + 2.4—noGenericPasswordB64 is that user’s password or access token, base64-encoded — obfuscation, not encryption.
generic_ca_b64string2.3 + 2.4—noGenericCAB64 is the mirror’s CA chain, PEM, base64-encoded — the AUTHORITATIVE copy, recorded from the file that generated it rather than learned from the network.
generic_ca_sha256string2.3 + 2.4—noGenericCASHA256 pins the mirror’s CA by SHA-256 (hex; a “sha256:” prefix and colons are accepted).
namespacestring2.3 + 2.4—noNamespace is the mirror project the artifacts land in.
include_deps*bool2.3 + 2.4—noIncludeDeps unions the non-F5 dependency artifacts (Jetstack cert-manager chart + images, the registry.k8s.io/kubectl node-labeler image) into the BOM.
source_service_account_b64string2.3 + 2.4—noSourceServiceAccountB64 is the FAR _json_key_base64 service-account JSON, base64-encoded, used as the replication SOURCE credential for repo.f5.com.

ResourceToggle

ResourceToggle is one create/reuse decision: Create=true provisions the resource (under its prefix-derived name); Create=false reuses an existing one named by Existing (when a live dependent consumes it).

keytypelinedefaultrequireddescription
createbool2.3 + 2.4—yesCreate decides whether roksbnkctl provisions this resource or adopts one that already exists.
existingstring2.3 + 2.4—noExisting is the name or ID of the resource to adopt when Create is false.

ResourcesCfg

ResourcesCfg holds the per-resource create toggles for a prefix-driven workspace (Sprint 26). The cluster itself is NOT here — it reuses the existing ClusterCfg.Create / ClusterCfg.Name (Name doubles as the existing id/name when Create=false, as today). Each toggle carries an Existing name/ID used when Create=false and a live dependent still needs to reference the resource by name.

keytypelinedefaultrequireddescription
transit_gatewayResourceToggle2.3 + 2.4—yesTransitGateway controls the transit gateway the cluster VPC attaches to.
registry_cosResourceToggle2.3 + 2.4—yesRegistryCOS controls the IBM Cloud Object Storage bucket that backs a mirror registry.
cert_managerResourceToggle2.3 + 2.4—yesCertManager controls whether cert-manager is INSTALLED or adopted.
bnkResourceToggle2.3 + 2.4—yesBNK controls whether the BIG-IP Next for Kubernetes install runs at all.
tgw_jumphostResourceToggle2.3 + 2.4—yesTGWJumphost controls the optional testing jumphost in the client VPC.
cluster_jumphostsResourceToggle2.3 + 2.4—yesClusterJumphosts controls the per-cluster jumphosts.
client_vpcResourceToggle2.3 + 2.4—yesClientVPC controls the testing client VPC that the jumphost lives in.
cluster_vpcResourceToggle2.3 + 2.4—yesClusterVPC controls the ROKS cluster’s OWN VPC.
client_regionstring2.3 + 2.4—noClientRegion is the region the testing client (TGW jumphost + client VPC) is installed in.
testing_client_vpc_namestring2.3 + 2.4tf-testing-vpcnoTestingClientVPCName names the testing client VPC when ClientVPC.Create is true (rendered as testing_client_vpc_name).
testing_ssh_key_namestring2.3 + 2.4—noTestingSSHKeyName is the IBM Cloud VPC SSH key name attached to the testing jumphosts (rendered as testing_ssh_key_name).
testing_jumphost_profilestring2.3 + 2.4—noJumphost sizing.
testing_min_vcpu_countint2.3 + 2.44noTestingMinVCPUCount is the vCPU floor for the jumphost auto-select.
testing_min_memory_gbint2.3 + 2.48noTestingMinMemoryGB is the memory floor for the same auto-select.
testing_jumphost_allowed_cidrs[]string2.3 + 2.4—noSecurity-group source CIDRs, following the flp_vsi module’s split between a management plane and an in-fabric plane.
testing_client_vpc_inbound_cidrs[]string2.3 + 2.4—noTestingClientVPCInboundCIDRs gates the client VPC’s DEFAULT security group, which the testing phase widens to all protocols and ports.
cluster_http_allowed_cidrs[]string2.3 + 2.4—noClusterHTTPAllowedCIDRs gates :80 on the cluster security group — the ingress/ALB path, which is meant to be publicly reachable, so empty means open.
cluster_vpc_default_sg_inbound_cidrs[]string2.3 + 2.4—noClusterVPCDefaultSGInboundCIDRs gates the cluster VPC’s DEFAULT security group, which the cluster phase widens to all protocols and ports.
copied_ssh_key_files[]string2.3 + 2.4—noCopiedSSHKeyFiles lists the ~/.ssh basenames roksbnkctl init ACTUALLY wrote when the user accepted the “copy the private key to ~/.ssh” prompt (only files it created — pre-existing files are skipped, never recorded).

StateCfg

GatewayCfg carries optional overrides for the Gateway phase (the BNK data-plane config). Every field is optional — unset values fall back to the terraform gateway module’s BNK install-guide defaults. Rendered as gateway_* tfvars. The phase itself is driven by roksbnkctl gateway up/down, not a toggle here. StateCfg selects where terraform state lives (PRD 16). Backend “” or “local” (the default) keeps per-phase local tfstate under the workspace dir — the original behaviour. “s3” stores each phase’s state in an S3-compatible bucket (IBM COS), so a stateless runner / parallel CI needs no shared volume, with native lockfile locking (terraform >= 1.10). Additive + omitempty — an absent state: block loads as the local default.

keytypelinedefaultrequireddescription
backendstring2.3 + 2.4—no“” | “local” | “s3”.
s3*StateS3Cfg2.3 + 2.4—noS3 configures an S3-compatible backend (IBM COS) for terraform state, so a workspace is not tied to one machine’s disk.

StateS3Cfg

StateS3Cfg configures the COS/S3 remote backend. The HMAC access/secret keys are NOT stored here — KeySource names the env var they come from (env-first), and roksbnkctl injects them as AWS_ env to the terraform child, never into the rendered HCL or the state object.

keytypelinedefaultrequireddescription
endpointstring2.3 + 2.4—yesCOS S3 endpoint URL.
bucketstring2.3 + 2.4—yespre-provisioned bucket.
regionstring2.3 + 2.4—yesCOS location / region.
key_prefixstring2.3 + 2.4—nodefault: the workspace name.
access_key_sourcestring2.3 + 2.4—noenv var name; default ROKSBNKCTL_COS_HMAC_ACCESS_KEY.
secret_key_sourcestring2.3 + 2.4—noenv var name; default ROKSBNKCTL_COS_HMAC_SECRET_KEY.

TFSourceCfg

TFSourceCfg picks where Terraform’s source tree comes from. Type drives which other fields apply: - embedded — uses the HCL bundled into the roksbnkctl binary via Go’s embed directive. No other fields needed. This is the default and what most users want; install one binary, get CLI + matched TF together. - github — downloads a tarball release from a GitHub repo. Repo (“owner/name”) and Ref (release tag) required. For testing forks or pinning to a specific upstream tag. - local — points Terraform at a directory on disk. Path required. For active development on the HCL itself. An empty Type (legacy / forgot-to-set) is treated as embedded.

keytypelinedefaultrequireddescription
typestring2.3 + 2.4—yesType selects where the Terraform comes from: “embedded” (the tree compiled into this binary — the default, and the only one guaranteed to match it), “github” (a released tree), or “local” (a path on disk, for testing a fork).
repostring2.3 + 2.4—noRepo is the owner/name to fetch from when Type is “github”.
refstring2.3 + 2.4—noRef is the tag, branch or commit to fetch when Type is “github”.
pathstring2.3 + 2.4—noPath is the local directory holding the Terraform tree when Type is “local”.

TargetCfg

TargetCfg is the on-disk shape of one entry under targets: in the workspace config. Lives in this package (rather than internal/remote) to avoid an import cycle: workspace.go needs to (de)serialise it, internal/remote needs to consume it. Keeping the wire shape here and the runtime Target type in internal/remote keeps the dep direction clean (remote → config, never the reverse).

keytypelinedefaultrequireddescription
hoststring2.3 + 2.4—yesHost is the target’s address, reached over SSH.
portint2.3 + 2.422noPort is its SSH port.
userstring2.3 + 2.4—yesUser is the SSH login.
key_pathstring2.3 + 2.4—nofile path (PEM).
key_sourcestring2.3 + 2.4—no“agent” | “tf-output:<name>”.

TestCfg

keytypelinedefaultrequireddescription
throughputThroughputCfg2.3 + 2.4—noThroughput configures the iperf3 bandwidth probe.
connectivityConnectivityCfg2.3 + 2.4—noConnectivity configures the reachability probes — the hosts roksbnkctl test tries to reach, and from where.
dnsDNSCfg2.3 + 2.4—noDNS configures the name-resolution probes, including which resolvers to ask.

ThroughputCfg

keytypelinedefaultrequireddescription
imagestring2.3 + 2.4networkstatic/iperf3:latestnodefault: networkstatic/iperf3:latest.
durationint2.3 + 2.430noseconds; default 30.
streamsint2.3 + 2.48noparallel; default 8.
default_modestring2.3 + 2.4north-southnonorth-south | east-west.

Terraform variable reference

Auto-generated by go run ./tools/refgen/tfvars-md > book/src/29-terraform-variable-reference.md. Re-run on every terraform/variables.tf change.

Every variable below is settable via terraform.tfvars, -var, -var-file, or (for sensitive values) the corresponding TF_VAR_<name> environment variable. Variables with _required_ defaults must be set explicitly. See Chapter 13 for how roksbnkctl threads these through the workspace config.

Root module variables

Source: terraform/variables.tf

VariableTypeDefaultDescriptionSensitive
ibmcloud_api_keystringrequiredIBM Cloud API keyyes
ibmcloud_cluster_regionstring"ca-tor"IBM Cloud region for all cluster resourcesno
ibmcloud_resource_groupstring"default"IBM Cloud resource group nameno
create_roks_clusterbooltrueCreate a new ROKS cluster. When false, supply roks_cluster_id_or_name instead.no
cluster_public_gatewaybooltrueAttach a public gateway to each cluster subnet for worker Internet egress. true (default) keeps current behavior; false builds a private, disconnected cluster with no egress (operator must supply private connectivity — VPEs / private service endpoints — for image pulls and IBM Cloud services).no
roks_cluster_id_or_namestring""ID or name of an existing ROKS cluster — used when create_roks_cluster = falseno
create_roks_transit_gatewaybooltrueCreate Transit Gateway and VPC connectionsno
create_roks_registry_cos_instancebooltrueCreate Cloud Object Storage instance for the OpenShift image registryno
deploy_cert_managerbooltrueWhen true, the cert_manager module’s helm/null_resource bring-up runs. Forced false by writeBnkPhaseOverrideAt when cluster-outputs.json exists (cluster phase already deployed cert_manager; trial phase consumes it via outputs that resolve to null on the bnk-phase apply, and downstream gates fall back to "direct-apply").no
roks_cluster_vpc_namestring"tf-cluster-vpc"Name of the cluster VPCno
openshift_cluster_namestring"tf-openshift-cluster"Name of the OpenShift clusterno
openshift_cluster_versionstring"4.21"OpenShift cluster version as a BARE major.minor prefix (e.g. “4.20”). Empty uses the latest available.no
roks_workers_per_zonenumber1Number of worker nodes per availability zoneno
roks_min_worker_vcpu_countnumber16Minimum vCPU count when auto-selecting the worker node flavorno
roks_min_worker_memory_gbnumber64Minimum memory in GB when auto-selecting the worker node flavorno
roks_cos_instance_namestring"tf-openshift-cos-instance"Name of the COS instance for the OpenShift image registryno
roks_transit_gateway_namestring"tf-tgw"Name of the Transit Gateway. Must reference an existing TGW when create_roks_transit_gateway = false and testing_create_tgw_jumphost = true.no
use_existing_cluster_vpcboolfalseReuse an existing cluster VPC instead of creating one. roksbnkctl sets this true in the second (bnk/testing) phase when cluster-outputs.json exists; the cluster phase leaves it false (create).no
existing_cluster_vpc_idstring""ID of the existing cluster VPC (used only when use_existing_cluster_vpc = true) — sourced from cluster-outputs.json vpc_id.no
install_cert_managerbooltrueInstall cert-manager. When false, cert_manager_namespace is passed directly to flo.no
cert_manager_namespacestring"cert-manager"Kubernetes namespace for cert-managerno
cert_manager_versionstring"v1.17.3"cert-manager Helm chart versionno
ibmcloud_cos_resource_groupstring""Resource group holding the supply-chain COS instance. Empty resolves the workspace’s own resource group, which is the pre-#295 behaviour. Set it when the supply chain lives centrally (typically default) and the workspace was placed elsewhere.no
ibmcloud_cos_bucket_regionstring"us-south"IBM Cloud region where the COS bucket is locatedno
ibmcloud_cos_instance_namestring"bnk-supply-chain"IBM Cloud COS instance nameno
ibmcloud_resources_cos_bucketstring"bnk-artifacts"IBM Cloud COS bucket containing FAR auth key and JWT filesno
deploy_bnkbooltrueDeploy BIG-IP Next for Kubernetes — creates flo, cne_instance, and license. When false all three modules are skipped.no
far_repo_urlstring"repo.f5.com"FAR repository URL for Docker and Helm imagesno
far_chart_repo_urlstring""Chart-pull host for the air-gap mirror (helm_release repository + manifest pull). Empty falls back to far_repo_url.no
far_image_repo_urlstring""Image-pull host for the air-gap mirror (image.repository + CNEInstance spec.registry.uri). Empty falls back to far_repo_url.no
use_registry_mirrorboolfalsePull charts + images from the registry mirror instead of FAR. The far-secret dockerconfig is dropped; how pods then authenticate depends on the mirror: an in-cluster/ICR mirror authorizes by RBAC and needs no pull secret, while an EXTERNAL mirror (Harbor, Artifactory) gets a mirror-secret dockerconfig built from registry_mirror_username/password. A private mirror therefore needs no anonymous/public project.no
registry_mirror_usernamestring""Basic-auth username for an EXTERNAL registry mirror (e.g. a Harbor robot/admin). Empty → the mirror is the in-cluster/ICR registry, which authenticates via the kube token / IAM key instead.no
registry_mirror_passwordstring""Basic-auth password/token for an external registry mirror. When set (with use_registry_mirror), chart and image pulls authenticate to the mirror with these credentials instead of the in-cluster kube token.yes
f5_bigip_k8s_manifest_versionstring"2.3.0-3.2598.3-0.0.170"Version of the f5-bigip-k8s-manifest chart (FLO and CIS versions are extracted from this)no
f5_cne_far_auth_filestring"f5-far-auth-key.tgz"FAR auth key filename in the COS bucket (.tgz)no
f5_cne_subscription_jwt_filestring"subscription.jwt"Subscription JWT filename in the COS bucket — used by flo and licenseno
use_cos_bucketbooltrueDownload the FAR auth tarball + subscription JWT from the orchestration COS. false = use the injected far_service_account_b64 / f5_cne_subscription_jwt content instead (local files).no
far_service_account_b64string""FAR _json_key_base64 service account (base64 of the .json), injected when use_cos_bucket = false. Empty on the COS path.yes
f5_cne_subscription_jwtstring""Subscription/license JWT token content, injected when use_cos_bucket = false. Empty on the COS path (downloaded from COS instead).yes
bnk_linestring"2.3"BNK release line driving per-release resource gating (2.3 or 2.4). Derived by roksbnkctl from bnk.manifest_version — set it by hand only for a standalone terraform run.no
flo_namespacestring"f5-bnk"Kubernetes namespace for the F5 Lifecycle Operatorno
flo_utils_namespacestring"f5-utils"Kubernetes namespace for F5 utility components — used by flo, cne_instance, and license. Set equal to flo_namespace to install into ONE namespace.no
bigip_usernamestring"admin"BIG-IP username for the CIS controllerno
bigip_passwordstring"admin"BIG-IP password for the CIS controlleryes
bigip_urlstring"192.168.1.245"BIG-IP URL for the CIS controllerno
flo_trusted_profile_idstring""IBM Cloud Trusted Profile ID created by flo — wired automatically from flo output; set here to overrideno
flo_trusted_profile_sa_namestring""Kubernetes service account name the CNE controller’s IBM Cloud Trusted Profile is linked to — i.e. which service account may ASSUME the profile and act on the VPC.no
flo_trusted_profile_roleslist(string)["Viewer", "Editor"]IAM roles granted to the CNE controller’s Trusted Profile, scoped to the cluster’s OWN VPC (serviceName=is, vpcId=<cluster vpc>).no
flo_cluster_issuer_namestring""Kubernetes ClusterIssuer name created by flo — wired automatically from flo output; set here to overrideno
cneinstance_network_attachmentslist(string)["ens3-ipvlan-l2", "macvlan-conf"]Network attachment names for cne_instance — wired automatically from flo output; set here to overrideno
cneinstance_deployment_sizestring""Deployment size for CNEInstance (Tiny, Small, Medium, Large). EMPTY takes the line default: Small on 2.3, Tiny on 2.4 (what the 2.4 install guide and F5’s reference cluster use). Tiny is what the BNK 2.4 install guide uses; it is passed through unvalidated, so a size a given manifest does not define is rejected by the operator, not here.no
cneinstance_gslb_datacenter_namestring""GSLB datacenter name for CNEInstance (optional)no
cneinstance_network_zoneslist(object({[]Per-zone subnet CIDRs + TMM self-IPs (empty = use install-guide defaults)no
cneinstance_vlan_prefixlen_externalnumber0External VLAN self-IP prefix length. 0 (default) → cneinstance_vlan_prefixlen, so a deployment that does not care keeps one knob and one value.no
cneinstance_vlan_prefixlen_internalnumber0Internal VLAN self-IP prefix length. 0 (default) → cneinstance_vlan_prefixlen. See the external variable for why the two can differ.no
cneinstance_vlan_prefixlennumber24TMM self-IP prefix length (spec.prefixlen_v4) for the external/internal F5SPKVlan CRsno
cneinstance_tmm_k8s_routesstring"172.17.0.0/18"Pod CIDR TMM routes to (advanced.tmm.env TMM_K8S_ROUTES). Default is the ROKS default pod subnet.no
license_modestring"connected"License operation mode (connected, disconnected, or f5licenseproxy)no
flp_license_server_urlstring""Base URL of the in-cluster F5 License Proxy service (FLP mode only; e.g. https://f5-license-proxy.<ns>.svc.cluster.local:8443)no
license_server_root_castring""PEM of the FLP root CA, written into the licenseserver-rootca Secret so CWC trusts the proxy (FLP mode only)no
testing_create_tgw_jumphostbooltrueCreate a jumphost in a client VPC connected to the cluster via the Transit Gatewayno
testing_create_cluster_jumphostsboolfalseCreate one jumphost per availability zone directly inside the cluster VPCno
testing_ssh_key_namestring""Name of the IBM Cloud SSH key to inject into all jumphostsno
testing_jumphost_profilestring""Instance profile for all jumphosts (leave empty to auto-select based on min_vcpu_count and min_memory_gb)no
testing_jumphost_total_volume_bandwidthnumber1000Volume bandwidth (Mbps) pinned on each jumphost. IBM computes this from the profile at creation and terraform carries the stored value into a later profile change, so an unmanaged value taken from an oversized profile makes a downsize impossible (#316). 1000 is ample for jumphosts, which run curl/iperf3 and no disk-heavy work, and is below the ceiling of every profile the auto-select can pick.no
testing_min_vcpu_countnumber4Minimum vCPU count when auto-selecting the jumphost instance profileno
testing_min_memory_gbnumber8Minimum memory in GB when auto-selecting the jumphost instance profileno
testing_create_client_vpcboolfalseCreate a new client VPC for the TGW jumphost. When false, testing_client_vpc_name must reference an existing VPC.no
testing_client_vpc_namestring"tf-testing-vpc"Name of the client VPC — created when testing_create_client_vpc = true, or looked up when falseno
testing_client_vpc_regionstring"ca-tor"IBM Cloud region for the client VPC and TGW jumphostno
testing_tgw_jumphost_namestring"tf-testing-jumphost-tgw"Name of the TGW-connected jumphost instanceno
testing_cluster_jumphost_name_prefixstring"tf-testing-jumphost-cluster"Name prefix for cluster jumphosts — zone name is appended (<prefix>-<zone>)no
kubeconfig_dirstring"/work/.bnk/scratch/kubeconfig"Parent directory where ibm_container_cluster_config writes admin kubeconfigs. Each submodule appends its name as a subdir. Default is the bnk runner image’s /work mount; override for direct-on-host runs.no
scratch_dirstring"/work/.bnk/scratch"Persistent scratch directory for FLO’s FAR/manifest cross-apply artifacts. Default is the bnk runner image’s /work mount; override for direct-on-host runs.no
deploy_gatewayboolfalseMaster toggle for the Gateway phase. Off in every other phase’s override; on only for gateway up.no
deploy_flpboolfalseMaster toggle for the F5 License Proxy phase. Off in every other phase’s override; on only for flp up.no
flp_namespacestring"f5-license-proxy"Namespace the F5 License Proxy is installed into (FLP phase).no
flp_chart_versionstring""Pin for the f5-license-proxy chart version (empty → registry latest).no
flp_storage_classstring"ibmc-vpc-block-metro-10iops-tier"Dynamic StorageClass for the FLP’s PVCs (FLP phase). Default = ROKS VPC block.no
gateway_app_namespacestring"f5-app"Application namespace the Gateway + HTTPRoute serve (created by the gateway module)no
gateway_class_namestring"gateway-class"GatewayClass name. Set it to run more than one BNK GatewayClass in a cluster — GatewayClass is cluster-scoped, so two installs sharing the name collide.no
gateway_route_exampleslist(string)[]Extra route kinds to create working examples of, alongside the default HTTPRoute. Valid on BNK 2.3: GRPCRoute, L4Route. Empty (default) leaves an existing deployment byte-identical. L4Route also adds a TCP listener to the Gateway, because an L4Route cannot attach to an HTTP one.no
gateway_l4_listener_portnumber8080Port for the TCP listener added when gateway_route_examples includes L4Routeno
gateway_controller_namestring""GatewayClass controllerName. Empty → derived as f5.com/<flo_namespace>-f5-cne-controller, which is the value the CNE controller answers to. Set it only to point the GatewayClass at a controller this deployment did not install; a wrong value fails silently (the GatewayClass is never Accepted and the apply still succeeds).no
gateway_backend_servicestring"nginx-service"HTTPRoute backend Service name in the app namespaceno
gateway_backend_portnumber80HTTPRoute backend Service portno
gateway_egress_modestring"snatpool"Egress SNAT strategy: snatpool (default), automap, or bothno
gateway_client_subnet_locallist(string)[]Local-VSI client subnet CIDRs the static routes reach (cluster-VPC clients; one route per entry × zone). Empty = no local client routes. gateway up auto-derives these from the cluster jumphost subnets when unset (PRD 12).no
gateway_client_subnet_remotelist(string)[]Remote-VSI client subnet CIDRs the static routes reach (client-VPC clients over the TGW; one route per entry × zone). Empty = no remote client routes.no
gateway_vxlan_portnumber6789Egress VXLAN UDP port (also opened on the cluster security group)no
flp_node_port_accessboolfalseExpose the F5 License Proxy outside its own cluster (NodePort + worker-node-IP cert SANs), so a BNK install in a different cluster can license through it.no
flp_node_port_source_cidrslist(string)[]With flp_node_port_access: open the proxy’s NodePort on the worker security group to these CIDRs. A LIST — a multi-zone VPC has one address prefix per zone, and a consuming pod in an unlisted zone is silently dropped.no
roksbnkctl_binarystring""Absolute path to the roksbnkctl binary. helm invokes it as the f5-license-proxy chart’s post-renderer (roksbnkctl flp postrender), so the FLP install needs no interpreter on the host. roksbnkctl sets this automatically via TF_VAR_roksbnkctl_binary; empty falls back to roksbnkctl on PATH.no
deploy_flp_vsiboolfalseDeploy the F5 License Proxy as a standalone VSI (podman pod, no k8s) instead of the helm chart. Set true only by the FLP phase in mode: vsi.no
flp_status_imagestring""Optional flp-status web UI image for the standalone FLP VSI (mirror/public ref). Empty = no status UI.no
flp_status_registry_hoststring""Registry host:port whose CA to trust so the FLP VSI can pull flp_status_image (e.g. Harbor’s <ip>).no
flp_status_registry_ca_b64string""Base64 CA cert for flp_status_registry_host.no
flp_vsi_ssh_keystring""Existing IBM Cloud VPC SSH key name to attach to the standalone FLP VSI (operator access). Empty = no key.no
flp_vsi_profilestring"bx2-4x16"VSI instance profile for the FLP (>= 4 vCPU / 8 GB).no
flp_vsi_zonestring""Zone for the FLP VSI. Empty → <region>-1.no
flp_vsi_boot_size_gbnumber100Boot volume size (GB) for the FLP VSI (>= 80).no
flp_vsi_floating_ipbooltrueAttach an operator floating IP to the FLP VSI for remote management (flp status + web UI + 8443 from another machine). Not the CWC endpoint. Reachability still gated by flp_vsi_allowed_cidrs. Default true.no
flp_vsi_management_allowed_cidrslist(string)[]Source CIDRs for the FLP VSI’s :80 flp-status web UI (read-only). Empty → 0.0.0.0/0 (open).no
flp_vsi_licensing_allowed_cidrslist(string)[]Source CIDRs for the FLP VSI’s :8443 proxy (+ :22 SSH). Empty → RFC-1918 private ranges.no
flp_vsi_allowed_cidrslist(string)[]DEPRECATED — legacy single list; seeds both management + licensing when set. Prefer the two per-plane variables.no
flp_prod_jwks_b64string""Optional override: base64 of F5’s public prod_jwks.txt. Empty → the flp_vsi module extracts it from the f5-license-proxy chart.no
flp_forward_proxy_hoststring""—no
flp_forward_proxy_portnumber0—no
flp_forward_proxy_protocolstring"http"—no
deploy_tgw_connectionboolfalseAttach the cluster’s VPC to an existing Transit Gateway. On only for the tgw phase; a no-op everywhere else.no
tgw_connection_targetstring""Existing Transit Gateway to attach the cluster VPC to, by NAME or ID. Multiple clusters passing the same value share one gateway.no
tgw_connection_namestring""Name for this cluster’s connection on the gateway (unique per gateway; prefix-derived so shared-gateway clusters don’t collide).no
cluster_absentboolfalseTrue only in the standalone FLP-VSI phase: no ROKS cluster exists or will be adopted, so all cluster data-source lookups + kube providers across modules are skipped (count=0).no
cluster_vpc_cidrstring""CIDR block the cluster VPC’s per-zone address prefixes are carved from (e.g. “10.241.0.0/16” → 10.241.0.0/18, 10.241.64.0/18, 10.241.128.0/18).no
use_existing_cluster_subnetsboolfalsePlace the cluster in subnets that already exist instead of creating them. Requires use_existing_cluster_vpc — a subnet cannot be adopted independently of its VPC.no
existing_cluster_subnet_idslist(string)[]Subnet ids to place the cluster in, one per zone, in zone order. Used only when use_existing_cluster_subnets = true. Their zones are read from the subnets themselves.no
flp_vsi_create_vpcboolfalseBuild the F5 License Proxy its own VPC instead of placing it in an existing one.no
flp_vsi_name_prefixstring""Prefix for the standalone FLP VSI’s resource names (e.g. bnk-ci → bnk-ci-flp-vsi). Empty (the default) keeps the legacy unprefixed names so an existing proxy is NOT replaced on upgrade. Set it to run more than one FLP in an account, and to bring the resources into roksbnkctl cleanup’s <prefix>-* sweep.no
flp_vsi_vpc_namestring""Name for the VPC created when flp_vsi_create_vpc = true.no
flp_vsi_subnet_cidrstring"10.250.0.0/24"Address prefix for the VPC created when flp_vsi_create_vpc = true.no
cluster_network_modestring"single-nic"How the cluster’s worker nodes are attached: single-nic (default) or multi-nic.no
testing_jumphost_allowed_cidrslist(string)[]Source CIDRs allowed to SSH (:22) to the testing jumphosts. Empty → 0.0.0.0/0 (open; access is key-only). Narrow to the operator’s public /32 on a shared account.no
testing_client_vpc_inbound_cidrslist(string)[]Source CIDRs allowed inbound to the testing client VPC’s default security group. Empty → the RFC-1918 private ranges (in-fabric test traffic arrives over the Transit Gateway).no
cluster_http_allowed_cidrslist(string)[]Source CIDRs allowed to reach :80 on the cluster security group. Empty → 0.0.0.0/0 (the ingress/ALB path is meant to be publicly reachable).no
cluster_vpc_default_sg_inbound_cidrslist(string)[]Source CIDRs allowed inbound (all protocols/ports) to the cluster VPC’s default security group. Empty → 0.0.0.0/0, the historical behaviour. Narrow to your private ranges unless a workload in this VPC needs a public source.no
cneinstance_tmm_resourcesmap(map(string)){}Overrides the TMM pod’s resource requests/limits, rendered as the CNEInstance’s advanced.tmm.resources (#203).no
cneinstance_advanced_envmap(map(string)){}Per-component environment passthrough for the BNK 2.4 CNEInstance’s advanced.<component>.env[] lists (#175).no
cneinstance_tmm_replicasnumber3Number of f5-tmm data-plane replicas (2.4). Reference: 3.no
cneinstance_watch_namespaceslist(string)["All"]Namespaces the CNE controller watches (2.4). Reference: ["All"].no
cneinstance_tmm_anti_affinitybooltrueRequire f5-tmm pods onto DIFFERENT NODES (2.4).no
cneinstance_tmm_zone_spreadbooltrueSpread f5-tmm pods across zones with maxSkew 1, DoNotSchedule (2.4). Reference: on.no
cneinstance_tmm_rolling_updatebooltruePin TMM’s rolling update to maxSurge 0 / maxUnavailable 1 (2.4).no
cneinstance_external_bigipboolfalseEnable the external BIG-IP controller (2.4). Reference: true.no
cneinstance_external_bigip_login_secretstring"f5-bigip-ctlr-login"Secret holding external BIG-IP credentials (2.4). Reference: f5-bigip-ctlr-login.no
cneinstance_cluster_identifierstring""CLUSTER_IDENTIFIER passed to the external BIG-IP controller (2.4). Empty derives from the cluster name.no
cneinstance_gateway_api_versionstring"1.5.0"GATEWAY_API_VERSION for the CNE controller (2.4). Reference: 1.5.0.no
gateway_api_bundle_urlstring""Where the Gateway API standard-install bundle is FETCHED from when the workspace has no registry mirror recorded (2.4 + gateway_api_mtls). Empty derives the upstream release asset for cneinstance_gateway_api_version.no
cneinstance_demo_modestring""advanced.demoMode.enabled.no
cneinstance_tmm_pod_labelstring"f5-tmm"Value of the app label the placement rules select f5-tmm pods by (2.4). Reference: f5-tmm.no
cneinstance_tmm_anti_affinity_topology_keystring"kubernetes.io/hostname"Node label the TMM anti-affinity rule spreads across (2.4).no
cneinstance_tmm_zone_topology_keystring"topology.kubernetes.io/zone"Node label the TMM zone spread uses (2.4). The IBM ROKS zone label. Reference: topology.kubernetes.io/zone.no
cneinstance_tmm_zone_max_skewnumber1maxSkew for the TMM zone topology-spread constraint (2.4). Reference: 1.no
cneinstance_tmm_zone_when_unsatisfiablestring"DoNotSchedule"whenUnsatisfiable for the TMM zone spread (2.4): DoNotSchedule or ScheduleAnyway. Reference: DoNotSchedule.no
cneinstance_tcp_settingsmap(string){}F5BigTcpSetting field overrides, as a flat map of field name to value.no
cneinstance_tcp_settings_namestring"sys-default-tcp"Name of the F5BigTcpSetting CR to write. Reference: sys-default-tcp.no
cneinstance_whole_cluster_overridestring""spec.wholeCluster, as a tri-state.no
roks_worker_flavorstring""Exact worker-node flavor, e.g. “cx3d.8x20”. Empty auto-selects.no
cneinstance_hugepagesboolfalseAllocate hugepages on the worker pool via the OpenShift Node Tuning Operator.no
cneinstance_hugepages_sizestring"2M"Hugepage size, e.g. 2M or 1G. TMM requests hugepages-2Mi, so 2M is the matching size.no
cneinstance_hugepages_countnumber2048Hugepages PER NODE. 2048 x 2M = 4Gi, which is what deploymentSize Small was observed to request.no
cneinstance_hugepages_node_rolestring"worker"machineconfiguration.openshift.io/role the Tuned profile applies to.no
cneinstance_hugepages_profile_namestring"bnk-hugepages"Name of the Tuned profile and CR.no
cneinstance_storage_class_namestring""StorageClass for the CNEInstance’s persistent volumes, TMM’s included. Empty leaves the CR’s own default, which resolves to the cluster default class.no

Module: cert_manager

Source: terraform/modules/cert_manager/variables.tf

VariableTypeDefaultDescriptionSensitive
ibmcloud_api_keystringrequiredIBM Cloud API Keyyes
ibmcloud_cluster_regionstring"ca-tor"IBM Cloud region where the cluster residesno
ibmcloud_resource_groupstring"default"IBM Cloud Resource Group name (leave empty to use account default)no
roks_cluster_name_or_idstringrequiredName or ID of the existing OpenShift ROKS cluster to deploy BNK ontono
cert_manager_namespacestring"cert-manager"Kubernetes namespace for cert-managerno
cert_manager_versionstring"v1.17.3"cert-manager Helm chart versionno
cert_manager_image_repositorystring""Override the cert-manager controller image repository (air-gap mirror image host). Empty leaves the chart default.no
create_roks_clusterboolfalseWhen true, cluster is being created by roks_cluster — skip plan-time cluster credential fetchno
deploy_cert_managerbooltrueWhen true, manage the cert_manager helm/null_resource bring-up. Set false in the bnk-phase override when cluster-outputs.json exists — cluster phase already provisioned cert_manager and the second phase must NOT re-manage it (would attempt kubectl delete namespace cert-manager on a subsequent bnk down).no
roks_cluster_dependency_idstringnullroks_cluster sentinel ID — when set, defers runtime_config fetch to apply time after roks_cluster completesno
kubeconfig_dirstring"/work/.bnk/scratch/kubeconfig/cert_manager"Persistent, writable dir for ibm_container_cluster_config kubeconfig downloads. Defaults to a host-bind-mounted, module-scoped path under .bnk/scratch.no
registry_mirror_usernamestring""Basic-auth username for an external registry mirror (a private Harbor/Artifactory). Empty → no mirror pull secret is created.no
registry_mirror_passwordstring""Basic-auth password for an external registry mirror. When set with image_repository, cert-manager’s pods pull with a dockerconfig secret built from it instead of anonymously.yes
cluster_absentboolfalseTrue in the standalone FLP-VSI phase: no ROKS cluster exists, so all cluster data-source lookups + kube providers are skipped (count=0).no

Module: cne_instance

Source: terraform/modules/cne_instance/variables.tf

VariableTypeDefaultDescriptionSensitive
bnk_linestring"2.3"BNK release line driving per-release resource gating (“2.3” or “2.4”).no
ibmcloud_api_keystringrequiredIBM Cloud API Keyyes
ibmcloud_cluster_regionstring"ca-tor"IBM Cloud region where the cluster residesno
ibmcloud_resource_groupstring"default"IBM Cloud Resource Group name (leave empty to use account default)no
roks_cluster_name_or_idstringrequiredName or ID of the existing OpenShift ROKS cluster to deploy BNK ontono
far_repo_urlstring"repo.f5.com"FAR Repository URL for Docker and Helm registryno
far_image_repo_urlstring""Image-pull host for the mirror (CNEInstance spec.registry.uri). Empty falls back to far_repo_url.no
use_registry_mirrorboolfalsePull charts + images from the registry mirror instead of FAR. The far-secret dockerconfig is dropped; how pods then authenticate depends on the mirror: an in-cluster/ICR mirror authorizes by RBAC and needs no pull secret, while an EXTERNAL mirror (Harbor, Artifactory) gets a mirror-secret dockerconfig built from registry_mirror_username/password. A private mirror therefore needs no anonymous/public project.no
flo_namespacestring"f5-bnk"Namespace for F5 Lifecycle Operatorno
flo_utils_namespacestring"f5-utils"Namespace for F5 utility componentsno
f5_bigip_k8s_manifest_versionstring"2.3.0-3.2598.3-0.0.170"Version of f5-bigip-k8s-manifest chart - used by flo, cneinstance modulesno
flo_trusted_profile_sa_namestring""The CNE controller service account; must match what the flo module linked.no
flo_trusted_profile_idstring""IBM IAM Trusted Profile ID for provisioning VPC routesno
flo_cluster_issuer_namestring""mTLS certificate issuer nameno
cneinstance_deployment_sizestring""Deployment size for CNEInstance (Tiny, Small, Medium, Large). EMPTY takes the line default: Small on 2.3, Tiny on 2.4 (what the 2.4 install guide and F5’s reference cluster use). Tiny is what the BNK 2.4 install guide uses.no
cneinstance_gslb_datacenter_namestring""GSLB datacenter name for CNEInstance (optional)no
cneinstance_network_attachmentslist(string)["ens3-ipvlan-l2", "macvlan-conf"]The Multus Network Attachment Definitions for the CNEInstance TMM deploymentsno
cneinstance_network_zoneslist(object({[]Per-zone subnet CIDRs + TMM self-IPs (empty = use the install-guide defaults)no
cneinstance_vlan_prefixlen_externalnumber0External VLAN self-IP prefix length; 0 inherits cneinstance_vlan_prefixlen.no
cneinstance_vlan_prefixlen_internalnumber0Internal VLAN self-IP prefix length; 0 inherits cneinstance_vlan_prefixlen.no
cneinstance_vlan_prefixlennumber24TMM self-IP prefix length (spec.prefixlen_v4) for the external/internal F5SPKVlan CRsno
cneinstance_tmm_k8s_routesstring"172.17.0.0/18"Pod CIDR TMM routes to (advanced.tmm.env TMM_K8S_ROUTES). Default is the ROKS default pod subnet.no
create_roks_clusterboolfalseWhen true, cluster is being created by roks_cluster — skip plan-time cluster credential fetchno
roks_cluster_dependency_idstringnullroks_cluster sentinel ID — when set, defers runtime_config fetch to apply time after roks_cluster completesno
flo_dependency_idstringnullflo_ready sentinel ID — pass module.flo.flo_ready_id to defer cne_instance until flo completes and CRDs are registeredno
deploy_bnkbooltrueDeploy BIG-IP Next for Kubernetes — when false the inner cneinstance module is disabled and no CNEInstance resources are createdno
kubeconfig_dirstring"/work/.bnk/scratch/kubeconfig/cne_instance"Persistent, writable dir for ibm_container_cluster_config kubeconfig downloads. Defaults to a host-bind-mounted, module-scoped path under .bnk/scratch.no
registry_mirror_passwordstring""Basic-auth password for an external registry mirror. When set with use_registry_mirror, the CNEInstance references the mirror-secret pull secret instead of pulling anonymously.yes
roksbnkctl_binarystring""Absolute path to the roksbnkctl binary; the CNE-instance phase invokes roksbnkctl tfx <verb> in place of host curl (no interpreter, so cmd.exe execs it on Windows). Empty falls back to roksbnkctl on PATH.no
cluster_absentboolfalseTrue in the standalone FLP-VSI phase: no ROKS cluster exists, so all cluster data-source lookups + kube providers are skipped (count=0).no
cneinstance_tmm_resourcesmap(map(string)){}TMM resource requests/limits override; rendered as advanced.tmm.resources (#203).no
cneinstance_advanced_envmap(map(string)){}Per-component advanced.<component>.env passthrough (#175). See the root variable of the same name.no
cneinstance_tmm_replicasnumber3Number of f5-tmm data-plane replicas (2.4). Reference: 3.no
cneinstance_watch_namespaceslist(string)["All"]Namespaces the CNE controller watches (2.4). Reference: ["All"].no
cneinstance_tmm_anti_affinitybooltrueRequire f5-tmm pods onto DIFFERENT NODES (2.4).no
cneinstance_tmm_zone_spreadbooltrueSpread f5-tmm pods across zones with maxSkew 1, DoNotSchedule (2.4). Reference: on.no
cneinstance_tmm_rolling_updatebooltruePin TMM’s rolling update to maxSurge 0 / maxUnavailable 1 (2.4).no
cneinstance_external_bigipboolfalseEnable the external BIG-IP controller (2.4). Reference: true.no
cneinstance_external_bigip_login_secretstring"f5-bigip-ctlr-login"Secret holding external BIG-IP credentials (2.4). Reference: f5-bigip-ctlr-login.no
cneinstance_cluster_identifierstring""CLUSTER_IDENTIFIER passed to the external BIG-IP controller (2.4). Empty derives from the cluster name.no
cneinstance_gateway_api_versionstring"1.5.0"GATEWAY_API_VERSION for the CNE controller (2.4). Reference: 1.5.0.no
cneinstance_demo_modestring""advanced.demoMode.enabled.no
cneinstance_tmm_pod_labelstring"f5-tmm"Value of the app label the placement rules select f5-tmm pods by (2.4). Reference: f5-tmm.no
cneinstance_tmm_anti_affinity_topology_keystring"kubernetes.io/hostname"Node label the TMM anti-affinity rule spreads across (2.4).no
cneinstance_tmm_zone_topology_keystring"topology.kubernetes.io/zone"Node label the TMM zone spread uses (2.4). The IBM ROKS zone label. Reference: topology.kubernetes.io/zone.no
cneinstance_tmm_zone_max_skewnumber1maxSkew for the TMM zone topology-spread constraint (2.4). Reference: 1.no
cneinstance_tmm_zone_when_unsatisfiablestring"DoNotSchedule"whenUnsatisfiable for the TMM zone spread (2.4): DoNotSchedule or ScheduleAnyway. Reference: DoNotSchedule.no
cneinstance_tcp_settingsmap(string){}F5BigTcpSetting field overrides, as a flat map of field name to value.no
cneinstance_tcp_settings_namestring"sys-default-tcp"Name of the F5BigTcpSetting CR to write. Reference: sys-default-tcp.no
cneinstance_whole_cluster_overridestring""spec.wholeCluster, as a tri-state.no
cneinstance_hugepagesboolfalseAllocate hugepages on the worker pool via the OpenShift Node Tuning Operator.no
cneinstance_hugepages_sizestring"2M"Hugepage size, e.g. 2M or 1G. TMM requests hugepages-2Mi, so 2M is the matching size.no
cneinstance_hugepages_countnumber2048Hugepages PER NODE. 2048 x 2M = 4Gi, which is what deploymentSize Small was observed to request.no
cneinstance_hugepages_node_rolestring"worker"machineconfiguration.openshift.io/role the Tuned profile applies to.no
cneinstance_hugepages_profile_namestring"bnk-hugepages"Name of the Tuned profile and CR.no
cneinstance_storage_class_namestring""StorageClass for the CNEInstance’s persistent volumes. See the leaf module variable of the same name (#189).no

Module: flo

Source: terraform/modules/flo/variables.tf

VariableTypeDefaultDescriptionSensitive
bnk_linestring"2.3"BNK release line driving per-release resource gating (“2.3” or “2.4”).no
ibmcloud_api_keystringrequiredIBM Cloud API Keyyes
ibmcloud_cluster_regionstring"ca-tor"IBM Cloud region where the cluster residesno
ibmcloud_resource_groupstring"default"IBM Cloud Resource Group name (leave empty to use account default)no
roks_cluster_name_or_idstringrequiredName or ID of the existing OpenShift ROKS cluster to deploy BNK ontono
far_repo_urlstring"repo.f5.com"FAR Repository URL for Docker and Helm registryno
far_chart_repo_urlstring""Chart-pull host for the mirror (helm_release repository + manifest pull). Empty falls back to far_repo_url.no
far_image_repo_urlstring""Image-pull host for the mirror (image.repository). Empty falls back to far_repo_url.no
use_registry_mirrorboolfalsePull charts + images from the registry mirror instead of FAR. The far-secret dockerconfig is dropped; how pods then authenticate depends on the mirror: an in-cluster/ICR mirror authorizes by RBAC and needs no pull secret, while an EXTERNAL mirror (Harbor, Artifactory) gets a mirror-secret dockerconfig built from registry_mirror_username/password. A private mirror therefore needs no anonymous/public project.no
registry_mirror_usernamestring""Basic-auth username for an external registry mirror (Harbor). Empty → in-cluster/ICR mirror.no
registry_mirror_passwordstring""Basic-auth password/token for an external registry mirror; chart pulls authenticate with it when set.yes
f5_bigip_k8s_manifest_versionstring"2.3.0-3.2598.3-0.0.170"Version of the f5-bigip-k8s-manifest chart (FLO/CIS versions are extracted from this)no
use_cos_bucketbooltrueFetch FAR auth key and JWT from IBM Cloud Object Storage instead of local variablesno
far_service_account_b64string""FAR _json_key_base64 service account (base64 of the .json), injected when use_cos_bucket = falseyes
ibmcloud_cos_bucket_regionstring"us-south"IBM Cloud region where the COS bucket is locatedno
ibmcloud_cos_instance_namestring"bnk-supply-chain"IBM Cloud COS instance nameno
ibmcloud_resources_cos_bucketstring"bnk-artifacts"IBM Cloud COS bucket containing the FAR auth key and JWT filesno
f5_cne_far_auth_filestring"f5-far-auth-key.tgz"FAR auth key filename in the COS bucket (.tgz)no
flo_trusted_profile_sa_namestring""Service account the CNE controller Trusted Profile is linked to.no
flo_trusted_profile_roleslist(string)["Viewer", "Editor"]IAM roles for the CNE controller Trusted Profile, scoped to the cluster VPC.no
flo_namespacestring"f5-bnk"Namespace for F5 Lifecycle Operatorno
flo_utils_namespacestring"f5-utils"Namespace for F5 utility componentsno
cert_manager_namespacestring"cert-manager"Kubernetes namespace for cert-manager - used by cert-manager, flo modulesno
bigip_usernamestring"admin"BIG-IP username for CIS controller loginno
bigip_passwordstring"admin"BIG-IP password for CIS controller loginyes
bigip_urlstring"https://192.168.1.245"BIG-IP URL for CIS controller loginno
create_roks_clusterboolfalseWhen true, cluster is being created by roks_cluster — skip plan-time cluster credential fetchno
roks_cluster_dependency_idstringnullroks_cluster sentinel ID — when set, defers runtime_config fetch to apply time after roks_cluster completesno
cert_manager_dependency_idstringnullcert_manager ready sentinel ID — when set, blocks flo inner module until cert-manager CRDs are availableno
deploy_bnkbooltrueDeploy BIG-IP Next for Kubernetes — when false the inner flo module is disabled and no FLO resources are createdno
kubeconfig_dirstring"/work/.bnk/scratch/kubeconfig/flo"Persistent, writable dir for ibm_container_cluster_config kubeconfig downloads. Defaults to a host-bind-mounted, module-scoped path under .bnk/scratch.no
scratch_dirstring"/work/.bnk/scratch"Persistent scratch directory for FAR/manifest cross-apply artifacts. Default is the bnk runner image’s /work mount.no
roksbnkctl_binarystring""Absolute path to the roksbnkctl binary; the FLO phase invokes roksbnkctl tfx <verb> in place of host curl/tar (no interpreter, so cmd.exe execs it on Windows). Empty falls back to roksbnkctl on PATH.no
cluster_absentboolfalseTrue in the standalone FLP-VSI phase: no ROKS cluster exists, so all cluster data-source lookups + kube providers are skipped (count=0).no
ibmcloud_cos_resource_groupstring""Resource group holding the supply-chain COS instance. Empty resolves the workspace’s own resource group (#295).no

Module: flp

Source: terraform/modules/flp/variables.tf

VariableTypeDefaultDescriptionSensitive
deploy_flpboolfalseDeploy the F5 License Proxy. Off in every other phase’s override; on only for flp up. When false the module is a complete no-op.no
create_roks_clusterboolfalseWhen true the cluster is being created in this same apply (FLP never runs then); gates the plan-time cluster-config fetch.no
roks_cluster_name_or_idstring""Name or id of the existing ROKS cluster to deploy the FLP into.no
roks_cluster_dependency_idanynullDependency handle from the cluster phase, deferring apply until the cluster is ready.no
kubeconfig_dirstring""Directory the ibm_container_cluster_config data source writes the kubeconfig into.no
ibmcloud_api_keystring""IBM Cloud API key (COS/IAM auth + provider config).yes
ibmcloud_cluster_regionstring""Region of the ROKS cluster (ibm provider).no
ibmcloud_resource_groupstring""Resource group containing the COS instance (empty = default group).no
ibmcloud_cos_instance_namestring""COS service instance holding the FAR auth tarball + subscription JWT.no
ibmcloud_resources_cos_bucketstring""COS bucket holding the FAR auth tarball + subscription JWT.no
ibmcloud_cos_bucket_regionstring""Region of the COS bucket.no
f5_cne_far_auth_filestring"f5-far-auth-key.tgz"FAR auth tarball object key in the COS bucket (the _json_key_base64 SA lives inside).no
f5_cne_subscription_jwt_filestring"subscription.jwt"Subscription JWT object key in the COS bucket — seeds flp-jwt-secret.no
scratch_dirstring"/tmp/roksbnkctl-flp"Working directory for the FAR-auth download/extract.no
far_repo_urlstring"repo.f5.com"FAR registry host (fallback when no mirror).no
far_chart_repo_urlstring""Mirror host for chart pulls (empty → coalesces to far_repo_url).no
far_image_repo_urlstring""Mirror host for image pulls (empty → coalesces to far_repo_url).no
use_registry_mirrorboolfalsePull charts + images from the registry mirror instead of FAR. The far-secret dockerconfig is dropped; how pods then authenticate depends on the mirror: an in-cluster/ICR mirror authorizes by RBAC and needs no pull secret, while an EXTERNAL mirror (Harbor, Artifactory) gets a mirror-secret dockerconfig built from registry_mirror_username/password. A private mirror therefore needs no anonymous/public project.no
registry_mirror_usernamestring""Basic-auth username for an EXTERNAL registry mirror (e.g. a Harbor robot/admin). Empty → in-cluster/ICR mirror (kube-token/IAM auth).no
registry_mirror_passwordstring""Basic-auth password/token for an external registry mirror. When set (with use_registry_mirror), chart + image pulls authenticate with it instead of the kube token.yes
flp_namespacestring"f5-license-proxy"Namespace to install the F5 License Proxy into.no
flp_chart_versionstring""Pin the f5-license-proxy chart version. Empty (the default) → resolved from the BNK manifest, which lists charts/f5-license-proxy for the release — same as the FLO and CIS charts. Set this only to override the manifest.no
f5_bigip_k8s_manifest_versionstring""BNK manifest version. The f5-license-proxy chart version is read out of this manifest when flp_chart_version is empty.no
flp_node_port_accessboolfalseExpose the proxy OUTSIDE its own cluster so a BNK install in a different cluster can license through it. The chart’s Service is already type NodePort (30001), but it hardcodes externalTrafficPolicy: Local — with one replica only the node running the pod answers — so this flips it to Cluster and adds the worker node IPs to the proxy’s server certificate (without them the remote CWC rejects the TLS handshake).no
flp_node_port_source_cidrslist(string)[]With flp_node_port_access: open the proxy’s NodePort on the cluster’s worker security group to these CIDRs (the consuming cluster’s subnets). A LIST, because a multi-zone VPC carries one address prefix per zone — allowing only one means a consuming pod scheduled in another zone is silently dropped at the security group. Empty leaves the security group untouched.no
flp_storage_classstring"ibmc-vpc-block-metro-10iops-tier"Dynamic StorageClass for the FLP’s PVCs. The chart ships hostPath PVs (incompatible with ROKS multi-node/non-root); a post-renderer drops them and repoints the PVCs here, so the CSI driver provisions block volumes chowned to fsGroup. Default is the ROKS VPC block default.no
roksbnkctl_binarystring""Absolute path to the roksbnkctl binary, which helm invokes as the f5-license-proxy chart’s POST-RENDERER (roksbnkctl flp postrender). roksbnkctl sets this to its own path automatically via TF_VAR_roksbnkctl_binary; empty falls back to roksbnkctl on PATH for a direct terraform apply. Replaces a generated python script, which made python3 an undeclared runtime dependency of the FLP phase — absent in the tools-runner container.no
ibmcloud_cos_resource_groupstring""Resource group holding the supply-chain COS instance. Empty resolves the workspace’s own resource group (#295).no

Module: flp_vsi

Source: terraform/modules/flp_vsi/variables.tf

VariableTypeDefaultDescriptionSensitive
deploy_flp_vsiboolfalseMaster toggle — when false the whole module is a no-op (count=0). Set true only by the FLP phase in mode: vsi.no
ibmcloud_api_keystringrequiredIBM Cloud API key (provider + COS/IAM REST auth).yes
ibmcloud_cluster_regionstringrequiredRegion of the ROKS cluster / where the FLP VSI is provisioned.no
ibmcloud_resource_groupstring""Resource group for the VSI + network (empty = account default).no
flp_vsi_profilestring"bx2-4x16"VSI instance profile (>= 4 vCPU / 8 GB).no
flp_vsi_zonestring""Zone for the VSI (e.g. us-south-1). Empty → <region>-1.no
flp_vsi_boot_size_gbnumber100Boot volume size in GB (>= 80).no
flp_vsi_floating_ipbooltrueAttach an operator floating IP to the FLP VSI for remote management — running roksbnkctl flp status and reaching the :80 web UI + :8443 proxy from another machine. NOT the CWC endpoint (the cluster always reaches the proxy privately). The floating IP is added to the leaf-cert SAN; reachability is still gated by flp_vsi_allowed_cidrs. Default true.no
flp_vsi_management_allowed_cidrslist(string)[]Source CIDRs allowed to reach the :80 flp-status web UI (read-only status). Empty → 0.0.0.0/0 (open — the page carries no secrets).no
flp_vsi_licensing_allowed_cidrslist(string)[]Source CIDRs allowed to reach the :8443 licensing proxy (and :22 SSH). Empty → the RFC-1918 private ranges (the cluster reaches the proxy privately over the VPC / Transit Gateway).no
flp_vsi_allowed_cidrslist(string)[]DEPRECATED — legacy single list. When set, seeds BOTH flp_vsi_management_allowed_cidrs and flp_vsi_licensing_allowed_cidrs. Prefer the two per-plane variables. Empty → the per-plane defaults apply.no
existing_cluster_vpc_idstring""The cluster VPC id (from cluster-outputs.json) the FLP VSI joins so the CWC reaches it directly.no
flp_vsi_name_prefixstring""Prefix for the FLP VSI’s resource names, e.g. "bnk-ci" yields bnk-ci-flp-vsi. Empty (the default) keeps the legacy unprefixed names, so an existing proxy is not replaced on upgrade. Set it to run more than one standalone FLP in an account, and to make the resources visible to roksbnkctl cleanup.no
far_repo_urlstring"repo.f5.com"FAR repository host for the FLP chart pulls. Same value as the root far_repo_url.no
flp_image_registrystring""FAR image host prefix. Empty (the default) derives <far_repo_url>/images, so the image host follows the chart host instead of pinning repo.f5.com independently.no
f5_bigip_k8s_manifest_versionstringrequiredBNK manifest version — the f5-license-proxy chart/image tag is resolved from it (like the helm path) when flp_chart_version is empty.no
flp_chart_versionstring""Pin the f5-license-proxy chart/image tag. Empty → resolved from the BNK manifest.no
flp_vault_image_tagstring"2.0.0"Tag for the vault image.no
f5_cert_urlstring"https://product.apis.f5.com/ee/v1"—no
f5_entitlement_urlstring"https://product-s.apis.f5.com/ee/v1"—no
f5_initial_config_urlstring"https://product-s.apis.f5.com/ee/v1"—no
mode_of_operationstring"connected"—no
flp_forward_proxy_hoststring""—no
flp_forward_proxy_portnumber0—no
flp_forward_proxy_protocolstring"http"—no
ibmcloud_cos_instance_namestring"bnk-supply-chain"—no
ibmcloud_resources_cos_bucketstring"bnk-artifacts"—no
ibmcloud_cos_bucket_regionstring"us-south"—no
f5_cne_far_auth_filestring"f5-far-auth-key.tgz"—no
f5_cne_subscription_jwt_filestring"subscription.jwt"—no
scratch_dirstring"/tmp/flp-vsi-scratch"Working dir for the FAR-auth download/extract + chart pull.no
flp_prod_jwks_b64string""Base64 of F5’s public prod_jwks.txt (JWT signature verification). Supplied by the FLP phase, which extracts it from the f5-license-proxy chart.no
roksbnkctl_binarystring""Absolute path to the roksbnkctl binary; the FLP-VSI phase invokes roksbnkctl tfx <verb> in place of host curl/tar (no interpreter, so cmd.exe execs it on Windows). Empty falls back to roksbnkctl on PATH.no
use_cos_bucketbooltrueTrue: pull the FAR tarball + subscription JWT from COS. False (disconnected): use far_service_account_b64 + f5_cne_subscription_jwt supplied by the root from local files.no
far_service_account_b64string""Base64 FAR service account (from bnk.far_auth_local_file), used when use_cos_bucket=false.no
f5_cne_subscription_jwtstring""Subscription JWT contents (from bnk.subscription_jwt_local_file), used when use_cos_bucket=false.no
flp_vsi_ssh_keystring""Existing VPC SSH key name to attach to the FLP VSI (operator access). Empty = no key.no
flp_vsi_create_vpcboolfalseBuild the proxy its own VPC, address prefix and public gateway instead of placing it in one that already exists. Default false keeps existing workspaces byte-identical.no
flp_vsi_vpc_namestring""Name for the VPC created when flp_vsi_create_vpc = true. Empty uses flp-vsi-vpc.no
flp_vsi_subnet_cidrstring"10.250.0.0/24"Address prefix for the VPC created when flp_vsi_create_vpc = true. Must not overlap anything the consuming clusters can already route to.no
ibmcloud_cos_resource_groupstring""Resource group holding the supply-chain COS instance. Empty resolves the workspace’s own resource group (#295).no

Module: gateway

Source: terraform/modules/gateway/variables.tf

VariableTypeDefaultDescriptionSensitive
bnk_linestring"2.3"BNK release line driving per-release resource gating (“2.3” or “2.4”).no
ibmcloud_api_keystringrequiredIBM Cloud API keyyes
ibmcloud_cluster_regionstringrequiredIBM Cloud region the cluster runs in (zone names derive as <region>-1/2/3)no
roks_cluster_name_or_idstringrequiredExisting ROKS cluster name or id the Gateway phase configuresno
kubeconfig_dirstring""Directory the IBM provider writes the cluster kubeconfig intono
create_roks_clusterboolfalseAlways false for the Gateway phase (it reuses an existing cluster)no
roks_cluster_dependency_idstringnullCluster-ready handle (apply-time ordering); unused gate kept for symmetryno
deploy_gatewayboolfalseMaster toggle — when false the whole Gateway phase is a no-op (count=0)no
flo_namespacestring"f5-bnk"BNK namespace (F5BnkGateway / SnatPool / Egress / StaticRoute live here)no
app_namespacestring"f5-app"Application namespace the Gateway + HTTPRoute serve (created by this module)no
cneinstance_network_zoneslist(object({[ { ext_vlan_cidr = "10.155.15.0/24" int_vlan_cidr = "10.254.99.0/24" int_snat_cidr = "10.10.11.0/24" int_vip_cidr = "10.135.15.0/24" external_selfip = "10.155.15.101" internal_selfip = "10.254.99.101" }, { ext_vlan_cidr = "10.156.16.0/24" int_vlan_cidr = "10.254.100.0/24" int_snat_cidr = "10.10.21.0/24" int_vip_cidr = "10.136.16.0/24" external_selfip = "10.156.16.101" internal_selfip = "10.254.100.101" }, { ext_vlan_cidr = "10.157.17.0/24" int_vlan_cidr = "10.254.101.0/24" int_snat_cidr = "10.10.31.0/24" int_vip_cidr = "10.137.17.0/24" external_selfip = "10.157.17.101" internal_selfip = "10.254.101.101" }, ]Per-zone subnet CIDRs (empty = install-guide defaults). Shared with the BNK phase.no
gateway_class_namestring"gateway-class"GatewayClass nameno
gateway_controller_namestring""GatewayClass controllerName. Empty (the default) DERIVES it from flo_namespace as “f5.com/<flo_namespace>-f5-cne-controller”, which is the only value the CNE controller answers to — it must equal the CNEInstance’s own name (modules/cne_instance/…/main.tf builds that as “<flo_namespace>-f5-cne-controller”). This used to be the literal “f5.com/f5-bnk-f5-cne-controller”, i.e. the DEFAULT namespace baked in as a constant, so any workspace with a non-default bnk.flo_namespace got a GatewayClass no controller ever accepted and a Gateway that never programmed. Set explicitly only to point at a controller this module did not deploy.no
gateway_bnkgateway_namestring"bnkgateway-cloud1"F5BnkGateway name (referenced by the Gateway parametersRef)no
gateway_namestring"http-gw"Gateway (gateway.networking.k8s.io) nameno
gateway_listener_portnumber80Gateway HTTP listener portno
gateway_vip_start_hostnumber100Host number in int_vip_cidr for F5BnkGateway listener startAddressno
gateway_vip_end_hostnumber120Host number in int_vip_cidr for F5BnkGateway listener endAddressno
gateway_route_namestring"http-route"HTTPRoute nameno
gateway_backend_servicestring"nginx-service"HTTPRoute backend Service name in the app namespaceno
gateway_backend_portnumber80HTTPRoute backend Service portno
gateway_route_exampleslist(string)[]Extra route kinds to create WORKING examples of, alongside the default HTTPRoute. Empty (the default) changes nothing about an existing deployment.no
gateway_l4_listener_portnumber8080Port for the TCP listener added when gateway_route_examples includes L4Routeno
gateway_egress_modestring"snatpool"Egress SNAT strategy: snatpool (default; creates the SnatPool + snatpool Egress), automap (automap Egress only), or both.no
gateway_snatpool_namestring"egress-snat-vx102"F5SPKSnatpool nameno
gateway_snat_hostnumber111Host number in int_snat_cidr for each zone’s SNAT addressno
gateway_egress_app_interfacestring"eth0"Application pod interface the Egress interceptsno
gateway_egress_tmm_interfacestring"external-vlan"TMM interface name the Egress VXLAN binds to (matches the external VLAN)no
gateway_egress_node_interfacestring"ens3"Node interface name for the Egress VXLANno
gateway_egress_mtunumber1460Egress VXLAN MTUno
gateway_vxlan_portnumber6789Egress VXLAN UDP port (also opened on the cluster security group)no
gateway_client_subnet_locallist(string)[]Local-VSI client subnet CIDRs the static routes reach (cluster-VPC clients; one F5SPKStaticRoute per entry × zone). Empty = no local client routes.no
gateway_client_subnet_remotelist(string)[]Remote-VSI client subnet CIDRs the static routes reach (client-VPC clients over the TGW; one F5SPKStaticRoute per entry × zone). Empty = no remote client routes.no
gateway_static_route_gw_hostnumber1Host number in ext_vlan_cidr used as each zone’s static-route gatewayno
gateway_infra_namestring"cloud-infra"Name of the BNK 2.4 Infra CR (gateway.k8s.f5.com/v1alpha1) in the FLO namespace.no
gateway_settings_namestring"gw-settings-common"Name of the BNK 2.4 GatewaySettings CR in the FLO namespace.no
gateway_nad_namestring"ens3-ipvlan-l2"NetworkAttachmentDefinition the 2.4 Infra CR’s external VLAN attaches to.no
gateway_egress_vxlan_portnumber6789egressDefaults.port on the 2.4 Infra CR. The CRD default is 4789; the install guide’s worked example uses 6789, which is what this defaults to so a stock install matches the vendor’s own manifest.no
gateway_egress_vxlan_subnetstring""egressDefaults.subnet on the 2.4 Infra CR. Empty leaves the controller’s default of 10.0.0.0/16.no

Module: license

Source: terraform/modules/license/variables.tf

VariableTypeDefaultDescriptionSensitive
flo_namespacestring"f5-bnk"Namespace the CNEInstance lives in, for the 2.4 post-licensing health check (#167).no
bnk_linestring"2.3"BNK release line (‘2.3’ or ‘2.4’) for per-release gating. See PRD 18 §4.no
ibmcloud_api_keystringrequiredIBM Cloud API Keyyes
ibmcloud_cluster_regionstring"ca-tor"IBM Cloud region where the cluster residesno
ibmcloud_resource_groupstring"default"IBM Cloud Resource Group name (leave empty to use account default)no
ibmcloud_cos_bucket_regionstring"us-south"IBM Cloud region where the COS bucket is locatedno
ibmcloud_cos_instance_namestring"bnk-supply-chain"IBM Cloud COS instance nameno
ibmcloud_resources_cos_bucketstring"bnk-artifacts"IBM Cloud COS bucket containing the FAR auth key and JWT filesno
roks_cluster_name_or_idstringrequiredName or ID of the existing OpenShift ROKS cluster to deploy BNK ontono
flo_utils_namespacestring"f5-utils"Namespace for F5 utility componentsno
f5_cne_subscription_jwt_filestring"subscription.jwt"Subscription JWT filename in the COS bucketno
use_cos_bucketbooltrueDownload the subscription JWT from COS. false = use the injected jwt_token content (local file).no
jwt_tokenstring""Subscription/license JWT content, injected when use_cos_bucket = false. Empty on the COS path.yes
license_modestring"connected"License operation mode (connected, disconnected, or f5licenseproxy)no
flp_license_server_urlstring""Base URL of the in-cluster F5 License Proxy (FLP mode only)no
license_server_root_castring""PEM of the FLP root CA, written to the licenseserver-rootca Secret (FLP mode only)no
create_roks_clusterboolfalseWhen true, cluster is being created by roks_cluster — skip plan-time cluster credential fetchno
roks_cluster_dependency_idstringnullroks_cluster sentinel ID — when set, defers runtime_config fetch to apply time after roks_cluster completesno
cneinstance_dependency_idstringnullcneinstance_ready_id from ws4 — when set, ensures License CRD is available before applying License CRno
deploy_bnkbooltrueDeploy BIG-IP Next for Kubernetes — when false the inner license module is disabled and no License resources are createdno
kubeconfig_dirstring"/work/.bnk/scratch/kubeconfig/license"Persistent, writable dir for ibm_container_cluster_config kubeconfig downloads. Defaults to a host-bind-mounted, module-scoped path under .bnk/scratch.no
roksbnkctl_binarystring""Absolute path to the roksbnkctl binary; the license phase invokes roksbnkctl tfx <verb> in place of host curl (no interpreter, so cmd.exe execs it on Windows). Empty falls back to roksbnkctl on PATH.no
cluster_absentboolfalseTrue in the standalone FLP-VSI phase: no ROKS cluster exists, so all cluster data-source lookups + kube providers are skipped (count=0).no
ibmcloud_cos_resource_groupstring""Resource group holding the supply-chain COS instance. Empty resolves the workspace’s own resource group (#295).no

Module: roks_cluster

Source: terraform/modules/roks_cluster/variables.tf

VariableTypeDefaultDescriptionSensitive
ibmcloud_api_keystringrequiredIBM Cloud API keyyes
ibmcloud_cluster_regionstringrequiredIBM Cloud region for all cluster resourcesno
ibmcloud_resource_groupstring"default"IBM Cloud resource group nameno
create_roks_clusterbooltrueCreate a new ROKS cluster. When false, supply roks_cluster_id_or_name instead.no
cluster_public_gatewaybooltrueAttach public gateways for worker Internet egress. true (default) = current behavior; false = private/disconnected cluster (no egress).no
roks_cluster_id_or_namestring""ID or name of an existing ROKS cluster — used when create_roks_cluster = falseno
create_roks_transit_gatewaybooltrueCreate Transit Gateway and VPC connectionsno
create_roks_registry_cos_instancebooltrueCreate Cloud Object Storage instance for the OpenShift image registryno
roks_cluster_vpc_namestring"tf-cluster-vpc"Name of the cluster VPCno
openshift_cluster_namestring"tf-openshift-cluster"Name of the OpenShift clusterno
openshift_cluster_versionstring"4.20"OpenShift cluster version as a BARE major.minor prefix (e.g. “4.20”). Empty uses the latest available.no
roks_workers_per_zonenumber1Number of worker nodes per availability zoneno
roks_min_worker_vcpu_countnumber16Minimum vCPU count when auto-selecting the worker node flavorno
roks_min_worker_memory_gbnumber64Minimum memory in GB when auto-selecting the worker node flavorno
roks_cos_instance_namestring"tf-openshift-cos-instance"Name of the COS instance for the OpenShift image registryno
roks_transit_gateway_namestring"tf-tgw"Name of the Transit Gatewayno
use_existing_cluster_vpcboolfalseReuse an existing cluster VPC instead of creating one (forwarded to module.cluster).no
existing_cluster_vpc_idstring""ID of the existing cluster VPC (used only when use_existing_cluster_vpc = true; forwarded to module.cluster).no
kubeconfig_dirstringrequiredDirectory where ibm_container_cluster_config writes the admin kubeconfig. Must be writable; set explicitly to avoid the provider’s HOME-derived default, which resolves empty under the roksbnkctl runner.no
roksbnkctl_binarystring""Absolute path to the roksbnkctl binary; the cluster phase invokes roksbnkctl tfx <verb> in place of host curl/kubectl (no interpreter, so cmd.exe execs it on Windows). roksbnkctl sets this via TF_VAR_roksbnkctl_binary; empty falls back to roksbnkctl on PATH.no
cluster_absentboolfalseTrue only in the standalone FLP-VSI phase: no ROKS cluster exists or will be adopted, so all cluster data-source lookups + kube providers across modules are skipped (count=0).no
cluster_vpc_cidrstring""CIDR block the cluster VPC’s per-zone address prefixes are carved from (e.g. “10.241.0.0/16” → 10.241.0.0/18, 10.241.64.0/18, 10.241.128.0/18).no
use_existing_cluster_subnetsboolfalsePlace the cluster in subnets that already exist instead of creating them. Requires use_existing_cluster_vpc — a subnet cannot be adopted independently of its VPC.no
existing_cluster_subnet_idslist(string)[]Subnet ids to place the cluster in, one per zone, in zone order. Used only when use_existing_cluster_subnets = true. Their zones are read from the subnets themselves.no
cluster_http_allowed_cidrslist(string)[]Source CIDRs allowed to reach :80 on the cluster security group. Empty → 0.0.0.0/0.no
cluster_vpc_default_sg_inbound_cidrslist(string)[]Source CIDRs allowed inbound (all protocols/ports) to the cluster VPC’s default security group. Empty → 0.0.0.0/0.no
roks_worker_flavorstring""Exact worker-node flavor, e.g. “cx3d.8x20”. Empty auto-selects.no
bnk_linestring"2.3"BNK release line (2.3 / 2.4), used to pick a VALIDATED default worker flavor per line.no

Module: testing

Source: terraform/modules/testing/variables.tf

VariableTypeDefaultDescriptionSensitive
ibmcloud_api_keystringrequiredIBM Cloud API Keyyes
ibmcloud_cluster_regionstring"ca-tor"IBM Cloud region where the referenced cluster residesno
ibmcloud_resource_groupstring""IBM Cloud Resource Group name (leave empty to use account default)no
roks_cluster_name_or_idstringrequiredName or ID of the existing OpenShift ROKS clusterno
testing_create_tgw_jumphostbooltrueCreate a jumphost in a client VPC and (optionally) connect it to the cluster via a Transit Gatewayno
testing_create_cluster_jumphostsboolfalseCreate one jumphost per availability zone directly inside the cluster VPCno
testing_ssh_key_namestring""Name of the SSH key to inject into all jumphosts. Must exist in client_vpc_region (for TGW jumphost) and in ibmcloud_cluster_region (for cluster jumphosts)no
testing_jumphost_profilestring""Instance profile for all jumphosts (leave empty to auto-select from min_vcpu_count and min_memory_gb)no
testing_jumphost_total_volume_bandwidthnumber1000Volume bandwidth (Mbps) pinned on each jumphost. IBM computes this from the profile at creation and terraform carries the stored value into a later profile change, so an unmanaged value taken from an oversized profile makes a downsize impossible (#316). 1000 is ample for jumphosts, which run curl/iperf3 and no disk-heavy work, and is below the ceiling of every profile the auto-select can pick.no
testing_min_vcpu_countnumber4Minimum vCPU count when auto-selecting the instance profileno
testing_min_memory_gbnumber8Minimum memory in GB when auto-selecting the instance profileno
testing_create_client_vpcboolfalseCreate a new client VPC for the TGW jumphost. When false, client_vpc_name must reference an existing VPCno
testing_client_vpc_namestring"tf-testing-vpc"Name of the client VPC — created when create_client_vpc = true, or looked up when create_client_vpc = falseno
testing_client_vpc_regionstring"ca-tor"IBM Cloud region for the client VPC and TGW jumphostno
testing_transit_gateway_namestring""Name of an existing Transit Gateway to connect the client VPC to (leave empty to skip TGW attachment)no
testing_tgw_jumphost_namestring"tf-testing-jumphost-tgw"Name of the TGW-connected jumphost instance (used as prefix for subnet, gateway, security group, and floating IP)no
testing_cluster_jumphost_name_prefixstring"tf-testing-jumphost-cluster"Name prefix for cluster jumphosts — zone name is appended (<prefix>-<zone>)no
roks_cluster_dependency_idstringnullroks_cluster sentinel ID — when set, defers cluster/TGW data source reads to apply time after roks_cluster completesno
create_roks_clusterboolfalseSet to true when the ROKS cluster is being created in this run — skips cluster-VPC-derived data sources that require a pre-existing clusterno
cluster_vpc_idstring""ID of the cluster VPC — pass module.roks_cluster.roks_cluster_vpc_id directly; avoids deriving via worker-pool subnet chain which is deferred to apply timeno
cluster_absentboolfalseTrue only in the standalone FLP-VSI phase: no ROKS cluster exists, so cluster data-source lookups are skipped (count=0).no
testing_jumphost_allowed_cidrslist(string)[]Source CIDRs allowed to SSH (:22) to the testing jumphosts. Empty → 0.0.0.0/0 (open — the jumphosts carry a public floating IP and are reached from wherever the operator happens to be; access is key-only). Narrow this to the operator’s public /32 on a shared account.no
testing_client_vpc_inbound_cidrslist(string)[]Source CIDRs allowed inbound to the testing client VPC’s DEFAULT security group. Empty → the RFC-1918 private ranges, which is where in-fabric test traffic arrives from (the cluster reaches the client VPC over the Transit Gateway). Set explicitly only if a test needs a public source.no

Module: tgw_connection

Source: terraform/modules/tgw_connection/variables.tf

VariableTypeDefaultDescriptionSensitive
deploy_tgw_connectionboolfalseAttach the cluster’s VPC to an existing Transit Gateway. Off in every other phase’s override; on only for tgw connect. When false the module is a complete no-op.no
ibmcloud_api_keystring""IBM Cloud API key (IAM auth + provider config).yes
ibmcloud_cluster_regionstring""Region of the cluster VPC (ibm provider + the ibm_is_vpc lookup).no
cluster_vpc_idstring""ID of the cluster’s VPC — the network attached to the Transit Gateway. Resolved to a CRN here; the connection needs the CRN, but cluster-outputs.json records the id.no
transit_gatewaystring""The EXISTING Transit Gateway to attach to, by NAME or by ID. Resolved against the account’s gateway list, so either works. Multiple clusters passing the same value share one gateway; each workspace owns its own connection.no
connection_namestring""Name for THIS cluster’s connection on the gateway. Must be unique per gateway, so it is prefix-derived — two clusters sharing one Transit Gateway get distinct connection names.no

Glossary

Plain-English definitions of the terms used across the book. Project-specific concepts, IBM-Cloud-specific products, OpenShift / Kubernetes admission concepts, and the F5 BIG-IP Next networking vocabulary all live here. Entries are deliberately one or two sentences; the deep-dive lives in the linked chapter where applicable.

A — D

api_key_b64 Base64-encoded IBM Cloud API key stored inline in the workspace config.yaml. Obfuscation, not encryption — anyone with file-read access decodes instantly. The field name deliberately doesn’t match the plaintext-secret rejection regex. See Chapter 14 §“Source 3”.

Backend (--backend) The execution context for a tool dispatch. One of local (os/exec on the host), docker (containerised), k8s (in-cluster ops pod or Job), or ssh:<target> (a registered SSH endpoint). See Chapter 17.

BNK BIG-IP Next for Kubernetes. F5’s Kubernetes-native CNF deployment of BIG-IP, made up of FLO (F5 Lifecycle Operator) + CNE Instance + License + CIS. The reason this CLI exists. See Chapter 1.

CIS Two unrelated CISes appear in this stack. Inside the cluster, CIS is F5’s Container Ingress Services — the F5 controller that watches Kubernetes Ingress + Route resources and programs the BIG-IP data plane. At the IBM Cloud account level, CIS is Cloud Internet Services — IBM’s DNS, CDN, WAF, and DDoS-protection product. Context disambiguates; when in doubt, “F5 CIS” vs “IBM Cloud CIS”.

ClusterIP (k8s) A Service type that gives a Service an internal cluster IP, reachable only from inside the cluster. Used by the throughput suite’s east-west mode.

ClusterRole (k8s) A cluster-scoped RBAC role granting verbs on resources. The ops pod’s least-privilege ClusterRole grants jobs.create in roksbnkctl-test but not pods.delete in default.

CNE Instance A Custom Resource defined by FLO. Represents one deployed instance of the BNK data plane (TMM pods + control plane). Sizing is Small/Medium/Large. See Chapter 10.

Cobra github.com/spf13/cobra — the Go CLI library roksbnkctl is built on. The command tree at internal/cli/ is a cobra command tree.

COS IBM Cloud Object Storage — S3-compatible object store. The BNK supply chain bucket lives on a COS instance. See Chapter 25.

cred resolver chain The ordered lookup for IBMCLOUD_API_KEY: env var → OS keychain → workspace api_key_b64 → interactive prompt. The chain stops at the first source that yields a non-empty value. See Chapter 14.

CRN Cloud Resource Name — IBM’s globally-unique resource identifier. Starts with crn:v1: and encodes account, region, service, and resource ID. Most roksbnkctl cos commands accept either a friendly name (resolved at runtime) or a CRN.

E — J

east-west Network direction term: traffic between two endpoints inside the cluster (pod-to-pod, service-to-service). The throughput suite’s --mode east-west measures CNI fabric throughput. See Chapter 22.

Embedded HCL The Terraform source tree compiled into the roksbnkctl binary via Go’s //go:embed directive. The default tf_source is embedded. Rebuilding the binary picks up HCL changes. See Chapter 31 §“The embedded HCL”.

envFrom (k8s) A Pod spec field that references a Secret or ConfigMap and projects all of its keys as environment variables into the container. The k8s backend’s ops pod uses envFrom: secretRef: roksbnkctl-ibm-creds to receive the API key without listing it in the manifest plaintext.

extra_hosts The workspace config’s list of additional URLs to probe under roksbnkctl test connectivity. The value is a bare []string of URLs; per-host method/expected-status overrides are deferred. See Chapter 20.

FAR F5 Application Runtime — the container-image distribution of the BIG-IP Next data plane. FLO pulls FAR images from repo.f5.com using the auth key in the COS supply-chain bucket.

FLO F5 Lifecycle Operator — the Kubernetes operator that owns the CNE Instance + License + supporting resources. The control plane piece of BNK. (The acronym sometimes also surfaces as “F5 Logging Operator” — context disambiguates; in this book it always means Lifecycle Operator.)

FQDN Fully Qualified Domain Name — the absolute form of a DNS name ending with a trailing dot (www.example.com.).

FAR auth key The credential tarball (f5-far-auth-key.tgz) that FLO uses to pull FAR images from repo.f5.com. Lives in the COS supply-chain bucket. Rotated periodically; see Chapter 25 §“Licence rotation”.

ghcr.io GitHub Container Registry — where the roksbnkctl-tools-* images are published. The k8s backend pulls from ghcr.io/jgruberf5/roksbnkctl-tools-{ibmcloud,iperf3}.

GSLB Global Server Load Balancing — DNS-driven traffic management where the answer a name returns depends on the requesting resolver’s network vantage. The thing Chapter 21 is built to validate.

--gslb-compare The DNS-probe flag that fans out across all configured backends in parallel and emits a comparison JSON with gslb_divergence: true|false. The signature workflow for “is the GSLB rule taking effect”. See Chapter 21 §“The –gslb-compare workflow”.

HCL HashiCorp Configuration Language — the syntax of Terraform .tf files. The upstream HCL is bundled into the binary; see Embedded HCL.

ibmcloud Two senses. The IBM Cloud CLI binary (which roksbnkctl ibmcloud … passes through to or replaces, depending on the backend). And the YAML block in config.yaml (ibmcloud:) holding region, resource group, and API key source.

ImagePullBackOff (k8s) A Pod status indicating the image couldn’t be pulled from the registry. Usually a network or auth problem; sometimes a tag-doesn’t-exist problem. See Chapter 26 §“ImagePullBackOff…”.

iperf3 mode (--mode) The throughput-suite flag selecting north-south (LoadBalancer Service, client outside the cluster) or east-west (ClusterIP Service, client inside the cluster). See Chapter 22 §“The two modes”.

JWT JSON Web Token — the signed-token format BNK uses for the subscription licence (subscription.jwt in the COS supply-chain bucket).

K — N

k (roksbnkctl k <verb>) The internalised kubectl subtree. roksbnkctl k get/apply/describe/delete/exec/logs/port-forward — built on k8s.io/client-go directly so no host kubectl binary is required. See Chapter 24.

kubeconfig The Kubernetes client-configuration file (clusters, contexts, credentials). Defaults to ~/.kube/config. roksbnkctl up auto-fetches the admin kubeconfig post-apply.

LoadBalancer (k8s Service type) A Service type that provisions an external endpoint (a cloud LB on managed Kubernetes; an external IP on bare-metal CNI). Used by the throughput suite’s north-south mode and by BNK’s exposed VIPs.

Long-lived ops pod The k8s backend’s persistent execution context. Deployed by roksbnkctl ops install; subsequent --backend k8s dispatches kubectl exec into the same pod rather than starting a fresh Pod each call. Contrasted with the one-shot Job pattern used for iperf3 and DNS probes. See Chapter 19.

BNK line / release line — The <major>.<minor> of a BNK release, e.g. 2.3. Not configured: derived from bnk.manifest_version. Selects the terraform overlay (Chapter 13) and gates the support-matrix check. An unset manifest_version derives the line from the built-in default rather than failing, because that field has always been optional.

Network mode (single-nic | multi-nic) — How a cluster’s worker nodes are attached. Fixed when the cluster is created and never converted in place. single-nic is the default and describes every cluster built with this tool today. multi-nic needs both IBM support that has not shipped and a BNK line that expresses it — and no shipped line does: 2.4 was expected to and, on the evidence of its EA install guide, does not.

Contract schema version — The schema_version in cluster-outputs.json, currently 2. Absent means 1. Declares the shape of the handoff record so a reader can tell what it is looking at.

Support matrix — internal/config/support_matrix.yaml: which BNK lines drive which network modes, and which contract versions each can read. Checked before planning. A known line with an unsupported mode is refused; a line the matrix has never heard of warns and proceeds, since that usually means the binary predates the release rather than that the pairing is wrong.

Manifest version (f5_bigip_k8s_manifest_version) The version pin on the f5-bigip-k8s-manifest Helm chart. Transitively pins both the FLO and CIS versions (both are extracted from the manifest chart). See Chapter 13. It also determines the derived BNK line above, so it selects the terraform overlay and gates the support-matrix check — a larger blast radius than the name suggests, and the reason there is no separate line field that could disagree with it.

mdBook rust-lang/mdBook — the static-site generator the book is built with. Markdown source under book/src/, HTML output under book/book/. See Chapter 31 §“The book build”.

miekg/dns github.com/miekg/dns — the Go DNS library the GSLB probe is built on. Same library CoreDNS uses; gives roksbnkctl test dns full record-type coverage and per-query server selection. See Chapter 21 §“The roksbnkctl test dns flag surface”.

north-south Network direction term: traffic crossing the cluster boundary — from outside the cluster to a pod inside, or vice versa. The throughput suite’s --mode north-south measures inbound LoadBalancer-path throughput. See Chapter 22.

NXDOMAIN DNS response code indicating “this name does not exist”. roksbnkctl test dns against a non-existent name exits 1 with rcode=NXDOMAIN.

O — R

--on <target> The persistent CLI flag dispatching an ibmcloud/exec/shell/kubectl/oc passthrough over SSH to a named target instead of running it locally. The other half of the SSH-client + --on feature alongside the SSH backend. See Chapter 16.

OpenShift Red Hat’s enterprise Kubernetes distribution. ROKS = managed OpenShift on IBM Cloud.

Ops pod Shorthand for the long-lived k8s-backend execution pod deployed in the roksbnkctl-ops namespace by roksbnkctl ops install. See Chapter 19.

passthrough A command that proxies its argv to an underlying tool. roksbnkctl ibmcloud … passes through to the ibmcloud CLI; roksbnkctl kubectl … passes through to kubectl. Passthroughs run on whatever backend is selected (local by default).

PRD Product Requirements Document. The project uses numbered PRDs under docs/prd/ to coordinate larger feature work. See Chapter 32 §“The PRD process”.

PHASE_FROM= The env-var resume mechanism on the e2e driver scripts. PHASE_FROM=L ./scripts/e2e-test-backends.sh fast-forwards past phases A-K. See Chapter 23 §“Resuming a partial run”.

RBAC Role-Based Access Control — the Kubernetes authorization model. The ops pod has a least-privilege RBAC binding; see ClusterRole.

restricted-v2 The default OpenShift PodSecurity policy / SCC at admission. Rejects pods that run as root, allow privilege escalation, or hold the ALL capability set. All roksbnkctl-managed pods (ops pod, iperf3 server, DNS probe Job) are written to satisfy restricted-v2. See Chapter 22 §“The bundled image and the runAsNonRoot constraint”.

redactor The output-stream wrapper at internal/exec/redact.go that masks the IBM API key value in any subprocess’s stdout/stderr before it reaches the user’s terminal or the log. The defence-in-depth net for credential leaks. See Chapter 14 §“The redactor”.

ROKS Red Hat OpenShift on IBM Cloud — IBM’s managed OpenShift offering. The cluster roksbnkctl up provisions. See Chapter 2.

runAsNonRoot A Pod / container securityContext field. Required true by restricted-v2. Images that have USER root in the Dockerfile fail admission with this set.

RTT Round-Trip Time — measured in milliseconds for each DNS query. roksbnkctl test dns -o json surfaces p50/p95/p99 across the run.

S — Z

Schematic JSON The deployer-rendered JSON document describing a BNK deployment. Lives in the COS supply-chain bucket; not consumed at install time, kept for forensics.

SCC Security Context Constraint — OpenShift’s pod-admission policy. restricted-v2 is the default; pods that violate it (e.g., by running as root) are rejected by the admission controller. See Chapter 22.

Secret (k8s) A namespaced resource holding key/value data, typically base64-encoded credentials. The k8s backend creates roksbnkctl-ibm-creds in the roksbnkctl-ops namespace at ops install time.

secretRef (k8s) The Pod spec form that references a Secret for environment-variable projection. Used together with envFrom for the ops pod’s credential injection.

Service (k8s sense) A Kubernetes resource that provides a stable endpoint for accessing one or more Pods. Types: ClusterIP (default), NodePort, LoadBalancer, ExternalName. See ClusterIP, LoadBalancer.

SPDY Speedy (protocol). The websocket-like, multiplexed-stream protocol Kubernetes uses for exec and port-forward. roksbnkctl k exec is a SPDY client implementation on top of k8s.io/client-go’s SPDY executor.

SSH backend The --backend ssh:<target> execution path. Runs the tool on a registered SSH endpoint via the internal/remote.Client wrapper. See Chapter 17 §“SSH backend”.

TGW Transit Gateway — IBM Cloud’s VPC-to-VPC connectivity service. The upstream HCL provisions a TGW between the cluster VPC and the testing-client VPC so the jumphost can reach the cluster’s internal endpoints.

tfvars (terraform.tfvars) Variable-value file for Terraform — assigns concrete values to the HCL’s variable blocks. roksbnkctl auto-renders one from config.yaml; user overrides layer on top via terraform.tfvars.user and --var-file. See Chapter 13.

tf_source The workspace config.yaml block selecting where the Terraform source comes from: embedded (compiled into the binary; the default), github (downloaded tarball), local (an on-disk directory). See Chapter 12 §“tf_source:”.

TLS (--insecure) Transport Layer Security. The --insecure flag on roksbnkctl test connectivity skips TLS certificate validation for every probe in the run (session-wide, not per-host).

TMM Traffic Management Microkernel — the BIG-IP data-plane process. BNK runs TMM as a Pod; the CNE Instance specifies how many and at what size.

TOFU Trust On First Use — the SSH-style host-key acceptance pattern. On first connection to a new SSH target, roksbnkctl prompts the user to verify the fingerprint; subsequent connections check against the saved fingerprint in ~/.roksbnkctl/known_hosts. A fingerprint mismatch refuses to connect. See Chapter 16 §“Host-key handling”.

Trusted Profile An IBM IAM construct that lets a Kubernetes ServiceAccount assume IBM Cloud permissions. FLO uses one to authenticate against IBM Cloud APIs without storing an API key in the cluster.

TTL Time To Live — DNS-record cache duration in seconds. roksbnkctl test dns -o json surfaces each answer’s TTL.

v1.0 The release this book is the launch deliverable for. All E2E phases pass on a clean dev box; doctor green-by-default with terraform-only required.

VPE Virtual Private Endpoint — IBM Cloud’s private-network access point for managed services. Sometimes left dangling after a cluster destroy (see Chapter 26 §“orphan IBM Cloud resources”).

VPC Virtual Private Cloud — IBM Cloud’s network-isolation primitive. The cluster lives in one VPC; the testing client jumphost lives in another, connected via TGW.

VSI Virtual Server Instance — IBM Cloud’s general-purpose VM. The jumphosts are VSIs.

workspace A named slot under ~/.roksbnkctl/<name>/ containing one config.yaml, one Terraform state directory, and (usually) one kubeconfig. The kubectl-style multi-environment isolation primitive. See Chapter 6.

ws / workspace The CLI subtree managing workspaces. roksbnkctl ws new/use/list/delete.

Cross-references

Building from source

This chapter is for contributors and operators who want to build roksbnkctl themselves — whether to test an unreleased change, to verify a release artefact, or to embed a custom HCL fork into the binary.

For users who just want to install the binary, Chapter 4 — Installation is the right page. This chapter is the build-side companion.

Go version requirement

The minimum Go version is the one pinned in go.mod:

go 1.25.0

We pin to a recent toolchain for two reasons:

  1. The IBM Cloud Go SDKs (go-sdk-core/v5, platform-services-go-sdk, ibm-cos-sdk-go) and k8s.io/client-go v0.30+ both make liberal use of Go’s modern generics — pre-1.21 toolchains won’t build.
  2. Several dependencies (miekg/dns, docker/docker) test on the current and previous minor only; we follow upstream.

Install Go via your package manager (brew install go, apt-get install golang-1.25, etc.) or from go.dev/dl. Verify with go version.

Quick build

The shortest path from a fresh clone to a working binary:

git clone https://github.com/jgruberf5/roksbnkctl.git
cd roksbnkctl
go build -o roksbnkctl ./cmd/roksbnkctl
./roksbnkctl --version

go build produces a static binary in the working directory. Cross-compilation to a different OS/arch needs GOOS / GOARCH set:

GOOS=linux   GOARCH=amd64 go build -o roksbnkctl-linux-amd64   ./cmd/roksbnkctl
GOOS=darwin  GOARCH=arm64 go build -o roksbnkctl-darwin-arm64  ./cmd/roksbnkctl
GOOS=windows GOARCH=amd64 go build -o roksbnkctl.exe          ./cmd/roksbnkctl

A full multi-platform build is easier through goreleaser:

goreleaser release --snapshot --clean
# Output lands in dist/

The --snapshot --clean flags produce a local build without trying to publish to GitHub. The release shape is described in .goreleaser.yml — Linux + macOS × amd64 + arm64, plus a Windows compile-only check.

Build via the Makefile

The repo’s Makefile wraps the common build steps and stamps version metadata into the binary:

make build              # builds bin/roksbnkctl with -ldflags version stamping
make test               # go test ./...
make vet                # go vet ./...
make tidy               # go mod tidy
make test-short         # go test -short ./...
make test-integration   # testcontainers-go-backed integration tests (needs Docker)
make test-cred-audit    # the security-spine regression suite
make lint               # gofmt + vet + staticcheck (if installed)

The version stamp comes from three ldflags variables baked into internal/cli:

var (
    Version   = "dev"
    Commit    = "none"
    BuildDate = "unknown"
)

make build passes -X github.com/jgruberf5/roksbnkctl/internal/cli.Version=$VERSION (and the others) so roksbnkctl --version reports the actual git-rev and build timestamp rather than the placeholders. Set VERSION explicitly when stamping a release:

VERSION=v1.0.0 make build

The embedded HCL

The Terraform source tree at terraform/ is compiled into the binary via Go’s //go:embed directive. The embed declaration lives at the repo root in embedded.go (and is wired through internal/tf/ to be served as the tf_source: embedded provider).

Two implications:

  1. Rebuilding the binary picks up HCL changes. If you’re hacking on the HCL, make build produces a binary that ships your changes embedded. No separate “deploy the HCL” step.
  2. The HCL is read-only at runtime. The binary extracts it to a temporary directory on first use; the extracted copy is what terraform operates on. The original embedded source is immutable.

For users who want to not use the embedded HCL, the tf_source: github or tf_source: local options in the workspace config bypass it entirely. See Chapter 12 §“tf_source:”.

The bundled tools images

The tools/docker/ directory holds Dockerfiles for the images the docker and k8s backends use, plus the release-time book builder:

tools/docker/
├── Makefile
├── ibmcloud/
│   └── Dockerfile      # roksbnkctl-tools-ibmcloud
├── iperf3/
│   └── Dockerfile      # roksbnkctl-tools-iperf3
└── mdbook/
    └── Dockerfile      # roksbnkctl-tools-mdbook (release-time book builder)

tools/docker/Makefile builds the three images locally as :dev:

cd tools/docker
make build-ibmcloud     # builds roksbnkctl-tools-ibmcloud:dev
make build-iperf3       # builds roksbnkctl-tools-iperf3:dev
make build-mdbook       # builds roksbnkctl-tools-mdbook:dev
make build-all          # all three

The :dev tag is what a from-source roksbnkctl resolves to when the binary’s Version is dev. A tag-released binary (v1.0.0) resolves to ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:v1.0.0 instead — the resolver logic lives in internal/exec/ (SetToolImageTag is wired in internal/cli/root.go::init). See Chapter 17 §“:dev tag resolution”.

The GitHub Actions workflow tools-images.yml builds and pushes the published images via a strategy.matrix.image over [ibmcloud, iperf3, mdbook]. Every push to main republishes the :dev tag for each image; every v* tag push publishes :<tagname> + :latest. workflow_dispatch is also wired for manual runs. The mdbook image is part of the matrix, so edits to tools/docker/mdbook/Dockerfile are republished automatically: the next push to main republishes ghcr.io/jgruberf5/roksbnkctl-tools-mdbook:dev with no manual make -C tools/docker build-mdbook + docker push step on the release-cut host.

The book build

The book is built with mdBook. Install:

cargo install mdbook
# or
brew install mdbook

The book source lives under book/src/ with book.toml as the config. Common operations:

make book-serve         # mdbook serve book/ --open
                        # opens http://localhost:3000 with live-reload
make book               # mdbook build book/
                        # static HTML at book/book/
make book-clean         # rm -rf book/book

The published site at https://jgruberf5.github.io/roksbnkctl/book/ is built and deployed by .github/workflows/book.yml on every push to main. The workflow runs mdbook build book/ and pushes the output to the gh-pages branch via peaceiris/actions-gh-pages.

For PR-time verification, .github/workflows/spellcheck.yml runs cspell on book/src/**/*.md — a warning, not a gate, but worth eyeballing the output before merging.

The auto-generated chapters

Two reference chapters are generated rather than hand-written. The generators live under tools/refgen/:

# Chapter 27 — command reference (walks the cobra command tree)
go run ./tools/refgen/cobra-md > book/src/27-command-reference.md

# Chapter 29 — terraform variable reference (parses terraform/variables.tf)
go run ./tools/refgen/tfvars-md > book/src/29-terraform-variable-reference.md

When to re-run:

  • Chapter 27: any change to the cobra command tree under internal/cli/ or cmd/roksbnkctl/ — new commands, renamed flags, edited Long: / Example: strings.
  • Chapter 29: any change to terraform/variables.tf or any submodule variables.tf referenced from the root — new variables, default-value changes, edited descriptions.

Both generators emit deterministic output — the same input HCL or cobra tree always produces the same markdown — so you can commit the rendered output to source control without worrying about spurious diff churn.

Cross-compile matrix

goreleaser covers the canonical matrix:

OSArchitectureStatus
Linuxamd64Fully supported
Linuxarm64Fully supported
macOSamd64 (Intel)Fully supported
macOSarm64 (Apple Silicon)Fully supported
Windowsamd64Compile-only; SSH TTY support degraded
Windowsarm64Compile-only; same caveat
FreeBSDamd64Not tested

The Windows caveat is real: golang.org/x/crypto/ssh’s PTY allocation isn’t complete on Windows, so roksbnkctl shell --on jumphost falls back to a non-TTY shell. The other commands (exec, ibmcloud, kubectl) work fine on Windows.

Output from goreleaser release --snapshot --clean lands in dist/:

dist/
├── roksbnkctl_linux_amd64_v1/
│   └── roksbnkctl
├── roksbnkctl_linux_arm64/
│   └── roksbnkctl
├── roksbnkctl_darwin_amd64_v1/
│   └── roksbnkctl
├── roksbnkctl_darwin_arm64/
│   └── roksbnkctl
└── ...

Each archive bundles the binary plus LICENSE, README.md, and the rendered book/book/ directory (when the snapshot is built from a tagged commit).

Release process

Tagged releases are cut on the main branch:

# Update CHANGELOG.md with the release notes
git tag -a v1.0.0 -m "v1.0.0 — book launch + full E2E coverage"
git push origin v1.0.0

The push triggers release.yml, which runs goreleaser release to:

  1. Cross-compile the binary for the supported OS/arch matrix.
  2. Build the matching tools images and push to ghcr.io/jgruberf5/roksbnkctl-tools-*:<tag>.
  3. Attach the binaries, checksums (checksums.txt), and the rendered book PDF (if mdbook-pdf is configured) to the GitHub release.
  4. Generate release notes from the CHANGELOG and the commits since the previous tag.

The release-gate criteria — what has to hold before tagging — are documented in PLAN.md §“v1.0 (M4)”. The most important: full E2E green for 3 consecutive nights on main.

Cross-references

Extending roksbnkctl

This chapter is the hacking guide for contributors. It covers the four most common extension shapes — adding a new execution backend, a new test suite, a new tool to an existing backend, a new chapter to the book — plus the PRD process the project uses to coordinate larger changes and the four-agent sprint-dispatch pattern Sprints 0-6 ran on.

For building the binary, see Chapter 31 — Building from source. For using the binary, see the rest of the book.

Adding a new execution backend

A backend is anything implementing the Backend interface in internal/exec/backend.go. The four backends shipped at v1.0 — local, docker, k8s, ssh:<target> — are each a single file under that package.

The end-to-end shape:

  1. Implement the interface. Create internal/exec/<your-backend>.go. The contract is Run(ctx context.Context, argv []string, opts RunOpts) (int, error). Honour opts.Stdin/Stdout/Stderr, opts.WorkDir, opts.Env, opts.Credentials, opts.HostMounts, and opts.RunAsUser. Return the subprocess exit code as the first return; second is for backend-side errors (couldn’t start, ctx cancelled, etc.).

  2. Register it. Call exec.Register(name string, b Backend) from the package’s init() block. The ResolveBackend(spec string) function in internal/exec/backend.go dispatches --backend <name> to the registered backend.

  3. Handle credentials safely. Read PRD 04 before touching opts.Credentials. The cardinal rule: never pass credential values via argv — they end up in ps output, container metadata, and process accounting. Pass by reference (env var by name, projected Secret, SSH SetEnv) and let the runtime do the value plumbing.

  4. Wire the redactor. Wrap opts.Stdout and opts.Stderr with internal/exec.NewRedactor before handing them to the subprocess. The redactor masks any credential value that leaks into the tool’s stdout/stderr. The local and docker backends do this in their wrappers; copy the pattern.

  5. Add a doctor check. Doctor’s per-backend availability check needs to recognise your backend. Add an entry under internal/cli/doctor_backend.go reporting whether your backend’s prerequisites are satisfied (e.g., “is the daemon running”, “is the SDK reachable”). Green-by-default on a stock dev box is the goal — yellow-skip rather than red-fail when prerequisites are missing.

  6. Add per-backend cred-audit assertions. The cred-leak audit at internal/exec/audit_test.go (and Phase M of the e2e plan) needs to know what surfaces your backend produces — container inspection, process listing, log files. Add a TestCredAudit_<YourBackend> subtest asserting the API key value never appears in any of them.

  7. E2E phase. Add a new phase to scripts/e2e-test-backends.sh with concrete pass/fail criteria. Cross-link from PRD 05 so the test plan stays the source of truth.

  8. Documentation. Add a deep-dive subsection to Chapter 17 — Execution backends and a decision-tree entry to Chapter 18 — Choosing a backend per tool. Without docs the backend doesn’t exist for users.

A backend PR that lands all eight steps is a complete contribution; one that lands the code but skips the audit and docs will get a “please come back with…” review comment.

Adding a new test suite

The test subtree (roksbnkctl test <suite>) holds three suites at v1.0: connectivity, dns, throughput. Adding a fourth (e.g., tls-handshake, latency, tcp-flowstate) follows a five-step recipe:

  1. Implement the runner. Create internal/test/<suite>.go. The suite produces results in the roksbnkctl.<suite>.v1 JSON schema — pick a top-level shape consistent with the existing suites (ProbeResult for single-probe, ProbeSuiteResult for an aggregate with results[]).

  2. Wire a subcommand. Add internal/cli/test_<suite>.go with a cobra command under test. The flag surface should mirror the existing suites’ patterns — --target, --iterations, -o json, --backend (when the suite is backend-aware).

  3. Pick the backends. Most test suites are backend-aware (the suite runs from a network vantage that the backend selects). DNS and throughput accept local / k8s / ssh:<target> and reject docker; connectivity is currently local-only. Decide which backends make sense for your suite — the deciding question is “does the vantage change the answer?”.

  4. Wire the JSON schema constant. Add roksbnkctl.<suite>.v1 to your suite’s output. CI assertions diff against this — bumping the version is a breaking change, document it in CHANGELOG.

  5. Add an E2E phase. New phase under PRD 05 and corresponding script section in scripts/e2e-test-backends.sh.

  6. Documentation. New chapter or major section in Part VI of the book (currently chapters 20-23). Cross-link from Chapter 23 — The E2E test plan and Chapter 18 — Choosing a backend per tool.

The DNS probe is the canonical worked example — read internal/test/dns.go + internal/cli/test.go to see all six steps in their landed form.

Adding a new tool to an existing backend

The docker, k8s, and ssh backends each maintain a map of tool-name → image / package. Adding a new tool (e.g., mtr, tcpdump, helm) means an entry in each backend’s map.

Docker backend

internal/exec/docker.go::toolImages maps tool names to image specs:

var toolImages = map[string]string{
    "ibmcloud":  "ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud",
    "iperf3":    "ghcr.io/jgruberf5/roksbnkctl-tools-iperf3",
    "terraform": "hashicorp/terraform:1.5.7",
    "<your>":    "<your-image-ref>",
}

Tag resolution is handled by SetToolImageTag (set in internal/cli/root.go::init) — a :dev tag for a from-source binary, :<release-tag> for a tagged release. If your image needs its ENTRYPOINT bypassed (e.g., for image-specific argv mangling), add a jobToolCmdOverride entry.

K8s backend

internal/exec/k8s.go holds two patterns — long-lived ops pod (for tools that share state, like ibmcloud) and one-shot Job (for tools that produce a single output, like iperf3 or DNS probes). New tools pick one pattern:

  • Ops pod: add the tool’s image to the ops pod’s container spec at install time, or kubectl exec into the existing ops pod and run the host-installed binary.
  • One-shot Job: build a Pod template using the same image conventions as iperf3, run, stream logs, capture exit code, delete. The Job pattern is the right call for tools where the result is the only thing that matters.

SSH backend

internal/exec/ssh.go maintains a map of tool names to apt-package names for the --bootstrap auto-install:

// toolPackage carries apt-repo metadata + package name; see the
// production form in internal/exec/ssh.go for the full struct shape
// (IBM repo URL + GPG key + apt-source line for ibmcloud-cli, etc.).
var toolPackages = map[string]toolPackage{
    "ibmcloud": { /* IBM apt repo + key + "ibmcloud-cli" */ },
    "iperf3":   { /* plain ubuntu-main "iperf3" */ },
    "<your>":   { /* repo + key + "<deb-package>" */ },
}

The bootstrap step runs apt-get install -y <packages> on the SSH target when the tool isn’t already on PATH. Non-Debian targets are out of scope for v1.0; the bootstrap fails clearly with a message pointing at the manual-install path.

For each backend, the implementation work is small (one map entry). The doctor checks, e2e coverage, and docs are the bulk — same shape as adding a new backend, scaled to the smaller surface.

Supporting a new BNK release

Two version axes vary independently here — the BNK release, and the IBM platform capability the cluster was built with. Adding a release means telling the tool about the new cell, not forking anything.

1. Add a row to internal/config/support_matrix.yaml.

  - bnk: "2.5"
    contract: [1, 2]                       # schema versions this line can read
    network_modes: [single-nic, multi-nic]
    notes: >-
      What changed, and anything a reader needs to know before trusting the row.

The line itself is derived from bnk.manifest_version (2.5.0-… → 2.5), so there is no new config field and nothing that can disagree with the manifest actually being installed.

2. Add terraform/lines/2.5/ only if the HCL genuinely differs. It mirrors the base tree and is layered on top at extraction: same path replaces, new path is added, nothing is removed. A difference a variable can express belongs in the base as a variable — an overlay is a fork of a file, and two copies of a file drift. An empty lines/ is the correct state whenever the base serves every supported release, which is the case today.

3. Add the CI coverage for the cell. From the matrix’s own header: “Adding a cell means adding a CI plan for it. A cell with no coverage is a claim nobody has checked.”

4. Mark it provisional until it has actually run. A row written before the release ships is an expectation, not an observation. Say so in the row — the 2.4 row does — so that a plan-time pass isn’t mistaken for verification.

Note what is not on this list: cutting a branch. That was the first answer and it was withdrawn. A branch per release forks the whole tool — CLI fixes, IBM module updates, docs, CI — to express a difference that lives in a handful of .tf files, and it makes every subsequent change pick a target branch.

Adding a new chapter to the book

The book is mdBook with markdown source under book/src/. Adding a chapter:

  1. Create book/src/<NN>-<slug>.md — the file. Numbered prefix for sort order.
  2. Add the chapter to book/src/SUMMARY.md — the TOC. Use the existing parts (Concepts, Getting Started, Cluster Lifecycle, …) or add a new part if it doesn’t fit.
  3. Run make book-serve to live-preview at http://localhost:3000 with auto-reload.
  4. Cross-link from related chapters at the bottom (the “Cross-references” section every chapter ends with).
  5. Push. .github/workflows/book.yml re-deploys to gh-pages on every merge to main.

The book follows a consistent style:

  • Lower-case prose, sentence-case section headers.
  • Code blocks for any command, inline code for filenames and identifiers.
  • Short paragraphs, one idea each.
  • Examples should be runnable as written.
  • PRD references use the full GitHub URL (https://github.com/jgruberf5/roksbnkctl/blob/main/docs/prd/03-EXECUTION-BACKENDS.md) to avoid the published-book 404 issue surfaced in Sprint 1.

The PRD process

The project uses numbered Product Requirements Documents under docs/prd/ for larger feature work — anything that touches multiple files, spans more than one sprint, or has open design questions that need to be settled before code lands.

When a feature warrants a PRD vs. a direct PR:

Use a PRUse a PRD
Single-file changeMulti-file change across internal/{exec,cli,config,…}
Bug fixNew subsystem (a new backend, a new test suite)
Doc fixNew surface that needs a stable contract (a JSON schema, a workspace-config field)
Refactor with no behaviour changeA change that needs threat-model thinking (creds, network, multi-tenancy)
Drive-by polishAnything cross-cutting >50 LOC

The PRD lifecycle:

  1. Draft: open as a markdown file under docs/prd/NN-<TITLE>.md. The structure should follow the existing PRDs (00-OVERVIEW, 01-SSH, 02-KUBECTL, 03-BACKENDS, 04-CREDS, 05-E2E): goal, approach, file-by-file plan, test plan, acceptance criteria, open questions.
  2. Review: open a PR adding the PRD. Discuss in the PR. Open questions get resolved by edit or by punting to a follow-up issue.
  3. Implement: the PRD becomes the implementation plan. Per-sprint tasks land in docs/PLAN.md referencing the PRD by number.
  4. Land: code PRs reference the PRD; the PRD itself is the spec, code is the implementation. When the implementation diverges from the PRD, the PRD gets updated to match — never the other way around (the binary’s behaviour is the source of truth).

The PLAN.md per-sprint planning rhythm interleaves code + tests + docs per sprint. Each sprint’s prompts translate the PLAN into concrete agent tasks.

The four-agent sprint dispatch

Larger sprints (Sprints 3-6) are dispatched as four parallel agents:

  • Architect — designs the surface, drafts the book chapters that explain it, files architect-side issues.
  • Staff engineer — writes the production Go and shell code, modifies the bundled HCL when needed.
  • Tech-writer — reviews the architect’s chapters for accuracy, fluency, and cross-link integrity. Files tech-writer-side issues.
  • Validator — writes / extends the e2e test scripts and CI workflows, files validator-side issues.

Each agent runs independently from its own prompt — one per agent — against the same repo snapshot. An integrator at the end folds the four agents’ outputs together, resolves the issues each filed against the others, and commits the aggregate.

When to dispatch four agents vs. just open a PR:

Direct PRFour-agent sprint
Single feature, single sprint, <10 filesMulti-feature sprint with code + docs + tests scope
Bug fixNew PRD landing
Drive-by improvementSprint-gate milestone work
You’re the only contributorCoordinating with reviewers who’d otherwise serialise

prompts/README.md documents the agent-coordination pattern. The sprint dispatch is the project’s way of running review-and-implementation in parallel rather than serial — it works when the surfaces are well-separated (code vs docs vs tests don’t conflict on file ownership) and the integrator has enough context to merge the four lanes.

Worked example: adding a new execution backend

End-to-end Part IX scenario: you want to add a podman backend (rootless container runtime as an alternative to docker) so users on Fedora/RHEL hosts that ship podman by default don’t have to install Docker just to use the --backend docker workflow. Same surface, different daemon. The walkthrough below tracks the eight-step recipe above with concrete file paths and a diff-shaped sketch of each change.

# 1. Implement the interface — new backend file
cat > internal/exec/podman.go <<'GO'
package exec

import (
    "context"
    "os/exec"
)

type podmanBackend struct{}

func (p *podmanBackend) Run(ctx context.Context, argv []string, opts RunOpts) (int, error) {
    args := append([]string{"run", "--rm"}, dockerStyleArgs(opts)...)
    args = append(args, opts.Image)
    args = append(args, argv...)
    cmd := exec.CommandContext(ctx, "podman", args...)
    return runWithRedactor(cmd, opts)
}

func init() {
    Register("podman", &podmanBackend{})
}
GO

# 2. Add the tool image mapping (podman uses the same OCI images as docker)
# Edit internal/exec/podman.go — add a toolImages map analogous to docker.go,
# or share the docker.go map by exporting it. The two registries are
# compatible; you'd typically share.

# 3. Doctor check — internal/cli/doctor_backend.go
# Add a `checkPodmanBackend()` function that runs `podman info` once with a
# 2s timeout. Green if exit 0, yellow if podman not found, red if podman
# present but daemon unreachable.

# 4. Wire credentials — re-use the docker backend's cred-propagation logic
# (the `-e VAR` pattern works identically for podman). Pass opts.Credentials
# by env-var reference, never by argv. See internal/exec/docker.go::
# dockerStyleArgs for the pattern to copy.

# 5. Add cred-audit test
cat > internal/exec/podman_audit_test.go <<'GO'
package exec_test

func TestCredAudit_Podman(t *testing.T) {
    // Run a no-op command via the podman backend with a known API key,
    // then inspect `podman inspect`'s output for the key value. Assert
    // the value never appears in the container's labels, env, or args.
}
GO

# 6. E2E phase — extend scripts/e2e-test-backends.sh
# Add Phase P (or extend Phase K) with a parallel sequence to K2-K6 but
# using --backend podman. Cross-link to PRD 05.

# 7. Documentation — chapters 17 + 18
# - Chapter 17: add a "Podman backend" section parallel to "Docker backend",
#   noting it's rootless-by-default and a drop-in alternative.
# - Chapter 18: add a row to the per-tool matrix; add a decision-tree entry
#   ("I'm on a podman-only host"); update the at-a-glance table.

# 8. Run the full test suite
go build ./...
go vet ./...
go test ./...
DRY_RUN=1 ./scripts/e2e-test-backends.sh

The PR should land all eight steps in one commit-set. A reviewer will look for: registered init(), doctor check, cred-audit test, e2e phase, and the two chapter additions. Without the audit + docs, the PR isn’t complete — see the cardinal rule at the top of the Adding a new execution backend section.

The same pattern applies to a new test suite, a new tool on an existing backend, or a new chapter — the eight-step recipe is the long version; the worked example is the copy-paste short version. Pick the shape that matches your contribution.

Cross-references

Agentic mode

Agentic mode lets you drive a ROKS + BNK trial with an AI coding agent (Claude Code, Gemini CLI, aider, …) acting under role-scoped personas, while roksbnkctl stays a deterministic tool. The binary embeds no LLM — you bring your own agent CLI; roksbnkctl provides the shared knowledge base (AGENTS.md), the role contracts (personas/), and a coordination substrate (journal/, decisions.md) scaffolded into the workspace.

It’s markdown plus a launcher. Nothing about agentic mode changes how the lifecycle commands behave.

Quick start

roksbnkctl -w acme agent init        # scaffold AGENTS.md + personas/ + journal/ into the workspace
roksbnkctl -w acme agent             # list supported CLIs + the workspace default
roksbnkctl -w acme agent claude      # print the invocation to launch Claude Code there

agent init writes into the workspace dir (~/.roksbnkctl/<workspace>/, alongside config.yaml):

AGENTS.md            shared reference every persona reads first
CLAUDE.md            one-line @AGENTS.md include for Claude Code
personas/            the four role contracts
decisions.md         decision-log seed
journal/             append-only timeline (handoffs live here)

Re-running agent init never clobbers your edits — existing files are left as-is.

The personas

Act as exactly one at a time. Each persona file defines an identity, ordered goals, a tool allowlist, what’s explicitly not allowed, a handoff protocol, and a phase checklist.

PersonaRoleTouches infra?Talks to customer?
solution-architectOwns scope (config.yaml) + decisions.md; drives the phasesNo (delegates)Yes (the only one)
cloud-operatorRuns the lifecycle (cluster up, bnk up, …), gates destructive stepsYesNo
test-engineerRuns connectivity/DNS/throughput probes, records evidenceNo (read + test)No
doc-specialistReads everything, writes report.mdNoNo

The boundaries are enforced by convention (the allowlists in each persona file), not by code — so the personas work with any agent CLI.

How they coordinate

config.yaml is the contract (scope/source of truth). The personas hand off through the append-only journal:

roksbnkctl -w acme journal add "cloud-operator: cluster up complete, nodes Ready"
roksbnkctl -w acme journal list      # chronological, with one-line summaries
roksbnkctl -w acme journal report    # assemble report.md from decisions.md + the journal

Never edit another persona’s journal entry — append a follow-up instead. The solution-architect requests infra work by journaling a request (with the consent tier); the cloud-operator executes and journals the result; the doc-specialist turns the timeline into the customer-facing report.md.

Choosing the CLI + endpoint

roksbnkctl agent <cli> supports claude, gemini, aider, openai, pi, opencode, and agy (Google). Set the workspace default and an optional LLM endpoint in config.yaml:

agent:
  default: claude
  llm_endpoint: ""   # OpenAI-/Anthropic-compatible base URL (cloud vendor or local vLLM); blank = the CLI's own config

The agy recipe seeds the persona with -i, which runs an initial prompt and then continues interactively. That is deliberate: pi and opencode auto-load AGENTS.md from the working directory, and their recipes rely on it, but agy does not advertise that behaviour — so its recipe passes the instruction rather than assuming the file is read.

roksbnkctl agent <cli> launches the agent, in the workspace directory, with your terminal attached. Add --show to print the invocation instead of running it — useful for putting it in a script, or for openai, whose recipe is guidance for whichever OpenAI-compatible REPL you use rather than a single command.

$ roksbnkctl agent agy            # launches agy in the workspace
$ roksbnkctl agent agy --show     # prints the invocation, runs nothing

roksbnkctl still embeds no LLM and sends nothing anywhere itself: it execs your CLI, exactly as roksbnkctl kubectl execs kubectl. Your CLI uses your credentials and endpoint.

Do not pipe the --show output into a shell. roksbnkctl agent <cli> --show | bash looks like the obvious shortcut and is silently wrong: bash’s standard input is the pipe carrying the recipe, so the agent’s first turn is the remaining comment lines of its own recipe. Nothing errors — the session simply begins as though you had typed them. If you need to run a printed recipe, use eval "$(roksbnkctl agent <cli> --show)", which leaves your terminal on standard input.

Running without --show is refused when standard output is not a terminal. An agent session needs one, and capturing it — $(...) or a pipe — would send the model’s output somewhere it does not belong.

Safety model

The personas inherit roksbnkctl’s existing guard-rails rather than replacing them:

  • Destructive commands (cluster up, down, cluster down, cleanup) still prompt unless --auto is passed — and the persona contracts forbid passing --auto to cluster up/cleanup without journaled consent.
  • Secret-tier commands (roksbnkctl apikey) and reading key/JWT/token material are off-limits to every persona except where a role genuinely needs it.
  • forge/kubeconfig.yaml carries the cluster’s admin client certificate/key (ROKS/OpenShift rejects raw IAM tokens) — treat it as secret-tier and never copy it into deliverables.

See the scaffolded AGENTS.md for the full operator reference (the phased lifecycle, the destructive-command gate contract, and the field-tested gotchas).

Appendix A — The four cluster topologies, end to end

BNK on ROKS comes in four shapes, and every one of them is the same roksbnkctl pipeline with different answers to two questions:

  1. Does roksbnkctl build the cluster, or adopt one that already exists?
  2. Do the workers have Internet egress, or not?

That is the whole taxonomy. This appendix walks all four end to end, twice: once driven by hand from the CLI, once driven as a CI pipeline by Argo Workflows — the same binary, the same commands, the same workspace layout, only the thing typing them changes.

#TopologyClusterWorker egressArtifacts fromLicensingTransit Gateway
ANew + connectedroksbnkctl creates ityes (public gateway)repo.f5.com directlyF5 directlyroksbnkctl creates its own
BNew + disconnectedroksbnkctl creates itnoprivate mirror (Harbor)F5 License Proxyjoins an existing shared one
CExisting + connectedyou already have ityesrepo.f5.com directlyF5 directlyalready attached
DExisting + disconnectedyou already have itnoprivate mirror (Harbor)F5 License Proxyalready attached

A and C need nothing else. They reach F5 over the Internet, so there is no mirror to fill and no proxy to build. If that is your case, skip straight to 1A or 1C — Part 0 does not apply to you.

B and D need the services infrastructure first — a Harbor registry holding a mirror of F5’s artifacts, and a standalone F5 License Proxy, both reachable from the cluster over a Transit Gateway. That is Part 0, and it is built once and shared by every disconnected cluster afterwards.

Related chapters: Air-gapped install, The F5 License Proxy, Sharing a Transit Gateway, Registering an existing cluster.

The topology — what reaches the Internet, and what doesn’t

graph TB
    subgraph net["Internet"]
        FAR["repo.f5.com<br/>FAR artifacts"]
        TEEM["F5 TEEM + licensing"]
    end
    subgraph svc["SERVICES VPC — built once, shared (topologies B + D)<br/>public gateway = the only egress"]
        Harbor["Harbor VSI<br/>OCI mirror · addressed by PRIVATE IP<br/>▶ the CLI operator runs HERE"]
        FLP["FLP VSI<br/>F5 License Proxy"]
    end
    TGW{{"Transit Gateway · global<br/>private RFC1918 only"}}
    subgraph conn["A / C — CONNECTED cluster VPC<br/>public_gateway: true"]
        CW["workers WITH egress"]
    end
    subgraph disc["B / D — DISCONNECTED cluster VPC<br/>public_gateway: false"]
        DW["workers, no egress"]
        Priv["IBM private service endpoints<br/>ICR · IAM · COS · master<br/>161.26.0.0/16 · 166.8.0.0/14"]
    end
    Harbor -->|egress| FAR
    FLP -->|egress| FAR
    FLP -->|egress| TEEM
    CW -->|"direct"| FAR
    CW -->|"direct"| TEEM
    Harbor --- TGW
    FLP --- TGW
    TGW --- DW
    DW -->|"images + charts"| Harbor
    DW -->|"licensing"| FLP
    DW --> Priv
    classDef vpc fill:#eef3fb,stroke:#4b6ea9,color:#000;
    classDef inet fill:#fdf2f2,stroke:#c0392b,color:#000;
    class svc,conn,disc vpc;
    class net inet;

Reachability summary

ComponentInternet?How it reaches what it needs
Harbor VSI (B, D)Yes — FAR pull onlypublic gateway (services VPC)
FLP VSI (B, D)Yes — F5 licensing + TEEM onlypublic gateway (services VPC)
Connected workers (A, C)Yestheir own public gateway
Disconnected workers (B, D) → Harbor / FLPNoprivate IPs over the Transit Gateway
Any workers → ICR / IAM / COS / masterNoIBM private service-endpoint CIDRs (161.26.0.0/16, 166.8.0.0/14) — routable without a public gateway
Operator (roksbnkctl, CLI)on the Harbor VSI for B/D; anywhere with egress for A/C—

Why no operator-built VPEs are needed for the platform. A public_gateway: false ROKS VPC cluster still reaches its IBM Cloud dependencies privately: system-image pulls use private ICR (private.icr.io), IAM/COS use their private endpoints, and the master is reached over the private service endpoint ROKS provisions itself. All of those resolve into IBM Cloud’s service-endpoint ranges (161.26.0.0/16, 166.8.0.0/14), which the VPC routes without a public gateway. Calico SNATs pod egress to the node, and the node uses those implicit routes. So the platform works private-only out of the box; you only add explicit VPEs for a specific service instance you want on a dedicated private endpoint (e.g. a particular COS bucket), not for the core ICR/IAM/COS/master path.

roksbnkctl does not build the services VPC, Harbor, or the FLP host image. It provisions no registries and no standalone VPCs (VPC creation only happens inside the cluster phase). Part 0 uses the ibmcloud CLI for those prerequisites; everything after it is roksbnkctl.

Prerequisites

All four topologies:

  • An IBM Cloud API key with VPC, Kubernetes Service and Transit Gateway authority.
  • terraform >= 1.10 on PATH (the floor is enforced — see Chapter 12a), plus roksbnkctl and kubectl. roksbnkctl doctor should be healthy.
  • The FAR supply chain: f5-far-auth-key.tgz and subscription.jwt, either staged as files or — better for CI — uploaded once into a COS bucket (see the box in S2).

Additionally for B and D (disconnected):

  • An existing global Transit Gateway (a global gateway spans regions; a local one does not).
  • An RSA VPC SSH key (IBM Cloud VPC rejects ed25519).
  • The ibmcloud CLI with the vpc-infrastructure and tg-cli plugins, plus jq, to build the services VPC in Part 0.

Versions this appendix was tested against

ComponentTestedFloor, and why
roksbnkctl / roksbnkctl-tools-runnerv1.42.0≥ v1.36.0 for registry adopt (B and D cannot work without it); ≥ v1.37.0 for bnkforge unregister; ≥ v1.39.0 for cluster.vpc_cidr; ≥ v1.40.1 fixes a variable validation that raised Invalid index on every terraform 1.10 plan; ≥ v1.42.0 retries the reachability gate, which matters most for topology B — it builds the VPC and joins the gateway in the same run as the install.
terraform1.10.5 (shipped inside the runner)≥ 1.10, enforced. Do not assume newer behaviour — 1.10 and 1.15 differ in ways that have shipped bugs here.
OpenShift (ROKS)4.20 (recommended default)cluster.openshift_version: "4.20" pins the minor; the latest patch within it is selected automatically. The end-to-end run recorded here was on 4.18.51 — the pipeline is version-agnostic, but that is the build the timings and screenshots come from. Check ibmcloud ks versions for what your account offers; IBM’s own default moves ahead of this.
Argo Workflows (Part 2)v4.0.8≥ v3.4 for the emissary executor and sidecars:.
Kubernetes hosting Argo (Part 2)k3s v1.36.3+k3s1any conformant cluster with an RWO StorageClass.

Addressing — give every VPC its own block

Read this before creating anything, because it cannot be changed afterwards.

A Transit Gateway routes on VPC address prefixes. Two attached VPCs claiming the same block make the route ambiguous, and the gateway resolves that by silently blackholing one of them — no error, no log line. It surfaces later as intermittent image-pull timeouts against the mirror while every security group and ACL in the path plainly allows the traffic.

It is easy to hit by accident: IBM’s default address-prefix management is auto, which gives every VPC in a region the identical three per-zone prefixes (10.241.0.0/18, 10.241.64.0/18, 10.241.128.0/18). So a second disconnected cluster on a shared gateway collides by construction.

Set cluster.vpc_cidr (env ROKSBNKCTL_CLUSTER_VPC_CIDR) per cluster. A worked allocation:

VPCBlockPer-zone prefixes
services (Harbor + FLP)existing, auto10.241.0.0/18, .64.0/18, .128.0/18
first disconnected cluster10.245.0.0/1610.245.0.0/18, .64.0/18, .128.0/18
second disconnected cluster10.247.0.0/1610.247.0.0/18, .64.0/18, .128.0/18

The block is split three ways, so /18 is the smallest usable value. Leaving it blank keeps auto, which is fine for exactly one cluster per gateway. Since v1.39.0 both cluster up and tgw connect refuse a detectable overlap up front and name the conflicting VPC — but only a distinct block prevents it. Full write-up: Chapter 9a.

Topology A creates its own gateway and is its only member, so the default is safe there.

Part 0 — The services infrastructure (topologies B and D only)

Built once with the ibmcloud CLI, then shared by every disconnected cluster. If you only ever run connected clusters, skip this entire part.

export IBMCLOUD_API_KEY=…
export TGW_NAME=my-global-tgw
export SVC_REGION=us-east SVC_ZONE=us-east-1
export SVC_PREFIX=bnk-svc RESOURCE_GROUP=default
export SSH_KEY_NAME=bnk-rsa-key                 # an RSA key already in IBM Cloud VPC
export HARBOR_ADMIN_PASSWORD=…                  # you choose it; it is a credential
ibmcloud login --apikey "$IBMCLOUD_API_KEY" -r "$SVC_REGION" -g "$RESOURCE_GROUP" -q
TGW_ID=$(ibmcloud tg gateways --output json | jq -r --arg n "$TGW_NAME" '.[]|select(.name==$n)|.id')

S1 — Services VPC + Harbor, attached to the TGW

# VPC + subnet + public gateway (the only egress) + 22/443 inbound
SVC_VPC_ID=$(ibmcloud is vpc-create "${SVC_PREFIX}-vpc" --resource-group-name "$RESOURCE_GROUP" --output json | jq -r .id)
SVC_VPC_CRN=$(ibmcloud is vpc "$SVC_VPC_ID" --output json | jq -r .crn)
SUBNET_ID=$(ibmcloud is subnet-create "${SVC_PREFIX}-subnet" "$SVC_VPC_ID" --zone "$SVC_ZONE" \
              --ipv4-address-count 256 --resource-group-name "$RESOURCE_GROUP" --output json | jq -r .id)
PGW_ID=$(ibmcloud is public-gateway-create "${SVC_PREFIX}-pgw" "$SVC_VPC_ID" "$SVC_ZONE" --output json | jq -r .id)
ibmcloud is subnet-update "$SUBNET_ID" --pgw "$PGW_ID"
SG=$(ibmcloud is vpc "$SVC_VPC_ID" --output json | jq -r .default_security_group.id)
for p in 22 443; do ibmcloud is security-group-rule-add "$SG" inbound tcp --port-min $p --port-max $p; done

# Attach to the gateway — the ONLY path from cluster workers to Harbor.
CONN_ID=$(ibmcloud tg connection-create "$TGW_ID" --name "${SVC_PREFIX}-conn" \
            --network-type vpc --network-id "$SVC_VPC_CRN" --output json | jq -r .id)
until [ "$(ibmcloud tg connection "$TGW_ID" "$CONN_ID" --output json | jq -r .status)" = attached ]; do sleep 6; done

Verify attached before moving on. If this connection silently fails, nothing complains until bnk up dies ~10 minutes in with cert_manager: context deadline exceeded — the nodes could not reach the mirror, and only nodes with a cached image survived.

Harbor’s TLS certificate must carry the floating IP in its SAN, so reserve the IP before rendering cloud-init:

ibmcloud is floating-ip-reserve "${SVC_PREFIX}-harbor-fip" --zone "$SVC_ZONE"
HARBOR_FIP=$(ibmcloud is floating-ips --output json | jq -r --arg n "${SVC_PREFIX}-harbor-fip" '.[]|select(.name==$n)|.address')
HARBOR_FIP_ID=$(ibmcloud is floating-ips --output json | jq -r --arg n "${SVC_PREFIX}-harbor-fip" '.[]|select(.name==$n)|.id')

export HARBOR_FIP HARBOR_ADMIN_PASSWORD HARBOR_VERSION=v2.11.1
envsubst '${HARBOR_FIP} ${HARBOR_VERSION} ${HARBOR_ADMIN_PASSWORD}' \
  < scripts/demos/disconnected-cluster-cli-demo/harbor-cloud-init.yaml.tmpl > /tmp/harbor.yaml

VSI=$(ibmcloud is instance-create "${SVC_PREFIX}-harbor" "$SVC_VPC_ID" "$SVC_ZONE" bx2-4x16 "$SUBNET_ID" \
        --image ibm-ubuntu-22-04-5-minimal-amd64-17 --keys "$SSH_KEY_NAME" \
        --resource-group-name "$RESOURCE_GROUP" --user-data @/tmp/harbor.yaml --output json)
HARBOR_VSI_ID=$(echo "$VSI" | jq -r .id)
VNI=$(echo "$VSI" | jq -r '.primary_network_attachment.virtual_network_interface.id')
ibmcloud is virtual-network-interface-floating-ip-add "$VNI" "$HARBOR_FIP_ID"

# The primary IP is 0.0.0.0 at create time — poll until it binds, or you bake 0.0.0.0
# into generic_host and every mirrored image resolves to https://0.0.0.0/…
until HARBOR_PRIVATE_IP=$(ibmcloud is instance "$HARBOR_VSI_ID" --output json \
        | jq -r '.primary_network_interface.primary_ip.address') \
      && [ -n "$HARBOR_PRIVATE_IP" ] && [ "$HARBOR_PRIVATE_IP" != 0.0.0.0 ]; do sleep 5; done
echo "Harbor private=$HARBOR_PRIVATE_IP floating=$HARBOR_FIP"

Give cloud-init 8–15 minutes, then create the projects:

until curl -sk --max-time 8 "https://${HARBOR_FIP}/api/v2.0/systeminfo" | grep -q harbor_version; do sleep 30; done
for pj in bnk-mirror bnk-status; do
  curl -sk -u "admin:${HARBOR_ADMIN_PASSWORD}" -X POST "https://${HARBOR_FIP}/api/v2.0/projects" \
       -H 'Content-Type: application/json' -d "{\"project_name\":\"$pj\",\"public\":true}"
done

bnk-mirror is public (anonymous pull). Harbor is network-isolated behind the gateway, so the network is the security boundary. Anonymous pull avoids a per-namespace pull-secret / ServiceAccount ordering race that can leave fresh cert-manager pods in ImagePullBackOff.

Finally, take the CA from the file that generated it — roksbnkctl refuses to adopt a self-signed CA it merely discovered over the wire:

ssh -i <key> ubuntu@"$HARBOR_FIP" 'sudo cat /opt/harbor/certs/harbor.crt' > harbor-ca.crt
HARBOR_CA_B64=$(base64 -w0 < harbor-ca.crt)

Put the operator on the VSI

For the CLI path, the Harbor VSI is the natural operator host: it is the only box with both Internet egress (to pull FAR) and private reach to Harbor.

This is not a convenience — it is a requirement. bnk up pulls the BNK manifest and the charts host-side (that is deliberate: pulling the archives itself keeps the terraform helm provider from doing an OCI login, whose credential-store step breaks on Windows). So the operator host must reach the mirror’s private address too. Running from a laptop that is not on the transit gateway gets as far as the manifest pull and then stops:

helm pull oci://10.243.0.4/bnk-mirror/release/f5-bigip-k8s-manifest
  dial tcp 10.243.0.4:443: i/o timeout

The node reachability gate does not cover this — it answers for the nodes, which is a different question from whether the operator can pull.

A stock Ubuntu VSI has none of the toolchain, so stage it once:

scp -i <key> "$(command -v roksbnkctl)" ubuntu@<HARBOR_FIP>:/home/ubuntu/roksbnkctl

# terraform >= 1.10 — the floor roksbnkctl enforces (native S3 state locking, Chapter 12a).
# Do not settle for whatever the distro ships; 1.9.x is rejected.
curl -fsSLo /tmp/tf.zip https://releases.hashicorp.com/terraform/1.10.5/terraform_1.10.5_linux_amd64.zip
sudo apt-get install -y unzip && unzip -o -q /tmp/tf.zip -d /tmp && sudo install -m0755 /tmp/terraform /usr/local/bin/

curl -fsSL https://get.helm.sh/helm-v3.16.3-linux-amd64.tar.gz -o /tmp/helm.tgz
tar xzf /tmp/helm.tgz -C /tmp && sudo install -m0755 /tmp/linux-amd64/helm /usr/local/bin/

Then either scp the two FAR artifacts across, or skip staging entirely with the COS supply chain below — which is what CI does, since a container has no local files.

S2 — Mirror FAR → Harbor

registry replicate pulls from repo.f5.com (the VSI’s egress) and pushes to Harbor at its private IP.

# mirror.yaml  (on the VSI)
ibmcloud: { region: us-east, resource_group: default }
prefix: mirror
tf_source: { type: embedded }
cluster: { create: false, name: none }
bnk:
  manifest_version: 2.3.0-3.2598.3-0.0.170
  far_auth_local_file: /root/f5-far-auth-key.tgz
  subscription_jwt_local_file: /root/subscription.jwt
registry:
  target: generic                            # ← see the warning below
  generic_host: <HARBOR_PRIVATE_IP>          # local to the VSI, private to the cluster
  generic_repo_prefix: bnk-mirror
  generic_username: admin
  generic_password_b64: <base64 of the Harbor admin password>
  generic_ca_b64: <HARBOR_CA_B64>            # from the file, not the wire
$ roksbnkctl -w mirror init --config-file mirror.yaml
$ roksbnkctl -w mirror registry bom          # the bill of materials to mirror
$ roksbnkctl -w mirror registry replicate    # FAR → Harbor (89 artifacts)
$ roksbnkctl -w mirror registry verify

registry.target: generic is not optional, and a --target flag is not a substitute. registry adopt — which topologies B and D run — has no --target flag; it reads registry.target from the workspace config. Left unset it defaults to icr, bnk up renders us.icr.io/<prefix>/…, and every pull on an air-gapped cluster fails with unauthorized: Authorization required. Passing --target generic to replicate fills Harbor correctly but writes nothing the later adopt reads, so the two silently disagree.

Local files vs. the COS supply chain. The config above points bnk.far_auth_local_file / bnk.subscription_jwt_local_file at files on the operator VSI — simplest when driving by hand. The alternative is to upload both once into a COS bucket and read them from there. This is mandatory for CI, where a container has no local files:

roksbnkctl cos object put bnk-artifacts-<acct>/f5-far-auth-key.tgz ./f5-far-auth-key.tgz --instance bnk-supply-chain
roksbnkctl cos object put bnk-artifacts-<acct>/subscription.jwt    ./subscription.jwt    --instance bnk-supply-chain

then drop the two *_local_file keys and add:

cos: { instance: bnk-supply-chain, bucket: bnk-artifacts-<acct>, region: us-south }
bnk:
  far_auth_file: f5-far-auth-key.tgz       # object keys in the bucket above
  subscription_jwt_file: subscription.jwt

The commands do not change — only the config keys. See Chapter 25 — the COS supply chain.

S3 — Standalone FLP licensing appliance

The FLP is a self-contained F5 licensing appliance — a VSI running the f5-license-proxy stack as a podman pod, with no cluster. It is the one box with controlled egress to F5: it pulls its own images from repo.f5.com and brokers licences (and sends TEEM telemetry) on the cluster’s behalf, over a private IP the disconnected cluster reaches via the gateway.

# flp.yaml  (on the VSI)
ibmcloud: { region: us-east, resource_group: default }
prefix: flp
tf_source: { type: embedded }
cluster: { create: false, name: none }
bnk:
  manifest_version: 2.3.0-3.2598.3-0.0.170
  far_auth_local_file: /root/f5-far-auth-key.tgz
  subscription_jwt_local_file: /root/subscription.jwt
flp:
  mode: vsi                       # mode:vsi + a non-empty vpc selects the cluster-less path
  vsi:
    vpc: <SVC_VPC_ID>
    zone: us-east-1
    profile: bx2-4x16
    ssh_key: <SSH_KEY_NAME>
    reach: private
    floating_ip: true             # operator address for `flp status` and the :80 UI
$ roksbnkctl -w flp flp init --config-file flp.yaml
$ roksbnkctl -w flp flp up --auto      # builds the VSI + the licence-proxy pod
$ roksbnkctl -w flp flp output         # → flp_external_endpoint + root CA

Keep both outputs — every disconnected cluster needs them as bnk.flp.external.url and bnk.flp.external.root_ca_b64.

The root CA is already base64. Pass it verbatim. Re-encoding it hands the CWC a corrupt CA and licensing fails in a way that looks like a network problem.

The status web UI (optional)

flp status queries a small status service on port 80 of the FLP. It is an optional add-on: it only exists if you built and pushed its image and set flp.vsi.status_image. The licence proxy works perfectly without it — do not let its absence fail a pipeline.

Part 1 — Driving it from the CLI

Four topologies, four subsections. Each is a complete config plus the commands, and each one is one roksbnkctl workspace (-w <name>) — a workspace’s terraform state describes exactly one cluster, so never point two clusters at the same one.

1A — New VPC + connected cluster (CLI)

The simplest shape, and the one to read first. Workers have egress, so BNK pulls charts and images straight from repo.f5.com and licenses directly against F5. Nothing is mirrored and nothing is proxied — there is no registry: block and no flp: block, and their absence is what makes it connected.

# connected.yaml
ibmcloud: { region: us-east, resource_group: default }
prefix: bnk-conn
tf_source: { type: embedded }
cluster:
  create: true
  name: bnk-conn                  # a NEW cluster is named from the prefix; see the note below
  openshift_version: "4.20"
  workers_per_zone: 1
  public_gateway: true            # ← CONNECTED: worker Internet egress
resources:
  transit_gateway:   { create: true }    # its own gateway; it is the only member
  registry_cos:      { create: true }    # REQUIRED when creating — see the warning
  cert_manager:      { create: true }
  bnk:               { create: true }
  tgw_jumphost:      { create: false }
  cluster_jumphosts: { create: false }
  client_vpc:        { create: false }
bnk:
  manifest_version: 2.3.0-3.2598.3-0.0.170
  far_auth_local_file: /root/f5-far-auth-key.tgz
  subscription_jwt_local_file: /root/subscription.jwt
  # license_mode defaults to `connected` — F5 directly
$ roksbnkctl -w bnkconn init --config-file connected.yaml
$ roksbnkctl -w bnkconn cluster up --auto     # ~45–55 min
$ roksbnkctl -w bnkconn bnk up --auto         # ~10 min
$ roksbnkctl -w bnkconn bnk status

⚠ registry_cos: { create: true } is mandatory when creating a cluster. ROKS-on-VPC refuses to provision without a COS instance backing its internal image registry (E7278). That is IBM Cloud COS over the private service-endpoint range — needed even air-gapped. create: false fails the cluster create outright.

A new cluster is named from prefix, not from cluster.name. cluster.name only selects which cluster to adopt (1C/1D). If you set a prefix of bnk-conn, the cluster comes up as bnk-conn — and that is the name 1C must later register.

1B — New VPC + disconnected cluster (CLI)

Requires Part 0. Three things make it disconnected, and all three are visible in the config: public_gateway: false, a registry: block pointing at Harbor, and license_mode: f5licenseproxy with the FLP endpoint.

It also joins an existing gateway rather than creating one — a new gateway of its own would isolate it from the very mirror it exists to install from.

# disconnected.yaml   (on the Harbor VSI)
ibmcloud: { region: us-south, resource_group: default }
prefix: bnk-disco
tf_source: { type: embedded }
cluster:
  create: true
  name: bnk-disco
  openshift_version: "4.20"
  workers_per_zone: 1
  public_gateway: false           # ← DISCONNECTED: no worker egress
  vpc_cidr: 10.242.0.0/16         # ← its own block; see "Addressing" above
resources:
  transit_gateway:   { create: false, existing: my-global-tgw }   # JOIN the mirror's gateway
  registry_cos:      { create: true }
  cert_manager:      { create: true }
  bnk:               { create: true }
  tgw_jumphost:      { create: false }
  cluster_jumphosts: { create: false }
  client_vpc:        { create: false }
registry:
  target: generic
  generic_host: <HARBOR_PRIVATE_IP>       # reached over the TGW
  generic_repo_prefix: bnk-mirror
  generic_username: admin
  generic_password_b64: <base64 of the Harbor admin password>
  generic_ca_b64: <HARBOR_CA_B64>
bnk:
  manifest_version: 2.3.0-3.2598.3-0.0.170
  far_auth_local_file: /root/f5-far-auth-key.tgz
  subscription_jwt_local_file: /root/subscription.jwt
  license_mode: f5licenseproxy
  flp:
    external:
      url: <flp_external_endpoint from S3>
      root_ca_b64: <flp_root_ca from S3>     # already base64 — verbatim
$ roksbnkctl -w bnkdisco init --config-file disconnected.yaml
$ roksbnkctl -w bnkdisco cluster up --auto     # ~45–55 min; TGW connect runs at the end
$ roksbnkctl -w bnkdisco tgw status
$ roksbnkctl -w bnkdisco registry adopt        # ← record the mirror in THIS workspace
$ roksbnkctl -w bnkdisco bnk up --auto
$ roksbnkctl -w bnkdisco bnk status

registry adopt is the step people miss. bnk up refuses to render against a mirror this workspace has no record of — otherwise BNK would be pointed at far_repo_url, which an air-gapped cluster cannot reach. Only registry replicate writes that record, and replicating again would need the FAR source this cluster has no route to. adopt writes the record from the registry config alone, with no source access. That is the entire reason it exists.

If you mirrored in a different workspace on the same host (S2 used -w mirror), copying the record works too: cp ~/.roksbnkctl/mirror/registry-mirror.json ~/.roksbnkctl/bnkdisco/.

Node CA trust is automatic. Before pulling, CRI-O on each node must trust Harbor’s self-signed cert or every pull fails x509. The usual OpenShift mechanism does not work on ROKS — image.config.openshift.io/cluster is HostedCluster-managed and a ValidatingAdmissionPolicy denies edits. bnk up installs the CA into each node’s /etc/containers/certs.d/<HARBOR_PRIVATE_IP>/ca.crt via a privileged DaemonSet using a node-cached image (so it needs no egress), and gates the install on the CA landing on every node. Because Harbor is addressed by an IP, the certs.d key is that IP — no node /etc/hosts entry needed.

Reachability is checked from every node first

The same DaemonSet also probes the mirror and the licence proxy from every node before anything tries to pull, and bnk up fails if the mirror is unreachable from any of them:

→ installing registry CA trust on all nodes (10.243.0.4) and checking reachability
  F5-License-Proxy: 3/3 nodes reachable
    ✓ kube-…-0000013a -> F5-License-Proxy (10.243.1.4:8443): dns=skipped-ip tcp=ok
  registry: 3/3 nodes reachable
    ✓ kube-…-0000013a -> registry (10.243.0.4:443): dns=skipped-ip tcp=ok
✓ registry CA installed on all nodes; 10.243.0.4 is trusted and reachable

It has to run on a node, not on the operator. The operator host sits on the services VPC with egress; the workers are air-gapped behind the gateway. A check from where roksbnkctl runs answers a different question and will return a confident green for a mirror the cluster cannot route to. Because a DaemonSet is one pod per node, this covers every availability zone without enumerating them — which matters, since a security group or subnet fixed in only two of three zones is a common shape and otherwise shows up merely as some pods failing to pull.

DNS and TCP are reported separately because the fixes differ: dns=FAILED is a resolver or hostname problem, tcp=FAILED is routing, security groups, or overlapping VPC address prefixes (which a gateway blackholes silently — Addressing).

The mirror is required: an unreachable one fails the install rather than deferring it to ImagePullBackOff and a helm context deadline exceeded that names neither the registry nor the node. The licence proxy is reported but not fatal, since a staged bring-up may legitimately point at an endpoint that is not up yet.

Two details worth knowing:

  • No CA is needed for the probe to run. A registry already trusted by the node bundle, or one with a publicly-signed certificate, skips the CA install but is still checked — being trusted and being reachable are different things, and only the second fails silently.
  • Every node must report. A node whose pod has not answered yet is not counted as a pass; the check waits for the DaemonSet’s full complement and fails if it cannot hear from them all, because a silent node is indistinguishable from a broken one.

It retries, and the budget is yours to set

A Transit Gateway attachment is asynchronous. IBM programs the routes some time after the connection reports attached, so a cluster built and installed in one run can probe a path that is correct but not yet live. This has been seen for real: a probe 73 seconds after attach failed on both targets, bnk up refused, and the same path was healthy minutes later — while a sibling cluster on the same gateway and the same blueprint passed, purely by landing on the other side of route programming.

“Unreachable” and “not reachable yet” are different claims, and one TCP failure cannot tell them apart. So each target is retried until its budget is spent; a success ends the retry immediately, so a healthy environment never pays for it. A failure then says how hard it tried:

✗ kube-…-0000013a -> registry (10.243.0.4:443): dns=skipped-ip tcp=FAILED
    (tcp connect failed -- refused, filtered, or no route
     (still failing after 19 attempts over 180s))

Both timers are workspace settings, because the right values are a property of the environment rather than of the tool:

bnk:
  preflight:
    reachability_retry_seconds: 180     # per target, before the verdict is believed
    reachability_timeout_seconds: 480   # for every node to report
Env varDefaultWhen to change it
reachability_retry_secondsROKSBNKCTL_REACHABILITY_RETRY_SECONDS180Raise when the gateway is attached in the same run as the install, or the fabric programs routes slowly. Set 0 for a static environment where a failure is never a race — one shot, fail immediately.
reachability_timeout_secondsROKSBNKCTL_REACHABILITY_TIMEOUT_SECONDS480Raise alongside a large retry budget. It is clamped to stay above the retry budget regardless, since a wait shorter than the retry would give up while the probe was still working.

The env vars matter for the CI path, where the workspace is built by init --non-interactive --override-from-env and there is no config.yaml to edit.

Each run gets a fresh verdict

The probe runs once per pod and then holds the pod Ready. That means the result bnk up reads is only as fresh as the pod that wrote it — so the gate rolls the DaemonSet on every run, even when the CA and the target list are unchanged. Without that, a re-run after fixing the routing re-read the original pod’s log and showed the same failure, with nothing to indicate it was a recording. The cost is one pod restart per node against a node-cached image, and the CA install is idempotent.

Keep SSL_CERT_FILE pointed at Harbor’s cert when running on the VSI — the chart pulls happen host-side: export SSL_CERT_FILE=/opt/harbor/certs/harbor.crt.

1C — Existing connected cluster (CLI)

roksbnkctl provisions nothing here. cluster register records the cluster’s identity — VPC, endpoints, registry COS — into cluster-outputs.json, which is what activates bnk up’s existing-cluster path.

# adopt-connected.yaml
ibmcloud: { region: us-east, resource_group: default }
prefix: bnk-conn
tf_source: { type: embedded }
cluster:
  create: false                   # ← ADOPT
  name: my-running-cluster        # ← the cluster to adopt, by name or id
resources:
  registry_cos:      { create: false }   # it already has one
  cert_manager:      { create: true }
  bnk:               { create: true }
  tgw_jumphost:      { create: false }
  cluster_jumphosts: { create: false }
  client_vpc:        { create: false }
bnk:
  manifest_version: 2.3.0-3.2598.3-0.0.170
  far_auth_local_file: /root/f5-far-auth-key.tgz
  subscription_jwt_local_file: /root/subscription.jwt
$ roksbnkctl -w bnkconn init --config-file adopt-connected.yaml
$ roksbnkctl -w bnkconn cluster register my-running-cluster   # writes cluster-outputs.json
$ roksbnkctl -w bnkconn kubeconfig --download                 # for kubectl
$ roksbnkctl -w bnkconn bnk up --auto
$ roksbnkctl -w bnkconn bnk status

cluster up also adopts. If cluster.create is false, roksbnkctl cluster up attaches to the cluster named in cluster.name — the same discovery and cluster-outputs.json write — instead of running the terraform create. One command covers both, which is what makes a single CI form able to drive either. See Chapter 9.

Re-installing over a live BNK? Watch the CWC. f5-spk-cwc uses a single-replica RWO PVC with a RollingUpdate strategy, so a re-install can deadlock: the new pod cannot mount the volume until the old one releases it, and the old one is not torn down first (Multi-Attach). Patch the Deployment to strategy: Recreate while bnk up runs. A fresh install has no prior CWC and does not hit this — it is specific to re-use. Part 2 automates it as a sidecar.

1D — Existing disconnected cluster (CLI)

1C plus the mirror and the proxy. Requires Part 0.

Take the 1C config and add the registry: block and the bnk.flp block exactly as in 1B, keeping cluster: { create: false, name: … } and registry_cos: { create: false }.

$ roksbnkctl -w bnkdisco init --config-file adopt-disconnected.yaml
$ roksbnkctl -w bnkdisco cluster register my-air-gapped-cluster
$ roksbnkctl -w bnkdisco kubeconfig --download
$ roksbnkctl -w bnkdisco tgw connect my-global-tgw      # idempotent; skip if already attached
$ roksbnkctl -w bnkdisco registry adopt
$ roksbnkctl -w bnkdisco bnk up --auto
$ roksbnkctl -w bnkdisco bnk status

An adopted cluster usually already runs cert-manager. OpenShift estates commonly install it as a day-1 add-on, and bnk up then stops with namespaces "cert-manager" already exists. That message names the collision but not the fix: set resources.cert_manager: { create: false } to adopt the one that is there. From CI, where there is no config.yaml to edit, the same thing is -e ROKSBNKCTL_CERT_MANAGER_CREATE=false. Check with kubectl get ns cert-manager before the first bnk up rather than after it.

Check the addressing before attaching. The existing cluster’s VPC must be on the same global gateway as Harbor and its address prefixes must not overlap the services VPC, or the gateway blackholes one of them. tgw connect refuses a detectable overlap and names the conflicting VPC. An existing VPC’s prefixes cannot be changed in place — moving a subnet’s CIDR replaces the subnet and destroys the cluster on it — so the remedy is to detach the other VPC, or rebuild this cluster on its own block.

Verifying any of the four

$ kubectl get pods -n f5-bnk                     # all Running
$ kubectl get pods -n f5-utils                   # the utility stack, incl. f5-spk-cwc
$ kubectl get license -n f5-utils                # STATE: Active
$ kubectl get cneinstance -n f5-bnk -o \
    jsonpath='{.items[0].status.conditions[?(@.type=="Available")].status}'   # True

The one check that distinguishes connected from disconnected is where the images came from:

$ kubectl -n cert-manager get pods \
    -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}'
quay.io/jetstack/cert-manager-controller:v1.17.3          # A / C — connected
10.241.0.4/bnk-mirror/jetstack/cert-manager-controller:…  # B / D — from the mirror

Teardown (CLI)

Per cluster workspace, then the shared services last:

$ roksbnkctl -w <ws> bnk down --auto            # BNK only; the cluster stays
$ roksbnkctl -w <ws> tgw disconnect --auto      # detach BEFORE destroying the cluster
$ roksbnkctl -w <ws> cluster down --auto        # the cluster + its VPC (+ its own TGW for topology A)

Then, only when no disconnected cluster is left:

$ roksbnkctl -w flp flp down --auto
$ ibmcloud is instance-delete "${SVC_PREFIX}-harbor" --force
$ ibmcloud tg connection-delete "$TGW_ID" "$CONN_ID"
$ ibmcloud is vpc-delete "$SVC_VPC_ID" --force   # after its subnet / pgw / floating IP are gone

Order matters, twice over. cluster down refuses while a TGW connection exists — the connection pins the VPC’s CRN and the VPC delete would fail — hence tgw disconnect in the middle. And disconnecting is not deleting: it removes only this cluster’s connection, so the shared gateway, the mirror’s connection and every other cluster’s stay. Topologies C and D adopted their clusters, so cluster down will not destroy them; remove BNK with bnk down and leave the cluster alone.

Part 2 — Driving it from CI with Argo Workflows

The same four topologies, same binary, same commands — as a pipeline. Each step is its own pod with its own status, logs and retries (the per-step visibility a single Job cannot give), and the workspace lives on a persistent PVC shared across the steps.

The complete, runnable manifests are in scripts/demos/blueprint-workflows-ci-demo. This part explains their shape; the repository has the YAML.

Everything is an environment variable

There is no config.yaml anywhere in Part 2 — the whole workspace is built from the environment by init --non-interactive --override-from-env, because that is the only shape available to a container module or an argv-only runner: no shell, no prompts, nowhere to stage a file.

CarrierHolds
bnk-env ConfigMapevery non-secret ROKSBNKCTL_* setting — readable in the Argo UI and via kubectl get cm bnk-env -o yaml
bnk-secrets SecretIBMCLOUD_API_KEY, ROKSBNKCTL_GENERIC_PASSWORD, and friends — never rendered, logged or printed

Adoption is environment-only too. A cluster the customer already built usually already has some of what bnk up would otherwise create, and each of those is a create toggle: ROKSBNKCTL_CERT_MANAGER_CREATE=false (the common one — see 1D), ROKSBNKCTL_REGISTRY_COS_CREATE=false when the mirror is external, and ROKSBNKCTL_CLUSTER_JUMPHOSTS_CREATE. Without these the CI path could express “adopt this cluster” but not “adopt what is on it”, and topologies C and D would stop on a collision no environment variable could clear.

Settings that define a topology — ROKSBNKCTL_CLUSTER_CREATE, ROKSBNKCTL_CLUSTER_PUBLIC_GATEWAY, ROKSBNKCTL_LICENSE_MODE — are pinned in the workflow YAML next to the steps they govern, not in the ConfigMap. That is why the connected and disconnected workflows differ in one line.

Bring your own controller — versions and constraints

Argo Workflows, not Argo CD. These are argo submit pipelines: no git repository, no Application, nothing to sync. If you already run Argo CD, it is not what drives this — you need the Argo Workflows controller alongside it. (Argo CD can own the lifecycle of the prerequisites below, since they are ordinary manifests.)

Versions tested are in the table above. The constraints your controller must satisfy — the first architectural, the rest each having cost a real run:

  • It must sit where it can reach the private addresses. For topologies B and D the controller pods pull the runner image from Harbor’s private IP and talk to the target cluster’s API. A hosted or SaaS Argo can do neither. Its nodes must be on, or routed to, the services VPC.

  • One workflow at a time per workspace. The PVC holds one terraform state, and ReadWriteOnce means one pod mounts it at a time anyway. Two runs against the same workspace fight over the state lock and the loser dies mid-apply. The shared bnk-env ConfigMap compounds it: it is re-rendered per run, and envFrom resolves per pod at creation, so an already-running workflow’s pending steps silently pick up the next run’s cluster name.

  • One roksbnkctl workspace per cluster — bnk for the mirror, bnkconn for the connected pair, bnkdisco for the disconnected pair, flp for the proxy. Terraform state describes one cluster, and registry-mirror.json is inherited by anything sharing the workspace, which is exactly how a connected cluster ends up rendering every image at a mirror it cannot reach.

  • ROKSBNKCTL_REGISTRY_TARGET=generic for B and D — see the warning in S2.

  • The namespace is bnk-ci in the manifests, hardcoded in each Workflow’s metadata.namespace.

The shared substrate

Applied once (workflows/00-prereqs.yaml): the namespace, the PVC, the ServiceAccount, and the executor RBAC.

apiVersion: v1
kind: Namespace
metadata: { name: bnk-ci }
---
apiVersion: v1
kind: PersistentVolumeClaim          # the persistent workspace every Workflow shares
metadata: { name: bnk-work, namespace: bnk-ci }
spec:
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 8Gi } }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: bnk-runner, namespace: bnk-ci }
---
apiVersion: rbac.authorization.k8s.io/v1        # Argo's emissary executor writes workflowtaskresults
kind: Role
metadata: { name: bnk-runner-executor, namespace: bnk-ci }
rules:
  - { apiGroups: ["argoproj.io"], resources: ["workflowtaskresults"], verbs: ["create", "patch"] }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: bnk-runner-executor, namespace: bnk-ci }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: bnk-runner-executor }
subjects: [{ kind: ServiceAccount, name: bnk-runner, namespace: bnk-ci }]

The PVC must not be an emptyDir. roksbnkctl keeps its config, terraform state and the mirror record under /work/.roksbnkctl, and every teardown reads that state. An emptyDir loses it when the pod ends — orphaning the IAM trusted profile and pull secrets, and leaving terraform nothing to destroy from.

A second, deliberately separate file (workflows/01-flp-handoff-rbac.yaml) grants write access to one named Secret, needed only if a workflow publishes the FLP handoff. It is separate so a blanket secret-write grant is never applied by accident.

Every step reuses one template:

    - name: rbk
      inputs: { parameters: [{ name: cmd }] }
      container:
        image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.42.0
        command: [sh, -ec]
        args: ["roksbnkctl {{inputs.parameters.cmd}}"]
        workingDir: /work
        envFrom:
          - configMapRef: { name: bnk-env }
          - secretRef:    { name: bnk-secrets }
        env:
          - { name: ROKSBNKCTL_HOME, value: "/work/.roksbnkctl" }
        volumeMounts: [{ name: work, mountPath: /work }]

For B and D the image is pulled from the mirror instead (<HARBOR_PRIVATE_IP>/bnk-mirror/roksbnkctl-tools-runner:v1.42.0), so the pipeline itself needs no public registry at run time. Pin by @sha256: digest in production.

2A — New VPC + connected cluster (Argo)

wf-new-cluster.yaml. Steps: init → cluster up → bnk up → bnk status.

        env:
          - { name: ROKSBNKCTL_CLUSTER_CREATE,         value: "true" }
          - { name: ROKSBNKCTL_CLUSTER_PUBLIC_GATEWAY, value: "true" }   # ← CONNECTED
          # Blanked ON PURPOSE — bnk-env carries a gateway name for the reuse workflows, and
          # inheriting it here would ADOPT the shared gateway instead of creating its own.
          # OverrideFromEnv skips empty values, so blank leaves transit_gateway at create:true.
          - { name: ROKSBNKCTL_TRANSIT_GATEWAY_NAME,   value: "" }
          # Blanked for the same reason: bnk-env carries the mirror host for the disconnected
          # workflows. Inherited, `init` would write registry.generic_host into this workspace
          # and every image would render at a mirror this cluster cannot reach.
          - { name: ROKSBNKCTL_GENERIC_HOST,           value: "" }

Uses workspace -w bnkconn.

2B — New VPC + disconnected cluster (Argo)

wf-new-cluster-disconnected.yaml. Steps: init → cluster up → registry adopt → bnk up → bnk status.

        envFrom:
          - configMapRef: { name: bnk-env }
          - secretRef:    { name: bnk-secrets }
          - secretRef:    { name: flp-handoff }    # written by the FLP workflow; last envFrom wins
        env:
          - { name: ROKSBNKCTL_CLUSTER_CREATE,         value: "true" }
          - { name: ROKSBNKCTL_CLUSTER_PUBLIC_GATEWAY, value: "false" }        # ← DISCONNECTED
          - { name: ROKSBNKCTL_LICENSE_MODE,           value: "f5licenseproxy" }
          # ROKSBNKCTL_TRANSIT_GATEWAY_NAME comes from bnk-env and is REQUIRED here.
          # ROKSBNKCTL_CLUSTER_VPC_CIDR likewise — this is the ONE workflow that creates a VPC
          # AND joins a shared gateway, so a second run collides by construction without it.

Uses workspace -w bnkdisco. The FLP endpoint and CA arrive as a Secret (flp-handoff) published by the FLP workflow rather than passing through a human.

2C — Existing connected cluster (Argo)

wf-existing-cluster.yaml. Steps: init → cluster register → kubeconfig --download → bnk up (with the CWC guard sidecar) → bnk status.

The kubeconfig fetch is new relative to 2A: the adopt path needs it so the k verbs and the guard can reach the cluster. Workspace -w bnkconn, with the same ROKSBNKCTL_GENERIC_* blanking.

    - name: bnk-up
      container:
        image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.42.0
        args: ["roksbnkctl -w bnkconn bnk up --auto"]
        # …envFrom / env / volumeMounts as the rbk template…
      sidecars:
        - name: cwc-guard
          image: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.42.0
          command: [sh, -ec]
          args:
            - |
              # Patch the single-replica RWO CWC Deployment to Recreate so the rolling
              # update cannot deadlock on Multi-Attach, then wait for the licence.
              kubectl -n "$CWC_NAMESPACE" patch deploy "$CWC_DEPLOYMENT" \
                -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'
              …

Why a sidecar and not a step. The deadlock it clears is what makes bnk up fail, so a guard running afterwards could never run at all. It has to be alongside. Observed working: cwc: strategy=Recreate applied / cwc: licence Active — guard done, while bnk up was still running. The guard is on the reuse workflows only — a fresh install has no prior CWC and does not deadlock.

2D — Existing disconnected cluster (Argo)

wf-existing-disconnected.yaml — 2C plus the mirror. Steps: init → cluster register → kubeconfig --download → registry adopt → bnk up (with the CWC guard sidecar) → bnk status. Workspace -w bnkdisco, flp-handoff in envFrom.

The two prerequisite workflows

Topologies B and D also need Part 0 filled, and that is itself two workflows:

WorkflowSteps
wf-far-mirror.yamlinit → registry target → registry bom → registry replicate --target generic → registry verify
wf-flp-vsi.yamlinit → flp up → (optional) flp status → publish the flp-handoff Secret

flp status is gated behind a parameter that defaults to false, because the status service is an optional add-on: asserting on a component that was never configured would fail a run whose real output — the handoff Secret — is perfectly good.

Teardown (CI)

A pod that mirrors the workflow container exactly — same image, same PVC, same env carriers — because anything less has no terraform state to destroy from:

$ ./blueprint-workflows-ci-demo.sh teardown bnkdisco    # bnk down → tgw disconnect → cluster down
$ ./blueprint-workflows-ci-demo.sh teardown bnkconn flp

The workspace filter matters: running all four topologies means two clusters with two different prefixes, and teardown rebuilds each workspace’s config from the current bnk-env — so each must come down under the environment it was built with, one at a time.

Timing — what to expect

PhaseTypical
Services VPC + Harbor (S1, incl. cloud-init)10–20 min
Mirror FAR → Harbor (S2)5–10 min
FLP VSI (S3)3–5 min
cluster up (a real ROKS cluster)45–55 min
bnk up8–12 min
bnk up on an already-installed cluster (2C/2D)1–3 min
cluster down10–20 min

The scripted walkthroughs

Everything above is automated in the repository, and the scripts are the executable form of this appendix:

ScriptCovers
disconnected-cluster-cli-demoPart 0 + topology D from the CLI
disconnected-cluster-ci-demothe same as a two-Workflow pipeline, plus the k3s + Argo VSI build
artifactory-adopt-ci-demotopology D with nothing created at all — an existing gateway, VPC and cluster, images from a JFrog Artifactory the customer already runs, and cert-manager adopted. No config.yaml exists: the workspace is built entirely from docker run -e. This is the shape most customers arrive in.
blueprint-workflows-ci-demoall four topologies as Argo Workflows, plus the mirror and FLP prerequisites

B — Replicating FAR into a JFrog Artifactory

This appendix mirrors the F5 Artifact Repository into a JFrog Artifactory you already run. It does not cover standing one up; that is the standalone Artifactory demo’s job.

Nothing here needs a Kubernetes cluster. registry replicate copies registry-to-registry from wherever roksbnkctl runs, so the mirror can be seeded long before a cluster exists — which is usually the point, since the cluster is air-gapped and the mirror is what makes it installable at all.

Everything below has been run end to end against a self-hosted Artifactory. Where a behaviour is edition-specific it says so, because the free and licensed editions differ in ways that are not obvious until something returns a 400.

What Artifactory has to provide

A local Docker repository. BNK’s artifacts are OCI images and OCI-packaged Helm charts, so the target must be a Docker repository — a local one. A remote repository is a read-through cache of an upstream and cannot be pushed to; a virtual repository accepts a push only if it has a local deployment target, so pointing at the local repository directly is simpler and fails more clearly when permissions are wrong.

Create it in the web console: Administration → Repositories → Add Repository → Local Repository → Docker. Any name works; the examples here use bnk-mirror.

Creating repositories is a console job on the free edition. The REST API answers 400 This REST API is available only in Artifactory Pro, so a repository cannot be created by script on JFrog Container Registry. The web UI works fine — it is the automated paths that are restricted. On Pro or Enterprise the REST call works normally.

A credential that can deploy. An access token is the right shape: it is validated independently of the account password, so rotating the password does not invalidate it. That matters because the same credential is handed to a pipeline.

curl -u admin:'<password>' -X POST \
  -d username=admin -d scope=applied-permissions/user -d expires_in=0 \
  https://artifactory.example.com/artifactory/api/security/token

expires_in=0 issues a non-expiring token. Omitting scope fails with Insufficient scope, which is not obvious from the error alone.

A scoped user needs a licensed edition. Users, groups and permission targets are Pro features; on JCR their REST APIs answer 400 … only in Artifactory Pro and the corresponding screens are absent, so the mirror runs as admin. That suits a single-purpose mirror host and does not suit a shared registry — on Pro, create a user, grant it read and write on the repository through a permission, and issue that user’s token instead.

Pointing roksbnkctl at it

roksbnkctl registry target generic
roksbnkctl registry target generic_host        artifactory.example.com
roksbnkctl registry target generic_repo_prefix bnk-mirror
roksbnkctl registry target generic_username    admin

echo "$ARTIFACTORY_TOKEN" | roksbnkctl registry target generic_password --password-stdin

--password-stdin keeps the token out of shell history, out of ps output, and out of any terminal recording. The password field accepts either an access token or the account password; a token is preferable for the reason above.

The repository name is the only thing that changes for a different repository. roksbnkctl composes references as <generic_host>/<generic_repo_prefix>/<image>, so the name is a path segment — bom, diff, replicate, verify, list, prune and delete are identical whatever it is called.

That composition is Artifactory’s repository-path method. If your instance is configured for the subdomain method instead, put the full subdomain in generic_host and leave generic_repo_prefix empty. Getting this wrong produces a 404 that reads like a bad prefix and sends you looking in the wrong place.

If Artifactory presents a certificate from a private CA, record the CA from the file that issued it:

roksbnkctl registry target generic_ca /etc/pki/ca-trust/source/anchors/corp-root.crt

Recording it from the file rather than capturing it from the host matters: a CA learned over the connection it is meant to authenticate proves nothing. With generic_ca set, replicate never dials the registry to discover trust. A publicly-trusted certificate needs none of this, and replicate will say so — no private CA captured is expected, not a warning to act on.

Doing the replication

roksbnkctl registry bom          # what BNK needs — offline, no registry required
roksbnkctl registry diff         # what is missing from Artifactory right now
roksbnkctl registry replicate    # copy it
roksbnkctl registry verify       # confirm every artifact is present + digest-matched

A successful run ends on two lines worth putting on screen:

✓ mirrored 89 artifacts into artifactory.example.com/bnk-mirror
✓ all 89 BOM artifacts present + digest-matched in the mirror

Gate on verify, not on replicate or diff. replicate reports what it pushed. verify independently re-reads the bill of materials and checks each artifact against the digest it should have, so a partial copy or a tag that moved underneath you is caught.

diff can be wrong when the recorded mirror is stale. A workspace records what it last replicated, and that record can describe a registry that has since been rebuilt or replaced. In that state diff has been observed reporting “mirror is in sync” against an empty registry, while verify correctly reported every artifact missing. If a workspace has been re-pointed at a different registry, trust verify.

Configuration and environment equivalents

Every field is config.yaml state:

registry:
  target: generic
  generic_host: artifactory.example.com
  generic_repo_prefix: bnk-mirror
  generic_username: admin
  generic_password_b64: <base64 of the access token>
  generic_ca_b64: <base64 of the CA PEM>       # only for a private CA

generic_password_b64 is obfuscation, not encryption — it exists so the value survives YAML and does not trip the plaintext-secret check. Treat the file as a secret: chmod 600, never commit it.

For CI, each has an environment override, so no config.yaml need be authored at all:

VariableField
ROKSBNKCTL_GENERIC_HOSTregistry.generic_host
ROKSBNKCTL_GENERIC_REPO_PREFIXregistry.generic_repo_prefix
ROKSBNKCTL_GENERIC_USERNAMEregistry.generic_username
ROKSBNKCTL_GENERIC_PASSWORDregistry.generic_password_b64 (raw; encoded for you)
ROKSBNKCTL_GENERIC_CA_B64registry.generic_ca_b64 (already base64)
ROKSBNKCTL_GENERIC_CA_SHA256registry.generic_ca_sha256

Three more are needed for an unattended run, and are easy to miss because init --non-interactive refuses without them rather than defaulting: ROKSBNKCTL_REGION, ROKSBNKCTL_RESOURCE_GROUP and ROKSBNKCTL_PREFIX.

The FAR pull credential is resolved from Cloud Object Storage, which needs ROKSBNKCTL_COS_INSTANCE, ROKSBNKCTL_COS_BUCKET and ROKSBNKCTL_COS_REGION. The bucket has no usable default — COS names are globally unique, so every account’s is suffixed (bnk-artifacts-<hex>), and the built-in default belongs to someone else. A wrong bucket fails with AccessDenied rather than NotFound, which reads like a bad API key and sends you to check the wrong credential.

The same mirror as an Argo workflow

Everything above is the interactive path: an operator with a workspace, testing a mirror by hand. Unattended, the same binary runs the same verbs in a container with no config.yaml anywhere — every setting arrives as an environment variable, because a CI runner has no shell, no prompts and nowhere to stage a file.

The credentials go in a Secret. Keep the token off the command line here too:

printf '%s' "$ARTIFACTORY_TOKEN" | kubectl -n bnk-ci create secret generic artifactory-mirror \
  --from-literal=host=artifactory.example.com \
  --from-literal=repo=bnk-mirror \
  --from-literal=username=admin \
  --from-file=password=/dev/stdin \
  --from-literal=ibmcloud-api-key="$IBMCLOUD_API_KEY"

The workflow runs the five verbs as pipeline steps and gates on verify:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: bnk-artifactory-mirror-
  namespace: bnk-ci
spec:
  entrypoint: mirror
  serviceAccountName: bnk-runner
  arguments:
    parameters:
      - {name: runner-image,   value: ghcr.io/jgruberf5/roksbnkctl-tools-runner:v1.50.0}
      - {name: bnk-version,    value: 2.3.0-3.2598.3-0.0.170}
      - {name: cos-instance,   value: bnk-supply-chain}
      - {name: cos-bucket,     value: ""}          # account-specific; no default
      - {name: cos-region,     value: us-south}
      - {name: region,         value: us-east}
      - {name: resource-group, value: default}
      - {name: prefix,         value: art-mirror}
  # No onExit teardown: a failed replicate should leave the workspace on the PVC
  # so the next run RESUMES. replicate is idempotent — an artifact already
  # present at the right digest is skipped — so a re-run costs only what did not
  # land, not the whole mirror again.
  volumes:
    - name: work
      persistentVolumeClaim: {claimName: bnk-work}
  templates:
    - name: mirror
      steps:
        - - {name: init,               template: rbk, arguments: {parameters: [{name: cmd, value: "init -w art --non-interactive --override-from-env"}]}}
        - - {name: show-target,        template: rbk, arguments: {parameters: [{name: cmd, value: "-w art registry target"}]}}
        - - {name: registry-bom,       template: rbk, arguments: {parameters: [{name: cmd, value: "-w art registry bom"}]}}
        - - {name: registry-replicate, template: rbk, arguments: {parameters: [{name: cmd, value: "-w art registry replicate --target generic"}]}}
        - - {name: registry-verify,    template: rbk, arguments: {parameters: [{name: cmd, value: "-w art registry verify"}]}}

    - name: rbk
      inputs:
        parameters: [{name: cmd}]
      container:
        image: "{{workflow.parameters.runner-image}}"
        command: [sh, -ec]
        args: ["roksbnkctl {{inputs.parameters.cmd}}"]
        workingDir: /work
        env:
          - {name: ROKSBNKCTL_REGION,           value: "{{workflow.parameters.region}}"}
          - {name: ROKSBNKCTL_RESOURCE_GROUP,   value: "{{workflow.parameters.resource-group}}"}
          - {name: ROKSBNKCTL_PREFIX,           value: "{{workflow.parameters.prefix}}"}
          - {name: ROKSBNKCTL_CLUSTER_CREATE,   value: "false"}
          - {name: ROKSBNKCTL_CLUSTER_NAME,     value: "none"}
          - {name: ROKSBNKCTL_REGISTRY_TARGET,  value: "generic"}
          - {name: ROKSBNKCTL_MANIFEST_VERSION, value: "{{workflow.parameters.bnk-version}}"}
          - {name: ROKSBNKCTL_COS_INSTANCE,     value: "{{workflow.parameters.cos-instance}}"}
          - {name: ROKSBNKCTL_COS_BUCKET,       value: "{{workflow.parameters.cos-bucket}}"}
          - {name: ROKSBNKCTL_COS_REGION,       value: "{{workflow.parameters.cos-region}}"}
          - name: ROKSBNKCTL_GENERIC_HOST
            valueFrom: {secretKeyRef: {name: artifactory-mirror, key: host}}
          - name: ROKSBNKCTL_GENERIC_REPO_PREFIX
            valueFrom: {secretKeyRef: {name: artifactory-mirror, key: repo}}
          - name: ROKSBNKCTL_GENERIC_USERNAME
            valueFrom: {secretKeyRef: {name: artifactory-mirror, key: username}}
          # The RAW token; roksbnkctl base64s it into generic_password_b64 itself.
          - name: ROKSBNKCTL_GENERIC_PASSWORD
            valueFrom: {secretKeyRef: {name: artifactory-mirror, key: password}}
          - name: IBMCLOUD_API_KEY
            valueFrom: {secretKeyRef: {name: artifactory-mirror, key: ibmcloud-api-key, optional: true}}
          # The workspace lives on the PVC so a later run — or registry delete —
          # sees the same mirror record.
          - {name: ROKSBNKCTL_HOME, value: "/work/.roksbnkctl"}
          - {name: HOME,            value: "/home/runner"}
        volumeMounts:
          - {name: work, mountPath: /work}
argo submit -n bnk-ci --wait wf-artifactory-mirror.yaml -p cos-bucket=bnk-artifacts-<hex>

It ends on the same two lines the CLI does, from a step that ran in a container that never saw a config file:

✓ mirrored 89 artifacts into artifactory.example.com/bnk-mirror
✓ all 89 BOM artifacts present + digest-matched in the mirror

Three variables are easy to miss and stop the run before any registry work begins. init --non-interactive requires ROKSBNKCTL_REGION, ROKSBNKCTL_RESOURCE_GROUP and ROKSBNKCTL_PREFIX, and refuses rather than defaulting — the error names them, which is the fastest way to diagnose it. The three ROKSBNKCTL_COS_* variables are needed for registry bom to resolve the FAR credential, and the bucket has no usable default.

A workflow with the same shape ships as scripts/demos/artifactory-mirror-demo/wf-artifactory-mirror.yaml.

When it goes wrong

UNAUTHORIZED on push, docker login works by hand. The account can read but not deploy. Docker surfaces this at the first blob upload, well after the login that is easy to mistake for proof of access.

404 or repository not found. Usually the repository-path/subdomain mismatch above. Confirm which method the instance uses before assuming the prefix is wrong.

x509: certificate signed by unknown authority. generic_ca is unset or does not chain to the certificate actually being served — which, on a load-balanced instance, may not be the one issued to the origin.

Storage. The BOM is 89 artifacts of container images and OCI-packaged charts. An instance with a repository quota, or a filestore near capacity, fails partway through replicate rather than cleanly. Re-run diff to see what landed, then replicate again; it copies only what is missing.

no mirror recorded — nothing to delete. registry delete works from the workspace’s recorded mirror, so it cannot clean a registry it has no record of — after the record has been cleared, or from a different workspace. Remove the artifacts through Artifactory instead.

C — Single-NIC sizing

Sizing figures here come from F5’s BIG-IP Next for Kubernetes 2.3.1 on Red Hat OpenShift on IBM Cloud — Single-NIC Sizing Guide, which derives them from reference testing of BNK 2.3.1/2.3.2 on ROKS. They are reproduced rather than computed: roksbnkctl does not model throughput, and nothing here is our measurement.

Only Small and Medium are ever needed

The CNEInstance’s deploymentSize accepts Tiny, Small, Medium, Large and Max. The reference designs use only Small and Medium. That is the single most useful thing to know about sizing, and it is easy to miss, because F5’s guide names three cluster sizes — Small, Medium and Large — and those names do not mean the same thing as deploymentSize.

The Large cluster runs the Medium BNK profile. It reaches ~31 Gbit/s not by asking BNK for a bigger profile, but by putting nine TMM pods on nine larger nodes, three per availability zone. Capacity comes from pod count and node size, not from a larger deploymentSize. So if you find yourself reaching for deploymentSize: Large, the guide’s answer is almost certainly more nodes instead.

SmallMediumLarge
Worker nodes6 (2 per AZ)6 (2 per AZ)9 (3 per AZ)
Reference-tested flavourcx3d.8x20cx2.16x32cx2.48x96
Recommended flavourbx2.8x32cx2.16x32cx2.48x96
deploymentSizeSmallMediumMedium
TMM pods3 (1 per AZ)3 (1 per AZ)9 (3 per AZ)
CPU free for applications38.8 %60 %82.9 %
L4 ingress per AZ~2.6 Gbit/s~3.5 Gbit/s~10.4 Gbit/s
L4 ingress, cluster~7.8 Gbit/s~10.4 Gbit/s~31 Gbit/s

Two footnotes on the flavours. For Small, the guide recommends bx2.8x32 over the reference-tested cx3d.8x20: the tested flavour works but leaves only ~24 GiB cluster-wide for applications, and the balanced variant buys four times that for a modest price step.

At the other end, the guide reports cx2.8x16 as leaving 0.1 % memory free and not holding the platform. roksbnkctl does not refuse it — you can set it, and the tool will build it. It is recorded here so the number is visible before you choose, not to make the decision for you. A per-node DaemonSet floor of ~2.9 vCPU (OpenShift/Calico, IBM observability, BNK node agents) consumes 36 % of an 8-vCPU node before anything of yours is scheduled, which is what makes the 16 GiB variant tight and why the small flavours generally trade application CPU for cost.

All of this is single-NIC. Multi-NIC is expected to change the picture, because per-pod throughput is bounded by TMM thread count rather than by the worker node’s NIC — so higher-performance nodes become worth adding in a way they are not here. Treat this appendix as describing the single-NIC case only, and do not extrapolate it.

Reference deployments

The figures above are F5’s. This section is ours: one clean deployment per sizing, on ROKS, recorded so a reader can tell what has actually been run from what has only been reproduced from a guide.

Every run below is a fresh install onto a fresh cluster, pinned to the manifest the BNK 2.4 IBM install guide documents — 2.4.0-EA, pulled from repo.f5.com — with containerPlatform: IBM and one namespace for every component.

One trap worth naming: BNK 2.4 is Early Access, and the ordinary production FAR token cannot see it. Both keys live in the same production project, so this is a GA-versus-non-GA content split rather than a prod-versus-test one:

FAR keycan pull
cne-gar-pull (standard production)2.3.0 only — no 2.4 version resolves
non-ga-prod-pull (non-GA production grant)2.4.0-EA

2.4.0, 2.4.0-GA and 2.4.1 do not resolve with either key, so until 2.4 goes GA the non-GA production grant is required. The failure mode is misleading: the wrong key authenticates happily and then reports f5-bigip-k8s-manifest:2.4.0-EA: not found, which reads like a wrong version rather than a wrong credential.

When 2.4 GAs this becomes a two-line change — bnk.manifest_version and the key file. Nothing about the configuration shape depends on it. A run counts as verified only if all of the following hold at once:

  • exactly one f5-* namespace exists, and no f5-utils
  • the CNEManifest version matches the reference build exactly
  • every component CR reports Available=True — CNEInstance, F5Tmm, CSRC, Cwc, DSSM, Coremond, Observer, RabbitMQ
  • CSRC pods are Running and the macvlan-internal NetworkAttachmentDefinition exists — CSRC creates it at runtime, so its absence is the visible symptom of a wrong containerPlatform
  • the licence reports Active
  • no pod anywhere is outside Running/Completed

scripts/sizing/sizing-matrix.sh builds and checks one sizing per invocation.

BaselineSmall clusterMedium clusterLarge cluster
Worker nodes3 (1 per AZ)6 (2 per AZ)6 (2 per AZ)9 (3 per AZ)
Flavourbx2.16x64bx2.8x32cx2.16x32cx2.48x96
deploymentSizeTinyTinyTinyTiny
tmmReplicas1339
Verifiedyesyesyesyes

All four are verified: every component CR reporting Available=True, every TMM pod 5/5 Running, macvlan-internal present, licence Active, and nothing outside Running/Completed. On the Large cluster the nine TMM pods land on nine distinct nodes, three per availability zone, which is the anti-affinity and topology-spread arrangement F5’s reference configuration specifies.

Note the third row. deploymentSize is Tiny everywhere — the column headings are cluster sizes, and the thing that changes between them is the node flavour and tmmReplicas, not the BNK profile. The “Baseline” column is the shape the BNK 2.4 IBM install guide itself describes (a 3-node 16-vCPU cluster); the other three are F5’s sizing-guide cluster shapes.

The configuration for each

Common to all four. manifest_version must be set explicitly — leaving it out installs the 2.3 line, and the first symptom is a CNEInstance validation error minutes into the apply complaining that deploymentSize: Tiny is unsupported, because Tiny does not exist on 2.3.

bnk:
  manifest_version: 2.4.0-EA
  far_auth_local_file: /path/to/non-ga-prod-pull-key.tgz
  subscription_jwt_local_file: /path/to/subscription.jwt
  flo_namespace: f5-bnk
  flo_utils_namespace: f5-bnk      # same name = one namespace for everything
  cneinstance_size: Tiny           # every size below; see the section above

far_auth_local_file and subscription_jwt_local_file are a pair — the no-COS path is gated on both being set. With only one, the tool silently falls back to the COS supply chain and the GA key, which fails much later with a 403 naming a GCP project rather than a config field.

Then the part that differs. Only two settings change between sizes, plus the node flavour:

Baseline — 3 nodes, the shape the BNK 2.4 IBM install guide describes:

cluster: {workers_per_zone: 1, worker_flavor: bx2.16x64}
bnk:     {tmm_replicas: 1}

Small cluster — 6 nodes, ~7.8 Gbit/s L4 ingress:

cluster: {workers_per_zone: 2, worker_flavor: bx2.8x32}
bnk:     {tmm_replicas: 3}

Medium cluster — 6 nodes, ~10.4 Gbit/s:

cluster: {workers_per_zone: 2, worker_flavor: cx2.16x32}
bnk:     {tmm_replicas: 3}

Large cluster — 9 nodes, ~31 Gbit/s:

cluster: {workers_per_zone: 3, worker_flavor: cx2.48x96}
bnk:     {tmm_replicas: 9}

As environment overrides

Every field above has a ROKSBNKCTL_* override, so a blueprint or CI job can select a sizing without editing config.yaml. init --non-interactive --override-from-env builds the workspace from these alone:

# common
export ROKSBNKCTL_MANIFEST_VERSION=2.4.0-EA
export ROKSBNKCTL_FAR_AUTH_LOCAL_FILE=/path/to/non-ga-prod-pull-key.tgz
export ROKSBNKCTL_SUBSCRIPTION_JWT_LOCAL_FILE=/path/to/subscription.jwt
export ROKSBNKCTL_FLO_NAMESPACE=f5-bnk
export ROKSBNKCTL_FLO_UTILS_NAMESPACE=f5-bnk
export ROKSBNKCTL_CNEINSTANCE_SIZE=Tiny

# per sizing — baseline / small / medium / large
export ROKSBNKCTL_WORKERS_PER_ZONE=1  ROKSBNKCTL_WORKER_FLAVOR=bx2.16x64 ROKSBNKCTL_TMM_REPLICAS=1
export ROKSBNKCTL_WORKERS_PER_ZONE=2  ROKSBNKCTL_WORKER_FLAVOR=bx2.8x32  ROKSBNKCTL_TMM_REPLICAS=3
export ROKSBNKCTL_WORKERS_PER_ZONE=2  ROKSBNKCTL_WORKER_FLAVOR=cx2.16x32 ROKSBNKCTL_TMM_REPLICAS=3
export ROKSBNKCTL_WORKERS_PER_ZONE=3  ROKSBNKCTL_WORKER_FLAVOR=cx2.48x96 ROKSBNKCTL_TMM_REPLICAS=9

ROKSBNKCTL_CNEINSTANCE_SIZE is listed for completeness rather than necessity: leaving it unset reaches Tiny on the 2.4 line anyway, and unset is what lets the line default apply. Set it when you want the choice recorded explicitly.

If the cluster shares a transit gateway, ROKSBNKCTL_CLUSTER_VPC_CIDR also matters — two VPCs with overlapping prefixes cannot both be routed, and the gateway blackholes one silently.

deploymentSize stays Tiny; capacity comes from replicas and node size

This is the single most important thing to know about running BNK 2.4 on ROKS, and it is easy to get wrong because F5’s sizing guide names its cluster sizes Small, Medium and Large — the same words deploymentSize uses.

They are not the same setting. The BNK 2.4 IBM install guide uses deploymentSize: "Tiny" with tmmReplicas: 3, and that is what every cluster size below runs. Capacity comes from how many TMM pods and how big the nodes are, never from a larger deploymentSize — which is exactly the point this appendix already makes about the Large cluster running the Medium profile.

deploymentSize above Tiny cannot run on ROKS at all, and the reason is worth recording because the failure is silent. Every size above Tiny requests hugepages, and the amount grows with the profile. Read from the controller-derived F5Tmm CR on a live 2.4 cluster by changing only deploymentSize:

deploymentSizeTMM cpuTMM memoryhugepages-2Mi
Tiny——none
Small11536Mi4 GiB
Medium42Gi8 GiB

The Large cluster runs the Medium profile, as the table above this section records, so it derives the same 8 GiB and is not a separate case.

TMM validates hugepages against the 2 MB page cgroup limit when it starts, and exits if that limit is zero:

<get_mem_info>  ERROR: No memory available based on 2MB page cgroup limit
<init_memory>   ERROR: Could not validate sufficient hugepages
<main>          ERROR: Failed to create cmdline args

It exits code 0, so this does not look like a crash — the pod reports 4/5 Running and the Deployment simply never becomes Available.

A ROKS cluster cannot allocate hugepages. Six approaches were measured on live 4.21 clusters and every one fails:

approachresult
bnk.hugepages (Node Tuning Operator, [bootloader])needs the Machine Config Operator; ROKS has 0 MachineConfigPools
Tuned with [sysctl] vm.nr_hugepagesROKS deletes the CR — created object returned, next read NotFound
privileged pod writing /proc/sys/vm/nr_hugepagesallocates (HugePages_Total: 2048) but kubelet still advertises allocatable: 0
advanced.tmm.resources with hugepages-2Mi: 0TMM schedules, then refuses to start (above)
advanced.tmm.env.TMM_MEMaccepted, but hugepages are still required
ibmcloud ks worker-poolno kernel-argument or sysctl option exists

Note that bnk.hugepages is therefore a no-op on ROKS, even though it is what the scheduling failure recommends. Setting it applies a Tuned CR that ROKS removes, which surfaces as terraform reporting a provider bug.

F5’s 2.3.1 sizing guide describes the reference-tested Small profile as “TMM: 1 thread, 1 vCPU / 1.5 GiB, hugepages disabled”, and its TMM resource table lists CPU and memory only. That configuration cannot be reproduced on the 2.4 build: Small derives a hugepages request, and TMM will not start without one. Whether the 2.3.1 profile still exists in 2.4 is an open question with F5 (issue #203).

Until it is answered, treat the Small, Medium and Large rows above as F5’s published figures rather than as configurations this tool can stand up on ROKS. Tiny is the only verified one, which is consistent with F5’s own reference cluster running Tiny.

Tiny is not in F5’s sizing guide — it is the profile the engineering reference cluster itself runs, recorded here as the smallest configuration known to work end to end, not as a recommendation for production traffic.

Two things worth knowing before you reproduce these

Set bnk.manifest_version explicitly. A workspace that leaves it unset installs the 2.3 line, and the first sign of it is a CNEInstance validation error several minutes into the apply complaining that deploymentSize: Tiny is unsupported — because Tiny does not exist on 2.3. The message names the size, not the version, which sends you looking in the wrong place entirely.

bnk down does not leave a clean cluster. It returns success while leaving the namespace and all of BNK’s CRDs installed. That is harmless when reinstalling the same release line, and actively misleading when changing manifest versions, because the previous line’s CRD schema stays in force. To reinstall from genuinely clean state, delete the BNK namespace and the *.k8s.f5.com CRDs after bnk down and before the next bnk up.

What this looks like for a real deployment

The sizing above assumes you are building the cluster. Most deployments are not like that. The common shape is that a platform team already has the VPC, the transit gateway and the ROKS cluster, built to their own standards, and wants roksbnkctl to install BNK onto it — cluster register rather than cluster up.

In that case the table is a specification to check your existing cluster against, not a thing to provision. Worth confirming before installing: node count and distribution across three AZs, the flavour’s memory variant against the “free for applications” column, and that the cluster is not on cx2.8x16.

Those clusters are also usually disconnected: no route to repo.f5.com, so BNK images come from a mirrored registry and licensing goes through an FLP rather than direct to F5. That combination — adopted cluster, adopted transit gateway, mirrored registry, FLP licensing — is the path most installations will actually take, and it is the one worth exercising hardest.

Coverage note. The five-run validation behind the 2.4 support claim used a cluster this tool created, with connected licensing. The adopted-cluster, disconnected, mirrored-registry, FLP path is supported and has its own demos, but it has not had the same repeated-run treatment. Read the 2.4 support status with that distinction in mind rather than assuming it covers the shape you are most likely to deploy.