Preface

Foreword

Standing up F5 BIG-IP Next for Kubernetes (BNK) on IBM Cloud Red Hat OpenShift (ROKS) used to be a multi-step deployment that hit a different surface at every step. A terraform init/plan/apply against an HCL tree somebody handed you. A manual ibmcloud ks cluster config to pull a kubeconfig. A separate IBM Cloud CLI install with its own apt-source dance. A manual oc adm policy add-scc-to-user privileged to let iperf3 actually run. SSH plumbing, env-var plumbing, kubeconfig plumbing — each one a small thing, and together a half-day of yak-shaving before BNK was even on the cluster.

roksbnkctl collapses that into a single static binary plus four interchangeable execution backends (local, docker, k8s, ssh:<target>) plus an opt-in in-cluster ops pod. One command brings a workspace up; one command tears it down; the connectivity, DNS, and throughput tests run from whichever network vantage the question actually requires. The tool exists because the manual path has too many moving parts for somebody who just wants to evaluate BNK or run a customer demo.

This book is the user-facing documentation for roksbnkctl. It ships alongside the v1.0 binary.

Who this book is for

Four audiences:

• BNK evaluators kicking the tires on F5 BIG-IP Next for Kubernetes who want a low-friction path to a working trial deployment.
• F5 sales engineers (SEs) who need a repeatable demo and proof-of-concept toolchain for customer engagements.
• Customer engineers standing up BNK in their own IBM Cloud account, either for evaluation or as the foundation of a production rollout.
• Contributors who want to extend roksbnkctl — add a backend, add a test suite, ship a new chapter. See Part IX — Contributing.

How to read this book

The book is organised so it can be read either way.

• Linear: Parts I-VII walk from concepts through your first deployment, day-2 operations, and the built-in test suite. New users should read in order.
• Reference: Part VIII (Command reference, Configuration reference, Terraform variable reference, Glossary) is exhaustive and indexed for lookups during day-to-day work. Search at the top of every page also reaches into reference material.

If you have 30 minutes and an IBM Cloud account, skip straight to Chapter 7 — Quick start. It’s the canonical “first cluster up” walkthrough and the rest of the book makes more sense after you’ve seen the happy path end-to-end.

Part IX is for contributors who want to build roksbnkctl from source or extend it.

Prerequisites

This book assumes:

• Basic familiarity with IBM Cloud — you have an account, you know what an API key is, and you’ve used the IBM Cloud console at least once.
• Basic familiarity with Kubernetes — you know what a pod, service, and namespace are; you’ve run kubectl (or oc) before.
• A working terminal on Linux or macOS. Windows is supported for roksbnkctl itself, with documented limitations around interactive SSH (see Chapter 16).

You do not need prior experience with:

• Terraform — roksbnkctl embeds a vetted HCL tree and drives terraform for you. You can ignore the underlying HCL until you want to customise it (Chapter 13).
• OpenShift specifics — the tool treats ROKS as Kubernetes with a thin SCC + project overlay; the few OpenShift-specific gotchas are called out in Chapter 22 and Chapter 26.
• F5 BIG-IP Next — BNK is the thing the book deploys; you don’t need to be a Big-IP engineer to evaluate it. Chapter 1 is the 5-minute “what is this product” primer.

Book conventions

• Code blocks: shell commands use bash syntax highlighting; YAML snippets use yaml; HCL fragments use hcl; sample command output is shown in plain text blocks to distinguish output from input.
• Cross-references: every chapter ends with a “Cross-references” section linking related chapters. Inline links use the form [Chapter 7 — Quick start](./07-quick-start.md) — a chapter number, an em-dash, the chapter title, and the relative path to the chapter source.
• PRD links: design documents under docs/prd/ are linked as full GitHub URLs (e.g. https://github.com/jgruberf5/roksbnkctl/blob/main/docs/prd/03-EXECUTION-BACKENDS.md) so they resolve from the published book at GitHub Pages. The PRDs are the design surface; the book is the user surface — read PRDs only if you’re contributing or want the why behind a design call.
• Forward references to post-v1.0 work: where a feature is explicitly queued for a v1.x release (e.g. terraform --backend k8s, multi-hop SSH ProxyJump), the prose flags it in future tense and points at docs/PLAN.md §“What’s deliberately deferred to post-v1.0” for the roadmap.

Welcome.

What is BIG-IP Next for Kubernetes (BNK)

F5 BIG-IP Next for Kubernetes (BNK) is F5’s containerised, Kubernetes-native re-imagining of the BIG-IP data plane. It runs the BIG-IP Traffic Management Microkernel (TMM) as pods inside a cluster, and exposes its configuration surface through Custom Resources rather than the classic TMSH / iControl REST APIs. The point of BNK is to give Kubernetes workloads the L4 and L7 traffic management features F5 customers already rely on — advanced load balancing, TLS termination, WAF policy, GSLB — without bolting an external appliance onto the cluster’s edge.

This chapter sets the context for the rest of the book. If you already deploy and operate BNK day-to-day you can skim it; if you arrived here knowing generic Kubernetes but new to F5’s product family, read it first.

Where BNK fits in F5’s product family

F5 has historically delivered traffic management as the BIG-IP appliance: a hardened Linux box (physical or virtual) running TMOS, with TMM as the data-plane kernel module. BIG-IP works extremely well at the cluster edge — north-south traffic — but it sits outside the cluster and is configured through its own control surface.

The next-generation lineage is BIG-IP Next: the same TMM data plane refactored to run as a regular Linux process, configurable through declarative APIs instead of imperative TMSH. BIG-IP Next ships in three deployment shapes:

• BIG-IP Next for VMs / Bare Metal — same form factor as classic BIG-IP, modernised control plane.
• BIG-IP Next Service Proxy for Kubernetes (SPK) — telco-focused, for 5G core workloads.
• BIG-IP Next for Kubernetes (BNK) — general-purpose, runs inside any conformant Kubernetes cluster.

BNK is the focus of this book. It is the option you pick when your workloads already live in Kubernetes and you want F5 traffic management without standing up a separate appliance fleet.

What problems BNK solves

A standard Kubernetes cluster ships with a basic Service / Ingress story: kube-proxy iptables rules, a community ingress controller, maybe an external load balancer in front. That covers the common case but falls short when you need:

• Real L7 traffic management for north-south traffic — fine-grained routing, header manipulation, TLS termination with custom cipher suites, mTLS enforcement, advanced HTTP/2 + HTTP/3 handling, WAF policy enforcement at the edge.
• East-west service mesh-style features without a sidecar — connection pooling, circuit breaking, retries, observability for pod-to-pod traffic, applied at a per-namespace or per-workload granularity.
• GSLB-style global traffic management — health-checked DNS responses that send a client to the nearest healthy cluster, integrated with the cluster’s own service health.
• Compliance and regulated workloads — DDoS mitigation, behavioural anomaly detection, audit logging that an enterprise security team will accept.

BNK delivers all of the above as cluster-native primitives. You install it once, and from then on you express traffic management intent through CRDs (F5BigIpCtx, F5IngressTls, F5GslbPool, etc.) committed alongside your application manifests.

The components

BNK isn’t a single binary; it’s a set of cooperating components installed into a cluster. The pieces you’ll see most often:

• TMM (Traffic Management Microkernel) — the data plane. Runs as DaemonSet pods on dedicated worker nodes. Every packet handled by BNK passes through TMM.
• FLO (F5 Lifecycle Operator) — the control-plane operator. Watches BNK CRDs and reconciles them into TMM data-plane configuration. Owns the lifecycle of the TMM pods themselves: image pulls, version upgrades, rolling restarts.
• CIS (Container Ingress Services) — Kubernetes-native ingress controller piece. Watches Ingress and BNK ingress CRDs, programmes TMM to terminate the corresponding traffic.
• CNE Instance — Cloud-Native Edge configuration, the umbrella resource that ties a BNK install to its tenant context.
• Cert-Manager — not strictly an F5 component, but a hard dependency. BNK uses cert-manager to mint and rotate the certificates TMM presents to clients.

Deeper chapters reference these names; you don’t need to memorise them now. The thing to take away is that BNK is an operator + DaemonSet pattern: the operator (FLO) reconciles your declarative intent into running data-plane pods (TMM).

Where BNK runs

BNK runs on a conformant Kubernetes cluster. F5 publishes a support matrix — read it for definitive answers — but in practice you’ll see BNK deployed on:

• Managed Kubernetes: ROKS (IBM Cloud’s managed OpenShift), OpenShift Dedicated, EKS, AKS, GKE.
• Self-managed OpenShift on bare metal or VMs.
• Upstream Kubernetes in private clouds, with an LB provider that BNK can integrate with.

This book targets ROKS specifically. The next chapter explains why. The decisions and patterns documented here will translate to other Kubernetes flavours, but the bundled Terraform that roksbnkctl ships only knows how to provision ROKS.

North-south and east-west, in one install

It’s worth calling out explicitly: BNK is not “just an ingress controller” and it’s not “just a service mesh data plane”. It’s both, in the same install:

• North-south (client outside the cluster talking to a workload inside) — BNK fronts a LoadBalancer-typed service, terminates TLS, applies WAF policy, routes to backend pods. Replaces the role a hardware BIG-IP or community ingress controller would play.
• East-west (pod-to-pod or namespace-to-namespace inside the cluster) — BNK can be inserted into the path with no application sidecar, providing per-workload connection pooling, retries, and observability.

A single BNK install can handle both at once. Customer architectures often start with the north-south story (the obvious replacement for an existing BIG-IP appliance), then expand into east-west as the team gets comfortable with the operator-driven configuration model.

Pointer to F5’s official docs

Everything in this chapter is intentionally a sketch — enough to make the rest of this book legible. For definitive and up-to-date product information, including the full CRD reference, version compatibility matrix, sizing guidance, and license model, see F5’s official BNK documentation: https://clouddocs.f5.com/bigip-next/latest/.

The rest of this book focuses on deploying BNK with roksbnkctl and validating that the deployment works end-to-end. It does not duplicate F5’s product documentation; it complements it.

For an at-a-glance view of how roksbnkctl’s components fit together — the four execution backends, the cluster, the jumphost, the IBM Cloud control plane — see the architecture diagram at the top of Chapter 17 — Execution backends. For the happy-path lifecycle from one command to the next, see Chapter 7 — Quick start.

Why ROKS (Red Hat OpenShift on IBM Cloud)

This book and the roksbnkctl tool target ROKS — IBM Cloud’s managed Red Hat OpenShift offering — specifically. Other Kubernetes flavours can run BNK, and most of the patterns you’ll learn here translate, but the bundled Terraform that roksbnkctl ships only knows how to provision a ROKS cluster.

This chapter explains the rationale behind that choice. If you’re already using ROKS, you can skim this. If you’re evaluating whether ROKS is the right substrate for your BNK trial, read in full.

What ROKS is

ROKS is short for Red Hat OpenShift on IBM Cloud. It’s IBM Cloud’s managed-OpenShift service: you ask IBM for a cluster, IBM provisions the masters, etcd, the OpenShift control plane, and a pool of worker nodes; you get a kubeconfig and start deploying.

ROKS clusters are real OpenShift. They run the same Operator Lifecycle Manager (OLM), the same oc CLI, the same SecurityContextConstraints (SCC) model, the same routes-and-services machinery you’d find on any OpenShift install. The only thing IBM has done differently is take responsibility for keeping the control plane and the underlying infrastructure healthy.

What IBM manages, what you manage

The boundary between “IBM’s responsibility” and “your responsibility” is the principal value proposition of any managed Kubernetes service. For ROKS the line falls roughly here:

The thing to internalise: with ROKS you do not rack hardware, install RHEL, run openshift-install, manage etcd backups, or chase CVE patches across a worker fleet. IBM does all of that. You start at “I have an OpenShift cluster” and go from there.

Why managed-OpenShift over self-managed for BNK evaluation

If you want to evaluate BNK quickly, the calculus is straightforward. Self-managed OpenShift is a multi-week lift before you have a cluster:

• Provision the underlying VMs (OpenStack / vSphere / bare metal).
• Run openshift-install and debug whatever doesn’t go right.
• Configure DNS, load balancers, container registry mirrors.
• Stand up monitoring + logging + cert-manager.
• Now you can start thinking about BNK.

ROKS compresses that to one Terraform apply of ~50 minutes. You get back a kubeconfig that authenticates against a real OpenShift cluster, with cert-manager already installable via OLM, and a worker pool of the size and zone topology you specified. From there, the BNK install is the same set of CRDs and Helm charts it would be on any OpenShift cluster.

For a sales-engineering demo or a customer proof-of-concept, “I have a cluster in 50 minutes” beats “I have a cluster in 2 weeks” every time. That trade-off is the reason this book exists in this shape.

Why OpenShift (not just any Kubernetes) for BNK

BNK runs on conformant Kubernetes generally, but it integrates more cleanly with OpenShift specifically because:

• Operator-driven install — BNK is shipped as a set of operators. OpenShift has Operator Lifecycle Manager (OLM) as a first-class citizen, so the install pattern is familiar to OpenShift admins.
• SecurityContextConstraints (SCC) — TMM pods need elevated capabilities (notably NET_ADMIN, raw socket access, hugepages). OpenShift’s SCC model formalises that grant; on upstream Kubernetes you’d be configuring PodSecurityAdmission policies by hand.
• Routes — OpenShift’s Route CRD predates and is more capable than Ingress. BNK can act as an alternate Route implementation, slotting into existing OpenShift application architectures without forcing teams to migrate.
• Image streams + the internal registry — useful for the BNK supply chain (FAR images, license bundles) which can be mirrored once and consumed by many installs.

If you’re already an OpenShift shop, BNK fits naturally. If you’re not, BNK still works but you’ll need to translate this book’s OpenShift-specific examples (SCCs, oc adm policy, Route) to your platform’s equivalents.

What’s out of scope for this book

A short list of Kubernetes flavours this book does not cover:

• EKS / AKS / GKE — BNK runs on these, but roksbnkctl up won’t provision them. You’d use cloud-specific tooling, then deploy BNK on top with the standard Helm charts F5 publishes.
• Self-managed OpenShift on bare metal or VMs — same: no roksbnkctl up. You’d use openshift-install, then deploy BNK.
• K3s, RKE2, microk8s — BNK’s not formally supported on these for production; useful for local dev work but outside this book’s scope.

The patterns from later chapters — workspaces, the --on flag, the connectivity / DNS / throughput tests — would still be useful on any of these, but the lifecycle commands (init, up, down, cluster register) assume ROKS.

What you need before continuing

To follow this book end-to-end you need:

• An IBM Cloud account with billing enabled. The free tier won’t provision a worker pool; you’ll need a Pay-As-You-Go or Subscription account.
• An IBM Cloud API key with permission to create ROKS clusters in the target account.
• A resource group to scope cluster resources to. The default Default group works fine for a single-user evaluation; production deployments tend to use a dedicated group per environment.

The next chapters walk through installation and the quick-start path. By the end of Chapter 7 you’ll have a deployed BNK trial on a fresh ROKS cluster.

What roksbnkctl does (and doesn’t do)

roksbnkctl is a single-binary CLI for deploying and validating F5 BIG-IP Next for Kubernetes (BNK) onto IBM Cloud ROKS. It exists to compress a multi-step deployment — clone the right Terraform, configure it, run terraform, fetch a kubeconfig, install BNK, run smoke tests — into a four-command lifecycle.

This chapter is about scope. What roksbnkctl owns, what it deliberately does not own, and what’s coming in future releases. Read it before you reach for the tool to do something it isn’t trying to do.

The 4-command lifecycle

The everyday user-facing flow is four commands:

roksbnkctl init        # answer a few prompts about region, RG, cluster name
roksbnkctl up          # terraform plan + apply (~50 min for fresh ROKS + BNK)
roksbnkctl test        # connectivity + DNS + throughput against the deployment
roksbnkctl down        # tear it all back down when you're done evaluating

That’s it. From “I have an IBM Cloud API key” to “deployed BNK with a passing throughput test” with no manual terraform apply, no hand-editing kubeconfig paths, no chasing down BNK Helm charts — then a clean tear-down when you’re done so you stop paying for the cluster.

Chapter 7 walks through this end-to-end with sample output.

What roksbnkctl owns

roksbnkctl’s scope is everything between “you have an IBM Cloud API key” and “you have a working BNK install you can run tests against”. Concretely:

• Workspace state — kubectl-style per-environment isolation under ~/.roksbnkctl/<workspace>/. Each workspace has its own config, terraform state, kubeconfig, scratch artefacts. Switch with roksbnkctl ws use <name> or override per-command with -w <name>.
• Terraform-exec orchestration — wraps HashiCorp’s terraform-exec library to drive terraform init/plan/apply/destroy with the right state file, the right TF_DATA_DIR, the right tfvars layering. You don’t run terraform directly; roksbnkctl up does.
• Kubeconfig fetch — after a successful up, fetches the admin kubeconfig from IBM Cloud’s container service API and writes it to ~/.kube/config at mode 0600. Retries on the 404s that happen during cluster propagation lag.
• COS supply chain — the BNK install needs FAR images and JWT licenses staged in IBM Cloud Object Storage. roksbnkctl cos instance/bucket/object handles instance creation, bucket lifecycle, and streaming object I/O (multipart for large files) without making you pip install the IBM COS SDK separately.
• Post-deploy validation — roksbnkctl test runs three suites: HTTP/HTTPS connectivity (built-in net/http, no external curl), DNS resolution (built-in net.Resolver, no external dig), and iperf3 throughput (deploys an iperf3 -s pod into the cluster, runs the client, parses JSON output, tears down).
• Credentials handling — IBM Cloud API key resolution chain: env vars (IBMCLOUD_API_KEY etc.), OS keychain (macOS Keychain / libsecret / Windows Credential Manager via zalando/go-keyring), opt-in base64 in workspace config, interactive prompt as last resort. Plaintext keys in config.yaml are rejected.

If any of those words don’t make sense yet, don’t worry — later chapters cover each in depth.

What roksbnkctl does not try to do

Equally important: the explicit non-goals. roksbnkctl deliberately stays out of these spaces because well-established tools already cover them:

• Not a generic IBM Cloud CLI. That’s ibmcloud. If you want to manage VPCs, IAM policies, classic infrastructure, Watson, or any of the hundred-plus other IBM Cloud services, use ibmcloud. roksbnkctl ibmcloud <args...> exists as a convenience passthrough that loads workspace credentials, but it doesn’t try to replace ibmcloud’s surface.
• Not a generic Kubernetes CLI. That’s kubectl. roksbnkctl kubectl <args...> is again a passthrough that loads the workspace’s kubeconfig; it does not try to be a kubectl re-implementation. (Phase 2 internalises a small subset — roksbnkctl k get/apply/logs/exec/port-forward — so the happy path doesn’t require a host kubectl binary, but that’s targeted convenience, not replacement.)
• Not an OpenShift admin tool. That’s oc. Same story: roksbnkctl oc <args...> passthrough, no attempt to re-implement.
• Not a BNK runtime UI. Once BNK is deployed, you configure it through its CRDs (F5BigIpCtx, F5IngressTls, etc.). roksbnkctl doesn’t ship a TUI / web UI for editing those — it gets you to a deployed BNK and steps out of the way.
• Not a Terraform authoring tool. The HCL lives in this repo’s terraform/ directory and is embedded into the binary at build time. roksbnkctl runs that HCL; it doesn’t help you write more of it. If you fork the HCL, point roksbnkctl at your fork via tf_source: github or tf_source: local.
• Not an arbitrary workload deployer. BNK is the workload. The iperf3 / nginx fixtures used by roksbnkctl test exist only to validate BNK; they’re not a general-purpose deployment surface.

The principle is “do one thing well”. roksbnkctl does BNK-on-ROKS lifecycle and validation. Every other concern is delegated to the right purpose-built tool.

The relationship to bundled HCL

A core design decision worth surfacing: the Terraform that drives the deployment lives in this repo under terraform/, and is embedded into the roksbnkctl binary at build time via Go’s embed package.

This means:

• One install gets you the CLI + a matched HCL pair. No “clone the right tag of the terraform repo separately” step.

• Versioning is unified. A roksbnkctl v1.0 release ships with a specific snapshot of the HCL. Upgrading the binary upgrades the HCL atomically. There’s no skew between “binary version” and “Terraform version”.

• Power users can override. The workspace config has a tf_source: block:

  tf_source:
    type: embedded     # default; uses HCL bundled into the binary
    # type: local
    # path: /path/to/your/terraform
    # type: github
    # repo: yourfork/roksbnkctl-terraform
    # ref: my-branch
  

  tf_source: local is the right setting if you’re iterating on the HCL itself. tf_source: github lets you point at a fork of the terraform repo if you’ve published one separately. The default — embedded — covers the everyday case.

Chapter 13 covers the tfvars layering rules; this is just the elevator pitch for “the HCL ships with the binary”.

What v1.0 ships and what’s queued for v1.x

This book ships with v1.0. The surface it documents:

• kubectl internalisation — roksbnkctl k get/apply/logs/exec/port-forward is a first-class verb talking to the cluster directly via client-go. Host kubectl is informational only; the only required prereq on PATH is terraform.
• Four execution backends — every external tool (ibmcloud, iperf3, terraform) selectable across local | docker | k8s | ssh via --backend. iperf3 runs entirely in-cluster by default; ibmcloud runs in a pinned-version Docker container if you don’t want to install it; any tool proxies through a jumphost via --backend ssh:<target>. Chapter 17 is the user-facing surface; PRD 03 is the design rationale.
• GSLB-aware DNS testing — the DNS probe is miekg/dns-based with multi-vantage support, so you can verify that BNK’s GSLB is returning different answers from different network locations. Chapter 21 covers it.
• Polished book — all 32 chapters, every code example verified, four Mermaid diagrams (architecture / lifecycle / GSLB cross-vantage / backend matrix), per-Part worked examples.

A handful of items are explicitly deferred to v1.x:

• terraform over --backend k8s and --backend ssh (state-file portability design needed).
• Multi-hop SSH ProxyJump for the --on and ssh:<target> paths.
• Windows full TTY (interactive shell on Windows ships as line-buffered; full PTY is a v1.x item).
• Typed OpenShift CRDs (today’s unstructured printer works; richer per-type output is queued).
• Cross-driver cluster-sharing for e2e-test-full.sh (each driver brings up its own cluster today).

See docs/PLAN.md §“What’s deliberately deferred to post-v1.0” for the full roadmap.

Pointers to the next chapters

• Chapter 4 — Installation gets the binary on your machine.
• Chapter 7 — Quick start walks the 4-command lifecycle with sample output.
• Chapter 16 — The –on flag and SSH jumphosts covers running passthrough commands over SSH against an auto-discovered jumphost — useful in customer-firewalled and air-gapped scenarios.

Installation

This chapter gets a roksbnkctl binary onto your machine and verifies it works. Two install paths are covered: build-from-source (native Go, the canonical path until release artefacts ship) and build-with-Docker (no host Go required).

Pre-built binaries are attached to every GitHub Release (Linux, macOS, Windows × amd64, arm64). The book also ships as an offline PDF (roksbnkctl-book-<tag>.pdf) on the same release page. A Homebrew tap is on the v1.x roadmap; until then macOS users grab the binary from the release page or build from source.

Prerequisites

• Linux or macOS for the day-to-day developer experience. Windows compiles cleanly but interactive features (TTY-bound SSH shell, ssh-agent integration) are not first-class on Windows yet.
• Git to clone the repository (only if building from source — not needed if you grab a pre-built binary).
• Go 1.25 or newer if you want a native build. If you don’t have Go (or have an older version), use the Docker-based build or a pre-built release binary.
• Terraform >= 1.5 on PATH at runtime — required for roksbnkctl up / plan / apply / down.
• Helm 3 on PATH at runtime — required during roksbnkctl up. The bundled terraform modules (cert_manager, flo, cne_instance) use null_resource + local-exec provisioners that shell out to helm upgrade --install; without helm the apply errors out with exit status 127 — helm: not found.

The remaining tools (ibmcloud, kubectl, oc, iperf3, docker) are optional and only needed for the corresponding passthrough or backend.

You do not need Docker installed to use roksbnkctl with the default local backend. Docker is required only if you opt in to --backend docker for terraform / ibmcloud. The k8s and ssh backends are alternatives that need neither host Docker nor host Go.

Installing prerequisites

Install paths per platform. terraform and helm are strictly required for v1.0 (helm is invoked by terraform’s local-exec provisioners during roksbnkctl up); the rest are optional, install only what you need.

macOS — Homebrew

brew install terraform               # required
brew install helm                    # required — terraform `local-exec` provisioner shells out to `helm`
brew install --cask ibmcloud-cli     # optional — only for `roksbnkctl ibmcloud …` passthrough
brew install kubectl                 # optional — only for `roksbnkctl kubectl …` passthrough (`roksbnkctl k *` is internalised)
brew install iperf3                  # optional — only for `--backend local`/`--backend ssh:<t>` throughput tests

# oc (Red Hat OpenShift CLI) — optional, only for `roksbnkctl oc …` passthrough.
# No brew formula; install via the Red Hat mirror tarball:
curl -sSL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-mac.tar.gz \
  | sudo tar -xz -C /usr/local/bin oc

If you installed ibmcloud-cli, add the plugins roksbnkctl uses:

ibmcloud plugin install kubernetes-service -f
ibmcloud plugin install cloud-object-storage -f

Linux — Ubuntu / Debian

# terraform — required
wget -qO- https://apt.releases.hashicorp.com/gpg \
  | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
  | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y terraform

# helm 3 — required (terraform's null_resource + local-exec provisioner for cert_manager / flo / cne_instance shells out to `helm`)
curl https://baltocdn.com/helm/signing.asc \
  | sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] \
https://baltocdn.com/helm/stable/debian/ all main" \
  | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update && sudo apt-get install -y helm

# ibmcloud CLI + plugins — optional, for `roksbnkctl ibmcloud …` passthrough with --backend local
curl -fsSL https://clis.cloud.ibm.com/install/linux | sudo sh
ibmcloud plugin install kubernetes-service -f
ibmcloud plugin install cloud-object-storage -f

# kubectl — optional, only for `roksbnkctl kubectl <args>` passthrough (`roksbnkctl k *` is internalised and needs no host install)
sudo snap install kubectl --classic
# or via direct download:
# curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
# chmod +x kubectl && sudo mv kubectl /usr/local/bin/

# oc (Red Hat OpenShift CLI) — optional, only for `roksbnkctl oc <args>` passthrough.
# No apt package; install via the Red Hat mirror tarball:
curl -sSL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz \
  | sudo tar -xz -C /usr/local/bin oc

# iperf3 — optional, only for `--backend local` / `--backend ssh:<t>` throughput tests
sudo apt-get install -y iperf3

Instructions above target Ubuntu and Debian. For other Linux distributions (RHEL, Fedora, Arch, openSUSE, Alpine, …), a quick online search for “install terraform on <your distro>” — and the same pattern for ibmcloud, kubectl, and iperf3 — yields the equivalent commands. HashiCorp ships an RPM repo at https://rpm.releases.hashicorp.com covering RHEL/Fedora, and most distributions package kubectl and iperf3 in their official repos; the IBM Cloud CLI installer at https://clis.cloud.ibm.com/install/linux is a single curl-pipe-sh that works across distros.

Windows — Chocolatey

choco install terraform
choco install kubernetes-helm  # required — terraform local-exec provisioner shells out to `helm`
choco install ibmcloud-cli     # optional
choco install kubernetes-cli   # optional, provides kubectl
choco install openshift-cli    # optional, provides oc (Red Hat OpenShift CLI)
choco install iperf3           # optional

Or via Scoop:

scoop install terraform helm ibmcloud-cli kubernetes-cli openshift-cli iperf3

If choco/scoop don’t carry openshift-cli for your version, grab the Windows tarball from the Red Hat mirror directly:

Invoke-WebRequest -Uri https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-windows.zip -OutFile oc.zip
Expand-Archive oc.zip -DestinationPath "$env:USERPROFILE\bin\"
# then add %USERPROFILE%\bin to your PATH

After installing ibmcloud-cli, add the plugins:

ibmcloud plugin install kubernetes-service -f
ibmcloud plugin install cloud-object-storage -f

Windows TTY-bound SSH features (the roksbnkctl shell --on <target> interactive path) have known limitations on Windows; file-based SSH keys + non-interactive commands work, but ssh-agent named-pipe integration is a v1.x item. See docs/PLAN.md §“What’s deliberately deferred to post-v1.0”.

Path A — native build (requires Go 1.25+)

If go version reports 1.25 or newer, this is the simplest path:

git clone https://github.com/jgruberf5/roksbnkctl.git
cd roksbnkctl

go mod tidy                          # first time only — populates go.sum
make build                           # → bin/roksbnkctl

# Install via roksbnkctl itself (recommended — copies into ~/.local/bin):
./bin/roksbnkctl install

That’s the whole thing. The install subcommand is idempotent and copies the running binary into a directory on your PATH. Default destination is ~/.local/bin/roksbnkctl.

Make targets you’ll use most often:

make build      # go build -ldflags ... -o bin/roksbnkctl ./cmd/roksbnkctl
make test       # go test ./...
make vet        # go vet ./...
make tidy       # go mod tidy
make clean      # rm -rf bin/

If make build fails, the most likely cause is Go too old. The module declares go 1.25.0 in go.mod (forced by transitive deps from the SSH/integration test layers); older versions error out with go: module requires Go 1.25. Either upgrade Go or fall back to the Docker path below.

Path B — Docker-based build (no host Go required)

This path is ideal for sealed CI workstations, custom VM images, or anywhere installing Go on the host is awkward. The official golang:1.25-alpine image has everything needed (Sprint 1 bumped the minimum Go version from 1.23 to 1.25 because of testcontainers-go and gliderlabs/ssh transitive dependencies); the build artefact lands in ./bin/ owned by your host user.

git clone https://github.com/jgruberf5/roksbnkctl.git
cd roksbnkctl

docker run --rm -v "$PWD:/work" -w /work \
  --user "$(id -u):$(id -g)" -e HOME=/tmp \
  golang:1.25-alpine sh -c 'go mod tidy && go build -o bin/roksbnkctl ./cmd/roksbnkctl'

./bin/roksbnkctl install

Anatomy of the docker invocation:

Cross-compile via Docker

Set GOOS / GOARCH env vars in the same docker run to produce binaries for other platforms:

# macOS arm64 (Apple Silicon)
docker run --rm -v "$PWD:/work" -w /work \
  --user "$(id -u):$(id -g)" -e HOME=/tmp \
  -e GOOS=darwin -e GOARCH=arm64 \
  golang:1.25-alpine sh -c 'go mod tidy && go build -o bin/roksbnkctl-darwin-arm64 ./cmd/roksbnkctl'

# Windows amd64 (compile-only; not tested at runtime)
docker run --rm -v "$PWD:/work" -w /work \
  --user "$(id -u):$(id -g)" -e HOME=/tmp \
  -e GOOS=windows -e GOARCH=amd64 \
  golang:1.25-alpine sh -c 'go mod tidy && go build -o bin/roksbnkctl.exe ./cmd/roksbnkctl'

Each binary is statically linked (Alpine + CGO_ENABLED=0 is the cross-compile default) so the produced file has no runtime library dependencies.

The install subcommand

roksbnkctl install [--dir PATH] [--force]

install copies the running binary into a directory on PATH. Defaults:

• Destination: ~/.local/bin/roksbnkctl — this directory is on the default PATH for most modern Linux and macOS user environments and does not require sudo.
• Mode: 0755.
• Idempotent: if the running binary is already at the destination, no-op (no error).

Override the destination with --dir:

./bin/roksbnkctl install --dir ~/bin
sudo ./bin/roksbnkctl install --dir /usr/local/bin

--force overwrites an existing file at the destination. Without it, install refuses if the destination is a different binary.

If ~/.local/bin is not on your PATH, add it. On bash:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
exec $SHELL -l

On zsh, swap ~/.bashrc for ~/.zshrc.

Verifying the install

Two quick checks: version (proves the binary runs) and doctor (proves the runtime environment is set up for actual work).

roksbnkctl version

roksbnkctl version

Sample output:

roksbnkctl v1.0.0 (commit abc1234, built 2026-05-10T14:22:08Z)
Docs: https://jgruberf5.github.io/roksbnkctl/book/

The version string is populated via -ldflags at build time; make build VERSION=v1.0.0 injects an explicit tag. A bare make build produces something like dev (commit abc1234, built ...). The Docs: URL is a compile-time constant (internal/cli/meta.go::DocsURL) — every binary built from this tree points at the same book URL.

roksbnkctl doctor

roksbnkctl doctor

doctor runs the prereq + credentials report. Sample output on a healthy machine looks like this (yours will differ depending on which optional binaries you have installed and whether you’ve initialised a workspace):

✓  terraform         /usr/bin/terraform (Terraform v1.15.2)                                   (required for `roksbnkctl up`)
✓  helm              /usr/local/bin/helm (v3.20.2)                                            (required for `roksbnkctl up`; terraform `local-exec` provisioners shell out to helm)
⚠  iperf3            not on PATH                                                              (needed for `roksbnkctl test throughput`)
✓  kubectl           /usr/local/bin/kubectl (clientVersion:)                                  (optional; `roksbnkctl kubectl` passthrough)
✓  oc                /usr/local/bin/oc (Client Version: 4.21.10)                              (optional; `roksbnkctl oc` passthrough)
✓  ibmcloud          /usr/local/bin/ibmcloud (ibmcloud 2.43.0 ...)                            (optional; `roksbnkctl ibmcloud` passthrough)
✓  kubeconfig        /home/jgruber/.kube/config                                               (needed for cluster-side ops)
✓  workspace         default                                                                  (per-environment config + state)
✓  ibmcloud api key  resolved via OS keychain                                                 (auth for terraform + IBM SDK calls)
✓  ibm cloud auth    OK (account: Main F5 Account)                                            (verifies API key works against IBM IAM)

Each row is <status> <name> <detail> <why we care>. Failures are red ✗ and exit non-zero; warnings are yellow ⚠ and don’t fail the run. terraform and helm are the hard-required checks at v1.0 — the rest are either optional passthroughs or specific to test suites. Chapter 5 walks through what each check is verifying and how to fix common failures.

OS support matrix

“First-class” means the v1.0 acceptance criteria are validated on those platforms; “compile-only” means the binary builds and runs but interactive features (notably TTY-bound SSH) have known limitations and are not part of the v1.0 release gate.

The Windows limitations are tracked in PRD 01 (the SSH client design) and largely come down to golang.org/x/crypto/ssh’s incomplete PTY handling on Windows and the absence of an SSH agent named-pipe protocol. File-based SSH keys work; full PTY and ssh-agent integration on Windows are on the v1.x roadmap (see docs/PLAN.md §“What’s deliberately deferred to post-v1.0”).

Required prerequisites — terraform and helm at v1.0

The v1.0 cluster lifecycle needs two binaries on PATH:

• terraform (>= 1.5) — hard-required for any cluster lifecycle command (up, down, plan, apply).
• helm (3.x) — hard-required during roksbnkctl up. The bundled terraform modules (cert_manager, flo, cne_instance) use null_resource + local-exec provisioners that shell out to helm upgrade --install. Without it, the apply fails with exit status 127 — helm: not found. (A v1.x effort to refactor those modules onto the helm_release terraform resource would eliminate the host requirement; tracked in docs/PLAN.md §“What’s deliberately deferred to post-v1.0”.)

Optional binaries — only needed for the corresponding passthrough or fallback path:

• iperf3 — only needed for --backend local and --backend ssh:<target> throughput modes. The default --backend k8s runs iperf3 entirely in cluster (no host binary needed).
• kubectl / oc — only needed for the roksbnkctl kubectl <args...> / roksbnkctl oc <args...> passthroughs. The everyday verbs (get, apply, describe, delete, logs, exec, port-forward) are internalised under roksbnkctl k and need no host binary.
• ibmcloud — only needed for the roksbnkctl ibmcloud <args...> passthrough on --backend local. The cluster-lifecycle path uses IBM Go SDKs internally and does not shell out to ibmcloud. The docker, k8s, and ssh backends ship their own ibmcloud — no host install needed.
• docker — only needed for --backend docker. Optional; the k8s and ssh backends are alternatives if docker isn’t available.

Run roksbnkctl doctor to see exactly what your environment is missing for the workflow you intend to run.

Updating

git pull && make build is the source-build update mechanism (or re-run the Docker build for the containerised path).

roksbnkctl self update upgrades from a tagged GitHub release. Use it once you’ve installed an initial v1.0 binary:

roksbnkctl self update
# Checks https://github.com/jgruberf5/roksbnkctl/releases/latest, downloads
# the matching asset for your OS+arch, verifies the checksum, swaps the
# binary atomically.

Next

With a working binary on PATH, Chapter 5 — Doctor explains what every doctor check is looking at, Chapter 6 — Workspaces explains the ~/.roksbnkctl/<workspace>/ layout, and Chapter 7 — Quick start walks the 4-command lifecycle end-to-end.

Doctor: checking your environment

roksbnkctl doctor is the prereq + credentials report. It runs in under five seconds, exits non-zero on any hard error, and prints a tabular report that maps one-to-one to the runtime dependencies the rest of the tool reaches for.

This chapter walks every check, explains what each row’s “why we care” blurb means, covers the post-Sprint 2 changes that move kubectl and oc from “needed” to “informational”, and describes the --target SSH probe added in Sprint 1.

What doctor checks

A bare roksbnkctl doctor runs the general checks: tooling on PATH, kubeconfig location, the resolved workspace, and the IBM Cloud authentication chain. Sample output on a healthy machine post-Sprint 2 looks like this:

roksbnkctl doctor
✓  terraform         /usr/bin/terraform (Terraform v1.15.2)                                   (required for `roksbnkctl up`)
✓  helm              /usr/local/bin/helm (v3.20.2)                                            (required for `roksbnkctl up`; terraform `local-exec` shells out to helm)
⚠  iperf3            not on PATH                                                              (needed for `roksbnkctl test throughput`)
✓  kubectl           /usr/local/bin/kubectl (clientVersion:)                                  (internalised in roksbnkctl k *; passthrough still works if installed)
✓  oc                /usr/local/bin/oc (Client Version: 4.21.10)                              (internalised in roksbnkctl k *; passthrough still works if installed)
✓  ibmcloud          /usr/local/bin/ibmcloud (ibmcloud 2.43.0 ...)                            (optional; `roksbnkctl ibmcloud` passthrough)
✓  kubeconfig        /home/jgruber/.kube/config                                               (needed for cluster-side ops)
✓  workspace         default                                                                  (per-environment config + state)
✓  ibmcloud api key  resolved                                                                 (auth for terraform + IBM SDK calls)
✓  ibm cloud auth    OK (account: 1a2b3c..., user: you@example.com)                           (verifies API key works against IBM IAM)

Each row has the same shape:

<status> <name> <detail> <why we care>

• status is one of ✓ (green / OK), ⚠ (yellow / warning), or ✗ (red / error). Skipped checks render as ⚠.
• name is the dependency or capability being checked.
• detail is the resolved value — usually a path, a version line, or an error message.
• why we care is a parenthetical clause naming the roksbnkctl feature that depends on this row.

The BackendName column on the underlying Check struct (internal/doctor/check.go) is reserved for the per-backend probes that land in Sprint 4 (PRD 03). Until then it stays empty for every general check.

Each check explained

terraform — required

One of two hard-required binaries for the roksbnkctl up happy path. roksbnkctl shells out to terraform via terraform-exec for plan/apply/destroy; without it nothing in the cluster lifecycle works.

Pass condition: a binary on PATH, version 1.5 or newer.

Failure mode: not on PATH. Fix: install Terraform from terraform.io, or your distro’s package manager, then re-run doctor.

helm — required

The second hard-required binary, added in v1.0.2. The bundled terraform modules (cert_manager, flo, cne_instance) use null_resource + local-exec provisioners that shell out to helm upgrade --install from inside terraform’s apply phase. Without helm on PATH, the apply fails partway through the cluster lifecycle with:

Error: local-exec provisioner error
Error running command 'helm upgrade --install cert-manager ...':
exit status 127. Output: /bin/sh: 1: helm: not found

Pass condition: a helm (v3.x) binary on PATH. Doctor parses helm version --short for the version detail.

Failure mode: not on PATH. Fix: install Helm 3 from helm.sh/docs/intro/install/, or via your distro’s package manager:

# Linux (Ubuntu/Debian — official Helm apt repo):
curl https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update && sudo apt-get install -y helm

# macOS:
brew install helm

# Windows:
choco install kubernetes-helm

A v1.x effort to refactor the cert_manager / flo / cne_instance modules onto the helm_release terraform resource type (which uses the hashicorp/helm provider’s embedded Helm 3 runtime) would eliminate this host requirement. Tracked in docs/PLAN.md §“What’s deliberately deferred to post-v1.0”.

iperf3 — informational

Used only by roksbnkctl test throughput in its host-iperf3 modes. After Sprint 4 lands the k8s execution backend (PRD 03), iperf3 moves entirely in-cluster and this row goes away for the everyday user.

Failure mode: not on PATH. Fix: install iperf3 if you plan to use the throughput test today; otherwise ignore.

kubectl — informational (Sprint 2 change)

Before Sprint 2: kubectl was an optional warning when missing — useful for the roksbnkctl kubectl passthrough.

After Sprint 2: kubectl is informational. The everyday verbs (get, apply, describe, delete, logs, exec, port-forward) are now native Go via client-go and live under roksbnkctl k. Missing host kubectl no longer disables the happy path; it only disables the roksbnkctl kubectl <args...> passthrough.

If kubectl is on PATH, the row is still ✓ and shows the version line. If it’s missing, the row is informational, not a warning, and the detail explains where the equivalent functionality lives.

oc — informational (Sprint 2 change)

Same story as kubectl — Sprint 2 internalises the OpenShift-relevant verbs (Phase 2.1 adds Project, Route, ImageStream to roksbnkctl k get). Host oc is preserved as an escape hatch; missing oc no longer warns.

ibmcloud — optional

Required only for the roksbnkctl ibmcloud <args...> passthrough. The cluster-lifecycle path uses IBM Go SDKs internally — roksbnkctl up does not shell out to ibmcloud — so you can skip this binary if you don’t need the passthrough.

kubeconfig

Resolves the kubeconfig path via $KUBECONFIG first, then ~/.kube/config. Cluster-side commands (status, logs, every k <verb>) need it.

roksbnkctl up writes the admin kubeconfig at ~/.kube/config (mode 0600) on a fresh apply. If you already have a multi-cluster ~/.kube/config, point $KUBECONFIG at the workspace’s state directory instead:

export KUBECONFIG=~/.roksbnkctl/<workspace>/state/kubeconfig

Failure mode: $KUBECONFIG and ~/.kube/config both missing. Fix: run roksbnkctl kubeconfig --download to fetch the admin kubeconfig from IBM Cloud.

workspace

Reports the resolved workspace name and whether its config.yaml exists.

• ✓ default — the current workspace pointer at ~/.roksbnkctl/config.yaml resolves and the named workspace has a populated config.yaml.
• ⚠ "default" not initialised — the directory may exist (created by roksbnkctl ws new) but config.yaml is empty. Run roksbnkctl init to populate.
• ✗ no config context — the global config can’t be loaded at all.

The one-off -w / --workspace flag overrides which workspace doctor reports against. See Chapter 6 — Workspaces.

ibmcloud api key

Resolves the API key via the chain documented in Chapter 14 — Credentials: env var → OS keychain → workspace config (base64) → TTY prompt.

Pass condition: the chain produces a non-empty key. The key value is never printed — only the source (“resolved”).

Failure mode: IBMCLOUD_API_KEY unset and no keychain entry for workspace "<name>". Fix: either export IBMCLOUD_API_KEY=... for the session, or re-run roksbnkctl init and accept the keychain-save prompt.

ibm cloud auth

Round-trips the resolved key against IBM IAM via the SDK (Verify() call). Confirms the key is not just present but actually authenticates.

Pass condition: IAM accepts the key; the row reports the resolved account and user identity.

Failure modes:

• BXNIM0415E: Provided API key could not be found — the key is malformed or has been deleted in IBM Cloud.
• network is unreachable / i/o timeout — your workstation can’t reach iam.cloud.ibm.com. Common in customer-firewall scenarios; route through a jumphost (Chapter 16) to confirm the key works from inside the customer network.

Common failures and how to fix them

The chapter readers most often land on. Each row maps a real-world symptom to its fix:

If a fix isn’t here, Chapter 26 — Troubleshooting covers the longer tail.

The --target <name> SSH check (Sprint 1)

Sprint 1’s --on jumphost flag introduced an optional second mode for doctor: probe an SSH target before you try to use it.

roksbnkctl doctor --target jumphost

This adds one row per resolved target:

✓  ssh:jumphost      ubuntu@169.45.91.177:22 (TOFU recorded)            (verifies the target is reachable)

The probe:

1. Resolves the target’s host, user, port, and key source from ~/.roksbnkctl/<workspace>/config.yaml.
2. Connects via the internal/remote SSH client.
3. Validates the host key against ~/.roksbnkctl/known_hosts (TOFU prompt on first contact, unless --insecure-host-key).
4. Runs a no-op command (true) to confirm the channel works end-to-end.

Failure modes specific to the SSH probe:

• host key mismatch — the target was rebuilt; edit ~/.roksbnkctl/known_hosts to clear the entry, then re-probe.
• unable to authenticate — the key source resolved but the remote rejected it. Check key_path / key_source in workspace config; if key_source: agent, verify ssh-add -l shows the right key.
• dial tcp: i/o timeout — the host:port is unreachable. Verify with nc -vz <host> 22 from a known-good network.

Pass --target all to probe every target listed in the workspace’s targets: block. Useful in CI when you want a single command that asserts every entry is reachable.

Reading the exit code

doctor exits with:

• 0 — all checks are green or warnings only. Warnings do not fail doctor. The everyday workflow can proceed.
• non-zero — at least one row produced an ✗ error. The first error string is also written to stderr so wrapper scripts can grep it.

This is the contract scripts/e2e-test.sh and the Makefile rely on: a script that runs roksbnkctl doctor && roksbnkctl up --auto will only proceed past doctor if the environment is genuinely ready.

The “warnings don’t fail” rule is deliberate. After Sprint 2, an iperf3 not on PATH warning is informational — the everyday up / test connectivity flow doesn’t need it. Forcing exit-1 on every warning would be too aggressive for the common case.

If you want to gate scripts strictly (e.g. CI workflows that must have iperf3 installed because they run the throughput suite), parse the output rather than relying on the exit code:

if ! roksbnkctl doctor | grep -q '^✓  iperf3'; then
  echo "iperf3 missing — install it before running test throughput" >&2
  exit 1
fi

What doctor is not

A few deliberate non-features worth naming:

• Not a fix-it tool. doctor reports; it never installs, never modifies workspace config, never calls IBM Cloud APIs that mutate state. The IAM verify call is read-only. If doctor could break things, users couldn’t run it freely — and “run doctor” needs to be a safe first move.
• Not a backend probe. Per-backend availability checks (docker daemon reachable, k8s ops pod healthy, ssh target reachable) ship as separate BackendName-tagged rows via doctor --backend <name> (PRD 03). The --target probe was the early prototype of that pattern.
• Not concurrent-safe. The CLI invokes doctor once per command; the side-channel for “why we care” blurbs in internal/doctor/doctor.go doesn’t synchronise. Don’t run two doctors against the same process.

Cross-references

• Chapter 4 — Installation introduces doctor as the post-install verification step.
• Chapter 6 — Workspaces explains the workspace row and the -w override.
• Chapter 14 — Credentials is the deep dive on the ibmcloud api key resolution chain.
• Chapter 16 — The --on flag covers the --target probe’s underlying SSH client.
• Chapter 24 — Day-2 ops is the canonical reference for the internalised k <verb> commands that make kubectl / oc informational.

Workspaces

A workspace is a per-environment bundle of config + state. The shape is modelled on kubectl contexts: you can have many of them, exactly one is “current” at a time, and a -w flag lets you address a specific one for a single command without flipping the pointer.

This chapter covers the on-disk layout, the everyday init / use / list flow, the full roksbnkctl workspaces command tree, the -w / --workspace override, and the “parking-lot” pattern the end-to-end test uses to delete the workspace it’s currently inside.

The on-disk layout

Every workspace lives under ~/.roksbnkctl/<name>/:

~/.roksbnkctl/
  config.yaml                          # global; current_workspace pointer
  known_hosts                          # SSH host keys (shared across workspaces)
  default/                             # workspace "default"
    config.yaml                        # this workspace's inputs
    cluster-outputs.json               # post-apply cluster identity (when present)
    state/                             # BNK trial state
      terraform.tfstate
      terraform.tfvars
      kubeconfig                       # admin kubeconfig (mode 0600)
      tf-source/                       # bundled HCL extracted to disk
      scratch/                         # docker bind-mounts, helm caches
    state-cluster/                     # cluster-phase state (separate tree)
      terraform.tfstate
      cluster-phase-override.tfvars
  prod/                                # workspace "prod"
    config.yaml
    state/
    ...

Three things are worth calling out:

• ~/.roksbnkctl/config.yaml is global — non-secret user-wide preferences plus the current_workspace pointer. It is not a workspace config; the per-workspace files live one level deeper.
• state/ and state-cluster/ are intentionally separate so roksbnkctl cluster up and roksbnkctl up don’t tangle their Terraform state. Most users won’t touch either directly.
• cluster-outputs.json is the persisted identity of the workspace’s ROKS cluster — written by cluster up or cluster register, read by roksbnkctl up so BNK trials don’t have to re-state cluster identity in every tfvars.

Override the base directory with the ROKSBNKCTL_HOME env var. Test fixtures use this; everyday users shouldn’t need it.

terraform.applied.tfvars — what’s deployed right now

v1.4.0 adds a per-phase snapshot of the effective Terraform var-file inputs that produced the workspace’s current state. After every successful terraform apply — roksbnkctl cluster up, roksbnkctl bnk up, or the legacy single-shape roksbnkctl up — roksbnkctl writes a canonical-HCL summary of “what var-files said” to the phase’s state directory. Re-create / audit / handoff workflows that previously needed config.yaml (or memory) now have a file-on-disk record of the inputs.

Where it lives

On ShapeLegacySingle, the file is a union of all sources (since the legacy shape doesn’t separate cluster and trial state) and the header comment records phase=legacy-single so the reader doesn’t mistake it for either a cluster-only or trial-only snapshot. See PRD 07 §“Design” for the format spec.

What it captures

A canonical HCL var-file: one assignment per line, variables sorted alphabetically within each source section. Each section is preceded by a comment line documenting which source contributed the values:

• # === from config.yaml === — vars derived from the workspace’s config.yaml (written to terraform.tfvars on disk).
• # === from terraform.tfvars.user === — the workspace-local user override file. If the file doesn’t exist, the section header is # === from terraform.tfvars.user (missing) === and the body is empty.
• # === from cluster-phase override === — state-cluster/cluster-phase-override.tfvars (cluster-phase snapshots only).

Source-attribution comments matter because the same variable can appear in multiple sources; the “winner” — the value Terraform actually used — is the last section to mention it. The comments let the reader trace why a particular value ended up live.

Lifecycle

• Written after every successful terraform apply. Plan flows don’t write the snapshot — the name terraform.applied.tfvars would mislead if a plan-time write existed.
• Overwritten each apply. If you want history, copy the file aside before re-running up or wire restic / a git commit hook against ~/.roksbnkctl/<workspace>/.
• Untouched by destroy. cluster down / bnk down leave the prior up’s snapshot in place; that’s what was last deployed. The file’s mtime + the absence of Terraform state is the “torn down on <date>” signal.
• Never read by roksbnkctl itself. The snapshot is an output for the user — never an input the tool depends on. Making it an input would create a feedback loop where redacted values get written back as the literal string <redacted>.

Redaction

Exactly one variable is redacted: ibmcloud_api_key. It’s the only var whose value comes from the cred resolver rather than being authored by the user in config.yaml or a tfvars file — so it’s the only value the snapshot would expose that the user didn’t put there themselves. See PRD 04 §“Cred tmpfile-bind-mount pattern” for why the API key isn’t in tfvars in the first place. The redacted line carries an inline comment:

ibmcloud_api_key = "<redacted>"  # source: cred resolver, not persisted

For team-handoff scenarios (a teammate receives this file out-of-band and wants to re-create the workspace): replace the <redacted> value with the teammate’s own API key, or simply remove the ibmcloud_api_key line so the cred resolver supplies it from the teammate’s own environment (keychain, shell env, ~/.bluemix/api_key, etc.) at apply time. Every other line round-trips verbatim.

The file mode is 0600 regardless. The non-redacted contents (workspace identifiers, region, resource group, cluster name, tunable values) aren’t credential-grade secrets, but aren’t world-readable-grade either. Tight permissions are the cheap default.

What it’s not

• Not an input to subsequent applies. The -var-file chain on the next apply is unchanged: config.yaml-derived → terraform.tfvars.user → phase overrides.
• Not a record of Terraform defaults. If variable "foo" { default = "bar" } and the user never set foo, the snapshot omits foo entirely. Capturing defaults would require running terraform output against the variables block — separate concern.
• Not a state-derived value capture. Computed expressions, resource references, locals, and data-source values aren’t var-file inputs and don’t appear. terraform console against the live state dir is the right tool for those.
• Not a TF_VAR_* env capture. roksbnkctl doesn’t set TF_VAR_* today — everything goes via -var-file — so the snapshot covers the complete input surface. A future cycle that starts using TF_VAR_* will need to extend this file.

Safe-to-commit guidance

The file is suitable for git commit alongside config.yaml after the user verifies the redaction matches their threat model. The standard reminder applies: the workspace dir may contain other semi-sensitive material — cluster-outputs.json records the cluster’s crn and admin identity hints; the state/ and state-cluster/ trees include terraform.tfstate (which contains resource IDs, IAM bindings, and any value Terraform’s provider exposed); the kubeconfig files are mode 0600 for a reason. Review the whole workspace dir with the same lens before committing.

roksbnkctl does not touch .gitignore. If you commit the workspace, you commit the workspace; if you don’t, you don’t. The tool stays out of that decision.

Worked example

For a ShapeSplit cluster phase apply, ~/.roksbnkctl/canada-roks/state-cluster/terraform.applied.tfvars looks like:

# Generated by roksbnkctl v1.4.0 at 2026-05-14T10:23:17Z after terraform apply on phase=cluster.
# Re-generated each apply. Do not edit by hand — your changes will be overwritten.

# === from config.yaml ===
cluster_name = "canada-roks"
ibmcloud_api_key = "<redacted>"  # source: cred resolver, not persisted
region = "ca-tor"
resource_group_name = "default"

# === from terraform.tfvars.user ===
worker_count = 4

# === from cluster-phase override ===
deploy_bnk = false

Re-applying from this snapshot alone reconstructs the inputs the user wrote; embedded Terraform module defaults are not captured (see §“What it’s not” above for the full list of what’s out of scope).

The header records the binary version and apply timestamp so the reader can correlate the snapshot to a specific roksbnkctl invocation. Alphabetic ordering within each section means re-running apply with identical inputs produces a byte-identical file (idempotency — handy for diffing snapshots across applies).

The everyday workspace routine

The minimum daily routine:

# Initialise (creates ~/.roksbnkctl/<name>/config.yaml; defaults to "default")
roksbnkctl init

# Switch which workspace is "current"
roksbnkctl ws use prod

# See all workspaces and which one is current
roksbnkctl ws list

roksbnkctl init -w <name> is the one-shot path that creates the directory and populates config.yaml interactively. Everything else (ws new, ws use, ws delete) is the deconstructed form for users who want finer-grained control.

The full command tree

roksbnkctl workspaces ...     # canonical name
roksbnkctl ws ...              # alias

ws new <name> — empty skeleton

Creates ~/.roksbnkctl/<name>/ with no config.yaml. Useful when you want the directory to exist (so ws use works) before you run init.

roksbnkctl ws new staging
# ✓ Created workspace "staging" (run `roksbnkctl init -w staging` to configure)

Most users skip this and use roksbnkctl init -w staging directly, which does both steps in one go.

ws use <name> — switch current

Sets the current_workspace pointer in ~/.roksbnkctl/config.yaml:

roksbnkctl ws use prod
# ✓ Current workspace: prod

roksbnkctl ws current
# prod

Refuses to point at a non-existent workspace. The pointer is the only thing that changes — workspace state stays put.

ws current — print the pointer

roksbnkctl ws current
# default

Prints the current workspace name on stdout. If no pointer is set, prints a hint like “no current workspace; run roksbnkctl ws use <name> or roksbnkctl init” to stderr and exits 0 with empty stdout — so WS=$(roksbnkctl ws current) produces an empty string in scripts rather than spurious output.

ws list — table view

roksbnkctl ws list
NAME      CURRENT  REGION    CLUSTER          TF SOURCE
default   *        us-south  bnk-quickstart   embedded@v1.0.0
prod               eu-de     bnk-prod         embedded@v1.0.0
staging            us-south  bnk-staging      local:./terraform

The * marker on CURRENT highlights the active workspace. Other columns reflect each workspace’s config.yaml. Rows where config.yaml is missing or unparseable still show the name, with the other columns blank — the list never errors out because of one corrupt workspace.

ws delete <name> [--force]

Removes the workspace directory and the OS-keychain entry for its API key. Two safety rails:

1. Refuses to delete the current workspace. You’d be left with a dangling current_workspace pointer, so delete errors out with: cannot delete current workspace "foo"; switch first: roksbnkctl ws use <other>.
2. Refuses if Terraform state lists provisioned resources (unless --force). Catches the foot-gun where you forget to run roksbnkctl down first.

roksbnkctl ws delete staging
# Delete workspace "staging"? [y/N]: y
# ✓ Deleted workspace "staging"

# Refused — state still has resources
roksbnkctl ws delete prod
# Error: terraform state lists 77 resources; run `roksbnkctl down` first or pass --force

# I really mean it
roksbnkctl ws delete prod --force
# ✓ Deleted workspace "prod"

--force skips both the prompt and the state-non-empty check. Use it sparingly — there’s no “undo” for rm -rf ~/.roksbnkctl/<name>/.

The current-workspace pointer

The pointer lives at ~/.roksbnkctl/config.yaml:

current_workspace: prod

Every command that doesn’t pass -w reads this pointer. roksbnkctl init writes it on first run (so the very first init makes default current automatically). ws use rewrites it. Nothing else touches it.

If the pointer references a workspace that doesn’t exist (e.g. someone rm -rf’d the directory by hand), roksbnkctl errors out with a clear message: workspace "prod" referenced by current_workspace does not exist; run roksbnkctl ws use <other>.

-w / --workspace for one-off overrides

Every command accepts -w <name> to override the current pointer for a single invocation:

# Doctor against "prod" without flipping the global pointer
roksbnkctl -w prod doctor

# Run init for a new workspace called "staging"
roksbnkctl init -w staging

# Get pods from the "default" cluster while currently on "prod"
roksbnkctl -w default k get pods -A

Use this when:

• You’re scripting against multiple workspaces in a single run (CI runner that exercises default + e2e-cleanup back-to-back).
• You want to run a one-off command against a different environment without losing your current context.
• You’re testing a fresh workspace before promoting it to current.

The flag only affects the running command — the pointer in ~/.roksbnkctl/config.yaml is unchanged. After the command exits, the next bare roksbnkctl reads the original pointer.

The parking-lot pattern

A subtle gotcha: ws delete refuses to remove the current workspace, but the end-to-end test suite needs to clean itself up after running against the default workspace.

The fix is the parking-lot pattern: have a throwaway workspace that exists only to be the “current” pointer while you delete other workspaces.

# End-to-end test cleanup (e2e-test.sh: Phase D destroys; Phase H runs the parking-lot dance below)

# Run the destroy against "default" (still current at this point)
roksbnkctl down --auto

# Park the pointer somewhere harmless
roksbnkctl ws new e2e-cleanup
roksbnkctl ws use e2e-cleanup

# Now we can drop the original workspace — it's no longer current
roksbnkctl ws delete default --force

# Optional: remove the parking lot too, by parking somewhere else first
roksbnkctl ws new tmp-park
roksbnkctl ws use tmp-park
roksbnkctl ws delete e2e-cleanup --force
roksbnkctl ws delete tmp-park --force   # leaves no current pointer

The pattern works because current_workspace only matters for commands that read workspace config. Once the pointer points elsewhere, the original workspace is just a directory and delete is happy to remove it.

If you want to delete every workspace including the parking lot, the last delete will leave you with an empty current_workspace. The next roksbnkctl init will populate it again with default.

Using a workspace’s environment in your shell

roksbnkctl shell drops you into a subshell with KUBECONFIG, IBMCLOUD_API_KEY, IC_API_KEY, and IBMCLOUD_REGION pre-loaded from the current workspace:

roksbnkctl shell
# (now in a subshell)
echo $KUBECONFIG
# /home/you/.roksbnkctl/default/state/kubeconfig
exit
# (back to the parent shell)

Same for -w:

roksbnkctl -w prod shell

Useful when you want to run host kubectl / host oc / arbitrary tools with the workspace context loaded. The Sprint 2 internalised verbs (roksbnkctl k get, etc.) read the same context automatically — you don’t need to be in a subshell to use them.

Common workspace patterns

A handful of patterns that come up in practice:

Workspaces are cheap. If a flow benefits from isolation, make a new one rather than fighting with --var-file overrides on the existing one.

Forward-link to Chapter 12

This chapter covers the workspace-as-a-unit: how to create, switch, list, delete. The schema of the per-workspace config.yaml itself — every field, default, valid range — is Chapter 12 — Workspace config.

Quick start: from API key to deployed BNK

This chapter walks the 4-command lifecycle (init → up → test → down) end-to-end. By the time you reach the bottom you’ll have a deployed BNK trial on a fresh ROKS cluster, a passing connectivity test, and a clean tear-down command ready when you’re done.

The lifecycle, at a glance:

sequenceDiagram
    autonumber
    actor User
    participant CLI as roksbnkctl
    participant TF as terraform-exec (embedded HCL)
    participant IBM as IBM Cloud API
    participant K8s as ROKS cluster + BNK
    User->>CLI: roksbnkctl init
    CLI->>CLI: write workspace config.yaml
    CLI-->>User: workspace ready
    User->>CLI: roksbnkctl up --auto
    CLI->>TF: terraform plan + apply
    TF->>IBM: provision cluster, VPC, jumphost
    IBM-->>TF: 77 resources created
    TF->>K8s: helm install cert-manager + flo + BNK
    K8s-->>CLI: cluster + BNK up
    CLI-->>User: kubeconfig saved, jumphost target auto-populated
    User->>CLI: roksbnkctl test
    CLI->>K8s: connectivity + DNS + (optional) throughput
    K8s-->>CLI: pass/fail per check
    CLI-->>User: green
    User->>CLI: roksbnkctl down --auto
    CLI->>TF: terraform destroy
    TF->>IBM: 77 resources destroyed
    CLI-->>User: workspace state retained

The walkthrough assumes:

• You have a roksbnkctl binary on PATH (Chapter 4).
• You have an IBM Cloud API key for an account with permission to create ROKS clusters.
• terraform >= 1.5 is on PATH and roksbnkctl doctor looks healthy (Chapter 5).

If roksbnkctl doctor is not green for terraform and IBMCLOUD_API_KEY resolves, fix those first — nothing below will work otherwise.

Note. The output blocks below are illustrative — version strings, cluster IDs, IPs, and timing all vary between runs. The shape of each step is what to look for.

Step 1 — set the API key

The cleanest way to make roksbnkctl see your API key is the IBMCLOUD_API_KEY environment variable. roksbnkctl init will offer to save it to your OS keychain afterwards, so you only paste it once.

export IBMCLOUD_API_KEY="ibmcloud-api-key-value-here"

If you’d rather not export it in your shell, roksbnkctl init will prompt for it on a TTY and offer the same keychain-save afterwards. See Chapter 14 for the full resolution chain.

Step 2 — roksbnkctl init

Initialises a workspace under ~/.roksbnkctl/default/ (or under <name>/ if you pass -w <name>). Verifies the API key against IBM IAM, resolves the resource group, and writes config.yaml.

roksbnkctl init

Sample interactive session:

roksbnkctl init
→ Verifying IBMCLOUD_API_KEY against IBM IAM ... ok (account: 1a2b3c..., user: you@example.com)
? Workspace name (default):
? Region (us-south):
? Resource group (Default):
→ Resolving resource group "Default" ... ok (id: ...)
? Cluster name (bnk-quickstart):
? OpenShift version (4.14_openshift):
? Worker zone (us-south-1):
? Worker count (2):
? Save IBMCLOUD_API_KEY to OS keychain for this workspace? (y/N): y
→ Saved to keychain (service: roksbnkctl, account: default/ibmcloud_api_key)
✓ Wrote ~/.roksbnkctl/default/config.yaml

What just happened:

• A workspace called default now exists at ~/.roksbnkctl/default/.
• config.yaml records the region, resource group, cluster name, OpenShift version, worker pool sizing, and BNK component defaults.
• The API key is saved to your OS keychain (macOS Keychain, libsecret on Linux, or Windows Credential Manager) under service roksbnkctl. Subsequent runs resolve it from there without prompting.

You can re-run roksbnkctl init to update workspace settings; existing values become the prompt defaults.

Step 3 — roksbnkctl up --auto

The deployment. Runs terraform plan, runs terraform apply, fetches the admin kubeconfig from IBM Cloud, writes it to ~/.kube/config at mode 0600. The --auto flag skips the plan-and-confirm gate; without it up shows the plan and asks “apply? [y/N]” before continuing.

roksbnkctl up --auto

Sample output (heavily abridged — a real run is ~50 minutes and prints terraform’s full plan + apply log):

roksbnkctl up --auto
→ Resolving terraform source ... embedded (v1.0.0)
→ Extracting bundled HCL to ~/.roksbnkctl/default/state/tf-source/embedded-terraform/
→ Pre-creating kubeconfig + scratch directories
→ Rendering auto-tfvars from config.yaml ... ok
→ terraform init -reconfigure
  Initializing provider plugins... done.
→ terraform apply (auto-approved)
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Creating...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [10m elapsed]
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [20m elapsed]
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [30m elapsed]
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Creation complete after 38m12s
  module.cert_manager.helm_release.cert_manager: Creation complete after 2m11s
  module.flo.helm_release.flo: Creation complete after 4m02s
  module.cne_instance.kubernetes_manifest.cne_instance: Creation complete after 1m42s
  module.license.helm_release.license: Creation complete after 2m18s
  module.testing.tls_private_key.jumphost_shared_key: Creation complete after 0s
  module.testing.ibm_is_instance.tgw_jumphost: Creation complete after 1m48s

  Apply complete! Resources: 77 added, 0 changed, 0 destroyed.

→ Fetching admin kubeconfig for cluster "<cluster-id>"
✓ Wrote /home/you/.kube/config (chmod 0600)
✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`

What just happened:

• 77 resources were created across ROKS, cert-manager, FLO, CNE Instance, BNK license, and a small testing footprint (the TGW jumphost).
• An admin kubeconfig was fetched directly from IBM Cloud’s container service API (no ibmcloud ks cluster config shell-out) and written at mode 0600.
• A jumphost target was auto-populated in your workspace config from terraform outputs. This makes Chapter 16’s --on jumphost flag work without any further configuration.

The actual elapsed time on a fresh run is dominated by ROKS cluster creation (~30-40 min) and cert-manager + FLO Helm install (~10 min). Re-runs are dramatically faster because terraform’s idempotence skips already-created resources.

Step 4 — roksbnkctl status

Quick sanity check: workspace pointer is right, cluster is reachable, BNK pods are healthy.

roksbnkctl status

Sample output:

Workspace: default
Region:    us-south
RG:        Default (id: ...)
Cluster:   bnk-quickstart (id: <cluster-id>) — Ready
TF source: embedded (v1.0.0)
Last apply: 2026-05-08T14:22:08Z
Nodes:     2/2 Ready
BNK pods:  flo (3/3), cis (1/1), cert-manager (3/3), cne-instance (1/1)

If anything is not green here, jump to Chapter 26 — Troubleshooting.

Step 5 — roksbnkctl test

Run the built-in validation suite. Bare test runs the connectivity + DNS checks (the throughput test takes a few minutes and is opt-in).

roksbnkctl test

Sample output:

roksbnkctl test
→ Suite: connectivity
  ✓ https://www.f5.com (200, 312ms)
  ✓ https://api.openshift.com (200, 88ms)
  ✓ https://us-south.containers.cloud.ibm.com (200, 142ms)
→ Suite: dns
  ✓ www.f5.com → 23.50.149.94 (A, 12ms)
  ✓ api.openshift.com → 35.190.27.231 (A, 18ms)

3 connectivity checks passed; 2 DNS checks passed; 0 failed.

For the throughput suite specifically:

roksbnkctl test throughput --mode east-west

Sample output:

→ Deploying iperf3 server pod into namespace "roksbnkctl-test"
✓ Pod ready (iperf3-server-...)
→ Exposing via ClusterIP service
✓ Service ready (cluster-ip: 172.21.45.108:5201)
→ Running iperf3 -c against the service from local
✓ throughput: 9.41 Gbits/sec (mean over 10s)
→ Tearing down iperf3 fixture
✓ pod and service deleted

The --mode east-west flag uses a ClusterIP service and runs the host iperf3 client through oc port-forward for in-cluster traffic; --mode north-south uses a LoadBalancer for outside-the-cluster traffic. See Chapter 22 for the full design.

The connectivity test uses Go’s built-in net/http — no external curl is shelled out — and similarly DNS uses Go’s net.Resolver. The --insecure flag on test connectivity skips TLS validation if you need to test against self-signed endpoints.

Step 6 — explore (optional)

A few useful follow-ups now that the cluster’s up:

# tail the F5 Lifecycle Operator logs
roksbnkctl logs flo -f

# drop into a shell with the workspace's KUBECONFIG + IBMCLOUD_API_KEY exported
roksbnkctl shell

# run a one-shot kubectl with the workspace context loaded
roksbnkctl kubectl get pods -A

# run ibmcloud through the auto-discovered jumphost
roksbnkctl ibmcloud --on jumphost ks cluster ls

The --on jumphost flag is covered in detail in Chapter 16. It lets you run any of the passthrough commands (exec, shell, kubectl, oc, ibmcloud) from inside the cluster’s network — useful when your workstation is behind a corporate firewall that can’t reach IBM Cloud directly.

Step 7 — roksbnkctl down --auto

Tear it all back down when you’re finished. The teardown is terraform destroy under the hood, with the same resilience to transient IBM API errors as up.

roksbnkctl down --auto

Sample output:

roksbnkctl down --auto
→ terraform destroy (auto-approved)
  module.testing.ibm_is_instance.tgw_jumphost: Destroying...
  ...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Destroying...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still destroying... [5m elapsed]
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Destruction complete after 8m16s

  Destroy complete! Resources: 77 destroyed.

✓ Workspace "default" state retained at ~/.roksbnkctl/default/
  (run `roksbnkctl ws delete default` to remove the workspace dir)

down retains the workspace dir and config so you can up again with the same settings. To remove the workspace entirely:

roksbnkctl ws delete default

This refuses if terraform state still lists resources (use --force to override) and cleans up the keychain entry.

What you just did

In effectively three commands you:

1. Provisioned a fresh ROKS cluster on IBM Cloud.
2. Installed cert-manager, F5 Lifecycle Operator, and a complete BNK trial on top of it.
3. Validated the deployment with HTTP connectivity + DNS resolution + (optionally) throughput tests.
4. Got an auto-discovered jumphost target ready for any --on jumphost follow-ups.

The same flow runs against multiple workspaces, multiple regions, and multiple resource groups — see Chapter 6 for the multi-environment patterns. From here, Chapter 16 covers the --on flag, Chapter 24 covers day-2 operations, and Chapter 26 covers what to do when one of the above steps doesn’t go right.

The cluster phase (cluster up/down)

A roksbnkctl workspace is two phases on top of each other: a durable cluster phase (the ROKS cluster + cluster-shared services that take 30+ minutes to provision) and a short-lived trial phase (the BNK trial that iterates on top in 5-10 minutes). The cluster phase is exposed as its own command pair, roksbnkctl cluster up / roksbnkctl cluster down, so the cluster survives across many BNK trial cycles.

As of v1.1.0, this two-phase shape is the default for every new workspace. A fresh roksbnkctl up provisions the cluster phase first, then the trial phase, against separate state directories. Tearing down only the trial — the common iteration case — uses roksbnkctl bnk down and leaves the cluster intact. The unscoped up / down verbs are now shape-aware composites that delegate to the right phase commands underneath.

Workspaces created against v1.0.x that have cluster modules and trial modules in the same terraform.tfstate (the legacy single-state shape) keep working — roksbnkctl up and down continue to operate against them in-place, byte-for-byte the way they did in v1.0. See § Legacy single-state workspaces at the bottom of the chapter to identify which shape a workspace is.

This chapter covers what each phase deploys, why the two state directories are separate, the deploy_bnk=false override that makes “cluster only” work, the cluster-outputs.json artefact written on success, a worked example, and the legacy single-state shape. The companion BNK-trial chapter, Chapter 10, covers roksbnkctl bnk up / bnk down for the trial layer.

What’s deployed where

The bundled HCL has roughly two halves. The cluster phase owns the durable, cluster-scoped resources:

• The ROKS cluster itself (VPC + subnets + worker pool)
• A transit gateway (so the test jumphost can reach cluster internals)
• The registry COS (Cloud Object Storage) instance — used by the BNK trial as its FAR image / license / schematic store
• cert-manager (Helm release into the cluster)
• The TGW jumphost VM (an Ubuntu VSI in the same VPC, used by --on jumphost)

The trial phase owns the BNK-specific resources:

• F5 Lifecycle Operator (flo) Helm release
• cne_instance Kubernetes manifest
• BNK license + admin certs
• Various cluster-side bits: ServiceAccounts, RoleBindings, Secrets

Two-phase split: cluster up provisions the first list; roksbnkctl up (the trial) provisions the second.

┌─────────────────────────────────────────────────────────┐
│  cluster phase (durable, reused across many trials)     │
│    ROKS cluster + VPC + transit gateway                 │
│    registry COS instance                                │
│    cert-manager (Helm)                                  │
│    TGW jumphost                                         │
├─────────────────────────────────────────────────────────┤
│  trial phase (one trial — destroyed by `roksbnkctl down`)│
│    flo (F5 Lifecycle Operator)                          │
│    cne_instance                                         │
│    license / admin cert / SCC bindings                  │
└─────────────────────────────────────────────────────────┘

The split exists because ROKS clusters take 30-50 minutes to provision and roughly $0.30/hour to run. Re-creating the cluster every time you want to re-test a BNK trial is wasteful; reusing one cluster for many trials cuts iteration time from “an hour” to “a few minutes”.

The two state directories

To keep cluster state and trial state from tangling, roksbnkctl uses separate Terraform state directories:

~/.roksbnkctl/<workspace>/
  state/                   # BNK trial state — written by `roksbnkctl up/down`
    terraform.tfstate
    terraform.tfvars
  state-cluster/           # cluster phase state — written by `roksbnkctl cluster up/down`
    terraform.tfstate
    cluster-phase-override.tfvars

Each phase’s commands read and write only their own state directory. Both phases use the same Terraform source (the bundled HCL) but with different effective tfvars — the trick is the deploy_bnk flag.

The deploy_bnk=false override

The bundled HCL has a top-level deploy_bnk boolean. When true, the BNK trial modules (flo, cne_instance, license) run; when false, they’re skipped and Terraform only provisions the cluster-phase resources.

roksbnkctl cluster up and roksbnkctl cluster down force deploy_bnk = false by writing a small auto-generated tfvars override into the cluster state directory:

# ~/.roksbnkctl/<workspace>/state-cluster/cluster-phase-override.tfvars
# Generated by roksbnkctl. Do not edit by hand.
# Cluster-phase override: BNK trial modules (flo / cne_instance /
# license) are skipped. cert-manager and the testing jumphost still run
# — they're cluster-shared singletons that belong with the cluster.
deploy_bnk = false

This file is layered onto the var-file chain after user-supplied --var-file flags so the override always wins. The user’s terraform.tfvars and --var-file <path> arguments still apply for everything else (region, RG, cluster name, worker count, …) — only deploy_bnk is forced.

roksbnkctl up doesn’t write this override file; its tfvars chain leaves deploy_bnk at the upstream default (true), so the trial modules run.

cluster-outputs.json — the cluster identity record

When roksbnkctl cluster up apply succeeds, it reads the relevant Terraform outputs (cluster name, ID, region, RG, VPC, registry COS) and writes them to a workspace-scoped JSON file:

~/.roksbnkctl/<workspace>/cluster-outputs.json

Sample contents:

{
  "cluster_name": "bnk-quickstart",
  "cluster_id": "cre6h4l20jjsg4kvt3a0",
  "region": "us-south",
  "resource_group_id": "abc123...",
  "vpc_id": "r006-...",
  "registry_cos_crn": "crn:v1:bluemix:public:cloud-object-storage:global:a/...",
  "registry_cos_name": "bnk-quickstart-cos-instance",
  "master_url": "https://c106.us-south.containers.cloud.ibm.com:31415",
  "openshift_version": "4.14_openshift",
  "source": "cluster-up",
  "recorded_at": "2026-05-08T14:22:08Z"
}

The source field discriminates between cluster-up (we created it) and cluster-register (we discovered an existing cluster — see Chapter 9). Subsequent commands read this file to learn the workspace’s cluster identity without hitting IBM APIs.

roksbnkctl cluster down deletes the file as part of its post-destroy cleanup. roksbnkctl cluster show pretty-prints it for human readers:

roksbnkctl cluster show
workspace:        default
source:           cluster-up
recorded_at:      2026-05-08T14:22:08Z

cluster_name:     bnk-quickstart
cluster_id:       cre6h4l20jjsg4kvt3a0
region:           us-south
resource_group:   abc123...
openshift:        4.14_openshift
master_url:       https://c106.us-south.containers.cloud.ibm.com:31415

vpc_id:           r006-...
registry_cos:     bnk-quickstart-cos-instance
registry_cos_crn: crn:v1:bluemix:public:cloud-object-storage:global:a/...

Worked example: cluster up → kubectl get nodes → cluster down

The cluster-only flow, end to end:

Step 1 — roksbnkctl init

If you don’t have a workspace yet, initialise one. This is the same init flow as the trial path; the cluster commands reuse the workspace’s config.

roksbnkctl init

Step 2 — roksbnkctl cluster up --auto

Provisions the cluster phase only:

roksbnkctl cluster up --auto

Sample output (heavily abridged):

→ terraform plan (cluster phase: deploy_bnk=false forced)
→ Layering user tfvars from ~/.roksbnkctl/default/state-cluster/cluster-phase-override.tfvars (overrides config.yaml-derived values)
→ terraform init
→ terraform apply
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Creating...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [10m elapsed]
  ...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Creation complete after 38m12s
  module.cert_manager.helm_release.cert_manager: Creation complete after 2m11s
  module.testing.tls_private_key.jumphost_shared_key: Creation complete after 0s
  module.testing.ibm_is_instance.tgw_jumphost: Creation complete after 1m48s

  Apply complete! Resources: 36 added, 0 changed, 0 destroyed.

✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json
✓ Wrote /home/you/.kube/config (chmod 0600)
✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`

Roughly 36 resources land — the cluster phase is about half the size of a full BNK trial. Time-to-ready is dominated by the ROKS cluster itself; everything else after the cluster comes up is fast.

Step 3 — verify the cluster works

The post-apply admin kubeconfig is fetched automatically (unless --no-kubeconfig). kubectl get nodes confirms reachability:

kubectl get nodes
# NAME           STATUS   ROLES           AGE   VERSION
# 10.243.0.4     Ready    master,worker   3m    v1.28.6+5e1b9a1
# 10.243.64.4    Ready    master,worker   3m    v1.28.6+5e1b9a1

Or, post-Sprint 2, the same thing through the internalised verb:

roksbnkctl k get nodes

roksbnkctl status reports cluster identity + reachability:

roksbnkctl status
Workspace:    default
Region:       us-south
Cluster:      bnk-quickstart  (attach existing)
TF source:    embedded@v1.0.0
Last apply:   2026-05-08 14:22:08 UTC  (3m ago)
Kubeconfig:   /home/you/.kube/config
Cluster:      2/2 nodes ready

Step 4 — (optional) deploy a BNK trial on top

Now that the cluster is up, roksbnkctl up deploys a BNK trial onto it. It reads cluster-outputs.json and reuses the cluster:

roksbnkctl up --auto

See Chapter 10 — Deploying BNK trials for the trial-phase walkthrough. You can run up / down many times against the same cluster — each cycle is ~5 minutes rather than the ~50 minutes of a fresh-cluster run.

Step 5 — roksbnkctl cluster down --auto

Tear down the cluster phase. In v1.1.0 cluster down is strictly scoped: it refuses with a hard error (rather than the v1.0.x warning-but-prompt) on any workspace whose trial state is non-empty, so an out-of-order destroy can’t accidentally orphan BNK resources. Destroy the trial first with roksbnkctl bnk down (or roksbnkctl down for both at once); see Chapter 11 for the full refusal catalogue.

roksbnkctl cluster down --auto

Sample output:

→ terraform destroy (cluster phase)
  module.testing.ibm_is_instance.tgw_jumphost: Destroying...
  module.cert_manager.helm_release.cert_manager: Destroying...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Destroying...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still destroying... [5m elapsed]
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Destruction complete after 8m16s

  Destroy complete! Resources: 36 destroyed.

Post-destroy, cluster-outputs.json is deleted. The workspace directory and its config.yaml survive — re-running cluster up against the same workspace re-creates the cluster with the same name and region.

Why split cluster from trial?

Two-phase is the default because the cost of conflating them is concrete. ROKS clusters take 30-50 minutes to provision and bill at roughly $0.30/hour; a BNK trial on top takes 5-10 minutes. Iterating on the trial — different flo versions, different cne_instance shapes, license bundle revisions — happens far more often than iterating on the cluster underneath. Splitting state means a bnk down / bnk up cycle is a five-minute round-trip instead of an hour.

Three scenarios this shape unlocks:

1. Many BNK trial iterations on one cluster. Run cluster up once, then loop bnk up / bnk down against the same cluster until you’ve covered all the trial permutations. Then cluster down once when you’re finished. This is the headline win of the v1.1.0 surface — see Chapter 10 §“Worked example — iterating on a BNK trial”.

2. Pre-provisioning for a workshop or demo. You want the cluster ready and warm before the demo starts; you’ll deploy the BNK trial live in front of the audience. cluster up the night before; bnk up during the demo.

3. Decoupling cluster lifecycle from trial lifecycle. A long-lived cluster used by multiple team members, where one person owns the cluster phase and others own the BNK trials. Cluster-phase outputs live in cluster-outputs.json; trials read it. Each trial can bnk up / bnk down without affecting the cluster.

For workspaces that just want “create a cluster, deploy BNK on it, test, tear it all down”, the unscoped roksbnkctl up / roksbnkctl down are still the right verbs — in v1.1.0 they’re shape-aware composites that drive the cluster + trial steps in the right order without you having to think about it.

Legacy single-state workspaces

Workspaces created against v1.0.x predate the split. Their terraform.tfstate under ~/.roksbnkctl/<workspace>/state/ contains both the cluster modules (module.roks_cluster, module.cert_manager, module.testing) and the trial modules (module.flo, module.cne_instance, module.license) in one file; state-cluster/ either doesn’t exist or is empty.

roksbnkctl calls this shape LegacySingle and identifies it by walking the trial state’s resource list for cluster-module addresses. To check a workspace’s shape from the outside, look at the state directories:

$ ls ~/.roksbnkctl/<workspace>/
config.yaml  state/  state-cluster/    # split (v1.1.0+) or cluster-only

$ ls ~/.roksbnkctl/<workspace>/
config.yaml  state/                    # legacy single-state, or empty

A state/terraform.tfstate that contains module.roks_cluster and friends is legacy single-state; a state-cluster/terraform.tfstate with content is the split shape.

The v1.1.0 binary handles both shapes:

• Legacy single-state workspaces: roksbnkctl up and roksbnkctl down operate monolithically the way they did in v1.0 — same plan output, same resource count, same byte-for-byte behaviour. The phase-scoped commands (cluster up/down, bnk up/down) refuse with a message pointing you back at the unscoped lifecycle verbs.
• Split workspaces (the new default): up / down are shape-aware composites that delegate to the phase commands underneath; cluster up/down and bnk up/down work directly.

The refusal messages on a legacy workspace look like:

$ roksbnkctl -w canada-roks cluster down
this workspace is legacy single-state; cluster and BNK trial share one state. Use `roksbnkctl down` to tear down both, or migrate the state first

$ roksbnkctl -w canada-roks bnk down
this workspace is legacy single-state; `bnk down` can't isolate the trial phase. Use `roksbnkctl down` to tear down both, or migrate the state first

The refusals print as a single line each — wrapping is a function of your terminal width. Grep against any of the inline punctuation (e.g. \bnk down` can’t isolate`) lands a clean match.

There is no automatic state-migration command in v1.1.0. The refusal text references migration (“or migrate the state first”) because a future roksbnkctl migrate is planned, but until it ships, legacy workspaces stay on the unscoped up / down flow that’s worked for them since v1.0. See Chapter 11 §“The phase-aware decision tree” for the full destruction-time decision matrix.

Cross-references

• Chapter 9 — Registering an existing cluster — the alternative to cluster up when you already have a ROKS cluster you want roksbnkctl to manage.
• Chapter 10 — Deploying BNK trials — roksbnkctl up and the bnk up / bnk down command group; the dispatch matrix; iteration walkthrough.
• Chapter 11 — Tearing down — phase-aware decision matrix and the refusal-message catalogue.
• Chapter 24 — Day-2 ops — roksbnkctl k get / apply / logs for working against the cluster after either phase.

Registering an existing cluster

roksbnkctl cluster register <name> wires roksbnkctl up to a ROKS cluster that already exists in your IBM Cloud account — one you didn’t provision via cluster up. After a successful register, the workspace behaves exactly as if you’d done cluster up: roksbnkctl up deploys BNK trials onto the registered cluster, roksbnkctl down tears those trials down, roksbnkctl status reports the cluster’s identity, and so on.

This chapter covers when registration is the right answer, what input is required vs auto-discovered, the COS naming convention, the cluster-outputs.json write, and a worked example.

When to use this

cluster register is the answer when all of these are true:

• A ROKS cluster already exists in the IBM Cloud account.
• You have IAM access to the cluster’s VPC + container service.
• You want roksbnkctl to deploy BNK trials onto that cluster.
• You don’t want roksbnkctl to own the cluster’s lifecycle (it shouldn’t be terraform destroy-able from your workstation).

Common scenarios:

1. Your team operates the ROKS cluster centrally. A platform team provisioned the cluster via their own Terraform / Pulumi / IBM Cloud Schematics; you just want to deploy BNK trials onto it. Register it; deploy trials; tear them back down. The cluster itself stays under the platform team’s ownership.

2. You’re attaching to an existing demo cluster. A workshop hosts a shared cluster that participants attach to. Each participant registers it in their own workspace and deploys their own trial — trials are isolated by namespace under the same cluster.

3. You provisioned the cluster manually for testing. You created a one-off cluster via ibmcloud ks cluster create vpc-gen2 ... and want to move forward with roksbnkctl rather than re-creating it.

If none of those apply — i.e. you want roksbnkctl to own cluster lifecycle end-to-end — use cluster up instead. Register and cluster up are mutually exclusive per workspace; the second one wins.

Required input vs auto-discovery

cluster register takes one positional argument (the cluster name or ID) and one optional flag (--registry-cos-name).

roksbnkctl cluster register <cluster-name-or-id> [--registry-cos-name <cos-instance-name>]

Everything else is auto-discovered via the IBM SDK:

The cluster lookup goes through the same container-service endpoint ibmcloud ks cluster get uses — no host ibmcloud install required. If the named cluster doesn’t exist in the account, the call returns a clear no cluster named <foo> error rather than a 404 stack trace.

A vpc-gen2 cluster is required. Classic infrastructure clusters return successfully but their vpc_id is empty, and cluster register refuses to write a record without one:

Error: cluster "old-classic" has no VPC — roksbnkctl only supports vpc-gen2 clusters

The COS naming convention

roksbnkctl up needs a Cloud Object Storage instance to act as the registry for FAR images, JWT licenses, and schematic state. cluster register verifies that this COS instance exists at registration time so a later up doesn’t fail mid-apply with a missing-instance error.

Default convention

The bundled HCL falls back to <cluster-name>-cos if the user’s tfvars don’t override roks_cos_instance_name. So cluster register defaults to looking up <cluster-name>-cos:

# Cluster name: "canada-roks" → expects COS instance "canada-roks-cos"
roksbnkctl cluster register canada-roks

Override with --registry-cos-name

If your team set roks_cos_instance_name to something else in their tfvars (or named the COS instance via the IBM Cloud console with a different convention), pass --registry-cos-name <name>:

roksbnkctl cluster register canada-roks \
  --registry-cos-name canada-roks-bnk-registry

The instance name is case-sensitive and must match exactly — Canada-ROKS-COS and canada-roks-cos are different instances.

What if the COS doesn’t exist yet?

cluster register errors out:

Error: registry COS instance "canada-roks-cos" not found in account: ...
  Either run `roksbnkctl cluster up` to create it, or pass --registry-cos-name <name>
  if your tfvars uses a different roks_cos_instance_name

You have two options:

1. Create the COS instance in the IBM Cloud console with the conventional name (<cluster>-cos), then re-run register. The instance can be empty — roksbnkctl up will populate it with the bucket structure it needs on its first apply.

2. Use a different name that already exists in the account, via --registry-cos-name <name>.

Either way, cluster register won’t write cluster-outputs.json until both the cluster and its registry COS instance exist.

The cluster-outputs.json write

On success, cluster register writes ~/.roksbnkctl/<workspace>/cluster-outputs.json — the same file cluster up writes. The contents look identical except for one field:

{
  "cluster_name": "canada-roks",
  "cluster_id": "cre6h4l20jjsg4kvt3a0",
  "region": "ca-tor",
  "resource_group_id": "abc123...",
  "vpc_id": "r038-...",
  "registry_cos_crn": "crn:v1:bluemix:public:cloud-object-storage:global:a/...",
  "registry_cos_name": "canada-roks-cos",
  "master_url": "https://c106.ca-tor.containers.cloud.ibm.com:31415",
  "openshift_version": "4.14_openshift",
  "source": "cluster-register",
  "recorded_at": "2026-05-08T14:22:08Z"
}

The source field is cluster-register (vs cluster-up for self-provisioned clusters). Downstream commands that care about provenance — for example, a future roksbnkctl cluster down would refuse to destroy a cluster-register-sourced cluster — read this field. Subnet IDs (subnet_ids) and transit gateway ID (transit_gateway_id) are left blank for registered clusters; the bundled HCL doesn’t need them when roksbnkctl up runs against a pre-existing cluster.

Worked example: register canada-roks

The full flow for attaching to a hypothetical canada-roks cluster.

Step 1 — create or pick a workspace

roksbnkctl ws new canada
roksbnkctl init -w canada
# (interactive — fill in region as ca-tor; cluster.name = canada-roks)

You can also run cluster register against the current workspace; the -w is just for clarity.

Step 2 — cluster register

roksbnkctl -w canada cluster register canada-roks

Sample output:

→ Looking up cluster "canada-roks"
✓ Cluster canada-roks (cre6h4l20jjsg4kvt3a0) — state: normal, masters: 4.14_openshift
✓ VPC r038-... (resource group prod-rg)
→ Verifying registry COS instance "canada-roks-cos"
✓ COS instance canada-roks-cos (abc-123-def-...)
✓ Wrote ~/.roksbnkctl/canada/cluster-outputs.json

If the COS naming was non-conventional:

roksbnkctl -w canada cluster register canada-roks \
  --registry-cos-name canada-bnk-registry

Step 3 — verify with cluster show

roksbnkctl -w canada cluster show
workspace:        canada
source:           cluster-register
recorded_at:      2026-05-08T14:22:08Z

cluster_name:     canada-roks
cluster_id:       cre6h4l20jjsg4kvt3a0
region:           ca-tor
resource_group:   abc123...
openshift:        4.14_openshift
master_url:       https://c106.ca-tor.containers.cloud.ibm.com:31415

vpc_id:           r038-...
registry_cos:     canada-roks-cos
registry_cos_crn: crn:v1:bluemix:public:cloud-object-storage:global:a/...

Step 4 — fetch the kubeconfig

cluster register does not automatically download the kubeconfig — it’s a metadata-only operation. Grab it explicitly:

roksbnkctl -w canada kubeconfig --download
# → Fetching admin kubeconfig for "canada-roks"
# ✓ Wrote /home/you/.kube/config (12345 bytes)

Step 5 — use the cluster as if you’d done cluster up

From here, the workflow is identical to a self-provisioned cluster:

# Verify reachability
roksbnkctl -w canada k get nodes

# Deploy a BNK trial onto it
roksbnkctl -w canada up --auto

# Tear the trial back down (cluster survives)
roksbnkctl -w canada down --auto

roksbnkctl up reads cluster-outputs.json and uses the cluster identity directly — no need to re-state cluster name/region/RG in the trial’s tfvars.

When register isn’t enough

Some scenarios where cluster register won’t get you over the line:

• The cluster is in a different IBM Cloud account. API keys are account-scoped; you’d need a key for the cluster’s account. cluster register doesn’t cross account boundaries.
• The cluster is private (no public master endpoint). roksbnkctl up needs to apply Helm charts and Kubernetes manifests against the master. If the master is only reachable from inside a VPN, route the apply through --on jumphost (Sprint 1) or wait for the SSH execution backend in Sprint 4.
• The cluster is a classic-infrastructure ROKS (not vpc-gen2). Registration refuses; classic clusters aren’t supported.
• The cluster’s worker pool is too small. BNK trials need at least 2 workers with adequate CPU/memory. The upstream HCL provisions appropriately-sized workers; an existing cluster might not.

For the first three, the cluster simply isn’t a candidate. For the last one, the apply may run but flo / cne_instance will fail to schedule — scale the worker pool first.

Re-registering and unregistering

To re-register with new data (e.g. you renamed the COS instance, or the master URL changed), just run cluster register again — it overwrites cluster-outputs.json in place.

To unregister without destroying anything, delete the file directly:

rm ~/.roksbnkctl/canada/cluster-outputs.json

The workspace’s config.yaml and state/ survive; only the cluster identity record is removed. The next roksbnkctl up will fail with workspace has no cluster-outputs.json until you either re-register or run cluster up.

There’s deliberately no roksbnkctl cluster unregister command. Deleting the JSON is a single-file operation that doesn’t deserve its own subcommand, and the absence of one nudges users toward “destroy the trial first, then deal with the cluster identity” rather than “unregister without thinking about the consequences”.

Cross-references

• Chapter 8 — The cluster phase — the alternative when you want roksbnkctl to provision the cluster.
• Chapter 10 — Deploying BNK trials — what roksbnkctl up does on top of a registered (or cluster up’d) cluster.
• Chapter 25 — COS supply chain management — the COS instance and bucket layout that --registry-cos-name points at.

Deploying BNK trials on top

roksbnkctl up deploys a BNK trial — F5’s Lifecycle Operator, the CNE Instance, license bundles, and the cluster-side glue that makes them work — onto a ROKS cluster that already exists. “Already exists” means either provisioned by cluster up or registered from a pre-existing cluster.

For workspaces where the cluster and the trial are managed as separate phases (the v1.1.0 default — see Chapter 8), the trial layer also gets its own command pair: roksbnkctl bnk up / bnk down. bnk down tears down only the trial; the cluster keeps running, so the next iteration starts in 5-10 minutes instead of an hour. The bnk group is documented in §“The bnk up / bnk down command group” below.

This chapter is the deeper-than-quick-start view of up: what each module does, the ~77-resource shape of a clean apply, the token-rotation observation when you re-run up against an existing cluster, how to read the Terraform plan output, and how the bnk group + the shape-aware composite up / down fit together.

Chapter 7 — Quick start shows the happy path end-to-end with sample output. This chapter goes deeper.

What “deploying BNK” means

A BNK trial is a deliberately small set of Kubernetes resources that share state with a cluster-shared cert-manager and a cluster-scoped registry COS. The components that roksbnkctl up is responsible for landing:

up does not own the cluster, cert-manager, the registry COS, or the jumphost — those are cluster-phase resources. See Chapter 8 for the split.

The 77-resource shape

A clean roksbnkctl up against a fresh cluster lands roughly 77 resources when the cluster phase is bundled in (i.e. cluster up and up were one combined run). Against a pre-existing cluster (cluster up then up), the trial-only count is smaller — roughly the difference, ~41 resources.

The number isn’t load-bearing; it shifts a few resources up or down between upstream HCL releases as the chart adds/removes null_resources and Secrets. Treat “77” as a sanity-check tag, not a contract.

A representative breakdown:

Cluster phase (~36 resources, owned by `cluster up`)
  ROKS cluster + worker pools          ~5
  VPC + subnets + security groups       ~6
  Transit gateway + connections          ~4
  Registry COS instance + bucket          ~3
  cert-manager Helm release               ~2
  TGW jumphost VSI + cloud-init         ~16

Trial phase (~41 resources, owned by `roksbnkctl up`)
  flo Helm release                       ~5
  cne_instance manifest + finalisers     ~4
  license Helm release                  ~10
  Cluster-side SAs / RoleBindings / SCC ~10
  null_resources for token bootstrap    ~12

The null_resources at the bottom of the list are interesting — they’re the ones that re-run on every apply (more on that below).

Apply timing

A clean up against a fresh cluster takes ~50 minutes:

• ROKS cluster provisioning: 30-40 min (the bulk of the wait)
• cert-manager + flo Helm install: ~5 min
• cne_instance reconcile: 1-2 min
• license bootstrap (token generation + activation): 2-3 min
• Cluster-side bits + finalisers: 2-3 min

Against a pre-existing cluster (already-up’d or registered), the trial-only run is 5-10 minutes. Most of that is Helm waiting for flo to stabilise and the license module’s null_resources running.

The token-rotation observation

If you re-run roksbnkctl up against an already-deployed BNK trial, you’ll see ~41 resources re-create or update in-place even though “nothing changed”. This is expected.

The license module rotates admin certificate tokens between runs — the JWT used to authenticate against the BNK control plane is short-lived and re-minted on each apply. A token rotation cascades into ~12 null_resources that exist solely to inject the new token into Helm-managed Secrets:

module.license.null_resource.cncf_admin_cert_token: Refreshing state... [id=8746234876]
module.license.null_resource.cncf_admin_cert_token: Destroying... [id=8746234876]
module.license.null_resource.cncf_admin_cert_token: Destruction complete after 0s
module.license.null_resource.cncf_admin_cert_token: Creating...
module.license.null_resource.cncf_admin_cert_token: Creation complete after 12s [id=9183746183]

That’s why the count of “destroyed + created” can hit ~41 even when no infrastructure-meaningful changes have been made.

The rotation is harmless — running pods aren’t restarted, traffic isn’t interrupted. The new token replaces the old in the relevant Secret; flo notices and updates its in-memory cache. From the BNK trial’s runtime perspective, the second up is a no-op.

If you want to skip the rotation cycle and just check “would this plan change anything significant?”, use roksbnkctl plan rather than up — it shows the plan without applying.

Reading the Terraform plan output

roksbnkctl up runs terraform plan first and prints its output. The plan summary at the end is the most useful part:

Plan: 77 to add, 0 to change, 0 to destroy.

Or, post-rotation:

Plan: 12 to add, 0 to change, 12 to destroy.

The body of the plan shows individual resource changes with one of three markers:

• + create — a new resource. Lines are green in a TTY.
• <= read — a data source the plan read but did not change. Common for data "ibm_resource_group" and similar lookups; effectively informational.
• # destroy — an in-progress destroy of an existing resource. Followed by a + create if it’s being replaced (the null_resource rotation case).
• ~ update in-place — a resource whose attributes are being mutated without re-creation.

The <= data sources are the ones that look like:

data "ibm_resource_group" "default" {
  name = "Default"
  id   = "abc123..." (will be read)
}

These are read-only — Terraform is just resolving the resource group’s ID at plan time so downstream modules can reference it. They show up in every plan, including no-op plans.

# destroy lines without a corresponding + create — i.e. resources actually leaving — should make you stop and read carefully. On a re-run of up, this generally means an upstream HCL change removed a resource. It’s rare but not zero.

When up doesn’t apply (no-op runs)

If the plan reports zero changes, up skips apply and prints:

✓ no changes

But it still does two best-effort post-actions:

1. Fetch the kubeconfig (unless --no-kubeconfig). Useful when the cluster exists but you’ve never grabbed the admin kubeconfig on this workstation.
2. Auto-register the jumphost target. Reads testing_tgw_jumphost_ip and jumphost_shared_key from Terraform outputs and writes a targets:jumphost entry in workspace config. Re-runs are idempotent.

So roksbnkctl up against an unchanged cluster is a useful “re-establish my workstation’s view of this workspace” verb — it can’t hurt anything (no apply runs), and it freshens local artefacts.

The --auto, --no-kubeconfig, --var-file flags

roksbnkctl up [--auto] [--no-kubeconfig] [--var-file <path>]...

--var-file is the canonical way to stage a non-default deploy. For example, deploying a BNK trial with a non-default cne_instance.replicas:

echo 'cne_replicas = 3' > ./more-replicas.tfvars
roksbnkctl up --auto --var-file ./more-replicas.tfvars

The var-file chain is, in order:

1. The auto-generated terraform.tfvars (rendered from config.yaml).
2. ~/.roksbnkctl/<workspace>/terraform.tfvars.user if present.
3. Each --var-file flag, left-to-right.

Later wins on conflict — same as Terraform’s own ordering.

Apply retries on transient errors

ROKS master endpoints take 1-5 minutes to fully propagate after the cluster reaches Ready. The cne_instance, license, and cert-manager modules all curl the master directly; on a fresh cluster, they sometimes race propagation and fail with exit status 7 (curl couldn’t connect) or Connection refused.

roksbnkctl up has built-in retry: up to 3 apply attempts, with a 60-second sleep between attempts, on any of these heuristic patterns:

• exit status 7 (curl couldn’t connect)
• Connection refused / connection refused
• i/o timeout
• no route to host
• network is unreachable
• no such host
• TLS handshake timeout
• failed to dial
• to download the config doesn't exist

If your apply hits one of these, you’ll see:

→ apply attempt 1 hit a transient-looking failure; waiting 60s and retrying...

Terraform’s idempotence means already-created resources are skipped on the retry; only the failed null_resources / data sources re-execute. After 3 attempts, up gives up:

✗ apply still failing after 3 attempts — giving up

At that point, fix the underlying cause (usually wait longer or re-run manually) and try again. The retry is for transient races, not persistent failures.

What happens on success

A successful up does five things in order:

1. Apply complete. Apply complete! Resources: 77 added, 0 changed, 0 destroyed.
2. Fetch the admin kubeconfig from IBM Cloud’s container service API. Written to $KUBECONFIG (or ~/.kube/config) at mode 0600.
3. Auto-register the jumphost target in workspace config (so --on jumphost works without manual config — see Chapter 16).
4. Stamp terraform.tfstate’s mtime. roksbnkctl status reads this as “last apply” timestamp.
5. Exit 0.

The kubeconfig fetch and jumphost registration are best-effort: they log warnings on failure but don’t fail the parent command. up succeeded if Terraform succeeded; the post-apply niceties are conveniences.

The bnk up / bnk down command group

New in v1.1.0. The roksbnkctl bnk group is the trial-only counterpart to roksbnkctl cluster — it operates on the trial state under state/ and leaves the cluster state under state-cluster/ untouched. The whole point is that iterating on a BNK trial no longer costs a 30-minute cluster rebuild: a bnk down / bnk up round-trip is the 5-10 minute trial-apply window, the cluster keeps running underneath.

roksbnkctl bnk up

Deploys the BNK trial against the workspace’s registered cluster.

• If the workspace already has a cluster phase (either from cluster up or from cluster register), bnk up runs the trial apply directly — same plan, same ~41 resources, same 5-10 minute window as the trial half of a full up.
• If the workspace is empty (no cluster registered yet), bnk up offers to bootstrap the cluster phase first with a confirmation prompt, then runs the trial apply. This keeps the new user’s quick-start path one command, even if they typed bnk up instead of up.
• On a legacy single-state workspace, bnk up refuses — there’s no way to isolate the trial phase when the trial and cluster share one state file.

Sample output of the bootstrap-prompt path:

$ roksbnkctl bnk up
No cluster registered for this workspace.
→ Provisioning the cluster phase first (ROKS cluster + transit gateway +
  registry COS + cert-manager + jumphost; ~30 min) before the BNK trial.
Continue? [y/N]: y
→ terraform plan (cluster phase: deploy_bnk=false forced)
...
✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json
→ terraform plan (trial phase)
...
Apply complete! Resources: 41 added, 0 changed, 0 destroyed.

Three prompts fire in the empty-workspace case — one for “do you want to bootstrap the cluster phase,” one for “apply this terraform plan” inside the nested cluster up, and a third when the trial-phase apply prompts. (On a non-empty workspace where bnk up skips the cluster bootstrap, only the latter two fire — and a ShapeClusterOnly/ShapeSplit bnk up is the common iteration case.) For a 30-minute operation we kept the prompts explicit rather than collapsing them. --auto skips all three:

$ roksbnkctl bnk up --auto

roksbnkctl bnk down

Destroys the trial only. The cluster phase keeps running.

• On a split workspace (cluster + trial both present), bnk down runs terraform destroy against the trial state — ~41 resources, the same as the trial half of a full down.
• On an empty or cluster-only workspace, bnk down refuses: there’s no trial to destroy.
• On a legacy single-state workspace, bnk down refuses: the cluster lives in the trial state so a trial-only destroy isn’t possible.

Sample output against a split workspace:

$ roksbnkctl bnk down --auto
→ terraform destroy (trial phase)
  module.license.helm_release.license: Destroying...
  module.cne_instance.kubernetes_manifest.cne: Destroying...
  module.flo.helm_release.flo: Destroying...
  ...
  Destroy complete! Resources: 41 destroyed.

✓ Trial phase destroyed. Cluster phase ~/.roksbnkctl/default/state-cluster/ is intact.
  Run `roksbnkctl bnk up` to deploy another trial against the same cluster.

The shape dispatch matrix

The unscoped roksbnkctl up / down verbs are now shape-aware composites — they detect the on-disk shape of the workspace and delegate to the right phase commands underneath. The full picture for all four shapes and all six commands:

The user-facing simplification: the unscoped up / down “just work” against every shape (including v1.0.x legacy state). The phase-scoped commands (bnk, cluster) only operate when the shape allows isolation and refuse loudly with an actionable message otherwise. Refusals always point at the resolution — see Chapter 11 §“Refusal messages” for the full catalogue.

The engineering version of this table — with the implementation details, the ShapeUnknown edge cases, and the rationale — lives in PRD 06 §“Dispatch table”.

Worked example — iterating on a BNK trial

The headline workflow the v1.1.0 surface unlocks. You’re testing different cne_instance parameter combinations against a stable cluster.

# Step 1 — one-time cluster provision (~38 minutes)
roksbnkctl cluster up --auto
# → terraform apply (cluster phase: deploy_bnk=false forced)
#   ...
#   Apply complete! Resources: 36 added, 0 changed, 0 destroyed.
# ✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json

# Step 2 — first BNK trial (~7 minutes — trial only, cluster is reused)
roksbnkctl bnk up --auto
# → terraform plan (trial phase)
#   Plan: 41 to add, 0 to change, 0 to destroy.
#   ...
#   Apply complete! Resources: 41 added, 0 changed, 0 destroyed.

# Step 3 — poke at the trial, find something to tune
roksbnkctl k get pods -n f5-bnk
roksbnkctl test connectivity

# Step 4 — destroy just the trial (~3 minutes — cluster persists)
roksbnkctl bnk down --auto
# → terraform destroy (trial phase)
#   Destroy complete! Resources: 41 destroyed.
# ✓ Trial phase destroyed. Cluster phase ~/.roksbnkctl/default/state-cluster/ is intact.

# Step 5 — edit config.yaml (or a --var-file) to change cne_instance settings
$EDITOR ~/.roksbnkctl/default/config.yaml

# Step 6 — second BNK trial against the same cluster (~7 minutes; the 30-minute
#          cluster provision from step 1 does NOT repeat)
roksbnkctl bnk up --auto
# → terraform plan (trial phase)
#   ...
#   Apply complete! Resources: 41 added, 0 changed, 0 destroyed.

The win is in step 6: the cluster persists across the bnk down / bnk up boundary, so the second trial deploy is ~7 minutes instead of the ~50 minutes a full down → up cycle would cost in v1.0.x. Across a day of iteration, that’s the difference between five trial permutations and one.

When you’re done with the whole session:

# Step 7 — tear down the cluster too
roksbnkctl cluster down --auto
# (or `roksbnkctl down` from any starting state — see the dispatch matrix above)

Cross-references

• Chapter 7 — Quick start — happy-path walkthrough end-to-end.
• Chapter 8 — The cluster phase — what cluster up provisions, the two state directories, and how to identify a legacy single-state workspace.
• Chapter 11 — Tearing down — phase-aware decision matrix; full refusal-message catalogue; orphan recovery.
• Chapter 13 — Terraform variables — full reference for what you can override via --var-file.
• Chapter 22 — Throughput testing — once BNK is deployed, validating its data plane.
• Chapter 26 — Troubleshooting — long-tail apply failures (SCC violations, propagation lag, kubeconfig 404s) and their fixes.

Tearing down

roksbnkctl down, roksbnkctl bnk down, and roksbnkctl cluster down are the three destroy verbs — the inverses of up, bnk up, and cluster up respectively. This chapter covers what each one removes, the ordering constraint between them, the refusal messages you’ll hit if you ask for the wrong one, what survives a destroy, the --auto flag for non-interactive runs, and the workspace-cleanup story.

The phase-aware decision tree

Which verb do you want? The shape of your workspace and your intent both matter. Start here:

I want to keep the cluster and just tear down the BNK trial:
    → roksbnkctl bnk down

I want to tear down everything (cluster + trial):
    → roksbnkctl down

I want to tear down only the cluster (no trial currently deployed):
    → roksbnkctl cluster down

I'm on a v1.0.x workspace (cluster + trial in one state):
    → roksbnkctl down       (tears down everything in one shot)
    → see Chapter 8 §"Legacy single-state workspaces" to confirm your shape

Quick shape check: ls ~/.roksbnkctl/<workspace>/ — if you see state-cluster/, you’re on the v1.1.0 split shape; if you see only state/, you’re on legacy single-state.

The big rule, stated up front: destroy in reverse of create. Trial first (bnk down), cluster second (cluster down). The unscoped roksbnkctl down does this ordering for you — on a split workspace it runs the trial destroy first and then the cluster destroy. On a legacy single-state workspace it runs a monolithic destroy (the v1.0.x behaviour, byte-for-byte). Either way you don’t have to think about ordering; down is the safe default.

The phase-scoped commands (bnk down, cluster down) are the precision tools — they let you keep one phase across many cycles of the other. They also refuse loudly if you ask them to do something that would orphan resources or that the shape doesn’t allow. The full refusal catalogue is in §“Refusal messages catalogue” below; the rule of thumb is that the error message always names the verb that would actually work.

The three destroys

There are three teardown verbs matching the three slices of state:

roksbnkctl down — shape-aware composite

The unscoped down is a shape-aware composite in v1.1.0: it detects the on-disk shape of the workspace and dispatches to the right phase destroys in the right order.

roksbnkctl down

This is the safe default — down always does the right thing regardless of shape, and it’s the only verb you can run on a legacy single-state workspace.

roksbnkctl bnk down — destroy the BNK trial only

New in v1.1.0. Tears down everything the trial phase created — the flo Helm release, cne_instance, the license module, cluster-side ServiceAccounts / RoleBindings / SCC bindings, and the null_resources that bootstrap admin tokens — and leaves the cluster running.

roksbnkctl bnk down

What survives:

• The ROKS cluster itself
• cert-manager
• The registry COS instance and its bucket contents (FAR images, license artefacts)
• The TGW jumphost
• All cluster-phase Terraform state under state-cluster/
• cluster-outputs.json (the cluster is still registered)
• The workspace’s config.yaml

Roughly 41 resources destroyed on a clean trial-only bnk down. Time is dominated by Helm’s pre-delete hooks and the cne_instance finaliser unwind — usually 2-5 minutes total.

bnk down refuses on Empty, ClusterOnly, and LegacySingle workspaces — there’s nothing to destroy on the first two, and the trial-only isolation isn’t possible on the third. See §“Refusal messages catalogue” for the exact text.

roksbnkctl cluster down — destroy the cluster phase

Tears down the cluster + cluster-shared services: the ROKS cluster, transit gateway, registry COS instance, cert-manager Helm release, and the TGW jumphost.

roksbnkctl cluster down

What survives:

• The workspace’s config.yaml
• ~/.roksbnkctl/<workspace>/state/ (now empty of resources but the directory persists)
• ~/.roksbnkctl/<workspace>/state-cluster/ Terraform state files (the cluster-side state itself is empty; the directory and terraform.tfstate persist)

Roughly 36 resources destroyed. The ROKS cluster destroy alone is 5-10 minutes; everything else is fast.

The post-destroy cleanup deletes cluster-outputs.json automatically — the workspace no longer has a registered cluster.

Order matters: trial first, then cluster

The upstream HCL’s resource graph requires this ordering. The trial-phase resources have implicit dependencies on cluster-phase resources (they live in the cluster, after all), and Terraform’s destroy graph traverses dependencies in reverse. If the cluster phase tries to destroy first, the trial phase’s resources are still there — finalisers block the destroy of the cluster’s namespaces, the cluster-side SCC bindings reference SCCs that are in the way, and so on.

In v1.1.0 roksbnkctl cluster down enforces this ordering with a hard refusal: if the trial state has any resources in it, cluster down errors out and points you at bnk down (or down) instead. The v1.0.x “warning-but-prompt” behaviour is gone — even --auto won’t bypass the guard, because correctness, not confirmation, is the issue. The full refusal text:

$ roksbnkctl cluster down
BNK trial state exists in this workspace; run `roksbnkctl bnk down` first
(or `roksbnkctl down` to tear down both phases)

So in practice, always destroy the trial before the cluster. The unscoped down does this ordering for you on a split workspace; the phase-scoped pair is bnk down then cluster down.

The clean teardown sequence — split workspace, explicit phase commands:

# 1. Destroy the BNK trial
roksbnkctl bnk down --auto

# 2. Now safe to destroy the cluster phase
roksbnkctl cluster down --auto

# 3. (Optional) Delete the workspace itself
roksbnkctl ws delete <name> --force

Or the one-shot equivalent:

# 1. Tear down both phases in order
roksbnkctl down --auto

# 2. (Optional) Delete the workspace itself
roksbnkctl ws delete <name> --force

If you roksbnkctl up against a registered cluster (one you didn’t cluster up yourself), step 2 doesn’t apply — the cluster wasn’t yours to destroy. Just bnk down the trial and stop there, then optionally unregister by deleting cluster-outputs.json.

Refusal messages catalogue

The phase-scoped destroy verbs refuse loudly when the shape doesn’t allow what you’ve asked for. Every refusal names the verb that would actually work. If you hit one in the wild, grep your terminal output for the message text and you should land here:

The “migrate the state first” references in two of the messages describe a future roksbnkctl migrate command that does not exist in v1.1.0. The refusals point at it so the wording stays valid once migrate ships; until then, the unscoped up / down is the working alternative for legacy workspaces.

What survives a destroy

The contract: roksbnkctl never destroys local state without explicit consent, and never destroys cloud resources outside its Terraform state.

After a successful down:

Re-running up against a down’d workspace re-creates everything from scratch. The workspace’s config.yaml is preserved precisely so this re-create can use the same inputs without re-prompting.

The COS bucket point is worth highlighting: the bundled HCL provisions the COS instance but generally does not provision the buckets inside it (those are written by post-apply provisioners or by the BNK runtime itself). When cluster down destroys the COS instance, the bucket goes with it — but if the COS instance was created out-of-band (e.g. by a registered cluster’s owner) and roksbnkctl is just attaching, then cluster down doesn’t apply and the COS survives.

--auto for non-interactive runs

All three destroy commands prompt for confirmation by default:

$ roksbnkctl down
This will destroy workspace "default"'s resources.
Continue? [y/N]: 

$ roksbnkctl bnk down
This will destroy the BNK trial for workspace "default". The cluster phase
will remain in place — run `roksbnkctl cluster down` to remove it too.
Continue? [y/N]: 

$ roksbnkctl cluster down
This will destroy the cluster phase for workspace "default" (ROKS + transit gateway + registry COS + cert-manager + jumphost).
Continue? [y/N]: 

--auto skips the prompt — required for CI / scripted pipelines:

roksbnkctl down --auto
roksbnkctl bnk down --auto
roksbnkctl cluster down --auto

--auto does not override the shape-based refusals (see §“Refusal messages catalogue” above) — those are correctness guards, not confirmation prompts. If trial state is present, cluster down --auto still refuses; on a legacy single-state workspace, bnk down --auto and cluster down --auto still refuse.

Like up, transient errors retry

down doesn’t share up’s explicit retry-on-transient-error logic, but Terraform’s destroy is naturally idempotent: re-running down after a partial destroy picks up where the previous run left off. If you see a transient network error during destroy, just re-run:

roksbnkctl down --auto
# (some resources destroyed, then transient error)

roksbnkctl down --auto
# (picks up where it left off, completes)

The same applies to cluster down. ROKS cluster destroy specifically can take longer than expected when the master is propagating its delete state — wait a few minutes and re-try if you see master-not-found errors.

Cleaning up workspaces

A successful down leaves the workspace directory in place. You usually want to clean that up too:

roksbnkctl ws delete <name> --force

Two safety rails on ws delete:

• Refuses to delete the current workspace. Use the parking-lot pattern if you need to drop your current workspace.
• Refuses if Terraform state still lists resources (unless --force). Catches the case where you forgot to run down first.

The --force flag overrides both checks — but if you ws delete --force a workspace that still has provisioned cloud resources, you’ll have leaked them. There’s no auto-recovery; you’d need to find them via the IBM Cloud console and delete them by hand.

The full clean-as-you-go pattern from scripts/e2e-test.sh (Phase D destroys; Phase H parks and deletes):

# 1. Destroy the trial
roksbnkctl down --auto

# 2. Destroy the cluster phase
roksbnkctl cluster down --auto

# 3. Park the current-workspace pointer somewhere harmless
roksbnkctl ws new e2e-cleanup
roksbnkctl ws use e2e-cleanup

# 4. Now the original workspace is no longer current — safe to delete
roksbnkctl ws delete default --force

# 5. (Optional) clean up the parking lot too
roksbnkctl ws delete e2e-cleanup --force

Step 3-5 is the parking-lot pattern from Chapter 6. It’s specifically necessary when the workspace you want to delete is currently the active one — ws delete refuses to remove the current workspace because that would leave a dangling current_workspace pointer.

Cost note: an undestroyed cluster keeps billing

ROKS clusters bill at roughly $0.30/hour per cluster + worker pool — call it $7/day for a 2-worker cluster, plus a few cents/day for the VPC / load balancers / COS / jumphost. A forgotten cluster can rack up real cost over a weekend.

To verify what’s still running in your account:

1. IBM Cloud console → Kubernetes → Clusters — every cluster, billing or not.
2. IBM Cloud console → VPC Infrastructure → VPCs — networks left over after a partial destroy.
3. IBM Cloud console → Resource list — exhaustive view of everything in the account, filterable by RG.

If you find a leaked cluster from a past roksbnkctl run, the right move is to re-attach to it via roksbnkctl cluster register <name> and then cluster down --auto — roksbnkctl cleans up cleanly when it has the cluster in its state. Manually deleting via the console works too but leaves dangling VPCs and security groups that the bundled HCL would have cleaned up.

roksbnkctl status and roksbnkctl cluster show both report the cluster identity recorded in cluster-outputs.json, but they don’t probe for “are there other clusters in this account?” — that’s deliberately not their job. The IBM Cloud console is the canonical source of truth for what’s billing.

Workspace deletion ≠ destroy

A subtle but important distinction. roksbnkctl ws delete removes the local workspace directory and the OS-keychain API key entry. It does not destroy any cloud resources. If you ws delete --force without first running down / cluster down, the cloud resources keep running and you’ve lost the local Terraform state that roksbnkctl would use to destroy them.

In that scenario, recovery is:

1. Find the leaked cluster in the IBM Cloud console.
2. Recreate the workspace: roksbnkctl init -w recovery.
3. Register the existing cluster: roksbnkctl cluster register <leaked-cluster-name>.
4. Then run roksbnkctl cluster down --auto to destroy it cleanly.

The Terraform state is regenerated implicitly during register + plan; the resources roksbnkctl would otherwise have tracked get re-discovered through the IBM SDK lookups. It’s not seamless, but it’s recoverable.

The ws delete --force flag’s “still has resources” check exists exactly to prevent this scenario — don’t bypass it without thinking about the consequences.

Worked example: register an existing cluster, deploy BNK, tear down

End-to-end Part III scenario: somebody on your team already provisioned a ROKS cluster manually via the IBM Cloud console (or via a different terraform tree); you need to deploy BNK on top of it using roksbnkctl, validate, and tear the whole thing down cleanly. The flow exercises Chapter 9, Chapter 10, and this chapter end-to-end.

# 1. Workspace bootstrap — same as a fresh deploy
roksbnkctl init -w preexisting
# (answer prompts for region + resource group; pick the values matching
#  the existing cluster's location)

# 2. Register the already-running cluster into the workspace
roksbnkctl cluster register existing-bnk-cluster -w preexisting
# Expected:
#   → Discovering cluster "existing-bnk-cluster" via IBM Cloud API ...
#   ✓ Cluster ID: <crn>
#   ✓ Wrote ~/.roksbnkctl/preexisting/cluster-outputs.json
#   ✓ Fetched admin kubeconfig to ~/.kube/config (chmod 0600)

# 3. Verify roksbnkctl sees the cluster
roksbnkctl status -w preexisting
# Expected: cluster Ready, workers count, no BNK pods yet

# 4. Deploy BNK on top — `up` is idempotent over the existing cluster
roksbnkctl up --auto -w preexisting
# Expected: terraform applies the cert-manager + flo + cne_instance +
# license modules only; the roks_cluster module sees the cluster already
# exists and skips. ~10-15 min vs ~50 min for a from-scratch up.

# 5. Validate
roksbnkctl test -w preexisting
# Expected: green across connectivity + dns

# 6. Tear down — destroys the BNK overlay; the registered cluster survives
roksbnkctl down --auto -w preexisting
# Expected:
#   → terraform destroy (auto-approved)
#   Destroy complete! Resources: N destroyed.
#   ✓ Workspace "preexisting" state retained at ~/.roksbnkctl/preexisting/

The destroy count N is the BNK overlay + jumphost only — typically 30-40 resources, not the from-scratch ~77 count. cluster register is a discovery-only path: terraform state holds the overlay modules (cert_manager, flo, cne_instance, license) and the testing jumphost, but not the roks_cluster module, because the cluster pre-existed roksbnkctl. down destroys only what terraform knows about, so the registered cluster survives untouched.

If you also want to release the underlying cluster, you have to tear it down through whatever provisioned it originally (the IBM Cloud console, or the separate terraform tree your teammate used). roksbnkctl cluster down only works against clusters roksbnkctl cluster up created in the first place — see Chapter 8 for the cluster-phase boundary.

The full register → up → test → down loop above is what Phase E + Phase H of the e2e plan exercise; see Chapter 23 for the CI version.

Cross-references

• Chapter 6 — Workspaces — ws delete mechanics and the parking-lot pattern.
• Chapter 8 — The cluster phase — what cluster up provisions and cluster down removes.
• Chapter 9 — Registering an existing cluster — the cluster register mechanics the walkthrough builds on.
• Chapter 10 — Deploying BNK trials — what up provisions and down removes.
• Chapter 26 — Troubleshooting — recovery from partial-destroy and orphan-state scenarios.

Workspace config (config.yaml)

This chapter is the field-by-field reference for the per-workspace config.yaml. If you’ve read Chapter 6 — Workspaces you’ve seen the on-disk layout; this chapter zooms in on the YAML file that drives everything else (init, up, down, cluster up, the test suite, the SSH targets, the new execution backends).

You don’t usually edit this file by hand. roksbnkctl init generates it interactively; later runs read it. But because every other knob in the tool reads from here, it’s worth knowing what every field means and what defaults apply when you leave one out.

File location

Each workspace’s config lives at:

~/.roksbnkctl/<workspace>/config.yaml

Override the base directory with the ROKSBNKCTL_HOME env var (test fixtures use this; everyday users shouldn’t need it). The file is created mode 0644 — readable by your user, the same trust posture as the surrounding workspace directory.

There’s also a global ~/.roksbnkctl/config.yaml at the top level — it holds the current_workspace pointer and other user-wide preferences. That’s a different file with a different schema; this chapter is about the per-workspace one.

When it gets written

Direct hand-editing is supported (the file is plain YAML) but discouraged for fields that have dedicated commands — adding an SSH target via roksbnkctl targets add keeps the schema validation in one place.

Top-level structure

ibmcloud:        # IBM Cloud account + auth
  region: ca-tor
  resource_group: default
  api_key_source: keychain
  # api_key_b64: <base64-of-api-key>   # OPTIONAL fallback when keychain unavailable

cluster:         # ROKS cluster identity
  create: true
  name: tf-openshift-cluster
  openshift_version: "4.18"
  workers_per_zone: 2

bnk:             # BNK trial knobs (optional; falls through to upstream HCL defaults)
  cneinstance_size: Small
  far_repo_url: repo.f5.com
  manifest_version: 2.3.0-3.2598.3-0.0.170

test:            # test-suite tuning (optional)
  throughput:
    duration: 30
    streams: 8
  connectivity:
    extra_hosts:
      - https://my.gslb.example.com

tf_source:       # where the Terraform HCL comes from
  type: embedded         # embedded | github | local

targets:         # SSH targets (see Chapter 15)
  jumphost:
    host: 169.45.91.177
    user: ubuntu
    key_source: tf-output:jumphost_shared_key

exec:            # per-tool execution backend defaults (see Chapter 17)
  ibmcloud:  { backend: local }
  iperf3:    { backend: k8s }
  terraform: { backend: local }

cos:             # optional COS supply-chain config
  instance: bnk-orchestration
  bucket: bnk-schematics-resources

Every block except ibmcloud:, cluster:, and tf_source: is optional. Omit a block and the tool falls through to either a documented default (covered below) or the upstream HCL’s own default for terraform variables.

ibmcloud:

ibmcloud:
  region: ca-tor
  resource_group: default
  api_key_source: keychain
  api_key_b64: ""

The plaintext field name api_key: is rejected at load time — roksbnkctl refuses to read a workspace config that contains it. The encoded api_key_b64: form is the only inline path. Full discussion in Chapter 14 — Credentials and the resolver chain.

cluster:

cluster:
  create: true
  name: tf-openshift-cluster
  openshift_version: "4.18"
  workers_per_zone: 2

The cluster: block translates to terraform variables create_roks_cluster, openshift_cluster_name, roks_cluster_id_or_name, openshift_cluster_version, roks_workers_per_zone — see Chapter 13 and Chapter 29 for the full mapping.

bnk:

bnk:
  cneinstance_size: Small
  far_repo_url: repo.f5.com
  manifest_version: 2.3.0-3.2598.3-0.0.170

Every field here is optional — leave the block out entirely and you get the upstream HCL’s defaults for all three.

test:

test:
  throughput:
    image: networkstatic/iperf3:latest
    duration: 30
    streams: 8
    default_mode: north-south
  connectivity:
    extra_hosts:
      - https://my.gslb.example.com
      - https://internal.example.test

tf_source:

tf_source:
  type: embedded

An empty type is treated as embedded (legacy / forgot-to-set).

roksbnkctl init --upgrade-tf is the helper for bumping the source between versions without retyping the rest of the config — see “Editing by hand vs helpers” below.

targets: — SSH targets

targets:
  jumphost:
    host: 169.45.91.177
    user: ubuntu
    key_source: tf-output:jumphost_shared_key
  bastion:
    host: ops.example.com
    user: jgruber
    key_path: ~/.ssh/id_ed25519

Each entry has host, user, optional port (default 22), and exactly one of key_path or key_source. The key_source enum supports agent and tf-output:<name>.

The deep reference is Chapter 15 — SSH targets, and the user-facing prose is Chapter 16 — The –on flag and SSH jumphosts. This chapter just notes the schema’s place in the overall config.

You don’t typically edit this block by hand. roksbnkctl up auto-populates jumphost post-apply, and roksbnkctl targets add ... populates the rest.

exec: — execution-backend defaults

exec:
  ibmcloud:  { backend: local }
  iperf3:    { backend: k8s }
  terraform: { backend: local }

Per-tool defaults for the --backend system. Each entry is keyed by the tool name (ibmcloud, iperf3, terraform, and others as the matrix grows) and selects which execution backend that tool uses by default. Allowed backend values:

A --backend <value> flag on the command line overrides the workspace config for that single invocation. The flag wins; the config sets the default.

The iperf3 default is k8s because measuring throughput from a laptop’s internet uplink isn’t useful — you want the test to run from a network location adjacent to or inside the cluster. The local default is wrong for that tool, so the workspace config flips it.

Chapter 17 — Execution backends covers the full backend matrix; Chapter 18 — Choosing a backend per tool is the decision tree.

cos: — COS supply-chain (optional)

cos:
  instance: bnk-orchestration
  bucket: bnk-schematics-resources
  upload:
    - source: ./local/f5-far-auth-key.tgz
      key: f5-far-auth-key.tgz
    - source: ./local/trial.jwt
      key: trial.jwt

The block is optional — if you’ve already populated COS by hand or via the upstream HCL’s roks_cos_instance_name variable, you don’t need it. Chapter 25 — COS supply chain management covers the full workflow.

Behaviour when fields are missing

roksbnkctl falls through three layers in order: workspace config → upstream HCL default → fail.

The general rule: if you don’t write it in config.yaml, roksbnkctl doesn’t write it into terraform.tfvars, and the upstream HCL’s default = ... clause takes over. The full upstream defaults are listed in Chapter 29.

How --var-file interacts with config.yaml

Both roksbnkctl up and roksbnkctl plan/apply/destroy accept the same --var-file flag terraform itself accepts (repeatable, later files win). The layering rule is:

1. config.yaml-derived terraform.tfvars        (written first by roksbnkctl)
2. ~/.roksbnkctl/<ws>/terraform.tfvars.user  (optional manual override)
3. --var-file <path>                           (CLI; repeatable)

Later layers override earlier. Concretely: config.yaml’s cluster.workers_per_zone: 2 writes roks_workers_per_zone = 2 into the generated tfvars. If you then pass --var-file ./bigger.tfvars containing roks_workers_per_zone = 5, terraform sees 5. The config.yaml value didn’t get re-applied; --var-file wins.

The terraform.tfvars.user middle layer is for when you want a workspace-local override that survives across runs without modifying config.yaml — it’s typically used for fields the YAML schema doesn’t model (rare; the schema covers the common knobs). Chapter 13 goes deep on this.

The IBMCLOUD_API_KEY is the one exception that never goes through tfvars on disk. It’s passed as a TF_VAR_ibmcloud_api_key env var on the terraform invocation. --var-file cannot supply the API key — the resolver chain in Chapter 14 is the only path.

Editing by hand vs helpers

Several commands manage subsets of config.yaml so you don’t have to:

When you do edit by hand, the load-time validators run on next roksbnkctl invocation:

• The plaintext-secret heuristic rejects an api_key: field (it must be api_key_b64: to be tolerated).
• Workspace name validation runs on directory access (workspace names must match [A-Za-z0-9][A-Za-z0-9_.-]{0,63}).
• YAML parse errors surface a line number.

If a hand edit breaks the file, every command that reads the workspace fails fast with the parse error path, so you’ll know within one invocation.

Worked example: bootstrap a workspace from scratch

End-to-end Part IV scenario: brand-new laptop, no roksbnkctl workspaces yet, an IBM Cloud API key in your password manager. Goal: a usable workspace with the key in the OS keychain, the right region + resource group resolved, and terraform.tfvars ready to drive the HCL.

# 1. roksbnkctl init — interactive bootstrap
$ roksbnkctl init
Workspace name [default]: dev
IBM Cloud region [ca-tor]:
IBM Cloud resource group [default]:
Enter IBM Cloud API key (input hidden):
Save the key for future runs? [Y/n]: y
  ✓ saved to OS keychain (service: roksbnkctl, account: dev/ibmcloud_api_key)
Cluster name [tf-openshift-cluster]: dev-cluster
Workers per zone [1]: 2
✓ Created workspace "dev"

The resulting ~/.roksbnkctl/dev/config.yaml:

ibmcloud:
  region: ca-tor
  resource_group: default
  api_key_source: keychain
cluster:
  create: true
  name: dev-cluster
  workers_per_zone: 2
tf_source:
  type: embedded

That’s the minimum. Everything else (bnk:, test:, targets:, exec:, cos:) is empty and falls through to defaults. The API key can also be supplied non-interactively from your password manager’s CLI by setting IBMCLOUD_API_KEY in the environment of the init invocation:

op here is the 1Password CLI; the op://... URI is its secret-reference scheme. Any password-manager CLI that prints a secret to stdout works the same way — Bitwarden (bw), gopass, aws secretsmanager get-secret-value, Doppler, etc. — the only thing roksbnkctl cares about is that IBMCLOUD_API_KEY is set in the environment when init runs.

# Alternative: pre-set IBMCLOUD_API_KEY so init resolves it from env rather than prompting
IBMCLOUD_API_KEY=$(op read 'op://Private/IBM Cloud/api-key') roksbnkctl init -w dev

Chapter 14 §“The IBMCLOUD_API_KEY resolver chain” covers the full env → keychain → workspace api_key_b64 → TTY-prompt order; this env-var path is the first link in that chain, so anything init resolves at bootstrap time follows the same precedence later invocations use. Once init has saved the key to the OS keychain (the default sink), no further prompting is needed. init still prompts interactively for the remaining workspace metadata (region, resource group, cluster name) — a fully non-interactive bootstrap is on the v1.x roadmap.

Now render terraform.tfvars so subsequent up runs have explicit HCL inputs to point --var-file at:

# 2. Render terraform.tfvars from config.yaml
$ roksbnkctl tfvars -w dev > ~/.roksbnkctl/dev/terraform.tfvars
$ head ~/.roksbnkctl/dev/terraform.tfvars
ibmcloud_region        = "ca-tor"
ibmcloud_resource_group = "default"
cluster_name           = "dev-cluster"
workers_per_zone       = 2
# ...

Chapter 13 covers the precedence rules between config.yaml, terraform.tfvars, and terraform.tfvars.user (the hand-edit overlay).

Finally, verify the workspace is healthy before the first real up:

# 3. Sanity-check
$ roksbnkctl doctor -w dev
✓ terraform     1.6.2  on PATH
✓ IBMCLOUD_API_KEY resolves via keychain
✓ region "ca-tor" accepts the key (IAM round-trip OK)
✓ resource group "default" exists (id: ...)
✓ workspace dev healthy

From here, roksbnkctl up --auto -w dev is the next step (see Chapter 7 — Quick start). You can layer on bnk:, test:, targets:, exec:, cos: blocks by hand-editing config.yaml whenever you need them — init only writes the minimum to keep first-run friction low.

Cross-references

• Chapter 13 — Terraform variables — the layering between config.yaml and terraform.tfvars.
• Chapter 14 — Credentials and the resolver chain — the api_key_* fields and how they’re resolved.
• Chapter 15 — SSH targets — the targets: block.
• Chapter 17 — Execution backends — the exec: block.
• Chapter 28 — Configuration reference — auto-generated complete field list.
• Chapter 29 — Terraform variable reference — the upstream HCL variables config.yaml translates to.

Terraform variables (terraform.tfvars)

roksbnkctl is a thin orchestration layer over a Terraform HCL bundle. The HCL has its own variables — well over 60 of them — declared in terraform/variables.tf. The workspace’s config.yaml covers the common knobs; for the rest, you reach into terraform.tfvars directly.

This chapter is the surface for that lower layer: where the example file lives, how roksbnkctl tfvars bootstraps a starter, what --var-file does, the layering rule between config.yaml-derived tfvars and your overrides, and the one variable that never goes on disk (ibmcloud_api_key).

Where the bundled HCL lives

The Terraform HCL is bundled into the roksbnkctl binary via go:embed. On first use of a workspace, it gets extracted to:

~/.roksbnkctl/<workspace>/state/tf-source/embedded-terraform/
├── main.tf
├── variables.tf
├── outputs.tf
├── providers.tf
├── versions.tf
├── terraform.tfvars.example
└── modules/

That terraform.tfvars.example file is the canonical reference for what’s tunable — every variable with a sensible starter value, grouped by module (ROKS cluster, cert-manager, FLO, CNEInstance, License, testing). terraform/variables.tf (linked at the GitHub canonical URL) is the formal declaration with types, descriptions, and defaults.

You don’t edit the example file in place. Copy or generate from it instead.

roksbnkctl tfvars — bootstrap a starter

The roksbnkctl tfvars subcommand prints a starter terraform.tfvars to stdout, populated from the current workspace state:

$ roksbnkctl tfvars > ~/.roksbnkctl/dev/terraform.tfvars.user

What gets pre-filled:

• Every field from config.yaml that maps to a tfvar (cluster name, region, workers, BNK fields, COS fields)
• The cluster’s identity from cluster-outputs.json if cluster up has already run
• A commented-out section for the variables you might want to tune next (jumphost profile, GSLB datacenter, license mode)

What’s deliberately excluded:

• ibmcloud_api_key — never on disk (see “The IBMCLOUD_API_KEY exception” below)
• Sensitive outputs (BIG-IP passwords, COS HMAC secrets) — left as upstream defaults

The starter is meant to be copied into ~/.roksbnkctl/<ws>/terraform.tfvars.user (the workspace-local override file) or into a --var-file path you keep alongside the workspace.

What you typically edit

The variables that matter for day-to-day BNK trial work, ordered by likely-to-touch:

For the full list with types and per-field descriptions, see terraform/variables.tf directly — link here — or the auto-generated Chapter 29 — Terraform variable reference.

The layering rule

When roksbnkctl up (or plan/apply/destroy) invokes Terraform, it composes three layers of tfvars in this order:

1. terraform.tfvars              (rendered by roksbnkctl from config.yaml)
2. terraform.tfvars.user         (workspace-local override, optional)
3. --var-file <path> ...         (CLI flag, repeatable, later file wins)

Later layers override earlier ones — same rule Terraform itself uses for -var-file chaining.

Concretely:

# config.yaml says cluster.workers_per_zone: 2
# ~/.roksbnkctl/dev/terraform.tfvars.user contains:
#   roks_workers_per_zone = 4
# Run with no flag:
roksbnkctl up
# → terraform sees 4 (.user wins over generated .tfvars)

# Pass a CLI override:
roksbnkctl up --var-file ./perf-test.tfvars
# perf-test.tfvars contains: roks_workers_per_zone = 8
# → terraform sees 8 (.var-file wins over .user)

# Multiple --var-files; later wins:
roksbnkctl up \
  --var-file ./base.tfvars \
  --var-file ./override.tfvars
# → values in override.tfvars win over base.tfvars,
#   which both win over .user, which wins over .tfvars

The --var-file flag matches Terraform’s own --var-file exactly — repeatable, paths interpreted relative to the working directory at invocation time.

The IBMCLOUD_API_KEY exception

The upstream HCL declares ibmcloud_api_key as a sensitive variable. Every other tfvar can land in a file on disk; this one never does.

Instead, the API key flows through the resolver chain (env → keychain → config-b64 → prompt — see Chapter 14), and roksbnkctl exports it as TF_VAR_ibmcloud_api_key in the environment of the terraform-exec child process. Terraform reads the env var and injects it as if it had been declared in tfvars, but no plaintext key ever touches the filesystem.

If you put ibmcloud_api_key = "..." in a hand-edited tfvars and run terraform directly (not via roksbnkctl), it works — Terraform itself is happy. But this is not how roksbnkctl runs Terraform, and putting the key in a .tfvars.user or --var-file is strongly discouraged: the file persists on disk, gets backed up, gets committed to git by accident, and gets read by other processes. The env-var path eliminates the on-disk window entirely.

Other secrets in scope:

• bigip_password — upstream HCL declares it as a regular string (not sensitive). If you set it in tfvars, the value lands on disk. Treat that file like a credential.
• COS HMAC keys — auto-generated by the roks_cluster module via the COS service-credentials resource; they live in terraform.tfstate (which is itself sensitive — chmod 0600, never commit, treat the workspace as a secret store).

Worked example: bigger cluster for a perf test

Default workspace, default cluster. You want to bump worker count for one perf-test run, then go back.

# 1. Confirm the current value comes from config.yaml
$ grep workers ~/.roksbnkctl/dev/config.yaml
  workers_per_zone: 2

# 2. Drop a one-off override into a file
$ cat > ~/perf-cluster.tfvars <<'EOF'
roks_workers_per_zone = 6
roks_min_worker_vcpu_count = 32
roks_min_worker_memory_gb = 128
EOF

# 3. Plan against it (note: --var-file passes through to terraform plan)
$ roksbnkctl plan --var-file ~/perf-cluster.tfvars

# 4. Apply
$ roksbnkctl apply --var-file ~/perf-cluster.tfvars

# 5. Run the throughput test
$ roksbnkctl test throughput

# 6. Roll back: re-apply WITHOUT the var-file
$ roksbnkctl apply
# → terraform sees workers_per_zone=2 again from config.yaml-derived tfvars

Notice step 6 — dropping the --var-file flag is the rollback. Terraform compares its current state to the new desired state (from config.yaml) and scales the worker pool back down. No special “undo” command needed.

For a more permanent override (you want this workspace to always run with bigger nodes), put the contents of perf-cluster.tfvars into ~/.roksbnkctl/dev/terraform.tfvars.user instead. Then every roksbnkctl up/apply picks it up automatically without a CLI flag.

When to edit config.yaml vs .tfvars.user vs --var-file

A rough decision matrix:

The schema in config.yaml covers about a third of the upstream HCL variables — the ones that nearly every workspace needs to set. The other two-thirds (jumphost details, every BNK module’s full surface, the testing module’s full surface) are reachable through the lower layers.

Cross-references

• Chapter 12 — Workspace config — what config.yaml covers vs what falls through to tfvars.
• Chapter 14 — Credentials and the resolver chain — why ibmcloud_api_key doesn’t go in tfvars.
• Chapter 29 — Terraform variable reference — auto-generated complete reference for terraform/variables.tf.
• The upstream terraform/variables.tf source: https://github.com/jgruberf5/roksbnkctl/blob/main/terraform/variables.tf
• The upstream starter file: https://github.com/jgruberf5/roksbnkctl/blob/main/terraform/terraform.tfvars.example

Credentials and the resolver chain

roksbnkctl handles four kinds of secrets: an IBM Cloud API key, a kubeconfig, an SSH private key, and the Terraform state file. Each has a different threat model, a different lookup chain, and a different rule for “what’s safe to commit to a workspace”.

This chapter is the user-facing distillation of PRD 04 — credential propagation. PRD 04 is the design surface for developers extending the credential system; this chapter is the operational surface for users who need to know “where does my key live, and how does the tool find it”.

The four secrets in scope

Each has its own discovery rules. Walk them in turn.

The IBMCLOUD_API_KEY resolver chain

The single most-used credential. Resolved by internal/cred/resolver.go (extracted this sprint from the formerly scattered logic in internal/config/secrets.go). The resolver walks four sources in order:

1. Environment variables (process-scoped, never persisted)
2. OS keychain (per-user, system-managed)
3. Workspace config api_key_b64 (per-workspace, base64 obfuscation)
4. Interactive prompt (TTY-only)

The first source that yields a non-empty value wins. The chain stops there — the key isn’t re-fetched from a “more authoritative” source on subsequent calls.

Source 1 — Environment

The resolver checks these env vars in order, returning the first non-empty value:

IBMCLOUD_API_KEY            # canonical
IC_API_KEY                  # short alias
TF_VAR_ibmcloud_api_key     # terraform passthrough form
TF_VAR_IBMCLOUD_API_KEY     # uppercase variant some pipelines use
TF_VAR_IC_API_KEY           # uppercase variant of the short alias

Env vars are first because they’re the most explicit path — if you’ve gone to the trouble of setting one, you’ve made a deliberate choice. Pre-existing CI pipelines, automation scripts, and direnv setups all live here. The resolver respects that ordering even when a keychain entry also exists.

Source 2 — OS keychain

roksbnkctl stores per-workspace API keys in the OS-native keychain via github.com/zalando/go-keyring:

Entries are namespaced under service roksbnkctl, with user <workspace>/ibmcloud_api_key:

# What `roksbnkctl init` writes (no-op shown for clarity)
$ keyring set roksbnkctl dev/ibmcloud_api_key

This is the recommended secure default. The OS handles process isolation; roksbnkctl only sees the value during the brief window between fetch and use.

Source 3 — Workspace api_key_b64

A base64-encoded blob in ~/.roksbnkctl/<workspace>/config.yaml:

ibmcloud:
  api_key_b64: ZW5jb2RlZC1hcGkta2V5LXZhbHVl

Important framing: base64 is obfuscation, not encryption. Anyone with read access to the file can decode it instantly:

echo -n "ZW5jb2RlZC1hcGkta2V5LXZhbHVl" | base64 -d
# → encoded-api-key-value

The encoding exists for two reasons:

1. Visual. A glancing cat config.yaml doesn’t surface the literal API key.
2. Format. API keys can contain = and other YAML-special characters that complicate inline storage. Base64 normalises them.

api_key_b64 is the fallback when the OS keychain isn’t available — most commonly WSL2 without libsecret, headless Linux servers, and CI runners where bringing up a keychain daemon is more friction than it’s worth. Treat the file like a plaintext credential: chmod 0600, never commit, never share.

File-mode note: roksbnkctl init writes config.yaml mode 0644 by default (chapter 12 §“File permissions”). When you populate api_key_b64, chmod 0600 the file yourself — and re-chmod after any subsequent roksbnkctl init that re-writes the file, since init doesn’t preserve the tightened mode. The keychain and env-var paths sidestep this entirely: nothing sensitive lands in config.yaml, so the default 0644 is fine.

The plaintext field name api_key: is rejected at workspace-load time:

$ roksbnkctl up
error: ~/.roksbnkctl/dev/config.yaml: plaintext secret detected (offset 47):
       workspace config.yaml must not contain credentials — use IBMCLOUD_API_KEY
       env var or the OS keychain (see `roksbnkctl init`)

The regex catches api_key, apikey, ibmcloud_api_key, password, token, secret_access_key, hmac_secret. The _b64 suffix is the documented escape — it’s the only inline form the loader tolerates.

Source 4 — Interactive prompt

When sources 1-3 all come up empty AND stdin is a TTY, the resolver prompts:

Enter IBM Cloud API key for workspace "dev": ********
Save the key for future runs? [Y/n]: y
  ✓ saved to OS keychain

The key is read with echo disabled (via golang.org/x/term). The prompt offers to persist — by default it tries the OS keychain first, falls back to api_key_b64 in config.yaml if the keychain is unavailable.

If stdin isn’t a TTY (CI runner, piped input, daemon process), the resolver errors instead of hanging:

error: no IBM Cloud API key available and stdin is not a TTY (cannot prompt;
       set IBMCLOUD_API_KEY or run `roksbnkctl init`)

Pinning a single source

The chain is the default. To force one specific source, set ibmcloud.api_key_source in config.yaml:

ibmcloud:
  api_key_source: keychain    # env | keychain | config | prompt

This is useful in two scenarios:

• CI: api_key_source: env makes a missing env var a hard error rather than falling through to a (locked / non-existent) keychain.
• Auditable single-source-of-truth: pinning to keychain documents that this workspace’s key lives in the OS keychain and nowhere else; reading the key from a different source becomes an error rather than a silent fallback.

kubeconfig discovery

Different chain, different rules. roksbnkctl discovers the kubeconfig the same way kubectl does — two sources, in this order:

1. KUBECONFIG environment variable (first existing path in a colon-separated list)
2. ~/.kube/config

This is the kubectl-standard discovery chain, implemented in internal/k8s/client.go::DefaultKubeconfigPath(). Whatever you’ve already taught kubectl to read, roksbnkctl reads too.

cluster up’s post-apply step writes the admin kubeconfig to ~/.kube/config (mode 0600) by default — so the second source in the chain is also the destination of the tool’s own output, and the same KUBECONFIG-overrides-everything rule applies. If KUBECONFIG is set when cluster up runs, the download lands at that path instead.

Note: there is also a ~/.roksbnkctl/<workspace>/state/kubeconfig/ directory under the workspace state dir. It’s a Terraform input (kubeconfig_dir tfvar) that the upstream HCL writes per-component sub-files into (cert_manager, cne_instance, flo, license); it is not a kubeconfig file the resolver reads. Don’t confuse the two.

When the file is missing

If neither source yields a kubeconfig, commands that need one error with:

error: no kubeconfig: KUBECONFIG env not set, ~/.kube/config not present.
       Run `roksbnkctl cluster up`, `roksbnkctl cluster register <name>`,
       or set KUBECONFIG.

The remediation message tells you which path to take. cluster register <name> is the path for an existing cluster you want to adopt without re-creating it (see Chapter 9).

File permissions

cluster up writes ~/.kube/config chmod 0600 (owner read/write only). It contains the cluster admin token; treat it like a credential. Don’t commit it, don’t email it, don’t cat it in screen-shared sessions.

SSH private keys

Per-target, not per-workspace. Each entry under targets: in config.yaml declares exactly one of:

The tf-output: form is the auto-discovered jumphost path — the upstream HCL provisions a tls_private_key resource per cluster create, marks it sensitive, and surfaces it as a terraform output. roksbnkctl reads the output via terraform output -raw at SSH-connect time, never persists it, and the key only exists in TF state plus in memory during a connect.

Chapter 15 — SSH targets is the deep reference for the targets: block; this chapter just notes the credential-side framing.

Terraform state

~/.roksbnkctl/<workspace>/state/terraform.tfstate is the workspace’s terraform state file. It contains:

• IBM Cloud admin tokens (cluster admin, COS HMAC credentials)
• Generated TLS private keys (the jumphost shared key referenced above)
• Sensitive outputs (FAR auth bundles, license JWTs)
• Every resource attribute terraform tracks

It is plaintext-credential-equivalent. The file mode is 0600; the parent directory is 0700. Backup the workspace dir intact, never commit it to git, treat compromise of the state file as compromise of every secret it contains.

There is no separate “TF state credential” — the file’s filesystem ACL is the only access control. PRD 04 covers the cross-backend story for moving state into a Docker bind-mount, a Kubernetes Secret, or an SCP’d remote temp directory; at v1.0 the local file is the only path (terraform --backend k8s / ssh are deferred to v1.x; see docs/PLAN.md §“What’s deliberately deferred to post-v1.0”).

What’s safe to commit vs not

A short rule:

SAFE TO COMMIT:    nothing in ~/.roksbnkctl/<workspace>/
NOT SAFE:          everything in ~/.roksbnkctl/<workspace>/

The longer version, by file:

The simplest policy: a .gitignore that excludes the entire ~/.roksbnkctl/ tree. If you really want to share a workspace skeleton with a colleague, send the config.yaml minus api_key_b64 and let them re-run roksbnkctl init against their own account.

How roksbnkctl init writes the API key

Walk through the writeable side of the resolver:

$ roksbnkctl init
Workspace name [default]: dev
IBM Cloud region [ca-tor]:
Enter IBM Cloud API key (input hidden): ********
Save the key for future runs? [Y/n]: y
  ✓ saved to OS keychain

What just happened:

1. init prompted for the key. Input echo was off; the key never appeared on screen.
2. The user said “save”.
3. SaveAPIKeyForWorkspace tried SaveAPIKeyToKeychain first.
4. The OS keychain accepted the entry (Linux + libsecret in this case). The success path returned "OS keychain" and init printed the confirmation.
5. The key was not written to terraform.tfvars (that’s the resolver’s job at terraform-invoke time, via the TF_VAR_ibmcloud_api_key env var).

If step 4 had failed (no keychain, WSL2 without libsecret), SaveAPIKeyForWorkspace would have fallen through to saveAPIKeyToConfig — base64-encoded the key, written it into config.yaml’s api_key_b64 field, returned "config.yaml (base64)". init would have printed:

  ✓ saved to config.yaml (base64)
  warning: base64 is obfuscation, not encryption — chmod 0600 the file

Both destinations work. The keychain path is the recommended default; the config-b64 path is the documented fallback.

What’s new in v1.2: the cred-tmpfile and trusted-profile paths

v1.2.0 closes the two longest-deferred items from PRD 04 §“Open questions”: roksbnkctl --backend docker no longer leaks IBMCLOUD_API_KEY in docker inspect, and roksbnkctl --backend k8s ops install auto-provisions an IBM Cloud trusted profile so the ops pod never sees a static API key. Both have fallbacks for environments where the new path doesn’t apply; v1.0.x / v1.1.x workspaces continue to work without change.

The tmpfile-bind-mount pattern (docker backend)

The docker backend writes the resolved IBMCLOUD_API_KEY to a 0600 tempfile on the host, bind-mounts that single file read-only at /run/secrets/ibmcloud_api_key inside the container, and points the container at the file via IBMCLOUD_API_KEY_FILE=/run/secrets/ibmcloud_api_key. The value never appears in the container’s stored env metadata — docker inspect <id> shows the path, not the key. The tempfile is owned by the calling user and is removed on backend exit (and on context cancellation, so an interrupted run still cleans up).

You don’t have to do anything to opt in — the pattern is the default for --backend docker on v1.2 and up. The engineering shape (lifecycle, the inline sh -c shim that re-exports the value into the legacy IBMCLOUD_API_KEY env name for tools that read from env, the why-not-just-use---secret discussion) lives in PRD 04 §“Resolved in Sprint 9” → “Cred tmpfile-bind-mount pattern (docker backend)”. For most users the takeaway is one line: in v1.2, --backend docker is docker inspect-clean.

The --trusted-profile flag (k8s backend)

New flag on roksbnkctl ops install that controls how the ops pod gets its IBM Cloud credential. Three values:

Chapter 19 — The in-cluster ops pod walks through the --trusted-profile=auto install flow, the verification commands (oc get serviceaccount roksbnkctl-ops -o yaml showing the trusted-profile annotation), the fallback warning shape, and how ops uninstall cleans up a provisioned profile.

Compatibility note

v1.0.x and v1.1.x workspaces continue to work without migration. The docker tmpfile pattern is a transparent replacement — the resolver chain is unchanged, the workspace config is unchanged, and no flag is required to opt in. The k8s --trusted-profile=auto default with auto-fallback means existing workspaces against an IAM-restricted key keep getting the static-key Secret as before, with one extra stderr warning line on ops install naming the fallback and how to silence it. Setting --trusted-profile=off reproduces the v1.0.x behaviour byte-for-byte (no warning, static-key Secret straight away).

Backend-specific cred propagation

The credential-propagation rules differ per backend. All four backends ship at v1.0:

Each backend’s “where creds live” surface is summarised in Chapter 17 — Execution backends; the design rationale is in PRD 04.

The user-facing invariant across all four: you put the key into one of the resolver chain’s sources, and roksbnkctl figures out the rest. You don’t have to learn four different credential APIs to use four different backends.

The redactor

roksbnkctl writes a fair amount to its own logs (stdout, stderr) — terraform plan output, ibmcloud CLI output, error traces. Anywhere we can plausibly print the IBM API key (because a downstream tool printed it, because an error message included it, because a debug trace dumped a struct), the redactor masks it before the bytes leave the binary.

What gets redacted:

• The IBM Cloud API key value, anywhere it appears in Stdout or Stderr of an exec backend’s RunOpts. Replaced with [REDACTED].
• The same value in roksbnkctl’s own log output (the lifecycle commands that wrap terraform-exec).

What does not get redacted:

• Output captured by callers via -o yaml/-o json for resources that legitimately contain the key (e.g., a Secret returned from roksbnkctl k get). The redactor doesn’t know about Kubernetes resource semantics; if you k get secret -o yaml, you’ll see the key. (The same is true of kubectl.)
• Output from a tool you ran outside roksbnkctl (e.g., piping to tee after invoking terraform directly). The redactor only sees bytes that pass through the exec backend’s Stdout/Stderr writers.
• The terraform state file. State is on-disk; the redactor is an in-memory stream filter.

The implementation is internal/exec/redact.go — a wrapping io.Writer with byte-comparison redaction and cross-write prefix buffering (so a secret split across two Write calls still gets masked). The matcher uses the resolved API key value verbatim (a known string at run-time) rather than a generic “looks like an IBM API key” pattern, to avoid false positives on legitimate output.

PRD 04’s acceptance criteria require that the API key never appears in docker inspect, ps -ef, kubectl get pods/events -o yaml, or kubectl describe pod. The redactor is the defence-in-depth layer; the per-backend cred-propagation rules are the primary control.

Cross-references

• PRD 04 — credential propagation across execution backends — the full design.
• Chapter 12 — Workspace config — the ibmcloud: block schema.
• Chapter 13 — Terraform variables — why ibmcloud_api_key doesn’t go in tfvars.
• Chapter 15 — SSH targets — the SSH key sources.
• Chapter 17 — Execution backends — backend-specific cred mechanics.
• internal/cred/resolver.go — the implementation extracted this sprint: https://github.com/jgruberf5/roksbnkctl/blob/main/internal/cred/resolver.go
• internal/config/secrets.go — the keychain + config-b64 helpers: https://github.com/jgruberf5/roksbnkctl/blob/main/internal/config/secrets.go

SSH targets

This chapter is the technical reference for the targets: system. Its companion is Chapter 16 — The –on flag and SSH jumphosts, which is the user-facing prose for “how do I run a command on the jumphost”. Chapter 16 introduces targets briefly; this chapter goes deeper into the schema, the key sources, the host-key trust model, the auto-discovery pipeline, and what the ssh execution backend layers on top of the lightweight --on flag.

If you arrived here from Chapter 16 looking for “where do I learn the full surface”, you’re in the right place.

The targets: block schema

Targets live under targets: in ~/.roksbnkctl/<workspace>/config.yaml:

targets:
  jumphost:
    host: 169.45.91.177
    port: 22
    user: ubuntu
    key_source: tf-output:jumphost_shared_key

  bastion:
    host: ops.example.com
    user: jgruber
    key_path: ~/.ssh/id_ed25519

  prod-jump:
    host: 10.0.0.5
    user: ec2-user
    key_source: agent

The Go struct backing it is internal/config.TargetCfg:

type TargetCfg struct {
    Host      string `yaml:"host"`
    Port      int    `yaml:"port,omitempty"`        // default 22
    User      string `yaml:"user"`
    KeyPath   string `yaml:"key_path,omitempty"`    // file path
    KeySource string `yaml:"key_source,omitempty"`  // "agent" | "tf-output:<name>"
}

Validation rules at load time:

• Exactly one of key_path or key_source must be set. Setting neither, or both, fails the load with a clear error.
• The target name (the YAML map key) must be non-empty and stable across YAML round-trips — roksbnkctl targets show <name> and roksbnkctl targets remove <name> look up by this name.

The TargetCfg type lives in internal/config rather than internal/remote to avoid an import cycle: the YAML (de)serialiser needs the wire shape, and the SSH client (internal/remote) needs to consume it. Keeping the shape in config and the runtime Target (parsed key, dialer config, etc.) in remote keeps the dependency direction one-way.

Key sources

The three options for telling roksbnkctl how to find the SSH private key for a target.

key_path: <file>

A PEM-encoded private key on disk:

bastion:
  host: ops.example.com
  user: jgruber
  key_path: ~/.ssh/id_ed25519

Standard OpenSSH formats are accepted: id_rsa, id_ed25519, id_ecdsa, id_dsa (deprecated but supported). Tilde expansion uses os.UserHomeDir() semantics — ~/ → user home, ~user/ is not supported (use an absolute path).

The file is read at SSH-connect time, not at config-load time. A missing or unreadable file fails the connect, not the workspace load. This matters for ergonomics: you can edit a target into config.yaml referencing a key path that doesn’t exist yet, then create the key separately, without roksbnkctl init/use failing in between.

Encrypted keys (passphrase-protected) are not currently supported in the SSH client — the agent path is the recommended workflow for keys that need a passphrase.

key_source: agent

Talks to ssh-agent over $SSH_AUTH_SOCK:

prod-jump:
  host: 10.0.0.5
  user: ec2-user
  key_source: agent

The agent presents whichever keys it currently holds; roksbnkctl tries each in turn against the target’s authorized_keys (via SSH’s standard publickey-authentication exchange). The first key the server accepts is the one that gets used.

This is the right setting when:

• Your team manages keys via 1Password / hardware tokens / gpg-agent and you don’t want a key file on disk.
• You’re on a shared workstation where putting the key file in ~/.ssh/ is undesirable.
• You’re already using ssh-agent for everything else and want consistent behaviour.

Platform note: ssh-agent integration is Linux/macOS-only. Windows users should use key_path to a file. The restriction is structural to the Go SSH library, which doesn’t wrap the Windows ssh-agent named-pipe protocol — see golang.org/x/crypto/ssh/agent and upstream tracking issues for status; full Windows support is a v2 item.

key_source: tf-output:<output-name>

Reads the key from the workspace’s terraform state output of that name:

jumphost:
  host: 169.45.91.177
  user: ubuntu
  key_source: tf-output:jumphost_shared_key

This is the auto-discovered jumphost path. The upstream HCL provisions a tls_private_key resource per cluster create, marks it sensitive, and surfaces it as a terraform output named jumphost_shared_key. roksbnkctl reads it via the equivalent of terraform output -raw <name> at SSH-connect time.

What this gets you that key_path doesn’t:

• No on-disk key file separate from terraform state. The key only exists in terraform.tfstate (which is already a sensitive workspace artefact) and in memory during the SSH handshake.
• Auto-rotation on cluster re-create. Destroy and re-create the cluster, terraform generates a new tls_private_key, and the next --on jumphost invocation picks up the new key without any manual rewriting of the workspace config.
• Single source of truth. The key value is in terraform state — the same place every other cluster-generated secret lives.

The terraform output must be a string-typed PEM-encoded private key. terraform output -raw <name> returns the value regardless of the sensitive flag (the flag just suppresses display; the data is still readable to anyone with state access).

Host-key TOFU and ~/.roksbnkctl/known_hosts

roksbnkctl keeps its own known_hosts file at ~/.roksbnkctl/known_hosts. It does not read or write ~/.ssh/known_hosts. The two files are independent.

Why a separate file

Three reasons:

1. Isolation. roksbnkctl’s SSH client is a different program from ssh(1); mixing host-key state between the two creates surprising behaviour (deleting a key from ~/.ssh/known_hosts doesn’t clear it from roksbnkctl’s view, or vice versa).
2. Audit. A roksbnkctl-managed file lets the tool’s behaviour be reasoned about without inspecting the user’s broader SSH state.
3. Cleanup. roksbnkctl ws delete <name> could theoretically scrub host-key entries on workspace destroy; mixing into ~/.ssh/known_hosts would mean editing a file the tool didn’t own.

The format matches OpenSSH’s ~/.ssh/known_hosts exactly (so future cross-pollination is technically possible), but the filenames are deliberately separate.

TOFU on first connect

The first time you connect to a target, roksbnkctl shows the host key fingerprint and asks whether to trust it:

$ roksbnkctl exec --on jumphost -- whoami
Add 169.45.91.177:22's key (SHA256:abc123def456ghi789jkl0mnopqrstuvwxyz/+=) to ~/.roksbnkctl/known_hosts? [y/N]: y
ubuntu

Answer y and the key is appended. Subsequent connects to the same host:port with the same server key trust silently.

Answer n and the connect aborts with exit code 126.

Mismatch behaviour

If the host key changes — re-provisioned VM, MITM attack, configuration drift — roksbnkctl refuses to connect:

error: host key mismatch: 169.45.91.177:22 known with SHA256:abc123... but
       server presented SHA256:zyx987...; if the host was rebuilt, edit
       ~/.roksbnkctl/known_hosts

Same model OpenSSH uses. The fix is the same: edit (or ssh-keygen -R) the file to remove the stale entry, then re-connect to re-trigger the TOFU prompt.

The default ssh-keygen binary works against ~/.roksbnkctl/known_hosts — pass -f:

ssh-keygen -R 169.45.91.177 -f ~/.roksbnkctl/known_hosts

--insecure-host-key for CI

Automation contexts can’t answer a TOFU prompt. The --insecure-host-key flag skips host-key verification entirely:

roksbnkctl exec --on jumphost --insecure-host-key -- whoami

This is insecure — anyone in the network path can MITM the connection — and is intended only for short-lived CI runs against ephemeral test infrastructure. Don’t use it where session content is sensitive.

The flag is per-invocation, not per-target. There’s deliberately no targets.<name>.insecure_host_key: true config knob: forcing the choice into the call site keeps the security implications visible at every invocation.

When to use it:

• E2E tests against a freshly-provisioned cluster jumphost where the host key is just-generated and changes per run.
• Pipeline runs against ephemeral test VMs that get torn down within minutes.
• Recovery scenarios where the known-hosts file is corrupt and you need to bootstrap.

When not to use it:

• Production jumphosts with stable identity.
• Customer environments where session integrity matters.
• Anything where the SSH session carries secrets you can’t afford to leak to a passive attacker.

roksbnkctl targets — full reference

Four subcommands. Chapter 16 introduces them with worked examples; here’s the complete flag surface.

roksbnkctl targets list

NAME       HOST                USER     KEY
jumphost   169.45.91.177:22    ubuntu   tf-output:jumphost_shared_key
bastion    ops.example.com:22  jgruber  file:~/.ssh/id_ed25519
prod-jump  10.0.0.5:22         ec2-user agent

Flags:

• --verbose / -v: also prints whether the target has a known-hosts entry recorded.
• -o json: machine-readable form. Schema: {"targets": [{"name": ..., "host": ..., "port": ..., "user": ..., "key_source": ...}]}.

The KEY column shows the source descriptor — never the key material. File-backed sources are prefixed file: to distinguish them visually from tf-output: and agent.

roksbnkctl targets show <name>

name:        jumphost
host:        169.45.91.177
port:        22
user:        ubuntu
key_source:  tf-output:jumphost_shared_key

Same restriction: key material itself is never printed.

-o json is supported for scripted callers.

roksbnkctl targets add <name> ...

Required flags: --host, --user, and exactly one of --key-path / --key-source.

# File-backed key
roksbnkctl targets add bastion \
  --host ops.example.com \
  --user jgruber \
  --key-path ~/.ssh/id_ed25519

# ssh-agent
roksbnkctl targets add prod-jump \
  --host 10.0.0.5 \
  --user ec2-user \
  --key-source agent

# Non-default port
roksbnkctl targets add custom \
  --host 10.0.0.5 \
  --user root \
  --key-path ~/.ssh/custom \
  --port 2222

# tf-output (rare; usually auto-populated by `up`)
roksbnkctl targets add backup-jump \
  --host 10.0.0.6 \
  --user ubuntu \
  --key-source tf-output:backup_jumphost_key

Refuses to add a target whose name collides with an existing entry — use targets remove <name> first, or pick a different name.

roksbnkctl targets remove <name>

roksbnkctl targets remove bastion

Removes the entry from config.yaml. Does not remove the corresponding host-key line from ~/.roksbnkctl/known_hosts — re-adding the same target later doesn’t re-trigger TOFU. This is deliberate; if you want to wipe the host key too, edit the known-hosts file by hand.

Auto-discovery from terraform outputs

The single most-used target — jumphost — is auto-populated post-roksbnkctl up. The flow:

1. roksbnkctl up runs terraform apply against the workspace’s HCL.

2. After successful apply, roksbnkctl reads three outputs: testing_tgw_jumphost_ip, testing_tgw_jumphost_user, jumphost_shared_key.

3. If testing_tgw_jumphost_ip is non-empty AND not the literal sentinel string "TGW jumphost not created" (which the upstream HCL emits when the testing module is disabled), roksbnkctl writes a jumphost target into config.yaml:

   targets:
     jumphost:
       host: <testing_tgw_jumphost_ip>
       user: <testing_tgw_jumphost_user || "ubuntu">
       key_source: tf-output:jumphost_shared_key
   

4. A confirmation line is printed:

   ✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`
   

The auto-population is idempotent — re-running up against an already-jumphost-populated workspace re-writes the same fields. If you’ve manually customised the entry (changed the user, swapped to key_path), the auto-population overwrites your changes. There’s no merge logic; the latest up wins.

If testing_create_tgw_jumphost = false in tfvars, the upstream HCL skips creating the jumphost VM and emits the sentinel output. Auto-population is then a no-op, and you’re free to create your own jumphost (or differently-named) entry via targets add.

Per-AZ cluster jumphosts (jumphost-<zone>)

When testing_create_cluster_jumphosts = true, the deploy builds one cluster jumphost per cluster-VPC availability zone in addition to the single TGW jumphost — each on its own floating IP, all sharing the same key as the TGW jumphost. Since v1.5.0, the same post-up hook that seeds the singular jumphost also auto-registers one target per AZ:

1. After a successful up, roksbnkctl reads the testing_cluster_jumphost_ips terraform output — a map { zone => floating-IP }.

2. For each zone => fip, it upserts a target named jumphost-<zone>, reusing the same shared key the singular jumphost uses:

   targets:
     jumphost-ca-tor-1:
       host: <ca-tor-1-fip>
       user: ubuntu
       key_source: tf-output:jumphost_shared_key
     # …one per AZ…
   

3. A summary line is printed:

   ✓ Auto-registered 3 per-AZ cluster jumphost targets (jumphost-ca-tor-1, jumphost-ca-tor-2, jumphost-ca-tor-3); use `roksbnkctl --on jumphost-<zone> ...`
   

Verify with roksbnkctl targets list — you should see jumphost plus one jumphost-<zone> per AZ. Each is a first-class --on target (full kubectl/oc/ibmcloud/shell passthrough, no SSH hop): roksbnkctl --on jumphost-ca-tor-2 kubectl get pods. Like the singular jumphost, registration is best-effort and idempotent — a parse/write failure logs a single warning: and does not fail up, and re-running up after a floating-IP rotation refreshes each jumphost-<zone> host in place. When testing_create_cluster_jumphosts = false (or the output is absent/empty), this is a silent no-op — only jumphost is seeded, with no warning noise.

Orphaned-target caveat (option (a) upsert-only). Auto-registration upserts but never prunes. If a later apply removes a zone, or testing_create_cluster_jumphosts is flipped to false, the now-orphaned jumphost-<oldzone> target points at a destroyed host and lingers in your config until you remove it by hand:

roksbnkctl targets remove jumphost-ca-tor-3

A host re-created on a recycled floating IP will also trip the host-key mismatch refusal — see §“Host-key TOFU and ~/.roksbnkctl/known_hosts” and clear the stale known_hosts line with ssh-keygen -R <fip> -f ~/.roksbnkctl/known_hosts. An automatic-prune (reconcile) mode that removes orphans on the next up is a deliberate post-v1.5.0 follow-up (it needs unambiguous “this target is auto-managed” ownership semantics so a hand-named jumphost-mybox is never deleted). See PRD 09.

Pre-v1.5.0 fallback. On a release before v1.5.0 the per-AZ jumphosts are not auto-registered — register each by hand. Look up the floating IPs with the read-only terraform command (v1.5.0+; Chapter 16 §“Per-AZ cluster jumphosts”):

roksbnkctl terraform output testing_cluster_jumphost_ips
roksbnkctl terraform output testing_cluster_jumphost_ssh_commands

…or, on an even older release without roksbnkctl terraform, the raw form cd ~/.roksbnkctl/<ws>/state && TF_DATA_DIR=$PWD/terraform terraform output testing_cluster_jumphost_ips. Then register one target per AZ — note --key-source tf-output:jumphost_shared_key is correct because one shared key covers all jumphosts (see §“key_source: tf-output:<output-name>”):

roksbnkctl targets add jumphost-ca-tor-1 \
  --host <ca-tor-1-fip> --user ubuntu \
  --key-source tf-output:jumphost_shared_key
# …repeat per zone…

Each new IP triggers a one-time host-key TOFU prompt on first connect (see §“Host-key TOFU and ~/.roksbnkctl/known_hosts”). Manually-added targets are not auto-managed: a destroy+recreate rotates the FIPs and you must re-targets add (contrast the v1.5.0 auto-registered targets, which up refreshes in place).

What is not auto-discovered

The auto-discovery flow registers the TGW jumphost (always) and, since v1.5.0, the per-AZ jumphost-<zone> targets when testing_create_cluster_jumphosts = true. It does not register:

• Per-AZ jumphosts by private IP. There is no top-level testing_cluster_jumphost_private_ips terraform output; the private-IP hop pattern in Chapter 16 §“Per-AZ cluster jumphosts” is a documented zero-setup technique, not an auto-registered target.
• Any non-jumphost host. Bastions, ops boxes, and the like are always targets add by hand.

Inspecting what the post-up flow saw

When the auto-population doesn’t happen and you expected it to, check:

roksbnkctl tf output testing_tgw_jumphost_ip
roksbnkctl tf output testing_tgw_jumphost_user
roksbnkctl tf output -json jumphost_shared_key | head -c 50

(The third one returns a JSON-encoded string for a sensitive output; truncate to confirm it’s non-empty without dumping the key.)

If all three are populated and the auto-write didn’t fire, that’s a bug — file an issue with the output values redacted.

What the SSH execution backend adds on top of --on

The --on <target> flag is the lightweight remote-exec path — one SSH session, one command, no remote state. The ssh:<target> execution backend layers more on top, reusing the same internal/remote.Client under internal/exec/ssh.go:

The targets: schema and the roksbnkctl targets commands are the same surface for both — the backend just uses each target in a heavier-weight way. Anything you set up for --on keeps working under --backend ssh:<target>.

The split between the lightweight --on path and the full ssh backend is deliberate: --on stays simple — one SSH session, one command, no remote state. The backend handles the heavier lifting (file materialisation, package installation, multi-step orchestration).

Cross-references

• Chapter 12 — Workspace config — where targets: fits in the overall schema.
• Chapter 14 — Credentials and the resolver chain — the SSH-key sources from a credential-discipline perspective.
• Chapter 16 — The –on flag and SSH jumphosts — the user-facing prose for “how do I use this”.
• Chapter 17 — Execution backends — where the SSH backend sits in the broader backend matrix.
• PRD 01 — SSH client + –on flag — the design rationale for targets: and the SSH client.
• internal/remote/ package: https://github.com/jgruberf5/roksbnkctl/tree/main/internal/remote
• internal/cli/targets.go: https://github.com/jgruberf5/roksbnkctl/blob/main/internal/cli/targets.go

The –on flag and SSH jumphosts

The --on <target> flag (most commonly --on jumphost) re-runs a roksbnkctl passthrough command (exec, shell, kubectl, oc, ibmcloud) on a remote SSH host instead of locally. After a successful roksbnkctl up, a jumphost target is auto-populated from the upstream HCL’s terraform outputs, so --on jumphost works with no manual configuration in the common case.

This chapter covers when to reach for --on, the targets: workspace config block, the auto-population behaviour, the roksbnkctl targets command tree for managing your own targets, and how host-key trust is established.

The full design rationale for this feature lives in PRD 01. This chapter is the user-facing distillation.

Why this exists

There are a handful of scenarios where running a command from your laptop is the wrong answer:

• Customer-firewall scenarios. Your customer’s network policy lets the corporate jumphost reach *.cloud.ibm.com but blocks your laptop’s egress to anything except web traffic. ibmcloud iam oauth-tokens works from the jumphost; from your laptop it times out.
• Air-gapped environments. The cluster lives in a VPC with no public ingress, accessible only via a bastion VM. The cluster API server isn’t reachable from your laptop at all; you need to be inside the network to talk to it.
• Pre-cluster operations. You want to run ibmcloud commands against the IBM Cloud API but your workstation doesn’t have ibmcloud installed and you’d rather not install it. The jumphost has it; route through there.

--on makes those scenarios one flag rather than “ssh to the jumphost, install your tools there, copy your kubeconfig over manually”. The SSH client is built into roksbnkctl (using golang.org/x/crypto/ssh); no host ssh binary is required.

The targets: workspace config block

Targets are stored in your workspace config at ~/.roksbnkctl/<workspace>/config.yaml under a targets: key:

targets:
  jumphost:                                # auto-populated after `roksbnkctl up`
    host: 169.45.91.177
    user: ubuntu
    key_source: tf-output:jumphost_shared_key
    port: 22                               # default; can be omitted

  bastion:                                 # user-defined
    host: ops.example.com
    user: jgruber
    key_path: ~/.ssh/id_ed25519

  prod-jump:
    host: 10.0.0.5
    user: ec2-user
    key_source: agent                      # use ssh-agent

Each entry has at minimum host and user. Port defaults to 22. Key resolution is determined by exactly one of key_path or key_source — see “Key sources” below.

You don’t typically edit this file by hand. The auto-discovery flow populates jumphost for you, and roksbnkctl targets add ... populates other entries.

Auto-discovery from roksbnkctl up

The upstream HCL provisions a small testing jumphost as part of every cluster apply. Two terraform outputs surface it:

• testing_tgw_jumphost_ip — the public IP of the jumphost VM.
• jumphost_shared_key — the private key (PEM) the jumphost was provisioned with, marked sensitive in the HCL.

After a successful roksbnkctl up, roksbnkctl reads both outputs and writes a jumphost target into your workspace config:

✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`

The auto-registered target uses user: ubuntu (the upstream HCL provisions an Ubuntu cloud image whose default user is ubuntu).

The key_source: tf-output:jumphost_shared_key form means the private key is read from terraform state at SSH-connect time rather than being copied into the workspace config or written to disk separately. The key only ever exists in terraform state and in memory during a connect; destroying and re-creating the cluster generates a new key, and roksbnkctl picks up the new one without any manual intervention.

If your cluster apply produced a testing_tgw_jumphost_ip output of "TGW jumphost not created" (the upstream HCL emits this string when the testing module is disabled) the auto-population is skipped. You can still add a jumphost target manually with roksbnkctl targets add if you have a different bastion in mind.

Key sources

Three ways to tell roksbnkctl how to find the SSH private key:

1. key_path: <path> — a file on disk. Standard OpenSSH key formats are accepted (~/.ssh/id_ed25519, ~/.ssh/id_rsa, etc.). Tilde expansion is honoured.

2. key_source: agent — talk to the user’s ssh-agent over the socket pointed at by $SSH_AUTH_SOCK. The agent presents whichever keys it currently holds; roksbnkctl tries each in turn against the target’s authorized_keys. This is the right setting if your team already manages keys via 1Password / hardware tokens / gpg-agent and you don’t want a key file on disk. Note: ssh-agent integration is Linux/macOS-only at v1.0; Windows users should use key_path instead. Windows ssh-agent named-pipe support is on the v1.x roadmap.

3. key_source: tf-output:<output-name> — read the key from the workspace’s terraform state output of that name. Used by the auto-discovered jumphost target. The terraform output must be a string-typed PEM-encoded private key; sensitive outputs work fine because terraform output -raw <name> returns the value regardless of the sensitive flag.

Exactly one of key_path or key_source must be set per target. roksbnkctl targets show <name> will tell you which is in use without printing the key material.

Host-key TOFU on first connect

The first time you connect to a target, roksbnkctl shows the host key fingerprint and asks whether to trust it. The prompt is a single line:

$ roksbnkctl exec --on jumphost -- whoami
Add 169.45.91.177:22's key (SHA256:abc123def456ghi789jkl0mnopqrstuvwxyz/+=) to ~/.roksbnkctl/known_hosts? [y/N]: y
ubuntu

Answer y and the key is appended to ~/.roksbnkctl/known_hosts (the same format as OpenSSH’s ~/.ssh/known_hosts). Subsequent connects trust silently.

Answer n and the connect fails with a clear “host key not trusted” error.

If the host key changes between runs — which would happen on a re-provisioned VM, or could happen as a man-in-the-middle attack — roksbnkctl refuses to connect:

error: host key mismatch: 169.45.91.177:22 known with SHA256:abc123... but server presented SHA256:zyx987...; if the host was rebuilt, edit ~/.roksbnkctl/known_hosts

This is “trust on first use” (TOFU) — the same model OpenSSH uses for new hosts. Exit code is 126 on host-key rejections.

--insecure-host-key for CI

In automation contexts where a TOFU prompt would block forever, pass --insecure-host-key to skip host-key verification entirely:

roksbnkctl exec --on jumphost --insecure-host-key -- whoami

This is insecure — anyone in the network path can MITM the connection — and is intended only for short-lived CI runs against ephemeral test infrastructure. Don’t use it in any context where the SSH session matters for security.

The roksbnkctl targets command tree

Four subcommands for managing target entries:

roksbnkctl targets list
roksbnkctl targets show <name>
roksbnkctl targets add <name> --host ... --user ... --key-path ...
roksbnkctl targets remove <name>

targets list

roksbnkctl targets list
NAME       HOST                USER     KEY
jumphost   169.45.91.177:22    ubuntu   tf-output:jumphost_shared_key
bastion    ops.example.com:22  jgruber  file:~/.ssh/id_ed25519

Prints every target in the current workspace’s config. The KEY column shows the key source — never the key material itself. File-backed keys are prefixed with file: so they’re visually distinct from tf-output: and agent sources.

targets show <name>

roksbnkctl targets show jumphost
name:        jumphost
host:        169.45.91.177
port:        22
user:        ubuntu
key_source:  tf-output:jumphost_shared_key

Prints the full record. Note that the key material itself is never printed — only the source descriptor (file path, ssh-agent, or terraform-output name).

targets add <name> ...

roksbnkctl targets add bastion \
  --host ops.example.com \
  --user jgruber \
  --key-path ~/.ssh/id_ed25519

# or with ssh-agent:
roksbnkctl targets add prod-jump \
  --host 10.0.0.5 \
  --user ec2-user \
  --key-source agent

# or with a non-default port:
roksbnkctl targets add custom \
  --host 10.0.0.5 \
  --user root \
  --key-path ~/.ssh/custom \
  --port 2222

Writes the new target into ~/.roksbnkctl/<workspace>/config.yaml. Refuses if a target of that name already exists (use targets remove first).

targets remove <name>

roksbnkctl targets remove bastion

Removes the entry from config.yaml. Does not remove the corresponding line from ~/.roksbnkctl/known_hosts — the host key stays recorded so re-adding the same target later doesn’t re-trigger a TOFU prompt.

Working examples

The everyday verbs:

# Run an arbitrary command on the jumphost
roksbnkctl exec --on jumphost -- whoami
# → ubuntu

roksbnkctl exec --on jumphost -- uname -a
# → Linux jumphost-vm 5.15.0-... #... SMP ... x86_64 GNU/Linux

# Interactive PTY shell
roksbnkctl shell --on jumphost
# → drops you into the jumphost's default shell as the configured user
# → exit returns you to your local prompt

# ibmcloud passthrough — runs `ibmcloud ks cluster ls` on the jumphost
# (handy when your laptop's network can't reach IBM Cloud APIs)
roksbnkctl ibmcloud --on jumphost ks cluster ls

# kubectl passthrough — same pattern
roksbnkctl kubectl --on jumphost get pods -A

# oc passthrough
roksbnkctl oc --on jumphost projects

Per-AZ cluster jumphosts

When your deploy sets testing_create_cluster_jumphosts = true, the upstream HCL builds one cluster jumphost per cluster-VPC availability zone in addition to the single TGW jumphost. Since v1.5.0 these are auto-registered by roksbnkctl up as jumphost-<zone> targets (alongside the singular jumphost) — see Chapter 15 §“Auto-discovery from terraform outputs”. They are first-class --on targets: full passthrough, no hop.

# verify what `up` registered:
roksbnkctl targets list
# → jumphost, jumphost-ca-tor-1, jumphost-ca-tor-2, jumphost-ca-tor-3

# run directly on a specific per-AZ cluster jumphost (no hop, full passthrough):
roksbnkctl kubectl --on jumphost-ca-tor-2 get nodes
roksbnkctl shell --on jumphost-ca-tor-1

Pre-v1.5.0 fallback. On a release before v1.5.0 the per-AZ jumphosts are not auto-registered. Look up their floating IPs and register each by hand — roksbnkctl terraform output testing_cluster_jumphost_ips (v1.5.0’s read-only terraform, Chapter 15), or on an even older release cd ~/.roksbnkctl/<ws>/state && TF_DATA_DIR=$PWD/terraform terraform output testing_cluster_jumphost_ips — then roksbnkctl targets add jumphost-<zone> --host <fip> --user ubuntu --key-source tf-output:jumphost_shared_key per zone. Chapter 15 §“Auto-discovery from terraform outputs” documents this fallback in full.

Hopping to a cluster jumphost via the registered jumphost (zero-setup, no roksbnkctl state)

The shared key is installed on every jumphost and the private key file is present on each box at /home/ubuntu/.ssh/id_rsa. So from the auto-registered TGW jumphost you can hop to any cluster jumphost by its private IP (the TGW jumphost reaches the cluster VPC over the Transit Gateway) with no key copying and nothing added to roksbnkctl state — handy for a one-off against a host you don’t want to register:

# the per-AZ private IPs are not a top-level terraform output; read them
# from inside the TGW jumphost (it sits in the routed network):
roksbnkctl exec --on jumphost -- \
  curl -s http://169.254.169.254/...   # or your own inventory source

# run a command on a cluster jumphost, hopping through the TGW jumphost,
# using the shared key already on the box (no scp):
roksbnkctl exec --on jumphost -- \
  ssh -i /home/ubuntu/.ssh/id_rsa -o StrictHostKeyChecking=accept-new \
  ubuntu@<cluster-jumphost-private-ip> kubectl get nodes

The inner ssh uses the on-box /home/ubuntu/.ssh/id_rsa (the same shared key the auto-registered targets use — one key for all jumphosts), so no key material is copied. StrictHostKeyChecking=accept-new on the inner hop avoids an interactive prompt the outer non-PTY session can’t answer. This is the zero-setup path: nothing is written to ~/.roksbnkctl/. For first-class repeated access prefer the auto-registered jumphost-<zone> targets above.

Orphaned-target caveat. The auto-registered jumphost-<zone> targets use option-(a) upsert-only registration: if a destroy removes a zone (or testing_create_cluster_jumphosts is flipped to false), the stale jumphost-<oldzone> entry lingers in your config until you roksbnkctl targets remove jumphost-<oldzone> by hand. A re-created host on a recycled floating IP will also trip the host-key TOFU mismatch — see Chapter 15 §“Host-key TOFU and ~/.roksbnkctl/known_hosts”. An automatic-prune (reconcile) mode is a tracked post-v1.5.0 follow-up.

Behaviour details worth knowing:

• Streaming I/O. stdout, stderr, stdin all stream in real time — the same as running the command locally. Long-running commands (oc adm top nodes, ibmcloud ks cluster get on a slow API call) work normally.
• Exit code propagation. The remote command’s exit code is the local exit code. A failing remote command produces a non-zero roksbnkctl exit; a succeeding remote command produces 0. CI scripts can rely on this.
• TTY auto-detection. roksbnkctl shell --on auto-allocates a PTY. Other verbs (exec, kubectl, oc, ibmcloud) run without a PTY at v1.0; if you need a PTY for top or another isatty()-sensitive command, fall back to roksbnkctl shell --on jumphost and run the command from the interactive shell.
• Environment passthrough. Only machine-portable values cross the SSH boundary: IBMCLOUD_API_KEY, IC_API_KEY, IBMCLOUD_REGION, and IBMCLOUD_VERSION_CHECK are propagated to the remote session via SSH SetEnv, so ibmcloud iam oauth-tokens on the jumphost authenticates with the same key your local workspace uses. The remote sshd must be configured to accept AcceptEnv IBMCLOUD_* etc. for this to work; the upstream HCL’s jumphost is already configured for it. KUBECONFIG is not forwarded — it is a local filesystem path, meaningless on the target, and forwarding it would shadow the jumphost’s own cloud-init-provisioned /home/ubuntu/.kube/config. Passthrough kubectl/oc on the target therefore use the target’s own kubeconfig, which is the correct behaviour. (Before v1.5.0 the local KUBECONFIG path was forwarded; after a successful local up that deterministically broke --on jumphost kubectl|oc with connection to the server localhost:8080 was refused — see the v1.5.0 changelog. Local roksbnkctl kubectl without --on still resolves KUBECONFIG via the local chain, unchanged.)

What --on doesn’t do (yet)

A few things deliberately deferred to later phases:

• Lifecycle commands (up, down, plan, apply) reject --on with a clear error at v1.0. Running terraform on a remote host has different state-handling considerations and is the job of the SSH execution backend (Chapter 17) — terraform over --backend ssh:<target> is itself deferred to v1.x (state-file portability).
• ProxyJump / multi-hop SSH. If your jumphost itself is reached through another bastion, that’s not directly supported at v1.0. The upstream HCL’s jumphost design lets the TGW jumphost reach cluster-internal VMs natively, so you usually don’t need multi-hop in practice. ProxyJump support is on the v1.x roadmap. (The per-AZ cluster jumphosts themselves are auto-registered as first-class jumphost-<zone> targets since v1.5.0 — see §“Per-AZ cluster jumphosts” above; the manual ssh -i … via the registered jumphost hop there is the zero-setup alternative for one-offs, not a roksbnkctl-native multi-hop.)
• ~/.ssh/config parsing. Targets must be defined explicitly in workspace config; roksbnkctl does not read your existing ~/.ssh/config.
• Password auth. Keys + agent only. Passwords are not supported and won’t be.
• SCP / SFTP. File transfer is the SSH execution backend’s job (handled via RunOpts.Files materialisation; see Chapter 17 §“SSH backend”). --on does one-shot remote exec only.
• Windows ssh-agent. The key_source: agent path is Linux/macOS only at v1.0; Windows users must use key_path to a file. Already noted in Key sources above; called out here so a Windows reader who skipped to this section doesn’t miss it.

Cross-reference

Chapter 17 — Execution backends extends the SSH client used here into a full execution backend with file materialisation, env-file fallback for sshd configurations that can’t AcceptEnv, and apt-bootstrap of missing tools on Ubuntu jumphosts. The --on flag stays as the lightweight one-shot path; --backend ssh is the deeper integration. The two share the same internal/remote.Client so what you learn here translates directly.

For the design rationale, edge cases, and open questions, read PRD 01 — this chapter is the user-facing surface; PRD 01 is the developer-facing surface.

Execution backends: local, docker, k8s, ssh

roksbnkctl runs a handful of external tools as part of its job — ibmcloud, terraform, iperf3, eventually dig-equivalents and others. By default each tool runs as a child process on your laptop. That’s fine for some tools and wrong for others: iperf3 from your laptop measures your laptop’s internet uplink, not the cluster’s bandwidth. Likewise, terraform via docker (the --backend docker mode covered below) lets you pin a frozen tool version for CI reproducibility without installing it on the host.

The execution-backend system lets you pick where each tool runs without changing the surface command. The same roksbnkctl ibmcloud ks cluster ls invocation can run as a local process, inside a vendored container, inside the cluster, or on a remote SSH host — selected by a flag or a per-tool default in your workspace config.

This chapter is the user-facing reference for all four backends. After the introduction, each backend gets its own deep-dive section covering the mechanics, the credential-propagation rules, the failure modes, and a short “when to use it” callout. Chapter 18 is the decision-tree companion that picks one for a given (tool, scenario) pair.

Architecture at a glance

The four backends sit between roksbnkctl (the binary on your laptop) and the external tools each backend runs. Every backend produces the same observable behaviour from the user’s point of view — the same roksbnkctl ibmcloud ks cluster ls invocation — but routes the actual tool execution to a different network vantage.

graph LR
    User[laptop<br/>roksbnkctl binary]
    subgraph local[local backend]
        L_tf[terraform]
        L_ibm[ibmcloud]
        L_iperf[iperf3]
        L_dns[dns probe]
    end
    subgraph docker[docker backend]
        D_ibm[ibmcloud<br/>frozen image]
        D_tf[terraform<br/>frozen image]
    end
    subgraph k8s[k8s backend]
        K_ops[ops pod<br/>ibmcloud]
        K_job[one-shot Job<br/>iperf3 / dns]
    end
    subgraph ssh[ssh:target backend]
        S_ibm[ibmcloud<br/>on jumphost]
        S_iperf[iperf3<br/>on jumphost]
    end
    Cluster[ROKS cluster<br/>cert-manager + flo + BNK]
    Jump[SSH jumphost<br/>auto-discovered from terraform]
    IBMAPI[IBM Cloud API<br/>+ IAM]

    User --> local
    User --> docker
    User --> k8s
    User --> ssh
    local --> IBMAPI
    docker --> IBMAPI
    k8s --> Cluster
    Cluster --> IBMAPI
    ssh --> Jump
    Jump --> IBMAPI
    Jump -.->|optional<br/>private path| Cluster

    classDef bk fill:#f4f4f4,stroke:#666,color:#000;
    class local,docker,k8s,ssh bk;

Chapter 18 is the decision-tree companion that picks a backend for a given scenario; the rest of this chapter is the per-backend mechanics.

The four backends at a glance

Each backend solves a different problem:

• local: fastest startup, simplest mental model, requires the host tool to exist on PATH.
• docker: reproducible across dev machines, no host install needed, frozen at a known-good tool version.
• k8s: network-correct (private IPs reachable, cluster-internal services accessible), zero host install, in-cluster identity via ServiceAccount.
• ssh: pre-cluster ops from a known-IP bastion, customer-firewall workflows, air-gapped environments where the laptop can’t reach IBM Cloud APIs but the jumphost can.

All four implementations conform to the same Go interface (internal/exec.Backend) so callers don’t branch on backend type — they just call backend.Run(ctx, argv, opts) and let the implementation handle the mechanics. That uniformity is what lets the same roksbnkctl ibmcloud ks cluster ls work across all four with no surface-level change.

The --backend CLI flag

Override the per-tool default for a single invocation:

# Local (the implicit default for ibmcloud + terraform)
roksbnkctl ibmcloud ks cluster ls

# Same command, in a vendored docker image
roksbnkctl ibmcloud --backend docker ks cluster ls

# Same command, in the cluster (requires `roksbnkctl ops install` first)
roksbnkctl ibmcloud --backend k8s ks cluster ls

# Same command, on a remote SSH host
roksbnkctl ibmcloud --backend ssh:jumphost ks cluster ls

Format:

--backend local|docker|k8s|ssh:<target>

The ssh:<target> form pins the SSH backend to a specific named target from roksbnkctl targets list (registered via roksbnkctl targets add; see Chapter 16).

The flag is persistent at the root — it works for any command that runs an external tool. Commands that don’t run external tools (like roksbnkctl ws list) ignore it.

The flag wins over the workspace-config default. If config.yaml says iperf3: { backend: k8s } and you pass --backend local, the local backend runs.

Backend-failure semantics

Each backend has a different failure surface. The convention is:

• Backend startup failure (Docker daemon unreachable, k8s API unreachable, SSH connect refused, binary not on PATH for local) ⇒ exit code 127, with a message naming the cause. No silent fallback to local. Silent fallback hides intent and produces confusing test results.
• Backend mid-run failure (the container started but couldn’t pull a sub-resource; the pod was OOMKilled before the wrapped tool ran; the SSH session died after apt-get install but before the tool exec) ⇒ exit code 126, distinct from 127 so CI can tell “we never got going” from “we got going then broke”.
• Tool exit code (the actual ibmcloud / terraform / iperf3 exit code, anything in 0-125 or 128-255) ⇒ propagated 1:1, including non-zero codes.
• Context cancellation / timeout ⇒ exit code 137 (the conventional SIGKILL-on-signal code).

This way, your CI script can tell “the tool said X failed” (typical exit codes) from “we never reached the tool” (127) from “we reached the tool, then the backend died mid-flight” (126) from “we ran out of time” (137).

Per-tool defaults from exec:

Workspace config carries the per-tool default backend in the exec: block:

# ~/.roksbnkctl/<workspace>/config.yaml
exec:
  ibmcloud:  { backend: local }
  iperf3:    { backend: k8s }
  terraform: { backend: local }

The defaults shipped today:

Chapter 12 — Workspace config covers the exec: block schema in detail; this chapter just notes its place in the backend system.

Chapter 18 — Choosing a backend per tool is the decision tree for “which backend should I pick for this tool in this scenario”.

Per-backend deep dives

local backend

The default for ibmcloud and terraform. os/exec.CommandContext(ctx, argv[0], argv[1:]...), inheriting the parent process’s environment, PATH, and working directory. Mechanically the simplest of the four — no container, no cluster, no network handshake.

os/exec shape

internal/exec/local.go resolves argv[0] via exec.LookPath, then builds a *exec.Cmd:

bin, err := exec.LookPath(argv[0])
// fall through to argv[0] verbatim if it's an absolute path that LookPath rejects
cmd := exec.CommandContext(ctx, bin, argv[1:]...)
cmd.Env    = effectiveEnv     // os.Environ() + opts.Env + Credentials.EnvVars()
cmd.Dir    = opts.WorkDir     // empty → inherit caller's CWD
cmd.Stdin  = opts.Stdin
cmd.Stdout = redactor(opts.Stdout, creds)
cmd.Stderr = redactor(opts.Stderr, creds)

The redactor wrap is defense-in-depth — see Chapter 14 §“The redactor”. If a wrapped tool ever prints IBMCLOUD_API_KEY value to stdout (a debug trace, an error message), the redactor replaces it with [REDACTED] before the bytes leave the binary.

Env propagation

Three sources, in order:

1. The host process’s environment (os.Environ()) — your shell’s PATH, HOME, KUBECONFIG, etc.
2. RunOpts.Env — caller-supplied KEY=VALUE strings (e.g., IBMCLOUD_REGION=ca-tor from the workspace config).
3. Credentials.EnvVars() — IBMCLOUD_API_KEY=… plus the legacy IC_API_KEY=… alias older ibmcloud versions accept.

os/exec documents that for duplicate keys the last entry wins. So caller-supplied vars override host env, and credential vars override caller-supplied — meaning a workspace’s API key always wins over a stale IBMCLOUD_API_KEY in your shell.

The local backend does not scrub the host env. If you have an unrelated AWS_ACCESS_KEY_ID in your shell, the wrapped tool sees it. That’s by design — local is the “trust the user’s shell” path; if you want a hermetic env, switch to docker.

Working directory

RunOpts.WorkDir becomes cmd.Dir. Empty → inherit the caller’s CWD (Cobra’s RootCmd.Run runs from wherever the user invoked roksbnkctl).

When RunOpts.Files is non-empty and WorkDir is empty, the local backend creates a tempdir under os.TempDir(), writes each Files entry as a 0600 file inside, and uses the tempdir as WorkDir. The tempdir is removed via defer after Run returns. This is mostly there for symmetry with the docker / k8s / ssh backends; today’s ibmcloud passthrough never uses it.

Signal handling

exec.CommandContext wires ctx cancellation to the child: when the ctx ticks past its deadline (or the user hits Ctrl-C and the root cobra command cancels), Go sends SIGKILL (the default Cmd.Cancel) to the child. The child has no opportunity to clean up; this is intentional — we’d rather kill a stuck terraform than wait on an indefinite hang.

The kill is process-only, not process-group. If terraform has spawned grandchildren (the IBM provider’s helpers, an SSH key generator, etc.) those grandchildren may outlive the ctx-cancel by a few seconds. We haven’t seen this matter in practice; if it does, a pgid kill is a small follow-up.

Exit-code mapping

Note the 126 vs 127 split: 127 means “we never reached the tool” (binary missing, daemon unreachable, SSH refused); 126 means “we reached the tool but the backend itself broke after that point” (couldn’t fork, container created but crashed, pod scheduled but evicted before exec). Sprint 3 collapsed both to 127 in the local + docker implementations; this sprint splits them per PRD 03 §“Backend interface”. CI scripts that distinguish “test infra broken” from “real test failure” can now key on the difference.

When to use it

• You have the tool installed and on PATH already.
• You want the fastest startup — no container daemon, no SSH handshake, no cluster API call.
• You’re running terraform against the workspace’s local state (the established workflow).
• You’re debugging and want the simplest mental model for “where did that output come from”.

Chapter 18 §“Decision tree” expands these into a per-(tool, scenario) walkthrough.

docker backend

Runs the tool inside a vendored container image, talking to the local docker daemon over its socket. docker on PATH is not required — roksbnkctl uses the official Docker Go SDK (github.com/moby/moby/client) and dials the socket directly.

roksbnkctl ibmcloud --backend docker ks cluster ls

Container shape

Mechanically (the ibmcloud passthrough; iperf3 client is similar with a different image and ports):

docker run --rm \
  -v <tempdir>/kubeconfig:/root/.kube/config:ro \  # if Credentials.KubeconfigBytes set
  -e IBMCLOUD_API_KEY \                            # bare name; value inherits
  -e IC_API_KEY \                                  # legacy alias
  ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:<v> \
  ks cluster ls

internal/exec/docker.go doesn’t shell out to docker run; it builds a container.Config + container.HostConfig and calls cli.ContainerCreate → ContainerStart → ContainerLogs(stream=true). The bash-style above is the conceptual equivalent.

There’s no workspace-wide bind-mount. Per-invocation mounts come from three sources only:

1. Credentials.KubeconfigBytes — written to <tempdir>/kubeconfig (mode 0600) on the host, bind-mounted as a single file at /root/.kube/config read-only. Single-file mount per PRD 04 §“Anti-patterns” — bind-mounting ~/.kube/ exposes other clusters’ configs.
2. RunOpts.Files — each name → bytes entry written to <tempdir>/<basename> and bind-mounted at /work/<basename>. The container’s WorkingDir is set to /work so callers can reference files by relative path. (ibmcloud passthrough doesn’t use this; it lands when the iperf3 client backend wants to ship iperf3.json to the pod, or when a future tool wants a config file.)
3. RunOpts.WorkDir — overrides WorkingDir if explicitly set.

The tempdir is removed via defer after Run returns, regardless of exit code or panic.

Credential propagation specifics

Three things matter, all enforced by internal/exec/creds.go::Credentials.DockerArgs(...):

1. --env IBMCLOUD_API_KEY (bare name, no =value). The docker daemon looks up the value from the daemon’s environment at container-create time, not from argv. So the literal API key string never appears in docker inspect, docker ps -a --format, or the daemon’s container metadata. PRD 04 §“Anti-patterns” calls out the --env IBMCLOUD_API_KEY=$KEY form as a leak vector — we don’t use it. See Chapter 14 — Credentials.DockerArgs() for the full call shape.
2. Single-file kubeconfig mount, read-only. Not the parent dir. The container can read exactly the kubeconfig you handed it — nothing else under ~/.kube/.
3. Stdout/stderr through the redactor. Same defense-in-depth as the local backend: if the wrapped tool prints the API key value (rare but possible), the redactor masks it before the bytes leave roksbnkctl’s process.

:dev tag resolution

The vendored images live at:

The <tag> for the vendored per-tool images (ibmcloud, iperf3) is resolved at runtime by internal/exec/docker.go::toolImageTag(). It reads the binary’s internal/version.Version (set via ldflags at build time): a release-built binary like v0.10.0 pulls :v0.10.0; a dev build (Version == "dev") pulls :dev. Sprint 4 landed this version-pinning in place of Sprint 3’s hard-coded :dev so a go install of a tagged release pulls a matching tagged image rather than a :dev that may not exist for the published binary. The terraform row is the exception — it points at the upstream hashicorp/terraform image and stays pinned to a specific version (currently 1.5.7) regardless of roksbnkctl’s own version.

The :dev tag is still the local-development idiom: cd tools/docker && make build-all builds and tags every tools image as :dev locally; a dev-build roksbnkctl finds them via the local docker cache without a ghcr.io round-trip.

If you’re cutting a custom tools image and want roksbnkctl to pick it up, the simplest path is docker tag your-image ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:dev locally — the docker backend pulls the local-cached version first.

Auto-remove and ctx-cancel-kill

Two cleanup mechanisms work together:

• AutoRemove: true in HostConfig. The docker daemon removes the container as soon as it exits, regardless of exit code. No docker ps -a clutter, no manual docker rm ever required.
• Ctx-cancel triggers ContainerKill. When ctx.Done() fires, the docker backend issues cli.ContainerKill(ctx, id, "SIGKILL") and waits a few seconds for the daemon to confirm. The --rm then takes care of removal. Net effect: hitting Ctrl-C during a stuck ibmcloud login doesn’t leave a zombie container behind.

Combined with the daemon’s own watchdog on the container, the worst case is a few seconds of “container is dying” between Ctrl-C and the container disappearing. We haven’t seen leaked containers in dev or CI.

Image build pipeline

Image versions are tagged in lock-step with roksbnkctl releases; the GitHub Actions workflow that builds + pushes them runs on every release tag. See Chapter 31 — Building from source for the build pipeline details.

terraform via docker

terraform is the second tool routed through the docker backend (alongside ibmcloud). The shape is similar to ibmcloud (docker run against a vendored image, single-file mounts for sensitive data, no creds in argv) but with two terraform-specific concerns: state persistence across runs, and host-user UID alignment so state files written inside the container stay readable on the host.

State persistence via bind-mount

Terraform’s local state file lives at terraform.tfstate in the working directory. For the docker backend the working directory has to be a host-side path bind-mounted into the container, not a container-internal path that disappears on --rm. The docker backend bind-mounts the workspace’s state directory into the container:

docker run --rm \
  -v ~/.roksbnkctl/<workspace>/state:/state \
  --workdir /state/tf-source/embedded-terraform \
  --user $(id -u):$(id -g) \
  hashicorp/terraform:1.5.7 \
  apply -auto-approve

Concretely:

• Host source: ~/.roksbnkctl/<workspace>/state/ — the same directory the local terraform backend writes state to today, so switching between --backend local and --backend docker against the same workspace doesn’t fork state.
• Container target: /state — the bind-mount root inside the container.
• Container working directory: /state/tf-source/<source>/ (e.g., /state/tf-source/embedded-terraform/ for the default embedded source) — the same path the local backend resolves to, so terraform sees the same main.tf either way.
• The HCL is bind-mounted alongside state. The embedded HCL is materialised at run time into ~/.roksbnkctl/<workspace>/state/tf-source/<source>/ (chapter 31 covers the embedded-source layout); since state/ is the bind-mount root, both terraform.tfstate and the HCL tree land inside the container together. There’s no separate HCL projection.

The bind-mount is read-write — terraform needs to write terraform.tfstate, rotate terraform.tfstate.backup, and populate the .terraform/ cache. Combined with --rm, the file lifecycle is: container creates state, container exits, --rm removes the container, state files persist on the host. Subsequent runs (re-mounted at the same host path) pick up where the prior run left off.

Image: hashicorp/terraform:1.5.7

The image is the official upstream hashicorp/terraform published by HashiCorp on Docker Hub, pinned to a literal version in internal/exec/docker.go’s toolImages map (currently 1.5.7). The pin is intentional — the embedded HCL has been validated against this terraform version, and the docker backend’s whole point is reproducibility. Bumping the pin is a deliberate change to the binary and lands as a release.

The vendored per-tool images (ibmcloud, iperf3) get their tag from the roksbnkctl binary’s own version (see § :dev tag resolution above). Terraform is the exception — the binary’s version doesn’t follow upstream terraform’s release cadence, so the pin stays literal.

The UID/GID alignment gotcha

Linux Docker containers run as root by default. With a root-owned container writing into a bind-mount, the resulting host files end up owned by root — and any subsequent local-backend terraform apply (or even a cat ~/.roksbnkctl/<ws>/state/terraform.tfstate) hits permission errors. The docker backend works around this by passing --user $(id -u):$(id -g) explicitly:

docker run --rm \
  --user 1000:1000 \                                     # host's caller-uid:caller-gid
  -v ~/.roksbnkctl/dev-tor/state:/state \
  --workdir /state/tf-source/embedded-terraform \
  hashicorp/terraform:1.5.7 \
  apply -auto-approve

The container process runs as the host user, so files written into the bind-mount are owned by the host user — same as a local-backend terraform apply would have produced. Switching backends mid-debug doesn’t strand state files behind a permission wall.

The UID/GID values are read from the host process at run time (Go’s os.Getuid() / os.Getgid()). On macOS this is mostly cosmetic — Docker Desktop’s VM normalises ownership on the host bind-mount automatically — but it’s required for clean Linux behaviour, so the backend always passes the flag.

Supported commands

The terraform docker backend honours --backend docker for the four lifecycle commands:

roksbnkctl up    --backend docker  [--var-file <path>] [--auto]
roksbnkctl plan  --backend docker  [--var-file <path>]
roksbnkctl apply --backend docker  [--var-file <path>] [--auto]
roksbnkctl down  --backend docker  [--var-file <path>] [--auto]

Flags that the local terraform backend honours (--var-file, --auto, plus the -w/--workspace selector) plumb through to the docker backend identically — the backend’s job is to spawn hashicorp/terraform:1.5.7 with the right argv; it doesn’t filter or rewrite the lifecycle commands’ flags. (--auto is roksbnkctl’s shorthand for terraform’s -auto-approve; the wrapper renames it for terseness and consistency across up/apply/down.)

roksbnkctl up --backend docker is the apply-with-auto-approve shorthand the existing local lifecycle uses; --backend docker switches the spawn target without changing the command shape.

Deferred: k8s and ssh terraform backends

--backend k8s and --backend ssh:<target> for terraform are not in v1.0. The blocker is state-handling: the local backend keeps state on the host filesystem, the docker backend bind-mounts the same path, but k8s (run terraform in a one-shot Job) and ssh:<target> (run terraform on a remote host) need a story for shipping state between the run vantage and the canonical workspace state dir. Designs under consideration include a versioned ConfigMap/Secret pair for k8s and an scp-pre-and-post atomic move for ssh; both are deferred to v1.x once the trade-offs have settled (see docs/PLAN.md §“What’s deliberately deferred to post-v1.0”).

PRD 03 §“State concerns” is the design spec; trying --backend k8s against terraform errors at parse time:

$ roksbnkctl up --backend k8s
error: terraform doesn't support backend `k8s` at v1.0 (state-handling design
       open; tracked in PRD 03 § State concerns); supported: local, docker

When to use it

• You’re on a clean dev machine without ibmcloud installed and don’t want to install it.
• You need a frozen tool version for CI reproducibility.
• You’re debugging a “works on my machine” issue and want to factor out the host install variable.

When docker is the wrong call:

• The tool needs network access that your laptop has but the container doesn’t (rare; default bridge networking usually preserves laptop’s egress).
• You’re running iperf3 and want a network-locality benefit — docker doesn’t give you that vs local. Use k8s instead.
• You’re running a DNS probe and want a different network vantage — same network identity as the host, no value-add. The DNS subcommand rejects --backend docker by design.
• You’re on Windows. Linux/macOS docker daemons are in scope; Windows Docker Desktop coverage is deferred to a future round.

k8s backend

Runs the wrapped tool inside the cluster. Two distinct execution patterns share the same Backend.Run interface:

The split mirrors the two latency budgets. Long-lived pods amortise the pod-startup cost across many invocations — perfect for ibmcloud iam oauth-tokens which you might run twenty times in a debugging session. One-shot Jobs are clean (no leftover state, no concurrency questions) — perfect for iperf3 -c <server> which runs once, emits its JSON, and exits.

Long-lived ops pod pattern

The pod is named roksbnkctl-ops in the roksbnkctl-ops namespace. roksbnkctl ops install deploys it (see Chapter 19 for the full lifecycle). The image bundles ibmcloud CLI plus kubectl as backup; future iterations may add oc, terraform, etc. The container inside the pod is named tools.

Backend.Run(ctx, argv, opts) for the ops-pod path is essentially:

exec, _ := remotecommand.NewSPDYExecutor(restConfig, "POST",
    clientset.CoreV1().RESTClient().Post().
        Resource("pods").Namespace("roksbnkctl-ops").Name("roksbnkctl-ops").
        SubResource("exec").
        VersionedParams(&corev1.PodExecOptions{
            Container: "tools",
            Command:   argv,
            Stdin:     opts.Stdin != nil,
            Stdout:    true,
            Stderr:    true,
            TTY:       opts.TTY,
        }, scheme.ParameterCodec).URL())
exec.StreamWithContext(ctx, remotecommand.StreamOptions{
    Stdin: opts.Stdin, Stdout: redactor(opts.Stdout, creds), Stderr: redactor(opts.Stderr, creds), Tty: opts.TTY,
})

The exit code comes back via the SPDY channel’s metav1.Status — the executor surfaces it as a exec.CodeExitError. We propagate that as the backend’s exit code, same as local propagates exec.ExitError.ExitCode().

opts.WorkDir is ignored for the ops pod path. The pod’s WorkingDir is fixed at container-spec time (/work); per-exec working-dir changes would require recreating the pod. Callers that need a specific cwd should cd <dir> && it into argv (the local backend’s symmetric escape hatch).

One-shot Job pattern

For each invocation, the backend builds a batchv1.Job spec, applies it, streams logs from the Job’s pod, waits for completion, reads the exit code from the pod’s container status, and lets ttlSecondsAfterFinished clean up.

Skeleton:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: roksbnkctl-iperf3-client-     # randomized; multiple runs don't collide
  namespace: roksbnkctl-test
spec:
  ttlSecondsAfterFinished: 60                  # auto-delete the Job + its Pod 60s after completion
  backoffLimit: 0                              # no retries; the test reports failure once and stops
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: iperf3-client
        image: ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<v>
        command: ["iperf3", "-c", "<server-svc>", "-J"]
        envFrom:
        - secretRef:
            name: roksbnkctl-job-creds-<random>   # projected per invocation
        volumeMounts:
        - name: files
          mountPath: /work
      volumes:
      - name: files
        projected:
          sources:
          - secret:
              name: roksbnkctl-job-files-<random>  # one Secret per invocation, holds RunOpts.Files

Three details to call out:

1. Projected Secret for cred propagation. Credentials.IBMCloudAPIKey (when set) becomes a one-shot Secret, mounted via envFrom: secretRef. Per PRD 04 §“In-cluster pod” this beats argv (which would show in kubectl describe pod) and beats inline env: blocks (which surface in kubectl get pod -o yaml). The Secret carries the same ttlSecondsAfterFinished-equivalent lifecycle: when the Job’s ttlSecondsAfterFinished deletes the Job, the owning controller’s GC sweeps the Secret too via ownerReferences.
2. Log streaming via client-go. Once the Job’s pod is in Running state, clientset.CoreV1().Pods(ns).GetLogs(name, &corev1.PodLogOptions{Follow: true}).Stream(ctx) returns an io.ReadCloser that we copy through the redactor into opts.Stdout. The stream stays open until the pod terminates or ctx cancels.
3. Exit-code extraction. When the pod transitions to Succeeded or Failed, we read pod.Status.ContainerStatuses[0].State.Terminated.ExitCode and return that as the backend’s exit code. A Failed pod with ExitCode: 0 (rare; usually OOMKilled or evicted) maps to backend exit code 126 — backend mid-run failure rather than tool failure.

The roksbnkctl-test namespace is a fresh namespace dedicated to one-shot test workloads. It’s separate from roksbnkctl-ops (the long-lived pod’s home) so RBAC can be scoped tighter — see Chapter 19 §“RBAC”.

iperf3 server side

Worth calling out because it’s the asymmetric piece. The iperf3 test deploys a server bare Pod + Service into roksbnkctl-test, then runs the client as the one-shot Job described above:

The bare-Pod (rather than Deployment) shape is intentional — the iperf3 server is single-shot, scoped to one test, torn down on completion; the controller-managed replica machinery a Deployment provides is unused and would only confuse the cleanup story. Service type is driven by --mode: north-south measures laptop-to-cluster bandwidth and needs a publicly reachable endpoint (LoadBalancer); east-west measures node-to-pod and stays in-cluster (ClusterIP). See Chapter 22 — Throughput testing for the user-facing flag surface.

The client Job’s argv is iperf3 -c <server-cluster-ip-or-lb> -J. The -J JSON flows back via log streaming, parsed in internal/test/throughput.go, surfaced as roksbnkctl test throughput JSON output.

The server pod’s securityContext is set to satisfy OpenShift’s restricted-v2 SCC: runAsNonRoot: true, allowPrivilegeEscalation: false, seccompProfile.type: RuntimeDefault, capabilities.drop: [ALL]. iperf3 listens on port 5201 (unprivileged) so no root is needed. The Sprint 3 cluster baseline tripped the SCC by missing one or more of these fields; the manifest the k8s backend emits this sprint sets all four.

When to use it

• You’re running iperf3 and want a number that reflects cluster fabric, not your office Wi-Fi.
• You’re running ibmcloud from a network that can reach the cluster but not *.cloud.ibm.com directly. The ops pod has both lines of sight; your laptop has only one.
• You want a cluster-side ad-hoc shell for debugging — roksbnkctl exec --backend k8s -- bash (when implemented) drops into the ops pod.

When k8s is the wrong call:

• The cluster doesn’t exist yet (roksbnkctl ops install requires a working kubeconfig). Use local or ssh for pre-cluster ops.
• You haven’t run roksbnkctl ops install. Run it first; it’s a one-time setup per cluster.
• You’re running terraform — --backend k8s for terraform is deferred to a future release pending a state-handling design (see PRD 03 §“State concerns”).

Chapter 19 is the full reference for the cluster-side mechanics: namespace, ServiceAccount, ClusterRole, Secret, lifecycle.

ssh backend

Runs the wrapped tool on a registered SSH target. Builds on Sprint 1’s internal/remote.Client (the same SSH client backing the --on flag); this section assumes you’ve read Chapter 16 for the target-config and host-key TOFU framing.

roksbnkctl ibmcloud --backend ssh:jumphost ks cluster ls
roksbnkctl ibmcloud --backend ssh:bastion --bootstrap iam oauth-tokens

Per-tool apt-bootstrap and the --bootstrap flag

Before exec’ing the wrapped tool, the SSH backend probes whether it’s installed:

ssh <target> 'command -v <tool>'

Exit 0 → tool present, proceed. Non-zero → tool missing. What happens next depends on --bootstrap:

• Without --bootstrap (the default). The backend errors with exit 127 and a clear message:

  error: tool `iperf3` not found on ssh target jumphost; re-run with --bootstrap to install via apt-get,
         or pre-install on the target manually
  

  No sudo apt-get ever runs. The backend won’t surprise the user with package-manager invocations or sudo password prompts on a remote they didn’t expect mutation on.

• With --bootstrap. The backend runs the per-tool bootstrap recipe. For Ubuntu (the only OS supported this round), the recipe is roughly:

  # ibmcloud needs IBM's apt repo + GPG key first
  curl -fsSL https://download.clis.cloud.ibm.com/Linux/Ubuntu/repo.gpg | sudo apt-key add -
  echo 'deb https://download.clis.cloud.ibm.com/Linux/Ubuntu jammy main' \
    | sudo tee /etc/apt/sources.list.d/ibmcloud.list
  sudo -n apt-get update -y
  sudo -n apt-get install -y ibmcloud-cli
  

  iperf3 is simpler — no repo addition, just sudo -n apt-get install -y iperf3.

The opt-in default reflects PRD 03 open question §“--bootstrap opt-in for SSH”: silent sudo apt-get on a remote host is the kind of surprise that erodes operator trust, especially when the remote is shared between teams. Make the user say “yes, install for me”.

Bootstrap failure modes split between the two backend-failure exit codes per §“Backend-failure semantics”: 127 when we never got going (couldn’t reach the repo, no apt mapping, tool missing without --bootstrap); 126 when we got partway in and then something broke (sudo / OS-detect / install).

File materialisation

RunOpts.Files entries are written to a per-invocation tempdir on the remote. The tempdir is /tmp/roksbnkctl.<random>/ where <random> is a fresh 16-byte hex string per Run:

# pseudo-flow
ssh <target> 'mkdir -m 0700 /tmp/roksbnkctl.<rand>'
scp <local-temp>/<basename> <target>:/tmp/roksbnkctl.<rand>/<basename>
ssh <target> '
  trap "rm -rf /tmp/roksbnkctl.<rand>" EXIT
  cd /tmp/roksbnkctl.<rand>
  <argv...>
'

The trap … EXIT is shell-builtin; it fires on normal exit, on set -e failure, on SIGINT (Ctrl-C), on SIGTERM. So even if the user kills their roksbnkctl invocation mid-run, the remote tempdir is cleaned up by the wrapper script’s own trap before the SSH session terminates.

The 0700 mode on the tempdir ensures only the SSH user can read it during the brief on-disk window. On shared bastions (multi-user jumphosts) this matters — and it’s why we materialise to /tmp (which the user owns) rather than /var/tmp or some shared scratch path.

Kubeconfig follows the same pattern: Credentials.KubeconfigBytes becomes <tempdir>/kubeconfig, the wrapper exports KUBECONFIG=<tempdir>/kubeconfig, the trap removes the file on exit. PRD 04 §“Kubeconfig options for SSH backend” calls this “Option A” — scp-and-cleanup. We picked it over the in-memory <() process-substitution alternative because it’s robust across remote shells and sshd configs.

Env propagation: SetEnv vs wrapper script

OpenSSH supports two ways to pass an env var to a remote command:

1. ssh -o SetEnv=KEY=VALUE target … — client tells the server “please add this to the env”. Works only if the server’s sshd_config has AcceptEnv KEY matching. Most stock sshd configs don’t enable AcceptEnv for arbitrary keys.
2. Wrapper script with export KEY=VALUE — the script writes the env var into its own process before exec "$@". Works regardless of sshd config, but the value lives briefly in a 0700 file on the remote.

The SSH backend tries SetEnv first. On the first connect to a new target, it sends a sentinel env var (ROKSBNKCTL_SETENV_TEST=ok) and runs echo "$ROKSBNKCTL_SETENV_TEST". If the output is ok, SetEnv works on this target — the result is cached in workspace state, and subsequent runs use SetEnv directly.

If the sentinel doesn’t surface, sshd silently dropped it (it logs refused setenv request on the server side, but clients don’t see that). The backend falls back to a wrapper script:

#!/bin/sh
# /tmp/roksbnkctl.<rand>/wrap.sh, mode 0700, owner-readable only
trap 'rm -f "$0"' EXIT
set +o history
export IBMCLOUD_API_KEY='<value>'
exec "$@"

Then: ssh <target> /tmp/roksbnkctl.<rand>/wrap.sh ibmcloud iam oauth-tokens.

The wrapper-script path is the Sprint 1 validator Issue 4 carry-over — the same shape --on uses for env passing today. Risks (file content includes the secret) are mitigated by:

• Mode 0700 so only the SSH user can read.
• set +o history so the value doesn’t leak into shell history.
• trap 'rm -f "$0"' EXIT deletes the wrapper as soon as it exits — including on Ctrl-C, since the trap covers SIGINT/SIGTERM by virtue of being in the script’s main process.
• The key is never in argv, so ps -ef on the remote doesn’t show it.

roksbnkctl targets show <name> reports which mechanism the target uses (e.g., env propagation: SetEnv (AcceptEnv ok) or env propagation: wrapper script (sshd refused SetEnv)) so users can choose to enable AcceptEnv server-side if they prefer.

Bootstrap failure modes (consolidated)

When to use it

• You’re running ibmcloud from a customer-firewalled office where the corporate jumphost can reach IBM Cloud APIs but your laptop can’t.
• You’re working in an air-gapped environment where roksbnkctl runs on your laptop but the IBM Cloud API conversations have to happen from a specific bastion’s IP.
• You want a low-overhead remote-exec path that doesn’t require a cluster (the k8s backend’s prereq).

When ssh is the wrong call:

• The target lacks the tool and you don’t want to mutate it. Skip --bootstrap; the backend errors clearly without installing anything.
• The target isn’t Ubuntu and you don’t want to pre-install. Bootstrap won’t work; pre-install or use local/docker/k8s.
• You’re running iperf3 to measure cluster bandwidth. SSH puts the client somewhere on the network path to the cluster but not necessarily adjacent to it — k8s is the right answer for that case.

Chapter 16 covers the lighter-weight --on jumphost predecessor that uses the same targets: config block. The SSH backend is the heavier-duty form: file materialisation, env propagation hardening, opt-in bootstrap. Chapter 18 is the decision tree.

The Backend interface

For the curious, the Go interface every backend conforms to:

package exec

type Backend interface {
    Run(ctx context.Context, argv []string, opts RunOpts) (int, error)
    Name() string
}

type RunOpts struct {
    Stdin           io.Reader
    Stdout, Stderr  io.Writer
    Env             []string         // KEY=VALUE pairs
    WorkDir         string           // best-effort; some backends ignore (k8s)
    TTY             bool             // request PTY where supported
    Files           map[string][]byte // files materialized at exec time
    Credentials     *Credentials     // routed via PRD 04's per-backend mechanism
}

type Credentials struct {
    KubeconfigBytes []byte
    IBMCloudAPIKey  string
}

All four implementations satisfy this interface. Call sites in cli/cluster.go, cli/test.go, etc., get a Backend from the registry and call Run(...) — no branching on backend type. The uniformity is what makes the system extensible without rewriting callers each time a backend lands.

The Credentials struct is the bridge between the resolver chain (env → keychain → config-b64 → prompt) covered in Chapter 14 and the per-backend propagation rules in PRD 04. Each backend translates the struct into the mechanism appropriate to where it runs: env vars for local, --env KEY (no =value) for docker, secretKeyRef for k8s, SetEnv or wrapper script for ssh.

Cross-references

• PRD 03 — pluggable execution backends — the design rationale and full per-backend spec.
• PRD 04 — credential propagation — the cred-passing rules every backend implements.
• Chapter 12 — Workspace config — the exec: block schema.
• Chapter 14 — Credentials and the resolver chain — how creds reach the backend in the first place.
• Chapter 16 — The –on flag and SSH jumphosts — the lightweight remote-exec predecessor to the SSH backend.
• Chapter 18 — Choosing a backend per tool — the decision tree.
• Chapter 19 — The in-cluster ops pod — deploy-time mechanics for the k8s backend.

Choosing a backend per tool

Chapter 17 covered the mechanics of each backend. This chapter is the decision tree: given a tool and a scenario, which of local / docker / k8s / ssh is the right call.

If you’re searching for “which backend should I use”, you’ve landed on the right page.

The four backends in one line each

If you’re skimming, the cheat-sheet is:

• local when you have the tool installed and the host’s network identity is correct for the call.
• docker when you don’t have the tool and don’t want to install it, or you need a frozen version for CI.
• k8s when the call’s network position matters and the cluster is the right vantage point.
• ssh:<target> when the call needs to originate from a specific external host (a customer bastion, an air-gapped bridge).

The rest of this chapter is the longer version.

Per-tool default backends

Every tool has a default backend baked into roksbnkctl. Workspace config (exec: block) can override the default per workspace; --backend overrides for a single invocation.

The defaults reflect “what’s the right answer for the most common scenario”:

• ibmcloud defaults to local because most users have it on PATH or are happy installing it. The compliance + firewall scenarios where ssh or docker are better are the minority of calls.
• iperf3 defaults to k8s because throughput from a laptop’s uplink isn’t the cluster’s bandwidth. The k8s backend places the iperf3 client in (or adjacent to) the cluster so the number reflects fabric, not Wi-Fi. Laptop-uplink-to-cluster is a real measurement too, but it’s the special case — opt in via --backend local.
• terraform defaults to local because the terraform-exec local path is the established workflow. State handling is simplest there. Frozen-version CI runs use --backend docker; non-local network-locality use cases (cluster-side, SSH-bastion-side) are deferred to a future release pending a state-handling design — see PRD 03 §“State concerns”.

To change a default per workspace, edit ~/.roksbnkctl/<workspace>/config.yaml:

exec:
  iperf3:    { backend: k8s }      # already the default; shown for clarity
  ibmcloud:  { backend: ssh:bastion }
  terraform: { backend: docker }

Chapter 12 §“exec:” covers the schema. The --backend CLI flag overrides whatever is in exec: for a single invocation.

Per-tool supported-backend matrix

Not every tool supports every backend. The authoritative matrix at v1.0:

Legend:

• yes — supported; same surface command works on this backend.
• yes (default) — this backend is the per-tool default; pass --backend other to override.
• not supported — rejected at CLI parse time with a clear error pointing at the right alternative.
• deferred to v1.x — a real design constraint, not a gap; see the cell text and PRD 03 §“State concerns”.
• internalised — roksbnkctl performs the operation via its own embedded library, not by shelling out; no backend selection applies.

The “no” entries are intentional design decisions, not gaps:

• iperf3 over docker is rejected because a Docker container running locally has the same network identity as the host — same NAT egress, same uplink, same observed bandwidth as --backend local. The user’s mental model would be “I picked docker, so the iperf3 must be hermetic now” but the throughput number wouldn’t actually differ. Better to refuse and force the user to pick local (deliberate laptop measurement) or k8s (cluster measurement).
• DNS probe over docker is rejected for the same reason. DNS resolution from a Docker container with default bridge networking goes through the same resolver as the host. There’s no GSLB-relevant network-locality difference. The probe subcommand errors with “DNS probe doesn’t benefit from docker; use local instead” when --backend docker is passed.
• terraform over k8s and ssh is deferred to v1.x. The state file is sensitive (admin tokens, generated TLS keys, license bundles); moving it into a Kubernetes Secret or scp’ing it pre/post-run requires a state-handling design that hasn’t shipped yet. PRD 03 §“State concerns” lays out the considerations; the roadmap entry lives in docs/PLAN.md §“What’s deliberately deferred to post-v1.0”.

Passing an unsupported (tool, backend) pair errors at the CLI layer before the backend is invoked:

$ roksbnkctl test throughput --backend docker
error: iperf3 doesn't support backend `docker` (same network identity as `local`,
       no value-add); supported: local, k8s, ssh:<target>

Decision tree

Pick the question that matches your scenario.

“I want to measure cluster bandwidth”

Use --backend k8s. The default for iperf3 is already k8s — the explicit flag is redundant unless you’ve overridden the default in workspace config:

roksbnkctl test throughput
# equivalent to:
roksbnkctl test throughput --backend k8s

The k8s backend deploys a server-side Deployment + LoadBalancer Service in roksbnkctl-test, runs the iperf3 client as a one-shot Job in the same namespace, collects the JSON output from the client pod’s logs, and tears down both. The bandwidth number reflects the cluster fabric.

If you instead want to measure your laptop’s uplink to the cluster:

roksbnkctl test throughput --backend local --endpoint <cluster-LB-ip>:5201

That’s a deliberately different measurement — useful when you suspect office Wi-Fi, not cluster fabric, is the bottleneck.

“I’m doing GSLB DNS validation”

Use both local and k8s. F5 BIG-IP Next’s GSLB returns different answers depending on the requesting resolver’s IP — geographic affinity, datacenter routing, health-check state. To validate that the GSLB is actually doing this, query from multiple network vantage points and compare.

The multi-vantage probe ships at v1.0 via roksbnkctl test dns --gslb-compare:

roksbnkctl test dns \
  --target www.example.com \
  --type A \
  --server gslb-vip.f5.example.com \
  --gslb-compare

--gslb-compare fans out to every configured vantage (local for your office IP, k8s for the cluster’s egress IP, ssh:<region-bastion> for a bastion in another region) in parallel and emits a single comparison JSON with a gslb_divergence boolean. Different answers across vantages are expected in a healthy GSLB; identical answers might mean the GSLB rules aren’t taking effect for the resolver positions you queried from.

Chapter 21 — DNS testing for GSLB is the full reference.

“I need to run ibmcloud from a customer-firewalled office”

Use --backend ssh:<bastion>. Your customer’s network policy lets the corporate jumphost reach *.cloud.ibm.com but blocks your laptop. The SSH backend ships your kubeconfig to the bastion (single file, mode 0600, removed via trap on session exit), runs ibmcloud there, streams the output back:

roksbnkctl ibmcloud --backend ssh:bastion ks cluster ls

If ibmcloud isn’t installed on the bastion, you’ll get a clear error:

error: tool `ibmcloud` not found on ssh target bastion; re-run with --bootstrap to install
       via apt-get, or pre-install on the target manually

Re-run with --bootstrap if you want roksbnkctl to sudo apt-get install -y ibmcloud-cli on the bastion. The opt-in default reflects “we don’t surprise users with sudo apt-get on a remote they didn’t expect mutation on” — see Chapter 17 §“SSH backend” for the bootstrap mechanics.

“I’m in CI and want a frozen toolchain version”

Use --backend docker. The vendored images are tagged in lock-step with roksbnkctl releases — ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:v1.0.0 is the exact same ibmcloud binary every CI run sees, regardless of when the runner image was built or what apt-get happens to ship that day:

roksbnkctl ibmcloud --backend docker iam oauth-tokens
roksbnkctl up --backend docker     # terraform inside hashicorp/terraform:<v>

For CI specifically, also pin ibmcloud.api_key_source: env in workspace config so the API key resolution is unambiguous (no keychain fallback to confuse a non-interactive runner) — see Chapter 14 §“Pinning a single source”.

“I’m on a clean dev machine without ibmcloud installed”

Use --backend docker. No apt-get install ibmcloud-cli, no IBM repo + GPG key dance, no upstream-package-version mismatch — docker pull ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:dev is the only setup, and roksbnkctl does that for you on first invocation.

Alternatively, if your laptop is the dev machine and you’ll run ibmcloud more than once, just install it. The local backend has lower per-invocation startup latency than docker (no container create/start/log-attach), so once you’ve paid the install cost the local path is faster for the rest of the session.

“I want a cluster-side ad-hoc shell”

Use --backend k8s with the long-lived ops pod. Once roksbnkctl ops install has run, --backend k8s for ibmcloud (or any future tool) routes through kubectl exec -n roksbnkctl-ops ops -- <argv>. The pod stays alive between invocations, so the second and subsequent commands skip pod-startup latency.

roksbnkctl ops install
roksbnkctl ibmcloud --backend k8s iam oauth-tokens
roksbnkctl ibmcloud --backend k8s ks cluster ls
roksbnkctl ibmcloud --backend k8s account list

Chapter 19 is the full reference for the ops pod lifecycle.

“I’m pre-cluster — there’s no cluster yet”

Use local or ssh:<target>. The k8s backend prereq is a working kubeconfig pointing at a running cluster; before roksbnkctl up has succeeded, that doesn’t exist. For pre-cluster ibmcloud + terraform calls (account inspection, IAM tinkering, the cluster-create itself), local and ssh:bastion are the only two paths.

When not to use a backend

Common foot-guns, in rough order of how often they come up:

--backend k8s without roksbnkctl ops install

The ops pod must exist before the k8s backend can route ibmcloud calls through it. First-time use:

roksbnkctl ops install         # one-time setup per cluster
roksbnkctl ibmcloud --backend k8s ks cluster ls

If you skip the install, the backend errors with a clear remediation:

error: ops pod not found in roksbnkctl-ops namespace; run `roksbnkctl ops install` first

Chapter 19 covers the install/show/uninstall lifecycle.

--backend docker for a network-locality test

iperf3 and the DNS probe both reject --backend docker because a local Docker container has the same network identity as the host (default bridge networking). The probe wouldn’t measure anything different. The CLI errors at parse time:

$ roksbnkctl test throughput --backend docker
error: iperf3 doesn't support backend `docker` (same network identity as `local`,
       no value-add); supported: local, k8s, ssh:<target>

If you actually want a hermetic-tools throughput test, --backend k8s is the right answer.

--backend ssh:host without --bootstrap on a fresh target

If ibmcloud (or iperf3) isn’t installed on the target, the SSH backend won’t silently sudo apt-get for you — --bootstrap is opt-in. The first call on a fresh target tells you exactly what’s needed:

error: tool `ibmcloud` not found on ssh target bastion; re-run with --bootstrap to install
       via apt-get, or pre-install on the target manually

Re-run with --bootstrap if mutation is OK; otherwise pre-install via your config-management of choice (Ansible, Salt, baked-in-image).

--backend ssh:host to a non-Ubuntu target with --bootstrap

The apt-bootstrap recipe is Ubuntu-only this round. RHEL / CentOS / Alpine targets need pre-installation via yum / dnf / apk — --bootstrap errors out cleanly:

error: auto-install only supports Ubuntu. Pre-install `ibmcloud-cli` on the target
       (RHEL: `yum install ibmcloud-cli`)

Once the tool is installed, --backend ssh:host works without --bootstrap.

--backend k8s for terraform

Deferred to v1.x. The terraform tool’s k8s + ssh backends require a state-handling design that hasn’t shipped — moving the state file into a Kubernetes Secret or scp’ing it pre/post-run is fiddly enough to be a feature in its own right (PRD 03 §“State concerns”; roadmap in docs/PLAN.md §“What’s deliberately deferred to post-v1.0”). For now, terraform supports local and docker only. If the network-locality use case (running terraform from a customer VPC for IP-egress reasons) is blocking, file an issue.

Mixing --on and --backend ssh:<target>

--on <target> is the Chapter 16 lightweight remote-exec — it runs the passthrough shape (exec, shell, kubectl, oc, ibmcloud) on the target by literally re-running the command via SSH. --backend ssh:<target> is the heavier-duty form — it routes through the Backend interface, which means file materialisation, env propagation hardening, opt-in apt-bootstrap, and the redactor are all wired in.

You generally want one or the other, not both. The supported precedence is “--backend ssh:<target> wins”; passing both flags on the same invocation surfaces a warning. If you’re calling roksbnkctl ibmcloud …, prefer --backend ssh:<target> for the same target — you get the better cred-handling story automatically.

Workspace config + --backend flag interaction

Recap of Chapter 12 §“exec:”:

The flag wins. If ~/.roksbnkctl/<ws>/config.yaml says:

exec:
  iperf3: { backend: k8s }

…and you run roksbnkctl test throughput --backend local, the local backend runs. The flag is the per-invocation override; the workspace config is the per-workspace default.

If neither is set, the per-tool default from the previous section applies (iperf3 → k8s, ibmcloud → local, terraform → local). The resolution order is exact:

1. --backend flag
2. exec.<tool>.backend in workspace config
3. Per-tool baked-in default

There’s no fallback chain inside this resolution — if you pass --backend k8s and the cluster is unreachable, the backend errors with “cluster API unreachable” (exit 127). It does not fall through to local. Silent fallback hides intent and produces confusing CI results; the failure-mode discipline in Chapter 17 §“Backend-failure semantics” applies here too.

Summary table

The decision-tree contents collapsed into one table:

Worked example: bare-metal + jumphost office workflow

End-to-end Part V scenario: you’re an F5 SE running a customer POC from a corporate-firewalled office. The laptop can’t reach *.cloud.ibm.com directly (the office proxy blocks it) but a customer-provisioned Ubuntu jumphost at 10.20.30.40 can. The jumphost was already auto-discovered by an earlier roksbnkctl up against this customer’s account, so targets list shows it. You need to: install the in-cluster ops pod, run ibmcloud from the bastion, and run a throughput test from inside the cluster — all without installing tools locally.

# 1. Verify jumphost is registered + reachable
$ roksbnkctl targets list -w customer
NAME       HOST          KEY_SOURCE         STATUS
jumphost   10.20.30.40   workspace/state    reachable

# 2. Run ibmcloud from the jumphost (Sprint 1 --on flag, lightweight)
$ roksbnkctl ibmcloud --on jumphost ks cluster ls
OK
Name              ID                                     State    Created     ...
customer-cluster  c4abc123def456                         normal   3 days ago  ...

# 3. For the same call routed through the Backend interface (cred-handling
# hardened, redactor wired, opt-in apt-bootstrap available), use --backend
$ roksbnkctl ibmcloud --backend ssh:jumphost ks cluster ls
# Same output; different code path. Prefer --backend ssh:<target> for
# everything except quick interactive shells where --on is faster to type.

# 4. Install the in-cluster ops pod (one-time per cluster)
$ roksbnkctl ops install
✓ Namespace roksbnkctl-ops created
✓ ServiceAccount + Role + RoleBinding applied
✓ Secret roksbnkctl-ibm-creds applied (envFrom secretRef)
✓ Pod roksbnkctl-ops Running (2.3s)

# 5. Same ibmcloud call routed through the ops pod (k8s backend)
$ roksbnkctl ibmcloud --backend k8s iam oauth-tokens
IAM token: Bearer eyJ...
# (The token comes from inside the cluster — different egress IP from the
# jumphost's, useful when IAM policy is IP-conditional.)

# 6. Throughput test using the cluster vantage (default for iperf3)
$ roksbnkctl test throughput
→ Deploying iperf3 server pod into namespace "roksbnkctl-test"
✓ Pod ready (iperf3-server-...)
→ Deploying iperf3 client Job in the same namespace
✓ Client Job complete
✓ throughput: 8.92 Gbits/sec (mean over 10s)
→ Tearing down iperf3 fixture
✓ pod, service, and Job deleted

# 7. Throughput test from the jumphost into the cluster (north-south, real
# customer-network bandwidth — not laptop wifi)
$ roksbnkctl test throughput --backend ssh:jumphost --mode north-south
✓ throughput: 936 Mbits/sec  (jumphost → cluster LB; customer's WAN)

# 8. Persist the per-tool routing in workspace config (one-time)
$ cat >> ~/.roksbnkctl/customer/config.yaml <<'YAML'
exec:
  ibmcloud:  { backend: ssh:jumphost }
  iperf3:    { backend: k8s }
  terraform: { backend: local }
YAML
# Subsequent runs skip the --backend flag — every ibmcloud call routes via
# the jumphost automatically; every iperf3 runs in-cluster.

The point of this walkthrough: with no tools installed locally beyond roksbnkctl itself, you’ve reached the IBM Cloud control plane from the customer’s bastion (compliance-correct egress), exercised the cluster fabric for throughput, and persisted the routing per workspace. The same laptop with the same roksbnkctl binary handles a different customer’s POC by pointing at a different workspace; nothing on the laptop is workspace-specific.

Chapter 19 covers the ops-pod lifecycle in detail; Chapter 22 covers the north-south vs east-west modes.

Cross-references

• Chapter 12 — Workspace config — the exec: block schema.
• Chapter 14 — Credentials and the resolver chain — how creds reach each backend.
• Chapter 16 — The --on flag and SSH jumphosts — the lightweight remote-exec predecessor to the SSH backend.
• Chapter 17 — Execution backends — per-backend mechanics.
• Chapter 19 — The in-cluster ops pod — the cluster-side prerequisite for --backend k8s.
• Chapter 22 — Throughput testing — iperf3-specific flags.
• PRD 03 — pluggable execution backends — the design spec.

The in-cluster ops pod

The k8s execution backend has two execution patterns: a long-lived ops pod for ad-hoc commands, and one-shot Jobs for throughput tests, DNS probes, and other per-invocation workloads. Chapter 17 §“K8s backend” covered the interface mechanics — how Backend.Run dispatches into either pattern.

This chapter is the reference for the pod itself: what roksbnkctl ops install deploys, what RBAC it grants, where credentials live, how to rotate them, and how to debug when something goes wrong.

If you’ve never run roksbnkctl ops install, you can read this chapter front-to-back; otherwise the § Operability section near the end is the troubleshooting jump-off point.

What the ops pod is

A long-lived pod in the roksbnkctl-ops namespace, running an image bundled with the tools roksbnkctl may want to invoke cluster-side: ibmcloud CLI plus kubectl as a fallback, with oc and terraform reserved for future iterations.

The pod sits idle waiting for kubectl exec calls. Each roksbnkctl ibmcloud --backend k8s … invocation routes through client-go’s SPDY executor, runs the wrapped tool inside the existing pod, streams stdout/stderr back, and returns the exit code. No pod create/start latency between invocations — a session of twenty ibmcloud commands pays the startup cost once.

Compared to the one-shot Job pattern (used for iperf3 and the DNS probe), the ops pod trades a bit of resource-usage idle-state for substantially lower per-call latency. It’s the right shape when you want to debug interactively or run many small commands.

roksbnkctl ops install

Idempotent setup. Run once per cluster; re-run any time you want to refresh the image, rotate the API key Secret, or recover from a partial uninstall.

roksbnkctl ops install

What it does, step by step:

1. Create the namespace

apiVersion: v1
kind: Namespace
metadata:
  name: roksbnkctl-ops
  labels:
    app.kubernetes.io/name: roksbnkctl
    app.kubernetes.io/component: ops-pod

The roksbnkctl-ops namespace is dedicated to the long-lived pod. Separate from roksbnkctl-test (where one-shot Jobs run) so RBAC can be scoped per namespace — see § RBAC below.

2. Create the ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: roksbnkctl-ops
  namespace: roksbnkctl-ops

The pod runs as this SA. Its projected token is auto-mounted at /var/run/secrets/kubernetes.io/serviceaccount/, which is what the bundled kubectl uses for in-cluster authentication. The IBM Cloud API key (a separate credential) reaches the pod through a Kubernetes Secret — see § Credential propagation below.

3. Create the ClusterRole + ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: roksbnkctl-ops
rules:
- apiGroups: [""]
  resources: ["pods", "pods/exec", "pods/log"]
  verbs:     ["get", "list", "watch", "create", "delete"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs:     ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs:     ["get", "list"]
  resourceNames: ["roksbnkctl-ibm-creds"]
- apiGroups: [""]
  resources: ["services"]
  verbs:     ["get", "list", "create", "delete"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs:     ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs:     ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: roksbnkctl-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: roksbnkctl-ops
subjects:
- kind: ServiceAccount
  name: roksbnkctl-ops
  namespace: roksbnkctl-ops

The full manifest lives at internal/exec/k8s_install.yaml (embedded into the binary). § RBAC walks through what each rule is for.

4. Create or update the credential Secret

v1.2+ note. This step describes the static-key Secret applied under --trusted-profile=off and under the auto-fallback when IAM perms don’t allow the trusted-profile path. Under --trusted-profile=auto success the Secret is still applied with empty data fields (placeholder so envFrom: secretRef always resolves); the cred propagation happens via the trusted-profile annotation on the SA + the IAM_PROFILE_ID env var instead. See §“Trusted-profile flow (v1.2+)” below.

apiVersion: v1
kind: Secret
metadata:
  name: roksbnkctl-ibm-creds
  namespace: roksbnkctl-ops
  annotations:
    helm.sh/resource-policy: keep            # don't sweep on accidental destroy
type: Opaque
stringData:
  IBMCLOUD_API_KEY: <resolved-key-value>

The key value comes from the workspace’s resolver chain (env → keychain → config-b64 → prompt) — see Chapter 14 for the resolution order. The Secret carries two keys (IBMCLOUD_API_KEY and the legacy alias IC_API_KEY) both populated from the same resolved value, so older ibmcloud CLI versions that look for the IC_ name find it.

If the Secret already exists (re-running ops install after a key rotation), roksbnkctl does a client-side Get + Update: the Secret’s data is overwritten with the freshly resolved value, the roksbnkctl.io/rotated-at annotation is stamped with the current timestamp, and the rest of the Secret’s metadata is left untouched. roksbnkctl ops show surfaces last cred rotation: <timestamp> by reading that annotation.

5. Create the Pod

apiVersion: v1
kind: Pod
metadata:
  name: roksbnkctl-ops
  namespace: roksbnkctl-ops
  labels:
    app: roksbnkctl-ops
    roksbnkctl.io/managed: "true"
spec:
  serviceAccountName: roksbnkctl-ops
  restartPolicy: Always
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: tools
    image: ${OPS_IMAGE}                  # resolved from roksbnkctl's version at install time
    imagePullPolicy: IfNotPresent
    command: ["sleep", "infinity"]
    envFrom:
    - secretRef:
        name: roksbnkctl-ibm-creds
    securityContext:
      allowPrivilegeEscalation: false
      runAsNonRoot: true
      capabilities:
        drop: ["ALL"]
    resources:
      requests: { cpu: 50m,  memory: 64Mi }
      limits:   { cpu: 500m, memory: 256Mi }

Three details to call out:

• command: ["sleep", "infinity"] — the pod’s own command. Each Backend.Run invocation issues a kubectl exec against this idle process, which means the pod’s main process never exits as long as the pod is healthy.
• securityContext is set explicitly for OpenShift’s restricted-v2 SCC. Pod-level runAsNonRoot + seccompProfile.type: RuntimeDefault; container-level allowPrivilegeEscalation: false + capabilities.drop: [ALL] + runAsNonRoot — the same fields the iperf3 server pod sets, for the same reason.
• envFrom: secretRef — the API key reaches the pod’s env without ever touching the pod manifest’s argv or env: block. kubectl describe pod roksbnkctl-ops shows the secret reference name but not the value, per PRD 04 §“In-cluster pod”.

6. Wait for readiness

roksbnkctl ops install waits for Pod.Status.Phase == Running and the container’s Ready condition before returning. Default timeout is 60 seconds; longer for clusters with slow image pulls (the ghcr.io image is ~80 MiB). Failures surface a kubectl describe pod roksbnkctl-ops excerpt for context.

Trusted-profile flow (v1.2+)

The static-key Secret described above is the v1.0.x / v1.1.x path. In v1.2.0 it becomes the fallback: the default ops install invocation auto-provisions an IBM Cloud IAM trusted profile linked to the ops pod’s ServiceAccount, and the static API key no longer needs to land in any Kubernetes Secret. PRD 04 §“Resolved in Sprint 9” → “Trusted-profile auto-provisioning (k8s backend)” is the design reference; this section is the operational walkthrough.

v1.3.0 closes both the provisioning and the runtime sides of this flow — ops install --trusted-profile=auto provisions the profile, and the in-pod ibmcloud login wrap detects the SA’s trusted-profile annotation and authenticates via the projected SA token at runtime. The v1.2.x partial-closure history (provisioning shipped, runtime deferred) is preserved in CHANGELOG v1.3.0 → ### Changed for readers who specifically want the chronology.

roksbnkctl ops install --trusted-profile=auto

--trusted-profile=auto is the default as of v1.2 — running roksbnkctl ops install with no flag picks the auto path. Naming the flag explicitly is useful in scripts that pin behaviour or in docs that want to read unambiguously:

$ roksbnkctl ops install --trusted-profile=auto
✓ Provisioned IAM trusted profile roksbnkctl-ops-canada-roks (iam-Profile-9f2…)
✓ created namespace roksbnkctl-ops
✓ created sa roksbnkctl-ops/roksbnkctl-ops
✓ created secret roksbnkctl-ops/roksbnkctl-ibm-creds
✓ created clusterrole roksbnkctl-ops
✓ created crb roksbnkctl-ops
✓ created pod roksbnkctl-ops/roksbnkctl-ops
→ Waiting for ops pod to be Ready (60s timeout)
✓ Ops pod is Ready (trusted profile roksbnkctl-ops-canada-roks)

Re-runs against an existing install emit updated <kind> … / <kind> … exists instead of created for each resource that already matches the desired state. The trusted-profile provisioning line above is the single line internal/cli/ops.go emits for the whole IBM IAM-side flow (perm probe + profile create + compute-resource link + SA annotation) — the work happens silently inside resolveTrustedProfileForInstall; the one line you see is the receipt.

What just happened, in order (the binary doesn’t narrate these steps but they’re what’s actually going on):

1. IAM perm probe. ops install calls IBM IAM Identity to confirm the resolved API key has iam-identity perms. On 403, the flag value drives the next step: auto falls back to the static-key Secret with a warning (see §“--trusted-profile=auto falling back” below); on errors out with a non-zero exit.
2. Profile creation. Names the profile roksbnkctl-ops-<workspace> so multiple workspaces against the same IBM Cloud account don’t race for a single shared name. The compute-resource link binds the profile to your cluster’s OIDC issuer URL + the roksbnkctl-ops/roksbnkctl-ops ServiceAccount specifically — other SAs on the same cluster can’t assume the profile.
3. Policy attachment. v1.2 ships with no default policies attached — the profile inherits whatever IAM policies your account has set up for trusted profiles in general (typically nothing, until you grant). A future cycle will surface ibmcloud.trusted_profile.policies as a workspace-config block; tracked under v1.x deferred. If you need the profile to actually authorise specific actions (Container Registry pulls, Cloud Object Storage reads), grant the policies via IBM Cloud Console or ibmcloud iam trusted-profile-policy-create after ops install returns.
4. SA annotation. The ServiceAccount gets iam.cloud.ibm.com/trusted-profile: roksbnkctl-ops-<workspace> plus the roksbnkctl.io/trusted-profile-managed: "true" marker that signals ops uninstall to delete the profile during cleanup.
5. Pod creation. The pod’s container always has envFrom: secretRef: roksbnkctl-ibm-creds; what changes between modes is the Secret’s contents. Under --trusted-profile=auto success the Secret is created with empty data — IBMCLOUD_API_KEY is the empty string — plus an extra IAM_PROFILE_ID env var pointing at the provisioned profile’s ID, and a projected ServiceAccount-token volume mounted at /var/run/secrets/tokens/token (audience iam) so the pod has a cluster-issued JWT the IBM IAM endpoint will accept. The in-pod ibmcloud login wrap detects the SA’s trusted-profile annotation and runs ibmcloud login -a https://cloud.ibm.com --cr-token @/var/run/secrets/tokens/token --profile "$IAM_PROFILE_ID" -r "${IBMCLOUD_REGION:-us-south}" --quiet — the --cr-token @<path> form reads the projected SA token from disk; IBM IAM validates that JWT against the trusted profile’s ROKS_SA compute-resource link (the link internal/ibm/trusted_profile.go::ensureLink provisions). The static API key never transits the pod env. Under --trusted-profile=off (or the auto-fallback) the Secret carries the resolved API key, no projected token volume is mounted, and the wrap runs ibmcloud login --apikey "$IBMCLOUD_API_KEY" — the v1.0.x path.

Verifying the profile is in use

The ServiceAccount carries the truth-of-record annotation:

$ oc get serviceaccount roksbnkctl-ops -n roksbnkctl-ops -o yaml
# or, kubectl-equivalent via the bundled passthrough:
$ roksbnkctl k get sa roksbnkctl-ops -n roksbnkctl-ops -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.cloud.ibm.com/trusted-profile: Profile-ccba11f2-3b1f-4b1a-b8a4-aeed2b7b3320  # ← the IBM IAM Profile ID
    roksbnkctl.io/trusted-profile-managed: "true"                                     # ← ops uninstall will delete it
    roksbnkctl.io/provisioned-at: "2026-05-13T14:08:33Z"
  name: roksbnkctl-ops
  namespace: roksbnkctl-ops

End-to-end smoke test of the runtime cred flow:

$ roksbnkctl --backend k8s ibmcloud iam oauth-tokens
IAM token:  Bearer eyJ…

The token is fresh-each-call: the in-pod wrap detects the SA’s iam.cloud.ibm.com/trusted-profile annotation, trades the pod’s projected SA token for an IAM token against the trusted profile, and returns it to the caller. No static API key transits the pod env.

The first invocation may take 30–60 seconds after ops install returns because IBM IAM needs to pick up the cluster’s OIDC issuer URL before it will accept the projected SA token as proof for the profile. The wrap absorbs this with a 3-attempt × 20s-backoff retry — up to ~40s of waiting inside the wrap before it surfaces the failure. On triple-fail the wrap prints trusted-profile login failed after 3 attempts: <captured-stderr> to your terminal (the captured stderr will include the underlying ibmcloud login diagnostic — typically the “Unable to authenticate” / FAILED banner shape). If your first smoke test produces that line, give IAM a few more seconds and re-run. After the first successful call the wrap’s auth state is cached for the pod’s lifetime, so subsequent roksbnkctl --backend k8s ibmcloud <subcommand> invocations don’t re-pay the propagation window.

--trusted-profile=auto falling back

auto falls back to the v1.0.x static-key Secret when any of three pre-conditions for trusted-profile provisioning aren’t met. The warning prints first (the fallback decision is made before any cluster-side resource is applied), then the rest of the install proceeds with the v1.0.x static-key shape:

$ roksbnkctl ops install
warning: IAM perm 'iam-identity' missing; using static-key Secret. Pass `--trusted-profile=off` to silence.
✓ created namespace roksbnkctl-ops
✓ created sa roksbnkctl-ops/roksbnkctl-ops
✓ created secret roksbnkctl-ops/roksbnkctl-ibm-creds
✓ created clusterrole roksbnkctl-ops
✓ created crb roksbnkctl-ops
✓ created pod roksbnkctl-ops/roksbnkctl-ops
→ Waiting for ops pod to be Ready (60s timeout)
✓ Ops pod is Ready (static-key Secret)

The three warning shapes (in source order — internal/cli/ops.go resolveTrustedProfileForInstall):

All three are non-fatal; the install completes and the pod works exactly as it did in v1.0.x. The warnings are terse on purpose — the actionable detail belongs in this chapter, not in every stderr line. Three ways to clear them permanently:

• Run cluster up or cluster register first if the warning names the missing cluster registration. ops install re-run after the registration completes will detect the cluster and switch to the trusted-profile path.
• Ask your IAM admin to grant iam-identity Operator role on the API key (or use a different key that already has it) if the warning names the missing IAM perm. Re-run ops install — the install detects the changed perm posture on re-run and replaces the static-key Secret with a trusted-profile binding.
• Opt out via --trusted-profile=off (next subsection) if you don’t want the warning every install.

--trusted-profile=off

Explicit opt-out. Skips the IAM perm check entirely and provisions the v1.0.x static-key Secret:

$ roksbnkctl ops install --trusted-profile=off
✓ created namespace roksbnkctl-ops
✓ created sa roksbnkctl-ops/roksbnkctl-ops
✓ created secret roksbnkctl-ops/roksbnkctl-ibm-creds
✓ created clusterrole roksbnkctl-ops
✓ created crb roksbnkctl-ops
✓ created pod roksbnkctl-ops/roksbnkctl-ops
→ Waiting for ops pod to be Ready (60s timeout)
✓ Ops pod is Ready (static-key Secret)

Use cases:

• Reproducing v1.0.x behaviour exactly — for byte-for-byte parity tests against an older deployment, or for scripts whose assertions match the v1.0.x ops show output verbatim.
• Air-gapped clusters that can’t reach the IBM IAM API at runtime — without that connectivity, the pod can’t trade its projected SA token for an IAM token, so the trusted-profile path is non-functional regardless of perms.
• Cred rotation runbooks that already automate the static-key path and aren’t yet ready to switch to the projected-token model.

The third value, --trusted-profile=on, is the inverse — it forces the trusted-profile path and refuses to fall back on perm-missing, returning a non-zero exit with the same warning text. Use it in CI to surface IAM-perm regressions explicitly.

Cleanup on ops uninstall

roksbnkctl ops uninstall --confirm honors the roksbnkctl.io/trusted-profile-managed: "true" annotation on the SA and deletes the IBM Cloud trusted profile alongside the cluster-side objects:

$ roksbnkctl ops uninstall --confirm
✓ deleted trusted profile roksbnkctl-ops-canada-roks
✓ deleted pod roksbnkctl-ops
✓ deleted secret roksbnkctl-ibm-creds
✓ deleted serviceaccount roksbnkctl-ops
✓ deleted clusterrolebinding roksbnkctl-ops
✓ deleted clusterrole roksbnkctl-ops
✓ deleted namespace roksbnkctl-ops
✓ deleted namespace roksbnkctl-test

The trusted profile is deleted first, before the cluster-side objects, so even if the cluster API becomes unreachable mid-uninstall the IBM Cloud-side state isn’t left orphaned. The Secret is always deleted regardless of mode (it’s always rendered by ops install, just with empty data under --trusted-profile=auto success).

The trusted-profile delete is best-effort — if the calling user’s API key has lost iam-identity perms in the meantime (or the key itself has rotated and the new key doesn’t have those perms), the cluster-side objects still delete and a warning line is printed instructing the user to delete the profile manually via the IBM Cloud console. The annotation remains correct documentation of what was provisioned; roksbnkctl ops install on a fresh cluster will pick a fresh profile name unconditionally so an orphaned profile from a prior install doesn’t collide.

--trusted-profile=off installs leave no trusted profile to clean up — ops uninstall --confirm just deletes the cluster-side objects + the static-key Secret as it did in v1.0.x.

roksbnkctl ops show

Reports current state without making any changes:

$ roksbnkctl ops show
namespace:    roksbnkctl-ops
pod:          roksbnkctl-ops
phase:        Running
ready:        true
image:        ghcr.io/jgruberf5/roksbnkctl-tools-ibmcloud:v0.9.0
rbac subject: system:serviceaccount:roksbnkctl-ops:roksbnkctl-ops
trusted-profile: Profile-ccba11f2-3b1f-4b1a-b8a4-aeed2b7b3320
secret:       roksbnkctl-ibm-creds (rotated 2026-05-10T11:03:17Z)

What each line surfaces:

1. Pod phase + readiness — Running + true is green; anything else means the pod is unhealthy and Backend.Run calls will fail. The container count is exactly one (tools); ready: true is a single bool, not a 2/2-style ratio.
2. Image — the :v… tag matches the roksbnkctl release the image was published with (resolved at install time from the binary’s version; see Chapter 17 §:dev tag resolution). Mismatched against your roksbnkctl --version means re-running ops install will pull the matching image.
3. RBAC subject — the SA the pod runs as. kubectl describe clusterrole roksbnkctl-ops prints the full ruleset (the ClusterRoleBinding is named the same as the role).
4. Trusted-profile line — reads the SA’s iam.cloud.ibm.com/trusted-profile annotation. The value is the IBM IAM Profile ID (Profile-<uuid>, the canonical IAM identifier — runOpsInstall annotates with tp.ID rather than the friendly roksbnkctl-ops-<workspace> name so the value is grep-friendly against IBM Cloud IAM audit logs; the parenthetical form on the ✓ Provisioned IAM trusted profile … install line cross-references both). Present + non-empty means ops install ran with --trusted-profile=auto or =on and the provisioning succeeded; the runtime cred flow is going via the projected SA token. Under --trusted-profile=off (or auto-fallback when IAM perms were missing) the line reads trusted-profile: (none — static-key Secret path).
5. Secret line — the cred Secret’s name + the roksbnkctl.io/rotated-at annotation that ops install stamps each time the Secret is applied. Always emitted (the Secret manifest is always rendered — empty data under trusted-profile success, populated under the static-key paths). If the Secret resource is missing entirely on the cluster, the line reads secret: (missing: …).

The current output is a fixed seven-line key/value block; a structured --output json mode is on the v1.x roadmap once ops show grows additional fields (image-id hash, env-hash reconciliation against the live pod, etc.). See docs/PLAN.md §“What’s deliberately deferred to post-v1.0”.

roksbnkctl ops uninstall

Full removal. Run when decommissioning the cluster, or when you want a clean re-install. The command is a destructive-action gate: by default it prints a preview of what would be deleted and exits successfully; the actual deletion only runs with --confirm.

$ roksbnkctl ops uninstall
Would delete (re-run with --confirm to proceed):
  - Pod        roksbnkctl-ops/roksbnkctl-ops
  - Secret     roksbnkctl-ops/roksbnkctl-ibm-creds
  - ServiceAccount roksbnkctl-ops/roksbnkctl-ops
  - ClusterRole/ClusterRoleBinding roksbnkctl-ops
  - Namespace  roksbnkctl-ops
  - Namespace  roksbnkctl-test

$ roksbnkctl ops uninstall --confirm
✓ deleted pod roksbnkctl-ops
✓ deleted secret roksbnkctl-ibm-creds
✓ deleted serviceaccount roksbnkctl-ops
✓ deleted clusterrolebinding roksbnkctl-ops
✓ deleted clusterrole roksbnkctl-ops
✓ deleted namespace roksbnkctl-ops
✓ deleted namespace roksbnkctl-test

Note that the cluster-scoped objects (ClusterRole, ClusterRoleBinding) get cleaned too — they’re not garbage-collected by namespace deletion since they live above the namespace. roksbnkctl ops uninstall --confirm makes this explicit so a stale roksbnkctl-ops ClusterRole can’t outlive a namespace removed via kubectl delete ns.

Both managed namespaces (roksbnkctl-ops and roksbnkctl-test) are deleted. The roksbnkctl-test namespace is where one-shot Job pods (iperf3 client, future probes) land — by the time you’re running uninstall you’ve already concluded those test workloads are finished, so removing the namespace alongside the ops-pod surface keeps the cluster clean.

When to run uninstall:

• Cluster decommission — the cluster is going away, clean up cluster-scoped objects before destroying it.
• Cred rotation when paranoid — the rotation story (next section) doesn’t require uninstall, but if you’re worried about old secrets persisting in etcd snapshots, an uninstall + re-install regenerates the Secret cleanly.
• Image upgrade with a major manifest change — if the embedded k8s_install.yaml evolves (new RBAC rule, security-context tweak), uninstall + install is the cleanest way to apply.

RBAC: the ClusterRole rules

The full ClusterRole rule set (transcribed from internal/exec/k8s_install.yaml):

Notably not granted:

• pods create / delete — the SA can list and watch pods but can’t create or delete them. Pod lifecycle is mediated by Jobs (which the SA does manage) and by the ops pod itself (which ops install creates with the user’s privilege, not the SA’s).
• secrets create / update / delete / list — the pod never writes Secrets, and can’t even list to discover which Secrets exist in the namespace. The install-time Secret creation is done by the user invoking ops install (whose kubeconfig has cluster-admin or comparable), not by the pod’s SA. Combined with the resourceNames filter on get, this is the tightest practical surface that still lets the pod consume its own cred.
• services, deployments, namespaces — the SA can’t touch these at all. The iperf3 server fixture (when the throughput test runs) is provisioned by roksbnkctl test throughput running on the caller’s side using the user’s kubeconfig, not by anything inside the ops pod.
• clusterroles, clusterrolebindings — the pod never modifies its own RBAC.
• * cluster-admin — explicitly avoided. The pod has exactly the verbs it needs and nothing else.

This matches PRD 04 §“Least privilege per backend” and PRD 03 §“K8s”: the ops pod is a powerful tool but its blast radius is bounded.

To audit the rules on a running cluster:

kubectl describe clusterrole roksbnkctl-ops
kubectl auth can-i --as=system:serviceaccount:roksbnkctl-ops:roksbnkctl-ops \
  '*' '*' --all-namespaces       # should print mostly "no"

Credential propagation

v1.2+ note. What follows is the static-key propagation path. As of v1.2 it’s the fallback rather than the default — --trusted-profile=auto installs assume an IBM Cloud trusted profile via the pod’s projected SA token and the static API key never lands in a Kubernetes Secret. See §“Trusted-profile flow (v1.2+)” above for that path. The hop-by-hop description below still describes what happens under --trusted-profile=off (and under the auto-fallback when IAM perms don’t allow the trusted-profile path).

The IBMCLOUD_API_KEY reaches the wrapped tool in three hops:

resolver chain (env → keychain → config-b64 → prompt)
       ↓                        on the laptop, at `roksbnkctl ops install` time
  Kubernetes Secret roksbnkctl-ibm-creds                in roksbnkctl-ops namespace
       ↓                        applied by `ops install` via kubectl-equivalent
  Pod env (IBMCLOUD_API_KEY=…)                          via `envFrom: secretRef`
       ↓                        kubelet reads Secret, sets env on container start
  Wrapped tool (`ibmcloud iam oauth-tokens`)            reads from os.Getenv

Three properties this gives you:

1. The key never appears in argv. kubectl describe pod roksbnkctl-ops shows envFrom: secretRef: name: roksbnkctl-ibm-creds, not the value. kubectl get pod roksbnkctl-ops -o yaml shows the same.
2. The key never appears in the pod’s own logs. The wrapped tool uses the env var; the env var name (not value) is what the pod’s startup logs print.
3. The redactor is the defense-in-depth backstop. If the wrapped tool ever prints the value (e.g., ibmcloud --debug), the SPDY stream from the pod is wrapped through internal/exec/redact.go before reaching the caller’s stdout — same as the local + docker backends.

The Secret carries two keys today — IBMCLOUD_API_KEY and the legacy IC_API_KEY alias older ibmcloud versions accept — both populated from the same resolved value. Names are stable; embedded in internal/exec/k8s_install.yaml. Future cluster-side credentials (an AWS access key, a GCP service-account JSON) will add new keys to the same Secret rather than spinning up new Secrets, simplifying RBAC.

Rotation: rotating the API key

v1.2+ note. Under --trusted-profile=auto / =on (default), there’s nothing to rotate — the ops pod’s IAM tokens are short-lived and the IBM IAM endpoint refreshes them transparently each time the SDK trades the projected SA token. Key rotation only matters when the install ran with --trusted-profile=off or fell back to the static-key Secret because the resolved key lacked iam-identity perms. The procedure below covers that static-key case.

When the IBM Cloud API key changes (key rotation, account takeover, key compromise), you need to update the cluster-side Secret. The flow:

# 1. Update the local resolver chain — pick whichever source you populated
#    initially (the chain order is: env > keychain > config-b64; see chapter 14):
export IBMCLOUD_API_KEY=<new-key>             # env (one-shot)
# or update the keychain entry directly: `keyring` / `secret-tool` / Keychain.app
# or edit ~/.roksbnkctl/<workspace>/config.yaml's api_key_b64 field

# 2. Re-run ops install — this re-resolves the key, updates the cluster
#    Secret, and rolls the pod
roksbnkctl ops install

What ops install does on re-run:

• The Secret roksbnkctl-ibm-creds is updated with the new value via a client-side Get + Update (the existing Secret’s data is overwritten, the roksbnkctl.io/rotated-at annotation is refreshed, the rest of the metadata is left alone).
• The pod’s env, however, is set at container-start time — kubelet reads the Secret value when the pod is created, not on every Secret update. So an updated Secret doesn’t propagate to the running pod’s env until the pod is recreated.
• ops install therefore deletes and recreates the bare ops pod after the Secret update. New pod → kubelet reads the updated Secret → env contains the new value. (Re-creation takes a few seconds for the image cache hit; up to ~30 seconds on a cold cluster.)

The ops pod is a bare Pod, not a Deployment or DaemonSet, so kubectl rollout restart won’t work on it (rollout restart only operates on controller resources). The canonical way to force a fresh pod is roksbnkctl ops install (idempotent — it’ll handle the delete-and-recreate). If you really want to do it by hand:

kubectl delete pod roksbnkctl-ops -n roksbnkctl-ops
# then re-run `roksbnkctl ops install` to create the replacement;
# the bare pod has no controller, so nothing else will recreate it.

roksbnkctl ops show will report phase: Pending briefly during recreation, then Running + ready: true once kubelet finishes projecting the updated Secret.

Operability

Things to know when something’s wrong.

Where pod logs go

roksbnkctl k logs -n roksbnkctl-ops roksbnkctl-ops
# or
kubectl logs -n roksbnkctl-ops roksbnkctl-ops

The pod’s main process is sleep infinity, so the log is mostly empty. Each kubectl exec invocation runs in its own ephemeral process — those processes’ stdout/stderr go back through the SPDY channel to the caller, not into the pod’s log. So kubectl logs is helpful for debugging pod startup (image pull failures, SCC denials, OOMKills) but not for “what did ibmcloud iam oauth-tokens actually print” — that’s just the caller’s stdout.

For a paper trail of recent invocations, capture roksbnkctl ibmcloud --backend k8s … 2>&1 | tee /tmp/ibmcloud.log on the calling side.

Debugging a stuck ops install

ops install waits up to 60 seconds for the pod to become Ready. If it times out:

roksbnkctl k describe -n roksbnkctl-ops pod/roksbnkctl-ops
roksbnkctl k get -n roksbnkctl-ops events --sort-by=.lastTimestamp | tail -20

Common causes:

Cluster API outage during ops install

If the kube-apiserver becomes unreachable mid-install (transient cloud-provider issue, kubeconfig expired, network partition), ops install fails fast at whichever step hit the apiserver:

✓ applied namespace roksbnkctl-ops
applying secret roksbnkctl-ops/roksbnkctl-ibm-creds       ... ERROR: Get "https://...": dial tcp: i/o timeout

The install is partial at that point — earlier steps succeeded, later steps didn’t. ops install is idempotent, so just re-run once the apiserver is back; the steps that already completed are no-ops the second time, the steps that didn’t will run.

If the apiserver is permanently gone (cluster destroyed): ops uninstall will fail the same way, since it also needs the apiserver. In that case the cluster-scoped objects (ClusterRole, ClusterRoleBinding) become orphans you can clean up manually if you ever rebuild the cluster, or ignore if you’re done with this cluster’s identity entirely.

Verifying the install end-to-end

A one-liner sanity check:

roksbnkctl ibmcloud --backend k8s iam oauth-tokens

If the SA/Secret/RBAC/Pod chain is healthy, this prints a fresh OAuth token. If it errors, the error message names which link in the chain broke (pod not found, Secret missing, exec denied, ibmcloud CLI exit non-zero).

Chapter 26 — Troubleshooting covers the broader “ops pod is unhappy” failure modes alongside other end-user troubleshooting.

Cross-references

• PRD 03 — pluggable execution backends, §“K8s” — the ops-pod design rationale.
• PRD 04 — credential propagation, §“In-cluster pod” — Secret-based propagation rules.
• Chapter 14 — Credentials and the resolver chain — where the IBMCLOUD_API_KEY value comes from before it lands in the Secret.
• Chapter 17 §“K8s backend” — the interface mechanics this chapter complements.
• Chapter 18 — Choosing a backend per tool — when --backend k8s is the right call.
• internal/exec/k8s_install.yaml — the embedded RBAC manifests: https://github.com/jgruberf5/roksbnkctl/blob/main/internal/exec/k8s_install.yaml
• internal/cli/ops.go — the roksbnkctl ops install/show/uninstall command implementation: https://github.com/jgruberf5/roksbnkctl/blob/main/internal/cli/ops.go

Connectivity testing

roksbnkctl test connectivity answers one question: can my workspace reach the HTTP/HTTPS endpoints I care about right now?

It’s the simplest of the three test suites — no cluster fixtures, no remote vantage, no JSON parsing harness. Each configured URL gets one HTTP GET, the suite reports pass/fail, and the runner exits 0 if every probe passed.

Use it as the first sanity check after roksbnkctl up, as a CI smoke step against a known-good fixture set, or as the “is it me or is it the network” baseline before reaching for curl -v or openssl s_client.

What the connectivity suite does

For each configured URL the runner:

1. Adds an https:// scheme if you didn’t write one.
2. Issues a single GET with a 10-second timeout and the user-agent roksbnkctl/test.
3. Records the HTTP status code, the wall-clock duration, and (for HTTPS) the negotiated TLS version.
4. Marks the probe pass if the status code is in [200, 400) (any 2xx or 3xx); fail for anything else, any TLS error, any DNS error, any timeout.
5. Aggregates the per-URL results into a suite result; the suite passes only when every URL passed.

That’s it. No retries, no expected-body matching, no configurable status assertions, no L4 reachability — those are deliberate non-goals (see § When connectivity is the wrong tool below).

Configuring extra_hosts

The list of URLs to probe lives in your workspace config under test.connectivity.extra_hosts:

# ~/.roksbnkctl/<workspace>/config.yaml
test:
  connectivity:
    extra_hosts:
      - https://my-bnk-cis-controller.example.com
      - https://bigip-next-admin.example.com:8443
      - https://gslb.example.com
      - my-bare-host.example.com    # scheme defaults to https://

The schema is intentionally minimal — extra_hosts is a []string of URLs (or bare hostnames; https:// is added when no scheme is present). One entry per line. The order in the file is the order the runner probes.

There’s no per-host method, no per-host expected-status, and no per-host TLS-trust override today. If you need to assert something more specific than “does HTTP work” — a particular status code, a custom header, a body match — curl is the right tool, not roksbnkctl test connectivity. A richer per-host schema is queued for v1.x; the v1.0 surface holds the YAML simple on purpose.

Chapter 12 — Workspace config covers the full test: block; this chapter expands the connectivity slice.

What extra_hosts typically holds

Three classes of URL show up most often in a real workspace:

• The BNK CIS controller — confirms the data-plane front-end is reachable and is returning a sane status code.
• The F5 BIG-IP Next admin endpoint — confirms the management plane is reachable from your seat (often :8443 rather than :443).
• The GSLB VIP that fronts the application — confirms the routed name actually serves a 2xx; pair with roksbnkctl test dns for the GSLB-aware DNS-side validation.

What doesn’t belong in extra_hosts: anything you only care about on a specific TLS error, anything that needs a request body, anything where pass/fail is more nuanced than “got a 2xx or 3xx”. Those are curl jobs, not connectivity-suite jobs.

The --insecure flag

Self-signed certs are common in pre-production BNK deployments — the F5 BIG-IP Next admin endpoint, the CIS controller, an internal GSLB VIP that hasn’t yet been re-fronted with a public CA cert. By default Go’s TLS stack rejects them and the probe fails with x509: certificate signed by unknown authority.

Pass --insecure to skip certificate verification for the run:

roksbnkctl test connectivity --insecure

What --insecure does:

• Sets tls.Config.InsecureSkipVerify = true on the HTTP client used by the connectivity suite.
• Applies for the duration of one invocation only.
• Affects every URL probed in that run.

What --insecure does not do:

• It does not change L4 / DNS behaviour. A name that won’t resolve still fails; a host that drops TCP still fails.
• It is not per-host — there’s no --insecure-only=foo.example.com. Once set, the run skips verification for everything in extra_hosts.
• It is not persisted. Setting it in one invocation does not affect the next.
• It is not the same as a config-level insecure_tls: true per host. The v1.0 schema doesn’t have that knob; the only way to skip cert verification today is the session-wide flag.

If you need different TLS-trust posture per endpoint (one URL strict, another lenient), run two invocations with two different extra_hosts lists in two workspaces — that’s the workaround until per-host trust lands.

Reading the output

Default output is human-readable on stderr; pass -o json for machine-readable on stdout.

Human-readable

$ roksbnkctl test connectivity
running connectivity ...
  PASS  https://my-bnk-cis-controller.example.com  200 OK in 142ms
  PASS  https://bigip-next-admin.example.com:8443  302 Found in 88ms
  FAIL  https://gslb.example.com                   Get "...": dial tcp: i/o timeout
connectivity FAIL (2/3 passed)
$ echo $?
1

A 3xx redirect counts as pass — the runner doesn’t follow redirects, but the redirect itself is a successful HTTP response, which is what the suite measures. If you specifically need the final 200 after a redirect chain, curl -L is the tool.

JSON

$ roksbnkctl test connectivity -o json

{
  "schema": "roksbnkctl.v1",
  "command": "test",
  "suite": "connectivity",
  "timestamp": "2026-05-10T14:32:01.123Z",
  "duration_ms": 235,
  "overall": "fail",
  "results": [
    {
      "suite": "connectivity",
      "name": "https://my-bnk-cis-controller.example.com",
      "status": "pass",
      "detail": "200 OK in 142ms",
      "duration_ms": 142,
      "extra": { "status_code": 200, "tls_version": "TLS 1.3" }
    },
    {
      "suite": "connectivity",
      "name": "https://gslb.example.com",
      "status": "fail",
      "detail": "Get \"https://gslb.example.com\": dial tcp: i/o timeout",
      "duration_ms": 10003
    }
  ]
}

Exit code follows the same rules as the human-readable form: 0 on overall: pass, 1 on overall: fail. CI runners can branch on the exit code; richer assertions (e.g., “I tolerate one fail out of five”) need to consume the JSON.

Running connectivity inside roksbnkctl test all

Connectivity is one of the suites the bare roksbnkctl test (or roksbnkctl test all) command dispatches. The runner walks every configured suite, prints per-suite summaries on stderr, and exits non-zero if any suite failed:

$ roksbnkctl test
running connectivity ...
  PASS  https://bnk-cis.dev-tor.example.com  200 OK in 174ms
running dns ...
  PASS  bnk-cis.dev-tor.example.com  resolved 1 address(es)
connectivity PASS (1/1 passed)
dns          PASS (1/1 passed)

PASS overall (2/2 suites passed)

In -o json mode, roksbnkctl test all emits an all-shape envelope with one suites[] entry per suite. CI assertions can pin to either the suite-level overall or to a specific probe’s status:

roksbnkctl test all -o json | jq -e '.suites[] | select(.suite=="connectivity") | .overall == "pass"'

The bare roksbnkctl test defaults to the all suite. To run connectivity in isolation:

roksbnkctl test connectivity            # explicit suite
roksbnkctl test connectivity --insecure # session-wide TLS skip

Exit codes and CI integration

exit 0  →  every probe passed (every URL returned 2xx or 3xx)
exit 1  →  any probe failed (non-2xx/3xx, TLS error, DNS error, timeout)

There’s no third “infra error” exit code from the connectivity suite specifically — the suite is straight Go HTTP, no external tooling, no backend dispatch. If roksbnkctl test connectivity exits non-zero, the cause is in the response from one of your configured URLs.

For a CI step that tolerates a known-flaky endpoint while still failing on the others, consume the JSON instead of relying on the exit code:

roksbnkctl test connectivity -o json \
  | jq -e '[.results[] | select(.name | test("flaky-staging.example") | not) | .status] | all(. == "pass")'

Worked example: probing a BNK deployment

A typical post-up config for a BNK trial — cover the data-plane VIP, the admin endpoint, and the GSLB front:

# ~/.roksbnkctl/dev-tor/config.yaml
test:
  connectivity:
    extra_hosts:
      - https://bnk-cis.dev-tor.bnkfun.example.com         # BNK CIS controller (data plane)
      - https://bigip-next-admin.dev-tor.bnkfun.example.com:8443  # F5 BIG-IP Next admin
      - https://gslb-vip.dev-tor.bnkfun.example.com        # the GSLB front

Then:

$ roksbnkctl test connectivity --insecure
running connectivity ...
  PASS  https://bnk-cis.dev-tor.bnkfun.example.com               200 OK in 174ms
  PASS  https://bigip-next-admin.dev-tor.bnkfun.example.com:8443 302 Found in 91ms
  PASS  https://gslb-vip.dev-tor.bnkfun.example.com              200 OK in 211ms
connectivity PASS (3/3 passed)

--insecure is needed here because BNK’s admin endpoint and the GSLB VIP are fronted by a self-signed cert in dev. Once the trial moves to a production cert chain, drop the flag — the strict path is what you want for staging and prod.

When connectivity is the wrong tool

roksbnkctl test connectivity is “does HTTP work”. For anything finer-grained, reach for the right tool:

The connectivity suite is intentionally a thin probe. When the answer to “is it broken” is “yes” and you need to know why, the suite has done its job — it’s flagged the URL — and the next step is one of the tools above.

Cross-references

• Chapter 12 — Workspace config — full test: block schema, including connectivity.extra_hosts.
• Chapter 21 — DNS testing for GSLB — when “the URL fails” actually means “the name doesn’t resolve from this vantage”.
• Chapter 22 — Throughput testing — the bandwidth-measurement companion suite.
• Chapter 26 — Troubleshooting — common patterns for diagnosing connectivity failures across BNK / ROKS deployments.

DNS testing for GSLB

roksbnkctl test dns is the diagnostic surface for DNS-driven traffic management — the kind of behaviour an F5 BIG-IP Next GSLB deployment depends on, where the answer a name returns isn’t a single global truth but a function of who’s asking from where.

This is the longest chapter in the testing section because the question it answers is the most subtle. Connectivity testing tells you “the URL works”; throughput testing tells you “the path is fast”. Both assume the name resolved. When the GSLB is the thing under test, “the name resolved” is itself the question — and the answer changes depending on the network vantage of whoever’s asking.

The flag surface, the JSON output, and the multi-vantage workflow on this page are all what v1.0 ships. The design rationale lives in PRD 03 §“DNS probe (GSLB-aware)”; read that for the why, this chapter for the how.

Three vantages, one comparison

--gslb-compare is the flagship workflow: a single roksbnkctl test dns invocation fans out across local, k8s, and (optionally) ssh:<target> vantages in parallel, asks each one to resolve the same name, and reports whether the answers diverged.

graph TB
    subgraph runner[roksbnkctl test dns --gslb-compare]
        cmd[fan-out parallel<br/>per-vantage probe]
        cmp[divergence detector<br/>+ JSON aggregate]
    end
    subgraph vantage_local[local vantage]
        L[laptop resolver<br/>public DNS / GSLB outside-the-cluster answer]
    end
    subgraph vantage_k8s[k8s vantage]
        K[ops-pod resolver<br/>cluster CoreDNS / cluster-routed GSLB answer]
    end
    subgraph vantage_ssh[ssh:jumphost vantage]
        S[remote-host resolver<br/>third network path's GSLB answer]
    end
    GSLB[F5 BIG-IP Next GSLB<br/>per-resolver dispatch rule]

    cmd --> L
    cmd --> K
    cmd --> S
    L --> GSLB
    K --> GSLB
    S --> GSLB
    GSLB -. region A IP .-> L
    GSLB -. region B IP .-> K
    GSLB -. region C IP .-> S
    L --> cmp
    K --> cmp
    S --> cmp
    cmp -->|gslb_divergence: true/false| out[stdout JSON<br/>roksbnkctl.dns.v1]

The point of the diagram: a single roksbnkctl invocation probes from network positions a dig from your laptop can’t reach. The cluster vantage answers from the cluster’s egress IP; the SSH vantage answers from a third network path. Comparing the three is exactly the assertion “is the GSLB rule taking effect” needs.

The design rationale for this shape lives in PRD 03 §“DNS probe (GSLB-aware)”. The rest of this chapter is the user-facing surface.

The GSLB problem

F5 BIG-IP Next’s GSLB (Global Server Load Balancing) returns different DNS answers depending on the requesting resolver’s IP address. The discrimination is a feature, not a bug — that’s the whole point of GSLB:

• Geographic affinity: a user in the US gets a US datacenter IP; a user in the EU gets an EU datacenter IP. The dispatch rule is per-region.
• Datacenter routing: a request from a known partner CIDR gets a private VIP; a request from the public internet gets a public VIP.
• Health-check state: when the primary pool member goes unhealthy, GSLB starts handing out the secondary pool member’s IP — but only after the resolver’s TTL on the prior answer expires.
• Anycast vs unicast: a name fronted by an anycast resolver fleet may return the same answer everywhere; the same name fronted by GSLB returns a per-region answer.

Validating GSLB means validating that the right answer comes back from the right vantage. From your laptop in Toronto, you should see the US datacenter IP. From a workload running in the EU, you should see the EU datacenter IP. From an east-Asia bastion, you should see whatever your GSLB rule says east-Asia gets.

The standard dig www.example.com from your laptop only ever tells you what your laptop’s resolver, talking to its configured upstream, gets back. That’s one vantage. To validate GSLB you need several, and you need to be able to compare the answers.