Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Troubleshooting

Common failure modes you’ll hit when running roksbnkctl against real IBM Cloud accounts, organised as symptom → root cause → fix. The entries here are mined from the issue logs accumulated over Sprints 0-5 plus the failure shapes documented in PRD 05 §“Risks”.

Use the page as a lookup table. If your symptom isn’t here, Chapter 23 — The E2E test plan lists what every phase asserts; reverse-engineering from the assertions can narrow your diagnosis. For deeper-than-here debugging, the per-phase log files under /tmp/roksbnkctl-e2e-backends/ are the first stop.

Install and init

Symptom: roksbnkctl init errors with plaintext secret detected

Root cause: an existing ~/.roksbnkctl/<workspace>/config.yaml has a credential value sitting in a field whose name matches the rejection regex (api_key, password, token, secret_access_key, hmac_secret). The rejection is a deliberate safety net — see Chapter 14 §“What’s safe to commit vs not”.

Fix: move the credential into IBMCLOUD_API_KEY (env var) or the OS keychain (roksbnkctl init writes it via zalando/go-keyring). For a single-user dev box, the supported plaintext-on-disk channel is ibmcloud.api_key_b64 — base64-encoded, which doesn’t trip the regex.

Symptom: roksbnkctl init interactive prompts loop forever asking for the API key

Root cause: you’re running under CI / a non-TTY shell and roksbnkctl can’t read stdin. The interactive prompt fallback is the last step in the credential resolver chain and it doesn’t gracefully skip when stdin is closed.

Fix: set IBMCLOUD_API_KEY in the env, or pre-populate the keychain entry. For batch / CI runs, the documented invocation is:

IBMCLOUD_API_KEY=$(cat /path/to/secret) roksbnkctl init -w my-workspace

Pre-setting IBMCLOUD_API_KEY skips the API-key prompt (it’s the first link in the resolver chain). init still prompts for the remaining workspace metadata (region, resource group, cluster name) on TTY-bound stdin — a fully non-interactive bootstrap is on the v1.x roadmap.

Symptom: doctor reports terraform: not found on a fresh dev box

Root cause: terraform is the only strictly-required host tool for v1.0 (everything else is internalised). Doctor checks PATH; if your shell session hasn’t sourced the install location it’ll miss.

Fix: install terraform via your package manager (brew install terraform, apt-get install terraform, etc.) and re-source the shell, or set the TERRAFORM_BIN env var pointing at the binary explicitly.

Symptom (Windows): roksbnkctl install succeeds but roksbnkctl version says the command is not recognized

Root cause: roksbnkctl version (like any command) only resolves if the binary sits in a directory on the Windows %PATH%. If the install landed the binary somewhere off %PATH% (for example ~/.local/bin, a Unix convention that is never on the Windows %PATH%), the copy succeeds but the command doesn’t resolve.

Fix: roksbnkctl install (and the install.ps1 one-liner) install into a directory already on %PATH% — preferring %LOCALAPPDATA%\Microsoft\WindowsApps (on the per-user PATH by default, no admin) — so the binary resolves immediately in the same session. Remove any stray copy left in a non-PATH directory with Remove-Item "$env:USERPROFILE\.local\bin\roksbnkctl.exe". If a run ever lands in a non-PATH directory, it prints the PowerShell one-liner ([Environment]::SetEnvironmentVariable('Path', ...,'User')) to add it.

roksbnkctl up lifecycle

Symptom: bnk up refuses with cluster "…" already has BNK installed … but workspace "…" has no terraform state for it

Root cause: exactly what it says — the cluster is serving a BNK install that this workspace’s state knows nothing about. This is the normal shape when a cluster moves between deployments: BNK Forge gives each project its own deployment-scoped volume, so the second project to adopt a cluster legitimately has no state for the first project’s install.

Before v1.42.0 there was no refusal. An empty state planned a full install — resources_to_add=64 over a cluster that already had all 64 — and ground on for about thirteen minutes before exiting 1 without naming the cause or the cluster, three times over with retries (#53). The refusal replaces that with a fast, named failure.

Fix: pick one —

  • run from the workspace that installed it, whose state can converge or remove the install;
  • remove BNK from the cluster first, then re-run here;
  • if the install is genuinely abandoned, clear it from the cluster deliberately rather than letting an apply collide with it.

roksbnkctl -w <ws> bnk status shows what is actually on the cluster.

The check is narrow by construction: it fires only when this workspace’s BNK state holds no managed resources and the cluster is actually serving an install. A workspace that owns an install converges exactly as before and never reaches it. If the cluster cannot be reached, or what the state holds cannot be determined — including an s3 remote backend whose state cannot be listed — the guard stays silent rather than inventing a refusal.

There is deliberately no bnk adopt yet. Deriving and validating a complete install record from a live cluster is a larger change than a guard, and wants its own testing against real installs.

Symptom: terraform apply errors timeout while waiting for state to become 'normal'

Root cause: IBM Cloud’s control plane is occasionally 5-15 minutes slow propagating cluster state — a known transient. The cluster was created; the API just hasn’t caught up to reporting it as Ready.

Fix: roksbnkctl up retries the apply automatically up to 3 attempts with a 60-second sleep between (see applyWithRetry in internal/cli/lifecycle.go). If all three retries fail, just re-run roksbnkctl up manually — terraform’s state is durable, and the second attempt skips every resource that’s already provisioned.

Symptom: roksbnkctl up returns success but roksbnkctl k get nodes says No resources found

Root cause: the ROKS cluster’s worker nodes take 5-10 minutes to provision after the cluster’s master endpoint returns Ready. Terraform considers the cluster “applied” as soon as the master is up; the workers come up asynchronously.

Fix: wait 5-10 minutes and re-run. If you want a deterministic gate, watch the IBM Cloud console’s cluster page until the worker count matches workers_per_zone × zones, then proceed. There’s no roksbnkctl wait command.

Symptom: roksbnkctl up post-apply hook fails fetching the admin kubeconfig with a 404

Root cause: the IBM Cloud kubeconfig API (/global/v2/applications/kubeconfig) returns 404 for ~30-60 seconds after the cluster create call returns. The cluster exists but the kubeconfig endpoint hasn’t materialised.

Fix: the binary retries with exponential backoff and usually succeeds within a minute. If it still 404s after the retry budget, run roksbnkctl kubeconfig --download -w <workspace> to retry just the fetch without re-applying.

Symptom: Error: Inappropriate value for attribute "kubeconfig_dir": directory does not exist

Root cause: the upstream HCL’s IBM provider doesn’t MkdirAll for the kubeconfig output directory; it expects the parent dir to exist already. The variable’s default (/work/.bnk/scratch/kubeconfig) is the in-container path; on a direct-on-host run it’s a path that doesn’t exist.

Fix: roksbnkctl writes a workspace-scoped override (kubeconfig_dir = ~/.roksbnkctl/<ws>/state/kubeconfig) and creates the dir at apply time. If you’re hand-rolling terraform without roksbnkctl up, mkdir -p ~/.roksbnkctl/<ws>/state/kubeconfig first.

Symptom: cert-manager (or another chart) fails with could not download chart: ... open <HOME>/.cache/helm/repository/<hash>-index.yaml: no such file or directory

Root cause: a helm_release whose repository is a bare chart-repo URL (cert-manager off https://charts.jetstack.io) resolves through Helm’s anonymous-repo path, which downloads the repo index into helmpath.CachePath("repository") — i.e. $HELM_CACHE_HOME/repository, defaulting to $HOME/.cache/helm/repository. Crucially, that path is governed by HELM_CACHE_HOME, not HELM_REPOSITORY_CACHE (which only covers the named-repo / helm repo CLI path). In a runner whose $HOME resolves to an empty or non-writable path — how the bnk-forge tools-runner image launches the phases — that directory can’t be created, so the download fails even though charts.jetstack.io is perfectly reachable (the index downloads on the fly once the dir is writable — no pre-helm repo add is needed). The sibling symptom is a non-fatal warning: creating <HOME>/.kube: mkdir <HOME>: permission denied from the post-apply admin-kubeconfig fetch.

Fix: roksbnkctl exports writable, $ROKSBNKCTL_HOME-relative Helm homes before invoking terraform/helm — HELM_CACHE_HOME, HELM_CONFIG_HOME, HELM_DATA_HOME (plus HELM_REPOSITORY_CACHE / HELM_REPOSITORY_CONFIG / HELM_REGISTRY_CONFIG) under $ROKSBNKCTL_HOME/.helm/, and KUBECONFIG at $ROKSBNKCTL_HOME/.kube/config when $HOME/.kube isn’t writable — and pre-creates those dirs (see prepareToolEnv in internal/tf/terraform.go). The terraform helm provider runs as a plugin subprocess that inherits this env, so the redirect reaches the chart download. $ROKSBNKCTL_HOME is the persisted workspace tree (/work/.roksbnkctl under bnk-forge), so the cache is writable and survives across the phased invocations. Each var is set only when unset, so you can still point the cache at an air-gap mirror by exporting your own value. Set ROKSBNKCTL_DEBUG=1 to log the resolved HELM_CACHE_HOME / KUBECONFIG before each phase’s terraform exec. No action needed on a normal host run — $HOME there is already writable.

Symptom: terraform destroy leaves orphan IBM Cloud resources (LBs, security groups, VPEs)

Root cause: ROKS occasionally leaves dangling cluster-owned resources after the cluster itself is destroyed — the destroy returns success but the IBM Cloud account still shows a load balancer or a Virtual Private Endpoint Gateway tagged with the deleted cluster’s ID.

Fix: run roksbnkctl cleanup. It sweeps the account for every resource named after the workspace prefix (<prefix>-*) — VPC instances/subnets/security-groups/floating-IPs/public-gateways/VPCs, the Transit Gateway, registry COS, the cluster, and the BNK trusted profile — and deletes them in dependency order. Preview first with roksbnkctl cleanup --dry-run; add --all-regions if resources landed in a region not recorded in config.yaml. (For one-off resource types not covered by the prefix sweep, roksbnkctl ibmcloud is load-balancers | grep <cluster-name> + ibmcloud is load-balancer-delete <id> still works as a manual fallback.)

Teardown

bnk down sits for four minutes, then the namespace hangs in Terminating

⚠ 7 BNK custom resource(s) in f5-bnk did not finalize within 4m0s.
⚠ namespace "f5-bnk" was stuck Terminating; cleared F5 finalizers on 2 object(s).
Error: context deadline exceeded

Fixed in v1.59.0. If you see this, you are on v1.58.0 or earlier.

The teardown drains BNK’s custom resources before destroying the namespace, so that the F5 Lifecycle Operator is still running to finalize them — a finalizer that outlives its controller blocks the namespace forever. Three things made that drain unable to finish:

  • It deleted the components FLO manages. cnecontrollers, f5tmms, dssms, afms, downloaders and cnemanifests are declared by the CNEInstance, and FLO recreates any of them within about five seconds for as long as the CNEInstance exists. Deleting the CNEInstance is the uninstall; the components go with it.
  • Two product defaults cannot be drained at all. f5-big-tcp-settings/sys-default-tcp is refused by BNK’s own admission webhook (Default TCP parameters CR cannot be deleted!) and f5-big-dns-caches/security-fqdn-dns-resolver is recreated on sight. Neither carries a finalizer, so the namespace delete removes them; waiting for them achieves nothing.
  • f5validate-<ns> blocks its own removal. It has failurePolicy: Fail and is served from the namespace being destroyed, so once the controller stops answering the API server refuses every DELETE of a k8s.f5net.com object. Removing it does not hold — FLO restores it in under a second.

If you are stuck on an older version, the manual escape is to delete the CNEInstance and let FLO remove the rest, rather than deleting the components individually:

kubectl -n f5-bnk delete cneinstance --all
kubectl get validatingwebhookconfiguration | grep f5validate   # delete if the namespace will not terminate

The teardown says a custom resource “did not finalize”

The message names the object. Two cases, and they need opposite responses:

  • cneinstances/... — the CNEInstance finalizer tears down the whole product, so this is legitimately slow work rather than a stall. Let it run.
  • anything else — check whether it is being recreated: kubectl -n f5-bnk get <kind> <name> -o jsonpath='{.metadata.uid}' before and after a delete. A new UID means a controller replaced it, and no amount of waiting will remove it while its controller is running.

Cluster-scoped objects remain after a successful bnk down

Expected. A BNK 2.4 install leaves 95 CustomResourceDefinitions, 10 APIServices and a ValidatingAdmissionPolicy behind, none of which a namespace delete touches. They are shared with any other BNK install on the cluster, so removing them is a deliberate operator decision rather than a side effect of one workspace’s teardown. Reinstalling over them works — verified across consecutive install/uninstall cycles at the same manifest version.

scripts/testing/cluster-residue-check.sh reports what is left, and excludes cert-manager: it is shared infrastructure, and removing its CRDs would delete every Certificate and Issuer on the cluster including ones BNK never created.

What must not remain is a ValidatingWebhookConfiguration served from the BNK namespace. It is named for one namespace and points at a Service that no longer exists, and because it matches by namespaceSelector, recreating a namespace of the same name re-arms a webhook whose backend will never answer.

Workspaces

Symptom: after roksbnkctl ws delete <name>, which workspace is current?

Behaviour (since v1.9.0): you can delete the current workspace directly — ws delete moves the current_workspace pointer to another existing workspace (alphabetically first), or clears it when you delete the last one. There’s no “switch first” requirement and no phantom default fallback.

If a command then reports no workspace selected: you deleted the last workspace, so nothing is current. Create one with roksbnkctl init, or point at an existing one with roksbnkctl ws use <name>. (If the message is instead workspace "<name>" is not initialised, the current_workspace pointer references a directory that was removed by hand — ws use <other> repoints it.) Documented in Chapter 6 — Workspaces.

Symptom: workspace "<name>" has terraform-managed resources; pass --force to delete anyway

Root cause: the workspace’s terraform.tfstate is non-empty — live infrastructure exists. roksbnkctl ws delete refuses to orphan the resources by removing the state file out from under them.

Fix: run roksbnkctl down -w <name> --auto first to destroy the resources, then roksbnkctl ws delete <name> (no --force needed once state is empty). If you genuinely want to abandon the infra and clean up by hand later, roksbnkctl ws delete --force skips the check.

Backends

Symptom: --backend docker errors with Cannot connect to the Docker daemon

Root cause: dockerd isn’t running, or your user isn’t in the docker group, or you’re on a system that needs a separate rootless-docker socket path.

Fix:

  • Linux with system docker: sudo systemctl start docker; add yourself to the docker group (sudo usermod -aG docker $USER) and log out + back in.
  • Linux with rootless docker: systemctl --user start docker; set DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock.
  • macOS / Windows: launch Docker Desktop / Rancher Desktop.

Verify with docker info | head -1 — if that fails, roksbnkctl --backend docker will too.

Symptom: --backend k8s errors with ops pod not found in roksbnkctl-ops namespace

Root cause: you haven’t run roksbnkctl ops install against the target cluster. The k8s backend dispatches into a long-lived ops pod that has to be provisioned first.

Fix:

roksbnkctl ops install

Verify with roksbnkctl k get pod -n roksbnkctl-ops — the pod should be Running. See Chapter 19 for the install model.

Symptom: --backend ssh:<target> errors with tool not found: iperf3 — run with --bootstrap to apt-install

Root cause: the SSH target doesn’t have the tool installed, and roksbnkctl doesn’t auto-install without explicit opt-in (because apt-installing on a production jumphost without consent is rude).

Fix: pass --bootstrap once per fresh target:

roksbnkctl --backend ssh:jumphost --bootstrap test throughput

The bootstrap step runs apt-get install -y <tool> (or the equivalent for ibmcloud — adding the IBM apt repo first). Subsequent calls skip the install check and run normally. See Chapter 17 §“SSH backend” for the bootstrap mechanism.

Symptom: --backend ssh:jumphost errors host key mismatch for jumphost (got SHA256:..., known_hosts has SHA256:...)

Root cause: the jumphost was re-provisioned (terraform destroy + apply) and now has a fresh host key, but ~/.roksbnkctl/known_hosts still has the old fingerprint. TOFU refuses to silently accept the change — that’s the threat model the prompt exists to defend against.

Fix: if you know the re-provision is legitimate, delete the stale entry:

ssh-keygen -R '<jumphost-ip>' -f ~/.roksbnkctl/known_hosts
# Or for the whole roksbnkctl known_hosts:
rm ~/.roksbnkctl/known_hosts

The next roksbnkctl --on jumphost call will TOFU-prompt with the new fingerprint. For CI use the --insecure-host-key flag, which records the key on first contact without prompting.

OpenShift and PodSecurity

Symptom: throughput test pod fails admission: violates PodSecurity "restricted:v1.x": runAsNonRoot != true

Root cause: the throughput suite’s default iperf3 image is networkstatic/iperf3:latest which runs as root. OpenShift’s restricted-v2 SCC rejects root pods.

Fix: set the workspace config to use the bundled image, which is built USER 1000:

# ~/.roksbnkctl/<workspace>/config.yaml
test:
  throughput:
    image: ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:v0.9.0

Chapter 22 §“The bundled image and the runAsNonRoot constraint” is the full backstory. The same chapter’s §“OpenShift SCC failure mode” lists the three error-message variants OpenShift produces.

Symptom: roksbnkctl ops install errors ServiceAccount "roksbnkctl-ops" forbidden: violates PodSecurityPolicy

Root cause: rare — the cluster is running PodSecurityPolicy (the deprecated predecessor to PodSecurity admission) and the ops pod’s ServiceAccount doesn’t have the SCC binding it needs.

Fix: the ops manifest assumes restricted-v2 is acceptable. If your cluster forces privileged, that’s a cluster-policy question outside roksbnkctl’s control — talk to your cluster admin about granting restricted-v2 to the roksbnkctl-ops namespace.

Symptom: ImagePullBackOff on the ops pod or throughput pod

Root cause: most commonly, the cluster can’t reach the image registry. Three sub-causes:

  1. The cluster’s egress NAT doesn’t route to ghcr.io (the image host for roksbnkctl-tools-*).
  2. The image tag doesn’t exist for the version you’re running (e.g., you built roksbnkctl from main at a commit between releases, and :dev isn’t published).
  3. ghcr.io itself is rate-limiting unauthenticated pulls (rare; usually only an issue for shared CI hosts hitting ghcr.io en masse).

Fix:

  • Check egress with roksbnkctl k exec <ops-pod> -- curl -sI https://ghcr.io — if that hangs, you have a network path issue, not a roksbnkctl issue.
  • Check the tag with docker manifest inspect ghcr.io/jgruberf5/roksbnkctl-tools-iperf3:<version> — if 404, pin to a tagged release version in workspace config rather than running from main head.
  • For rate-limit issues, pre-pull images to a local registry mirror and override the workspace test.throughput.image to point there.

DNS

Symptom: roksbnkctl test dns returns NXDOMAIN against an internal GSLB record that you know exists

Root cause: your laptop’s resolver chain doesn’t have a route to the internal GSLB VIP. The default --server system uses your /etc/resolv.conf, which resolves against your office or ISP resolver — neither of which knows about the cluster-private GSLB.

Fix: query the GSLB VIP explicitly, or query from inside the cluster:

# Query the GSLB VIP directly
roksbnkctl test dns --target www.example.com --type A --server 169.45.91.5

# Or run the probe from inside the cluster (the cluster's resolvers reach the GSLB)
roksbnkctl test dns --target www.example.com --type A --backend k8s --server cluster

Chapter 21 §“Server resolution” is the full --server reference.

Symptom: --gslb-compare always reports gslb_divergence: false against a target you expect to diverge

Root cause: the chosen target’s GSLB rule isn’t differentiating your local vantage (laptop) from your k8s vantage (cluster). Two common shapes:

  1. The name is fronted by an anycast resolver fleet (Cloudflare, Google Public DNS) — same answer everywhere by design.
  2. Your laptop and your cluster are both in the same geographic region from GSLB’s perspective (both in North America hitting the same datacenter).

Fix: pick a target known to be geo-resolved (www.google.com is the canonical “different IPs from different regions” example), or add an SSH-based vantage (--backend ssh:eu-bastion) to bring in a third region. Chapter 21 §“GSLB cross-vantage compare” covers the multi-vantage workflow.

Symptom: roksbnkctl test dns --backend docker errors DNS probe doesn't benefit from docker

Root cause: design choice. Docker containers share the host’s network namespace by default, so a docker-backend probe has the same network identity as a --backend local probe — no GSLB-relevant vantage difference.

Fix: use --backend local, --backend k8s, or --backend ssh:<target> instead.

Cluster registration

Symptom: roksbnkctl cluster register <name> errors cluster not found

Root cause: the cluster name doesn’t exist in the workspace’s resource group, or the API key doesn’t have visibility into the resource group.

Fix: verify the name with roksbnkctl ibmcloud ks cluster ls --output json | jq '.[].name', and verify the resource-group scope in workspace config matches where the cluster lives. If the cluster is in a different resource group, set ibmcloud.resource_group in the workspace config to that group.

Symptom: register succeeds but roksbnkctl k get nodes immediately errors Unauthorized

Root cause: the kubeconfig was fetched but the auth token has already expired, or the IAM-based token that the kubeconfig embeds doesn’t match the API key that’s currently in env. Common after a 1-hour idle window.

Fix:

roksbnkctl kubeconfig --download --cluster <name>

The token refresh is automatic on every up/apply, but register against a cluster you didn’t just provision sometimes lands you with a stale token in the kubeconfig.

COS supply chain

Symptom: FLO fails to start with failed to pull FAR image: 403 Forbidden

Root cause: the f5-far-auth-key.tgz object in the bucket has stale credentials (the F5-side pull key was rotated, but the bucket still has the old one).

Fix: re-issue the key on the F5 side and upload to COS:

roksbnkctl cos object put bnk-artifacts/f5-far-auth-key.tgz \
  ./new-f5-far-auth-key.tgz \
  --instance bnk-supply-chain

# Restart FLO so it re-reads
roksbnkctl k delete pod -n f5-bnk -l app=flo

See Chapter 25 §“Worked example” for the full flow.

Symptom: cos object put for a 3 GB file errors midway with RequestTimeout

Root cause: the multipart upload SDK encountered a transient COS HTTP timeout on one of the part uploads. Multipart uploads aren’t currently resumed from the failure point — they restart from zero.

Fix: re-run the cos object put. If it fails reproducibly on the same part, the underlying network is the problem (your egress link is saturated, or COS is having a regional outage — check the IBM Cloud status page).

Symptom: cos bucket delete errors Bucket not empty

Root cause: COS requires buckets to be empty before delete; there’s no --recursive flag on bucket delete today.

Fix: list and delete each object, then delete the bucket:

roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain | \
  awk 'NR>1 {print $1}' | \
  xargs -I{} roksbnkctl cos object delete "bnk-artifacts/{}" --instance bnk-supply-chain

roksbnkctl cos bucket delete bnk-artifacts --instance bnk-supply-chain

Don’t forget to abort any pending multipart uploads first — they don’t appear in the standard object list but they do prevent bucket deletion. The workaround for now is ibmcloud cos list-multipart-uploads followed by ibmcloud cos abort-multipart-upload until v1.x lands a native command.

Shared VPC, Transit Gateway, and quotas

Symptom: roksbnkctl up fails at apply with a VPC or Transit Gateway quota error (Quota: 20, or cannot add more than 10 gateways)

Root cause: IBM Cloud caps VPCs at 20 per region and Transit Gateways at 10 per account (defaults). A full region/account fails the create — often ~40 minutes into a cluster build.

Fix: run roksbnkctl doctor first — the ibm cloud quota check reports VPCs-in-region and account-wide Transit Gateways vs the limits and warns when you’re at the wall. The init interview shows the same count at the create-VPC / create-TGW prompts; answer no and adopt an existing VPC/gateway (the picker lists them), or request a quota increase from IBM. See Reusing an existing VPC and Sharing a Transit Gateway.

Symptom: image pulls from the mirror time out intermittently, but every security group and ACL allows the traffic

Root cause: two VPCs on the same Transit Gateway have overlapping address prefixes. A gateway routes on VPC address prefixes; when two attached VPCs claim the same block the route is ambiguous and the gateway silently drops traffic for one of them. Nothing logs an error, which is what sends people into hours of firewall archaeology. The classic tell is that connectivity is partial or flaps — some pulls land, some hang — rather than failing cleanly.

It happens by accident because IBM’s default address-prefix management is auto, which gives every VPC in a region the identical three per-zone prefixes (10.241.0.0/18, 10.241.64.0/18, 10.241.128.0/18). A second roksbnkctl-created cluster joining a shared gateway therefore collides by construction.

Confirm it — list the prefixes of each VPC on the gateway and look for repeats:

ibmcloud tg connections <gateway-id> --output json | jq -r '.[]|select(.network_type=="vpc")|.network_id'
ibmcloud is vpc-address-prefixes <vpc-id>     # per VPC from that list

Fix: the prefixes of a live VPC cannot be edited — moving a subnet’s CIDR replaces the subnet, destroying the cluster on it. Either detach one (roksbnkctl -w <other-workspace> tgw disconnect --auto; the cluster survives), or rebuild one cluster on its own block, set before cluster up:

cluster:
  vpc_cidr: 10.242.0.0/16     # ROKSBNKCTL_CLUSTER_VPC_CIDR for CI

Set it before cluster up, not after. Editing vpc_cidr in a workspace whose cluster already exists does not re-address anything — an existing VPC’s prefixes cannot be changed without replacing its subnets. It now says so:

! cluster.vpc_cidr is a CREATE-time setting and it has changed.
  Cluster "x" was created from 10.242.0.0/16; this workspace now says 10.99.0.0/16.
  The new value is ignored — ...
  A future release will refuse this rather than ignore it; ...

The warning fires only on a real disagreement with the block the cluster was created from, so a workspace that set it once and left it alone stays silent. A workspace built by cluster register also stays silent, because the adopt path records no block to disagree with — see Chapter 9.

Prevention (v1.39.0+): cluster up refuses before terraform creates anything when the VPC it is about to build would overlap one already on the gateway, and tgw connect refuses before attaching — each naming the other VPC and the colliding prefixes. See Give each cluster VPC its own address block.

Symptom: cluster ... was created as a single-nic cluster, but this workspace now asks for multi-nic

Root cause: cluster.network_mode is a create-time setting and it changed after the cluster was built. There is no in-place conversion between modes.

Why it refuses rather than warns: terraform would happily plan this — as a destroy and create of the running cluster, rendered as ~60 lines that read like an update. The refusal happens before planning, so you never get the chance to approve it by mistake.

Fix: set cluster.network_mode back to what the cluster is, or build the other mode as a new cluster in a new workspace. What the cluster actually is comes from cluster-outputs.json, not from config.yaml.

Note that removing the setting is not the same as changing it: an unset network_mode defers to the record and is never refused. See Chapter 8a §Which side wins.

Symptom: BNK 2.3 does not support multi-nic clusters (it supports single-nic)

Root cause: the BNK release — derived from bnk.manifest_version — cannot drive a cluster of that network mode. 2.3 does not express the multi-NIC network attachments or CNEInstance options.

Fix: use a cluster built in a mode this line supports. There is currently no line to move to for multi-nic — 2.4 was expected to add it and, on the evidence of its EA install guide, does not, so its row was corrected to single-nic. See Chapter 28. The pairing is checked at plan time precisely so this is a message rather than a failed apply against real infrastructure.

Symptom: ! unknown BNK release line: BNK 2.5 is not in this build's support matrix

This is a warning, and the run continues. It is expected, not a defect.

Root cause: the support matrix ships inside the binary, so a release newer than your build is unknown to it. That is missing information, not a known incompatibility — refusing on it would make every build refuse every BNK release that shipped after it, which is unhelpful when the binary is pinned (the BNK Forge modules pin the runner image by digest, so “use a newer build” is not an action available to whoever is choosing the release).

What it means: the combination is unverified here, not blessed. Upgrade roksbnkctl if you want it actually checked.

Symptom: down fails with The VPC is in use and cannot be deleted. Subnets [...]

Root cause: this workspace created the VPC, and another workspace that adopted it still has a cluster (its subnets) inside. The VPC owner must be torn down last — deleting it first can’t remove a non-empty VPC, and also deletes the shared per-zone public gateways the other cluster still depends on.

Fix (v1.26.1+): down refuses this up front, naming the foreign subnets. Tear down the workspace(s) sharing the VPC first, then re-run down on the owner. See the teardown-order note in Reusing an existing VPC.

Symptom: down fails with UnsetSubnetPublicGateway ... the specified subnet has no public gateway

Root cause: an adopter cluster’s teardown tried to detach its subnets from a shared public gateway that the VPC-owning cluster had already deleted (the owner was torn down first — see above).

Fix: v1.26.1+ clusters can’t hit this — subnets attach their gateway inline, so deleting the subnet detaches implicitly. For an older cluster stuck mid-teardown, drop the dead attachment from state and retry: cd ~/.roksbnkctl/<ws>/state-cluster && terraform state rm 'module.roks_cluster.module.cluster.ibm_is_subnet_public_gateway_attachment.cluster_subnet_gateway_zone1[0]' (repeat for zone2/zone3), then roksbnkctl -w <ws> cluster down.

Symptom: roksbnkctl bnk up hangs on licensing / the License CR never reaches Active

Root cause: BNK licensing completes only when the License CR reports status.state: Active (or the LicenseActive condition), which roksbnkctl polls for deterministically (~15-minute bound). A hang usually means the FAR supply chain is wrong — a missing/expired JWT, or the JWT uploaded under the wrong object key — or, in f5licenseproxy mode, the F5 License Proxy isn’t reachable.

Fix: confirm the JWT is present under the exact object key subscription.jwt (roksbnkctl cos object list <bucket> --instance bnk-supply-chain) and not expired; for FLP mode check roksbnkctl flp status before bnk up. See Licensing BNK with the FLP.

Since v1.41.0 the reachability gate catches the network half of this before the install starts, so a licensing hang that is really an unreachable proxy now surfaces as a named per-node failure instead — see the next entry.

Symptom: bnk up fails the reachability gate on a cluster whose network you believe is correct

✗ kube-…-0000013a -> registry (10.243.0.4:443): dns=skipped-ip tcp=FAILED
    (tcp connect failed -- refused, filtered, or no route)

Root cause: most often the gate is right and something (a security group, an ACL, a missing Transit Gateway attachment, or overlapping VPC address prefixes) genuinely blocks the path from that node.

But there is one case where the network is correct and the probe is early. A Transit Gateway attachment is asynchronous: IBM programs the routes some time after the connection reports attached. A workspace that creates the VPC, joins the gateway, and installs in a single run can probe a path that is correct but not yet live. A probe 73 seconds after attach has failed on a path that was healthy minutes later, while a sibling cluster on the same gateway and the same blueprint passed simply by landing on the other side of route programming (#57).

Fix: read the attempt count in the detail first — it is what tells the two apart:

  • (still failing after 19 attempts over 180s) — it kept trying for the whole budget. That is a real break; go look at the security groups, the TGW attachment, and the VPC prefixes.

  • No attempt clause, or a low count — it gave up early. Raise the budget:

    bnk:
      preflight:
        reachability_retry_seconds: 600
    

    or ROKSBNKCTL_REACHABILITY_RETRY_SECONDS=600 on a CI runner. See Chapter 28 §bnk.preflight.

Check dns before tcp: dns=FAILED is a resolver or hostname problem and no retry budget will fix it.

Re-running gives you a fresh answer. Since v1.42.0 the gate rolls its DaemonSet on every run, so the verdict always belongs to the run you just started. Before that, a re-run after fixing the routing re-read the original pod’s log and showed the same failure with nothing to indicate it was a recording.

Networking

Symptom: roksbnkctl test connectivity reports Get "https://...": dial tcp: i/o timeout for an internal-only URL

Root cause: connectivity probes run from --backend local by default. From your laptop, internal-only URLs (cluster-private VIPs, internal GSLB names) aren’t reachable.

Fix: route the probe through the cluster’s network — either via --backend k8s (when it lands for the connectivity suite — currently k8s-backend is iperf3 + DNS only; connectivity stays local for v1.0) or via an SSH target inside the cluster’s VPC (--backend ssh:cluster-jumphost).

Symptom: roksbnkctl test connectivity fails with x509: certificate signed by unknown authority against a self-signed internal endpoint

Root cause: the URL’s TLS cert isn’t in the host’s trust store.

Fix: pass --insecure (session-wide; skips TLS validation for every probe in the run). The flag is deliberately session-wide rather than per-host — see Chapter 20 §“Mixed TLS-trust posture”. For mixed trust posture across multiple internal endpoints, run two separate test connectivity invocations, one per trust group.

CI-specific

Symptom: nightly e2e run fails on phase D with Error: Provider configuration is missing

Root cause: a terraform init cache invalidation under ~/.roksbnkctl/<ws>/state/.terraform/ left a partial provider download. Happens after a CI worker is recycled mid-init.

Fix: rm -rf ~/.roksbnkctl/<ws>/state/.terraform/ then re-run roksbnkctl up. Terraform-init re-downloads the providers cleanly. For CI workers that get recycled often, add a pre-step that purges .terraform/ before each run.

Symptom: cred audit (phase M) reports IBMCLOUD_API_KEY found in docker inspect output

Root cause: real stop-ship — credentials leaked into a docker container’s runtime env. Check internal/exec/docker.go::buildEnvArgv for any code path that passes the credential by value (-e IBMCLOUD_API_KEY=<value>) rather than by reference (-e IBMCLOUD_API_KEY — let docker pull from the caller’s env).

Fix: file an issue immediately, do not tag a release until this is green. Phase M is the v1.0 release gate; a leak here means the redactor or the cred-passing logic regressed. See PRD 04 for the threat model.

Getting more help

When the symptom isn’t on this page:

  1. Re-run with --verbose (-v) — the verbose output usually surfaces the root cause directly.
  2. Check /tmp/roksbnkctl-e2e-backends/<phase>-<ts>.log for the per-phase trail.
  3. Cross-reference Chapter 23 — The E2E test plan — the phase-by-phase pass criteria usually narrow down where the breakage lives.
  4. File an issue on github.com/jgruberf5/roksbnkctl with the verbose output, the roksbnkctl --version stamp, and the per-phase log if there is one.