Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Doctor: checking your environment

roksbnkctl doctor is the prereq + credentials report. It runs in under five seconds, exits non-zero on any hard error, and prints a tabular report that maps one-to-one to the runtime dependencies the rest of the tool reaches for.

This chapter walks every check, explains what each row’s “why we care” blurb means, covers why kubectl and oc are informational rather than required, and describes the --target SSH probe.

What doctor checks

A bare roksbnkctl doctor runs the general checks: tooling on PATH, kubeconfig location, the resolved workspace, and the IBM Cloud authentication chain. Sample output on a healthy machine looks like this:

roksbnkctl doctor
✓  terraform         /usr/bin/terraform (Terraform v1.15.2)                                   (required for `roksbnkctl up`)
✓  helm              /usr/local/bin/helm (v3.20.2)                                            (required for `roksbnkctl up`; terraform `local-exec` shells out to helm)
⚠  iperf3            not on PATH                                                              (needed for `roksbnkctl test throughput`)
✓  kubectl           /usr/local/bin/kubectl (clientVersion:)                                  (internalised in roksbnkctl k *; passthrough still works if installed)
✓  oc                /usr/local/bin/oc (Client Version: 4.21.10)                              (internalised in roksbnkctl k *; passthrough still works if installed)
✓  ibmcloud          /usr/local/bin/ibmcloud (ibmcloud 2.43.0 ...)                            (optional; `roksbnkctl ibmcloud` passthrough)
✓  kubeconfig        /home/jgruber/.kube/config                                               (needed for cluster-side ops)
✓  workspace         default                                                                  (per-environment config + state)
✓  workspace permissions  owner-only                                                          (credentials in the workspace are protected by file mode alone)
✓  ibmcloud api key  resolved                                                                 (auth for terraform + IBM SDK calls)
✓  ibm cloud auth    OK (account: 1a2b3c..., user: you@example.com)                           (verifies API key works against IBM IAM)

Each row has the same shape:

<status> <name> <detail> <why we care>
  • status is one of ✓ (green / OK), ⚠ (yellow / warning), or ✗ (red / error). Skipped checks render as ⚠.
  • name is the dependency or capability being checked.
  • detail is the resolved value — usually a path, a version line, or an error message.
  • why we care is a parenthetical clause naming the roksbnkctl feature that depends on this row.

The BackendName column on the underlying Check struct (internal/doctor/check.go) is reserved for the per-backend probes that land in Sprint 4 (PRD 03). Until then it stays empty for every general check.

Each check explained

terraform — required

One of two hard-required binaries for the roksbnkctl up happy path. roksbnkctl shells out to terraform via terraform-exec for plan/apply/destroy; without it nothing in the cluster lifecycle works.

Pass condition: a binary on PATH, version 1.5 or newer.

Failure mode: not on PATH. Fix: install Terraform from terraform.io, or your distro’s package manager, then re-run doctor.

helm — required

The second hard-required binary (doctor flags it required). It is needed because FAR chart-version discovery — the data.external.versions lookup — runs helm registry login + helm pull to read the FLO/CIS chart versions during roksbnkctl up. Without helm on PATH, the apply fails partway through the cluster lifecycle with:

Error: local-exec provisioner error
Error running command 'helm upgrade --install cert-manager ...':
exit status 127. Output: /bin/sh: 1: helm: not found

Pass condition: a helm (v3.x) binary on PATH. Doctor parses helm version --short for the version detail.

Failure mode: not on PATH. Fix: install Helm 3 from helm.sh/docs/intro/install/, or via your distro’s package manager:

# Linux (Ubuntu/Debian — official Helm apt repo):
curl https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update && sudo apt-get install -y helm

# macOS:
brew install helm

# Windows:
choco install kubernetes-helm

The BNK charts (cert_manager / flo / cne_instance) install via the helm_release terraform resource, which uses the hashicorp/helm provider’s embedded Helm 3 runtime — no host helm is shelled out for the chart installs themselves. Doctor still flags helm required because FAR chart-version discovery shells out to host helm (helm registry login + helm pull) to read the FLO/CIS versions. See Chapter 10 §“The terraform-native deployment model”.

iperf3 — informational

Used only by roksbnkctl test throughput in its host-iperf3 modes. After Sprint 4 lands the k8s execution backend (PRD 03), iperf3 moves entirely in-cluster and this row goes away for the everyday user.

Failure mode: not on PATH. Fix: install iperf3 if you plan to use the throughput test today; otherwise ignore.

kubectl — informational

kubectl is informational. The everyday verbs (get, apply, describe, delete, logs, exec, port-forward) are native Go via client-go and live under roksbnkctl k. Missing host kubectl does not disable the happy path; it only disables the roksbnkctl kubectl <args...> passthrough.

If kubectl is on PATH, the row is still ✓ and shows the version line. If it’s missing, the row is informational, not a warning, and the detail explains where the equivalent functionality lives.

oc — informational

Same story as kubectl — the OpenShift-relevant verbs are internalised (Project, Route, and ImageStream are available via roksbnkctl k get). Host oc is preserved as an escape hatch; a missing oc is informational, not a warning.

ibmcloud — optional

Required only for the roksbnkctl ibmcloud <args...> passthrough. The cluster-lifecycle path uses IBM Go SDKs internally — roksbnkctl up does not shell out to ibmcloud — so you can skip this binary if you don’t need the passthrough.

kubeconfig

Resolves the kubeconfig path via $KUBECONFIG first, then ~/.kube/config. Cluster-side commands (status, logs, every k <verb>) need it.

roksbnkctl up writes the admin kubeconfig at ~/.kube/config (mode 0600) on a fresh apply.

If you already have a multi-cluster ~/.kube/config, do not point $KUBECONFIG at ~/.roksbnkctl/<workspace>/state/kubeconfig. That is a directory of per-phase kubeconfigs Terraform writes as a side effect of config_dir, each carrying a roughly one-hour IAM token that nothing refreshes. An hour after an apply they authenticate as nobody, and the only symptom is:

error: You must be logged in to the server (the server has asked for the client to provide credentials)

Use the workspace-scoped verbs instead — they select the kubeconfig that addresses this workspace’s cluster, and skip any whose credential has expired:

roksbnkctl -w <workspace> k get pods -A
roksbnkctl -w <workspace> kubectl get nodes

To refresh the admin kubeconfig at ~/.kube/config, run roksbnkctl kubeconfig --download. roksbnkctl doctor reports expired phase kubeconfigs under workspace phase kubeconfigs.

Failure mode: $KUBECONFIG and ~/.kube/config both missing. Fix: run roksbnkctl kubeconfig --download to fetch the admin kubeconfig from IBM Cloud.

workspace

Reports the resolved workspace name and whether its config.yaml exists.

  • ✓ default — the current workspace pointer at ~/.roksbnkctl/config.yaml resolves and the named workspace has a populated config.yaml.
  • ⚠ "default" not initialised — the directory may exist (created by roksbnkctl ws new) but config.yaml is empty. Run roksbnkctl init to populate.
  • ✗ no config context — the global config can’t be loaded at all.

The one-off -w / --workspace flag overrides which workspace doctor reports against. See Chapter 6 — Workspaces.

workspace permissions

Reports whether the workspace tree is owner-only, and tightens it if not.

config.yaml can hold ibmcloud.api_key_b64 and registry.generic_password_b64; state/terraform.tfstate holds whatever those resolve to. All of it is base64 at most, which is obfuscation and not encryption — the file mode is the only protection, so config.yaml is 0600 and the directories are 0700.

The check is read-only. Loading a workspace already tightens what it can, so anything still loose by the time doctor looks is something the repair could not fix — which is exactly what is worth reporting.

  • ✓ owner-only — nothing to do.
  • ✗ <path> readable by other users and could not be tightened (N path(s)) — the only case that needs your hands. A read-only mount, or a filesystem with no POSIX modes (a WSL DrvFs mount without metadata, an SMB share). Move the workspace onto a filesystem that can hold 0600, or use the keychain or env-var credential paths so nothing sensitive lands in the file at all.

The row is absent on Windows: Go’s Chmod there only toggles the read-only bit, so there is no owner-only mode to assert and a permanent unactionable warning would be worse than no row.

The scope is the workspace directory and its immediate children: config.yaml, registry-mirror.json, cluster-outputs.json, ssh/, and every state-* tree. Children are enumerated rather than matched against a list of known names, so a state tree added by a future phase is covered the day it appears. Owner-only on a state directory makes everything inside it unreachable regardless of those files’ own modes, which is why the check does not walk deeper.

The repair happens on load, not here, and announces itself once:

⚠ workspace "prod" was readable by other users on this host; tightened 3 path(s) to owner-only.
  config.yaml can hold your IBM Cloud API key and registry password. If this host is shared, rotate them.

Workspaces written by a release before v1.50.0 are 0644/0755 on disk and will produce that line the first time they are read. The exposure that already happened is not undone — on a shared host, rotate the API key.

ibmcloud api key

Resolves the API key via the chain documented in Chapter 14 — Credentials: env var → OS keychain → workspace config (base64) → TTY prompt.

Pass condition: the chain produces a non-empty key. The key value is never printed — only the source (“resolved”).

Failure mode: IBMCLOUD_API_KEY unset and no keychain entry for workspace "<name>". Fix: either export IBMCLOUD_API_KEY=... for the session, or re-run roksbnkctl init and accept the keychain-save prompt.

ibm cloud auth

Round-trips the resolved key against IBM IAM via the SDK (Verify() call). Confirms the key is not just present but actually authenticates.

Pass condition: IAM accepts the key; the row reports the resolved account and user identity.

Failure modes:

  • BXNIM0415E: Provided API key could not be found — the key is malformed or has been deleted in IBM Cloud.
  • network is unreachable / i/o timeout — your workstation can’t reach iam.cloud.ibm.com. Common in customer-firewall scenarios; route through a jumphost (Chapter 16) to confirm the key works from inside the customer network.

Common failures and how to fix them

The chapter readers most often land on. Each row maps a real-world symptom to its fix:

SymptomLikely causeFix
terraform not on PATHnot installedinstall Terraform >= 1.5; re-run doctor
kubeconfig: $KUBECONFIG and ~/.kube/config both missingnever ran up against this workspaceroksbnkctl kubeconfig --download or run roksbnkctl up
ibmcloud api key: ... no keychain entrynew shell, key not exportedexport IBMCLOUD_API_KEY=... or re-run roksbnkctl init
ibm cloud auth: BXNIM0415Ebad / rotated keyregenerate the key in the IBM Cloud console; update the keychain via roksbnkctl init
ibm cloud auth: i/o timeoutcorp-firewalled workstationuse --on jumphost to test from inside the customer network
workspace "foo" not initialisedws new was run but init was notrun roksbnkctl init -w foo
workspace: no config context~/.roksbnkctl/config.yaml corruptinspect the file; worst case delete it and re-run init

If a fix isn’t here, Chapter 26 — Troubleshooting covers the longer tail.

The --target <name> SSH check

The --on jumphost flag adds an optional second mode for doctor: probe an SSH target before you try to use it.

roksbnkctl doctor --target jumphost

This adds one row per resolved target:

✓  ssh:jumphost      ubuntu@169.45.91.177:22 (TOFU recorded)            (verifies the target is reachable)

The probe:

  1. Resolves the target’s host, user, port, and key source from ~/.roksbnkctl/<workspace>/config.yaml.
  2. Connects via the internal/remote SSH client.
  3. Validates the host key against ~/.roksbnkctl/known_hosts (TOFU prompt on first contact, unless --insecure-host-key).
  4. Runs a no-op command (true) to confirm the channel works end-to-end.

Failure modes specific to the SSH probe:

  • host key mismatch — the target was rebuilt; edit ~/.roksbnkctl/known_hosts to clear the entry, then re-probe.
  • unable to authenticate — the key source resolved but the remote rejected it. Check key_path / key_source in workspace config; if key_source: agent, verify ssh-add -l shows the right key.
  • dial tcp: i/o timeout — the host:port is unreachable. Verify with nc -vz <host> 22 from a known-good network.

Pass --target all to probe every target listed in the workspace’s targets: block. Useful in CI when you want a single command that asserts every entry is reachable.

Reading the exit code

doctor exits with:

  • 0 — all checks are green or warnings only. Warnings do not fail doctor. The everyday workflow can proceed.
  • non-zero — at least one row produced an ✗ error. The first error string is also written to stderr so wrapper scripts can grep it.

This is the contract scripts/e2e-test.sh and the Makefile rely on: a script that runs roksbnkctl doctor && roksbnkctl up --auto will only proceed past doctor if the environment is genuinely ready.

The “warnings don’t fail” rule is deliberate. An iperf3 not on PATH warning is informational — the everyday up / test connectivity flow doesn’t need it. Forcing exit-1 on every warning would be too aggressive for the common case.

If you want to gate scripts strictly (e.g. CI workflows that must have iperf3 installed because they run the throughput suite), parse the output rather than relying on the exit code:

if ! roksbnkctl doctor | grep -q '^✓  iperf3'; then
  echo "iperf3 missing — install it before running test throughput" >&2
  exit 1
fi

What doctor is not

A few deliberate non-features worth naming:

  • Not a fix-it tool. doctor reports; it never installs, never modifies workspace config, never calls IBM Cloud APIs that mutate state. The IAM verify call is read-only. If doctor could break things, users couldn’t run it freely — and “run doctor” needs to be a safe first move.
  • Not a backend probe. Per-backend availability checks (docker daemon reachable, k8s ops pod healthy, ssh target reachable) ship as separate BackendName-tagged rows via doctor --backend <name> (PRD 03). The --target probe was the early prototype of that pattern.
  • Not concurrent-safe. The CLI invokes doctor once per command; the side-channel for “why we care” blurbs in internal/doctor/doctor.go doesn’t synchronise. Don’t run two doctors against the same process.

Cross-references