Doctor: checking your environment
roksbnkctl doctor is the prereq + credentials report. It runs in under five seconds, exits non-zero on any hard error, and prints a tabular report that maps one-to-one to the runtime dependencies the rest of the tool reaches for.
This chapter walks every check, explains what each row’s “why we care” blurb means, covers why kubectl and oc are informational rather than required, and describes the --target SSH probe.
What doctor checks
A bare roksbnkctl doctor runs the general checks: tooling on PATH, kubeconfig location, the resolved workspace, and the IBM Cloud authentication chain. Sample output on a healthy machine looks like this:
roksbnkctl doctor
✓ terraform /usr/bin/terraform (Terraform v1.15.2) (required for `roksbnkctl up`)
✓ helm /usr/local/bin/helm (v3.20.2) (required for `roksbnkctl up`; terraform `local-exec` shells out to helm)
⚠ iperf3 not on PATH (needed for `roksbnkctl test throughput`)
✓ kubectl /usr/local/bin/kubectl (clientVersion:) (internalised in roksbnkctl k *; passthrough still works if installed)
✓ oc /usr/local/bin/oc (Client Version: 4.21.10) (internalised in roksbnkctl k *; passthrough still works if installed)
✓ ibmcloud /usr/local/bin/ibmcloud (ibmcloud 2.43.0 ...) (optional; `roksbnkctl ibmcloud` passthrough)
✓ kubeconfig /home/jgruber/.kube/config (needed for cluster-side ops)
✓ workspace default (per-environment config + state)
✓ workspace permissions owner-only (credentials in the workspace are protected by file mode alone)
✓ ibmcloud api key resolved (auth for terraform + IBM SDK calls)
✓ ibm cloud auth OK (account: 1a2b3c..., user: you@example.com) (verifies API key works against IBM IAM)
Each row has the same shape:
<status> <name> <detail> <why we care>
- status is one of
✓(green / OK),⚠(yellow / warning), or✗(red / error).Skippedchecks render as⚠. - name is the dependency or capability being checked.
- detail is the resolved value — usually a path, a version line, or an error message.
- why we care is a parenthetical clause naming the
roksbnkctlfeature that depends on this row.
The BackendName column on the underlying Check struct (internal/doctor/check.go) is reserved for the per-backend probes that land in Sprint 4 (PRD 03). Until then it stays empty for every general check.
Each check explained
terraform — required
One of two hard-required binaries for the roksbnkctl up happy path. roksbnkctl shells out to terraform via terraform-exec for plan/apply/destroy; without it nothing in the cluster lifecycle works.
Pass condition: a binary on PATH, version 1.5 or newer.
Failure mode: not on PATH. Fix: install Terraform from terraform.io, or your distro’s package manager, then re-run doctor.
helm — required
The second hard-required binary (doctor flags it required). It is needed because FAR chart-version discovery — the data.external.versions lookup — runs helm registry login + helm pull to read the FLO/CIS chart versions during roksbnkctl up. Without helm on PATH, the apply fails partway through the cluster lifecycle with:
Error: local-exec provisioner error
Error running command 'helm upgrade --install cert-manager ...':
exit status 127. Output: /bin/sh: 1: helm: not found
Pass condition: a helm (v3.x) binary on PATH. Doctor parses helm version --short for the version detail.
Failure mode: not on PATH. Fix: install Helm 3 from helm.sh/docs/intro/install/, or via your distro’s package manager:
# Linux (Ubuntu/Debian — official Helm apt repo):
curl https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update && sudo apt-get install -y helm
# macOS:
brew install helm
# Windows:
choco install kubernetes-helm
The BNK charts (cert_manager / flo / cne_instance) install via the helm_release terraform resource, which uses the hashicorp/helm provider’s embedded Helm 3 runtime — no host helm is shelled out for the chart installs themselves. Doctor still flags helm required because FAR chart-version discovery shells out to host helm (helm registry login + helm pull) to read the FLO/CIS versions. See Chapter 10 §“The terraform-native deployment model”.
iperf3 — informational
Used only by roksbnkctl test throughput in its host-iperf3 modes. After Sprint 4 lands the k8s execution backend (PRD 03), iperf3 moves entirely in-cluster and this row goes away for the everyday user.
Failure mode: not on PATH. Fix: install iperf3 if you plan to use the throughput test today; otherwise ignore.
kubectl — informational
kubectl is informational. The everyday verbs (get, apply, describe, delete, logs, exec, port-forward) are native Go via client-go and live under roksbnkctl k. Missing host kubectl does not disable the happy path; it only disables the roksbnkctl kubectl <args...> passthrough.
If kubectl is on PATH, the row is still ✓ and shows the version line. If it’s missing, the row is informational, not a warning, and the detail explains where the equivalent functionality lives.
oc — informational
Same story as kubectl — the OpenShift-relevant verbs are internalised (Project, Route, and ImageStream are available via roksbnkctl k get). Host oc is preserved as an escape hatch; a missing oc is informational, not a warning.
ibmcloud — optional
Required only for the roksbnkctl ibmcloud <args...> passthrough. The cluster-lifecycle path uses IBM Go SDKs internally — roksbnkctl up does not shell out to ibmcloud — so you can skip this binary if you don’t need the passthrough.
kubeconfig
Resolves the kubeconfig path via $KUBECONFIG first, then ~/.kube/config. Cluster-side commands (status, logs, every k <verb>) need it.
roksbnkctl up writes the admin kubeconfig at ~/.kube/config (mode 0600) on a fresh apply.
If you already have a multi-cluster ~/.kube/config, do not point $KUBECONFIG at
~/.roksbnkctl/<workspace>/state/kubeconfig. That is a directory of per-phase
kubeconfigs Terraform writes as a side effect of config_dir, each carrying a
roughly one-hour IAM token that nothing refreshes. An hour after an apply they
authenticate as nobody, and the only symptom is:
error: You must be logged in to the server (the server has asked for the client to provide credentials)
Use the workspace-scoped verbs instead — they select the kubeconfig that addresses this workspace’s cluster, and skip any whose credential has expired:
roksbnkctl -w <workspace> k get pods -A
roksbnkctl -w <workspace> kubectl get nodes
To refresh the admin kubeconfig at ~/.kube/config, run roksbnkctl kubeconfig --download.
roksbnkctl doctor reports expired phase kubeconfigs under workspace phase kubeconfigs.
Failure mode: $KUBECONFIG and ~/.kube/config both missing. Fix: run roksbnkctl kubeconfig --download to fetch the admin kubeconfig from IBM Cloud.
workspace
Reports the resolved workspace name and whether its config.yaml exists.
✓ default— the current workspace pointer at~/.roksbnkctl/config.yamlresolves and the named workspace has a populatedconfig.yaml.⚠ "default" not initialised— the directory may exist (created byroksbnkctl ws new) butconfig.yamlis empty. Runroksbnkctl initto populate.✗ no config context— the global config can’t be loaded at all.
The one-off -w / --workspace flag overrides which workspace doctor reports against. See Chapter 6 — Workspaces.
workspace permissions
Reports whether the workspace tree is owner-only, and tightens it if not.
config.yaml can hold ibmcloud.api_key_b64 and registry.generic_password_b64; state/terraform.tfstate holds whatever those resolve to. All of it is base64 at most, which is obfuscation and not encryption — the file mode is the only protection, so config.yaml is 0600 and the directories are 0700.
The check is read-only. Loading a workspace already tightens what it can, so anything still loose by the time doctor looks is something the repair could not fix — which is exactly what is worth reporting.
✓ owner-only— nothing to do.✗ <path> readable by other users and could not be tightened (N path(s))— the only case that needs your hands. A read-only mount, or a filesystem with no POSIX modes (a WSL DrvFs mount without metadata, an SMB share). Move the workspace onto a filesystem that can hold0600, or use the keychain or env-var credential paths so nothing sensitive lands in the file at all.
The row is absent on Windows: Go’s Chmod there only toggles the read-only bit, so there is no owner-only mode to assert and a permanent unactionable warning would be worse than no row.
The scope is the workspace directory and its immediate children: config.yaml, registry-mirror.json, cluster-outputs.json, ssh/, and every state-* tree. Children are enumerated rather than matched against a list of known names, so a state tree added by a future phase is covered the day it appears. Owner-only on a state directory makes everything inside it unreachable regardless of those files’ own modes, which is why the check does not walk deeper.
The repair happens on load, not here, and announces itself once:
⚠ workspace "prod" was readable by other users on this host; tightened 3 path(s) to owner-only.
config.yaml can hold your IBM Cloud API key and registry password. If this host is shared, rotate them.
Workspaces written by a release before v1.50.0 are 0644/0755 on disk and will produce that line the first time they are read. The exposure that already happened is not undone — on a shared host, rotate the API key.
ibmcloud api key
Resolves the API key via the chain documented in Chapter 14 — Credentials: env var → OS keychain → workspace config (base64) → TTY prompt.
Pass condition: the chain produces a non-empty key. The key value is never printed — only the source (“resolved”).
Failure mode: IBMCLOUD_API_KEY unset and no keychain entry for workspace "<name>". Fix: either export IBMCLOUD_API_KEY=... for the session, or re-run roksbnkctl init and accept the keychain-save prompt.
ibm cloud auth
Round-trips the resolved key against IBM IAM via the SDK (Verify() call). Confirms the key is not just present but actually authenticates.
Pass condition: IAM accepts the key; the row reports the resolved account and user identity.
Failure modes:
BXNIM0415E: Provided API key could not be found— the key is malformed or has been deleted in IBM Cloud.network is unreachable/i/o timeout— your workstation can’t reachiam.cloud.ibm.com. Common in customer-firewall scenarios; route through a jumphost (Chapter 16) to confirm the key works from inside the customer network.
Common failures and how to fix them
The chapter readers most often land on. Each row maps a real-world symptom to its fix:
| Symptom | Likely cause | Fix |
|---|---|---|
terraform not on PATH | not installed | install Terraform >= 1.5; re-run doctor |
kubeconfig: $KUBECONFIG and ~/.kube/config both missing | never ran up against this workspace | roksbnkctl kubeconfig --download or run roksbnkctl up |
ibmcloud api key: ... no keychain entry | new shell, key not exported | export IBMCLOUD_API_KEY=... or re-run roksbnkctl init |
ibm cloud auth: BXNIM0415E | bad / rotated key | regenerate the key in the IBM Cloud console; update the keychain via roksbnkctl init |
ibm cloud auth: i/o timeout | corp-firewalled workstation | use --on jumphost to test from inside the customer network |
workspace "foo" not initialised | ws new was run but init was not | run roksbnkctl init -w foo |
workspace: no config context | ~/.roksbnkctl/config.yaml corrupt | inspect the file; worst case delete it and re-run init |
If a fix isn’t here, Chapter 26 — Troubleshooting covers the longer tail.
The --target <name> SSH check
The --on jumphost flag adds an optional second mode for doctor: probe an SSH target before you try to use it.
roksbnkctl doctor --target jumphost
This adds one row per resolved target:
✓ ssh:jumphost ubuntu@169.45.91.177:22 (TOFU recorded) (verifies the target is reachable)
The probe:
- Resolves the target’s
host,user,port, and key source from~/.roksbnkctl/<workspace>/config.yaml. - Connects via the
internal/remoteSSH client. - Validates the host key against
~/.roksbnkctl/known_hosts(TOFU prompt on first contact, unless--insecure-host-key). - Runs a no-op command (
true) to confirm the channel works end-to-end.
Failure modes specific to the SSH probe:
host key mismatch— the target was rebuilt; edit~/.roksbnkctl/known_hoststo clear the entry, then re-probe.unable to authenticate— the key source resolved but the remote rejected it. Checkkey_path/key_sourcein workspace config; ifkey_source: agent, verifyssh-add -lshows the right key.dial tcp: i/o timeout— thehost:portis unreachable. Verify withnc -vz <host> 22from a known-good network.
Pass --target all to probe every target listed in the workspace’s targets: block. Useful in CI when you want a single command that asserts every entry is reachable.
Reading the exit code
doctor exits with:
0— all checks are green or warnings only. Warnings do not faildoctor. The everyday workflow can proceed.- non-zero — at least one row produced an
✗error. The first error string is also written to stderr so wrapper scripts can grep it.
This is the contract scripts/e2e-test.sh and the Makefile rely on: a script that runs roksbnkctl doctor && roksbnkctl up --auto will only proceed past doctor if the environment is genuinely ready.
The “warnings don’t fail” rule is deliberate. An iperf3 not on PATH warning is informational — the everyday up / test connectivity flow doesn’t need it. Forcing exit-1 on every warning would be too aggressive for the common case.
If you want to gate scripts strictly (e.g. CI workflows that must have iperf3 installed because they run the throughput suite), parse the output rather than relying on the exit code:
if ! roksbnkctl doctor | grep -q '^✓ iperf3'; then
echo "iperf3 missing — install it before running test throughput" >&2
exit 1
fi
What doctor is not
A few deliberate non-features worth naming:
- Not a fix-it tool.
doctorreports; it never installs, never modifies workspace config, never calls IBM Cloud APIs that mutate state. The IAM verify call is read-only. Ifdoctorcould break things, users couldn’t run it freely — and “rundoctor” needs to be a safe first move. - Not a backend probe. Per-backend availability checks (docker daemon reachable, k8s ops pod healthy, ssh target reachable) ship as separate
BackendName-tagged rows viadoctor --backend <name>(PRD 03). The--targetprobe was the early prototype of that pattern. - Not concurrent-safe. The CLI invokes
doctoronce per command; the side-channel for “why we care” blurbs ininternal/doctor/doctor.godoesn’t synchronise. Don’t run twodoctors against the same process.
Cross-references
- Chapter 4 — Installation introduces
doctoras the post-install verification step. - Chapter 6 — Workspaces explains the
workspacerow and the-woverride. - Chapter 14 — Credentials is the deep dive on the
ibmcloud api keyresolution chain. - Chapter 16 — The
--onflag covers the--targetprobe’s underlying SSH client. - Chapter 24 — Day-2 ops is the canonical reference for the internalised
k <verb>commands that makekubectl/ocinformational.