The cluster phase (cluster up/down)
A workspace is three phases — Cluster / BNK / Testing. This chapter covers the Cluster phase specifically; the BNK trial and the testing jumphosts are separate sibling phases that deploy in parallel on top. The “trial phase” language in this chapter means the BNK phase, and the jumphosts live in their own Testing phase. See Chapter 8a — The three-phase lifecycle for the full model, parallel
up, androksbnkctl testing up/down.
A roksbnkctl workspace is two phases on top of each other: a durable cluster phase (the ROKS cluster + cluster-shared services that take 30+ minutes to provision) and a short-lived trial phase (the BNK trial that iterates on top in 5-10 minutes). The cluster phase is exposed as its own command pair, roksbnkctl cluster up / roksbnkctl cluster down, so the cluster survives across many BNK trial cycles.
This two-phase shape is the default for every new workspace. A fresh
roksbnkctl upprovisions the cluster phase first, then the trial phase, against separate state directories. Tearing down only the trial — the common iteration case — usesroksbnkctl bnk downand leaves the cluster intact. The unscopedup/downverbs are shape-aware composites that delegate to the right phase commands underneath.
This chapter covers what each phase deploys, why the two state directories are separate, the deploy_bnk=false override that makes “cluster only” work, the cluster-outputs.json artefact written on success, and a worked example. The companion BNK-trial chapter, Chapter 10, covers roksbnkctl bnk up / bnk down for the trial layer.
What’s deployed where
The bundled HCL has roughly two halves. The cluster phase owns the durable, cluster-scoped resources:
- The ROKS cluster itself (VPC + subnets + worker pool)
- A transit gateway (so the test jumphost can reach cluster internals)
- The registry COS (Cloud Object Storage) instance — used by the BNK trial as its FAR image / license / schematic store
cert-manager(Helm release into the cluster)- The TGW jumphost VM (an Ubuntu VSI in the same VPC, used by
--on jumphost)
The trial phase owns the BNK-specific resources:
- F5 Lifecycle Operator (
flo) Helm release cne_instanceKubernetes manifest- BNK license + admin certs
- Various cluster-side bits: ServiceAccounts, RoleBindings, Secrets
Two-phase split: cluster up provisions the first list; roksbnkctl up (the trial) provisions the second.
┌─────────────────────────────────────────────────────────┐
│ cluster phase (durable, reused across many trials) │
│ ROKS cluster + VPC + transit gateway │
│ registry COS instance │
│ cert-manager (Helm) │
│ TGW jumphost │
├─────────────────────────────────────────────────────────┤
│ trial phase (one trial — destroyed by `roksbnkctl down`)│
│ flo (F5 Lifecycle Operator) │
│ cne_instance │
│ license / admin cert / SCC bindings │
└─────────────────────────────────────────────────────────┘
The split exists because ROKS clusters take 30-50 minutes to provision and roughly $0.30/hour to run. Re-creating the cluster every time you want to re-test a BNK trial is wasteful; reusing one cluster for many trials cuts iteration time from “an hour” to “a few minutes”.
The two state directories
To keep cluster state and trial state from tangling, roksbnkctl uses separate Terraform state directories:
~/.roksbnkctl/<workspace>/
state/ # BNK trial state — written by `roksbnkctl up/down`
terraform.tfstate
terraform.tfvars
state-cluster/ # cluster phase state — written by `roksbnkctl cluster up/down`
terraform.tfstate
cluster-phase-override.tfvars
Each phase’s commands read and write only their own state directory. Both phases use the same Terraform source (the bundled HCL) but with different effective tfvars — the trick is the deploy_bnk flag.
The deploy_bnk=false override
The bundled HCL has a top-level deploy_bnk boolean. When true, the BNK trial modules (flo, cne_instance, license) run; when false, they’re skipped and Terraform only provisions the cluster-phase resources.
roksbnkctl cluster up and roksbnkctl cluster down force deploy_bnk = false by writing a small auto-generated tfvars override into the cluster state directory:
# ~/.roksbnkctl/<workspace>/state-cluster/cluster-phase-override.tfvars
# Generated by roksbnkctl. Do not edit by hand.
# Cluster-phase override: BNK trial modules (flo / cne_instance /
# license) are skipped. cert-manager and the testing jumphost still run
# — they're cluster-shared singletons that belong with the cluster.
deploy_bnk = false
This file is layered onto the var-file chain after user-supplied --var-file flags so the override always wins. The user’s terraform.tfvars and --var-file <path> arguments still apply for everything else (region, RG, cluster name, worker count, …) — only deploy_bnk is forced.
roksbnkctl up doesn’t write this override file; its tfvars chain leaves deploy_bnk at the upstream default (true), so the trial modules run.
Reusing an existing VPC (multiple clusters in one VPC)
By default the cluster phase creates a prefix-named VPC for the cluster it builds. But you can point a new cluster at a VPC that already exists — so several clusters (each its own workspace) live in one shared VPC: the shape behind a shared licensing cluster or a common services network.
Decline “Create a new cluster VPC?” in init and it discovers the VPCs in the
chosen region and lets you pick one by number (since v1.26.0):
$ roksbnkctl -w app-b init
...
Create a new cluster VPC? [Y/n] n
→ Discovering existing VPCs...
Existing VPCs in us-south:
1) shared-services-vpc (available)
2) app-a-cluster-vpc (available)
Use an existing cluster VPC — pick a number (0 = none / create a new one): 1
Picking 0 (or a region with no VPCs) falls back to creating one — a cluster must
have a VPC. In config that is (the value is the VPC id, unlike the
transit-gateway adopt-by-name):
resources:
cluster_vpc:
create: false
existing: r006-6fe0b20a-... # the VPC id
which renders use_existing_cluster_vpc = true + existing_cluster_vpc_id. Two
things make this safe for more than one cluster in a VPC: the cluster’s subnets are
named per-cluster (<cluster>-subnet-zoneN) so they don’t collide, and per-zone
public gateways are reused when the VPC already has them (IBM allows exactly
one per zone per VPC). Make sure the VPC has free address-prefix space — each
cluster adds three /24 subnets.
Combine this with an adopted Transit Gateway: clusters that share a VPC share that VPC’s single connection to the gateway, so N clusters can reach each other over one VPC and one gateway.
Teardown order matters in a shared VPC. The cluster that created the VPC owns its per-zone public gateways (IBM allows only one gateway per zone per VPC, so the adopters attach to the owner’s). The owner must be destroyed last — tear the adopters down first.
roksbnkctlenforces this:cluster down/downrefuses to destroy a workspace whose VPC still holds another cluster’s subnets, naming them, rather than failing mid-destroy withThe VPC is in use. (Sincev1.26.1; subnets also attach their gateway inline now, so an adopter’s teardown no longer errors even if the owner’s gateway is already gone.)
cluster-outputs.json — the cluster identity record
When roksbnkctl cluster up apply succeeds, it writes a workspace-scoped JSON file describing the cluster. Most of it comes from Terraform outputs (cluster name, ID, region, RG, VPC, registry COS); the create-time settings — network_mode, and vpc_cidr when this tool created the VPC — come from the workspace config instead, because that is the only place they are knowable and the moment they become settled:
~/.roksbnkctl/<workspace>/cluster-outputs.json
Sample contents:
{
"cluster_name": "bnk-quickstart",
"cluster_id": "cre6h4l20jjsg4kvt3a0",
"region": "us-south",
"resource_group_id": "abc123...",
"vpc_id": "r006-...",
"schema_version": 2,
"network_mode": "single-nic",
"vpc_cidr": "10.242.0.0/16",
"registry_cos_crn": "crn:v1:bluemix:public:cloud-object-storage:global:a/...",
"registry_cos_name": "bnk-quickstart-cos-instance",
"master_url": "https://c106.us-south.containers.cloud.ibm.com:31415",
"openshift_version": "4.14_openshift",
"source": "cluster-up",
"recorded_at": "2026-05-08T14:22:08Z"
}
The source field discriminates between cluster-up (we created it) and cluster-register (we discovered an existing cluster — see Chapter 9). Subsequent commands read this file to learn the workspace’s cluster identity without hitting IBM APIs.
Reading this file from your own tooling
The file is a versioned contract, and the rules for reading it matter more than the field list, because every rule exists to keep a file written by one version readable by another.
schema_versionabsent means 1. Every record written before the field existed describes a real cluster and must stay readable; treating a missing value as0would invalidate all of them.network_modeabsent meanssingle-nic. That is what every cluster built before the field existed actually is.vpc_cidrabsent means unknown, not “no block was set.” It is recorded only when this tool created the VPC; on the adopt path the setting never applied, so recording the configured value would record something that was never true.node_interfacesis declared but written by nothing today. Multi-NIC ROKS has not shipped, so no code path can populate it honestly yet. Empty means unknown — never “this cluster has no extra interfaces”. Both readings are true of every cluster that exists right now, which is exactly what makes the wrong one easy to adopt and expensive to unwind later.- Unknown fields are ignored, deliberately. New keys are only ever added, so an older reader keeps working against a newer file.
For the create-time settings, this file — not config.yaml — is the authority.
See Chapter 8a for what happens
when the two disagree.
roksbnkctl cluster down deletes the file as part of its post-destroy cleanup. roksbnkctl cluster show pretty-prints it for human readers:
roksbnkctl cluster show
workspace: default
source: cluster-up
recorded_at: 2026-05-08T14:22:08Z
cluster_name: bnk-quickstart
cluster_id: cre6h4l20jjsg4kvt3a0
region: us-south
resource_group: abc123...
openshift: 4.14_openshift
master_url: https://c106.us-south.containers.cloud.ibm.com:31415
vpc_id: r006-...
registry_cos: bnk-quickstart-cos-instance
registry_cos_crn: crn:v1:bluemix:public:cloud-object-storage:global:a/...
Worked example: cluster up → kubectl get nodes → cluster down
The cluster-only flow, end to end:
Step 1 — roksbnkctl init
If you don’t have a workspace yet, initialise one. This is the same init flow as the trial path; the cluster commands reuse the workspace’s config.
roksbnkctl init
Step 2 — roksbnkctl cluster up --auto
Provisions the cluster phase only:
roksbnkctl cluster up --auto
Sample output (heavily abridged):
→ terraform plan (cluster phase: deploy_bnk=false forced)
→ Layering user tfvars from ~/.roksbnkctl/default/state-cluster/cluster-phase-override.tfvars (overrides config.yaml-derived values)
→ terraform init
→ terraform apply
module.roks_cluster.ibm_container_vpc_cluster.cluster: Creating...
module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [10m elapsed]
...
module.roks_cluster.ibm_container_vpc_cluster.cluster: Creation complete after 38m12s
module.cert_manager.helm_release.cert_manager: Creation complete after 2m11s
module.testing.tls_private_key.jumphost_shared_key: Creation complete after 0s
module.testing.ibm_is_instance.tgw_jumphost: Creation complete after 1m48s
Apply complete! Resources: 36 added, 0 changed, 0 destroyed.
✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json
✓ Wrote /home/you/.kube/config (chmod 0600)
✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`
Roughly 36 resources land — the cluster phase is about half the size of a full BNK trial. Time-to-ready is dominated by the ROKS cluster itself; everything else after the cluster comes up is fast.
Step 3 — verify the cluster works
The post-apply admin kubeconfig is fetched automatically (unless --no-kubeconfig). kubectl get nodes confirms reachability:
kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# 10.243.0.4 Ready master,worker 3m v1.28.6+5e1b9a1
# 10.243.64.4 Ready master,worker 3m v1.28.6+5e1b9a1
Or the same thing through the internalised verb:
roksbnkctl k get nodes
roksbnkctl status reports cluster identity + reachability:
roksbnkctl status
Workspace: default
Region: us-south
Cluster: bnk-quickstart (attach existing)
TF source: embedded@v1.0.0
Last apply: 2026-05-08 14:22:08 UTC (3m ago)
Kubeconfig: /home/you/.kube/config
Cluster: 2/2 nodes ready
Step 4 — (optional) deploy a BNK trial on top
Now that the cluster is up, roksbnkctl up deploys a BNK trial onto it. It reads cluster-outputs.json and reuses the cluster:
roksbnkctl up --auto
See Chapter 10 — Deploying BNK trials for the trial-phase walkthrough. You can run up / down many times against the same cluster — each cycle is ~5 minutes rather than the ~50 minutes of a fresh-cluster run.
Step 5 — roksbnkctl cluster down --auto
Tear down the cluster phase. cluster down is strictly scoped: it refuses with a hard error on any workspace whose trial state is non-empty, so an out-of-order destroy can’t accidentally orphan BNK resources. Destroy the trial first with roksbnkctl bnk down (or roksbnkctl down for both at once); see Chapter 11 for the full refusal catalogue.
roksbnkctl cluster down --auto
Sample output:
→ terraform destroy (cluster phase)
module.testing.ibm_is_instance.tgw_jumphost: Destroying...
module.cert_manager.helm_release.cert_manager: Destroying...
module.roks_cluster.ibm_container_vpc_cluster.cluster: Destroying...
module.roks_cluster.ibm_container_vpc_cluster.cluster: Still destroying... [5m elapsed]
module.roks_cluster.ibm_container_vpc_cluster.cluster: Destruction complete after 8m16s
Destroy complete! Resources: 36 destroyed.
Post-destroy, cluster-outputs.json is deleted. The workspace directory and its config.yaml survive — re-running cluster up against the same workspace re-creates the cluster with the same name and region.
Why split cluster from trial?
Two-phase is the default because the cost of conflating them is concrete. ROKS clusters take 30-50 minutes to provision and bill at roughly $0.30/hour; a BNK trial on top takes 5-10 minutes. Iterating on the trial — different flo versions, different cne_instance shapes, license bundle revisions — happens far more often than iterating on the cluster underneath. Splitting state means a bnk down / bnk up cycle is a five-minute round-trip instead of an hour.
Three scenarios this shape unlocks:
-
Many BNK trial iterations on one cluster. Run
cluster uponce, then loopbnk up/bnk downagainst the same cluster until you’ve covered all the trial permutations. Thencluster downonce when you’re finished. This is the headline win of the phase split — see Chapter 10 §“Worked example — iterating on a BNK trial”. -
Pre-provisioning for a workshop or demo. You want the cluster ready and warm before the demo starts; you’ll deploy the BNK trial live in front of the audience.
cluster upthe night before;bnk upduring the demo. -
Decoupling cluster lifecycle from trial lifecycle. A long-lived cluster used by multiple team members, where one person owns the cluster phase and others own the BNK trials. Cluster-phase outputs live in
cluster-outputs.json; trials read it. Each trial canbnk up/bnk downwithout affecting the cluster.
For workspaces that just want “create a cluster, deploy BNK on it, test, tear it all down”, the unscoped roksbnkctl up / roksbnkctl down are the right verbs — they’re shape-aware composites that drive the cluster + trial steps in the right order without you having to think about it.
Cross-references
- Chapter 9 — Registering an existing cluster — the alternative to
cluster upwhen you already have a ROKS cluster you wantroksbnkctlto manage. - Chapter 10 — Deploying BNK trials —
roksbnkctl upand thebnk up/bnk downcommand group; the dispatch matrix; iteration walkthrough. - Chapter 11 — Tearing down — phase-aware decision matrix and the refusal-message catalogue.
- Chapter 24 — Day-2 ops —
roksbnkctl k get/apply/logsfor working against the cluster after either phase.