Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

The cluster phase (cluster up/down)

A workspace is three phases — Cluster / BNK / Testing. This chapter covers the Cluster phase specifically; the BNK trial and the testing jumphosts are separate sibling phases that deploy in parallel on top. The “trial phase” language in this chapter means the BNK phase, and the jumphosts live in their own Testing phase. See Chapter 8a — The three-phase lifecycle for the full model, parallel up, and roksbnkctl testing up/down.

A roksbnkctl workspace is two phases on top of each other: a durable cluster phase (the ROKS cluster + cluster-shared services that take 30+ minutes to provision) and a short-lived trial phase (the BNK trial that iterates on top in 5-10 minutes). The cluster phase is exposed as its own command pair, roksbnkctl cluster up / roksbnkctl cluster down, so the cluster survives across many BNK trial cycles.

This two-phase shape is the default for every new workspace. A fresh roksbnkctl up provisions the cluster phase first, then the trial phase, against separate state directories. Tearing down only the trial — the common iteration case — uses roksbnkctl bnk down and leaves the cluster intact. The unscoped up / down verbs are shape-aware composites that delegate to the right phase commands underneath.

This chapter covers what each phase deploys, why the two state directories are separate, the deploy_bnk=false override that makes “cluster only” work, the cluster-outputs.json artefact written on success, and a worked example. The companion BNK-trial chapter, Chapter 10, covers roksbnkctl bnk up / bnk down for the trial layer.

What’s deployed where

The bundled HCL has roughly two halves. The cluster phase owns the durable, cluster-scoped resources:

  • The ROKS cluster itself (VPC + subnets + worker pool)
  • A transit gateway (so the test jumphost can reach cluster internals)
  • The registry COS (Cloud Object Storage) instance — used by the BNK trial as its FAR image / license / schematic store
  • cert-manager (Helm release into the cluster)
  • The TGW jumphost VM (an Ubuntu VSI in the same VPC, used by --on jumphost)

The trial phase owns the BNK-specific resources:

  • F5 Lifecycle Operator (flo) Helm release
  • cne_instance Kubernetes manifest
  • BNK license + admin certs
  • Various cluster-side bits: ServiceAccounts, RoleBindings, Secrets

Two-phase split: cluster up provisions the first list; roksbnkctl up (the trial) provisions the second.

┌─────────────────────────────────────────────────────────┐
│  cluster phase (durable, reused across many trials)     │
│    ROKS cluster + VPC + transit gateway                 │
│    registry COS instance                                │
│    cert-manager (Helm)                                  │
│    TGW jumphost                                         │
├─────────────────────────────────────────────────────────┤
│  trial phase (one trial — destroyed by `roksbnkctl down`)│
│    flo (F5 Lifecycle Operator)                          │
│    cne_instance                                         │
│    license / admin cert / SCC bindings                  │
└─────────────────────────────────────────────────────────┘

The split exists because ROKS clusters take 30-50 minutes to provision and roughly $0.30/hour to run. Re-creating the cluster every time you want to re-test a BNK trial is wasteful; reusing one cluster for many trials cuts iteration time from “an hour” to “a few minutes”.

The two state directories

To keep cluster state and trial state from tangling, roksbnkctl uses separate Terraform state directories:

~/.roksbnkctl/<workspace>/
  state/                   # BNK trial state — written by `roksbnkctl up/down`
    terraform.tfstate
    terraform.tfvars
  state-cluster/           # cluster phase state — written by `roksbnkctl cluster up/down`
    terraform.tfstate
    cluster-phase-override.tfvars

Each phase’s commands read and write only their own state directory. Both phases use the same Terraform source (the bundled HCL) but with different effective tfvars — the trick is the deploy_bnk flag.

The deploy_bnk=false override

The bundled HCL has a top-level deploy_bnk boolean. When true, the BNK trial modules (flo, cne_instance, license) run; when false, they’re skipped and Terraform only provisions the cluster-phase resources.

roksbnkctl cluster up and roksbnkctl cluster down force deploy_bnk = false by writing a small auto-generated tfvars override into the cluster state directory:

# ~/.roksbnkctl/<workspace>/state-cluster/cluster-phase-override.tfvars
# Generated by roksbnkctl. Do not edit by hand.
# Cluster-phase override: BNK trial modules (flo / cne_instance /
# license) are skipped. cert-manager and the testing jumphost still run
# — they're cluster-shared singletons that belong with the cluster.
deploy_bnk = false

This file is layered onto the var-file chain after user-supplied --var-file flags so the override always wins. The user’s terraform.tfvars and --var-file <path> arguments still apply for everything else (region, RG, cluster name, worker count, …) — only deploy_bnk is forced.

roksbnkctl up doesn’t write this override file; its tfvars chain leaves deploy_bnk at the upstream default (true), so the trial modules run.

Reusing an existing VPC (multiple clusters in one VPC)

By default the cluster phase creates a prefix-named VPC for the cluster it builds. But you can point a new cluster at a VPC that already exists — so several clusters (each its own workspace) live in one shared VPC: the shape behind a shared licensing cluster or a common services network.

Decline “Create a new cluster VPC?” in init and it discovers the VPCs in the chosen region and lets you pick one by number (since v1.26.0):

$ roksbnkctl -w app-b init
...
Create a new cluster VPC? [Y/n] n
→ Discovering existing VPCs...
  Existing VPCs in us-south:
     1) shared-services-vpc       (available)
     2) app-a-cluster-vpc         (available)
Use an existing cluster VPC — pick a number (0 = none / create a new one): 1

Picking 0 (or a region with no VPCs) falls back to creating one — a cluster must have a VPC. In config that is (the value is the VPC id, unlike the transit-gateway adopt-by-name):

resources:
  cluster_vpc:
    create: false
    existing: r006-6fe0b20a-...   # the VPC id

which renders use_existing_cluster_vpc = true + existing_cluster_vpc_id. Two things make this safe for more than one cluster in a VPC: the cluster’s subnets are named per-cluster (<cluster>-subnet-zoneN) so they don’t collide, and per-zone public gateways are reused when the VPC already has them (IBM allows exactly one per zone per VPC). Make sure the VPC has free address-prefix space — each cluster adds three /24 subnets.

Combine this with an adopted Transit Gateway: clusters that share a VPC share that VPC’s single connection to the gateway, so N clusters can reach each other over one VPC and one gateway.

Teardown order matters in a shared VPC. The cluster that created the VPC owns its per-zone public gateways (IBM allows only one gateway per zone per VPC, so the adopters attach to the owner’s). The owner must be destroyed last — tear the adopters down first. roksbnkctl enforces this: cluster down / down refuses to destroy a workspace whose VPC still holds another cluster’s subnets, naming them, rather than failing mid-destroy with The VPC is in use. (Since v1.26.1; subnets also attach their gateway inline now, so an adopter’s teardown no longer errors even if the owner’s gateway is already gone.)

cluster-outputs.json — the cluster identity record

When roksbnkctl cluster up apply succeeds, it writes a workspace-scoped JSON file describing the cluster. Most of it comes from Terraform outputs (cluster name, ID, region, RG, VPC, registry COS); the create-time settings — network_mode, and vpc_cidr when this tool created the VPC — come from the workspace config instead, because that is the only place they are knowable and the moment they become settled:

~/.roksbnkctl/<workspace>/cluster-outputs.json

Sample contents:

{
  "cluster_name": "bnk-quickstart",
  "cluster_id": "cre6h4l20jjsg4kvt3a0",
  "region": "us-south",
  "resource_group_id": "abc123...",
  "vpc_id": "r006-...",
  "schema_version": 2,
  "network_mode": "single-nic",
  "vpc_cidr": "10.242.0.0/16",
  "registry_cos_crn": "crn:v1:bluemix:public:cloud-object-storage:global:a/...",
  "registry_cos_name": "bnk-quickstart-cos-instance",
  "master_url": "https://c106.us-south.containers.cloud.ibm.com:31415",
  "openshift_version": "4.14_openshift",
  "source": "cluster-up",
  "recorded_at": "2026-05-08T14:22:08Z"
}

The source field discriminates between cluster-up (we created it) and cluster-register (we discovered an existing cluster — see Chapter 9). Subsequent commands read this file to learn the workspace’s cluster identity without hitting IBM APIs.

Reading this file from your own tooling

The file is a versioned contract, and the rules for reading it matter more than the field list, because every rule exists to keep a file written by one version readable by another.

  • schema_version absent means 1. Every record written before the field existed describes a real cluster and must stay readable; treating a missing value as 0 would invalidate all of them.
  • network_mode absent means single-nic. That is what every cluster built before the field existed actually is.
  • vpc_cidr absent means unknown, not “no block was set.” It is recorded only when this tool created the VPC; on the adopt path the setting never applied, so recording the configured value would record something that was never true.
  • node_interfaces is declared but written by nothing today. Multi-NIC ROKS has not shipped, so no code path can populate it honestly yet. Empty means unknown — never “this cluster has no extra interfaces”. Both readings are true of every cluster that exists right now, which is exactly what makes the wrong one easy to adopt and expensive to unwind later.
  • Unknown fields are ignored, deliberately. New keys are only ever added, so an older reader keeps working against a newer file.

For the create-time settings, this file — not config.yaml — is the authority. See Chapter 8a for what happens when the two disagree.

roksbnkctl cluster down deletes the file as part of its post-destroy cleanup. roksbnkctl cluster show pretty-prints it for human readers:

roksbnkctl cluster show
workspace:        default
source:           cluster-up
recorded_at:      2026-05-08T14:22:08Z

cluster_name:     bnk-quickstart
cluster_id:       cre6h4l20jjsg4kvt3a0
region:           us-south
resource_group:   abc123...
openshift:        4.14_openshift
master_url:       https://c106.us-south.containers.cloud.ibm.com:31415

vpc_id:           r006-...
registry_cos:     bnk-quickstart-cos-instance
registry_cos_crn: crn:v1:bluemix:public:cloud-object-storage:global:a/...

Worked example: cluster up → kubectl get nodes → cluster down

The cluster-only flow, end to end:

Step 1 — roksbnkctl init

If you don’t have a workspace yet, initialise one. This is the same init flow as the trial path; the cluster commands reuse the workspace’s config.

roksbnkctl init

Step 2 — roksbnkctl cluster up --auto

Provisions the cluster phase only:

roksbnkctl cluster up --auto

Sample output (heavily abridged):

→ terraform plan (cluster phase: deploy_bnk=false forced)
→ Layering user tfvars from ~/.roksbnkctl/default/state-cluster/cluster-phase-override.tfvars (overrides config.yaml-derived values)
→ terraform init
→ terraform apply
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Creating...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still creating... [10m elapsed]
  ...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Creation complete after 38m12s
  module.cert_manager.helm_release.cert_manager: Creation complete after 2m11s
  module.testing.tls_private_key.jumphost_shared_key: Creation complete after 0s
  module.testing.ibm_is_instance.tgw_jumphost: Creation complete after 1m48s

  Apply complete! Resources: 36 added, 0 changed, 0 destroyed.

✓ Wrote ~/.roksbnkctl/default/cluster-outputs.json
✓ Wrote /home/you/.kube/config (chmod 0600)
✓ Auto-registered target jumphost (169.45.91.177); use `roksbnkctl --on jumphost ...`

Roughly 36 resources land — the cluster phase is about half the size of a full BNK trial. Time-to-ready is dominated by the ROKS cluster itself; everything else after the cluster comes up is fast.

Step 3 — verify the cluster works

The post-apply admin kubeconfig is fetched automatically (unless --no-kubeconfig). kubectl get nodes confirms reachability:

kubectl get nodes
# NAME           STATUS   ROLES           AGE   VERSION
# 10.243.0.4     Ready    master,worker   3m    v1.28.6+5e1b9a1
# 10.243.64.4    Ready    master,worker   3m    v1.28.6+5e1b9a1

Or the same thing through the internalised verb:

roksbnkctl k get nodes

roksbnkctl status reports cluster identity + reachability:

roksbnkctl status
Workspace:    default
Region:       us-south
Cluster:      bnk-quickstart  (attach existing)
TF source:    embedded@v1.0.0
Last apply:   2026-05-08 14:22:08 UTC  (3m ago)
Kubeconfig:   /home/you/.kube/config
Cluster:      2/2 nodes ready

Step 4 — (optional) deploy a BNK trial on top

Now that the cluster is up, roksbnkctl up deploys a BNK trial onto it. It reads cluster-outputs.json and reuses the cluster:

roksbnkctl up --auto

See Chapter 10 — Deploying BNK trials for the trial-phase walkthrough. You can run up / down many times against the same cluster — each cycle is ~5 minutes rather than the ~50 minutes of a fresh-cluster run.

Step 5 — roksbnkctl cluster down --auto

Tear down the cluster phase. cluster down is strictly scoped: it refuses with a hard error on any workspace whose trial state is non-empty, so an out-of-order destroy can’t accidentally orphan BNK resources. Destroy the trial first with roksbnkctl bnk down (or roksbnkctl down for both at once); see Chapter 11 for the full refusal catalogue.

roksbnkctl cluster down --auto

Sample output:

→ terraform destroy (cluster phase)
  module.testing.ibm_is_instance.tgw_jumphost: Destroying...
  module.cert_manager.helm_release.cert_manager: Destroying...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Destroying...
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Still destroying... [5m elapsed]
  module.roks_cluster.ibm_container_vpc_cluster.cluster: Destruction complete after 8m16s

  Destroy complete! Resources: 36 destroyed.

Post-destroy, cluster-outputs.json is deleted. The workspace directory and its config.yaml survive — re-running cluster up against the same workspace re-creates the cluster with the same name and region.

Why split cluster from trial?

Two-phase is the default because the cost of conflating them is concrete. ROKS clusters take 30-50 minutes to provision and bill at roughly $0.30/hour; a BNK trial on top takes 5-10 minutes. Iterating on the trial — different flo versions, different cne_instance shapes, license bundle revisions — happens far more often than iterating on the cluster underneath. Splitting state means a bnk down / bnk up cycle is a five-minute round-trip instead of an hour.

Three scenarios this shape unlocks:

  1. Many BNK trial iterations on one cluster. Run cluster up once, then loop bnk up / bnk down against the same cluster until you’ve covered all the trial permutations. Then cluster down once when you’re finished. This is the headline win of the phase split — see Chapter 10 §“Worked example — iterating on a BNK trial”.

  2. Pre-provisioning for a workshop or demo. You want the cluster ready and warm before the demo starts; you’ll deploy the BNK trial live in front of the audience. cluster up the night before; bnk up during the demo.

  3. Decoupling cluster lifecycle from trial lifecycle. A long-lived cluster used by multiple team members, where one person owns the cluster phase and others own the BNK trials. Cluster-phase outputs live in cluster-outputs.json; trials read it. Each trial can bnk up / bnk down without affecting the cluster.

For workspaces that just want “create a cluster, deploy BNK on it, test, tear it all down”, the unscoped roksbnkctl up / roksbnkctl down are the right verbs — they’re shape-aware composites that drive the cluster + trial steps in the right order without you having to think about it.

Cross-references