Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

BNK 2.4

roksbnkctl drives BNK 2.3 and 2.4 from one build. Which one you get is decided by bnk.manifest_version and nothing else — there is deliberately no bnk.line field and no ROKSBNKCTL_BNK_LINE override, because a second way to say which release this is, is a second thing that can disagree with the manifest being installed.

bnk:
  manifest_version: 2.3.0-3.2598.3-0.0.170   # -> the 2.3 line
  # manifest_version: 2.4.0-EA               # -> the 2.4 line

The derived value is rendered as a bnk_line terraform variable ("2.3" or "2.4"), and the modules gate per-release resources with count on it. There is no forked copy of the tree: the 2.4 changes are additive resources, which count expresses cleanly and two copies of a file do not.

2.4.0-EA carries no build suffix, where 2.3 versions have four dot/dash segments (2.3.0-3.2598.3-0.0.170). Anything you write that parses a manifest version should key on the major.minor prefix rather than the segment count.

What actually changes

2.4 replaces the data-plane configuration model. It runs the CNEInstance with USE_GATEWAY_SETTINGS=true, and under that flag the CNE controller ignores the cloud-network-mapping ConfigMap and the entire F5SPK* custom-resource family that a 2.3 install creates, reading two new CRs instead.

2.32.4
cloud-network-mapping ConfigMapInfra.spec.networks[] + networkAttachments[]
F5SPKVlan × 2Infra.spec.networks[type=vlan]
F5SPKStaticRoute × NInfra.spec.staticRoutes[]
F5SPKSnatpoolGatewaySettings.spec.sourceNATPools[]
F5SPKEgress × 2GatewaySettings.spec.egressConfigs[] + Infra.spec.egressDefaults
F5BnkGatewayGatewaySettings (the Gateway’s parametersRef target)
VIP address arithmeticIPAM allocation, read from Gateway.status.addresses

roksbnkctl emits both CRs for you. The addressing comes from the per-zone CIDRs you already configure — nothing new to fill in for a standard install:

your existing configthe 2.4 IPAM pool it becomes
ext_vlan_cidrexternal-vlan-ipam — TMM VLAN self-IPs
int_vip_cidrvip-listener-ipam — ingress VIPs
int_snat_cidregress-snat-ipam — egress SNAT addresses

gateway_egress_mode means the same thing on both lines: snatpool becomes a Pool egress config backed by egress-snat-ipam, automap becomes an Automap config, and both produces both.

Things 2.4 stops needing

Three of the four items in the 2.4 guide’s own “Changes From BNK Release-2.3” section are subtractions, which is easy to miss when reading a list of additions. On the 2.4 line roksbnkctl no longer creates:

  • a separate CIS Helm release — CIS is integrated into FLO, and externalBigip.enabled on the CNEInstance is what turns it on;
  • the node-labeler Job — app=f5-tmm labelling is not required;
  • the macvlan-conf NetworkAttachmentDefinition — the product creates its own internal NAD as macvlan-internal on dummy0, owned by the F5Tmm CR, and that name is reserved;
  • eighteen of nineteen privileged SCC bindings. FLO grants its components what they need, so the install requires one binding rather than nineteen.

The guide also asks for one privileged SCC binding per application namespace (oc adm policy add-scc-to-user privileged -n <ns> -z default). roksbnkctl does not create those: it knows the FLO and utils namespaces and nothing about where you will run workloads. That one stays an operator step.

The Trusted Profile

2.4’s CNE controller authenticates to IBM Cloud with a Trusted Profile and uses it to write into your VPC — adding the Infra IPAM CIDRs as VPC address prefixes, creating a routing table, and advertising TMM virtual-IP routes to the transit gateway.

roksbnkctl creates the profile, links it to the cluster’s service account, and passes its id to the CNEInstance as advanced.cneController.env.CLOUD_TRUSTED_PROFILE. Two access policies are attached: Viewer + Editor on the cluster’s VPC, so the controller can manage the network objects it creates, and Viewer on the cluster itself, so it can read the topology it next-hops routes against.

Because those IPAM CIDRs become real VPC address prefixes, they must not collide with what the cluster VPC already allocates — see Transit gateway sharing.

Moving between lines

You cannot flip a deployed workspace from 2.3 to 2.4 in place. bnk up and gateway up both refuse it, and the refusal names the reasons, because none of them is visible in a terraform plan:

  • GTM object naming differs. 2.3 writes server_<tmm_selfip>; 2.4 writes server_<digitalassetid>_<cluster-name>_<ns>_<ip>. Those objects live on an external BIG-IP — outside the cluster and outside terraform state — so nothing here removes them, and leaving both formats in place causes device IP conflicts.
  • The network model is replaced, not migrated. The F5SPK* objects a 2.3 install created are simply no longer read, and they stay on the cluster.
  • The CWC licence secrets survive a teardown and must be cleared before a reinstall licenses cleanly. bnk down now removes them for you.

To run 2.4, use a new workspace and a new install. Changing CLUSTER_IDENTIFIER on a deployed workspace needs the same manual GTM cleanup, for the same reason.

mTLS and the Gateway API bundle

A base 2.4 install works with whatever Gateway API bundle OpenShift ships: its crd-installer logs a graceful skip and leaves the platform’s CRDs alone.

mTLS is the exception. It needs Gateway API 1.5.0 standard, and installing that means deleting the OpenShift ingress-operator’s ValidatingAdmissionPolicy and its binding — which the platform recreates quickly, so it is a race. Opt in:

bnk:
  gateway_api_mtls: true      # ROKSBNKCTL_GATEWAY_API_MTLS=true

Off by default, because deleting a platform admission policy on a cluster that does not need the newer bundle is a change nobody asked for. Ignored on 2.3, where that sweep always runs — there the crd-installer does force its own CRDs and is blocked without it.

Where the bundle comes from

With gateway_api_mtls on, bnk up installs the upstream kubernetes-sigs/gateway-api standard-install bundle itself — eight CRDs, a ValidatingAdmissionPolicy of its own, and no container images — at the release named by bnk.gateway_api_version (default 1.5.0, the same number the CNE controller is given as GATEWAY_API_VERSION, so the two cannot disagree). It is applied server-side with --force-conflicts, inside the window where the admission sweep is running, and before terraform starts.

The bytes are always checked against a sha256 that ships in roksbnkctl, not in your configuration. A release with no pin in your build is refused rather than fetched unverified.

It is fetched from, in order:

  1. The registry mirror, whenever the workspace has one. The bundle is a normal BOM artifact, so registry bom lists it, registry replicate copies it and registry verify proves it arrived. This is the path a disconnected cluster needs: in the CI flow roksbnkctl runs as a pod inside that cluster, so its egress is the cluster’s and github.com is not reachable.

    The bundle enters the BOM only when gateway_api_mtls is on, so a mirror replicated before you set it does not carry the bundle — re-run registry replicate.

  2. bnk.gateway_api_bundle_url, if set — an internal proxy of the upstream release, for an estate that blocks github.com but has no mirror yet. It moves only where the bytes come from; the pin still applies.

  3. The upstream release asset, derived from the version.

Conforming to F5’s reference CNEInstance

The 2.4 spec this tool renders is checked against F5’s own reference install — the live capture in /mnt/d/roksbnkctl-gap-2-3-to-2-4. Every default below is F5’s value, not an invented one.

Pod placement replaced the node labeller

2.4 removed the node-labeler job, and spec.placement is the mechanism that took over. For a while this tool had neither: the labeller was gone and placement was never emitted, so TMM landing one-per-node and spread across zones was left to the scheduler’s discretion rather than required.

bnk:
  tmm_anti_affinity: true                                  # one TMM per node
  tmm_anti_affinity_topology_key: kubernetes.io/hostname   # the ROKS node label
  tmm_zone_spread: true                                    # spread across zones
  tmm_zone_topology_key: topology.kubernetes.io/zone       # the ROKS zone label
  tmm_zone_max_skew: 1
  tmm_zone_when_unsatisfiable: DoNotSchedule
  tmm_pod_label: f5-tmm

The topology keys are IBM ROKS node labels, and they are settings rather than constants on purpose: hard-coding them is the assumption the node-labeller baked in, and a cluster that labels its topology differently should still be configurable.

Turn both toggles off and no placement key is emitted at all — an operator who does not want the constraint gets no empty object for the CR to interpret.

The rest of the conformance set

settingdefaultwhat it is
tmm_replicas3data-plane replica count
watch_namespaces["All"]namespaces the controller watches
tmm_rolling_updatetruepins maxSurge 0 / maxUnavailable 1
external_bigipfalseexternal BIG-IP controller
external_bigip_login_secretf5-bigip-ctlr-loginits credentials secret
cluster_identifierderivedpassed to that controller
gateway_api_version1.5.0GATEWAY_API_VERSION on the controller
demo_modefalse on 2.4advanced.demoMode.enabled

Three of these are worth a sentence each.

tmm_rolling_update is the same shape as the cwc Multi-Attach deadlock: an unconstrained rolling update on a workload holding a single-attach resource can wedge, with the old pod holding the volume the new one is waiting for.

gateway_api_version was previously set nowhere, so the controller ran on whatever the operator defaulted to — v1.4.1 on the verified cluster, against F5’s 1.5.0. The 2.4 EA guide requires the 1.5 bundle for mTLS. Note that roksbnkctl still does not install the bundle; see the mTLS caveat above.

demo_mode was being set true on every install, on both lines. On 2.4 it now defaults to false, matching the reference. On 2.3 it stays true — that is what has shipped and been exercised there, and changing a working line’s behaviour without a run to back it would be a guess. Set it explicitly to pin it on either line.

Every one of these is a config.yaml field, a ROKSBNKCTL_* override, and a row in Chapter 28 with its line and default.

Hugepages, and why deploymentSize stays Tiny

The defaults aim at a working cluster with no overrides at all. On 2.4 that means deploymentSize: Tiny, OCP 4.21, and no hugepages — and on ROKS that is not merely a sensible default, it is the only configuration that runs.

deploymentSize decides how much TMM asks for, hugepages included. Measured on live 4.21 clusters by changing only that field:

deploymentSizeTMM cpuTMM memoryhugepages-2Mi
Tiny——none
Small11536Mi4 GiB
Medium42Gi8 GiB

A stock ROKS worker reports zero, and there is no supported way to change that.

bnk.hugepages does not work on ROKS

This is worth stating plainly, because the setting exists and reads like the answer. It is not. Six approaches were measured and all six fail:

approachresult
bnk.hugepages (Node Tuning Operator, [bootloader])needs the Machine Config Operator; ROKS has 0 MachineConfigPools
a Tuned CR with [sysctl] vm.nr_hugepagesROKS deletes the CR — the create returns a real object, the next read is NotFound
a privileged pod writing /proc/sys/vm/nr_hugepagesallocates (HugePages_Total: 2048), but kubelet only reads hugepages at startup so the node still advertises 0
advanced.tmm.resources with hugepages-2Mi: 0TMM schedules, then refuses to start
advanced.tmm.env.TMM_MEMaccepted, requirement remains
ibmcloud ks worker-poolno kernel-argument or sysctl option exists

IBM manages the worker OS on ROKS, which is what removes the first two. TMM validates hugepages against the 2 MB page cgroup limit when it starts and exits if that limit is zero — with exit code 0, so it never looks like a crash: the pod reports 4/5 Running and the Deployment simply never becomes Available.

Setting bnk.hugepages on ROKS now fails with an explanation naming this, rather than surfacing as a terraform “provider produced inconsistent result” error that blames the kubectl provider (issue #203).

So how do you get more capacity?

With more TMM pods on bigger nodes, which is what F5’s own sizing guide does. Appendix C’s Large cluster reaches ~31 Gbit/s at the Medium profile by running nine TMM pods on nine larger nodes — capacity comes from pod count and node size, never from a larger deploymentSize.

cluster:
  workers_per_zone: 2
  worker_flavor: cx2.16x32
bnk:
  cneinstance_size: Tiny     # or leave unset — Tiny is the 2.4 default
  tmm_replicas: 3            # one TMM per worker node
ROKSBNKCTL_WORKERS_PER_ZONE=2
ROKSBNKCTL_WORKER_FLAVOR=cx2.16x32
ROKSBNKCTL_TMM_REPLICAS=3

All four cluster shapes in Appendix C are verified this way, at 1, 3, 3 and 9 replicas.

cneinstance_size stays configurable rather than hardcoded, deliberately: if IBM ships a worker profile that supports hugepages, only the platform’s answer changes and nothing about config.yaml or the ROKSBNKCTL_* surface has to.

What happens if you ask for a size that cannot run

bnk up fails immediately, with the numbers and the alternatives, rather than waiting fifteen minutes to report Available=False. The scheduler’s verdict is already in the CNEInstance condition; the wait recognises that no amount of waiting creates node capacity and stops there.

Tuning the data-plane TCP profile

F5’s approved reference cluster carries a hand-applied sys-default-tcp F5BigTcpSetting in the BNK namespace. Despite the “sys-default” name it was applied by engineering rather than created by the product, which is why it is surfaced here.

bnk:
  tcp_settings:
    idleTimeout: "300"
    congestionControl: high-speed
    abc: "true"
ROKSBNKCTL_BNK_TCP_idleTimeout=300
ROKSBNKCTL_BNK_TCP_congestionControl=high-speed

A map rather than 54 config fields, because the CR has 54 of them across bool, int and string. Enumerating each would be 54 rows nobody reads, and it would go stale the moment F5 adds one.

Values are text — a config file and an environment variable cannot carry anything else — and are coerced on the way out by YAML’s own scalar rules: 300 becomes an integer, true a boolean, and anything else is quoted as a string. Field names are case-sensitive and used verbatim, because the CR is case-sensitive and guessing a mapping would silently write a field it ignores.

Setting nothing writes no CR at all. The product manages its own default profile, and emitting an empty one would fight it.

Per-component settings

Field specs for everything below live in Chapter 28 — Configuration reference, which is generated from the struct and test-checked. The line column there says which release line each field applies to.

The 2.4 CNEInstance takes an environment list for each of its ~26 components. Anything not surfaced as a first-class field is reachable through the passthrough, in YAML or from the environment:

bnk:
  advanced:
    tmm:
      env:
        TMM_DEFAULT_MTU: "9000"
    cneController:
      env:
        GSLB_DATACENTER_NAME: "datacenter-eu-gb"
ROKSBNKCTL_BNK_ADV_TMM_ENV_TMM_DEFAULT_MTU=9000
ROKSBNKCTL_BNK_ADV_CNECONTROLLER_ENV_GSLB_DATACENTER_NAME=datacenter-eu-gb

An entry here replaces the default of the same name rather than adding a second copy of it — the CNEInstance spec is read by the lifecycle operator, not by kubelet, so “the last duplicate wins” is not a rule to rely on. That is what makes this the way to change a default rather than only to add to it.

USE_GATEWAY_SETTINGS is deliberately not the example here, though earlier drafts of this page used it. roksbnkctl sets it on 2.4 itself, because on 2.4 there is no working configuration without it: without the flag the controller never reads the Infra and GatewaySettings CRs that gateway up applies, and they sit at Accepted=Unknown forever. You can still override it through this passthrough — that is what the replace-not-append rule buys you — but you should not have to.

The component name is matched case-insensitively, and one that roksbnkctl does not recognize is passed through rather than refused — the component set belongs to the product, and this tool should not be the reason a newly shipped one is unreachable.

Checking a 2.4 install

bnk status reports the CNEInstance’s aggregate Available condition plus any component condition that is not True, which is what turns a timeout into a diagnosis:

cneinstance:            Available=False (ComponentsNotReady)
cneinstance_not_ready:  F5TmmAvailable=False

That distinction matters on 2.4. CNEControllerAvailable=True can sit next to Available=False with TMM at 0/3 and nothing passing traffic.

The gate itself is CNEControllerAvailable on both lines, and deliberately so: waiting on the aggregate at the gate deadlocks 2.4, because the aggregate cannot go True until the License CR exists and the License CR is created after the gate. The aggregate is checked separately, after licensing, which is where it can actually be satisfied.

If a 2.4 install looks broken in a way that has nothing to do with what you changed, check the License CR first. The controller refuses to run any resource controller until it is Active, logging License is not enabled.. skip Resource controllers.

gateway status reports the Infra and GatewaySettings CRs alongside the Gateway, and reads the IPAM-allocated VIP from the Gateway’s status — on 2.4 there is no address to compute, only one to read.

Support status

2.4 is verified against 2.4.0-EA: bnk up reaches all eighteen CNEInstance conditions with TMM at 5/5 and the licence Active, on a single-NIC OCP 4.20 cluster with connected licensing.

What that run covered, and what it did not, because the difference matters:

exercisedbnk up on an adopted cluster — manifest 2.4.0-EA, FAR repo.f5.com with the non-GA service account, three workers, single-NIC, connected licensing
exercisedthe gateway phase — gateway up applies Infra and GatewaySettings, and both reach Accepted, ResolvedRefs and Programmed = True, with the CNEInstance still 18/18 and the licence still Active afterwards
exercisedfive consecutive clean cluster-lifecycle CLI demo runs on a cluster reproducing F5’s approved reference (6 × cx3d.8x20, OCP 4.21), each a cold start through all six phases
not exercisedthe CI demo — it is docker-driven and needs its own five runs
not exercisedtraffic — every check above reads CR conditions, and Accepted / ResolvedRefs / Programmed are all computed without calling IBM Cloud. #313 was a 2.4 install where the CNE controller could not assume its Trusted Profile, so no VPC address prefixes and no routes were created: all four CRs reported Programmed=True and nothing passed traffic in either direction. A condition-based verification cannot see that class of defect
not exercisedmTLS, multi-NIC, disconnected licensing, EgressGateway CRs, and the contract boundary

Every conformance setting was confirmed on the cluster, not inferred from a plan: deploymentSize: Tiny, tmmReplicas: 3, wholeCluster: false paired with watchNamespaces: ["All"], pod placement present, demoMode off, GATEWAY_API_VERSION: 1.5.0, and the three TMM environment settings.

The teardown namespace repair fired in all five runs and in every teardown between them. That is worth stating plainly: the 2.4 finalizer deadlock is not intermittent, so without the repair a 2.4 cluster cannot be reinstalled into at all.

Running the gateway phase is what found the three defects that made it impossible before it was run: the gateway plan failed outright on 2.4, nothing set USE_GATEWAY_SETTINGS so no controller ever reconciled the CRs, and the cneinstance_advanced_env override meant to let you set it by hand was read by no terraform at all. None of the three was visible in the source — the third least of all, since the variable’s name appears everywhere it should.

2.4 EA is single-NIC throughout — one NetworkAttachmentDefinition, one external-vlan network in the Infra CR, and no second data-plane interface anywhere. An earlier expectation that 2.4 would bring multi-NIC is withdrawn, and the verified run was single-NIC, so it is evidence for single-NIC and nothing else.

support_matrix.yaml records the same thing in the same terms. Its rule is worth repeating, because it is what keeps that file useful: a cell with no coverage is a claim nobody has checked.