BNK 2.4
roksbnkctl drives BNK 2.3 and 2.4 from one build. Which one you get is
decided by bnk.manifest_version and nothing else — there is deliberately no
bnk.line field and no ROKSBNKCTL_BNK_LINE override, because a second way to
say which release this is, is a second thing that can disagree with the manifest
being installed.
bnk:
manifest_version: 2.3.0-3.2598.3-0.0.170 # -> the 2.3 line
# manifest_version: 2.4.0-EA # -> the 2.4 line
The derived value is rendered as a bnk_line terraform variable ("2.3" or
"2.4"), and the modules gate per-release resources with count on it. There is
no forked copy of the tree: the 2.4 changes are additive resources, which count
expresses cleanly and two copies of a file do not.
2.4.0-EAcarries no build suffix, where 2.3 versions have four dot/dash segments (2.3.0-3.2598.3-0.0.170). Anything you write that parses a manifest version should key on themajor.minorprefix rather than the segment count.
What actually changes
2.4 replaces the data-plane configuration model. It runs the CNEInstance with
USE_GATEWAY_SETTINGS=true, and under that flag the CNE controller ignores
the cloud-network-mapping ConfigMap and the entire F5SPK* custom-resource
family that a 2.3 install creates, reading two new CRs instead.
| 2.3 | 2.4 |
|---|---|
cloud-network-mapping ConfigMap | Infra.spec.networks[] + networkAttachments[] |
F5SPKVlan × 2 | Infra.spec.networks[type=vlan] |
F5SPKStaticRoute × N | Infra.spec.staticRoutes[] |
F5SPKSnatpool | GatewaySettings.spec.sourceNATPools[] |
F5SPKEgress × 2 | GatewaySettings.spec.egressConfigs[] + Infra.spec.egressDefaults |
F5BnkGateway | GatewaySettings (the Gateway’s parametersRef target) |
| VIP address arithmetic | IPAM allocation, read from Gateway.status.addresses |
roksbnkctl emits both CRs for you. The addressing comes from the per-zone CIDRs
you already configure — nothing new to fill in for a standard install:
| your existing config | the 2.4 IPAM pool it becomes |
|---|---|
ext_vlan_cidr | external-vlan-ipam — TMM VLAN self-IPs |
int_vip_cidr | vip-listener-ipam — ingress VIPs |
int_snat_cidr | egress-snat-ipam — egress SNAT addresses |
gateway_egress_mode means the same thing on both lines: snatpool becomes a
Pool egress config backed by egress-snat-ipam, automap becomes an Automap
config, and both produces both.
Things 2.4 stops needing
Three of the four items in the 2.4 guide’s own “Changes From BNK Release-2.3”
section are subtractions, which is easy to miss when reading a list of
additions. On the 2.4 line roksbnkctl no longer creates:
- a separate CIS Helm release — CIS is integrated into FLO, and
externalBigip.enabledon the CNEInstance is what turns it on; - the node-labeler Job —
app=f5-tmmlabelling is not required; - the
macvlan-confNetworkAttachmentDefinition — the product creates its own internal NAD asmacvlan-internalondummy0, owned by theF5TmmCR, and that name is reserved; - eighteen of nineteen privileged SCC bindings. FLO grants its components what they need, so the install requires one binding rather than nineteen.
The guide also asks for one privileged SCC binding per application namespace (
oc adm policy add-scc-to-user privileged -n <ns> -z default).roksbnkctldoes not create those: it knows the FLO and utils namespaces and nothing about where you will run workloads. That one stays an operator step.
The Trusted Profile
2.4’s CNE controller authenticates to IBM Cloud with a Trusted Profile and
uses it to write into your VPC — adding the Infra IPAM CIDRs as VPC address
prefixes, creating a routing table, and advertising TMM virtual-IP routes to the
transit gateway.
roksbnkctl creates the profile, links it to the cluster’s service account, and
passes its id to the CNEInstance as
advanced.cneController.env.CLOUD_TRUSTED_PROFILE. Two access policies are
attached: Viewer + Editor on the cluster’s VPC, so the controller can manage
the network objects it creates, and Viewer on the cluster itself, so it can
read the topology it next-hops routes against.
Because those IPAM CIDRs become real VPC address prefixes, they must not collide with what the cluster VPC already allocates — see Transit gateway sharing.
Moving between lines
You cannot flip a deployed workspace from 2.3 to 2.4 in place. bnk up and
gateway up both refuse it, and the refusal names the reasons, because none of
them is visible in a terraform plan:
- GTM object naming differs. 2.3 writes
server_<tmm_selfip>; 2.4 writesserver_<digitalassetid>_<cluster-name>_<ns>_<ip>. Those objects live on an external BIG-IP — outside the cluster and outside terraform state — so nothing here removes them, and leaving both formats in place causes device IP conflicts. - The network model is replaced, not migrated. The
F5SPK*objects a 2.3 install created are simply no longer read, and they stay on the cluster. - The CWC licence secrets survive a teardown and must be cleared before a
reinstall licenses cleanly.
bnk downnow removes them for you.
To run 2.4, use a new workspace and a new install. Changing
CLUSTER_IDENTIFIER on a deployed workspace needs the same manual GTM cleanup,
for the same reason.
mTLS and the Gateway API bundle
A base 2.4 install works with whatever Gateway API bundle OpenShift ships: its
crd-installer logs a graceful skip and leaves the platform’s CRDs alone.
mTLS is the exception. It needs Gateway API 1.5.0 standard, and installing
that means deleting the OpenShift ingress-operator’s
ValidatingAdmissionPolicy and its binding — which the platform recreates
quickly, so it is a race. Opt in:
bnk:
gateway_api_mtls: true # ROKSBNKCTL_GATEWAY_API_MTLS=true
Off by default, because deleting a platform admission policy on a cluster that
does not need the newer bundle is a change nobody asked for. Ignored on 2.3,
where that sweep always runs — there the crd-installer does force its own
CRDs and is blocked without it.
Where the bundle comes from
With gateway_api_mtls on, bnk up installs the upstream
kubernetes-sigs/gateway-api standard-install bundle itself — eight CRDs, a
ValidatingAdmissionPolicy of its own, and no container images — at the release
named by bnk.gateway_api_version (default 1.5.0, the same number the CNE
controller is given as GATEWAY_API_VERSION, so the two cannot disagree). It is
applied server-side with --force-conflicts, inside the window where the
admission sweep is running, and before terraform starts.
The bytes are always checked against a sha256 that ships in roksbnkctl, not in your configuration. A release with no pin in your build is refused rather than fetched unverified.
It is fetched from, in order:
-
The registry mirror, whenever the workspace has one. The bundle is a normal BOM artifact, so
registry bomlists it,registry replicatecopies it andregistry verifyproves it arrived. This is the path a disconnected cluster needs: in the CI flow roksbnkctl runs as a pod inside that cluster, so its egress is the cluster’s andgithub.comis not reachable.The bundle enters the BOM only when
gateway_api_mtlsis on, so a mirror replicated before you set it does not carry the bundle — re-runregistry replicate. -
bnk.gateway_api_bundle_url, if set — an internal proxy of the upstream release, for an estate that blocksgithub.combut has no mirror yet. It moves only where the bytes come from; the pin still applies. -
The upstream release asset, derived from the version.
Conforming to F5’s reference CNEInstance
The 2.4 spec this tool renders is checked against F5’s own reference install —
the live capture in /mnt/d/roksbnkctl-gap-2-3-to-2-4. Every default below is
F5’s value, not an invented one.
Pod placement replaced the node labeller
2.4 removed the node-labeler job, and spec.placement is the mechanism that
took over. For a while this tool had neither: the labeller was gone and
placement was never emitted, so TMM landing one-per-node and spread across zones
was left to the scheduler’s discretion rather than required.
bnk:
tmm_anti_affinity: true # one TMM per node
tmm_anti_affinity_topology_key: kubernetes.io/hostname # the ROKS node label
tmm_zone_spread: true # spread across zones
tmm_zone_topology_key: topology.kubernetes.io/zone # the ROKS zone label
tmm_zone_max_skew: 1
tmm_zone_when_unsatisfiable: DoNotSchedule
tmm_pod_label: f5-tmm
The topology keys are IBM ROKS node labels, and they are settings rather than constants on purpose: hard-coding them is the assumption the node-labeller baked in, and a cluster that labels its topology differently should still be configurable.
Turn both toggles off and no placement key is emitted at all — an operator who
does not want the constraint gets no empty object for the CR to interpret.
The rest of the conformance set
| setting | default | what it is |
|---|---|---|
tmm_replicas | 3 | data-plane replica count |
watch_namespaces | ["All"] | namespaces the controller watches |
tmm_rolling_update | true | pins maxSurge 0 / maxUnavailable 1 |
external_bigip | false | external BIG-IP controller |
external_bigip_login_secret | f5-bigip-ctlr-login | its credentials secret |
cluster_identifier | derived | passed to that controller |
gateway_api_version | 1.5.0 | GATEWAY_API_VERSION on the controller |
demo_mode | false on 2.4 | advanced.demoMode.enabled |
Three of these are worth a sentence each.
tmm_rolling_update is the same shape as the cwc Multi-Attach deadlock: an
unconstrained rolling update on a workload holding a single-attach resource can
wedge, with the old pod holding the volume the new one is waiting for.
gateway_api_version was previously set nowhere, so the controller ran on
whatever the operator defaulted to — v1.4.1 on the verified cluster, against
F5’s 1.5.0. The 2.4 EA guide requires the 1.5 bundle for mTLS. Note that
roksbnkctl still does not install the bundle; see the mTLS caveat above.
demo_mode was being set true on every install, on both lines. On 2.4 it
now defaults to false, matching the reference. On 2.3 it stays true — that is
what has shipped and been exercised there, and changing a working line’s
behaviour without a run to back it would be a guess. Set it explicitly to pin it
on either line.
Every one of these is a config.yaml field, a ROKSBNKCTL_* override, and a row
in Chapter 28 with its line and default.
Hugepages, and why deploymentSize stays Tiny
The defaults aim at a working cluster with no overrides at all. On 2.4 that
means deploymentSize: Tiny, OCP 4.21, and no hugepages — and on ROKS that
is not merely a sensible default, it is the only configuration that runs.
deploymentSize decides how much TMM asks for, hugepages included. Measured on
live 4.21 clusters by changing only that field:
deploymentSize | TMM cpu | TMM memory | hugepages-2Mi |
|---|---|---|---|
Tiny | — | — | none |
Small | 1 | 1536Mi | 4 GiB |
Medium | 4 | 2Gi | 8 GiB |
A stock ROKS worker reports zero, and there is no supported way to change that.
bnk.hugepages does not work on ROKS
This is worth stating plainly, because the setting exists and reads like the answer. It is not. Six approaches were measured and all six fail:
| approach | result |
|---|---|
bnk.hugepages (Node Tuning Operator, [bootloader]) | needs the Machine Config Operator; ROKS has 0 MachineConfigPools |
a Tuned CR with [sysctl] vm.nr_hugepages | ROKS deletes the CR — the create returns a real object, the next read is NotFound |
a privileged pod writing /proc/sys/vm/nr_hugepages | allocates (HugePages_Total: 2048), but kubelet only reads hugepages at startup so the node still advertises 0 |
advanced.tmm.resources with hugepages-2Mi: 0 | TMM schedules, then refuses to start |
advanced.tmm.env.TMM_MEM | accepted, requirement remains |
ibmcloud ks worker-pool | no kernel-argument or sysctl option exists |
IBM manages the worker OS on ROKS, which is what removes the first two. TMM
validates hugepages against the 2 MB page cgroup limit when it starts and exits
if that limit is zero — with exit code 0, so it never looks like a crash:
the pod reports 4/5 Running and the Deployment simply never becomes Available.
Setting bnk.hugepages on ROKS now fails with an explanation naming this,
rather than surfacing as a terraform “provider produced inconsistent result”
error that blames the kubectl provider (issue #203).
So how do you get more capacity?
With more TMM pods on bigger nodes, which is what F5’s own sizing guide
does. Appendix C’s Large cluster reaches ~31 Gbit/s at the Medium profile by
running nine TMM pods on nine larger nodes — capacity comes from pod count and
node size, never from a larger deploymentSize.
cluster:
workers_per_zone: 2
worker_flavor: cx2.16x32
bnk:
cneinstance_size: Tiny # or leave unset — Tiny is the 2.4 default
tmm_replicas: 3 # one TMM per worker node
ROKSBNKCTL_WORKERS_PER_ZONE=2
ROKSBNKCTL_WORKER_FLAVOR=cx2.16x32
ROKSBNKCTL_TMM_REPLICAS=3
All four cluster shapes in Appendix C are verified this way, at 1, 3, 3 and 9 replicas.
cneinstance_size stays configurable rather than hardcoded, deliberately: if
IBM ships a worker profile that supports hugepages, only the platform’s answer
changes and nothing about config.yaml or the ROKSBNKCTL_* surface has to.
What happens if you ask for a size that cannot run
bnk up fails immediately, with the numbers and the alternatives, rather
than waiting fifteen minutes to report Available=False. The scheduler’s
verdict is already in the CNEInstance condition; the wait recognises that no
amount of waiting creates node capacity and stops there.
Tuning the data-plane TCP profile
F5’s approved reference cluster carries a hand-applied sys-default-tcp
F5BigTcpSetting in the BNK namespace. Despite the “sys-default” name it was
applied by engineering rather than created by the product, which is why it is
surfaced here.
bnk:
tcp_settings:
idleTimeout: "300"
congestionControl: high-speed
abc: "true"
ROKSBNKCTL_BNK_TCP_idleTimeout=300
ROKSBNKCTL_BNK_TCP_congestionControl=high-speed
A map rather than 54 config fields, because the CR has 54 of them across bool, int and string. Enumerating each would be 54 rows nobody reads, and it would go stale the moment F5 adds one.
Values are text — a config file and an environment variable cannot carry
anything else — and are coerced on the way out by YAML’s own scalar rules:
300 becomes an integer, true a boolean, and anything else is quoted as a
string. Field names are case-sensitive and used verbatim, because the CR is
case-sensitive and guessing a mapping would silently write a field it ignores.
Setting nothing writes no CR at all. The product manages its own default profile, and emitting an empty one would fight it.
Per-component settings
Field specs for everything below live in Chapter 28 — Configuration reference, which is generated from the struct and test-checked. The
linecolumn there says which release line each field applies to.
The 2.4 CNEInstance takes an environment list for each of its ~26 components. Anything not surfaced as a first-class field is reachable through the passthrough, in YAML or from the environment:
bnk:
advanced:
tmm:
env:
TMM_DEFAULT_MTU: "9000"
cneController:
env:
GSLB_DATACENTER_NAME: "datacenter-eu-gb"
ROKSBNKCTL_BNK_ADV_TMM_ENV_TMM_DEFAULT_MTU=9000
ROKSBNKCTL_BNK_ADV_CNECONTROLLER_ENV_GSLB_DATACENTER_NAME=datacenter-eu-gb
An entry here replaces the default of the same name rather than adding a second copy of it — the CNEInstance spec is read by the lifecycle operator, not by kubelet, so “the last duplicate wins” is not a rule to rely on. That is what makes this the way to change a default rather than only to add to it.
USE_GATEWAY_SETTINGS is deliberately not the example here, though earlier
drafts of this page used it. roksbnkctl sets it on 2.4 itself, because on 2.4
there is no working configuration without it: without the flag the controller
never reads the Infra and GatewaySettings CRs that gateway up applies, and
they sit at Accepted=Unknown forever. You can still override it through this
passthrough — that is what the replace-not-append rule buys you — but you should
not have to.
The component name is matched case-insensitively, and one that roksbnkctl does
not recognize is passed through rather than refused — the component set belongs
to the product, and this tool should not be the reason a newly shipped one is
unreachable.
Checking a 2.4 install
bnk status reports the CNEInstance’s aggregate Available condition plus any
component condition that is not True, which is what turns a timeout into a
diagnosis:
cneinstance: Available=False (ComponentsNotReady)
cneinstance_not_ready: F5TmmAvailable=False
That distinction matters on 2.4. CNEControllerAvailable=True can sit next to
Available=False with TMM at 0/3 and nothing passing traffic.
The gate itself is CNEControllerAvailable on both lines, and deliberately
so: waiting on the aggregate at the gate deadlocks 2.4, because the aggregate
cannot go True until the License CR exists and the License CR is created after
the gate. The aggregate is checked separately, after licensing, which is where
it can actually be satisfied.
If a 2.4 install looks broken in a way that has nothing to do with what you changed, check the
LicenseCR first. The controller refuses to run any resource controller until it is Active, loggingLicense is not enabled.. skip Resource controllers.
gateway status reports the Infra and GatewaySettings CRs alongside the
Gateway, and reads the IPAM-allocated VIP from the Gateway’s status — on 2.4
there is no address to compute, only one to read.
Support status
2.4 is verified against 2.4.0-EA: bnk up reaches all eighteen CNEInstance
conditions with TMM at 5/5 and the licence Active, on a single-NIC OCP 4.20
cluster with connected licensing.
What that run covered, and what it did not, because the difference matters:
| exercised | bnk up on an adopted cluster — manifest 2.4.0-EA, FAR repo.f5.com with the non-GA service account, three workers, single-NIC, connected licensing |
| exercised | the gateway phase — gateway up applies Infra and GatewaySettings, and both reach Accepted, ResolvedRefs and Programmed = True, with the CNEInstance still 18/18 and the licence still Active afterwards |
| exercised | five consecutive clean cluster-lifecycle CLI demo runs on a cluster reproducing F5’s approved reference (6 × cx3d.8x20, OCP 4.21), each a cold start through all six phases |
| not exercised | the CI demo — it is docker-driven and needs its own five runs |
| not exercised | traffic — every check above reads CR conditions, and Accepted / ResolvedRefs / Programmed are all computed without calling IBM Cloud. #313 was a 2.4 install where the CNE controller could not assume its Trusted Profile, so no VPC address prefixes and no routes were created: all four CRs reported Programmed=True and nothing passed traffic in either direction. A condition-based verification cannot see that class of defect |
| not exercised | mTLS, multi-NIC, disconnected licensing, EgressGateway CRs, and the contract boundary |
Every conformance setting was confirmed on the cluster, not inferred from a
plan: deploymentSize: Tiny, tmmReplicas: 3, wholeCluster: false paired with
watchNamespaces: ["All"], pod placement present, demoMode off,
GATEWAY_API_VERSION: 1.5.0, and the three TMM environment settings.
The teardown namespace repair fired in all five runs and in every teardown between them. That is worth stating plainly: the 2.4 finalizer deadlock is not intermittent, so without the repair a 2.4 cluster cannot be reinstalled into at all.
Running the gateway phase is what found the three defects that made it
impossible before it was run: the gateway plan failed outright on 2.4, nothing
set USE_GATEWAY_SETTINGS so no controller ever reconciled the CRs, and the
cneinstance_advanced_env override meant to let you set it by hand was read by
no terraform at all. None of the three was visible in the source — the third
least of all, since the variable’s name appears everywhere it should.
2.4 EA is single-NIC throughout — one NetworkAttachmentDefinition, one
external-vlan network in the Infra CR, and no second data-plane interface
anywhere. An earlier expectation that 2.4 would bring multi-NIC is withdrawn, and
the verified run was single-NIC, so it is evidence for single-NIC and nothing
else.
support_matrix.yaml records the same thing in the same terms. Its rule is worth
repeating, because it is what keeps that file useful: a cell with no coverage is
a claim nobody has checked.