Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

COS supply chain management

BIG-IP Next for Kubernetes (BNK) pulls its runtime artefacts — the F5 Application Runtime (FAR) container images, the JWT licence used at install + renewal time, the f5-bigip-k8s-manifest Helm chart, and the schematic JSON the deployer renders — from IBM Cloud Object Storage (COS). The COS bucket is the supply chain: it’s how artefacts produced upstream (F5 build pipeline, licence-issuing service, schematic generator) reach the cluster.

roksbnkctl cos is the management surface for that supply chain. Three command levels — cos instance, cos bucket, cos object — cover the full CRUD on COS resources without touching the ibmcloud CLI; everything (most visibly cos object put for uploads and cos object get for downloads) goes through the IBM Cloud Go SDKs (go-sdk-core, platform-services-go-sdk, ibm-cos-sdk-go).

What COS is in this stack

COS is IBM’s S3-compatible object store. Two layers matter here:

  • Instance: a service instance under Resource Controller. Instances are global; they don’t pin to a region. A workspace typically uses one COS instance per environment (dev, staging, prod) and shares it across multiple buckets.
  • Bucket: an S3-style bucket with regional affinity, hosted on a COS instance. Buckets carry the storage class (standard, vault, cold, smart) and the access policy (HMAC keys, service-instance creds, public ACLs — the latter are off-limits for BNK supply-chain use).

The BNK supply chain reads from one COS bucket per cluster’s BNK install. The bucket holds:

ObjectWhat it isConsumed by
f5-far-auth-key.tgzFAR repository pull credentials — the F5-internal artefact key that lets FLO download FAR container imagesflo module at install time
subscription.jwt (or production equivalent)BNK subscription JWT — the licence the CNE Instance presentsflo and license modules
schematic-<v>.jsonThe deployer’s schematic JSON for the deployed BNK versioninformational, not directly mounted into the cluster
(optional) FAR image tarballsPre-pulled FAR images for air-gapped installsflo when running in disconnected mode

The bucket structure is defined by the upstream HCL — concretely by the ibmcloud_resources_cos_bucket variable. When you leave cos.bucket unset, init provisions (or discovers and reuses) an account-scoped bucket named bnk-artifacts-<first-12-of-account-id>: COS bucket names share one global namespace (like S3), so the account suffix keeps the name unique and lets a second workspace run from the same account find the bucket the first one created — no duplicate, no re-upload. The instance defaults to bnk-supply-chain. (The bare bnk-artifacts used in the manual cos examples below is just an illustrative name; supply your own.)

Local files instead of COS (no bucket needed)

COS is not mandatory for the two artefacts the BNK phase actually needs at apply time — the FAR auth tarball and the subscription JWT. You can point the workspace at local files and skip COS entirely:

bnk:
  far_auth_local_file: /path/to/f5-far-auth-key.tgz
  subscription_jwt_local_file: /path/to/subscription.jwt

When both are set, roksbnkctl reads them at render time — extracting the FAR _json_key_base64 service account from the tarball in Go — and injects the content directly into the flo and license modules (use_cos_bucket = false), so no COS instance, bucket, or S3 endpoint is contacted. This is useful when the account has no orchestration COS, or when the COS S3 endpoint isn’t reachable from where you run roksbnkctl.

roksbnkctl init wires this up automatically: it checks COS first, and on any COS error (including the dial tcp … no such host you get when a cluster region such as eu-fr2 has no COS S3 endpoint) it falls back to prompting for these local files and records them on the workspace. You can also set them by hand. When they’re unset, the COS path below is used unchanged.

The three command levels

roksbnkctl cos instance {create|delete|list}
roksbnkctl cos bucket   {create|delete|list|get} --instance <name-or-CRN>
roksbnkctl cos object   {put|get|delete|list} --instance <name-or-CRN>

All three layers resolve credentials through the standard credential resolver chain — env var, OS keychain, workspace api_key_b64, prompt. There’s no separate “COS credential”; the IBM API key authenticates against Resource Controller (instance ops) and IAM-signed S3 requests (bucket and object ops).

cos instance

Manages COS service instances at the account level via Resource Controller.

# Create a Standard-plan instance under the workspace's resource group
roksbnkctl cos instance create bnk-supply-chain --plan standard

# Override the plan by catalog UUID when roksbnkctl hasn't mapped the tier
roksbnkctl cos instance create bnk-supply-chain --plan-id <uuid>

# List instances in the account
roksbnkctl cos instance list

# Delete an instance (default: recursive — removes bound HMAC keys, service creds)
roksbnkctl cos instance delete bnk-supply-chain
roksbnkctl cos instance delete bnk-supply-chain --no-recursive --auto
FlagDefaultNotes
--planstandardFriendly name (standard, lite); maps to a Resource Controller plan UUID internally.
--plan-id—Catalog UUID — bypasses the friendly-name mapping. Use when IBM ships a plan tier roksbnkctl hasn’t seen yet.
--targetglobalCOS instances are global; this is left as a flag for forward compatibility.
--no-recursive(off)On delete, do NOT remove bound HMAC keys and service credentials. Hardly ever what you want.
--auto(off)On delete, skip the y/N confirmation.

The resource group is read from the workspace’s ibmcloud.resource_group field (defaulting to default when unset).

cos bucket

Manages buckets within a named instance. The --instance flag is required for every bucket and object call — buckets aren’t globally unique, only unique within an instance.

# Create a standard-class bucket
roksbnkctl cos bucket create bnk-artifacts \
  --instance bnk-supply-chain \
  --class standard

# List buckets on the instance
roksbnkctl cos bucket list --instance bnk-supply-chain

# Delete (the bucket must be empty first; cos object delete --recursive isn't implemented yet)
roksbnkctl cos bucket delete bnk-artifacts --instance bnk-supply-chain

# Snapshot the entire bucket to a local directory (recursive, streaming, byte-for-byte).
# Useful before rotating supply-chain artefacts — keeps a known-good set you can roll back to.
roksbnkctl cos bucket get bnk-artifacts ./snapshot \
  --instance bnk-supply-chain
FlagDefaultNotes
--instance(required)Instance name or CRN — the CRN starts with crn:v1: and is used as-is; a bare name is looked up via Resource Controller.
--regionworkspace regionThe IBM Cloud region the bucket is pinned to. Override only when you’re crossing regions deliberately.
--classstandardStorage class: standard (frequently accessed), vault (infrequent), cold (archive), smart (auto-tiered). The BNK supply chain uses standard because FLO reads at install + every restart.
--no-clobber(off)On cos bucket get: skip any object whose local path already exists. Default overwrites. Files-only flag; ignored by other cos bucket verbs.

cos object

Manages objects (files) within a bucket. The key syntax is <bucket>/<key/with/slashes> — the parser splits on the first slash, so bucket/dir/file.tgz parses as bucket bucket, key dir/file.tgz.

# Upload (streaming; multipart auto-engages for large files)
roksbnkctl cos object put bnk-artifacts/f5-far-auth-key.tgz \
  ./local/f5-far-auth-key.tgz \
  --instance bnk-supply-chain

# Download (streaming)
roksbnkctl cos object get bnk-artifacts/f5-far-auth-key.tgz \
  ./downloaded.tgz \
  --instance bnk-supply-chain

# Delete
roksbnkctl cos object delete bnk-artifacts/old-subscription.jwt \
  --instance bnk-supply-chain

# List (with an optional key prefix)
roksbnkctl cos object list bnk-artifacts \
  --instance bnk-supply-chain

roksbnkctl cos object list bnk-artifacts/schematics/ \
  --instance bnk-supply-chain

The list output is a tab-separated KEY SIZE MODIFIED table — pipe through column -t for readability or cut -f1 to extract just the keys.

The BNK supply chain shape

A typical bnk-artifacts bucket after a clean install looks like:

$ roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain
KEY                                     SIZE        MODIFIED
f5-far-auth-key.tgz                     2412        2026-05-08T14:12:33Z
subscription.jwt                               1857        2026-05-08T14:12:34Z
schematic-2.3.0-3.2598.3-0.0.170.json   18432       2026-05-08T14:13:01Z

Three pieces of metadata in the upstream HCL (terraform/variables.tf) pin the bucket layout:

HCL variableDefaultObject
f5_cne_far_auth_filef5-far-auth-key.tgzFAR pull credentials
f5_cne_subscription_jwt_filesubscription.jwtSubscription JWT
f5_bigip_k8s_manifest_version2.3.0-3.2598.3-0.0.170Schematic filename inferred from this

Changing any of these in terraform.tfvars (or the workspace bnk: block, which renders into tfvars) changes which COS keys FLO will look for. The HCL doesn’t auto-discover key names — they’re literal.

For air-gapped installs where the cluster can’t reach repo.f5.com, additional pre-pulled FAR image tarballs go in the same bucket and the far_repo_url variable points at a COS-backed proxy. That topology is out of scope for v1.0; the supply chain shape described here is the connected-mode happy path.

Multipart upload and streaming download

FAR image tarballs run 1-5 GB. cos object put streams the input file into the bucket in 5 MB parts using S3-style multipart uploads:

  • For files under 5 MB, a single-part PutObject is used (the SDK’s default).
  • For files over 5 MB, the SDK auto-engages multipart upload — the file is split into 5 MB parts, each uploaded in parallel (up to 4 concurrent parts), and finalised with a CompleteMultipartUpload call.

The split is transparent — there’s no --multipart flag to set. The SDK handles it under the hood. If you want to verify multipart is happening for a specific file, watch with roksbnkctl --verbose cos object put … and the SDK’s debug logging surfaces the part count.

cos object get is similarly streaming: the SDK pipes the body straight to the destination file without buffering it in memory. Multi-gigabyte downloads on a memory-constrained jumphost are safe.

If a multipart upload is interrupted (network drop, ^C), the partial-upload state lingers on COS until cleaned up. Today roksbnkctl doesn’t expose a “list and abort orphan multipart uploads” command — that’s a v1.x addition. The workaround is to use ibmcloud cos list-multipart-uploads directly via Chapter 17’s docker backend or the IBM Cloud console.

Workspace config integration

The workspace cos: block is optional — if the bucket is already populated (manually, or by an external CI pipeline), the block can be omitted entirely. When set, it triggers an auto-upload at roksbnkctl up time so the FAR pull and licence land before FLO needs them.

# ~/.roksbnkctl/<workspace>/config.yaml
cos:
  instance: bnk-supply-chain
  bucket: bnk-artifacts
  upload:
    - source: ./local/f5-far-auth-key.tgz
      key: f5-far-auth-key.tgz
    - source: ./local/subscription.jwt
      key: subscription.jwt

The block maps directly to internal/config/workspace.go::COSCfg:

FieldTypePurpose
instancestringCOS instance name or CRN. Looked up via Resource Controller at runtime.
bucketstringBucket name within the instance.
uploadlist of {source, key}Optional pre-flight uploads. source is a host filesystem path (relative or absolute); key is the destination object key in the bucket.

Pre-flight uploads run before terraform apply, so FLO sees the artefacts when it pulls. Idempotent: re-running up re-uploads, which COS treats as overwrite — safe.

When the supply chain matters

Three lifecycle moments where the COS bucket is in play:

Install time

roksbnkctl up provisions FLO, which queries the bucket for f5-far-auth-key.tgz and subscription.jwt. Missing either object → FLO fails to start → terraform apply retries (per internal/cli/lifecycle.go::applyWithRetry) for ~3 attempts before erroring. The fix is always “put the missing object in the bucket and re-run up”; the lifecycle retry hides transient bucket-policy propagation lag but won’t paper over a genuinely-empty bucket.

Upgrade time

When bnk.manifest_version (or the f5_bigip_k8s_manifest_version HCL variable) bumps, FLO pulls a new FAR image tarball and re-renders the CNE Instance. If the new manifest version references a FAR image that isn’t already in repo.f5.com’s public registry (rare, but happens for pre-release builds), the bucket holds the air-gapped fallback. Standard upgrades — the connected-mode case — don’t touch the bucket; they just pull from repo.f5.com using the credentials in f5-far-auth-key.tgz.

Licence rotation

When the trial expires or a production licence arrives, swap subscription.jwt for the new file:

roksbnkctl cos object put bnk-artifacts/subscription.jwt \
  ./new-license.jwt \
  --instance bnk-supply-chain

# Force FLO to re-read the licence (delete the CNE Instance's License resource;
# FLO's reconciler re-creates it from the updated JWT)
roksbnkctl k delete license -n f5-bnk --all

FLO picks up the new JWT within 60-90 seconds. No roksbnkctl up re-run required for licence rotation alone.

Worked example: rotating COS supply-chain assets

End-to-end Part VII scenario: the FAR auth key on file is about to expire, a new one arrived from the F5 distribution side, and you need to rotate it without taking BNK down. The same flow handles licence-JWT rotation (swap subscription.jwt for the production JWT) and FAR-image-tarball uploads for air-gapped clusters. Cross-link to Chapter 14 for the API-key half of the rotation story; this walkthrough focuses on the COS object half.

# 1. Sanity-check the current state
roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain

# 2. Upload the new auth key (overwrites the existing file)
roksbnkctl cos object put bnk-artifacts/f5-far-auth-key.tgz \
  ./new-far-auth-key.tgz \
  --instance bnk-supply-chain

# 3. Verify the upload
roksbnkctl cos object list bnk-artifacts --instance bnk-supply-chain
# Expected: the f5-far-auth-key.tgz row's MODIFIED timestamp is now

# 4. (optional, air-gapped only) Upload the FAR image tarball
roksbnkctl cos object put bnk-artifacts/far-2.3.0-images.tgz \
  ./far-2.3.0-images.tgz \
  --instance bnk-supply-chain

# 5. Force FLO to re-read the supply chain
roksbnkctl k delete pod -n f5-bnk -l app=flo
# (FLO's controller restarts; the new pod re-pulls f5-far-auth-key.tgz on first reconcile)

# 6. Verify FLO is healthy with the new key
roksbnkctl logs flo
# Expected: no "failed to pull FAR image: unauthorized" lines

The third step is the verification gate. If FLO’s logs still show auth failures after the pod restart, the new auth key was rejected by repo.f5.com — re-issue the key on the F5 side, not in the bucket.

Cross-references