BKE
This is the documentation for BKE 2.4.0. The latest release is BKE 2.6.0. Read this page for BKE 2.6.0.

Upgrading

What the version number tells you

BKE versions are classified by what the upgrade does to a running cluster, not by which piece of software moved. So you can decide whether an upgrade needs a maintenance window from the number alone, without reading a changelog.

Digit that moved What you may conclude
third (patch) component pods rolled. No drain. Your manifests are unaffected
second (minor) a drain and a maintenance window are possible. Your manifests are unaffected, except for the resources of a component whose major version moved, which the release notes name
first (major) schedule it. Your manifests may break. One Kubernetes minor at a time

In detail:

Tier What may move Effect
major the Kubernetes minor API removals are possible, so your manifests can break. Nodes drain
minor the Kubernetes patch, Calico, containerd, component minors and component majors Nodes may drain, the dataplane may restart. No API removals
patch component patches, k9s Workload pods roll. No control-plane change, no drain

A Kubernetes patch is a BKE minor, not a BKE patch, and the naming is the weaker argument here. A component patch is a helm upgrade where that component’s pods roll. A Kubernetes patch is a kubeadm upgrade where the API server and kubelet restart and each node drains in turn. If both were patches, then “patch” would sometimes mean a few pods and sometimes mean a maintenance window — and anyone running an auto-apply-patches policy would receive an unscheduled drain. Calling it a minor fails safe, because the worst case you must already assume for a minor includes a window.

The same holds for urgency: a Kubernetes CVE fix ships as a minor. How urgent a change is does not alter what it does to your cluster.

The most invasive change in a release decides its tier. A release carrying a Kubernetes patch and a Longhorn patch is a minor, and the Longhorn patch travels inside it. That is what makes bundling possible, so several changes can go out in one window instead of each claiming its own.

containerd is upgraded only as part of the node upgrade, which runs on a drained node, so a containerd change is always a minor.

A component’s major version is a BKE minor, not a BKE major. The BKE major is reserved for the Kubernetes minor. A component major can still change how that component treats your own resources, so the release notes of a minor that carries one name the component and list those changes.

You can skip patches. You cannot skip minors.

A BKE patch release can be skipped. A BKE minor release cannot. Upgrade to every minor in turn. Within a minor, go straight to its latest patch: there is no need to apply the patches before it.

From To  
2.0.1 2.0.4 Allowed. Patches may be skipped
2.0.1 2.1.3 Allowed. One minor forward, to any of its patches
2.0.1 2.2.0 Refused. It skips 2.1. Upgrade to the latest 2.1 first

A new major starts from the last minor of the major before it, and arrives at its first minor. To reach 3.0, a cluster must be on the last 2.x minor. Moving to 3.1 directly from 2.x is refused, because it skips 3.0.

check.sh --version <target> --mode upgrade fails on a move that skips a minor, and names the version to upgrade to first. It reads the cluster’s current BKE version from its own record, so run it on a control-plane node. A cluster with no record yet cannot be assessed, and check.sh warns that it was not, rather than passing. Run it days before the window rather than at the start of it.

You also cannot skip a Kubernetes minor, which is why each BKE major moves Kubernetes by exactly one minor. check.sh fails on that too, and names the two versions. kubeadm refuses as well, but it refuses once the upgrade is already under way.

From 2.3.x to 2.4.0

Upgrade from any 2.3 patch; the latest is not required.

If check.sh fails a Longhorn disk below the floor, free space first. Run crictl rmi --prune on that node to remove the images no container uses, or give Longhorn a disk of its own. The upgrade’s lower image thresholds then keep the disk free. From 2.4.0, check.sh --version 2.4.0 reports a Longhorn disk with less free space than Longhorn’s minimal available percentage as a failure, because Longhorn places no new replica there and a replica lost in a drain is not rebuilt on that node.

The kubelet removes unused images sooner. A cluster whose kubelet configuration sets no image garbage collection threshold moves from the kubelet’s 85% and 80% to 70% and 60% at its upgrade. Set kubelet.imageGC in config.yaml to keep other values; a threshold your cluster already sets is left as it is. The plan before --commit names each threshold that changes. Because images are removed sooner, a node pulls more images again after a drain, so confirm that every node can reach the registry before the window.

Daily backups do not start with the upgrade. A backup target says where Longhorn writes backups; it schedules none. To have BKE create a daily backup job, set components.longhorn.settings.backup.daily.enabled: true. Storage describes the job and what check.sh reports about it.

The Release notes list everything else that changes.

Plan before the window

curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- \
  --role first --version 2.4.0

Nothing is upgraded without --commit. Every BKE script works this way: the default reports, and --commit acts.

Run this days before the maintenance window, not at the start of it. It runs the same preflight the real upgrade runs and then stops. A preflight failure found in preparation costs nothing; the same failure found mid-window costs a stalled cluster.

Like apply.sh, this changes nothing on the node. It downloads the target kubeadm into a temporary directory and runs the upgrade plan with it, fetches the container runtime package the upgrade would install, and compares the containerd configuration. It then lists what --commit will change on that node: the apt repository, kubeadm, the container runtime, and, where the node’s /etc/containerd/config.toml differs from the target version’s, that file, with the number of lines that differ. It is safe to run on a node that is carrying traffic.

With --commit, the containerd configuration and the runtime package change in one step, so containerd restarts once; running containers keep running while it does.

Run check.sh --version <target> --mode upgrade on every node at the same time. --mode upgrade makes the cluster checks required rather than skipped, which is what a release test should do.

Read the report before the window, too

curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.4.0

Without --commit this changes nothing, and it answers the questions the window is being booked for:

  • which versions move, including the ones that do not — “netdata is not moving” is worth knowing;
  • whether any node drains, and how many;
  • whether it can be rolled back — charts can, a Kubernetes upgrade cannot;
  • what changes in each of your values files, and where an edit of yours and one of ours have landed on the same setting;
  • which Kubernetes objects actually move as a result;
  • which values your file holds back — a setting your file keeps at the old chart’s default, and a setting BKE makes that your file leaves out;
  • whether the new software accepts its configuration, checked on the node before anything is applied.

Three findings stop an upgrade, and all three are far better found now than in the window: a values conflict, values the new chart will not accept, and a pre-flight check the new software fails (both below). See Installing components for what the four values outcomes mean.

What the objects section is for

A values diff tells you what the file says. It does not tell you what the cluster does about it, and the two come apart in both directions:

  kong    4 changed, 1 added, 1 removed, of 13 objects
  kuma    values changed, NO objects change (of 26)

The second line is the more important of the two. Helm never warns about a values key no template consumes — it is not an error, it is just nothing — so a careful-looking edit can reach nothing at all. The first line is the other half: a one-word change that rewrites six objects.

Both are produced by rendering your values against the pinned chart on each side and comparing the results. Object names and counts only, never bodies, so this section stays safe to paste into a ticket: a rendered manifest contains the values Helm is about to put in a Secret, and some charts generate their own certificates.

A component that could not be rendered says so, by name and with the reason. That line is not a failure of the upgrade, and it does not stop the run, but it does mean you have no answer for that component rather than a reassuring one.

Values your file holds back

Two things can keep a setting where it is without any line of your file changing, so neither values diff can show them. The report lists both, for every field alike, and neither stops the run.

A value at the old chart’s default. If your file sets a value to exactly what the old chart used by default, and the new chart’s default is different, your file keeps the old value and the setting does not move. A file written from a running installation usually carries every image tag this way:

  longhorn
    image.longhorn.manager.tag
      your file v1.0.0 (the old default), new default v1.1.0

The chart is upgraded and the software is not. To take the new default, remove the line from your file. To keep the old value, leave it.

A BKE setting your file leaves out. BKE’s values set some settings for every cluster. If your file does not contain one, the chart’s own default applies instead, and the report says what that is. If you removed it on purpose, nothing needs doing; to restore it, add the line back with the value shown.

Values the new chart will not accept

Most charts publish a schema for their values, and Helm checks your file against it before it changes anything. A new chart version can remove a key that the previous one accepted. If your file still sets that key, the report marks the component REFUSED, lists each key under Values the new chart refuses, and stops: apply.sh applies components one at a time, so --commit would upgrade the components before that one and then fail, leaving the cluster part-upgraded.

  cert-manager, against chart v1.21.2
    - at '/prometheus/servicemonitor': additional properties 'path',
      'targetPort' not allowed

Each path is a key in /etc/bke/values/<component>.yaml. Additional properties not allowed means the chart no longer has that key. Check whether the setting still does anything on your cluster, remove or correct each key named, and run the report again. A file that was taken over from a cluster BKE did not install often carries a full copy of an earlier chart’s defaults, and those copies are the usual source.

The section also states what it cannot see, and those limits are real: CRDs the chart installs once and never updates, objects a chart renders only when some other API is already present, and anything decided at apply time rather than at render time.

Components whose files change nothing are summarised in one line each rather than printed in full — six lines saying unchanged are what make the seventh readable. Add --show-unchanged if you want all of them. It changes the report only; the same upgrade happens either way, which is why it is safe to put in a runbook.

Pre-flight checks: what the new software will not accept

A new version of a component can refuse a configuration that the old version accepted. The values are then the same before and after, so no diff shows anything, and a chart’s schema has no view of a free-text configuration file. The only reliable answer comes from the new software itself. So before the report is printed, apply.sh checks every component that the run changes, on the node it runs on:

  • Every image exists. Only the image’s manifest is fetched, a few kilobytes, so a missing tag is found without downloading anything large. An image that is pulled from outside the BKE mirror is listed as a warning.
  • The API server accepts every object, in a server-side dry run with strict field validation. Nothing is stored.
  • The software accepts its own configuration, where it can say so. Each check runs the component’s new image against the configuration it would be given, in a container that has no network and only the files the check provides. A ConfigMap or Secret of your own that the pod mounts, such as a Kong plugin listed in plugins.configMaps, is read from your cluster on the node and given to the check as the pod receives it; it does not leave the node:
Component The check
Fluent Bit its own configuration test, --dry-run
Kong kong check, kong prepare (which runs nginx -t), then a start and stop
Kong ingress controller, cert-manager, metrics-server, Kuma start-up against an API server that does not answer: a refused configuration exits before it ever tries to connect
Netdata start-up, then its effective configuration is read back; any option it ignores is a warning, because Netdata never refuses a configuration
Longhorn none beyond the two checks above; Longhorn’s own pre-flight covers what can be known in advance

A check that fails prints the software’s own error lines under it, and the report stops:

  fluent-bit
    FAILED   Fluent Bit configuration test (--dry-run) (exit 1)
               [error] [parser] parser named 'apache' already exists, skip.
               [error] [config] could not configure service property
                 'parsers_file'
               [error] configuration file contains errors, aborting.

Correct /etc/bke/values/<component>.yaml and run the report again. In this example the file defines a parser that the new Fluent Bit image already ships under the same name. The older version logged the duplicate and continued; the newer one stops. Removing the duplicate definitions from config.customParsers is the correction.

A check reproduces the pod outside the cluster, and a reproduction can be wrong where the software is not. If you are certain that a failed check is wrong, accept it by name, and the run proceeds with the failure still printed:

BKE_ACCEPT_VALUES_RISK="preflight" \
  sh -c 'curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.4.0 --commit'

A check that cannot be run never stops the run. No ctr on the node, a registry that asks for credentials, an image that will not download: each is reported as not run, with the reason, and the upgrade proceeds exactly as it would have without the checks.

What the checks cost. Each image under test is downloaded once into a separate containerd namespace, bke-preflight, and removed when the checks finish, together with the namespace. Where every component changes, as in a minor upgrade, that is several hundred megabytes on the node where apply.sh runs, and a few minutes. A component that does not change is not checked: the software running it has already accepted its configuration.

What the checks cannot see. A configuration test checks that a configuration is valid; it does not connect to a log destination or a database, so a wrong host name still passes. An option the software silently ignores passes too, except in Netdata, which reports it. The dry run does not check NodePort ranges, and it skips a namespace that the apply has yet to create.

What leaves the node. The checks are prepared by the BKE API from the same values the report is built from, and their results are sent back to it so that the report can include them: each check’s exit status and the last part of its output. Nothing is stored.

The sequence

  1. apply.sh --commit at the current version, to confirm the cluster is reconciled and nothing has drifted.
  2. upgrade.sh --role first --commit on one control-plane node. This moves the control plane. If that node runs workloads (it has no control-plane taint), drain it first and uncordon it afterwards, as in step 3.
  3. On every remaining node — control-plane and worker alike — one at a time: drain it, run upgrade.sh --role other --commit on it, uncordon it, and wait until it is Ready and every Longhorn volume is healthy before the next node.
  4. apply.sh --version <new>, then --commit, to bring the components to the new graph.

upgrade.sh does not drain the node it runs on. The drain and the uncordon are yours, so that you decide when a node’s workloads move.

# step 2, on one control-plane node
curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --role first --version 2.4.0 --commit

# step 3, for each remaining node: from a control-plane node
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=15m
# then on that node
curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --role other --version 2.4.0 --commit
# then from a control-plane node
kubectl uncordon <node>
kubectl get volumes.longhorn.io -n longhorn-system \
  -o custom-columns=NAME:.metadata.name,ROBUSTNESS:.status.robustness

# step 4, once, from a control-plane node - read it, then commit it
curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.4.0
curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.4.0 --commit

Step 1 also sets where the control plane is pulled from. kubeadm upgrade pulls the control-plane images, kube-proxy and CoreDNS from the registry recorded in the kubeadm-config ConfigMap in kube-system. A cluster created before BKE pinned that registry, or created by another installer, records the upstream registry there. apply.sh shows the one-line change that corrects it, and --commit stores it. Storing it restarts nothing; the registry takes effect at step 2. If your cluster runs a different Kubernetes patch from the one this BKE version pins, apply.sh says so and makes the change at the first apply after an upgrade instead.

--role first and --role other describe order, not importance. The first node is whichever control-plane node you choose to go first; every other node, including other control-plane nodes, is other.

During

You drain the nodes in turn. Workloads without a PodDisruptionBudget and without enough replicas will be unavailable while their node is drained — that is ordinary Kubernetes behaviour, and the upgrade is when you find out whether your budgets are right.

What to expect in an upgrade that drains nodes:

  • In a release that moves Calico, Calico restarts on every node while upgrade.sh runs on the first control-plane node, before any other node is drained. Pod traffic pauses briefly on each node, and writes to a Longhorn volume can pause for around 20 seconds and then resume without error.
  • A Longhorn volume that moves with a drained node pauses for as long as it takes to detach and attach again on another node, typically under a minute.
  • Kong answers only on nodes running a Kong pod, because its proxy Service keeps the client’s address (externalTrafficPolicy: Local). A drained node stops answering on ports 32080 and 32443, and Kubernetes does not move pods back after an uncordon. A load balancer that health-checks the NodePort sends traffic only to nodes that answer, and with Kong’s two replicas on different nodes, one node answers throughout.

upgrade.sh collects warnings as it goes and prints them at the end rather than burying them in the middle of the output. Read that block; it is where anything that succeeded-but-not-cleanly ends up.

--ignore-preflight-errors

upgrade.sh passes this through to kubeadm, and every check you ignore is logged by name. Use it when you know why a check is failing and have decided it is acceptable — not to make an unexplained failure go away.

The one case a customised CoreDNS meets: a Corefile carrying an import line fails the preflight with CoreDNSUnsupportedPlugins on an upgrade that moves CoreDNS to a new version, because kubeadm cannot migrate a Corefile that has one. An upgrade that leaves CoreDNS where it is does not inspect the Corefile, and passes without the flag. check.sh compares the CoreDNS running on your cluster with the version the upgrade installs, warns days ahead when it moves, and prints the exact flag to bring:

--ignore-preflight-errors=CoreDNSUnsupportedPlugins

Do not delete the import to satisfy the preflight. The configuration it pulls in is live, and its loss is silent — nothing fails when it disappears, DNS answers simply change.

--kernel-modules

A node installed by BKE has dm_crypt loaded and persisted for every boot — Longhorn’s encrypted volumes need it, and without it an encrypted volume fails at attach time, a long way from the cause. A node that came to BKE some other way may not, and an upgrade does not change how your node boots unless you ask it to.

--kernel-modules asks: the upgrade then loads the module immediately and writes /etc/modules-load.d/bke.conf so it survives a reboot, the same as an installed node. Run it once per node, on the same upgrade run you are already doing; it is idempotent, and it does nothing on a run without --commit. If you do not use encrypted StorageClasses, you do not need it.

Afterwards

curl -sL https://bke.maml.uk/version.sh | sh
kubectl get nodes

version.sh reports what a node is running and changes nothing. Every node should report the new BKE version, and kubectl get nodes should show the new Kubernetes version on all of them. A node left behind is a node that will drift further with the next upgrade.

Rolling back

There is no supported downgrade path. Kubernetes does not support downgrading a control plane, and a BKE version that moves backwards is not a route back to where you were.

BKE does not hard-block it, deliberately — a downgrade during recovery has to stay possible — but it does tell you loudly. check.sh fails on a component that would move backwards, and explains the specific danger: charts carry their CRDs, so a chart moving backwards replaces newer CRD schemas with older ones underneath running workloads, while those workloads keep the newer image, because image tags are pinned separately.

So the protection is on the other side of the upgrade, not behind it. Before you start:

  • take an etcd snapshot — etcdctl snapshot save on a control-plane node. On a cluster that encrypts Secrets, the snapshot can be restored only with the key in /etc/kubernetes/bke/encryption-config.yaml, so confirm you hold a copy of it, stored apart from the snapshot;
  • confirm your Longhorn backups are current, rather than assuming it.

Storage covers how to confirm the second one, and why a backup target that has never worked looks exactly like one that is idle.