This is the documentation for BKE 2.0.0. The latest release is BKE 2.6.0. Read this page for BKE 2.6.0.

Upgrading

What the version number tells you

BKE versions are classified by what the upgrade does to a running cluster, not by which piece of software moved. So you can decide whether an upgrade needs a maintenance window from the number alone, without reading a changelog.

Digit that moved	What you may conclude
third (patch)	component pods rolled. No drain. Your manifests are unaffected
second (minor)	a drain and a maintenance window are possible. Your manifests are unaffected, except for the resources of a component whose major version moved, which the release notes name
first (major)	schedule it. Your manifests may break. One Kubernetes minor at a time

In detail:

Tier	What may move	Effect
major	the Kubernetes minor	API removals are possible, so your manifests can break. Nodes drain
minor	the Kubernetes patch, Calico, containerd, component minors and component majors	Nodes may drain, the dataplane may restart. No API removals
patch	component patches, k9s	Workload pods roll. No control-plane change, no drain

A Kubernetes patch is a BKE minor, not a BKE patch, and the naming is the weaker argument here. A component patch is a helm upgrade where that component’s pods roll. A Kubernetes patch is a kubeadm upgrade where the API server and kubelet restart and each node drains in turn. If both were patches, then “patch” would sometimes mean a few pods and sometimes mean a maintenance window — and anyone running an auto-apply-patches policy would receive an unscheduled drain. Calling it a minor fails safe, because the worst case you must already assume for a minor includes a window.

The same holds for urgency: a Kubernetes CVE fix ships as a minor. How urgent a change is does not alter what it does to your cluster.

The most invasive change in a release decides its tier. A release carrying a Kubernetes patch and a Longhorn patch is a minor, and the Longhorn patch travels inside it. That is what makes bundling possible, so several changes can go out in one window instead of each claiming its own.

containerd is upgraded only as part of the node upgrade, which runs on a drained node, so a containerd change is always a minor.

A component’s major version is a BKE minor, not a BKE major. The BKE major is reserved for the Kubernetes minor. A component major can still change how that component treats your own resources, so the release notes of a minor that carries one name the component and list those changes.

You can skip patches. You cannot skip minors.

A BKE patch release can be skipped. A BKE minor release cannot. Upgrade to every minor in turn. Within a minor, go straight to its latest patch: there is no need to apply the patches before it.

From	To
2.0.1	2.0.4	Allowed. Patches may be skipped
2.0.1	2.1.3	Allowed. One minor forward, to any of its patches
2.0.1	2.2.0	Refused. It skips 2.1. Upgrade to the latest 2.1 first

A new major starts from the last minor of the major before it, and arrives at its first minor. To reach 3.0, a cluster must be on the last 2.x minor. Moving to 3.1 directly from 2.x is refused, because it skips 3.0.

check.sh --version 2.0.0 --mode upgrade fails on a move that skips a minor, and names the version to upgrade to first. It reads the cluster’s current BKE version from its own record, so run it on a control-plane node. A cluster with no record yet cannot be assessed, and check.sh warns that it was not, rather than passing. Run it days before the window rather than at the start of it.

You also cannot skip a Kubernetes minor, which is why each BKE major moves Kubernetes by exactly one minor. check.sh fails on that too, and names the two versions. kubeadm refuses as well, but it refuses once the upgrade is already under way.

Plan before the window

curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- \
  --role first --version 2.0.0

Nothing is upgraded without --commit. Every BKE script works this way: the default reports, and --commit acts.

Run this days before the maintenance window, not at the start of it. It runs the same preflight the real upgrade runs and then stops. A preflight failure found in preparation costs nothing; the same failure found mid-window costs a stalled cluster.

Like apply.sh, this changes nothing on the node. It downloads the target kubeadm into a temporary directory and runs the upgrade plan with it, fetches the container runtime package the upgrade would install, and compares the containerd configuration. It then lists what --commit will change on that node: the apt repository, kubeadm, and the container runtime, whose upgrade restarts containerd. It is safe to run on a node that is carrying traffic.

Run check.sh --version 2.0.0 --mode upgrade on every node at the same time. --mode upgrade makes the cluster checks required rather than skipped, which is what a release test should do.

Read the report before the window, too

curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.0.0

Without --commit this changes nothing, and it answers the questions the window is being booked for:

which versions move, including the ones that do not — “netdata is not moving” is worth knowing;
whether any node drains, and how many;
whether it can be rolled back — charts can, a Kubernetes upgrade cannot;
what changes in each of your values files, and where an edit of yours and one of ours have landed on the same setting;
which Kubernetes objects actually move as a result.

The values conflict is the one that can stop an upgrade, and it is far better found now than at 2am. See Installing components for what the four outcomes mean.

What the objects section is for

A values diff tells you what the file says. It does not tell you what the cluster does about it, and the two come apart in both directions:

  kong    4 changed, 1 added, 1 removed, of 13 objects
  kuma    values changed, NO objects change (of 26)

The second line is the more important of the two. Helm never warns about a values key no template consumes — it is not an error, it is just nothing — so a careful-looking edit can reach nothing at all. The first line is the other half: a one-word change that rewrites six objects.

Both are produced by rendering your values against the pinned chart on each side and comparing the results. Object names and counts only, never bodies, so this section stays safe to paste into a ticket: a rendered manifest contains the values Helm is about to put in a Secret, and some charts generate their own certificates.

A component that could not be rendered says so, by name and with the reason. That line is not a failure of the upgrade — this section informs a decision and never refuses one — but it does mean you have no answer for that component rather than a reassuring one.

The section also states what it cannot see, and those limits are real: CRDs the chart installs once and never updates, objects a chart renders only when some other API is already present, and anything decided at apply time rather than at render time.

Components whose files change nothing are summarised in one line each rather than printed in full — six lines saying unchanged are what make the seventh readable. Add --show-unchanged if you want all of them. It changes the report only; the same upgrade happens either way, which is why it is safe to put in a runbook.

The sequence

apply.sh --commit at the current version, to confirm the cluster is reconciled and nothing has drifted.
upgrade.sh --role first --commit on one control-plane node. This moves the control plane. If that node runs workloads (it has no control-plane taint), drain it first and uncordon it afterwards, as in step 3.
On every remaining node — control-plane and worker alike — one at a time: drain it, run upgrade.sh --role other --commit on it, uncordon it, and wait until it is Ready and every Longhorn volume is healthy before the next node.
apply.sh --version 2.0.0, then --commit, to bring the components to the new graph.

upgrade.sh does not drain the node it runs on. The drain and the uncordon are yours, so that you decide when a node’s workloads move.

# step 2, on one control-plane node
curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --role first --version 2.0.0 --commit

# step 3, for each remaining node: from a control-plane node
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=15m
# then on that node
curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --role other --version 2.0.0 --commit
# then from a control-plane node
kubectl uncordon <node>
kubectl get volumes.longhorn.io -n longhorn-system \
  -o custom-columns=NAME:.metadata.name,ROBUSTNESS:.status.robustness

# step 4, once, from a control-plane node - read it, then commit it
curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.0.0
curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.0.0 --commit

Step 1 also sets where the control plane is pulled from. kubeadm upgrade pulls the control-plane images, kube-proxy and CoreDNS from the registry recorded in the kubeadm-config ConfigMap in kube-system. A cluster created before BKE pinned that registry, or created by another installer, records the upstream registry there. apply.sh shows the one-line change that corrects it, and --commit stores it. Storing it restarts nothing; the registry takes effect at step 2. If your cluster runs a different Kubernetes patch from the one this BKE version pins, apply.sh says so and makes the change at the first apply after an upgrade instead.

--role first and --role other describe order, not importance. The first node is whichever control-plane node you choose to go first; every other node, including other control-plane nodes, is other.

During

You drain the nodes in turn on a minor or major. Workloads without a PodDisruptionBudget and without enough replicas will be unavailable while their node is drained — that is ordinary Kubernetes behaviour, and the upgrade is when you find out whether your budgets are right.

upgrade.sh collects warnings as it goes and prints them at the end rather than burying them in the middle of the output. Read that block; it is where anything that succeeded-but-not-cleanly ends up.

`--ignore-preflight-errors`

upgrade.sh passes this through to kubeadm, and every check you ignore is logged by name. Use it when you know why a check is failing and have decided it is acceptable — not to make an unexplained failure go away.

The one case a customised CoreDNS meets: a Corefile carrying an import line fails the preflight with CoreDNSUnsupportedPlugins on an upgrade that moves CoreDNS to a new version, because kubeadm cannot migrate a Corefile that has one. An upgrade that leaves CoreDNS where it is does not inspect the Corefile, and passes without the flag. check.sh compares the CoreDNS running on your cluster with the version the upgrade installs, warns days ahead when it moves, and prints the exact flag to bring:

--ignore-preflight-errors=CoreDNSUnsupportedPlugins

Do not delete the import to satisfy the preflight. The configuration it pulls in is live, and its loss is silent — nothing fails when it disappears, DNS answers simply change.

`--kernel-modules`

A node installed by BKE has dm_crypt loaded and persisted for every boot — Longhorn’s encrypted volumes need it, and without it an encrypted volume fails at attach time, a long way from the cause. A node that came to BKE some other way may not, and an upgrade does not change how your node boots unless you ask it to.

--kernel-modules asks: the upgrade then loads the module immediately and writes /etc/modules-load.d/bke.conf so it survives a reboot, the same as an installed node. Run it once per node, on the same upgrade run you are already doing; it is idempotent, and it does nothing on a run without --commit. If you do not use encrypted StorageClasses, you do not need it.

Afterwards

curl -sL https://bke.maml.uk/version.sh | sh
kubectl get nodes

version.sh reports what a node is running and changes nothing. Every node should report the new BKE version, and kubectl get nodes should show the new Kubernetes version on all of them. A node left behind is a node that will drift further with the next upgrade.

Rolling back

There is no supported downgrade path. Kubernetes does not support downgrading a control plane, and a BKE version that moves backwards is not a route back to where you were.

BKE does not hard-block it, deliberately — a downgrade during recovery has to stay possible — but it does tell you loudly. check.sh fails on a component that would move backwards, and explains the specific danger: charts carry their CRDs, so a chart moving backwards replaces newer CRD schemas with older ones underneath running workloads, while those workloads keep the newer image, because image tags are pinned separately.

So the protection is on the other side of the upgrade, not behind it. Before you start:

take an etcd snapshot — etcdctl snapshot save on a control-plane node;
confirm your Longhorn backups are current, rather than assuming it.

Storage covers how to confirm the second one, and why a backup target that has never worked looks exactly like one that is idle.