BKE Bahriya Kubernetes Engine Start a 14-day trial

Upgrading

What the version number tells you

BKE versions are classified by what the upgrade does to a running cluster, not by which piece of software moved. So you can decide whether an upgrade needs a maintenance window from the number alone, without reading a changelog.

Digit that moved What you may conclude
third (patch) component pods rolled. No drain. Your manifests are unaffected
second (minor) a drain and a maintenance window are possible. Your manifests are unaffected
first (major) schedule it. Your manifests may break. One Kubernetes minor at a time

In detail:

Tier What may move Effect
major the Kubernetes minor API removals are possible, so your manifests can break. Nodes drain
minor the Kubernetes patch, Calico, component minors Nodes may drain, the dataplane may restart. No API removals
patch component patches, k9s Workload pods roll. No control-plane change, no drain

A Kubernetes patch is a BKE minor, not a BKE patch, and the naming is the weaker argument here. A component patch is a helm upgrade where that component’s pods roll. A Kubernetes patch is a kubeadm upgrade where the API server and kubelet restart and each node drains in turn. If both were patches, then “patch” would sometimes mean a few pods and sometimes mean a maintenance window — and anyone running an auto-apply-patches policy would receive an unscheduled drain. Calling it a minor fails safe, because the worst case you must already assume for a minor includes a window.

The same holds for urgency: a Kubernetes CVE fix ships as a minor. How urgent a change is does not alter what it does to your cluster.

The most invasive change in a release decides its tier. A release carrying a Kubernetes patch and a Longhorn patch is a minor, and the Longhorn patch travels inside it. That is what makes bundling possible, so several changes can go out in one window instead of each claiming its own.

Major versions are adjacent only. You cannot skip a Kubernetes minor. check.sh --version <target> --mode upgrade fails on an attempt to, and names the two versions — which is one of the reasons to run it days before the window rather than at the start of it. kubeadm refuses as well, but it refuses once the upgrade is already under way.

Plan before the window

curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- \
  --role first --version 2.1.0

Nothing is upgraded without --commit. Every BKE script works this way: the default reports, and --commit acts.

Run this days before the maintenance window, not at the start of it. It runs the same preflight the real upgrade runs and then stops. A preflight failure found in preparation costs nothing; the same failure found mid-window costs a stalled cluster.

Unlike apply.sh, this is not a no-op, and it says so rather than implying otherwise. To plan an upgrade accurately it needs the kubeadm it is planning for, so it points apt at the new Kubernetes minor, upgrades the kubeadm binary, and brings the container runtime onto the version the new BKE version pins — which on a node behind that pin restarts containerd. It does not touch kubelet, the control plane, or any workload. Run it on a node you are willing to have containerd restart on.

Run check.sh --version <target> --mode upgrade on every node at the same time. --mode upgrade makes the cluster checks required rather than skipped, which is what a release test should do.

Read the report before the window, too

curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.1.0

Without --commit this changes nothing, and it answers the questions the window is being booked for:

  • which versions move, including the ones that do not — “netdata is not moving” is worth knowing;
  • whether any node drains, and how many;
  • whether it can be rolled back — charts can, a Kubernetes upgrade cannot;
  • what changes in each of your values files, and where an edit of yours and one of ours have landed on the same setting;
  • which Kubernetes objects actually move as a result.

The values conflict is the one that can stop an upgrade, and it is far better found now than at 2am. See Installing components for what the four outcomes mean.

What the objects section is for

A values diff tells you what the file says. It does not tell you what the cluster does about it, and the two come apart in both directions:

  kong    4 changed, 1 added, 1 removed, of 13 objects
  kuma    values changed, NO objects change (of 26)

The second line is the more important of the two. Helm never warns about a values key no template consumes — it is not an error, it is just nothing — so a careful-looking edit can reach nothing at all. The first line is the other half: a one-word change that rewrites six objects.

Both are produced by rendering your values against the pinned chart on each side and comparing the results. Object names and counts only, never bodies, so this section stays safe to paste into a ticket: a rendered manifest contains the values Helm is about to put in a Secret, and some charts generate their own certificates.

A component that could not be rendered says so, by name and with the reason. That line is not a failure of the upgrade — this section informs a decision and never refuses one — but it does mean you have no answer for that component rather than a reassuring one.

The section also states what it cannot see, and those limits are real: CRDs the chart installs once and never updates, objects a chart renders only when some other API is already present, and anything decided at apply time rather than at render time.

Components whose files change nothing are summarised in one line each rather than printed in full — six lines saying unchanged are what make the seventh readable. Add --show-unchanged if you want all of them. It changes the report only; the same upgrade happens either way, which is why it is safe to put in a runbook.

The sequence

  1. apply.sh --commit at the current version, to confirm the cluster is reconciled and nothing has drifted.
  2. upgrade.sh --role first --commit on one control-plane node. This moves the control plane.
  3. upgrade.sh --role other --commit on every remaining node — control-plane and worker alike — one at a time, waiting for each to return Ready.
  4. apply.sh --version <new>, then --commit, to bring the components to the new graph.
# step 2, on one control-plane node
curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --role first --version 2.1.0 --commit

# step 3, on each remaining node
curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --role other --version 2.1.0 --commit

# step 4, once, from a control-plane node - read it, then commit it
curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.1.0
curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.1.0 --commit

--role first and --role other describe order, not importance. The first node is whichever control-plane node you choose to go first; every other node, including other control-plane nodes, is other.

During

Nodes drain in turn on a minor or major. Workloads without a PodDisruptionBudget and without enough replicas will be unavailable while their node is drained — that is ordinary Kubernetes behaviour, and the upgrade is when you find out whether your budgets are right.

upgrade.sh collects warnings as it goes and prints them at the end rather than burying them in the middle of the output. Read that block; it is where anything that succeeded-but-not-cleanly ends up.

--ignore-preflight-errors

upgrade.sh passes this through to kubeadm, and every check you ignore is logged by name. Use it when you know why a check is failing and have decided it is acceptable — not to make an unexplained failure go away.

The one case a customised CoreDNS meets on every upgrade: a Corefile carrying an import line fails the preflight with CoreDNSUnsupportedPlugins, because kubeadm cannot migrate a Corefile that has one. check.sh warns about this days ahead and prints the exact flag to bring:

--ignore-preflight-errors=CoreDNSUnsupportedPlugins

Do not delete the import to satisfy the preflight. The configuration it pulls in is live, and its loss is silent — nothing fails when it disappears, DNS answers simply change.

--kernel-modules

A node installed by BKE has dm_crypt loaded and persisted for every boot — Longhorn’s encrypted volumes need it, and without it an encrypted volume fails at attach time, a long way from the cause. A node that came to BKE some other way may not, and an upgrade does not change how your node boots unless you ask it to.

--kernel-modules asks: the upgrade then loads the module immediately and writes /etc/modules-load.d/bke.conf so it survives a reboot, the same as an installed node. Run it once per node, on the same upgrade run you are already doing; it is idempotent, and it does nothing on a run without --commit. If you do not use encrypted StorageClasses, you do not need it.

Afterwards

curl -sL https://bke.maml.uk/version.sh | sh
kubectl get nodes

version.sh reports what a node is running and changes nothing. Every node should report the new BKE version, and kubectl get nodes should show the new Kubernetes version on all of them. A node left behind is a node that will drift further with the next upgrade.

Rolling back

There is no supported downgrade path. Kubernetes does not support downgrading a control plane, and a BKE version that moves backwards is not a route back to where you were.

BKE does not hard-block it, deliberately — a downgrade during recovery has to stay possible — but it does tell you loudly. check.sh fails on a component that would move backwards, and explains the specific danger: charts carry their CRDs, so a chart moving backwards replaces newer CRD schemas with older ones underneath running workloads, while those workloads keep the newer image, because image tags are pinned separately.

So the protection is on the other side of the upgrade, not behind it. Before you start:

  • take an etcd snapshot — etcdctl snapshot save on a control-plane node;
  • confirm your Longhorn backups are current, rather than assuming it.

Storage covers how to confirm the second one, and why a backup target that has never worked looks exactly like one that is idle.