BKE

Upgrading

What the version number tells you

BKE versions are classified by what the upgrade does to a running cluster, not by which piece of software moved. So you can decide whether an upgrade needs a maintenance window from the number alone, without reading a changelog.

Digit that moved What you may conclude
third (patch) component pods rolled. No drain. Your manifests are unaffected
second (minor) a drain and a maintenance window are possible. Your manifests are unaffected, except for the resources of a component whose major version moved, which the release notes name
first (major) schedule it. Your manifests may break. One Kubernetes minor at a time

In detail:

Tier What may move Effect
major the Kubernetes minor API removals are possible, so your manifests can break. Nodes drain
minor the Kubernetes patch, Calico, containerd, component minors and component majors Nodes may drain, the dataplane may restart. No API removals
patch component patches, k9s Workload pods roll. No control-plane change, no drain

A Kubernetes patch is a BKE minor, not a BKE patch, and the naming is the weaker argument here. A component patch is a helm upgrade where that component’s pods roll. A Kubernetes patch is a kubeadm upgrade where the API server and kubelet restart and each node drains in turn. If both were patches, then “patch” would sometimes mean a few pods and sometimes mean a maintenance window — and anyone running an auto-apply-patches policy would receive an unscheduled drain. Calling it a minor fails safe, because the worst case you must already assume for a minor includes a window.

The same holds for urgency: a Kubernetes CVE fix ships as a minor. How urgent a change is does not alter what it does to your cluster.

The most invasive change in a release decides its tier. A release carrying a Kubernetes patch and a Longhorn patch is a minor, and the Longhorn patch travels inside it. That is what makes bundling possible, so several changes can go out in one window instead of each claiming its own.

containerd is upgraded only as part of the node upgrade, which runs on a drained node, so a containerd change is always a minor.

A component’s major version is a BKE minor, not a BKE major. The BKE major is reserved for the Kubernetes minor. A component major can still change how that component treats your own resources, so the release notes of a minor that carries one name the component and list those changes.

You can skip patches. You cannot skip minors.

A BKE patch release can be skipped. A BKE minor release cannot. Upgrade to every minor in turn. Within a minor, go straight to its latest patch: there is no need to apply the patches before it.

From To  
2.0.1 2.0.4 Allowed. Patches may be skipped
2.0.1 2.1.3 Allowed. One minor forward, to any of its patches
2.0.1 2.2.0 Refused. It skips 2.1. Upgrade to the latest 2.1 first

A new major starts from the last minor of the major before it, and arrives at its first minor. To reach 3.0, a cluster must be on the last 2.x minor. Moving to 3.1 directly from 2.x is refused, because it skips 3.0.

check.sh --version <target> --mode upgrade fails on a move that skips a minor, and names the version to upgrade to first. It reads the cluster’s current BKE version from its own record, so run it on a control-plane node. A cluster with no record yet cannot be assessed, and check.sh warns that it was not, rather than passing. Run it days before the window rather than at the start of it.

You also cannot skip a Kubernetes minor, which is why each BKE major moves Kubernetes by exactly one minor. check.sh fails on that too, and names the two versions. kubeadm refuses as well, but it refuses once the upgrade is already under way.

From 2.5.x: apply.sh only

Upgrade from any 2.5 patch. 2.6.0 moves no Kubernetes, containerd or Calico version, so no node is upgraded and no node drains: there is no upgrade.sh step. Run apply.sh once, from a control-plane node. Read the report, then commit it:

curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.6.0
curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.6.0 --commit

Of what 2.6.0 adds, aybak is installed only when config.yaml asks for it (see Feeding your load balancer). Without an aybak block the report says it is off, and nothing of it is installed. Ubuntu support changes nothing on a node you already have.

check.sh at 2.6.0 fails a node on which no time service is running. Earlier versions read only the kernel’s synchronised flag, which stays set after the service stops, so a node that passed before can fail now. Run check.sh --version 2.6.0 on every node before the upgrade, and start the node’s time service where it names one.

A node that keeps time with chrony takes compliance.time.servers once apply.sh has run at 2.6.0. Run upgrade.sh --node-settings --commit on that node to give them to it at once.

cloud-init and the hostname. A node installed at 2.6.0 or later is given preserve_hostname: true for cloud-init (see Before you begin). A node installed earlier is not, because apply.sh changes no file on a node. It is exposed only if its hostname equals the name the provider gave the server, and the server is later renamed at the provider or rebuilt from a snapshot. To close it on such a node, as root:

printf 'preserve_hostname: true\n' > /etc/cloud/cloud.cfg.d/99-bke-preserve-hostname.cfg

From 2.5.0, every Netdata pod restarts once, because Netdata moves to v2.12.3 and the parent’s configuration changes, as in 2.5.1. From 2.5.1 Netdata does not change.

After that restart, a node can be missing from the parent for up to ten minutes, or fifteen where the parent holds ten or more nodes. An agent that restarts before the new parent accepts connections has its first connection time out, and Netdata then waits before it tries again: a random 5 to 10 minutes, or 5 to 15 minutes where the parent holds ten or more nodes. The parent lists that node as stale, and its alarms are not evaluated until it reconnects. The wait is fixed in Netdata itself: its reconnect delay setting does not apply to a connection that timed out, and no setting shortens it.

Moving a cluster from Debian to Ubuntu

A cluster may run Debian and Ubuntu nodes together, so it moves one node at a time: add an Ubuntu node, then take a Debian node out. Add before you remove, so the cluster never runs short of the nodes its workloads and its etcd quorum need. Run every kubectl command below on a control-plane node.

A worker.

  1. Add the Ubuntu worker as Adding nodes describes, and wait until it is Ready.

  2. If Longhorn is installed, have it move the old worker’s replicas first, so that no volume runs on a single copy while you work:

    kubectl -n longhorn-system patch nodes.longhorn.io <old-worker> --type merge \
  -p '{"spec":{"allowScheduling":false,"evictionRequested":true}}'
kubectl -n longhorn-system get replicas.longhorn.io \
  -o custom-columns=VOLUME:.spec.volumeName,NODE:.spec.nodeID

    Wait until no replica lists the old worker, and every volume in kubectl -n longhorn-system get volumes.longhorn.io is healthy.

  3. Drain it, reset it, and remove it:

    kubectl drain <old-worker> --ignore-daemonsets --delete-emptydir-data
# on the old worker, as root:
kubeadm reset -f
# then, on a control-plane node:
kubectl delete node <old-worker>

    Longhorn removes its own record of the node once the Node is gone and scheduling on it is off.

A control-plane node.

  1. Add the Ubuntu node with upgrade.sh --add-node control-plane, giving --address as the existing nodes see the new one, as Adding nodes describes, and wait until it is Ready.

  2. Drain the old node, then reset it. kubeadm reset removes the node’s own member from etcd, so the remaining members keep their quorum:

    kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
# on the old node, as root:
kubeadm reset -f
# then, on another control-plane node:
kubectl delete node <old-node>

Do not take out the node your control-plane endpoint points at while that endpoint is the node’s own address. The endpoint is in every kubeadm and kubeconfig file and on your licence. Where it is a load balancer or a DNS name in front of the control-plane nodes, as Reference architecture recommends, point it at the new node first.

Run check.sh --version 2.6.0 on every node when you are done.

From 2.4.x to 2.5.1

Upgrade from any 2.4 patch; the latest is not required. These steps take a 2.4 cluster directly to 2.5.1, and 2.5.0 need not be applied first. No node is drained for this version: each node’s upgrade restarts its kubelet, and nothing else.

Run the plan on every node before the window. upgrade.sh --version 2.5.1 without --commit lists the kernel settings, modules and files each node will take. It stops on a node with a file in /etc/sysctl.d, or /etc/sysctl.conf, that would replace one of BKE’s settings in 90-bke.conf at the next boot, naming the file and the setting. Correct that file, or set the setting to BKE’s value, before the window.

With --commit, the kernel settings come first. Each node writes the settings and modules, reads every setting back, and stops, with its kubelet untouched, if one did not take. A node that runs in a container, or on a host that does not let it change a setting, stops here. Only after that does upgrade.sh change the kubelet configuration and restart the kubelet with protectKernelDefaults on. To keep the earlier behaviour, set kubelet.protectKernelDefaults: false in config.yaml before the upgrade.

A node where br_netfilter was not loaded has it loaded at its upgrade. Traffic that crosses a Linux bridge on that node passes through iptables from then on. Pod traffic routed by Calico does not cross a bridge.

A setting of your own that differs from BKE’s needs the protection off. With kubelet.protectKernelDefaults: false in config.yaml, name the setting in BKE_ACCEPT_KERNEL_OVERRIDE, on the shell’s side of the pipe:

curl -sL https://bke.maml.uk/upgrade.sh | BKE_ACCEPT_KERNEL_OVERRIDE="kernel.panic" sh -s -- \
  --role other --version 2.5.1

The variable has no effect while the protection is on, because the kubelet would then not start. A node installing and not yet joined cannot read the cluster’s setting, so the refusal stands there too; correct the file, install the node and restore your setting afterwards.

With the protection off, the kubelet sets its six settings itself each time it starts. A value of your own for one of those six does not last past the kubelet’s next start, even with your file in place. The variable lets the upgrade continue; it does not keep the value.

A node whose kubelet refuses to start can go on showing Ready. The kubelet starts, renews its lease and exits, so kubectl get nodes and the node’s lease can show nothing wrong for minutes. On the node, systemctl is-active kubelet reads activating rather than active, systemctl show -p NRestarts --value kubelet keeps rising, and journalctl -u kubelet names the setting as an invalid kernel flag.

upgrade.sh --commit watches the kubelet for 40 seconds after restarting it, and stops with the reason and the way back if it does not stay running. The version is then not recorded on that node.

A file in /etc/sysctl.d written after the upgrade takes effect at the next reboot, with no upgrade running to see it. check.sh --version 2.5.1 on the node catches this: it fails on a kubelet that is restarting, and names the setting. Run it after a node’s reboot, since Ready will not show it. The kubelet check takes 40 seconds on each node.

To bring it back, remove the file that sets the wrong value, then:

sysctl -qp /etc/sysctl.d/90-bke.conf
systemctl restart kubelet

This works on a control-plane node after a reboot as well. If such a file is in place when a control-plane node reboots, its kubelet refuses to start and so its API server does not return until this is done. On a cluster with one control-plane node, the Kubernetes API is unavailable for that time.

We recommend rebooting each node once after its upgrade, inside the window. It is not required. The settings are in force from the upgrade, and a reboot proves that the settings and modules persist, while you are there to see the node come back with its kubelet running. Reboot one node at a time, and wait for every Longhorn volume to be healthy before the next: with volumes of two replicas, rebooting nodes less than a minute apart can leave volumes faulted or degraded for about two minutes, until they recover.

A reboot can cause request failures, so check what each node runs first. A workload with one replica on the node fails its requests until the node has returned and the workload is ready again. On a node holding a Kong Pod, requests can also fail briefly until the load balancer stops sending to that node.

kubectl get pods -A -o wide --field-selector spec.nodeName=<node>

For such a node, move the workload first or defer the reboot to a quiet period.

The Release notes list everything else that changes.

Plan before the window

curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- \
  --role first --version 2.4.0

Nothing is upgraded without --commit. Every BKE script works this way: the default reports, and --commit acts.

Run this days before the maintenance window, not at the start of it. It runs the same preflight the real upgrade runs and then stops. A preflight failure found in preparation costs nothing; the same failure found mid-window costs a stalled cluster.

Like apply.sh, this changes nothing on the node. It downloads the target kubeadm into a temporary directory and runs the upgrade plan with it, fetches the container runtime package the upgrade would install, and compares the containerd configuration. It then lists what --commit will change on that node: the apt repository, kubeadm, the container runtime, and, where the node’s /etc/containerd/config.toml differs from the target version’s, that file, with the number of lines that differ. It is safe to run on a node that is carrying traffic.

With --commit, the containerd configuration and the runtime package change in one step, so containerd restarts once; running containers keep running while it does.

Run check.sh --version <target> --mode upgrade on every node at the same time. --mode upgrade makes the cluster checks required rather than skipped, which is what a release test should do.

Read the report before the window, too

curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.4.0

Without --commit this changes nothing, and it answers the questions the window is being booked for:

  • which versions move, including the ones that do not — “netdata is not moving” is worth knowing;
  • whether any node drains, and how many;
  • whether it can be rolled back — charts can, a Kubernetes upgrade cannot;
  • what changes in each of your values files, and where an edit of yours and one of ours have landed on the same setting;
  • which Kubernetes objects actually move as a result;
  • which values your file holds back — a setting your file keeps at the old chart’s default, and a setting BKE makes that your file leaves out;
  • whether the new software accepts its configuration, checked on the node before anything is applied.

Three findings stop an upgrade, and all three are far better found now than in the window: a values conflict, values the new chart will not accept, and a pre-flight check the new software fails (both below). See Installing components for what the four values outcomes mean.

What the objects section is for

A values diff tells you what the file says. It does not tell you what the cluster does about it, and the two come apart in both directions:

  kong    4 changed, 1 added, 1 removed, of 13 objects
  kuma    values changed, NO objects change (of 26)

The second line is the more important of the two. Helm never warns about a values key no template consumes — it is not an error, it is just nothing — so a careful-looking edit can reach nothing at all. The first line is the other half: a one-word change that rewrites six objects.

Both are produced by rendering your values against the pinned chart on each side and comparing the results. Object names and counts only, never bodies, so this section stays safe to paste into a ticket: a rendered manifest contains the values Helm is about to put in a Secret, and some charts generate their own certificates.

A component that could not be rendered says so, by name and with the reason. That line is not a failure of the upgrade, and it does not stop the run, but it does mean you have no answer for that component rather than a reassuring one.

Values your file holds back

Two things can keep a setting where it is without any line of your file changing, so neither values diff can show them. The report lists both, for every field alike, and neither stops the run.

A value at the old chart’s default. If your file sets a value to exactly what the old chart used by default, and the new chart’s default is different, your file keeps the old value and the setting does not move. A file written from a running installation usually carries every image tag this way:

  longhorn
    image.longhorn.manager.tag
      your file v1.0.0 (the old default), new default v1.1.0

The chart is upgraded and the software is not. To take the new default, remove the line from your file. To keep the old value, leave it.

A BKE setting your file leaves out. BKE’s values set some settings for every cluster. If your file does not contain one, the chart’s own default applies instead, and the report says what that is. If you removed it on purpose, nothing needs doing; to restore it, add the line back with the value shown.

Values the new chart will not accept

Most charts publish a schema for their values, and Helm checks your file against it before it changes anything. A new chart version can remove a key that the previous one accepted. If your file still sets that key, the report marks the component REFUSED, lists each key under Values the new chart refuses, and stops: apply.sh applies components one at a time, so --commit would upgrade the components before that one and then fail, leaving the cluster part-upgraded.

  cert-manager, against chart v1.21.2
    - at '/prometheus/servicemonitor': additional properties 'path',
      'targetPort' not allowed

Each path is a key in /etc/bke/values/<component>.yaml. Additional properties not allowed means the chart no longer has that key. Check whether the setting still does anything on your cluster, remove or correct each key named, and run the report again. A file that was taken over from a cluster BKE did not install often carries a full copy of an earlier chart’s defaults, and those copies are the usual source.

The section also states what it cannot see, and those limits are real: CRDs the chart installs once and never updates, objects a chart renders only when some other API is already present, and anything decided at apply time rather than at render time.

Components whose files change nothing are summarised in one line each rather than printed in full — six lines saying unchanged are what make the seventh readable. Add --show-unchanged if you want all of them. It changes the report only; the same upgrade happens either way, which is why it is safe to put in a runbook.

Pre-flight checks: what the new software will not accept

A new version of a component can refuse a configuration that the old version accepted. The values are then the same before and after, so no diff shows anything, and a chart’s schema has no view of a free-text configuration file. The only reliable answer comes from the new software itself. So before the report is printed, apply.sh checks every component that the run changes, on the node it runs on:

  • Every image exists. Only the image’s manifest is fetched, a few kilobytes, so a missing tag is found without downloading anything large. An image that is pulled from outside the BKE mirror is listed as a warning.
  • The API server accepts every object, in a server-side dry run with strict field validation. Nothing is stored.
  • The software accepts its own configuration, where it can say so. Each check runs the component’s new image against the configuration it would be given, in a container that has no network and only the files the check provides. A ConfigMap or Secret of your own that the pod mounts, such as a Kong plugin listed in plugins.configMaps, is read from your cluster on the node and given to the check as the pod receives it; it does not leave the node:
Component The check
Fluent Bit its own configuration test, --dry-run
Kong kong check, kong prepare (which runs nginx -t), then a start and stop
Kong ingress controller, cert-manager, metrics-server, Kuma start-up against an API server that does not answer: a refused configuration exits before it ever tries to connect
Netdata start-up, then its effective configuration is read back; any option it ignores is a warning, because Netdata never refuses a configuration
Longhorn none beyond the two checks above; Longhorn’s own pre-flight covers what can be known in advance

A check that fails prints the software’s own error lines under it, and the report stops:

  fluent-bit
    FAILED   Fluent Bit configuration test (--dry-run) (exit 1)
               [error] [parser] parser named 'apache' already exists, skip.
               [error] [config] could not configure service property
                 'parsers_file'
               [error] configuration file contains errors, aborting.

Correct /etc/bke/values/<component>.yaml and run the report again. In this example the file defines a parser that the new Fluent Bit image already ships under the same name. The older version logged the duplicate and continued; the newer one stops. Removing the duplicate definitions from config.customParsers is the correction.

A check reproduces the pod outside the cluster, and a reproduction can be wrong where the software is not. If you are certain that a failed check is wrong, accept it by name, and the run proceeds with the failure still printed:

BKE_ACCEPT_VALUES_RISK="preflight" \
  sh -c 'curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.4.0 --commit'

A check that cannot be run never stops the run. No ctr on the node, a registry that asks for credentials, an image that will not download: each is reported as not run, with the reason, and the upgrade proceeds exactly as it would have without the checks.

What the checks cost. Each image under test is downloaded once into a separate containerd namespace, bke-preflight, and removed when the checks finish, together with the namespace. Where every component changes, as in a minor upgrade, that is several hundred megabytes on the node where apply.sh runs, and a few minutes. A component that does not change is not checked: the software running it has already accepted its configuration.

What the checks cannot see. A configuration test checks that a configuration is valid; it does not connect to a log destination or a database, so a wrong host name still passes. An option the software silently ignores passes too, except in Netdata, which reports it. The dry run does not check NodePort ranges, and it skips a namespace that the apply has yet to create.

What leaves the node. The checks are prepared by the BKE API from the same values the report is built from, and their results are sent back to it so that the report can include them: each check’s exit status and the last part of its output. Nothing is stored.

The sequence

  1. apply.sh --commit at the current version, to confirm the cluster is reconciled and nothing has drifted.
  2. upgrade.sh --role first --commit on one control-plane node. This moves the control plane. If that node runs workloads (it has no control-plane taint), drain it first and uncordon it afterwards, as in step 3.
  3. On every remaining node — control-plane and worker alike — one at a time: drain it, run upgrade.sh --role other --commit on it, uncordon it, and wait until it is Ready and every Longhorn volume is healthy before the next node.
  4. apply.sh --version <new>, then --commit, to bring the components to the new graph.

upgrade.sh does not drain the node it runs on. The drain and the uncordon are yours, so that you decide when a node’s workloads move.

# step 2, on one control-plane node
curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --role first --version 2.4.0 --commit

# step 3, for each remaining node: from a control-plane node
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=15m
# then on that node
curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --role other --version 2.4.0 --commit
# then from a control-plane node
kubectl uncordon <node>
kubectl get volumes.longhorn.io -n longhorn-system \
  -o custom-columns=NAME:.metadata.name,ROBUSTNESS:.status.robustness

# step 4, once, from a control-plane node - read it, then commit it
curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.4.0
curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.4.0 --commit

Step 1 also sets where the control plane is pulled from. kubeadm upgrade pulls the control-plane images, kube-proxy and CoreDNS from the registry recorded in the kubeadm-config ConfigMap in kube-system. A cluster created before BKE pinned that registry, or created by another installer, records the upstream registry there. apply.sh shows the one-line change that corrects it, and --commit stores it. Storing it restarts nothing; the registry takes effect at step 2. If your cluster runs a different Kubernetes patch from the one this BKE version pins, apply.sh says so and makes the change at the first apply after an upgrade instead.

--role first and --role other describe order, not importance. The first node is whichever control-plane node you choose to go first; every other node, including other control-plane nodes, is other.

During

You drain the nodes in turn. Workloads without a PodDisruptionBudget and without enough replicas will be unavailable while their node is drained — that is ordinary Kubernetes behaviour, and the upgrade is when you find out whether your budgets are right.

What to expect in an upgrade that drains nodes:

  • In a release that moves Calico, Calico restarts on every node while upgrade.sh runs on the first control-plane node, before any other node is drained. Pod traffic pauses briefly on each node, and writes to a Longhorn volume can pause for around 20 seconds and then resume without error.
  • A Longhorn volume that moves with a drained node pauses for as long as it takes to detach and attach again on another node, typically under a minute.
  • Kong answers only on nodes running a Kong pod, because its proxy Service keeps the client’s address (externalTrafficPolicy: Local). A drained node stops answering on ports 32080 and 32443, and Kubernetes does not move pods back after an uncordon. A load balancer that health-checks the NodePort sends traffic only to nodes that answer, and with Kong’s two replicas on different nodes, one node answers throughout.

upgrade.sh collects warnings as it goes and prints them at the end rather than burying them in the middle of the output. Read that block; it is where anything that succeeded-but-not-cleanly ends up.

--ignore-preflight-errors

upgrade.sh passes this through to kubeadm, and every check you ignore is logged by name. Use it when you know why a check is failing and have decided it is acceptable — not to make an unexplained failure go away.

The one case a customised CoreDNS meets: a Corefile carrying an import line fails the preflight with CoreDNSUnsupportedPlugins on an upgrade that moves CoreDNS to a new version, because kubeadm cannot migrate a Corefile that has one. An upgrade that leaves CoreDNS where it is does not inspect the Corefile, and passes without the flag. check.sh compares the CoreDNS running on your cluster with the version the upgrade installs, warns days ahead when it moves, and prints the exact flag to bring:

--ignore-preflight-errors=CoreDNSUnsupportedPlugins

Do not delete the import to satisfy the preflight. The configuration it pulls in is live, and its loss is silent — nothing fails when it disappears, DNS answers simply change.

--kernel-modules

A node installed by BKE has dm_crypt loaded and persisted for every boot — Longhorn’s encrypted volumes need it, and without it an encrypted volume fails at attach time, a long way from the cause. A node that came to BKE some other way may not, and an upgrade does not change how your node boots unless you ask it to.

--kernel-modules asks: the upgrade then loads the module immediately and writes /etc/modules-load.d/bke.conf so it survives a reboot, the same as an installed node. Run it once per node, on the same upgrade run you are already doing; it is idempotent, and it does nothing on a run without --commit. If you do not use encrypted StorageClasses, you do not need it.

Afterwards

curl -sL https://bke.maml.uk/version.sh | sh
kubectl get nodes

version.sh reports what a node is running and changes nothing. Every node should report the new BKE version, and kubectl get nodes should show the new Kubernetes version on all of them. A node left behind is a node that will drift further with the next upgrade.

Rolling back

There is no supported downgrade path. Kubernetes does not support downgrading a control plane, and a BKE version that moves backwards is not a route back to where you were.

BKE does not hard-block it, deliberately — a downgrade during recovery has to stay possible — but it does tell you loudly. check.sh fails on a component that would move backwards, and explains the specific danger: charts carry their CRDs, so a chart moving backwards replaces newer CRD schemas with older ones underneath running workloads, while those workloads keep the newer image, because image tags are pinned separately.

So the protection is on the other side of the upgrade, not behind it. Before you start:

  • take an etcd snapshot — etcdctl snapshot save on a control-plane node. On a cluster that encrypts Secrets, the snapshot can be restored only with the key in /etc/kubernetes/bke/encryption-config.yaml, so confirm you hold a copy of it, stored apart from the snapshot;
  • confirm your Longhorn backups are current, rather than assuming it.

Storage covers how to confirm the second one, and why a backup target that has never worked looks exactly like one that is idle.