Storage

Longhorn is the only component whose failure mode is data loss, so it is the only one BKE refuses to touch when the cluster is not in a fit state.

Before it installs

components:
  longhorn:
    enabled: true
    namespace: longhorn-system
    secrets:
      - longhorn-backup
    settings:
      backup:
        target: s3://bke-backups@lon/prod
        credentialSecret: longhorn-backup
      defaultClassReplicaCount: 3
      storageMinimalAvailablePercentage: 15

A cluster can run without a backup store. Set both backup.target and backup.credentialSecret to "" and leave the secrets: list out; no Secret and no /etc/bke/secrets.d/ directory are then needed. This suits an evaluation cluster. For a cluster holding real data, configure a backup target before the data exists: Longhorn is the one component whose failure mode is data loss, and the backup-target refusal below can only protect backups that happen.

defaultClassReplicaCount is worth deciding rather than defaulting. 3 is right for production and needs at least three nodes Longhorn can use; 1 is for a single-node or test cluster and gives you no redundancy at all. Count only the nodes that run workloads: a control-plane node that keeps its control-plane taint holds no replicas. A count higher than the number of such nodes leaves every volume permanently degraded. BKE applies it to the default StorageClass, which every PersistentVolumeClaim without a class of its own uses, and to a volume created in the Longhorn UI without a count. Changing it later affects new volumes, not existing ones; change an existing volume’s count on the volume, in the Longhorn UI.

check.sh compares each volume’s replica count with its StorageClass. A volume whose class sets numberOfReplicas passes at that count or above and is a warning below it, so a class you made for two replicas is not reported. A volume whose class sets no count is a warning below three, because during a drain such a volume has no spare replica.

dm_crypt must be available for encrypted volumes. BKE enables it at install time on every node it provisions. A node provisioned before that existed, or one whose kernel lacks the module, will attach ordinary volumes fine and fail to attach encrypted ones.

Longhorn will use every schedulable node for replicas unless you tell it otherwise. If some nodes are not meant to carry storage, decide that before installing rather than after replicas have been placed.

The seven refusals

apply.sh checks the cluster’s storage state before the component loop, not during it, so a refusal happens before anything has been changed. Each condition has a name, and the name is what you type back if you decide to proceed anyway.

Condition	Refuses when
`degraded-volume`	a volume is not fully replicated. Upgrading now risks the only healthy replica
`engine-skew`	volumes are running an engine image more than one minor behind the current one. One minor behind warns; beyond one refuses, because Longhorn supports the current engine and the previous one only
`backup-target`	the backup target is unavailable, or has not synced in over 24 hours
`incomplete-backup`	a backup is in progress
`node-not-ready`	a Longhorn node is not ready, so its replicas are not available
`minor-skip`	the upgrade would skip a Longhorn minor version
`downgrade`	the target is older than what is installed

Each refusal names the condition and says what is wrong, on one line, so the name stays attached to its reason.

Proceeding anyway

BKE_ACCEPT_LONGHORN_RISK="backup-target" \
  sh -c 'curl -sL https://bke.maml.uk/apply.sh | sh -s -- --version 2.6.0 --commit'

The sh -c wrapper is required, not a style. Written as BKE_ACCEPT_LONGHORN_RISK="..." curl ... | sh, the setting applies to curl and never reaches the script on the other side of the pipe — so the refusal appears again and nothing explains why. The same is true of every setting on this page and of BKE_ACCEPT_VALUES_RISK. Exporting it first works equally well.

It takes condition names, not a blanket yes. You can accept a stale backup target without also accepting a degraded volume. Accepting one is logged as a warning naming what was accepted.

There is no override for minor-skip or downgrade that makes them safe — accepting them means you have decided the risk is yours.

Backups

Check your backup target actually works, once, when you first configure it:

kubectl -n longhorn-system get backuptargets.longhorn.io default \
  -o custom-columns=URL:.spec.backupTargetURL,AVAILABLE:.status.available,SYNCED:.status.lastSyncedAt

AVAILABLE must be true and SYNCED must be recent. A target that has never worked looks exactly like one that is idle, and the reason is only in the longhorn-manager log:

kubectl -n longhorn-system logs -l app=longhorn-manager --tail=200 | grep -i backup

Two mistakes account for most failures:

A missing scheme on AWS_ENDPOINTS. s3.example.com is rejected as an invalid URI; https://s3.example.com is accepted. The symptom is a target that never becomes available.

A region that does not apply. If your object storage is not AWS, the region in the target URI is whatever that vendor uses, and copying one from an AWS example gives you a target that authenticates and then cannot find the bucket.

BKE checks the target’s availability and staleness as part of its preflight, so once it is configured you will hear about it going stale. It cannot check it before it exists.

Daily backups

A backup target says where backups go. It does not schedule any. Without a schedule, Longhorn writes a backup only when someone asks for one. BKE creates a daily backup job when you turn it on in config.yaml:

components:
  longhorn:
    settings:
      backup:
        target: s3://<bucket>@<region>/<path>
        credentialSecret: longhorn-backup
        daily:
          enabled: true          # off unless set
          schedule: "0 2 * * *"  # five-field cron, UTC; this is the default
          retain: 7              # backups kept per volume; this is the default
        maxAgeHours: 168         # check.sh fails beyond this; this is the default

With daily.enabled: true, apply.sh --commit creates the Longhorn RecurringJob bke-daily-backup in Longhorn’s namespace. It backs up each volume on the schedule, one volume at a time, and keeps the newest retain backups of each. Daily backups need a backup target: apply.sh refuses daily.enabled: true without one, because a job with nowhere to write would fail on every run. Setting daily.enabled back to false removes bke-daily-backup and nothing else.

Retention is by count, not by age. Longhorn can also keep backups for a period of time. A job whose backups have been failing for longer than that period would then delete the last good backups and could leave a volume with none. A count keeps the newest backups that exist, however old they are.

Which volumes it covers. The job is in Longhorn’s default group: it runs on every volume labelled recurring-job-group.longhorn.io/default=enabled. Longhorn writes that label itself onto every volume that has no recurring-job label, when the volume is created and whenever it is updated. It does not remove the label when you add one of your own.

To keep a volume out of BKE’s job, give it a recurring job of your own and remove the default label, in one command:

kubectl -n longhorn-system label volumes.longhorn.io <volume> \
  recurring-job.longhorn.io/<your-job>=enabled \
  recurring-job-group.longhorn.io/default-

Longhorn does not put the default label back while the volume carries a recurring-job label of its own. If the volume’s PersistentVolumeClaim is labelled recurring-job.longhorn.io/source=enabled, Longhorn takes the volume’s recurring-job labels from the claim instead, so label the claim.

BKE does not change or remove a recurring job it did not create. A job of your own, including one that is also in the default group, runs beside bke-daily-backup; a volume in both is backed up by both. A job of your own named bke-daily-backup stops apply.sh rather than being replaced.

What check.sh reports. Where bke-daily-backup exists, check.sh fails a volume it covers when that volume’s newest completed backup, from any job, is older than maxAgeHours, or when the volume has no completed backup. A volume is not due a backup until maxAgeHours have passed since the later of its own creation and the job’s, so turning the job on does not fail every existing volume before the job’s first run. Where bke-daily-backup does not exist, check.sh reports the age check as skipped.

Choose a schedule outside your maintenance windows. apply.sh refuses to start while a Longhorn backup is in progress (incomplete-backup, above), because an upgrade of Longhorn restarts the replicas a running backup reads from.

Engine images do not move on their own

This is the one that surprises people.

Upgrading the Longhorn chart does not upgrade the engine image of existing volumes. Each volume pins the engine it was created with, and it keeps it. After a chart upgrade you can have a cluster running the new Longhorn with every volume still on the old engine — indefinitely.

The setting that moves them is Longhorn’s concurrent-automatic-engine-upgrade-per-node-limit, and it defaults to 0, meaning never.

This is a Longhorn setting, and BKE deliberately leaves it to you. Upgrading a volume’s engine detaches and reattaches it, and that is not something to trigger as a side effect of installing components — a cluster that is fine one moment should not lose volume attachments because someone applied a chart. BKE reports the skew on every run and refuses an upgrade that would outrun it; the moment to move the engines is yours to choose.

Our recommendation is 1:

kubectl -n longhorn-system patch settings.longhorn.io   concurrent-automatic-engine-upgrade-per-node-limit   --type=merge -p '{"value":"1"}'

One at a time per node, so volumes are interrupted gradually rather than together. A higher number is not faster in any way that helps you.

Because BKE does not manage this setting, your change is not overwritten by the next apply.sh — and equally is not restored if something else removes it. Check it after any Longhorn change:

kubectl -n longhorn-system get settings.longhorn.io   concurrent-automatic-engine-upgrade-per-node-limit   -o jsonpath='{.value}'

check.sh reports this value, and says plainly when it is 0 that Longhorn will not upgrade engines on its own.

Check what your volumes are actually on:

kubectl -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,ENGINE:.status.currentImage,STATE:.status.state

More than one version in that column is skew. One minor behind is reported as outstanding work; more than one minor behind refuses the next upgrade, because Longhorn itself supports only the current engine and the one before it.

The reporting starts at the first minor deliberately. Leaving this setting at 0 is exactly how a cluster reaches three minors of skew without anyone noticing, and by then the fix is large. Told at one minor, it is small.

Longhorn’s internal network policies

Longhorn creates six NetworkPolicies in its own namespace, and BKE leaves them on: longhorn-manager, instance-manager, longhorn-webhook, backing-image-manager, backing-image-data-source and longhorn-recovery-backend. They restrict traffic into Longhorn’s own pods to the Longhorn components that need it. Nothing else in the cluster is affected, and Calico enforces them.

Their purpose is that a pod outside Longhorn can no longer reach longhorn-manager on port 9500. That port serves Longhorn’s API, which can delete volumes and start restores, as well as its metrics.

A metrics scraper outside Longhorn’s namespace is blocked too. If your own monitoring reads longhorn-manager on port 9500, let it through in one of two ways. Either name it in /etc/bke/values/longhorn.yaml, as narrowly as the scraper allows, with a namespace and a pod selector in one entry:

networkPolicies:
  metricsScrapeSources:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
      podSelector:
        matchLabels:
          app.kubernetes.io/name: prometheus

or add a NetworkPolicy of your own in Longhorn’s namespace. NetworkPolicies are additive, so one of yours opens a path without changing Longhorn’s:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-metrics-from-monitoring
  namespace: longhorn-system
spec:
  podSelector:
    matchLabels:
      app: longhorn-manager
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 9500
          protocol: TCP

Replace monitoring and the pod labels with your scraper’s.

Two paths stay open to any source. Port 3260 (iSCSI) on the instance managers, and the admission webhooks on 9501 and 9502, unless you set networkPolicies.v1DataEngineInitiatorSourceCIDRs and networkPolicies.kubeAPIServerSourceCIDRs in the same file to the addresses that must reach them.

Turning them off. If Longhorn misbehaves after the upgrade in a way you suspect is network-related, such as a volume that will not attach, set this in /etc/bke/values/longhorn.yaml and run apply.sh --commit:

networkPolicies:
  restrictInternalTraffic: false

The policies are removed and no stored data changes. Remove the line again to turn them back on.

networkPolicies.enabled is a different setting, for a policy on Longhorn’s web interface on other Kubernetes distributions. BKE leaves it off.

The web interface

Longhorn includes a web interface for its volumes, replicas, backups and engine images. It is not exposed outside the cluster by default. To open it, forward its port to your own machine:

kubectl -n longhorn-system port-forward svc/longhorn-frontend 8080:80

then open http://localhost:8080. Substitute the namespace your config.yaml names. The interface can change volumes as well as show them, so publish it outside the cluster only behind authentication.

Before a cluster upgrade

Longhorn is the reason to take the ordering in Upgrading seriously:

Confirm the backup target is available and recently synced.
Confirm no volume is degraded — kubectl -n longhorn-system get volumes.longhorn.io and look for anything not attached/healthy.
Confirm every volume is on the same engine image.
Then start.

All four are what the preflight checks. Checking them yourself beforehand means that a refusal, if it comes, arrives before the maintenance window rather than during it.