Before you begin
BKE turns a set of Debian or Ubuntu servers into a Kubernetes cluster. It installs upstream software — kubelet, containerd, Calico, Helm — pinned to versions we have tested together, and then installs the components you have chosen on top.
This page is the one worth reading in full before you touch a machine. Most of what goes wrong in an installation was decided before it started.
If control plane, worker and kubectl are new words, read How a cluster fits together first — every page from here on assumes them, and it also carries the table of which command runs on which machine.
What a machine needs
Debian 12 (bookworm), Debian 13 (trixie), Ubuntu 24.04 LTS (noble) or Ubuntu 26.04 LTS (resolute), on amd64 or arm64. All four are tested against every release before it is published, and they are the only four this version installs on.
Another release is refused. BKE installs the container runtime from a
specific package that each BKE version pins for a named release, and this
version names only these four. There is no fallback: installing another
release’s package would put it on a machine that did not ask for it. check.sh
reports this, and install.sh stops before it changes anything on the machine.
Ubuntu’s interim releases, such as 25.10, are refused for the same reason: BKE
supports Ubuntu’s long-term support releases only.
Earlier BKE versions took the runtime from a third-party apt repository, which is why an untested Debian used to be a warning rather than a refusal. That repository also meant the same BKE version could install a different runtime depending on the day it ran. The pin removes both.
Anything that is neither Debian nor Ubuntu is refused for the same reason, and is caught earlier still — before the version graph is even fetched.
A cluster may mix Debian and Ubuntu nodes, for example while you move it from one to the other. Each node installs the runtime package built for its own release, and every package one BKE version pins carries the same containerd version, so the nodes run the same runtime. Every node must be on a release the BKE version supports.
Kubernetes packages come from pkgs.k8s.io, which is the only third party a
node reaches.
Root. Every script is run as root or under sudo. They set hostnames, write
to /etc/, manage apt, and load kernel modules.
amd64 or arm64. Helm is fetched for the architecture the node reports.
Anything else is refused rather than guessed at.
Disk, in two places that fill at different rates:
| Path | Refuses below | Comfortable |
|---|---|---|
/ |
5 GB | 10 GB |
/var/lib |
20 GB | 40 GB |
/var/lib is the one to size generously. Every container image the cluster ever
pulls lands there, and it grows with what you run rather than with what you
installed.
curl on PATH. Nothing can be fetched without it. Everything else BKE
needs — jq, kubectl, tar, etcd-client — is installed for you.
Swap will be turned off. install.sh runs swapoff -a and comments the swap
entries out of /etc/fstab. This is not a preference; the kubelet refuses to
start with swap enabled. If a machine is running something else that expects
swap, it is not a candidate for a cluster node.
A clock within 60 seconds of correct. Certificate validation tolerates some
drift; kubeadm tolerates about five minutes. Beyond a minute you are relying on
that tolerance, and the failure — a node that joins and is then rejected — does
not name the clock as the cause. Run systemd-timesyncd, chrony or ntpsec
before anything else. check.sh fails a node on which none of the three is
running, because the kernel can go on reporting the clock as synchronised long
after its time service has stopped.
On Ubuntu
The universe component must be enabled, or the packages etcd-client and
s3cmd otherwise available, because install.sh installs both and Ubuntu
carries them in universe. Ubuntu’s server images enable it by default.
check.sh names either package if apt cannot find it, before you install.
needrestart is left as you have configured it. Ubuntu runs needrestart after apt installs a package, and in its default mode it restarts the services it judges affected, which can include containerd and the kubelet. BKE suspends it for the packages BKE itself installs, and restarts those services itself at the point in each step where a restart is safe. Package updates you run yourself, including unattended upgrades, are handled by needrestart as your configuration says, and BKE does not change that configuration. In our testing, one unattended-upgrades run on Ubuntu 24.04 restarted containerd; every node runs the same schedule, so the nodes can restart it at about the same time. To prevent that, add this drop-in on each Ubuntu node, and restart the two services yourself, a node at a time, when an update needs it:
printf '%s\n' '$nrconf{override_rc}{qr(^containerd\.service$)} = 0;' \
'$nrconf{override_rc}{qr(^kubelet\.service$)} = 0;' > /etc/needrestart/conf.d/kubernetes.conf
check.sh says on each Ubuntu node whether the drop-in is in place.
Ubuntu 26.04 keeps time with chrony, not systemd-timesyncd, from Ubuntu’s own servers over NTS, which authenticates the time. BKE configures whichever of the two the node runs. If you name your own time servers, read Compliance and hardening first: chrony keeps to Ubuntu’s authenticated servers unless yours offer NTS too.
The hostname, where cloud-init runs
cloud-init sets a cloud server’s hostname at its first boot, and resets it at
later boots if the name is still the one cloud-init set. A node whose BKE
hostname equals the name the provider gave it would therefore be renamed at
boot if the server were later renamed at the provider, and Kubernetes refuses a
node whose name has changed. install.sh closes this by writing
/etc/cloud/cloud.cfg.d/99-bke-preserve-hostname.cfg, which sets cloud-init’s
preserve_hostname: true, on every node where cloud-init is installed. It is the
one change BKE makes to a node’s cloud-init configuration. Leave the file in
place. Upgrading says how to add it to a node installed before 2.6.0.
What BKE reaches over the network
Every node, control-plane and worker alike, needs outbound HTTPS to all of these. There is no offline installation, and no bundle that removes the requirement.
| Host | What comes from it |
|---|---|
bke.maml.uk |
the scripts themselves, and the version graph |
mirrors.bke.maml.uk |
Calico manifests, Helm, the containerd package, calicoctl and k9s |
charts.bke.maml.uk |
the Helm chart for every component |
api.maml.uk |
your licence, and the plan and values for your cluster |
pkgs.k8s.io |
kubelet, kubeadm, kubectl |
prod-cdn.packages.k8s.io |
where pkgs.k8s.io redirects the package downloads |
1x.ax |
every container image |
| your Debian or Ubuntu mirror | base packages |
The two k8s.io names travel together: the Kubernetes project’s repository
answers the package request with a redirect to its content delivery network,
and apt follows it. A rule that allows only pkgs.k8s.io passes every
check and then fails at the download step of an install or upgrade.
Every container image BKE installs comes from 1x.ax — the Kubernetes
control plane, all seven components, and the pod sandbox image the container
runtime pulls on its own account. They are copies of the upstream images, served
from one host so that the rule you write does not change when a component does.
This is why the list above is short. Pulled from their own publishers, those images would need four registries and, because a registry answers a pull from more than one host, five further names for token services and content delivery networks — one of which is chosen by where the node is, so it cannot be written down in advance.
Your own workloads are unaffected. Nothing here changes how an image you
deploy is resolved: a Pod that asks for nginx:latest still pulls it from
Docker Hub. Whatever registries your own applications use remain your firewall
rule to write, exactly as they were.
github.com, get.helm.sh and download.docker.com are no longer reached at
all. Helm, the containerd package, calicoctl and k9s are served from
mirrors.bke.maml.uk alongside the Calico manifests. Every one of them is a
version a BKE release pins, and a pin whose bytes a third party can move or
withdraw is not a pin — mirroring them is what makes the pin true, and it
shortens the list above at the same time.
k9s and calicoctl remain operator conveniences rather than part of the cluster: if the mirror is unreachable when they are fetched, provisioning finishes and prints the command to install them later.
If your estate reaches the internet through a proxy, these are the hosts to allow
before you start rather than after the first failure. check.sh probes every one
of them and names any that did not answer.
What BKE opens on every node
install.sh configures UFW on every node it provisions. Two kinds of port are
opened:
Open to any address:
| Port | For |
|---|---|
| 6443 | the Kubernetes API server, on control-plane nodes |
| 22 | SSH, or whichever ports sshd is configured for and listening on |
Open only to the cluster’s own nodes:
| Port | For |
|---|---|
| 2379, 2380 | etcd (client and peer), on control-plane nodes |
| 10250 | the kubelet |
| 179 | BGP, for Calico |
| 4789 | VXLAN |
| 5473 | Calico Typha |
| 51820, 51821 | WireGuard, for encrypted pod traffic |
The second group is admitted from the cluster’s node addresses. A
control-plane node reads them from the cluster every 15 seconds, so a node that
joins is admitted within that time; each worker reads the same list from a
ConfigMap the cluster keeps current. To fix the sources instead, for example to
a private network, set network.nodeCidrs in config.yaml; config.yaml
describes it. The node addresses stay inside your cluster and are never sent to
us.
Kong’s ports, 32080 and 32443, are not in this list. They need no UFW rule,
and UFW cannot restrict them; Reference architecture explains why, and where to
decide what may reach them.
BKE does not change your SSH configuration
/etc/ssh/sshd_config is yours. install.sh does not rewrite it, does not
move the port, does not change your authentication policy, and does not restart
sshd. Nothing about running Kubernetes requires any of that.
UFW opens every port sshd is configured for, as sshd -T reports it, which
includes a port set in a file under /etc/ssh/sshd_config.d/, and every port
sshd is listening on when install.sh runs. The ports are read rather than
assumed, so enabling the firewall does not lock you out of a machine you are
connected to.
If you would rather BKE owned it, install.sh --manage-ssh installs our
configuration: password authentication disabled, root permitted to log in by key
only, and --ssh-port to choose the port. It applies on that node and on that
run, and it replaces the file rather than merging into it. Before using it,
confirm that key-based login already works and that the port you choose is open
on anything in front of the machine.
Opening them in UFW is not the same as them being reachable. If the machines sit behind a cloud security group, a hardware firewall or separate subnets, those also have to allow this traffic, and nothing in BKE can see that they do. This is the single most common cause of a first node that comes up fine and a second node that will not join.
Three decisions you cannot change afterwards
These are arguments to install.sh, and all three are baked into the cluster the
moment the first control-plane node comes up. Changing any of them later is a
cluster rebuild, not an edit.
The node hostname becomes the machine’s hostname and its --node-name in
Kubernetes, and it appears in the node’s certificates.
The control-plane endpoint — host:port, no https:// — is the stable
address of the API server. It goes into every kubeconfig ever generated for this
cluster, and it is the address your licence is issued against. A licence covers
one cluster, and this is how we know which. Reference architecture describes
what this address should point at in production — a load balancer across the
control-plane nodes — and is worth reading before you choose it.
The service DNS suffix is what your services resolve under, as
<service>.<namespace>.svc.<this>. cluster.local is the Kubernetes convention.
There is no default: the installer requires you to state it.
Pick a naming convention now rather than for the first cluster only. A shape that
works well is <role>.<cluster>.k8s.<your-domain> for node hostnames —
m1.<cluster> for the first control-plane node, w-<city>.<cluster> for
workers — and a separate <cluster>.bke.<your-domain> for the endpoint, so the
address of the control plane is not tied to any one machine’s name. None of this
is enforced; it is simply the shape that stops the second cluster from being
awkward.
Then check, before you install
check.sh runs on every node, before an install and before an upgrade. It reads
only — no package is installed, nothing is applied to a cluster, nothing is left
behind. It is safe to run at any time, including in the middle of an upgrade.
curl -sL https://bke.maml.uk/check.sh | sh -s -- --version 2.6.0
It verifies every host above from this machine, checks the disk floors, the clock, swap and the tools, confirms your licence will not refuse later, and — on a control-plane node of an existing cluster — checks that the cluster is fit to be changed.
| Exit | Meaning |
|---|---|
0 |
ready |
1 |
a check failed; the output names which and what to do |
2 |
the preflight could not run at all |
The difference between 1 and 2 is deliberate. A failed check is information
about your node. Exit 2 means the check itself could not reach a conclusion, and
a preflight that cannot run has told you nothing — it must not be mistaken for one
that passed.
Skipped checks are counted and printed for the same reason. If a check does not apply to this machine, you should be able to see that it did not apply rather than assume it passed.
What check.sh does not tell you
It runs on one machine and reports on that machine. It cannot see:
- whether your nodes can reach each other on the ports above — it tests outbound reachability to the internet, not node-to-node connectivity;
- whether your load balancer or DNS points at the endpoint you are about to declare;
- whether the hardware is adequate for what you intend to run, as opposed to adequate to install.
Run it on every node, not only the first. A node that cannot reach what it needs will fail part-way through its own installation.