BKE
This is the documentation for BKE 2.5.1. The latest release is BKE 2.6.0. Read this page for BKE 2.6.0.

The first control-plane node

This is the node that creates the cluster. Everything else joins what it starts. What a control-plane node is, and why this one is called the first, is covered in How a cluster fits together. Every command on this page runs on this one machine, as root.

Run check.sh on it first, and register the licence, before you get here.

Two files must be in place on this node before you run the command below:

File Why this node needs it
/etc/bke/license mode 0600, from Register your cluster
/etc/bke/config.yaml cluster.podNetworkCidr, auth.enabled and encryption in it decide how the cluster is created

install.sh --role master asks the API how to create this cluster before it changes anything on the machine, and it stops if it cannot. It is not reading config.yaml itself and it is not guessing — see Cluster login. A worker needs neither file: it joins a cluster whose shape was decided here.

The command

curl -sL https://bke.maml.uk/install.sh | sh -s -- \
  --role master \
  --version 2.5.1 \
  --hostname m1.prod.k8s.example.com \
  --endpoint prod.bke.example.com:6443 \
  --dns-domain cluster.local

--version is the BKE version, not a Kubernetes version. BKE names its own versions, and each one decides which Kubernetes minor, which Calico, which Helm and which container images it means. The release notes in the console list the published versions and what each one pins.

The three fixed arguments

Flag Format Fixed for the life of the cluster
--hostname bare host or FQDN yes — it is in the node’s certificates
--endpoint host:port, no scheme yes — it is in every kubeconfig, and on your licence
--dns-domain DNS suffix yes — every service name resolves under it

--endpoint takes a host and a port, not a URL. kubeadm wants prod.bke.example.com:6443, and https://prod.bke.example.com:6443 will not work.

Point the endpoint at whatever will still be correct when you have three control-plane nodes — a load balancer, a floating address, a DNS record you control. Pointing it at this machine’s own name works today and becomes the thing you cannot change when you add the second one. Reference architecture gives the load balancer configuration this address points at in production.

What the script does

In order:

  1. Asks the API how this cluster should be created, before anything on the machine changes. A node that cannot reach the API stops here, with nothing installed and nothing reconfigured.
  2. Sets the hostname.
  3. Installs base packages — curl, jq, git, etcd-client, open-iscsi, wireguard, and others.
  4. Enables the kernel modules the cluster needs at boot, including dm_crypt for encrypted Longhorn volumes.
  5. Adds the Kubernetes apt repository and installs kubelet, kubeadm and kubectl, and containerd from the BKE mirror, at the versions BKE 2.5.1 pins. It writes BKE 2.5.1’s containerd configuration to /etc/containerd/config.toml; settings of your own go in a file under /etc/containerd/conf.d/, as described below.
  6. Configures UFW, disables swap and writes BKE’s kernel settings and modules, described below, and reads every setting back. It stops, with kubeadm not yet run, if a setting does not take or if another file in /etc/sysctl.d would replace one at the next boot. It creates the audit log directory, /var/log/kubernetes/audit, readable by root only. It opens 6443 and every port sshd is configured for, including a port set in a file under /etc/ssh/sshd_config.d/, and every port sshd is listening on, and does not change sshd’s configuration. The node-to-node ports are admitted only from network.nodeCidrs or, without it, from the cluster’s node addresses, which this node records for the others before any of them joins.
  7. Unless config.yaml turns Secret encryption off, generates the cluster’s encryption key and writes it to /etc/kubernetes/bke/encryption-config.yaml, so that the cluster’s first Secret is already stored encrypted. It writes /etc/kubernetes/bke/required beside it, the list of files a further control-plane node must hold.
  8. Runs kubeadm init --upload-certs with a configuration file carrying your three arguments, the pod network from cluster.podNetworkCidr in config.yaml, the encryption file when the cluster encrypts and, when auth.enabled is true, your identity provider.
  9. Writes the admin kubeconfig.
  10. Installs Calico via the Tigera operator with its address pool set to the same pod network, waits for it, and enables WireGuard encryption for pod traffic.
  11. Installs k9s.
  12. Records the BKE version at /etc/bke/version.

If you have not read the SSH warning on Before you begin, read it before step 5 happens to you.

Afterwards

Back up /etc/kubernetes/bke/encryption-config.yaml now, and store the copy apart from your etcd snapshots. The file exists unless config.yaml turned Secret encryption off. It holds the key your Secrets are encrypted with: an etcd snapshot cannot be restored without it, and a snapshot stored with it is not protected by it. install.sh ends with a notice saying the same. Encryption at rest explains both halves.

kubectl get nodes
kubectl get pods -A

The node will be Ready once Calico is up. calicoctl node status reports the BGP peerings; with a single node there is nothing to peer with yet, and that is expected.

k9s is also installed — a terminal interface over the same API kubectl talks to, useful for watching pods start and reading logs without composing commands. Run k9s as root and it works immediately: provisioning copied the administrator kubeconfig to ~/.kube/config, which is the file both kubectl and k9s read. On nodes added later that copy is yours to make — Adding nodes carries the command.

/etc/bke/version now records the BKE version this node is on. It is a record of what happened, not a setting — nothing reads it back to decide anything.

Your own containerd settings

/etc/containerd/config.toml belongs to BKE. A BKE upgrade that moves to a new containerd configuration replaces it, keeping the previous file beside it, so an edit made there does not survive the upgrade.

Put a setting of your own, such as a registry mirror, in a file of its own under /etc/containerd/conf.d/. BKE’s configuration reads every *.toml file there, after its own settings, and an upgrade does not touch that directory. Start each file with version = 4. For example, to have containerd read registry hosts from /etc/containerd/certs.d:

version = 4
[plugins.'io.containerd.cri.v1.images'.registry]
  config_path = '/etc/containerd/certs.d'

Then restart containerd with systemctl restart containerd, on each node that carries the file.

A setting in a conf.d file replaces the same setting in BKE’s file. That is how a deliberate change is made, and it is also how one is made by accident: do not repeat a BKE setting there unless you mean to change it. containerd config dump shows the result.

The kernel settings and modules

Every node holds three files of BKE’s, which the script writes on every node before kubeadm runs, and which apply at every boot:

File What it holds
/etc/modules-load.d/bke-kubernetes.conf br_netfilter and iscsi_tcp, loaded at boot
/etc/sysctl.d/90-bke.conf Settings required exactly: IP forwarding, the two net.bridge settings, and the six the kubelet requires (vm.overcommit_memory = 1, vm.panic_on_oom = 0, kernel.panic = 10, kernel.panic_on_oops = 1, kernel.keys.root_maxkeys = 1000000, kernel.keys.root_maxbytes = 25000000)
/etc/sysctl.d/10-bke-minimum.conf vm.max_map_count = 262144, a minimum

The kubelet runs with protectKernelDefaults on, so it does not start on a node whose settings differ from the six it requires. Do not set any setting in 90-bke.conf to another value in a file of your own: a file in /etc/sysctl.d whose name sorts after 90-bke.conf, or /etc/sysctl.conf, would replace BKE’s value at the next boot, and BKE refuses to install or upgrade a node that has one. To run the kubelet without the protection, set kubelet.protectKernelDefaults: false in config.yaml (see config.yaml); BKE_ACCEPT_KERNEL_OVERRIDE then accepts a setting you name, as Upgrading describes.

vm.max_map_count is a minimum. A higher value set in a file whose name sorts after 10-bke-minimum.conf, as some cloud images set, is the value the node holds, and BKE leaves it as it is.

What is not installed yet

No components. metrics-server, cert-manager, Longhorn, Kong, Kuma, Fluent Bit and netdata are installed separately, once, for the whole cluster, after the workers have joined. That is apply.sh, and it needs a config.yaml. It shows you what it would install before it installs anything.

No authentication. The cluster is reachable with the kubeadm administrator kubeconfig and nothing else. Cluster login is a separate, deliberate step.