Adding nodes

Every additional node is two commands. The first runs on an existing control-plane node and prints the second. The second runs on the new node: it installs BKE and joins the node to the cluster. Nothing is copied between the two machines by hand. Whether a node should be a control-plane member or a worker, and why control-plane nodes come in odd numbers, is covered in How a cluster fits together.

Run check.sh on the new node first.

On an existing control-plane node

As root, naming the new node and the BKE version the cluster runs:

curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- \
  --add-node worker \
  --hostname w1.prod.k8s.example.com \
  --version 2.6.0

Without --commit it says what it would do and changes nothing. With --commit it:

• allows join tokens to read the two things a joining node needs: the cluster’s public discovery information, and the encrypted copy of the files every control-plane node holds. Anonymous requests stay limited to the health checks, as Compliance and hardening describes;
• creates a join token, valid for two hours;
• for a control-plane node, stores those files in the cluster, encrypted, and uploads the certificates kubeadm shares between control-plane nodes;
• prints the command for the new node.

For a control-plane node, use --add-node control-plane, and give the new node’s address with --address, as the existing nodes will see it:

curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- \
  --add-node control-plane \
  --hostname m2.prod.k8s.example.com \
  --address 10.0.0.12 \
  --version 2.6.0

A new control-plane node joins etcd before it appears in the cluster’s list of nodes, so the existing control-plane nodes cannot yet admit it from that list. With --commit, they admit --address on etcd’s ports (2379 and 2380) until the join token expires, and the command is printed only once every one of them has. For a node with an IPv4 and an IPv6 address, give both, separated by a comma. With network.nodeCidrs set, the address must be inside it, and nothing further is admitted.

--hostname is fixed for the life of the node: it is in the node’s certificates.

On the new node

Run the command --add-node --commit printed, as root, within two hours. It looks like this:

BKE_JOIN='...' \
  sh -c 'curl -sL https://bke.maml.uk/install.sh | sh -s -- --role worker --version 2.6.0 --hostname w1.prod.k8s.example.com'

BKE_JOIN carries a join token. It is passed in the environment rather than as an argument, because an argument is readable by every local user. Do not store it, and do not paste it anywhere but the new node.

Before it changes anything on the machine, install.sh checks BKE_JOIN, connects to the cluster, verifies the cluster against the CA certificate BKE_JOIN carries, and confirms that the token is accepted. A control-plane node also confirms that it can reach etcd on every existing control-plane node; if it cannot, the address given to --address is not the one the existing nodes see, and --add-node must be run again with the right one. If any of this fails it stops, with nothing changed. A token that has expired is refused by name: run --add-node again for a new one.

It then provisions the node exactly as the first node was provisioned (packages, kernel modules, containerd, UFW, swap off, sysctls) and joins it.

A control-plane node is joined in two steps. kubeadm first downloads the certificates the control-plane nodes share, which include the cluster CA’s private key. install.sh then decrypts the files every control-plane node holds, which can include the Secret encryption key, and places them in /etc/kubernetes/bke. Only then does the rest of the join start the node’s API server, which reads them. The node also receives the time servers, and the administrator kubeconfig is set up for kubectl and k9s.

A worker is joined in one step. It holds no licence and none of those files. It takes the time servers config.yaml names from the cluster, once apply.sh has recorded them.

How the files reach a control-plane node

The files every control-plane node holds are stored in the cluster as one Secret, kube-system/bke-control-plane-files, encrypted with AES-256 and authenticated with HMAC-SHA256, under keys derived from the cluster CA’s private key. Every control-plane node holds that key, and nothing else does: a cluster administrator reading the Secret sees only ciphertext. A file that does not authenticate under this cluster’s key is refused, and nothing in it is written.

The Secret also holds the licence and config.yaml of the node apply.sh runs on, so that any control-plane node can run the operations that need them. A control-plane node is given them only where it has none of its own. A copy you put on a node yourself is never replaced.

The same Secret is how the other control-plane nodes receive a change. When upgrade.sh or a Secret encryption step changes these files on one node, the next node to run that step takes them from the cluster.

If your cluster’s CA private key is not on its nodes, because you created the cluster with an external CA, BKE cannot store the files and says so. Copy /etc/kubernetes/bke from an existing control-plane node to the new one before running install.sh --role control-plane on it:

tar -C /etc/kubernetes -cpf - bke | ssh root@<new-node> tar -C /etc/kubernetes -xpf -

Control-plane nodes come in odd numbers

etcd needs a majority to accept writes, so three members tolerate one failure and four still tolerate only one. Two is worse than one: either failure takes the cluster down.

Your --endpoint must already resolve to something that can reach all of them before you add the second one. If it points at the first node’s own address, fix that first: it cannot be changed afterwards.

Check

On any control-plane node:

kubectl get nodes -o wide

Every node should reach Ready. A node stuck in NotReady is almost always one of two things, in this order:

1. Node-to-node ports are blocked. UFW admits them from the cluster’s node addresses within about 15 seconds of the node registering, or from network.nodeCidrs when it is set; a node outside network.nodeCidrs is not admitted. /var/lib/bke/node-rules on each node records the outcome of the last read. Something BKE cannot see, such as a cloud security group or a firewall between subnets, may also block the traffic.
2. Calico has not finished on the new node. kubectl get pods -n calico-system -o wide shows whether its pods are running there.

Workers hold no administrator credential, and that is correct: a worker holds only its own kubelet’s identity. Run kubectl and k9s from a control-plane node; Cluster login covers access for people who do not hold root on one.

Labelling

BKE does not label or taint nodes for you. Roles beyond control-plane, such as storage nodes and ingress nodes, are your convention, applied with kubectl label.

One case is worth knowing before you install components: Longhorn will use every schedulable node for replicas unless you tell it otherwise. If you intend some nodes to carry storage and others not, decide that before Longhorn is installed rather than after it has placed replicas.