Adding nodes
Every additional node is two commands. The first runs on an existing control-plane node and prints the second. The second runs on the new node: it installs BKE and joins the node to the cluster. Nothing is copied between the two machines by hand. Whether a node should be a control-plane member or a worker, and why control-plane nodes come in odd numbers, is covered in How a cluster fits together.
Run check.sh on the new node first.
On an existing control-plane node
As root, naming the new node and the BKE version the cluster runs:
curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- \
--add-node worker \
--hostname w1.prod.k8s.example.com \
--version 2.3.0
Without --commit it says what it would do and changes nothing. With
--commit it:
- allows join tokens to read the two things a joining node needs: the cluster’s public discovery information, and the encrypted copy of the files every control-plane node holds. Anonymous requests stay limited to the health checks, as Compliance and hardening describes;
- creates a join token, valid for two hours;
- for a control-plane node, stores those files in the cluster, encrypted, and uploads the certificates kubeadm shares between control-plane nodes;
- prints the command for the new node.
For a control-plane node, use --add-node control-plane, and give the new
node’s address with --address, as the existing nodes will see it:
curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- \
--add-node control-plane \
--hostname m2.prod.k8s.example.com \
--address 10.0.0.12 \
--version 2.3.0
A new control-plane node joins etcd before it appears in the cluster’s list of
nodes, so the existing control-plane nodes cannot yet admit it from that list.
With --commit, they admit --address on etcd’s ports (2379 and 2380) until
the join token expires, and the command is printed only once every one of them
has. For a node with an IPv4 and an IPv6 address, give both, separated by a
comma. With network.nodeCidrs set, the address must be inside it, and nothing
further is admitted.
--hostname is fixed for the life of the node: it is in the node’s
certificates.
On the new node
Run the command --add-node --commit printed, as root, within two hours. It
looks like this:
BKE_JOIN='...' \
sh -c 'curl -sL https://bke.maml.uk/install.sh | sh -s -- --role worker --version 2.3.0 --hostname w1.prod.k8s.example.com'
BKE_JOIN carries a join token. It is passed in the environment rather
than as an argument, because an argument is readable by every local user. Do
not store it, and do not paste it anywhere but the new node.
Before it changes anything on the machine, install.sh checks BKE_JOIN,
connects to the cluster, verifies the cluster against the CA certificate
BKE_JOIN carries, and confirms that the token is accepted. A control-plane
node also confirms that it can reach etcd on every existing control-plane node;
if it cannot, the address given to --address is not the one the existing
nodes see, and --add-node must be run again with the right one. If any of
this fails it stops, with nothing changed. A token that has expired is refused
by name: run --add-node again for a new one.
It then provisions the node exactly as the first node was provisioned (packages, kernel modules, containerd, UFW, swap off, sysctls) and joins it.
A control-plane node is joined in two steps. kubeadm first downloads the
certificates the control-plane nodes share, which include the cluster CA’s
private key. install.sh then decrypts the files every control-plane node
holds, which can include the Secret encryption key, and places them in
/etc/kubernetes/bke. Only then does the rest of the join start the node’s
API server, which reads them. The node also receives the time servers, and the
administrator kubeconfig is set up for kubectl and k9s.
A worker is joined in one step. It holds no licence and none of those
files. It takes the time servers config.yaml names from the cluster, once
apply.sh has recorded them.
How the files reach a control-plane node
The files every control-plane node holds are stored in the cluster as one
Secret, kube-system/bke-control-plane-files, encrypted with AES-256 and
authenticated with HMAC-SHA256, under keys derived from the cluster CA’s
private key. Every control-plane node holds that key, and nothing else does: a
cluster administrator reading the Secret sees only ciphertext. A file that
does not authenticate under this cluster’s key is refused, and nothing in it is
written.
The Secret also holds the licence and config.yaml of the node apply.sh
runs on, so that any control-plane node can run the operations that need them.
A control-plane node is given them only where it has none of its own. A copy
you put on a node yourself is never replaced.
The same Secret is how the other control-plane nodes receive a change. When
upgrade.sh or a Secret encryption step changes these files on one node, the
next node to run that step takes them from the cluster.
If your cluster’s CA private key is not on its nodes, because you created
the cluster with an external CA, BKE cannot store the files and says so. Copy
/etc/kubernetes/bke from an existing control-plane node to the new one before
running install.sh --role control-plane on it:
tar -C /etc/kubernetes -cpf - bke | ssh root@<new-node> tar -C /etc/kubernetes -xpf -
Control-plane nodes come in odd numbers
etcd needs a majority to accept writes, so three members tolerate one failure and four still tolerate only one. Two is worse than one: either failure takes the cluster down.
Your --endpoint must already resolve to something that can reach all of them
before you add the second one. If it points at the first node’s own address, fix
that first: it cannot be changed afterwards.
Check
On any control-plane node:
kubectl get nodes -o wide
Every node should reach Ready. A node stuck in NotReady is almost always one
of two things, in this order:
- Node-to-node ports are blocked. UFW admits them from the cluster’s node
addresses within about 15 seconds of the node registering, or from
network.nodeCidrswhen it is set; a node outsidenetwork.nodeCidrsis not admitted./var/lib/bke/node-ruleson each node records the outcome of the last read. Something BKE cannot see, such as a cloud security group or a firewall between subnets, may also block the traffic. - Calico has not finished on the new node.
kubectl get pods -n calico-system -o wideshows whether its pods are running there.
Workers hold no administrator credential, and that is correct: a worker holds
only its own kubelet’s identity. Run kubectl and k9s from a control-plane
node; Cluster login covers access for people who do not hold root on one.
Labelling
BKE does not label or taint nodes for you. Roles beyond control-plane, such as
storage nodes and ingress nodes, are your convention, applied with kubectl label.
One case is worth knowing before you install components: Longhorn will use every schedulable node for replicas unless you tell it otherwise. If you intend some nodes to carry storage and others not, decide that before Longhorn is installed rather than after it has placed replicas.