This is the documentation for BKE 2.3.0. The latest release is BKE 2.6.0. Read this page for BKE 2.6.0.

Encryption at rest

Kubernetes stores every Secret in etcd. Unless it is told otherwise, it stores them as written: anyone who can read etcd’s data, or a copy of it, can read every Secret in the cluster.

A cluster created with BKE 2.3.0 encrypts its Secrets before they are written to etcd. It is on unless config.yaml turns it off. A cluster created with an earlier version does not encrypt until you turn it on with upgrade.sh, as described below.

The key is generated on your first control-plane node and is never sent to us. It is a file on each control-plane node.

What it protects, and what it does not

It protects what is stored. Secrets are encrypted in etcd’s database on each control-plane node’s disk, and therefore in every etcd snapshot taken of it. A disk, a snapshot or a backup that leaves your control over it does not carry your Secrets with it, unless the key goes too.

It does not change who can read a Secret through the API. The API server decrypts a Secret for anyone permitted to read it. Your RBAC decides that, as before. A person or a workload with get secrets reads plain values.

It does not protect against root on a control-plane node. The key is on that node, readable by root, because the API server must read it. Whoever holds root there can read the key, and with it everything the key protects.

It does not encrypt Secrets once they reach a pod. A Secret given to a container as an environment variable or a mounted file is plain to that container and on the node that runs it.

It encrypts Secrets, and ConfigMaps if you ask for them. Nothing else stored in etcd is encrypted.

The key

The key is held in /etc/kubernetes/bke/encryption-config.yaml on every control-plane node, mode 0600, owned by root. The API server on each control-plane node reads it. Every control-plane node holds the same file.

Do not edit the file. BKE writes it and recognises it. upgrade.sh refuses to work on a file that differs from the one it wrote, and check.sh reports one.

Back it up, apart from your etcd snapshots

Back up the file as soon as the cluster is created, and store the copy somewhere other than where your etcd snapshots are stored. install.sh says so in a notice at the end of its output, the last thing on the screen.

The two halves of that instruction protect against opposite failures:

• Without the key, a snapshot cannot be restored. The Secrets in it are encrypted, and no other key decrypts them. If every control-plane node is lost, the key is lost with them unless you hold a copy.
• A snapshot stored with its key is not protected by it. Anyone who can read both can read every Secret in the snapshot.

BKE cannot check that you hold a copy, so on an existing cluster upgrade.sh asks you to confirm it before anything is written encrypted, or, where the same run generates the key, to confirm that you will copy it when the run ends (see Conditions you accept by name below).

When the cluster is created

The first control-plane node decides, from encryption in config.yaml, whether the cluster encrypts. With encryption on, which is the default, install.sh --role master generates the key and writes the file before kubeadm init, so the cluster’s first Secret is already written encrypted.

If /etc/kubernetes/bke/encryption-config.yaml already exists on that machine, install.sh stops without changing anything. BKE never replaces a key, because a Secret written with one key can be read with no other.

Joining a control-plane node

kubeadm join copies the cluster’s certificates to a new control-plane node and nothing else. It does not copy the encryption file, and a control-plane node without it has an API server that does not start.

BKE gives the new node the file. A node added with upgrade.sh --add-node control-plane takes /etc/kubernetes/bke, the key included, from the cluster as it joins, before its API server starts. The directory is stored in the cluster encrypted under keys derived from the cluster CA’s private key, which only control-plane nodes hold. Adding nodes describes the join and how the files are stored.

The directory holds a list, required, of the files this cluster’s API servers read, and the join checks the node against it: it stops before the API server starts if a file the list names is missing, or if the key is not mode 0600 and owned by root.

If your cluster’s CA private key is not on its nodes, copy the directory by hand instead, as Adding nodes describes, over a connection you would trust with the key.

ConfigMaps

encryption:
  secrets: true
  configMaps: false

encryption.configMaps adds ConfigMaps to what is encrypted. It is off by default, and it can be on only while secrets is on. Turn it on if your applications keep anything sensitive in ConfigMaps; leave it off otherwise. config.yaml gives both keys.

Turning it on for a cluster that already exists

Run this on a control-plane node, in a maintenance window:

curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --enable-secret-encryption

Nothing changes without --commit. Without it, the command performs every check, says which run comes next and what --commit would do on this node, and stops. Run it first.

It needs /etc/bke/config.yaml and your licence, as apply.sh does. A control-plane node that holds neither takes them from the cluster, where the node apply.sh runs on has stored them (Adding nodes describes how). It encrypts what encryption in config.yaml asks for: Secrets if the block is absent. It takes no other option but --pass and --commit, and moves no version.

Each run restarts the API server on the node it runs on. Running workloads are not restarted. On a cluster with one control-plane node, the cluster’s API is unavailable until that API server is back; on a cluster with more, the others continue to answer.

One control-plane node: a single run

BKE_ACCEPT_ENCRYPTION_RISK="key-backup" \
  sh -c 'curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --enable-secret-encryption --commit'

It generates the key, writes the file, restarts the API server, confirms that the API server has loaded the file, and then stores every existing Secret again so that each is encrypted.

Copy the key as soon as the run completes. The key does not exist until this run generates it, so key-backup here is your undertaking to copy /etc/kubernetes/bke/encryption-config.yaml when the run ends, before your next etcd snapshot. Snapshots taken before the run hold the Secrets unencrypted and are restored without it; every snapshot taken after it needs the key.

More than one control-plane node: three passes

Every API server must be able to read what any other writes. So encryption is turned on in three passes, and each pass is completed on every control-plane node, one node at a time, before the next pass begins:

Pass	Where	What it does
1	every control-plane node	The API server can read encrypted Secrets, and still writes them unencrypted
2	every control-plane node	The API server writes Secrets encrypted
3	once, on any control-plane node	Every existing Secret is stored again, so that each is encrypted

curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --enable-secret-encryption --pass 1 --commit

Pass 1 on the first node generates the key, and stores it in the cluster. Each other control-plane node takes it from there, with the licence and config.yaml where it holds none of its own, at the start of its own pass 1. Nothing is copied by hand.

Pass 2 asks you to confirm that you hold a copy of the key, because from that pass on Secrets are written encrypted:

BKE_ACCEPT_ENCRYPTION_RISK="key-backup" \
  sh -c 'curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --enable-secret-encryption --pass 2 --commit'

You do not need to keep track of the passes. Without --pass, the command says which pass comes next. It works this out by asking each API server which configuration it has loaded, and it refuses a pass that would come too early, naming the nodes that have not completed the one before. A pass that has already completed on a node can be run there again safely.

What it checks, and what it leaves behind

Before it changes anything, it:

• checks that the administrator kubeconfig authenticates against the cluster;
• checks that every API server is running a configuration this node’s file produces, and stops if one holds another key;
• on the node’s first run, asks us for your cluster’s own kubeadm configuration with the encryption setting added to it, renders the API server configuration kubeadm would write, and compares it with the one the node is running. If anything other than encryption differs, it prints the difference and stops.

It records the setting in your cluster’s own kubeadm configuration, so that a later control-plane upgrade keeps it.

A copy of the API server configuration as it was is left at /etc/bke/kube-apiserver.yaml.before-secret-encryption, and a copy of the encryption file before a run that changes it at /etc/bke/encryption-config.yaml.before-secret-encryption.

Storing Secrets again does not change their contents. If a Secret changes while it is being stored again, that Secret is reported as a conflict; it has already been stored, encrypted, by the change. Run the same pass again.

Conditions you accept by name

Each is named, and each is accepted by name in BKE_ACCEPT_ENCRYPTION_RISK. There is no blanket yes.

Name	What it means
key-backup	We cannot check that you hold a copy of the key apart from your etcd snapshots. Confirm that you do, or, on a single control-plane node where the run itself generates the key, that you will copy it as soon as the run completes. Asked on the run that starts writing Secrets encrypted
manifest-drift	Regenerating the API server would change more than encryption, usually because its configuration has been edited by hand since the cluster was created. The difference is printed. Read it before accepting it
kubeadm-skew	The kubeadm on the node is from another Kubernetes minor than the cluster is running, and would write that release’s API server configuration. Complete the upgrade first rather than accepting this

Set the variable before sh, as shown, and not before curl: set before curl, it never reaches the script.

Adding ConfigMaps later, or stopping

To add ConfigMaps to a cluster that already encrypts Secrets, set encryption.configMaps: true and run --enable-secret-encryption again, in the same passes.

To stop encrypting ConfigMaps, set it back to false and run it again. ConfigMaps written from then on are stored unencrypted, and those stored encrypted remain readable.

BKE does not turn Secret encryption off. With encryption.secrets: false on a cluster whose Secrets are encrypted, --enable-secret-encryption refuses.

Rotating the key

Replace the key with a new one if you believe the old one has been exposed, or on whatever schedule your security policy sets:

curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --rotate-encryption-key

As with enabling, nothing changes without --commit, the command says which run comes next, and it takes no other option but --pass and --commit. It needs your licence on the node, and does not need config.yaml. Nothing is sent to us.

On one control-plane node it is a single run: the new key is generated and made the one that encrypts, every Secret is stored again under it, etcd is read to confirm that nothing remains under the old key, and the old key is removed.

BKE_ACCEPT_ENCRYPTION_RISK="key-backup" \
  sh -c 'curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --rotate-encryption-key --commit'

On more than one control-plane node it is four passes, each completed on every control-plane node, one at a time, before the next begins:

Pass	Where	What it does
1	every control-plane node	Adds the new key. The old key still encrypts; both decrypt
2	every control-plane node	The new key encrypts. Asks you to confirm key-backup
3	once, on any control-plane node	Every Secret is stored again under the new key
4	every control-plane node	Removes the old key

Pass 1 on the first node generates the new key. Each other control-plane node takes it from the cluster on its own pass 1, as when enabling.

Pass 4 reads etcd before it removes anything, and refuses while etcd still holds anything under the old key. Removing a key that something is still stored under would leave that thing unreadable.

Pass 3 restarts nothing. Each other pass restarts the API server on the node it runs on.

Back up the new key as soon as pass 1 has run on the first node, apart from your etcd snapshots, as for the first key.

The old key is kept

After the old key is removed, a copy of the file that held it is kept on each control-plane node at /etc/bke/encryption-config.yaml.before-key-rotation, until the next rotation. An etcd snapshot taken before the rotation can be restored only with the old key, so keep a copy of that file with the snapshots it can read, and apart from them, for as long as you keep those snapshots.

The plaintext files in /etc/bke/secrets.d/

The credentials BKE creates Secrets from are files on the node where you run apply.sh (see Secrets). Encryption at rest protects the Secrets in etcd. It does nothing for those files, which stay on the node, unencrypted, until you remove them. check.sh warns about each one it finds on an encrypting cluster.

BKE does not delete them for you. apply.sh creates each Secret from its directory on every run, and refuses if the directory is missing. To remove a directory, hand the Secret over to the cluster first:

1. Run apply.sh --commit once, so that the Secret is in the cluster.
2. Add its name to secrets.unmanaged in config.yaml. Leave it in the component’s secrets: list as well.
3. Delete /etc/bke/secrets.d/<name>.

From then on apply.sh checks that the Secret exists and leaves it alone.

Changing such a credential is then yours to do in the cluster, for example with kubectl edit secret or by applying a new Secret of the same name in the component’s namespace. BKE does not restart anything when a Secret it does not manage changes, so restart the workloads that use it yourself, for example with kubectl rollout restart. To hand the Secret back to BKE, remove its name from secrets.unmanaged and recreate the directory.

What check.sh reports

On a control-plane node, check.sh reports, for this node and the cluster:

• whether the cluster encrypts, and a warning if it does not;
• a failure if the cluster’s API servers are started with the encryption file and this node does not hold it: its API server will not start after the next control-plane upgrade or restart;
• the file’s mode and owner, and whether BKE wrote it;
• whether every API server runs the same configuration, and which nodes are part-way through enabling or rotation;
• how many Secrets are stored encrypted, read from the etcd member on that node. Secrets stored unencrypted, or under the previous key during a rotation, are a warning. Secrets under a key this node does not hold are a failure.

It never prints the key.