config.yaml
One file at /etc/bke/config.yaml describes your cluster: what it is called,
which components you want, and how each is configured.
apply.sh sends it to api.maml.uk, which checks it and returns what to
install. An error in it comes back as a sentence naming what is wrong, before
anything on your cluster is touched.
This file leaves your cluster; /etc/bke/secrets.d/ never does. That is why
credentials live in a separate directory rather than in here.
Two files leave your cluster, and this is one of them. The other is
/etc/bke/values/<component>.yaml — see Installing components. Neither is
stored. secrets.d/ is sent nowhere, ever.
It describes the cluster. It does not say which BKE version you are running —
that is --version on whichever script you are running, because it is what you
are doing today rather than a property of the cluster.
The shape
schemaVersion: 1
cluster:
name: prod
region: lon
podNetworkCidr: 192.168.0.0/16
secrets:
unmanaged:
- longhorn-backup
auth:
enabled: false
encryption:
secrets: true
configMaps: false
components:
metrics-server:
enabled: true
namespace: kube-system
# ...
A complete file to start from
The file below is a full config.yaml for a new cluster, with every component
enabled and each in its conventional namespace. Copy it to
/etc/bke/config.yaml and replace every value written as <...> with your
own. The file is ready when
grep '<' /etc/bke/config.yaml
finds nothing. A placeholder you miss does not install something wrong: a value
config.yaml must set and does not is refused before your cluster is touched.
schemaVersion: 1
cluster:
# The name appears in your logs, your Longhorn backup paths and your netdata
# hostnames. Choose one that will still be recognisable in an alert.
name: <cluster-name>
region: <region>
# The pod network, decided before the first node is installed and fixed
# once the cluster exists. 192.168.0.0/16 is a reasonable choice unless it
# overlaps a network your nodes or your pods must reach. See the table below.
podNetworkCidr: 192.168.0.0/16
auth:
# Read *Cluster login* before setting this to true.
enabled: false
encryption:
# Secrets are encrypted in etcd. Back up the key the first node writes,
# apart from your etcd snapshots - see *Encryption at rest*.
secrets: true
configMaps: false
reporting:
# A daily version report to us. Off is supported and costs nothing - see
# the reporting section on this page.
enabled: true
compliance:
# Each control below is described in *Compliance and hardening*. The values
# shown are the defaults, written out so that the file states them.
audit:
enabled: true
maxAgeDays: 30
maxBackups: 10
maxSizeMB: 100
podSecurity:
enabled: true
audit: restricted
warn: restricted
exemptNamespaces: []
controlPlane:
enabled: true
anonymousAccess:
restricted: true
meshCni:
enabled: true
# Name your time servers here to have every node use them; empty leaves
# each node's own time configuration as it is.
time:
servers: []
# From outside the cluster, each node then accepts only the API server, SSH,
# the NodePort range and ICMP. List every port sshd listens on, on every
# node, before the first apply.sh --commit - see the compliance section on
# this page.
hostFirewall:
enabled: true
sshPorts: [22]
# Limits the NodePort range to your load balancers. Leave it off until you
# list their addresses: on with an empty list is refused.
nodePortFilter:
enabled: false
sources: []
components:
metrics-server:
enabled: true
namespace: kube-system
cert-manager:
enabled: true
namespace: cert-manager
settings:
leaderElectionNamespace: kube-system
longhorn:
enabled: true
namespace: longhorn-system
# BKE creates this Secret from /etc/bke/secrets.d/longhorn-backup/ -
# create that directory with the target's credentials in it first.
# See *Secrets* for the directory's shape and *Storage* for the keys.
# To evaluate without a backup store, set both backup values to ""
# and remove this secrets list; no Secret is then needed.
secrets:
- longhorn-backup
settings:
backup:
target: s3://<bucket>@<region>/<path>
credentialSecret: longhorn-backup
# Replicas per volume: 1 on a single node, 3 in production.
defaultClassReplicaCount: 3
storageMinimalAvailablePercentage: 15
kong:
enabled: true
namespace: kong
kuma:
enabled: true
namespace: kuma-system
fluent-bit:
enabled: true
namespace: logging
settings:
output:
# A complete [OUTPUT] section, verbatim, written flush left within the
# block. *Logging with Fluent Bit* carries worked examples for
# OpenSearch, Loki and S3-compatible stores.
config: |
[OUTPUT]
Name loki
Match *
Host <log-destination-host>
Port 3100
netdata:
enabled: true
namespace: netdata
settings:
cloud:
# As written, this cluster is not claimed into Netdata Cloud and the
# named Secret need not exist. To claim it: set enabled to true, set
# room to your room UUID, put the claim token under
# /etc/bke/secrets.d/netdata-cloud/, and list netdata-cloud under this
# component's secrets - see *Secrets*.
enabled: false
room: ""
secret: netdata-cloud
Every key in it is explained below. Two habits worth keeping from the first
day: a component you do not want is enabled: false rather than deleted, so
the file still names it the day you change your mind; and nothing that is a
credential goes in this file — /etc/bke/secrets.d/ exists for those and
never leaves your cluster.
Top level
| Key | Required | Notes |
|---|---|---|
schemaVersion |
yes | Must be 1 |
cluster.name |
yes | Appears in your logs, your Longhorn backup paths and your netdata hostnames. Choose a name that will still be recognisable in an alert |
cluster.region |
yes | A label you choose. BKE does not interpret it |
cluster.podNetworkCidr |
yes | The network pods take their addresses from. It must be an IPv4 network of /24 or larger (a prefix of 24 or less), written as its network address (10.240.0.0/16, not 10.240.1.0/16), and it must not overlap 10.96.0.0/12, the network Kubernetes services use. Choose it before the first control-plane node is installed: the first node’s installation uses it, and it cannot be changed once the cluster exists. Changing it here afterwards does not change the cluster |
network.nodeCidrs |
no | The networks or addresses the node-to-node ports are admitted from. Without it, each node admits the cluster’s own node addresses. See network below |
secrets.unmanaged
Names Secrets that your cluster already owns and BKE must not create. Use it when a Secret is produced by sealed-secrets, the External Secrets Operator, or by hand.
Every name listed here must also appear in some component’s secrets:
list. That declaration is what tells BKE which component needs the Secret —
and therefore which namespace to look for it in. secrets.unmanaged then
decides who provides it: for a name listed here, BKE checks the Secret exists
and creates nothing; for a declared name not listed here, BKE creates it from
/etc/bke/secrets.d/.
A name here that no component declares is refused — it is almost always a
spelling that no longer matches the secrets: list it was meant to pair with.
A Secret that is only referenced in settings — a credentialSecret value,
netdata’s cloud.secret — is not a declaration, and BKE neither creates nor
checks it.
A declared name that resolves to neither a directory under
/etc/bke/secrets.d/ nor an unmanaged entry is a pre-flight refusal —
before the first chart is fetched, not part-way through. See Secrets.
auth
auth:
enabled: false
oidc:
- issuerUrl: https://login.example.com/v2.0
clientId: bke-prod
usernameClaim: email
groupsClaim: groups
groupsPrefix: "oidc:"
rbac:
clusterAdminGroups: [oidc:platform-admins]
adminGroups: [oidc:team-leads]
editGroups: [oidc:developers]
viewGroups: []
namespaces:
team-a:
adminGroups: [oidc:team-a-leads]
editGroups: [oidc:team-a-developers]
team-b:
viewGroups: [oidc:team-b]
The four lists directly under rbac grant across every namespace there is.
namespaces is how a group is given access to one namespace and no other; the
keys under a namespace are the same three, without clusterAdminGroups. Every
namespace named must already exist, and one that does not is a pre-flight
refusal naming each missing name. See Cluster login, which also explains why
listing a group in both places does not restrict it.
enabled defaults to false, and the rbac groups can be listed while it is
off — the bindings are created and simply bind groups nobody can present yet.
That pairing is deliberate: on an existing cluster, enabling authentication
restarts your API server, and it should be a separate decision from installing
components.
enabled decides how your cluster is created, so on a new cluster it has to
be right before install.sh --role master runs. With it off, the cluster is
created exactly as it always was. usernamePrefix is optional and does for
usernames what groupsPrefix does for groups.
Read Cluster login before setting enabled: true. There is a limitation
there that will decide whether your identity provider can be used at all, and a
break-glass procedure to carry out first rather than afterwards.
encryption
encryption:
secrets: true
configMaps: false
| Key | Default | Meaning |
|---|---|---|
secrets |
true |
Secrets are encrypted before they are stored in etcd |
configMaps |
false |
ConfigMaps are encrypted too. Only with secrets: true; configMaps: true with secrets: false is refused |
The block takes these two keys and nothing else, and each is true or false.
The key itself never appears in this file, and a block that tries to carry
one is refused: the key is generated on your first control-plane node and is not
sent to us.
On a new cluster, secrets decides how the cluster is created, so it has to
be right before install.sh --role master runs. Absent, it is true.
On a cluster that already exists, nothing changes until you run
upgrade.sh --enable-secret-encryption, which encrypts what this block asks
for. That is also how configMaps is turned on or off later. BKE does not turn
Secret encryption off: secrets: false on a cluster whose Secrets are
encrypted is refused by that command, and changes nothing.
Encryption at rest covers what it protects, backing up the key, and rotating it.
reporting
reporting:
enabled: true
enabled defaults to true and is the only key. Once a day, a CronJob named
bke-report in the bahriya-system namespace sends us a short report about
this cluster. It is the one recurring thing that leaves your cluster, which is
why it has a first-class off switch.
Setting enabled: false removes the job, along with its ServiceAccount and
all of its RBAC, on the next apply.sh --commit. It is not merely no longer
installed: a cluster that reported yesterday stops reporting today.
Turning it off is a supported choice, not a degraded one. Nothing expires for silence and nothing is withheld.
Cluster reporting is the whole of it: every field that is sent, what the job’s permissions do and do not allow it to read, how to run it by hand and read the result on your own cluster, and what to check after turning it off.
compliance
The controls that security frameworks and regulators commonly ask of a Kubernetes platform. Compliance and hardening describes what each one does, how to forward the audit log, and how to change a control on a cluster that already exists.
| Key | Default | Meaning |
|---|---|---|
audit.enabled |
true |
The API server writes an audit log on each control-plane node |
audit.maxAgeDays |
30 |
Days a rotated audit log file is kept on the node |
audit.maxBackups |
10 |
Rotated audit log files kept on the node |
audit.maxSizeMB |
100 |
Size at which the audit log is rotated |
audit.policyFile |
absent | An absolute path on every control-plane node, to an audit policy of your own used in place of BKE’s |
podSecurity.enabled |
true |
Pod Security defaults apply to every namespace without its own labels |
podSecurity.enforce |
baseline on a new cluster, privileged on an upgraded one |
The level Pods are refused below |
podSecurity.audit |
restricted |
The level below which a Pod is recorded in the audit log |
podSecurity.warn |
restricted |
The level below which a Pod’s creator is warned |
podSecurity.exemptNamespaces |
empty | Namespaces exempt from Pod Security, added to BKE’s own exemptions |
controlPlane.enabled |
true |
Profiling off, and TLS 1.2 or later with AEAD ciphers only, on the control plane and every kubelet |
anonymousAccess.restricted |
true |
A request without a credential reaches only /livez, /readyz and /healthz |
meshCni.enabled |
true |
Kuma is installed with its CNI plugin, so meshed Pods can meet baseline. Has effect only when Kuma is enabled |
time.servers |
empty | Host names or addresses every node’s systemd-timesyncd uses. Empty leaves each node as it is |
hostFirewall.enabled |
false |
Described below |
hostFirewall.sshPorts |
[22] |
Described below |
nodePortFilter.enabled |
false |
Described below |
nodePortFilter.sources |
empty | Described below |
Every enabled key, and anonymousAccess.restricted, takes true or false.
podSecurity.enforce, audit and warn each take privileged, baseline or
restricted. The block takes these keys and nothing else; a key that is not listed here is refused,
because a misspelt control would otherwise be silently off.
At a BKE version earlier than 2.3.0, these controls do not exist. A control set explicitly to on there is refused, rather than ignored, because the cluster would not have it.
compliance.hostFirewall and compliance.nodePortFilter
compliance:
hostFirewall:
enabled: true
sshPorts: [22]
nodePortFilter:
enabled: false
sources: []
Two switches that close your nodes to traffic from outside the cluster. Calico, which every BKE cluster already runs, enforces both. A node that joins later is covered as soon as it joins, with nothing to run on it.
| Key | Default | Meaning |
|---|---|---|
hostFirewall.enabled |
false |
Each node accepts from outside the cluster only the API server (6443), the SSH ports, the NodePort range (30000-32767) and ICMP, over IPv4 and IPv6 |
hostFirewall.sshPorts |
[22] |
Every port sshd listens on, on every node |
nodePortFilter.enabled |
false |
The NodePort range accepts only sources, the cluster’s nodes and its Pods |
nodePortFilter.sources |
empty | The addresses your load balancers connect to the nodes from, as addresses or CIDRs, IPv4 or IPv6 |
Both are off when this block does not mention them, and the file at the
top of this page turns hostFirewall on. An upgrade alone therefore never
closes a port on a cluster that already exists.
Traffic from inside the cluster is not affected. The nodes reach each other on every port, and Pods reach the nodes on every port, as before. Traffic to a Pod is not affected either.
What the host firewall closes is everything else on the node: etcd, the
kubelet, Calico’s own ports, and any service you run on the node itself outside
Kubernetes. To admit such a service, add a Calico GlobalNetworkPolicy of your
own with an order below 1000, for example:
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: allow-monitoring
spec:
order: 500
selector: "projectcalico.org/created-by == 'calico-kube-controllers'"
types:
- Ingress
ingress:
- action: Allow
protocol: TCP
source:
nets: ["203.0.113.10/32"]
destination:
ports: [9100]
SSH and the API server cannot be closed by a policy. BKE makes the SSH
ports and 6443 Calico failsafe ports, which Calico admits before it consults
any policy, yours or BKE’s. apply.sh refuses to turn the host firewall on
while sshd on the node it runs on listens on a port sshPorts does not list.
It cannot see the other nodes’ sshd, so check them yourself before the first
apply.sh --commit.
Turning the host firewall on for the first time restarts Calico on every
node once, to filter IPv6 as well as IPv4. The report apply.sh prints
without --commit says so. IPv6 filtering then stays on even if you turn the
host firewall off again, because it is what keeps IPv6 as open as IPv4 once the
firewall is gone.
nodePortFilter is for clusters behind load balancers. Without it, every
NodePort answers any address that can reach a node. With it, list the address
each load balancer connects from: the address on the network it reaches your
nodes over, which may be a private one. enabled: true with no source is
refused, because it would close every NodePort to everything outside the
cluster. 0.0.0.0/0 and ::/0 are refused, because they filter nothing.
UFW is set apart from these switches. The node-to-node rules described
under network.nodeCidrs apply whichever way they are set. With the host
firewall on, Calico decides first; with it off, UFW is the only filter on the
node.
Changing either switch needs only apply.sh, without and then with
--commit. Turning a switch off removes what BKE added, and nothing else:
Calico settings and policies of your own are left as they are.
network
network.nodeCidrs
Optional. The networks, or single addresses, the cluster’s nodes are in. UFW admits the node-to-node ports (etcd on control-plane nodes, the kubelet, BGP, VXLAN, Typha and WireGuard) only from these.
Without it, each node admits the cluster’s own node addresses instead: the
internal and external addresses each node reports, and the address Calico uses
for each node. Calico may use a different interface from the one the node
reports, such as a private network, and the nodes’ BGP sessions run from it. A
control-plane node reads the addresses from the cluster every 15 seconds; each
worker reads them from the ConfigMap bke-node-addresses in bahriya-system,
which the job node-ip-sync keeps current every minute. This suits nodes on
scattered public addresses. The addresses stay inside your cluster and are
never sent to us.
WireGuard and VXLAN are admitted by Calico before UFW sees them. Calico accepts WireGuard (UDP 51820) from any address, and VXLAN (UDP 4789) only from the cluster’s nodes, ahead of UFW’s rules, so UFW’s rules for those two ports do not decide them. WireGuard discards every packet that does not come from a node holding one of the cluster’s keys.
With it, the rules are fixed to the networks listed and nothing is read. This suits nodes on one private network:
network:
nodeCidrs: [10.0.0.0/24]
Nodes on two sites, or dual-stack:
network:
nodeCidrs: [10.0.0.0/24, 10.1.0.0/24, 2001:db8:10::/64]
Pinned addresses, one per node:
network:
nodeCidrs: [203.0.113.10, 203.0.113.11, 2001:db8::10]
A bare address is read as that one address (/32, or /128 for IPv6), and
every entry is normalised, so 10.0.0.7/24 is read as 10.0.0.0/24. Where your
provider gives each server an IPv6 /64, list the /64 rather than one
address, so that a change of address within it does not cut the node off. A
prefix of 0 is refused, because it would admit every address.
A node outside the list is not admitted. Add its address or network here,
run apply.sh --commit, and then upgrade.sh --node-settings --commit on each
existing node, before the new node joins.
Who can change the list can change the rules. Without nodeCidrs, whoever
can update bke-node-addresses changes what every worker admits. BKE grants
that right to node-ip-sync alone, by name; do not grant it to anyone else. The
list admits single addresses only, never a network.
Applying it to a node that exists
upgrade.sh --node-settings on each node reports what it would change, and
--commit changes it: the node-to-node rules are admitted from the list, the
rules earlier versions opened to every address are then removed, and the
reader is installed or removed as nodeCidrs requires. check.sh warns about
a node that still has a port open to every address.
components
Each component takes enabled, namespace, optionally releaseName, optionally
secrets, and optionally settings.
components:
longhorn:
enabled: true
namespace: longhorn-system
secrets:
- longhorn-backup
settings:
backup:
target: s3://bke-backups@lon/prod
credentialSecret: longhorn-backup
defaultClassReplicaCount: 3
storageMinimalAvailablePercentage: 15
namespace is required for every enabled component and has no default. That
is not an oversight. Four of the seven components are conventionally installed in
a namespace that is not the component’s own name — kube-system,
longhorn-system, kuma-system, logging — so a default would be wrong more
often than right, and being wrong means quietly installing a second copy
somewhere you were not looking.
releaseName defaults to bke-<component> and exists for clusters that came
before BKE. A new cluster gets bke-longhorn, bke-kong and so on, which is
what makes BKE’s releases obvious in helm list without a selector.
A cluster you already run almost certainly calls them something else — plain
longhorn, plain kong, or a name of your own. Name it here and BKE takes that
release over in place:
components:
kong:
enabled: true
namespace: kong
releaseName: kong
Renaming a Helm release means deleting and recreating it, which on Longhorn
or an ingress gateway is an outage, so BKE never does it. The name is a
convenience; what marks a release as BKE’s is the
app.kubernetes.io/managed-by=bke label, which is neither optional nor
configurable.
This is also why BKE takes over the release you name rather than one it
recognises. A release already called kong that BKE did not install is adopted
only because you asked for Kong in this file — never because something scanned
your cluster and found a familiar name.
Naming a component this BKE version does not ship is refused, not skipped. Asking for something this BKE version does not describe and receiving everything else would leave you believing you had it.
Per-component settings
| Component | Key | Meaning |
|---|---|---|
| cert-manager | leaderElectionNamespace |
Where cert-manager takes its leader-election lease. Conventionally kube-system |
| longhorn | backup.target |
The backup target URI, e.g. s3://bucket@region/path. Set it to "" to run without backups |
| longhorn | backup.credentialSecret |
Name of the Secret holding the target’s credentials. Set it to "" when backup.target is "" |
| longhorn | defaultClassReplicaCount |
Replicas per volume in the default StorageClass, and for a volume created in the Longhorn UI without a count. 1 on a single node; 3 in production. No more than the number of nodes Longhorn can use. A change applies to new volumes |
| longhorn | storageMinimalAvailablePercentage |
Longhorn stops scheduling to a disk below this |
| fluent-bit | output.config |
A complete [OUTPUT] section, verbatim. Write it flush left; it is re-indented for you. Logging with Fluent Bit carries worked examples |
| netdata | cloud.enabled |
Whether to claim this cluster into Netdata Cloud |
| netdata | cloud.room |
The Netdata Cloud room UUID to claim into |
| netdata | cloud.secret |
Name of the Secret holding the claim token |
Running without backups is written down, never inferred. To install
Longhorn with no backup store — reasonable while evaluating, not for a cluster
holding data you cannot recreate — set both backup.target and
backup.credentialSecret to "", leave secrets: out of the longhorn block,
and create nothing under /etc/bke/secrets.d/. Omitting the backup keys
entirely is refused rather than read as this choice, so a forgotten backup
block cannot quietly become a cluster that does not back up.
secrets: and settings: are two different statements. secrets declares
that the component needs a Secret by that name — it is the list BKE works from,
with secrets.unmanaged deciding per name whether BKE creates it from
/etc/bke/secrets.d/ or verifies it and touches nothing.
settings.*.credentialSecret and its like are references inside the
component’s configuration: they tell the component which Secret to use, and BKE
does not read them as declarations.
So for a Secret produced by something else — sealed-secrets, the External
Secrets Operator, your own kubectl — name it in settings, declare it in
secrets, and list it in secrets.unmanaged. BKE then checks it exists before
anything is applied and never creates, changes or deletes it. Declaring it
without the unmanaged entry tells BKE to manage it: its contents become
whatever /etc/bke/secrets.d/<name>/ holds, and every workload referencing it
restarts when that changes — the opposite of what you meant for a Secret
something else owns.
What an error looks like
A sentence in plain text, naming what is wrong and what to do about it.
It arrives before your cluster is touched. Everything in config.yaml is
checked while apply.sh is still gathering what it needs, so a mistake here
costs you a re-run and nothing else. BKE would rather stop than install
something half-described.