This is the documentation for BKE 2.4.0. The latest release is BKE 2.6.0. Read this page for BKE 2.6.0.

Compliance and hardening

The compliance block of config.yaml turns on the controls that security frameworks and regulators commonly ask of a Kubernetes platform. Each control is a separate switch, so a control that does not fit your environment can be turned off without affecting the others.

Every control is on unless you turn it off, except the two network controls. A new cluster is created with each of them, and a cluster that already exists receives each of them when it is upgraded to this version. The host firewall and the NodePort filter are off unless config.yaml turns them on, because closing ports on a node can stop a service you run there outside Kubernetes.

Control	Switch	When absent
API audit logging	`compliance.audit.enabled`	on
Pod Security defaults	`compliance.podSecurity.enabled`	on
Control-plane and kubelet hardening	`compliance.controlPlane.enabled`	on
Anonymous requests restricted to health checks	`compliance.anonymousAccess.restricted`	on
Kuma’s CNI plugin	`compliance.meshCni.enabled`	on
Time servers	`compliance.time.servers`	not managed
Host firewall	`compliance.hostFirewall.enabled`	off
NodePort filter	`compliance.nodePortFilter.enabled`	off

The full block, with every key and its default, is on the config.yaml page.

Turning a control off is a supported choice. check.sh then reports it as a warning that names the control your cluster no longer provides, never as a failure.

Nothing on this page sends anything new out of your cluster. The audit log stays on your control-plane nodes until you forward it, and the record of which controls are in force is a ConfigMap inside the cluster.

API audit logging

The API server records the requests made to it in /var/log/kubernetes/audit/audit.log on each control-plane node, one JSON object per line.

What is logged

BKE’s policy records who did what, and keeps the contents of sensitive objects out of the log. In the table, machine identities are the nodes, every service account and the control plane’s own components. Your workloads call the API as service accounts, so they are machine identities too.

Requests	Recorded at
Any request for a Secret, a ConfigMap or a service account token, reads included, by anyone	`Metadata`: who, what, when and the result; never the object
Token reviews and access reviews made by people	`Metadata`
Changes to RBAC objects and to namespaces, by anyone	`RequestResponse`: the request and the response in full
Other changes made by people	`Request`: the request in full
Other changes made by machine identities	`Metadata`
Other reads made by people	`Metadata`
Other reads, and token and access reviews, made by machine identities	not recorded

Health checks, metrics, API discovery, leader-election leases and events are not recorded. A read of a Secret by a machine identity is still recorded, at Metadata.

The policy itself is published at https://bke.maml.uk/etc/kubernetes/audit-policy.yaml, and the copy in force is /etc/kubernetes/bke/audit-policy.yaml on each control-plane node.

Rotation, and why the log must be forwarded

Key	Default	Meaning
`audit.maxSizeMB`	`100`	Size at which the file is rotated
`audit.maxBackups`	`10`	Rotated files kept on the node
`audit.maxAgeDays`	`30`	Days a rotated file is kept, whichever limit is reached first

At the defaults the log uses at most about 1.1 GB on each control-plane node. That is a bound on disk use, not a retention period. A busy cluster can fill it in less than a day, after which the oldest records are deleted. Retention belongs in the system you forward the log to.

Forwarding the log to your SIEM

BKE writes the log and does not forward it, because the destination is yours. The Fluent Bit that BKE installs can forward it with two additions to /etc/bke/values/fluent-bit.yaml, which is your file and which apply.sh keeps across upgrades (see Installing components):

Run Fluent Bit on the control-plane nodes too, where the log is written. Add a toleration for the control-plane taint at the top level of the file:
```
tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule
```
If your control-plane nodes carry no taint, Fluent Bit already runs on them and this step is not needed.

Add an input for the log at the end of config.inputs:

config:
  inputs: |
    # ... the inputs already in the file, unchanged ...

    [INPUT]
        Name tail
        Path /var/log/kubernetes/audit/audit.log
        Parser json
        Tag audit.*
        Mem_Buf_Limit 20MB
        Skip_Long_Lines On
        Read_from_Head On
        DB /var/log/flb_audit.db

DB records how far Fluent Bit has read, so a restarted Fluent Bit Pod continues where it stopped rather than sending the log again, and the input follows the file across the API server’s rotation. Read_from_Head sends what is already in the log the first time.

Then send the audit.* tag where it belongs. An [OUTPUT] with Match * already sends it with everything else; to send it to a different destination, add a second [OUTPUT] section with Match audit.* to components.fluent-bit.settings.output.config in config.yaml, and exclude it from the first with a narrower Match. Logging with Fluent Bit has examples of each destination. Run apply.sh, read the report, then run it with --commit.

Fluent Bit buffers in memory and is not a guaranteed delivery path. If your regulator requires that no audit record is lost, size the rotation so that the node holds the log for longer than the longest outage you expect of the destination, and alert on Fluent Bit’s delivery failures as described in Logging with Fluent Bit.

Retention

The periods below are those commonly cited for each framework. They are offered to help you plan, not as a statement of what applies to you: confirm the requirement with your compliance function or regulator.

Framework	Audit log retention commonly required
NCA Essential Cybersecurity Controls (Saudi Arabia)	12 months
SAMA Cyber Resilience Framework (Saudi Arabia)	one year
UAE Information Assurance Regulation	one year
QCB cyber security requirements (Qatar)	180 days

Using your own policy

To use an audit policy of your own in place of BKE’s, put it on every control-plane node at the same path and name that path:

compliance:
  audit:
    policyFile: /etc/bke/audit-policy.yaml

install.sh and upgrade.sh copy it into /etc/kubernetes/bke/ and stop, having changed nothing, when it is missing on the node they run on. check.sh reports whether BKE’s policy or yours is in force. Write your policy so that Secrets are recorded at Metadata at most: at Request or above, their values are written to the log.

Turning it off

audit.enabled: false removes the API server’s audit settings and BKE’s copy of the policy. Log files already written are left on the node.

Pod Security

Kubernetes’ Pod Security admission compares each Pod with three levels: privileged (no restrictions), baseline (no host access, no privilege escalation, no added capabilities beyond a safe set) and restricted (baseline, plus running as a non-root user with every capability dropped). BKE sets the cluster-wide defaults:

	`enforce`	`audit`	`warn`
A cluster created at this version	`baseline`	`restricted`	`restricted`
A cluster upgraded to this version	`privileged`	`restricted`	`restricted`

enforce refuses a Pod that does not meet the level. warn lets it run and returns a warning to whoever created it. audit lets it run and records the violation in the audit log. An upgraded cluster refuses nothing it did not refuse before, and tells you what would be refused at restricted.

The defaults apply to every namespace without its own pod-security.kubernetes.io/ labels. A namespace’s own labels take precedence, so a decision you have already made for a namespace is kept.

Namespaces that are exempt

kube-system, tigera-operator, calico-system and calico-apiserver are exempt, because the control plane and the network need host access. So are the namespaces of Longhorn, Kuma, Fluent Bit and netdata while each is enabled, because each needs host access to do its job: Longhorn manages disks, Fluent Bit reads the node’s logs, netdata reads the node’s metrics, and Kuma’s CNI plugin writes the node’s network configuration. metrics-server, cert-manager and Kong meet baseline and are not exempt.

To exempt a namespace of your own, list it:

compliance:
  podSecurity:
    exemptNamespaces: [legacy-agent]

The list is added to BKE’s exemptions and never replaces them. Labelling the namespace pod-security.kubernetes.io/enforce=privileged has the same effect for one namespace and does not need a change to config.yaml.

Reading the warnings

A Pod that does not meet the warn level is created, and the client that created it prints the reason:

Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false
(container "app" must set securityContext.allowPrivilegeEscalation=false), ...

The same violation appears in the audit log as the annotation pod-security.kubernetes.io/audit-violations on the request. check.sh lists, for baseline and for restricted, every namespace whose running Pods would be refused.

Moving a namespace to enforcement

Ask the API server which running Pods a level would refuse, without changing anything:

kubectl label --dry-run=server --overwrite namespace retail \
  pod-security.kubernetes.io/enforce=restricted

Each warning names a Pod and the setting it lacks. When there are none, apply the label without --dry-run. Enforcement applies to Pods created afterwards; a Pod already running is not stopped.

To enforce baseline everywhere on an upgraded cluster, set it once all namespaces meet it:

compliance:
  podSecurity:
    enforce: baseline

audit and warn take the same three levels.

Meshed Pods meet baseline only with Kuma’s CNI plugin. Without it, Kuma adds an init container that needs the NET_ADMIN and NET_RAW capabilities, which baseline forbids. Keep meshCni on wherever baseline or restricted is enforced; check.sh warns when it is off while either is.

Control-plane and kubelet hardening

With controlPlane.enabled, the default:

Where	Setting
API server, controller manager, scheduler	profiling endpoints off
API server and every kubelet	TLS 1.2 or later, and only ECDHE key exchange with AES-GCM or ChaCha20-Poly1305

Without the cipher list, the API server and the kubelet both accept CBC-mode ciphers with SHA-1.

Kubernetes already sets the following on a BKE cluster, and BKE leaves them as they are: anonymous access to the kubelet is off, the kubelet authorises requests through the API server, kubelet certificates rotate, the kubelet’s read-only port is closed, and etcd requires client and peer certificates. check.sh verifies each of them on every run, so that a change in a later Kubernetes release is reported.

With anonymousAccess.restricted, also the default, a request that presents no credential can reach only /livez, /readyz and /healthz. A load balancer’s health check on /readyz keeps working. Everything else, including the API discovery paths, requires a credential.

Time servers

Accurate time is what makes audit records from different nodes comparable. check.sh reports, on each node, whether the clock is synchronised and which servers it uses.

BKE does not change a node’s time configuration unless you name servers:

compliance:
  time:
    servers: [ntp1.example.com, ntp2.example.com]

systemd-timesyncd on every node is then pointed at them, through /etc/systemd/timesyncd.conf.d/bke.conf. Each node receives them as follows:

The first control-plane node, when it is installed. It also records the list in /etc/kubernetes/bke/time-servers.
A further control-plane node, when it joins, from the copy of /etc/kubernetes/bke it takes from the cluster.
A worker, when it joins. A worker holds no licence and no config.yaml, so it reads the list from the cluster with its kubelet’s credential, once apply.sh has recorded it there. A worker that joined before then takes it with upgrade.sh --node-settings --commit.

On a cluster that exists, apply.sh --commit records the list in the cluster, and each node takes it at its next upgrade.sh, or at once with upgrade.sh --node-settings --commit. That command moves no version, needs no licence, and restarts nothing but the time service. Run it on every node after you change the list. check.sh warns on a node whose servers differ from the list the cluster holds, and prints the command.

Remove the list to stop: the file is removed and the node returns to its own configuration.

A node that keeps time with chrony is left alone, and install.sh and upgrade.sh say so; point chrony at the servers yourself.

Kuma’s CNI plugin

With meshCni.enabled, the default, Kuma is installed with its CNI plugin chained after Calico’s. A meshed Pod’s traffic is redirected to its proxy when the node sets up the Pod’s network, rather than by an init container with added capabilities, which is what lets meshed Pods meet baseline. In place of the init container, a meshed Pod carries a validation container that drops every capability and stops the Pod from starting if its traffic is not redirected.

The plugin runs as a DaemonSet in kube-system. Until it is ready on a node, Kuma taints that node so that no Pod is scheduled where a meshed Pod could not be set up. check.sh reports a node left in that state. The switch has effect only when the Kuma component is enabled.

The host firewall and the NodePort filter

Both are described on the config.yaml page, with what each closes and how to admit a service of your own.

The record of what is in force

apply.sh --commit writes the ConfigMap bke-platform in bahriya-system. It records the BKE and Kubernetes versions and, for each control, both what config.yaml asks for and what the cluster is observed to run, read from each control-plane node’s API server:

kubectl -n bahriya-system get configmap bke-platform -o yaml

A value that could not be read is recorded as unknown, never as the configured value. The ClusterRole bke-platform-reader grants get on this ConfigMap and nothing else, for a compliance tool or an auditor’s account to bind to. The time source is not in it, because it can be read only on each node; check.sh reports it there.

Verifying image signatures

Every image BKE installs from 1x.ax is signed with BKE’s image signing key. The public key is published at https://bke.maml.uk/signing/bke-images-2026.pub. To verify an image with cosign:

curl -fsSLO https://bke.maml.uk/signing/bke-images-2026.pub
cosign verify --key bke-images-2026.pub --insecure-ignore-tlog=true \
  1x.ax/bke/library/pause:3.10.2

The signatures are not recorded in a public transparency log, so that they can be verified on a network with no route to the internet; --insecure-ignore-tlog tells cosign not to look for one, and cosign prints a warning saying so. The check that matters is the one it reports: The cosign claims were validated. An image that is not signed with this key fails with no signatures found.

The same key can be given to an admission policy, such as Kyverno’s image verification, to refuse images from 1x.ax that do not carry a valid signature. When the key is replaced, both keys are published while images are signed again with the new one, and the release notes say so.

Changing a control on a cluster that already exists

Edit compliance in /etc/bke/config.yaml.
Run upgrade.sh at the version the cluster already runs: --role first on the control-plane node apply.sh runs on, then --role other on each other node, without and then with --commit. The files a control needs are written on the first node and stored in the cluster, and each other control-plane node takes them from there on its own run.
Run apply.sh, without and then with --commit. It records the change in bke-platform, and upgrades Kuma with the new setting if meshCni changed.

This is the upgrade sequence without moving the version, so there is nothing new to learn. No package changes, so no node needs draining. A control-plane component whose settings did not change is not restarted; the API server on each control-plane node restarts once if its settings changed. upgrade.sh restarts the kubelet on every node, which does not stop the Pods running there. Turning a control off removes what that control added and nothing else.

upgrade.sh refuses, without changing anything, when the control-plane manifests it would write differ from the running ones by anything other than these controls’ own settings, because the difference would be applied too. The refusal names the condition, manifest-drift, and shows the difference. kubeadm-skew means the kubeadm on the node is not the one the cluster runs. Each can be accepted by name once you have read it:

BKE_ACCEPT_COMPLIANCE_RISK="manifest-drift" upgrade.sh ...

The host firewall, the NodePort filter and meshCni need only step 3, because they are cluster-wide objects rather than settings on a node.

The time servers need step 3, then upgrade.sh --node-settings --commit on every node. apply.sh records the list in the cluster, and each node reads it from there, so step 2 run before step 3 gives each node the previous list. No other step is needed for a change to the time servers alone.

What `check.sh` reports

check.sh has a “Compliance settings” section. For each control it reports whether it is on, and where it is on:

audit: the settings on the API server, that the log was written within the last ten minutes, whose policy is in force, and the free disk under /var/log/kubernetes/audit against the rotation settings;
Pod Security: the defaults in force, and each namespace whose running Pods would be refused at baseline and at restricted;
hardening and anonymous requests: each setting on every control-plane node, including the ones Kubernetes sets itself;
Kuma’s CNI plugin: the plugin present in this node’s CNI configuration, and any node Kuma has tainted;
time: synchronisation and the servers in use on this node;
the host firewall and the NodePort filter: BKE’s policies, the failsafe ports, and IPv6 filtering where the node has an IPv6 address;
the record: bke-platform present and current.

A control that is off, or on and not yet in effect on every node, is a WARN naming what is missing. check.sh reports what it finds on the node it runs on and in the cluster; run it on each control-plane node to see each one’s settings.