BKE

Compliance and hardening

The compliance block of config.yaml turns on the controls that security frameworks and regulators commonly ask of a Kubernetes platform. Each control is a separate switch, so a control that does not fit your environment can be turned off without affecting the others.

Every control is on unless you turn it off, except the two network controls. A new cluster is created with each of them, and a cluster that already exists receives each of them when it is upgraded to this version. The host firewall and the NodePort filter are off unless config.yaml turns them on, because closing ports on a node can stop a service you run there outside Kubernetes.

Control Switch When absent
API audit logging compliance.audit.enabled on
Pod Security defaults compliance.podSecurity.enabled on
Control-plane and kubelet hardening compliance.controlPlane.enabled on
Anonymous requests restricted to health checks compliance.anonymousAccess.restricted on
Kuma’s CNI plugin compliance.meshCni.enabled on
Time servers compliance.time.servers not managed
Host firewall compliance.hostFirewall.enabled off
NodePort filter compliance.nodePortFilter.enabled off

The full block, with every key and its default, is on the config.yaml page.

Turning a control off is a supported choice. check.sh then reports it as a warning that names the control your cluster no longer provides, never as a failure.

Nothing on this page sends anything new out of your cluster. The audit log stays on your control-plane nodes until you forward it, and the record of which controls are in force is a ConfigMap inside the cluster.

API audit logging

The API server records the requests made to it in /var/log/kubernetes/audit/audit.log on each control-plane node, one JSON object per line. The directory is mode 0700 and owned by root, and each file in it 0600, so only root on that node can read the log. A tool that ships the log elsewhere runs as root or is given read access by you.

What is logged

BKE’s policy records who did what, and keeps the contents of sensitive objects out of the log. In the table, machine identities are the nodes, every service account and the control plane’s own components. Your workloads call the API as service accounts, so they are machine identities too.

Requests Recorded at
Any request for a Secret, a ConfigMap or a service account token, reads included, by anyone Metadata: who, what, when and the result; never the object
Token reviews and access reviews made by people Metadata
Changes to RBAC objects and to namespaces, by anyone RequestResponse: the request and the response in full
Other changes made by people Request: the request in full
Other changes made by machine identities Metadata
Other reads made by people Metadata
Other reads, and token and access reviews, made by machine identities not recorded

Health checks, metrics, API discovery, leader-election leases and events are not recorded. A read of a Secret by a machine identity is still recorded, at Metadata.

The policy itself is published at https://bke.maml.uk/etc/kubernetes/audit-policy.yaml, and the copy in force is /etc/kubernetes/bke/audit-policy.yaml on each control-plane node.

Rotation, and why the log must be forwarded

Key Default Meaning
audit.maxSizeMB 100 Size at which the file is rotated
audit.maxBackups 10 Rotated files kept on the node
audit.maxAgeDays 30 Days a rotated file is kept, whichever limit is reached first

At the defaults the log uses at most about 1.1 GB on each control-plane node. That is a bound on disk use, not a retention period. A busy cluster can fill it in less than a day, after which the oldest records are deleted. Retention belongs in the system you forward the log to.

Forwarding the log to your SIEM

BKE writes the log and does not forward it, because the destination is yours. The Fluent Bit that BKE installs can forward it with two additions to /etc/bke/values/fluent-bit.yaml, which is your file and which apply.sh keeps across upgrades (see Installing components):

  1. Run Fluent Bit on the control-plane nodes too, where the log is written. Add a toleration for the control-plane taint at the top level of the file:

    tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule

    If your control-plane nodes carry no taint, Fluent Bit already runs on them and this step is not needed.

  2. Add an input for the log at the end of config.inputs:

    config:
  inputs: |
    # ... the inputs already in the file, unchanged ...

    [INPUT]
        Name tail
        Path /var/log/kubernetes/audit/audit.log
        Parser json
        Tag audit.*
        Mem_Buf_Limit 20MB
        Skip_Long_Lines On
        Read_from_Head On
        DB /var/log/flb_audit.db

    DB records how far Fluent Bit has read, so a restarted Fluent Bit Pod continues where it stopped rather than sending the log again, and the input follows the file across the API server’s rotation. Read_from_Head sends what is already in the log the first time.

Then send the audit.* tag where it belongs. An [OUTPUT] with Match * already sends it with everything else; to send it to a different destination, add a second [OUTPUT] section with Match audit.* to components.fluent-bit.settings.output.config in config.yaml, and exclude it from the first with a narrower Match. Logging with Fluent Bit has examples of each destination. Run apply.sh, read the report, then run it with --commit.

Fluent Bit buffers in memory and is not a guaranteed delivery path. If your regulator requires that no audit record is lost, size the rotation so that the node holds the log for longer than the longest outage you expect of the destination, and alert on Fluent Bit’s delivery failures as described in Logging with Fluent Bit.

Retention

The periods below are those commonly cited for each framework. They are offered to help you plan, not as a statement of what applies to you: confirm the requirement with your compliance function or regulator.

Framework Audit log retention commonly required
NCA Essential Cybersecurity Controls (Saudi Arabia) 12 months
SAMA Cyber Resilience Framework (Saudi Arabia) one year
UAE Information Assurance Regulation one year
QCB cyber security requirements (Qatar) 180 days

Using your own policy

To use an audit policy of your own in place of BKE’s, put it on every control-plane node at the same path and name that path:

compliance:
  audit:
    policyFile: /etc/bke/audit-policy.yaml

install.sh and upgrade.sh copy it into /etc/kubernetes/bke/ and stop, having changed nothing, when it is missing on the node they run on. check.sh reports whether BKE’s policy or yours is in force. Write your policy so that Secrets are recorded at Metadata at most: at Request or above, their values are written to the log.

Turning it off

audit.enabled: false removes the API server’s audit settings and BKE’s copy of the policy. Log files already written are left on the node.

Pod Security

Kubernetes’ Pod Security admission compares each Pod with three levels: privileged (no restrictions), baseline (no host access, no privilege escalation, no added capabilities beyond a safe set) and restricted (baseline, plus running as a non-root user with every capability dropped). BKE sets the cluster-wide defaults:

  enforce audit warn
A cluster created at this version baseline restricted restricted
A cluster upgraded to this version privileged restricted restricted

enforce refuses a Pod that does not meet the level. warn lets it run and returns a warning to whoever created it. audit lets it run and records the violation in the audit log. An upgraded cluster refuses nothing it did not refuse before, and tells you what would be refused at restricted.

The defaults apply to every namespace without its own pod-security.kubernetes.io/ labels. A namespace’s own labels take precedence, so a decision you have already made for a namespace is kept.

Namespaces that are exempt

kube-system, tigera-operator, calico-system and calico-apiserver are exempt, because the control plane and the network need host access. So are the namespaces of Longhorn, Kuma, Fluent Bit and netdata while each is enabled, because each needs host access to do its job: Longhorn manages disks, Fluent Bit reads the node’s logs, netdata reads the node’s metrics, and Kuma’s CNI plugin writes the node’s network configuration. metrics-server, cert-manager and Kong meet baseline and are not exempt.

To exempt a namespace of your own, list it:

compliance:
  podSecurity:
    exemptNamespaces: [legacy-agent]

The list is added to BKE’s exemptions and never replaces them. Labelling the namespace pod-security.kubernetes.io/enforce=privileged has the same effect for one namespace and does not need a change to config.yaml.

Reading the warnings

A Pod that does not meet the warn level is created, and the client that created it prints the reason:

Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false
(container "app" must set securityContext.allowPrivilegeEscalation=false), ...

The same violation appears in the audit log as the annotation pod-security.kubernetes.io/audit-violations on the request. check.sh lists, for baseline and for restricted, every namespace whose running Pods would be refused.

Moving a namespace to enforcement

Ask the API server which running Pods a level would refuse, without changing anything:

kubectl label --dry-run=server --overwrite namespace retail \
  pod-security.kubernetes.io/enforce=restricted

Each warning names a Pod and the setting it lacks. When there are none, apply the label without --dry-run. Enforcement applies to Pods created afterwards; a Pod already running is not stopped.

To enforce baseline everywhere on an upgraded cluster, set it once all namespaces meet it:

compliance:
  podSecurity:
    enforce: baseline

audit and warn take the same three levels.

Meshed Pods meet baseline only with Kuma’s CNI plugin. Without it, Kuma adds an init container that needs the NET_ADMIN and NET_RAW capabilities, which baseline forbids. Keep meshCni on wherever baseline or restricted is enforced; check.sh warns when it is off while either is.

Control-plane and kubelet hardening

With controlPlane.enabled, the default:

Where Setting
API server, controller manager, scheduler profiling endpoints off
API server and every kubelet TLS 1.2 or later, and only ECDHE key exchange with AES-GCM or ChaCha20-Poly1305

Without the cipher list, the API server and the kubelet both accept CBC-mode ciphers with SHA-1.

Kubernetes already sets the following on a BKE cluster, and BKE leaves them as they are: anonymous access to the kubelet is off, the kubelet authorises requests through the API server, kubelet certificates rotate, the kubelet’s read-only port is closed, and etcd requires client and peer certificates. check.sh verifies each of them on every run, so that a change in a later Kubernetes release is reported.

With anonymousAccess.restricted, also the default, a request that presents no credential can reach only /livez, /readyz and /healthz. A load balancer’s health check on /readyz keeps working. Everything else, including the API discovery paths, requires a credential.

Time servers

Accurate time is what makes audit records from different nodes comparable. check.sh reports, on each node, whether the clock is synchronised and which servers it uses. It asks the time service that is running, which is chrony, systemd-timesyncd or ntpsec, rather than the kernel’s synchronised flag, which can stay set long after the service has stopped. A node on which none of the three is running fails the check.

BKE does not change a node’s time configuration unless you name servers:

compliance:
  time:
    servers: [ntp1.example.com, ntp2.example.com]

Every node’s time service is then pointed at them:

  • systemd-timesyncd, the time service on Debian and Ubuntu 24.04, through /etc/systemd/timesyncd.conf.d/bke.conf.
  • chrony, the time service on Ubuntu 26.04 and an alternative on the others, through /etc/chrony/sources.d/bke.sources, which the chrony.conf Debian and Ubuntu ship reads. The servers are marked prefer. Any source your own chrony.conf names stays, and check.sh reports the source chrony actually follows. A chrony.conf that does not read /etc/chrony/sources.d is left alone, and install.sh and upgrade.sh say so.

On Ubuntu 26.04, chrony uses Ubuntu’s own servers unless you change it. Ubuntu configures them with NTS, which authenticates the time each server sends, and chrony does not select a server without NTS while an authenticated one is usable. Servers named in compliance.time.servers are given to chrony, but most time servers do not offer NTS, so chrony keeps following Ubuntu’s. BKE does not weaken that protection on your behalf. check.sh warns on such a node, names the source chrony follows, and says why. To have your own servers used, name servers that offer NTS, or change chrony’s configuration on the node yourself, for example by removing /etc/chrony/sources.d/ubuntu-ntp-pools.sources where your own servers are the trusted source.

Each node receives the list as follows:

  • The first control-plane node, when it is installed. It also records the list in /etc/kubernetes/bke/time-servers.
  • A further control-plane node, when it joins, from the copy of /etc/kubernetes/bke it takes from the cluster.
  • A worker, when it joins. A worker holds no licence and no config.yaml, so it reads the list from the cluster with its kubelet’s credential, once apply.sh has recorded it there. A worker that joined before then takes it with upgrade.sh --node-settings --commit.

On a cluster that exists, apply.sh --commit records the list in the cluster, and each node takes it at its next upgrade.sh, or at once with upgrade.sh --node-settings --commit. That command moves no version, needs no licence, and restarts nothing but the time service. Run it on every node after you change the list. check.sh warns on a node whose servers differ from the list the cluster holds, and prints the command.

Remove the list to stop: the file is removed and the node returns to its own configuration.

Before 2.6.0, BKE left a node that keeps time with chrony alone. Such a node takes the servers at its next upgrade.sh, or at once with upgrade.sh --node-settings --commit, once apply.sh has run at 2.6.0.

Kuma’s CNI plugin

With meshCni.enabled, the default, Kuma is installed with its CNI plugin chained after Calico’s. A meshed Pod’s traffic is redirected to its proxy when the node sets up the Pod’s network, rather than by an init container with added capabilities, which is what lets meshed Pods meet baseline. In place of the init container, a meshed Pod carries a validation container that drops every capability and stops the Pod from starting if its traffic is not redirected.

The plugin runs as a DaemonSet in kube-system. Until it is ready on a node, Kuma taints that node so that no Pod is scheduled where a meshed Pod could not be set up. check.sh reports a node left in that state. The switch has effect only when the Kuma component is enabled.

The host firewall and the NodePort filter

Both are described on the config.yaml page, with what each closes and how to admit a service of your own.

The record of what is in force

apply.sh --commit writes the ConfigMap bke-platform in bahriya-system. It records the BKE and Kubernetes versions and, for each control, both what config.yaml asks for and what the cluster is observed to run, read from each control-plane node’s API server:

kubectl -n bahriya-system get configmap bke-platform -o yaml

A value that could not be read is recorded as unknown, never as the configured value. The ClusterRole bke-platform-reader grants get on this ConfigMap and nothing else, for a compliance tool or an auditor’s account to bind to. The time source is not in it, because it can be read only on each node; check.sh reports it there.

Verifying image signatures

Every image BKE installs from 1x.ax is signed with BKE’s image signing key. The public key is published at https://bke.maml.uk/signing/bke-images-2026.pub. To verify an image with cosign:

curl -fsSLO https://bke.maml.uk/signing/bke-images-2026.pub
cosign verify --key bke-images-2026.pub --insecure-ignore-tlog=true \
  1x.ax/bke/library/pause:3.10.2

The signatures are not recorded in a public transparency log, so that they can be verified on a network with no route to the internet; --insecure-ignore-tlog tells cosign not to look for one, and cosign prints a warning saying so. The check that matters is the one it reports: The cosign claims were validated. An image that is not signed with this key fails with no signatures found.

The same key can be given to an admission policy, such as Kyverno’s image verification, to refuse images from 1x.ax that do not carry a valid signature. When the key is replaced, both keys are published while images are signed again with the new one, and the release notes say so.

Changing a control on a cluster that already exists

  1. Edit compliance in /etc/bke/config.yaml.
  2. Run upgrade.sh at the version the cluster already runs: --role first on the control-plane node apply.sh runs on, then --role other on each other node, without and then with --commit. The files a control needs are written on the first node and stored in the cluster, and each other control-plane node takes them from there on its own run.
  3. Run apply.sh, without and then with --commit. It records the change in bke-platform, and upgrades Kuma with the new setting if meshCni changed.

This is the upgrade sequence without moving the version, so there is nothing new to learn. No package changes, so no node needs draining. A control-plane component whose settings did not change is not restarted; the API server on each control-plane node restarts once if its settings changed. upgrade.sh restarts the kubelet on every node, which does not stop the Pods running there. Turning a control off removes what that control added and nothing else.

upgrade.sh refuses, without changing anything, when the control-plane manifests it would write differ from the running ones by anything other than these controls’ own settings, because the difference would be applied too. The refusal names the condition, manifest-drift, and shows the difference. kubeadm-skew means the kubeadm on the node is not the one the cluster runs. Each can be accepted by name once you have read it:

BKE_ACCEPT_COMPLIANCE_RISK="manifest-drift" upgrade.sh ...

The host firewall, the NodePort filter and meshCni need only step 3, because they are cluster-wide objects rather than settings on a node.

The time servers need step 3, then upgrade.sh --node-settings --commit on every node. apply.sh records the list in the cluster, and each node reads it from there, so step 2 run before step 3 gives each node the previous list. No other step is needed for a change to the time servers alone.

What check.sh reports

check.sh has a “Compliance settings” section. For each control it reports whether it is on, and where it is on:

  • audit: the settings on the API server, that the log was written within the last ten minutes, whose policy is in force, and the free disk under /var/log/kubernetes/audit against the rotation settings;
  • Pod Security: the defaults in force, and each namespace whose running Pods would be refused at baseline and at restricted;
  • hardening and anonymous requests: each setting on every control-plane node, including the ones Kubernetes sets itself;
  • Kuma’s CNI plugin: the plugin present in this node’s CNI configuration, and any node Kuma has tainted;
  • time: synchronisation and the servers in use on this node;
  • the host firewall and the NodePort filter: BKE’s policies, the failsafe ports, and IPv6 filtering where the node has an IPv6 address;
  • the record: bke-platform present and current.

A control that is off, or on and not yet in effect on every node, is a WARN naming what is missing. check.sh reports what it finds on the node it runs on and in the cluster; run it on each control-plane node to see each one’s settings.