Logging with Fluent Bit

Fluent Bit runs on every node and ships logs off the cluster. BKE decides what is collected; you decide where it goes, and until you do, the component has no destination and nothing is shipped. This page is how to make that decision and write it down.

The destination side of that line is yours: the log platform, the output configuration and its credentials are your infrastructure, and the examples on this page are starting points offered to help you rather than part of BKE.

What is collected

Three streams leave every node, each under its own tag:

Tag	Contents
kube.*	every container’s output, with its Kubernetes context attached — namespace, pod, labels — so your platform can filter by application
host.*	the node agent’s own service log — the machine-level view when a node misbehaves
k8s_events.*	cluster events: pods scheduled, images pulled, restarts, failures — stamped with your cluster’s name and region so two clusters sharing one platform stay distinguishable

Naming the destination

The destination is one setting in /etc/bke/config.yaml: a complete [OUTPUT] section in Fluent Bit’s own configuration syntax, written flush left. Whatever the section names is where the logs go, and every capability of the plugin you choose is available — there is no reduced list of supported settings.

components:
  fluent-bit:
    enabled: true
    namespace: logging
    settings:
      output:
        config: |
          [OUTPUT]
              Name  opensearch
              Match *
              Host  logs.example.com
              Port  9200
              tls   On
              HTTP_User  ingest
              HTTP_Passwd ${OPENSEARCH_PASSWORD}
              Logstash_Format On
              Suppress_Type_Name On

Run apply.sh after changing it, as with any config.yaml change. The ${OPENSEARCH_PASSWORD} reference is deliberate and explained below — the one rule on this page is no password in this block.

Grafana Loki

[OUTPUT]
    Name   loki
    Match  *
    Host   loki.example.com
    Port   3100
    tls    On
    labels job=fluent-bit, cluster=prod
    line_format json

An S3-compatible object store

[OUTPUT]
    Name             s3
    Match            *
    bucket           cluster-logs
    region           lon
    endpoint         https://s3.example.com
    total_file_size  50M
    upload_timeout   10m

The s3 plugin reads its credentials from the standard AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables — provide them by the method below.

Another Fluent Bit or Fluentd

[OUTPUT]
    Name   forward
    Match  *
    Host   collector.example.com
    Port   24224

Splitting the streams

Match selects by tag, and the block may contain several [OUTPUT] sections — for example containers to your log platform, cluster events to object storage:

[OUTPUT]
    Name  opensearch
    Match kube.*
    Host  logs.example.com
    Port  9200
    tls   On
    HTTP_User  ingest
    HTTP_Passwd ${OPENSEARCH_PASSWORD}

[OUTPUT]
    Name    s3
    Match   k8s_events.*
    bucket  cluster-events
    region  lon
    endpoint https://s3.example.com

Keeping the credential out of config.yaml

config.yaml is sent to us so your configuration can be checked; /etc/bke/secrets.d/ never leaves the node. A log platform’s password therefore does not belong in the output block — it belongs in a Secret, reaching Fluent Bit as an environment variable, with the block referring to it as ${NAME}. The reference is resolved on your nodes when Fluent Bit starts; the value never appears in config.yaml.

Three steps, using the same Secret machinery as every other component:

1. Declare the Secret in config.yaml and put the value on the node:

   components:
     fluent-bit:
       enabled: true
       namespace: logging
       secrets:
         - fluent-bit-output
   

   install -d -m 0700 /etc/bke/secrets.d/fluent-bit-output
   printf '%s' 'the-password' > /etc/bke/secrets.d/fluent-bit-output/OPENSEARCH_PASSWORD
   chmod 0600 /etc/bke/secrets.d/fluent-bit-output/OPENSEARCH_PASSWORD
   

   The file’s name is the variable’s name. printf rather than echo, so no newline is appended to the password.

2. Hand the Secret to Fluent Bit as environment, in /etc/bke/values/fluent-bit.yaml on the control-plane node where you run apply.sh — the per-component file that is yours to edit (Installing components explains these files, and why changes to them are applied with apply.sh rather than with Helm directly):

   envFrom:
     - secretRef:
         name: fluent-bit-output
   

3. Refer to it in the output block as ${OPENSEARCH_PASSWORD}, as in the examples above, and run apply.sh.

When the destination is down

Logs buffer on each node and shipping resumes when the destination returns; each node’s buffer is capped, so a long outage costs a gap in the logs rather than a full disk. Design for that honestly: this is log shipping, not a guaranteed audit trail.

One behaviour is worth knowing before it pages someone. When the destination stays unreachable, every Fluent Bit pod in the cluster reports itself not ready — the component’s health follows its ability to deliver. A whole fleet of NotReady Fluent Bit pods therefore usually means one remote problem, not a cluster problem: check the destination and the path to it first.

Verifying it works

The component’s own log names delivery failures — the wrong host, a refused certificate, a rejected credential:

kubectl -n logging logs -l app.kubernetes.io/name=fluent-bit --tail=50

Substitute the namespace your config.yaml names. A healthy pod logs little after startup; repeated retry lines name the destination and the error. Then confirm arrival on the other side — a search for the cluster’s own namespaces in your log platform is the test that matters, because it exercises the whole path.

After changing the output block, confirm the setting took by checking the pods rolled: kubectl -n logging get pods — their age should be recent.