Reference architecture

A cluster serves two kinds of traffic, and each needs an address that outlives any single machine. This page describes the shape we recommend for production, and gives reference configurations to start from.

This page is guidance, not part of BKE. The load balancer is your infrastructure: BKE does not install it, does not manage it, and does not depend on which product you choose. The configurations below are starting points offered to help you — adapt them, and own what results. A BKE support plan covers the cluster BKE installs; the balancer is operated by you or by whoever runs your network.

                       your users
                           |
                     443 / 80 (TCP)
                           v
              +--------------------------+
              |   ingress load balancer  |
              +--------------------------+
                  |        |        |
             :32443   :32443   :32443          (and :32080)
                  v        v        v
              worker-1  worker-2  worker-3      Kong pods
                  |        |        |
                  v        v        v
              your Services and your pods


          administrators, kubectl, nodes
                           |
                      6443 (TCP)
                           v
              +--------------------------+
              | control-plane balancer   |
              +--------------------------+
                  |        |        |
              :6443    :6443    :6443
                  v        v        v
                 cp-1     cp-2     cp-3         API servers

Both balancers work at layer 4 — they forward TCP connections and do not open them. TLS for your applications terminates at Kong, inside the cluster, where cert-manager keeps the certificates current. Terminating at the balancer as well would give you a second place certificates live and expire.

One balancer appliance or pair can serve both roles; they are separated here because their backends differ — control-plane nodes on one side, workers on the other.

Address one: the control-plane endpoint

The --endpoint you give install.sh is the address of the API server, and it is fixed for the life of the cluster. Every kubeconfig carries it, every node joins through it, and it is the address on your licence.

With one control-plane node, the endpoint can be a DNS record pointing at that node. The moment you plan a second control-plane node, the endpoint must be an address that reaches all of them and notices when one is down — which is a load balancer.

Forward TCP 6443 to port 6443 on every control-plane node. Do not terminate TLS; the API server’s own certificate must reach the client.
Health-check GET /readyz over HTTPS on each backend, without verifying the certificate — the balancer is not a client, and the API server’s certificate is not issued to the balancer’s address. 200 means the API server is serving.
Decide this before the cluster exists. The endpoint cannot be changed afterwards, so create the balancer (or at least the DNS name that will point at it) first, and give install.sh that name.

Address two: your applications

Kong receives HTTP and HTTPS for every application you expose. It listens on two fixed ports on every worker node that runs a Kong pod:

Port	Traffic
`32080`	HTTP
`32443`	HTTPS

Forward TCP 443 to 32443, and TCP 80 to 32080, on your worker nodes. Control-plane nodes are not backends: on a cluster BKE creates they do not run application workloads, so no Kong pod runs there.
Health-check with an HTTP request to 32080 and accept any HTTP response, including 404. Kong answers 404 for a request no application has claimed; an answer — any answer — means Kong is alive on that node. This check matters more than usual here: a node without a running Kong pod does not forward this traffic, so the balancer must stop sending to it. Checking the TCP handshake alone is not sufficient.
Point DNS for your applications at this balancer — one record per hostname you expose, or a wildcard.

Two ports you must open yourself

install.sh configures the firewall on each node for cluster traffic, but it does not open Kong’s two ports — which machines may reach your applications is your decision, not a property of the cluster. On each worker node, allow your balancer in:

ufw allow in from 10.0.0.0/24 to any port 32080 proto tcp
ufw allow in from 10.0.0.0/24 to any port 32443 proto tcp

Replace 10.0.0.0/24 with the network your balancer addresses the nodes from. If a cloud security group or a hardware firewall sits between the balancer and the nodes, it needs the same two rules.

Reference: HAProxy

One HAProxy instance serving both roles. Debian’s packaged HAProxy is fine.

# /etc/haproxy/haproxy.cfg

defaults
    mode    tcp
    timeout connect 5s
    timeout client  30m
    timeout server  30m

# --- the control-plane endpoint -------------------------------------
frontend kubernetes_api
    bind :6443
    default_backend control_plane

backend control_plane
    balance roundrobin
    option httpchk GET /readyz
    http-check expect status 200
    default-server check check-ssl verify none inter 5s fall 3 rise 2
    server cp-1 10.0.1.11:6443
    server cp-2 10.0.1.12:6443
    server cp-3 10.0.1.13:6443

# --- your applications, through Kong --------------------------------
frontend apps_https
    bind :443
    default_backend kong_https

frontend apps_http
    bind :80
    default_backend kong_http

backend kong_https
    balance roundrobin
    option httpchk GET /
    http-check expect rstatus (2|3|4)[0-9][0-9]
    default-server check port 32080 inter 5s fall 3 rise 2
    server w-1 10.0.2.21:32443
    server w-2 10.0.2.22:32443
    server w-3 10.0.2.23:32443

backend kong_http
    balance roundrobin
    option httpchk GET /
    http-check expect rstatus (2|3|4)[0-9][0-9]
    default-server check port 32080 inter 5s fall 3 rise 2
    server w-1 10.0.2.21:32080
    server w-2 10.0.2.22:32080
    server w-3 10.0.2.23:32080

The health check for the Kong backends runs against 32080 in plain HTTP even for the HTTPS backend — the point is whether Kong on that node answers, and checking the HTTP port answers it without the balancer needing TLS anywhere.

A single HAProxy is a single point of failure. The standard remedy is a pair of machines running identical configurations with keepalived holding one floating address between them (VRRP): the address moves to the survivor when a machine fails. Both the control-plane endpoint’s DNS and your application DNS point at floating addresses, not at either machine.

Reference: a hardware or cloud balancer

The same shape translates directly. On an F5 BIG-IP, or the L4 load balancer any cloud provides:

Two virtual servers (F5) or two listeners/target groups (cloud): TCP 6443 with the control-plane nodes as the pool, and TCP 443 (plus 80) with the worker nodes as the pool on ports 32443/32080.
Performance (Layer 4) type, no TLS profile on either. The balancer forwards; it does not terminate.
Health monitors as above: HTTPS GET /readyz expecting 200 for the control plane; HTTP GET / on 32080 accepting any HTTP status for Kong.
Cloud clusters: a NodePort is exactly what a cloud load balancer’s target group expects. Register the worker instances on 32443/32080 and let the target-group health check use the same HTTP check.

The client address, stated honestly

A balancer that forwards TCP opens its own connections to the nodes, so the address Kong sees — and logs, and rate-limits on — is the balancer’s, not the caller’s. Decide which of these you want before your first application depends on client addresses:

Accept it. For internal applications, or where the balancer’s own logs record callers, this is often fine, and it needs no configuration.
PROXY protocol. The balancer prefixes each connection with the caller’s real address, and Kong reads it. Enable it on both ends — one end alone breaks all traffic, because Kong then requires the prefix on every connection:
- HAProxy: add send-proxy to each server line in both Kong backends. F5 and cloud balancers have an equivalent option.
- Kong: in /etc/bke/values/kong.yaml, add:
```
env:
  proxy_listen: "0.0.0.0:8000 proxy_protocol, 0.0.0.0:8443 http2 ssl proxy_protocol"
  real_ip_header: proxy_protocol
  trusted_ips: "10.0.0.0/24"
```
  and run apply.sh again. Set trusted_ips to your balancer’s network and nothing wider. It names who is believed about the caller’s address; 0.0.0.0/0 would let anything that can reach the port claim any address, which defeats logging and rate limiting in one stroke.
DNS straight at the workers. Round-robin DNS on the node addresses, ports open to the world, no balancer. The client address arrives intact without any of the above. The cost is that DNS does not notice a dead node, and clients keep trying it until the record changes. Acceptable for evaluation; not what we recommend for production.

The smallest viable shapes

Not every cluster starts at three and three behind a pair of balancers.

Evaluation, one node: no balancers. The endpoint is a DNS name pointing at the machine. Set that DNS name rather than the machine’s own hostname, and a later move to the full shape is a re-point rather than a rebuild.
Small production, three control-plane nodes and two workers: the balancers matter more than extra nodes. A single HAProxy machine in front is still a single point of failure, but an honest one you can see; add keepalived and a second machine when the applications warrant it.