Cluster login

Read this page before you plan cluster authentication, not while you are configuring it. One limitation below decides whether your identity provider can be used at all, and one section decides when you do this rather than how.

The two halves

Authorisation. BKE creates four ClusterRoleBindings from auth.rbac.* in your config.yaml, binding your groups to Kubernetes’ built-in roles:

config.yaml	Kubernetes role	The object BKE creates	Grants
clusterAdminGroups	cluster-admin	bke-cluster-admin	everything
adminGroups	admin	bke-admin	full control within a namespace
editGroups	edit	bke-edit	create and modify workloads, not RBAC
viewGroups	view	bke-view	read-only

kubectl describe clusterrolebinding bke-admin shows you exactly which group strings a cluster is currently honouring, which is the first thing to look at when someone logs in successfully and can do nothing.

These are created even when auth.enabled is false, which is deliberate: the bindings can exist harmlessly, binding groups nobody can present yet, so that turning authentication on later is one change rather than two.

Authentication. With auth.enabled: true, BKE configures your API server to trust your identity provider. Your people then log in as themselves, and the bindings above decide what they can do.

Setting auth.enabled is a decision about the API server, not a preference. What it changes, exactly:

	enabled: false	enabled: true
the four ClusterRoleBindings	created	created
your API server	untouched	started with --authentication-config
who can log in	nobody, through OIDC	anyone your provider authenticates
creating the cluster	six kubeadm flags, as always	a configuration file BKE renders

On a new cluster it costs nothing extra, because the API server is being started for the first time anyway. If you know you want cluster login, set it before you run install.sh --role master and there is no second event.

On a cluster that already exists, turning it on restarts the API server. That belongs in a maintenance window. Turning it on for a cluster that already exists below is the procedure; read Break-glass first, not afterwards.

The limitation that may rule out your provider

BKE grants all cluster access through group membership. There is no mechanism to grant a role to an individual user — auth.rbac.* takes group names and nothing else.

So an identity provider that cannot emit a groups claim is unsupported. A user would authenticate successfully against it and then be able to do nothing at all, because no group means no binding means no permission.

BKE refuses a configuration like that outright:

auth.oidc[0] sets no groupsClaim, and BKE cannot support an identity provider that does not emit groups. All cluster access is granted through group membership (auth.rbac.*), so a login with no groups would authenticate and then be able to do nothing.

Check now, before anything else on this page matters:

1. Can your provider emit a groups claim in an ID token? Entra ID, Okta, Google Workspace and Keycloak all can. Some smaller or embedded providers cannot.
2. Does it emit groups without needing a second call? Kubernetes reads claims from the token. A provider that requires a /userinfo lookup to reveal group membership will not work.
3. Are the groups stable identifiers? If your provider emits group names that an administrator can rename, your bindings break when someone renames a group. Object IDs are safer, if less readable.

If the answer to the first is no, cluster login is not available for that provider in this BKE version, and no amount of configuration will change it.

Configuring it

auth:
  enabled: true
  oidc:
    - issuerUrl: https://login.example.com/v2.0
      clientId: bke-prod
      usernameClaim: email
      groupsClaim: groups
      groupsPrefix: "oidc:"
  rbac:
    clusterAdminGroups:
      - oidc:platform-admins
    adminGroups:
      - oidc:team-leads
    editGroups:
      - oidc:developers
    viewGroups: []

oidc is a list even with one issuer. Kubernetes supports several, and the shape does not change when you add a second.

groupsPrefix is worth setting. It prefixes every group from this issuer, so a group called admins at your provider becomes oidc:admins in Kubernetes and cannot collide with a name from anywhere else. Whatever you choose, the names in auth.rbac.* must include it — oidc:platform-admins, not platform-admins.

usernameClaim decides what shows up in audit logs. email is readable; sub is stable but opaque. Pick for whoever will read those logs.

usernamePrefix is optional and defaults to none. It does for usernames what groupsPrefix does for groups. Leaving it unset is normal; set it if you have more than one issuer and their usernames could collide.

issuerUrl must be https://. Kubernetes refuses anything else, and it refuses it when the API server starts — which on a new cluster is after kubeadm has already created it. BKE refuses it earlier, where it costs you a re-run.

Access to one namespace

The four lists above grant across the whole cluster. A group in editGroups can modify workloads in every namespace that exists and in every namespace created afterwards. For a platform team that is what you want; for an application team it is not.

auth.rbac.namespaces grants within a single namespace:

auth:
  rbac:
    clusterAdminGroups:
      - oidc:platform-admins
    namespaces:
      team-a:
        adminGroups:
          - oidc:team-a-leads
        editGroups:
          - oidc:team-a-developers
      team-b:
        viewGroups:
          - oidc:team-b

The keys under a namespace are the same three, and they name the same Kubernetes roles — admin, edit and view — applied to that namespace alone. BKE creates one RoleBinding per namespace and role, named bke-admin, bke-edit and bke-view, so kubectl describe rolebinding bke-edit -n team-a answers who can change things in team-a.

There is no clusterAdminGroups under a namespace, and it is refused by name rather than ignored. Granting cluster-admin within one namespace confers exactly what admin already confers there, including control of that namespace’s own RBAC, so the key would appear to scope cluster administration to a namespace and would not.

A group listed in both places is not restricted by the second

This is the one thing on this page that catches people out, and it is a property of Kubernetes rather than of BKE.

Kubernetes RBAC is purely additive. There is no deny. Permissions from every binding that matches are added together, and nothing subtracts. So:

    viewGroups:
      - oidc:developers          # view, on EVERY namespace
    namespaces:
      team-a:
        editGroups:
          - oidc:developers      # edit, on team-a

gives oidc:developers read access to the entire cluster and write access to team-a. It does not confine them to team-a. If that is what you intended, remove the group from viewGroups: there is no way to write a rule that takes the cluster-wide grant back.

The practical form is to keep the four cluster-wide lists for the people who genuinely administer the cluster, leave the rest empty, and grant everything else per namespace.

What BKE removes

These bindings are declarative in both directions. Taking a namespace out of auth.rbac.namespaces, or taking the last group out of one of its roles, removes the RoleBinding at the next apply.sh --commit. Access does not linger because it was granted once.

BKE removes only the three bindings it creates, identified by name and by its own app.kubernetes.io/managed-by=bke label. RoleBindings you wrote yourself are never touched, including any that happen to carry that label.

The namespace must already exist

A namespace named here is one of yours, holding your workloads, and BKE does not create it. apply.sh refuses before it fetches anything, naming every namespace it could not find rather than the first:

ERROR: auth.rbac.namespaces in /etc/bke/config.yaml names namespaces this
       cluster does not have:
         team-b
         team-c
       BKE does not create them - a namespace holding your own workloads is
       yours to create. Create each one, or correct the name in config.yaml.

Creating it for you would hide the failure this refusal exists to catch. A mistyped name would produce an empty namespace, the binding would be applied to it successfully, and the team it was meant for would discover the problem by being unable to work — with nothing anywhere reporting a fault.

What happens on the node

On a new cluster, install.sh --role master asks the API how this cluster should be created, before it changes anything on the machine:

• with auth.enabled: false it is told there is nothing to do, and the cluster is created exactly as it always was;
• with auth.enabled: true it is handed a kubeadm configuration file and an authentication configuration, writes the second to /etc/kubernetes/bke/authentication-config.yaml, and runs kubeadm init --config.

This is why the first control-plane node now needs /etc/bke/config.yaml and your licence file in place before you run install.sh. It is not guessing whether you want cluster login, and it is not reading your configuration file to find out — it asks, and it stops if it cannot.

On a cluster that already exists, the same two files are needed for the same reason, and the node sends its cluster’s own kubeadm configuration along with them: what comes back is that configuration with cluster login added, so nothing your cluster was created with is replaced by something we rendered.

check.sh checks both endpoints along with everything else, so a node that cannot reach them says so in the preflight rather than half-way through an install or a maintenance window.

Break-glass

The kubeadm administrator kubeconfig written at install time is your fallback, and it does not depend on any identity provider.

Verify it works before you reconfigure anything, not after. Enabling authentication restarts your API server, and if groups do not arrive as you expected, that certificate is the only way back in. Keep a copy somewhere that is not the cluster.

BKE checks the half of this it can check: if that kubeconfig does not authenticate against your cluster today, --enable-cluster-login refuses outright and is not overridable. It cannot check that you hold a copy somewhere else, so it asks you to say so — see below.

Canary one cluster before doing this to all of them.

Turning it on for a cluster that already exists

Run this on a control-plane node, in a maintenance window:

BKE_ACCEPT_AUTH_RISK="break-glass" \
  sh -c 'curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --enable-cluster-login --commit'

Nothing changes without --commit. Drop it and the same command performs every check, renders what it would write, shows you how your API server would differ, and stops. Do that first — days before the window if you can: it is where you find out that your break-glass kubeconfig does not authenticate, or that your API server has been edited by hand, while it is still cheap.

It takes no other option but --commit, moves no version and installs no package. Enabling cluster login restarts the API server and so does a control-plane upgrade; doing both in one run would leave two candidates for a control plane that did not come back.

Every control-plane node needs it, one at a time. The API server’s configuration is a file on each node. Wait for each node’s API server to answer before starting the next.

Before it changes anything, it:

• checks your break-glass kubeconfig against the live API server, and refuses if it does not authenticate;
• checks that the kubeadm on the node is from the Kubernetes minor the cluster is running, because kubeadm is what writes the API server’s configuration and one from another release writes that release’s;
• asks us for your cluster’s own kubeadm configuration with cluster login added to it — your configuration, with two settings added and nothing else changed;
• renders the API server configuration it would write, and compares it with the one the node is running. If anything other than cluster login differs, it prints the difference and stops.

It then writes the authentication configuration, records the change in your cluster’s own kubeadm configuration so a later upgrade keeps it, regenerates the API server, waits for it to answer, and confirms that what came back is configured for cluster login.

A copy of the API server configuration as it was is left at /etc/bke/kube-apiserver.yaml.before-cluster-login. Restoring it over /etc/kubernetes/manifests/kube-apiserver.yaml undoes the change on that node.

The two conditions it refuses on, and how to accept one

Each is named, and each is accepted by name. There is no blanket yes: accepting one is a decision about one risk rather than a decision to stop checking.

Name	What it means
break-glass	We cannot check that you hold a copy of the administrator kubeconfig somewhere that is not this cluster. Confirm that you do.
manifest-drift	Regenerating the API server would change more than cluster login — usually because its configuration has been edited by hand since the cluster was created. The difference is printed. Read it before accepting it.

BKE_ACCEPT_AUTH_RISK="break-glass manifest-drift" \
  sh -c 'curl -sL https://bke.maml.uk/upgrade.sh | sh -s -- --enable-cluster-login --commit'

An administrator kubeconfig that does not authenticate is not in this table, and is not overridable. A fallback that does not work today will not work after the restart.

What it does not do

It does not create the access bindings — apply.sh does that, from auth.rbac.*, whether cluster login is on or off. They are already there.

It does not change any other cluster. Canary one, confirm your people can log in and have the permissions you expect, and only then do the rest.

Giving your people access

BKE distributes no kubeconfigs and issues no credentials. A person’s access is their identity at your provider plus the group bindings above, and the kubeconfig they use contains no secret: it names your cluster, names your issuer, and names a command to run when a token is needed. That is what makes it safe to hand the same file to everyone, commit it to a repository, or put it on a shared drive.

Do not hand out copies of the administrator kubeconfig as a substitute. It is a client certificate. It cannot be revoked short of rotating the cluster CA, it identifies every holder as the same person in your audit logs, and it bypasses every binding on this page.

The rest of this section is one worked example. It assumes you administer Debian or Ubuntu machines and are comfortable with Kubernetes without being a specialist in it. There are three parts: a client registered once at your identity provider, a tool installed on each person’s own machine, and a kubeconfig you write once and distribute.

1. Register a client at your identity provider

This is a second client, separate from anything your applications use. Register it once; every person and every cluster shares it.

Setting	Value
Client type	public — no client secret
Authorization code flow	enabled
Redirect URIs	http://localhost:8000, http://localhost:18000, http://127.0.0.1:8000, http://127.0.0.1:18000
Groups claim	must appear in the ID token
PKCE	leave it to be negotiated; do not require it

A public client is correct here, not a compromise. The software holding these credentials runs on a laptop, so a client secret placed in it would be readable by whoever holds the laptop and would not be a secret.

The four redirect URIs are one URI with two port numbers and two spellings of the same address. The login tool starts a web server on the person’s own machine to receive the response from your provider; it listens on 127.0.0.1:8000 and falls back to 127.0.0.1:18000 when something else already holds the first port. Registering both spellings as well costs nothing and avoids discovering which one was used while people are waiting to log in.

Do not configure the client to require PKCE. The login tool reads your provider’s discovery document and uses PKCE whenever the provider offers it, so there is nothing to switch on and no protection to gain by forcing it. Forcing it has a cost: anyone logging in from a machine with no browser uses the device authorization flow, for which PKCE parameters are not defined at all, and a provider that insists on them refuses the request before a browser is ever opened. The refusal reads:

{"error":"invalid_request","error_description":"Missing parameter: code_challenge_method"}

The groups claim must be in the ID token. The login tool presents the ID token to your API server, and your API server reads claims from the token it is given. A group membership that is only visible at your provider’s /userinfo endpoint is not visible to Kubernetes. If your provider emits groups only when a particular scope is requested, note the scope name — it is needed in step 3.

2. Install the login tool on each person’s machine

kubectl oidc-login is a kubectl plugin named kubelogin, published at https://github.com/int128/kubelogin. It is not part of kubectl, and BKE neither installs nor requires it: it runs on the machine the person is typing on, and nothing about it reaches your cluster. Any client that can perform an OIDC login and return a token to kubectl will do; this one is the common choice.

Debian and Ubuntu carry no package for it. Install the release binary:

sudo apt-get install -y unzip

VERSION=v1.36.4
ARCH=amd64                      # arm64 on a Raspberry Pi or an Apple Silicon VM

cd "$(mktemp -d)"
curl -fsSLO "https://github.com/int128/kubelogin/releases/download/${VERSION}/kubelogin_linux_${ARCH}.zip"
curl -fsSLO "https://github.com/int128/kubelogin/releases/download/${VERSION}/kubelogin_linux_${ARCH}.zip.sha256"
sha256sum -c "kubelogin_linux_${ARCH}.zip.sha256"

unzip -q "kubelogin_linux_${ARCH}.zip"
sudo install -m 0755 kubelogin /usr/local/bin/kubectl-oidc_login

The installed name matters and is not the name of the file in the archive. kubectl discovers plugins by looking on PATH for executables called kubectl-<something>, so the binary must be installed as kubectl-oidc_login for kubectl oidc-login to exist. The underscore is not a typo: kubectl converts it to the hyphen in the subcommand name.

Confirm it before going further:

kubectl oidc-login --version

You do not need krew. krew is kubectl’s plugin manager and is one way to install this plugin, but installing the binary under the right name is the whole of what krew would have done. kubectl krew install oidc-login on a machine without krew fails with unknown command "krew" for "kubectl", which reads as a problem with kubectl and is not one.

On macOS, brew install kubelogin installs a different tool. That formula name belongs to Microsoft’s kubelogin for Azure, which is unrelated. The one described here is:

brew tap int128/kubelogin
brew install int128/kubelogin/kubelogin

3. Write the kubeconfig

Three pieces: the cluster, the credential, and a context joining them. Run these on the person’s machine, or run them once and distribute the resulting file.

The cluster. You need two facts from a control-plane node: the API server’s address, which is the server: line in the administrator kubeconfig, and the cluster’s CA certificate, which is /etc/kubernetes/pki/ca.crt. The CA certificate is a public certificate and is safe to distribute; the private key beside it is not, and is not needed here.

kubectl config set-cluster prod \
  --server=https://k8s.example.com:6443 \
  --certificate-authority=./ca.crt \
  --embed-certs=true

Both halves are required. Without --server every command goes to localhost:8080 and reports connection refused; without the CA certificate the connection is refused for an unknown authority. --embed-certs=true writes the certificate into the kubeconfig, so the finished file is self-contained and can be copied to another machine.

The credential. This stanza runs the login tool rather than storing anything:

kubectl config set-credentials oidc \
  --exec-interactive-mode=Never \
  --exec-api-version=client.authentication.k8s.io/v1 \
  --exec-command=kubectl \
  --exec-arg=oidc-login --exec-arg=get-token \
  --exec-arg=--oidc-issuer-url=https://login.example.com/v2.0 \
  --exec-arg=--oidc-client-id=bke-prod

--oidc-issuer-url and --oidc-client-id must match auth.oidc[0].issuerUrl and auth.oidc[0].clientId in your config.yaml exactly. The API server validates the issuer and the audience against what it was configured with, and a trailing slash on one side and not the other is a mismatch.

--exec-interactive-mode=Never is required, not a preference. Without it kubectl refuses every command with interactiveMode must be specified for oidc to use exec authentication plugin. Never is correct even though a browser opens: it tells kubectl that the plugin does not need kubectl’s own terminal, which is true — the plugin runs its own browser and its own local web server.

If your provider needs an extra scope before it will emit groups, add it here:

  --exec-arg=--oidc-extra-scope=groups

The context, and then use it:

kubectl config set-context prod-oidc --cluster=prod --user=oidc
kubectl config use-context prod-oidc

The first command run against this context opens a browser. After signing in, the token is cached under ~/.kube/cache/oidc-login and reused until it expires; rm -rf ~/.kube/cache/oidc-login forces a fresh login, which is what to do after changing someone’s group membership.

4. Confirm it worked

Two commands, in this order, and the first is the one that matters:

kubectl auth whoami
kubectl auth can-i --list
kubectl auth can-i --list -n team-a    # for access granted per namespace

auth whoami prints the username and the groups the API server derived from the token. This is the only direct evidence that authentication is working as configured, and the group strings it prints are what you compare against auth.rbac.*.

They must match character for character, groupsPrefix included. A group printed as oidc:/platform/admins is not bound by a config.yaml listing oidc:platform-admins: the login succeeds, the groups arrive, no binding matches, and the person can do nothing. That failure looks identical to authentication not working at all, which is why auth whoami comes first.

Then test someone who should have no access, and treat it as part of the work rather than as an optional extra. A successful login by an administrator proves your provider is trusted. Only a login by someone in none of the groups in auth.rbac.* — who authenticates, and can then do nothing — proves that the groups are being read rather than ignored.

Logging in from a machine with no browser

Someone working over SSH on a jump host has no browser for the flow above. Add one argument to the credential stanza:

  --exec-arg=--grant-type=device-code

The tool then prints a URL and a short code; the person opens them on a machine that does have a browser. This is the flow that your provider’s PKCE setting can break, as described in step 1.

When something does not work

What you see	What it is
unknown command "krew" for "kubectl"	krew is not installed, and is not needed. Install the binary as kubectl-oidc_login instead.
interactiveMode must be specified for oidc to use exec authentication plugin	--exec-interactive-mode=Never is missing from the credential stanza.
The connection to the server localhost:8080 was refused	The kubeconfig has no cluster entry. set-credentials does not create one; run set-cluster as well.
Missing parameter: code_challenge_method	The client at your provider requires PKCE, and the device flow cannot supply it. Stop requiring it.
x509: certificate signed by unknown authority	The cluster entry has no CA certificate, or the wrong one.
Login succeeds, auth can-i --list shows almost nothing	The groups arriving do not match auth.rbac.*. Compare kubectl auth whoami with kubectl describe clusterrolebinding bke-admin, prefix included.
Someone granted one namespace can read the whole cluster	They are also in a group listed in the cluster-wide viewGroups. RBAC adds; it never subtracts. See Access to one namespace.
Login succeeds, auth whoami shows no groups at all	Your provider is not putting groups in the ID token. An extra scope may be required; see step 1.