Moving an estate to BKE.

The principle: build beside, then move

A migration to BKE builds a new cluster beside the one in service and moves workloads across in stages. It does not convert the existing cluster in place. No distribution can be converted into another safely while it carries production, and a parallel build has a further advantage: the source cluster remains the rollback position until each workload has been proven on the target.

The method has six stages. The platform guides add what is specific to each source, but the stages do not change.

1. Take an inventory

List everything the source cluster runs and everything it provides, before deciding anything. The inventory has two parts.

What runs on it. Namespaces, workloads, persistent volumes and the size of the data on each, and every object of a type that is not part of upstream Kubernetes. The last list is the most important, because it is the measure of how far the estate has adopted the source platform’s own way of working:

# Every API group the cluster serves. Groups outside *.k8s.io and the core
# Kubernetes groups belong to the platform or to something installed on it.
kubectl api-resources --verbs=list -o wide | sort -k3

# How many objects of one such type exist, across every namespace.
kubectl get <resource>.<group> -A --no-headers | wc -l

What it provides. Identity and login, ingress and certificates, storage, the image registry, build pipelines, monitoring, logging, the service mesh, backups, and anything the platform’s console does for your teams. Each of these is a service a workload depends on, whether or not any manifest names it.

2. Classify every item

Each item in the inventory falls into one of four groups, and the proportion in each is a reliable estimate of the effort involved.

3. Map the platform’s services to BKE

BKE installs seven components, each opt-in, and deliberately leaves the rest to you. For every service in the inventory, decide which of the following applies.

A service listed as your own is not necessarily new work. A registry, a CI system or a Prometheus installation that the estate already operates can usually continue to serve the BKE cluster unchanged. Where the source platform provided the registry and CI itself, the Mamluk Enterprise Helm Charts provide both, so that the estate does not have to assemble them.

4. Build the target cluster

The target cluster is built from the documentation in the ordinary way, with three points of particular importance in a migration.

Fix the decisions that cannot change. The node hostnames, the control-plane endpoint and the service DNS suffix are fixed when the cluster is created. Before you begin describes each. If any application addresses a service by its full in-cluster name, choose the same DNS suffix as the source cluster, which is usually cluster.local.

Enable cluster login at creation. Turning it on later restarts the API server. If the source platform authenticates users itself, confirm first that your identity provider can emit a groups claim, because BKE grants all access through groups.

Choose the pod network range with the estate in view. The range set as cluster.podNetworkCidr in config.yaml is also fixed when the cluster is created. It must not overlap any network the workloads need to reach, including the networks of the source cluster’s nodes while both clusters are in service.

5. Move in waves

Move one pilot application first, chosen because it is representative rather than because it is simple: it should use ingress, a Secret and, ideally, a persistent volume. Then move the remainder in waves, grouped by dependency so that services which call each other move together.

For each application:

1. Apply the translated manifests to the target cluster, with the application scaled to zero or held out of service.
2. Move the data, using one of the methods below.
3. Test the application on the target through a temporary hostname or a hosts-file entry, without changing public DNS.
4. Move traffic, either by changing DNS or by changing the backends of the load balancer in front of the cluster. Lower the DNS time-to-live in advance.
5. Keep the source deployment available, scaled down, until the agreed observation period has passed.

Moving data. The method depends on what holds the data, and the order of preference is the same on every platform.

• Replication at the application level. A database that supports replication can run a replica on the target, follow the source until the cutover, and then be promoted. This gives the shortest interruption and requires no knowledge of either platform’s storage.
• Dump and restore. A logical export and import, during a maintenance window sized to the data.
• Volume-level copy. A backup tool that reads volumes through the file system, such as Velero with its file-system backup, can copy a volume from one storage system to another. It works without the application’s co-operation, but the application must be stopped, or the data quiesced, for the copy to be consistent.

Object storage that the application reaches by URL, rather than through a volume, does not move at all.

6. Decommission

When every application has passed its observation period, remove the source deployments, retain the final backups for the period your policy requires, and only then retire the source cluster and its licences. Record the final state of the target’s config.yaml and values files in source control, if they are not there already. They are the complete description of the new cluster.

Help with the work

The migration remains your team’s project, but it need not be carried out alone. Support can be purchased with consulting time, which can be applied to the inventory, the plan and the cutover. The cost-of-ownership example shows one such arrangement.

Other sources

The same six stages apply to any other source. From a managed cloud service such as EKS, AKS or GKE, the work is principally in what the cloud provided: its load balancers, its storage classes, and its identity integration for workloads. From another upstream distribution, such as kubeadm, Talos or k3s, most manifests move unchanged, and the inventory will be short.