CKA Cluster Lifecycle: kubeadm Upgrades, etcd Recovery, and High Availability

Introduction

CKA cluster lifecycle management builds on installing a Kubernetes cluster with kubeadm. This lesson continues the Cluster Architecture, Installation & Configuration domain (25% of the CKA). The official curriculum asks you to manage the lifecycle of a Kubernetes cluster using kubeadm: upgrade clusters, implement a highly available control plane, and operate etcd including backup and restore. These are classic hands-on exam tasks: "upgrade the control plane node to version X", "take an etcd snapshot to this path", "drain this node before maintenance". Each of them has a precise command sequence, and small ordering mistakes cost points.

After this lesson you will be able to plan and perform a kubeadm upgrade, drain and cordon nodes safely, back up and restore etcd, reason about HA topologies and quorum, and inspect and renew cluster certificates.

The Linux Foundation FAQ consulted for this review lists Kubernetes v1.37 for CKA. The 1.34-to-1.35 versions below are explicit historical training examples, not the current exam target or a recommendation to install an old patch. Use the task’s target version and matching upgrade documentation, package repository, and etcd tooling.

Learning objectives

• Apply the version skew policy and plan a one-minor-version-at-a-time upgrade.
• Upgrade a control plane node and worker nodes with kubeadm upgrade, in the correct order.
• Use kubectl cordon, drain and uncordon, and explain what blocks a drain.
• Take an etcd snapshot with etcdctl and restore it with etcdutl, then point the static Pod at the restored data.
• Compare stacked and external etcd topologies and compute etcd quorum.
• Check and renew certificates with kubeadm certs.

Planning an upgrade: the version skew policy

Kubernetes releases a new minor version (1.34, 1.35, ...) roughly three times a year. A kubeadm cluster must be upgraded one minor version at a time: from 1.33 to 1.35 you must pass through 1.34. Patch upgrades (1.35.0 to 1.35.3) can jump directly. The skew rules you should know:

The practical consequence is the order: control plane first, then workers. Check all reachable API server versions in HA: the oldest instance constrains kubelet and kube-proxy upgrades, while kubectl may be one minor newer. Tool-specific restrictions still apply.

Draining and cordoning nodes

Before upgrading or repairing a node you move its workload away.

• kubectl cordon <node> marks the node unschedulable (SchedulingDisabled). Existing Pods keep running.
• kubectl drain <node> cordons the node and evicts its Pods through the Eviction API, honouring PodDisruptionBudgets (PDBs).
• kubectl uncordon <node> makes it schedulable again.

kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data
# ... maintenance ...
kubectl uncordon worker-2

A drain may need the following flags; choose them from the workloads and accepted data-loss conditions rather than adding them automatically:

What else blocks a drain? A bare Pod (not owned by a controller) cannot be recreated elsewhere; drain refuses unless you add --force, and the Pod is lost permanently. A PDB that allows zero disruptions makes eviction retry; the default drain timeout is unlimited unless a finite --timeout is supplied. Pods stuck terminating, or local storage, are other causes.

Worked example: the drain that never finishes

Scenario. Contoso drains node-3 before a kernel patch. The command prints cannot evict pod as it would violate the pod's disruption budget for the checkout Deployment, which has 2 replicas and a PDB with minAvailable: 2.

Reasoning.

1. With 2 replicas and minAvailable: 2, no voluntary disruption is allowed, so any eviction violates the budget.
2. Inspect: kubectl get pdb -A and kubectl get deploy checkout.
3. The right fix is to restore capacity: scale to 3 replicas (kubectl scale deploy checkout --replicas=3), and wait for the extra replica to become Ready on available capacity outside the draining node. Then one Pod may be evicted while two remain available.
4. Alternatively relax the PDB (minAvailable: 1) if the business accepts reduced redundancy.

Why not the alternatives. Deleting the Pods directly bypasses the budget and may cause an outage; --disable-eviction does the same and should be a last resort; deleting the node object removes it from the API but does not stop its containers.

Upgrading a cluster with kubeadm

The official sequence for the first control plane node (assume the target is v1.35.2 on a Debian-style system):

# 1. Enable the target minor repository first (v1.35 for this historical example).
# See the official package-repository change instructions, then find the package:
sudo apt-get update
apt-cache madison kubeadm | head

# 2. Upgrade kubeadm itself first
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm='1.35.2-*'
sudo apt-mark hold kubeadm
kubeadm version

# 3. Verify the plan and apply
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.35.2

# 4. Drain, upgrade kubelet and kubectl, restart, uncordon
kubectl drain cp1 --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet='1.35.2-*' kubectl='1.35.2-*'
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl uncordon cp1

Key points to remember:

• kubeadm first. kubeadm upgrade apply runs the new kubeadm's logic; the old binary cannot upgrade to a newer version.
• kubeadm upgrade plan shows what it can upgrade to, component versions and whether manual steps (for example CoreDNS or config migration) are required.
• kubeadm upgrade apply upgrades local control-plane static Pod manifests and normally renews kubeadm-managed leaf certificates. Since v1.28, addon upgrades (CoreDNS and kube-proxy) wait until all control-plane instances have been upgraded. It does not upgrade the kubelet package; that is your step 4.
• Other control plane nodes run sudo kubeadm upgrade node (not apply) after upgrading the kubeadm package.
• Worker nodes: upgrade the kubeadm package, run sudo kubeadm upgrade node, then drain from the control plane, upgrade kubelet and kubectl, restart the kubelet and uncordon. Do one worker at a time, or in small batches, to preserve capacity.
• Verify with kubectl get nodes; the VERSION column reports the kubelet version, which is why a node still shows the old version until its kubelet package is upgraded and restarted.

On some exam tasks, etcd upgrade is skipped (kubeadm upgrade apply --etcd-upgrade=false) when the task says not to touch it.

Worked example: upgrade order on a 1 + 2 node cluster

Task. A cluster runs v1.34.4 on cp1, w1, w2. Upgrade it to v1.35.2 with minimal downtime for a 4-replica web Deployment.

Steps.

1. Confirm the jump is a single minor version (34 to 35): allowed.
2. Upgrade cp1 as above (kubeadm, upgrade plan, upgrade apply v1.35.2, drain, kubelet and kubectl, restart, uncordon). While cp1 is being upgraded the API server restarts, but Pods already running keep serving traffic.
3. On w1: install kubeadm 1.35.2, sudo kubeadm upgrade node. From cp1, kubectl drain w1 --ignore-daemonsets --delete-emptydir-data. On w1 upgrade kubelet, restart. From cp1, kubectl uncordon w1.
4. Wait until replacement Pods are Ready and the required available replica count is restored, then repeat for w2.
5. Check kubectl get nodes shows v1.35.2 everywhere.

Why not upgrade both workers at once? Draining both would take away all capacity; with only a single node left the 4 replicas may not fit, and the Deployment is partly unavailable. Why not workers first? A v1.35 kubelet talking to a v1.34 API server violates the skew policy.

Backing up and restoring etcd

etcd stores Kubernetes API state; PersistentVolume payloads, external databases, and files on nodes require separate backups. If it is lost without a backup, the cluster state is lost. In a kubeadm cluster etcd runs as a static Pod (/etc/kubernetes/manifests/etcd.yaml), and its data lives in /var/lib/etcd on the control plane node.

Taking a snapshot

Use an etcd client certificate trusted by the endpoint, such as kubeadm’s healthcheck-client.crt and matching key. Inspect the manifests and PKI rather than assuming a server-only certificate has client-auth usage. Secure the snapshot because it contains sensitive API state.

sudo install -d -m 700 /opt/backup
sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  snapshot save /opt/backup/etcd-snapshot.db

Check the file: sudo etcdutl snapshot status /opt/backup/etcd-snapshot.db --write-out=table. If the host lacks the tools, obtain compatible official etcd binaries or inspect what the container image actually provides; do not assume every image includes both utilities. Protect and copy the verified snapshot off the node, and retain the PKI and manifests needed for recovery.

Restoring a snapshot

Restoring is done offline with etcdutl (the etcdctl snapshot restore subcommand is deprecated and removed in recent etcd releases). For Kubernetes restores with supported etcd tooling, a revision bump and compaction invalidate stale informer caches. The following single-member example assumes member name cp1, peer URL https://10.0.0.11:2380, and a new empty destination. Substitute the actual manifest identity and peer URL; choose a bump large enough for writes since the snapshot:

sudo etcdutl snapshot restore /opt/backup/etcd-snapshot.db \
  --data-dir /var/lib/etcd-restore \
  --name cp1 \
  --initial-cluster cp1=https://10.0.0.11:2380 \
  --initial-advertise-peer-urls https://10.0.0.11:2380 \
  --bump-revision 1000000000 --mark-compacted

Stop the API server writers and old etcd process using the documented static-Pod maintenance procedure before switching to restored state. Keep manifests and original data outside the watched directory as recovery copies. Then tell the etcd static Pod to use the new directory by editing /etc/kubernetes/manifests/etcd.yaml: in the volumes section, change the hostPath.path of the volume named etcd-data from /var/lib/etcd to /var/lib/etcd-restore. The kubelet notices the change and recreates the etcd Pod, then the API server reconnects. (Do not forget that the --data-dir argument inside the container still refers to the container path /var/lib/etcd; only the host path changes.) The restore creates a new cluster identity, so for a full multi-member recovery, restore the same snapshot into separate member directories with consistent new membership, distinct names and peer URLs. Loss of one member with quorum intact is normally repaired by member replacement, not a full-cluster snapshot restore.

Worked example: undoing an accidental namespace deletion

Scenario. An engineer at Fabrikam deleted the billing namespace by mistake at 14:05. A snapshot was taken at 13:30. The cluster has one control-plane node; the restore is a full API-state rollback, not a namespace-only restore.

Reasoning.

1. Confirm that a restore is justified: all changes after 13:30 will be lost, including everything created in other namespaces. Get agreement.
2. Move the API server and etcd manifests out of the watched directory and wait for their containers to stop, checking with sudo crictl ps. Keep recovery copies outside that directory.
3. Restore into a new directory with the single-member command above, substituting this cluster’s member name and peer URL; include the supported revision-bump and compaction options.
4. Update the saved etcd manifest’s etcd-data hostPath to /var/lib/etcd-restore, return that manifest, and verify etcd health before restarting API server writers.
5. Return the API server manifest and verify namespace/API state and controller recovery. Restore or reconcile application data separately; etcd does not restore volume contents.

Why other choices are weaker. Re-applying manifests from Git recreates objects but not lost API metadata or manually created Secrets; neither method restores the contents of external PersistentVolumes; restoring over the running data directory without stopping etcd corrupts the store; deleting /var/lib/etcd first would destroy the only copy.

Highly available control plane

A single control plane node is a single point of failure for management operations (running containers can continue and kubelets can restart local containers, but scheduling and control-plane reconciliation are unavailable). An HA cluster has three or more control plane nodes behind a load balancer. kubeadm supports two topologies:

In both cases, clients use the API server through a load balancer (or virtual IP such as keepalived plus HAProxy) listening on the address given to --control-plane-endpoint. Only the API server is accessed through it: the scheduler and controller manager use leader election, so only one instance is active at a time.

etcd quorum

etcd uses Raft. For the voting members in this example, a write requires a majority (quorum): quorum = floor(n/2) + 1. Learners do not count as voting capacity.

An even number adds cost without adding tolerance (4 members still tolerate only 1 failure), so use odd sizes: 3 or 5. Without quorum, writes and normal linearizable reads cannot proceed. First restore connectivity or enough existing members; permanent majority loss may require snapshot disaster recovery. It is not a usable read-only Kubernetes mode.

Building the HA cluster with kubeadm

1. Provision a load balancer for k8s-api.contoso.example:6443.
2. On the first node: sudo kubeadm init --control-plane-endpoint "k8s-api.contoso.example:6443" --upload-certs --pod-network-cidr=10.244.0.0/16.
3. Install the CNI plugin.
4. On each additional control plane node, run the join command that includes --control-plane --certificate-key <key>.
5. Join workers with the normal join command.

If the uploaded certificate Secret has expired (normally two hours), re-upload with sudo kubeadm init phase upload-certs --upload-certs.

Certificate management

By default, kubeadm creates CAs valid for 10 years and kubeadm-managed leaf certificates valid for 1 year; these durations are configurable. Kubelet client certificates have a separate rotation lifecycle. A cluster that is never upgraded stops working when they expire: kubectl shows x509: certificate has expired, and the API server and kubelets fail to authenticate each other.

sudo kubeadm certs check-expiration        # table with expiry dates and residual time
sudo kubeadm certs renew all               # renew every kubeadm-managed certificate
sudo kubeadm certs renew apiserver         # renew only one

After renewing, restart the control plane static Pods so they load the new files; the simplest method is to move the manifests out of /etc/kubernetes/manifests and back after a short pause, waiting for the kubelet’s manifest scan interval and verifying termination/recreation. Restarting kubelet alone does not guarantee its static Pod containers restart. Copy the refreshed admin.conf to ~/.kube/config. Normal kubeadm upgrades renew managed leaf certificates unless renewal is disabled; this does not renew CA certificates, external-CA material, or every copied kubeconfig. Continue monitoring expiry. Kubelet client certificates rotate automatically when rotateCertificates is enabled (the default for kubeadm clusters).

If you need a certificate signed for a new identity, the Kubernetes CertificateSigningRequest API (kubectl certificate approve) lets the cluster CA sign it; this is used in the RBAC lesson to create a user.

Worked example: expired certificates

Scenario. Tailwind Traders' lab cluster has not been touched for 14 months. kubectl get pods fails with Unable to connect to the server: x509: certificate has expired or is not yet valid.

Reasoning.

1. Check system time and inspect sudo kubeadm certs check-expiration on the control-plane node. Identify which leaf or client certificate expired; the CA may still be valid. Confirm local signing-key availability before renewal.
2. sudo kubeadm certs renew all.
3. Restart the control plane: temporarily move the four manifests out of /etc/kubernetes/manifests, wait 20 seconds, move them back.
4. sudo cp /etc/kubernetes/admin.conf ~/.kube/config so kubectl uses the fresh client certificate.
5. Verify kubectl get nodes. Worker kubelets, if their client certificates were rotated, reconnect on their own; otherwise follow the documented kubelet client-certificate recovery procedure rather than immediately resetting the node.

Why this order. Renewing without restarting leaves the old certificate in memory; keeping the old kubeconfig keeps the expired client certificate.

Common pitfalls and exam tips

• Skipping a minor version is unsupported: always go 1.34 to 1.35 to 1.36.
• Upgrading kubelet before kubeadm upgrade apply or workers before the control plane violates the skew policy.
• Using kubeadm upgrade apply on secondary control plane or worker nodes: use kubeadm upgrade node there.
• Forgetting --ignore-daemonsets makes drain fail immediately; forgetting to uncordon leaves the node unschedulable (a frequent exam deduction).
• A restored etcd data directory that is never referenced in etcd.yaml changes nothing.
• A stray backup file in /etc/kubernetes/manifests runs as an extra static Pod or breaks the kubelet's parse.
• Even-sized etcd clusters are a design smell.
• Exam tip: the etcd certificate paths are in /etc/kubernetes/manifests/etcd.yaml; copy them with grep instead of typing from memory.
• Exam tip: verify each phase with kubectl get nodes, kubectl get pods -n kube-system, sudo crictl ps.

Mini case study: maintenance window at Fabrikam

Situation. Fabrikam runs a three-node stacked HA control plane (cp1-cp3) and five workers on v1.34.3. They plan to move to v1.35.2 this weekend, and a security audit asks for an etcd backup policy.

Questions.

1. In what order should nodes be upgraded?
2. How many control plane nodes can fail before the cluster loses write ability?
3. Which command do the second and third control plane nodes use?
4. Before upgrading, what must be done for the data?

Solution, commented.

1. Take an etcd snapshot, then upgrade cp1 (kubeadm upgrade apply), cp2 and cp3 (kubeadm upgrade node) one at a time, then the workers in small batches with drain/uncordon.
2. One. Quorum for three members is two; with two failures only one member remains and writes stop.
3. sudo kubeadm upgrade node: the cluster configuration is already updated by the first node.
4. Back up etcd to a location off the node (for example object storage) with etcdctl snapshot save and verify it with etcdutl snapshot status; a backup that was never verified is not yet a backup.

Use Kubernetes RBAC and identity checks to verify administrative access after recovery, and cluster and component troubleshooting to investigate unhealthy control-plane Pods.

Key terms

• Version skew: the allowed version difference between components.
• Cordon / drain / uncordon: mark unschedulable / evict Pods / re-enable scheduling.
• PodDisruptionBudget (PDB): limit on voluntary disruptions.
• Snapshot: point-in-time backup of etcd.
• Quorum: majority of etcd members needed to commit writes.
• Stacked / external etcd: etcd co-located with control plane nodes / on separate hosts.
• Leader election: mechanism letting only one scheduler or controller manager act at a time.

Summary checklist

• Upgrade one minor version at a time, control plane first, then workers.
• kubeadm package, kubeadm upgrade plan, kubeadm upgrade apply (first node) or kubeadm upgrade node (others).
• Drain with --ignore-daemonsets --delete-emptydir-data, upgrade kubelet and kubectl, restart kubelet, uncordon.
• etcd: etcdctl snapshot save with endpoint and three certificate flags; etcdutl snapshot restore --data-dir; update the etcd-data hostPath.
• HA: odd number of etcd members, load balancer at --control-plane-endpoint, --upload-certs.
• kubeadm certs check-expiration and renew, restart the control plane, refresh kubeconfig.

Official sources

• Upgrading kubeadm clusters: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ (opens in a new tab)
• Version skew policy: https://kubernetes.io/releases/version-skew-policy/ (opens in a new tab)
• Safely drain a node: https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/ (opens in a new tab)
• Operating etcd clusters: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/ (opens in a new tab)
• HA topology options: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/ (opens in a new tab)
• Certificate management with kubeadm: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/ (opens in a new tab)
• CKA curriculum: https://github.com/cncf/curriculum/tree/master/cka (opens in a new tab)
• CKA environment FAQ: https://docs.linuxfoundation.org/tc-docs/certification/faq-cka-ckad-cks (opens in a new tab)
• etcd snapshot recovery and revision bumps: https://etcd.io/docs/v3.6/op-guide/recovery/ (opens in a new tab)