Kubernetes Cluster Troubleshooting: Nodes, Kubelet, and Control Plane

Introduction

Kubernetes cluster troubleshooting starts by distinguishing client access, control-plane health, node agents and workload failures. It belongs to the Troubleshooting domain, the largest part of the CKA at 30%. This lesson covers clusters, nodes and components; the following lessons address monitoring, container output and networking. The official exam page currently lists Kubernetes v1.37; confirm the version for your sitting and use matching documentation.

Original lab scenarios include a NotReady worker, unscheduled Pods and an unavailable API endpoint. On a kubeadm Linux/systemd lab, use observe, locate the layer, read logs, apply the smallest justified fix, verify. Managed services, external etcd and different runtimes need their own procedures; do not assume SSH or static control-plane manifests are available everywhere.

After this lesson you will be able to diagnose a NotReady node, repair kubelet and container runtime problems, fix a broken control plane static Pod, use systemctl, journalctl and crictl, and recognise common configuration faults without assuming that a symptom proves a single cause.

Learning objectives

• Apply a layered method to cluster failures (API reachable? nodes Ready? system Pods running?).
• Diagnose NotReady nodes through conditions, kubelet status and logs.
• Explain how static Pods work and fix manifests in /etc/kubernetes/manifests.
• Use systemctl, journalctl and crictl when kubectl cannot help.
• Identify frequent causes: wrong kubeconfig, expired certificates, bad flags, swap, wrong paths and a stopped runtime.

A layered method

Work from the outside in and stop at the first layer that is broken.

If API access fails, API-based checks may be unavailable through that endpoint. First distinguish a client/context issue from an actual server failure; another healthy endpoint or authorised client may still work. Use node-shell tools when the API path is genuinely unavailable.

When kubectl fails

The message The connection to the server ... was refused has a few typical causes:

1. Wrong kubeconfig or context. Check kubectl config get-contexts and the server: URL. For an authorised administrator on a kubeadm control-plane host, /etc/kubernetes/admin.conf is a possible diagnostic kubeconfig. It grants powerful access: use the credentials appropriate to the task and protect the file.
2. The API server is down. On the control plane node run crictl ps -a | grep kube-apiserver, then read its logs with crictl logs <id>.
3. etcd is unavailable. The API server may fail startup/readiness or requests; inspect its logs, configured etcd endpoints and, for stacked etcd, crictl ps -a | grep etcd. External etcd is not necessarily a local container.
4. The kubelet is down. Existing containers may keep running, but static Pod reconciliation and node status reporting stop; do not infer container absence from service state alone.
5. Certificate or clock problems. A TLS/x509 error differs from TCP connection refused. Check time and kubeadm certs check-expiration. Renew only certificates covered by the diagnosed kubeadm/local-CA workflow, then restart affected components using the documented procedure. renew all does not rotate CAs or repair every external/kubelet certificate.

Nodes and the kubelet

Every node runs the kubelet (a systemd service), a container runtime (containerd or CRI-O) and normally kube-proxy and a CNI agent as Pods. The kubelet reports node conditions, shown by kubectl describe node:

Ready=Unknown with the message Kubelet stopped posting node status means the control plane stopped hearing from the kubelet: the service is stopped, the node is down or the network is broken. Ready=False with container runtime network not ready points to the CNI.

Worked example: node worker-2 is NotReady

Scenario. kubectl get nodes shows worker-2 NotReady. Pods on it are Unknown.

Steps.

1. Read the conditions: kubectl describe node worker-2. The Ready condition reads Unknown, "Kubelet stopped posting node status". That points at the kubelet or the host, not at a scheduling problem.
2. Reach the node: ssh worker-2, then sudo -i.
3. Check the service: systemctl status kubelet. It reports inactive (dead).
4. Start and enable it: systemctl enable --now kubelet. If it fails, read the logs: journalctl -u kubelet --no-pager -n 50.
5. In this lab, failSwapOn is true and the log reports swap is enabled unexpectedly. Confirm swapon --show, disable the unintended swap with swapoff -a, and remove its boot-time activation (the lab uses /etc/fstab). Check other swap units/generators if applicable. This restores the intended no-swap configuration; current Linux Kubernetes can instead support deliberately configured swap with failSwapOn: false and an appropriate memorySwap policy.
6. Restart kubelet and verify service, journal, runtime and node conditions. Watch kubectl get nodes -w for Ready; recovery time depends on the fault and heartbeat/reconciliation intervals.

Why this order? The describe output narrows the layer before you log in. The service status shows whether the problem is "not running" or "running but unhealthy". The journal gives the precise error. Editing files before reading the log risks fixing the wrong thing.

Typical kubelet failures

Kubelet configuration lives in /var/lib/kubelet/config.yaml; startup flags and the kubeconfig path come from the systemd drop-in /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf (path may be /etc/systemd/system/... on some distributions). After editing a unit or drop-in, run systemctl daemon-reload and then systemctl restart kubelet; after editing only config.yaml, restart is enough.

systemctl, journalctl and crictl

systemctl manages services: status, start, stop, restart, enable, disable, is-active, and cat kubelet prints the effective unit with drop-ins.

journalctl reads logs of services: journalctl -u kubelet -f follows, -n 100 limits lines, --since "10 min ago" limits time, -p err shows only errors.

crictl talks directly to the CRI runtime, which is essential when the API server is down. Point it at the runtime with /etc/crictl.yaml (runtime-endpoint: unix:///run/containerd/containerd.sock).

crictl inspects the local runtime, not cluster-level controllers such as Deployments. CRI Pod sandboxes carry Kubernetes namespace metadata, and crictl pods displays a NAMESPACE column. Filter by Pod/container name or inspect sandbox metadata; a healthy runtime on this host says nothing about another node.

Control plane components

In a typical kubeadm cluster with stacked etcd, the API server, controller manager, scheduler and local etcd are static Pods: the kubelet reads manifests from /etc/kubernetes/manifests (the staticPodPath in the kubelet config) and runs them without the API server. Mirror Pods show them in kubectl get pods -n kube-system, with the node name suffix, for example kube-scheduler-cp1. The API mirror is not the authoritative workload specification. Editing or deleting a mirror Pod does not update or restart its static workload. Edit the local manifest to change that workload, then check kubelet/runtime reconciliation; timing is configurable.

Components and their typical symptoms:

Worked example: Pods stay Pending, scheduler broken

Scenario. A new Deployment's Pods are Pending, and kubectl describe pod shows no events at all (not even FailedScheduling).

Reasoning. The scheduler normally emits scheduling events, so inspect it when new unassigned Pods make no progress. No events alone is not proof: events can expire, access may be restricted, or a Pod may name a different scheduler. Check spec.nodeName, spec.schedulerName, events and component health before editing manifests.

1. kubectl get pods -n kube-system | grep scheduler shows kube-scheduler-cp1 0/1 CrashLoopBackOff.
2. kubectl logs -n kube-system kube-scheduler-cp1 shows unknown flag: --kubeconfigg (or unable to load kubeconfig: ... /etc/kubernetes/scheduler.conf).
3. On the control plane node edit /etc/kubernetes/manifests/kube-scheduler.yaml: fix the typo in the flag, or the path under hostPath volumes.
4. Save. The kubelet recreates the Pod; watch kubectl get pods -n kube-system -w.
5. The Pending Pods get scheduled. Verify with kubectl get pods -o wide.

Why not restart the Pod? Deleting the API mirror only makes kubelet recreate the mirror object; it does not fix or restart the underlying static Pod. The file is the source of truth.

Frequent injected faults in manifests

• A wrong flag name or value in command (typos, bad port).
• A wrong hostPath for certificates, kubeconfig or data directory (for example etcd pointing to /var/lib/etcd-wrong).
• A wrong image tag.
• YAML indentation errors; the kubelet ignores an unparsable manifest, so the Pod simply vanishes. Check journalctl -u kubelet for parse errors.
• The API server --etcd-servers pointing to the wrong port (etcd client port is 2379).
• The API server --service-cluster-ip-range or certificate paths changed.

A safe edit routine: copy the manifest to another directory outside /etc/kubernetes/manifests as a backup (not inside, or the kubelet would run it too), edit, then watch with crictl ps -a and crictl logs.

Comparison: where to look for which failure

Use Service and network troubleshooting when direct Pod IP, Service IP and DNS tests give different results. A node can report Ready while a particular application path is broken.

Worked example: classify client errors before repairing the server

Scenario. An engineer says the cluster is down because a kubectl command failed. A second authorised workstation can list namespaces. You must identify whether the first workstation has a transport, TLS, authentication or authorisation problem.

1. Record the exact command, timestamp, current context and endpoint. Run kubectl config current-context and kubectl config get-contexts; inspect the selected context locally without printing raw private keys or tokens into shared incident logs. A command targeting an old lab endpoint does not establish a production outage.
2. A connection-refused error means the TCP connection was rejected at that address; a timeout can involve routing, firewalls, a load balancer or an unresponsive process. Neither message alone proves etcd corruption. Compare the documented endpoint and an authorised healthy client before logging onto nodes.
3. An x509 error means certificate validation failed. Check the endpoint hostname, trust chain, certificate validity and clock. Do not disable TLS verification as a repair. An unauthorised response instead indicates missing or rejected client identity; a forbidden response normally means the request reached authorisation but that identity lacks the required permission.
4. If the API is reachable but a readiness request returns a failed check, read that check and component logs. A Forbidden response to /readyz may simply reflect permissions and is different from a readiness failure. Use an appropriately authorised diagnostic identity.
5. After a justified fix, repeat the original failing command and a representative operation. A successful command from a different context is not evidence that the original path is repaired. Preserve the error and the reason for the change so another operator can assess the result.

Why not the alternatives? Renewing every certificate will not fix a wrong server address. Restarting kubelet will not grant RBAC permission. Deleting an etcd data directory destroys evidence and risks cluster loss without addressing client configuration. The smallest correct repair follows the failing layer.

Pressure conditions, evictions, and OOM kills

Configured memory, disk-space/inode and process thresholds drive node pressure; there is no universal disk percentage such as 92% that always triggers it. Kubelet first tries to reclaim node resources. If eviction is necessary, selection depends on the starved resource, use exceeding requests, Pod priority and relative excess use; QoS is not a strict eviction queue.

Inspect kubectl describe node <node>, df -h, df -i, runtime logs and the relevant filesystem layout. For example, 30 GB of container logs can exhaust nodefs even if a separate imagefs has space. Check supported log rotation and stop the noisy source; use kubelet/runtime-supported garbage collection instead of deleting arbitrary runtime files. Removing images from another filesystem will not fix a full log filesystem. Preserve incident evidence before any approved cleanup.

Evicted Pods have phase Failed and reason Evicted; retain their status/log evidence until diagnosis is complete, then delete only the selected obsolete objects. An all-namespaces delete of every Failed Pod also removes failures unrelated to this incident. Controllers can create replacements, which still need sufficient healthy capacity.

OOMKilled means the container was killed by an out-of-memory mechanism; it can arise from a cgroup limit or a node-wide OOM. MemoryPressure=False at inspection time does not rule out an earlier or faster node OOM. Correlate timestamps, kernel/runtime logs, actual memory limits and usage; distinguish this from a kubelet eviction. Use resource monitoring and container output to collect evidence rather than assume that raising a limit is always the fix.

Node maintenance and recovery notes

• Cordon and drain. kubectl cordon <node> stops new scheduling. For planned maintenance, kubectl drain <node> --ignore-daemonsets uses eviction and can be blocked by PodDisruptionBudgets or unmanaged Pods. DaemonSets and mirror Pods remain; add --delete-emptydir-data only when loss of that Pod-local data is acceptable. Uncordon only after the node and workloads are healthy. A node that is Ready,SchedulingDisabled is cordoned, not broken.
• Taints from node problems. Unhealthy nodes get taints such as node.kubernetes.io/not-ready and node.kubernetes.io/unreachable, whose NoExecute effects can lead to eviction after matching tolerations expire. Many Pods default to 300 seconds for these two health taints, while DaemonSets normally tolerate them indefinitely; explicit policies can differ.
• Replace a node. First isolate/fence an unreachable worker, account for workloads and attached storage, and preserve diagnostics. Deleting its Node object does not power off the machine or stop old processes. For an approved replacement, drain when possible, remove the stale object and join a prepared worker using a fresh, protected join command from the administrator.
• Certificates. Kubelet client certificates are rotated automatically when rotateCertificates is true; diagnose expired control-plane certificates against the actual CA ownership before using kubeadm renewal. External CA-managed certificates need that CA's process; after appropriate renewal, restart affected static components and refresh copied administrator kubeconfigs where used. See cluster lifecycle, etcd recovery, and certificate maintenance.
• Container runtime health. systemctl status containerd and crictl info. If images cannot be pulled, also check proxy and registry settings.

Common pitfalls and exam tips

• Forgetting which node you are on. Many fixes require ssh to the right node; check hostname and prompt.
• Editing the wrong file. Static Pod manifests are /etc/kubernetes/manifests/*.yaml; kubelet settings are /var/lib/kubelet/config.yaml; the service drop-in is separate.
• Skipping daemon-reload after unit changes.
• Not reading the logs. journalctl -u kubelet -n 50 and crictl logs give the answer in seconds.
• Backups inside the manifests directory run as extra Pods or confuse the kubelet.
• Waiting too little. The kubelet may take 20 to 60 seconds to recreate a static Pod and for the node to become Ready.
• Assuming the API is available. If kubectl fails, use crictl.
• Exam tip: move quickly through a fixed checklist: describe node, systemctl status, journalctl, manifests; write down the namespace and node name in the task before starting; verify with kubectl get nodes and kubectl get pods -A when done.

Mini case study: Contoso's broken cluster after maintenance

After a weekend maintenance at Contoso, the on-call engineer finds:

1. kubectl get nodes hangs, then prints connection refused.
2. On cp1, systemctl status kubelet is active (running), but crictl ps -a lists kube-apiserver as Exited.
3. crictl logs <apiserver-id> ends with dial tcp 127.0.0.1:2379: connect: connection refused.
4. crictl ps -a | grep etcd shows etcd restarting. Inspection shows an unexpected empty data directory inside the container: the manifest mounts hostPath /var/lib/etcd-new at /var/lib/etcd, while the known existing member data is on the host at /var/lib/etcd. Assume this controlled lab uses one stacked etcd member, with a known-good original manifest and backup. A wrong empty directory can also lead to unintended initialisation, so do not assume one exact log message is inevitable.

Solution.

1. Preserve both directories and the manifest before changing anything. Confirm the expected member identity and volumeMount/hostPath/--data-dir relationship against the known-good configuration. Restore the incorrect hostPath to the existing member directory; do not delete member data, initialise a new cluster or overwrite it with an unrelated snapshot. For replicated etcd, follow a quorum- and membership-aware recovery plan instead of copying this single-member lab repair.
2. Wait for the kubelet to recreate etcd; check crictl ps for a running etcd, then the API server recovers by itself.
3. Verify etcd health and API readiness, then kubectl get nodes, kubectl get pods -A and application behaviour. A running container alone is not proof of a healthy datastore.
4. After the incident: store a copy of manifests outside the manifests directory and schedule an etcd backup (covered in the cluster lifecycle lesson).

Commentary. The failure looked like "the API server is down", but the root cause was two layers deeper. Following the log messages layer by layer found it quickly, without guessing.

Key terms

• Node condition: status reported by the kubelet (Ready, MemoryPressure, DiskPressure, PIDPressure).
• Kubelet: node agent that runs Pods and reports status.
• Static Pod: Pod defined by a file read directly by the kubelet.
• Mirror Pod: read-only API object representing a static Pod.
• crictl: CLI to inspect containers through the CRI.
• journalctl / systemctl: log reader and service manager of systemd.
• Cordon / drain: mark unschedulable / evict Pods.

Summary checklist

• Work in layers: client, nodes, control plane, node agents, workloads.
• kubectl describe node explains NotReady through conditions.
• systemctl status kubelet and journalctl -u kubelet find kubelet failures.
• Control plane components are static Pods in /etc/kubernetes/manifests; edit the file, not the Pod.
• crictl ps -a and crictl logs work when the API is down.
• Unassigned Pods without progress suggest checking the intended scheduler; missing ReplicaSets suggest checking Deployment reconciliation and its events.
• Run systemctl daemon-reload after unit changes.
• Verify with kubectl get nodes and kubectl get pods -A.

Official sources

• Troubleshooting clusters: https://kubernetes.io/docs/tasks/debug/debug-cluster/ (opens in a new tab)
• Troubleshooting kubeadm: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/ (opens in a new tab)
• Debugging Kubernetes nodes with crictl: https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/ (opens in a new tab)
• Create static Pods: https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/ (opens in a new tab)
• Node status: https://kubernetes.io/docs/reference/node/node-status/ (opens in a new tab)
• Certificate management with kubeadm: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/ (opens in a new tab)
• Swap memory management: https://kubernetes.io/docs/concepts/cluster-administration/swap-memory-management/ (opens in a new tab)
• Node-pressure eviction and OOM handling: https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/ (opens in a new tab)
• CKA domains and current version: https://training.linuxfoundation.org/certification/certified-kubernetes-administrator-cka/ (opens in a new tab)
• CKA curriculum: https://github.com/cncf/curriculum (opens in a new tab)