Skip to content
FR

Kubernetes Monitoring: Resource Usage, Logs, and Events

Step 21 of 24

CKA Kubernetes Monitoring: kubectl top, Logs, and Events

Introduction

Kubernetes monitoring with kubectl top, logs and events supports the Troubleshooting domain (30% of the CKA). This lesson covers resource usage and container output: how to identify a busy workload, collect evidence from a failed container and distinguish usage from reservations. Use documentation matching the Kubernetes version of your lab and exam sitting.

Original exercises below find the highest-usage Pod, save selected container output and investigate repeated restarts. They practise published competencies rather than predict exact exam tasks. Confirm context, namespace and output requirements before collecting evidence.

After this lesson you will be able to install and verify metrics-server, read kubectl top output correctly, sort and filter it, read logs of running, crashed and multi-container Pods, use events and describe to explain failures, and understand where container output actually goes.

Learning objectives

  • Explain how the resource metrics pipeline works and verify that metrics-server is healthy.
  • Use kubectl top node, kubectl top pod, --containers, --sort-by and -A effectively.
  • Read logs with kubectl logs, including --previous, -c, --all-containers, -f, --tail, --since and label selectors.
  • Use kubectl describe and kubectl get events to explain scheduling, image, probe and restart problems.
  • Describe where container stdout and stderr are stored and what log rotation does.
  • Know what to do when the API is unavailable (read logs on the node).

The resource metrics pipeline

Kubernetes does not ship with a metrics store. The kubelet exposes resource metrics, including through /metrics/resource, using runtime/CRI statistics or cAdvisor as appropriate to the runtime. The add-on metrics-server scrapes every kubelet (by default every 15 seconds), keeps the latest values only in memory, and publishes them in the API server as the Metrics API (metrics.k8s.io). Three consumers use it: kubectl top, the Horizontal Pod Autoscaler, and the Vertical Pod Autoscaler.

Important consequences:

  1. kubectl top needs a working Metrics API provider, normally metrics-server. A missing or unavailable provider can produce Metrics API not available.
  2. No durable history. Metrics-server provides recent resource measurements for autoscaling. Use a separate monitoring system with retention for trends and alerts; one top sample cannot establish a trend.
  3. Short delay. A new Pod may show no metrics for one or two minutes.
  4. Values are for autoscaling, not accounting. CPU is shown in millicores (250m = a quarter of a core) and memory in Mi, as the working set of the container, which can differ from what top on the node says.

Installing and verifying metrics-server

If installation is required, select an official metrics-server release compatible with the Kubernetes version and review its aggregation-layer, kubelet authentication, TLS and networking prerequisites. The following assumes the lab supplies that reviewed, version-pinned manifest locally:

Bash
kubectl apply -f ./metrics-server-components.yaml
kubectl get deployment metrics-server -n kube-system
kubectl get apiservice v1beta1.metrics.k8s.io

Check APIService availability, Deployment readiness and scrape errors; AVAILABLE=True alone does not prove metrics exist for every node. Follow the task instructions rather than assume installation is required. Prefer valid kubelet serving certificates trusted by metrics-server and matching the selected node addresses. The --kubelet-insecure-tls flag disables identity verification; it is a deliberate disposable-lab exception, not a required or general fix for self-signed certificates.

When kubectl top fails, work through the chain:

Symptom Likely cause Check
Metrics API not available metrics-server missing or APIService unavailable kubectl get apiservice v1beta1.metrics.k8s.io, kubectl get pods -n kube-system -l k8s-app=metrics-server
ServiceUnavailable for pods.metrics.k8s.io API server cannot reach metrics-server Pod (network, CNI, firewall) kubectl describe apiservice v1beta1.metrics.k8s.io, Pod logs
Some nodes show <unknown> metrics-server Pod cannot scrape that kubelet (TLS error, port 10250 blocked) kubectl logs -n kube-system deploy/metrics-server
metrics-server Pod Pending taints, resources, or no node matches kubectl describe pod
New Pods have no values normal scrape delay wait one to two minutes

kubectl top

Bash
kubectl top node
kubectl top pod
kubectl top pod -A
kubectl top pod -n batch --sort-by=cpu
kubectl top pod -n batch --sort-by=memory
kubectl top pod mypod --containers
kubectl top pod -l app=web

Sample output:

Text
NAME      CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
cp1       210m         10%      1450Mi          38%
worker-1  1840m        92%      3100Mi          81%
worker-2  95m          4%       900Mi           23%

Reading it correctly:

  • CPU(%) and MEMORY(%) for a node are relative to allocatable resources by default; --show-capacity changes the denominator to raw capacity.
  • For Pods, the figure is usage, not what the Pod requested. A Pod can use less than its request or more than its request (up to its limit).
  • --sort-by accepts only cpu or memory, and sorts in descending order.
  • --containers splits a Pod into its containers, which is how you find the noisy sidecar.
  • kubectl top shows nothing for Pods that have just been created or for Jobs that have already completed.

To compare usage with reservation, combine two views:

Bash
kubectl describe node worker-1      # "Allocated resources" shows requests and limits totals
kubectl top node worker-1           # shows live usage

A node may have 95% of allocatable CPU requested while usage is 10%; a new Pod is rejected by CPU-capacity filtering only if its effective request exceeds the remaining budget. CPU limits drive throttling, while node-pressure eviction considers resource-specific thresholds, requests, priority and usage. See Pod scheduling, requests, and HPA for those different decisions.

Worked example: find the heaviest Pod and record it

Scenario. The task: "In namespace shop, find the Pod with the highest memory use and write its name (only the name) to /opt/top-memory.txt."

Steps.

  1. kubectl top pod -n shop --sort-by=memory lists Pods, highest first.
  2. To avoid the header and extra columns: kubectl top pod -n shop --sort-by=memory --no-headers | head -1 | awk '{print $1}' > /opt/top-memory.txt.
  3. Verify: cat /opt/top-memory.txt shows a single Pod name such as cart-7d9c8b6f5-xk2lp.

Why this approach? The built-in sort removes manual comparison errors, --no-headers keeps the header line out of the answer, and checking the file confirms the format required by the task. Some choices are worse: kubectl get pods cannot show usage, and kubectl describe shows requests and limits, not consumption. Without -n, kubectl uses the namespace in the current context, falling back to default only if none is configured. Check for command failure or an empty result before accepting the generated answer file.

Container logs

Containers write to stdout and stderr. The container runtime captures both streams and the kubelet stores them as files on the node, under /var/log/pods/<namespace>_<pod>_<uid>/<container>/<n>.log, with symlinks in /var/log/containers/. kubectl logs asks the kubelet for those files through the API server. Kubernetes does not itself provide durable cluster-wide log retention. A logging agent can ship output to a separate store; collection and retention policies must be designed explicitly.

A container that writes logs only to a file inside its filesystem is invisible to kubectl logs. The 12-factor practice is to log to stdout/stderr; for legacy apps use a sidecar that tails the file and prints it to its own stdout.

The essential flags

Goal Command
Logs of a Pod kubectl logs mypod
A specific container kubectl logs mypod -c sidecar
All containers kubectl logs mypod --all-containers=true
Follow live kubectl logs -f mypod
Last N lines kubectl logs mypod --tail=50
Last period kubectl logs mypod --since=15m
With timestamps kubectl logs mypod --timestamps
Previous (crashed) instance kubectl logs mypod --previous (alias -p)
Init container kubectl logs mypod -c init-db
Through a Deployment kubectl logs deploy/web (picks one Pod)
By label, all matching Pods, all retained lines in the current log file kubectl logs -l app=web --all-containers --prefix --tail=-1
Save to a file kubectl logs mypod > /opt/mypod.log

Details that matter:

  • --previous returns the logs of the last terminated instance of a container. It is the key command for CrashLoopBackOff: the current container may have just started and have no useful output, while the previous one contains the crash message.
  • Multi-container Pods. Select -c explicitly. Current kubectl can use the default-container annotation or default to the first application container with a notice; behaviour and errors can vary across client versions and resource selection. Do not rely on an omitted container name to collect the intended output.
  • Label selectors. With a selector, --tail defaults to 10 lines per container, not the whole file; use --tail=-1 when appropriate. --max-log-requests defaults to 5 concurrent log-follow requests, not a five-Pod limit on all snapshot queries. --prefix identifies each source. Current kubectl also supports logs deploy/web --all-pods=true to select all its Pods.
  • Redirect, do not copy. Saving output with > avoids transcription mistakes; check the file with wc -l or head.

Worked example: a Pod in CrashLoopBackOff

Scenario. Pod api-6f7d in namespace prod shows CrashLoopBackOff with 7 restarts. Find the reason and save the last 20 lines of the failed instance to /opt/api-crash.txt.

Steps.

  1. kubectl get pod api-6f7d -n prod confirms CrashLoopBackOff.
  2. kubectl describe pod api-6f7d -n prod shows, under Last State, Terminated with Exit Code: 1 and Reason: Error (not OOMKilled), so the application itself exits.
  3. kubectl logs api-6f7d -n prod prints almost nothing because the new instance has just started.
  4. kubectl logs api-6f7d -n prod -c app --previous --tail=20 > /opt/api-crash.txt captures the crash. The file ends with FATAL: environment variable DATABASE_URL is not set.
  5. The message identifies a missing environment variable. Correct the intended Deployment configuration in prod, preferably referencing the appropriate Secret for credentials. Review Deployments, ConfigMaps, and Secrets for how environment changes reach replacement containers.

Why not the other options? kubectl logs -f would only follow the new instance and would stop at the next crash. kubectl exec fails because the container is not running. Exit code, reason, events and prior logs help distinguish application exits, OOM termination and probe-triggered restarts; none of these clues alone identifies every underlying cause.

Logs of control plane and node components

Control plane components of a kubeadm cluster are Pods, so kubectl logs -n kube-system kube-apiserver-cp1 works. When the API is down use crictl logs <id> (previous lesson). The kubelet and containerd are systemd services: journalctl -u kubelet, journalctl -u containerd. Static Pod log files also exist directly: ls /var/log/pods/.

Rotation and retention

The kubelet configures log rotation with containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5). Only the latest log file is exposed through kubectl logs; retained rotated files are not concatenated by that command. --previous selects the previous container instance if it still exists, not a deleted Pod or a complete history. Node-local files can disappear through Pod deletion/eviction or garbage collection; after the API Pod is gone, kubectl logs cannot retrieve its output even if forensic remnants briefly remain. Use a separate durable logging system for reliable history.

Events and describe

kubectl describe assembles the object spec, its status and its events. Events are Kubernetes objects that record what controllers, the scheduler and the kubelet did, such as Scheduled, Pulling, Failed, BackOff, Unhealthy. The API server event TTL defaults to one hour and is configurable; events are not a durable audit log.

Bash
kubectl describe pod mypod
kubectl get events -n shop --sort-by=.metadata.creationTimestamp
kubectl get events -A --field-selector type=Warning
kubectl get events --field-selector involvedObject.name=mypod
kubectl events -n shop --for pod/mypod

(kubectl get events does not sort by time unless you ask.) Common event messages and their meaning:

Event / state Meaning Typical cause and fix
FailedScheduling Scheduler found no node Insufficient CPU or memory, taints, selector or affinity, unbound PVC
ErrImagePull / ImagePullBackOff Image cannot be pulled Wrong name or tag, private registry without imagePullSecrets
CrashLoopBackOff Container keeps exiting Read --previous logs and exit code
OOMKilled (termination reason) OOM mechanism killed the container; cgroup limit or node OOM is possible Correlate timestamps, limits, usage and kernel logs before choosing a fix
Unhealthy Probe failed Wrong port or path, slow start: adjust the probe
FailedMount Volume cannot be mounted Missing Secret, ConfigMap or PVC, wrong claim
Evicted Node pressure Free resources, see previous lesson
CreateContainerConfigError Reference to a missing ConfigMap or Secret key Create or correct the reference

Container states in describe: Waiting (with reason), Running, Terminated (with reason, exit code, start and finish time). Last State shows the previous termination and Restart Count the number of restarts.

Worked example: Pending Pod with a clear reason

Scenario. Pod report-0 is Pending. kubectl describe pod report-0 ends with: Warning FailedScheduling ... 0/3 nodes are available: 3 Insufficient memory.

Reasoning. The scheduler considers requests, not usage. kubectl top node might show the nodes mostly idle, but kubectl describe node shows "Allocated resources" with memory requests at 95% on every node. The Pod requests 6Gi, which no node can still reserve.

Solutions. Lower the request if it is oversized, free capacity by removing or right-sizing other workloads, or add a node. You do not fix this by restarting anything.

kubectl debug and exec for output you cannot see

Sometimes logs are not enough. Useful tools:

  • kubectl exec -it mypod -- sh to run commands in a running container, with -c to select a container.
  • kubectl debug mypod -it --image=busybox:1.36 --target=app adds an ephemeral debugging container with its own tools. It uses the Pod network namespace; targeting the app process namespace requires runtime support and does not automatically mount the app filesystem. RBAC and admission policy must permit it.
  • kubectl debug node/worker-1 -it --image=busybox:1.36 can start a node-debug Pod with the host filesystem mounted at /host, subject to access and admission controls. Host-file access still depends on permissions and the chosen debug profile.
  • kubectl cp mypod:/var/log/app.log ./app.log copies a file out (requires tar in the image).

Comparison: which tool answers which question

Question Tool
How much CPU or memory is used right now? kubectl top
How much is reserved and what is left to schedule? kubectl describe node (Allocated resources)
What did the application print? kubectl logs
Why did the previous container die? kubectl logs --previous and describe (Last State)
What did Kubernetes do to this object? describe events, kubectl get events
What happens inside the container? kubectl exec, kubectl debug
Is the metrics pipeline healthy? kubectl get apiservice v1beta1.metrics.k8s.io

Common pitfalls and exam tips

  • Wrong namespace. kubectl top pod and kubectl logs use the current namespace; use -n for logs and Pod metrics, or -A for metrics across namespaces. Many failed answers are simply the wrong namespace.
  • Forgetting -c. Explicitly select the desired container even when kubectl chooses a default.
  • Reading the current logs of a crash loop. Use --previous.
  • Header line in the answer file. Use --no-headers when the output must contain only names.
  • Mixing up usage and requests. top is usage; describe node shows reservations; scheduling uses requests.
  • Expecting instant metrics. Wait a minute after creating Pods.
  • Expired events. If events are gone, reproduce the failure or rely on logs and state.
  • Exam tip: set the output with a redirect (>), then cat the file to confirm. Use kubectl explain or --help for flags rather than memory; the kubernetes.io documentation is allowed during the exam.
  • Exam tip: kubectl logs and kubectl top are cheap. Use them before changing anything.

Mini case study: the slow checkout at Fabrikam

Fabrikam's checkout Deployment (3 replicas, namespace shop) is slow and sometimes restarts. The on-call engineer collects:

  1. kubectl top pod -n shop --sort-by=cpu shows checkout-5c9-a at 980m, while its siblings show about 120m.
  2. kubectl get pod -n shop shows checkout-5c9-a with 4 restarts.
  3. kubectl describe pod checkout-5c9-a -n shop shows Limits: cpu 1, memory 512Mi, and Last State: Terminated, Reason: OOMKilled, Exit Code: 137.
  4. kubectl top pod checkout-5c9-a -n shop --containers shows the container app at 470Mi and the sidecar log-shipper at 9Mi.
  5. kubectl logs checkout-5c9-a -n shop -c app --previous --tail=30 ends with repeated cache warm-up messages.
  6. In this lab, the timestamped kernel/cgroup record confirms the app hit its own memory limit; node-wide OOM is ruled out by the correlated evidence, not by one top snapshot.

Solution.

  1. The Pod is the CPU outlier; the correlated cgroup evidence identifies a 512Mi limit OOM. OOMKilled and current moderate node usage alone would not establish that conclusion.
  2. The previous logs point to a cache warm-up loading too many entries only on this replica (for example because it received a large tenant's traffic).
  3. If measurements and node capacity justify a temporary 1Gi limit, update the intended resource configuration, including an appropriate memory request, and watch the rollout. Confirm existing CPU settings and sidecar resources remain as intended; a higher limit does not fix the underlying leak or guarantee no restarts.
  4. Verify: kubectl top pod -n shop shows balanced values and kubectl get pod -n shop shows restart counts stable at 0 for the new Pods.
  5. Longer term: fix the warm-up logic and set requests close to real usage, since only a request lets the scheduler place the Pod correctly.

Commentary. The investigation combines usage, state, prior output and host evidence. OOMKilled identifies an OOM termination, while the cgroup record establishes which boundary was hit. Use cluster and node troubleshooting when the evidence points to host pressure or component failure.

Key terms

  • metrics-server: add-on that collects current CPU and memory from kubelets and serves the Metrics API.
  • Metrics API (metrics.k8s.io): API consumed by kubectl top and autoscalers.
  • cAdvisor: container statistics collector built into the kubelet.
  • stdout / stderr: output streams captured by the container runtime.
  • --previous: logs of the last terminated container instance.
  • Event: short-lived record of an action or problem related to an object.
  • Ephemeral container: temporary debugging container added to a running Pod.
  • OOMKilled: OOM termination; distinguish a container-limit event from node-wide OOM.

Summary checklist

  • kubectl top needs metrics-server and a healthy v1beta1.metrics.k8s.io APIService.
  • kubectl top pod -A --sort-by=cpu|memory finds heavy consumers; --containers splits Pods.
  • kubectl logs supports -c, -f, --tail, --since, --previous, -l, --all-containers.
  • Use --previous and describe (Last State, exit code) for crash loops.
  • Events explain scheduling, image, probe and mount failures; they expire after about an hour.
  • Logs live on the node under /var/log/pods and are rotated by the kubelet.
  • Usage (top) differs from requests (describe node).
  • Redirect output to the requested file and verify it.

Official sources