Kubernetes Pod Scheduling, Admission, and HPA for CKA

Introduction

Kubernetes Pod scheduling and Horizontal Pod Autoscaling (HPA) belong to the Workloads & Scheduling domain (15% of the CKA). Learn how admission decides whether a Pod is accepted, how resource requests and placement rules determine its node, and how an HPA adjusts replicas. Original practice tasks include selecting SSD nodes, applying namespace quotas, isolating batch placement and targeting 60% CPU utilisation. These exercises illustrate published competencies rather than predicting exact exam tasks.

The official exam page currently lists Kubernetes v1.37; confirm your sitting's version and match the documentation to your cluster. Private registry images in snippets must be supplied by the lab.

After this lesson you will be able to set requests and limits and predict the resulting QoS class, constrain consumption with LimitRanges and ResourceQuotas, steer Pods with node selectors, affinity, anti-affinity, taints and tolerations, create a Horizontal Pod Autoscaler, and apply Pod Security admission to a namespace.

Learning objectives

• Explain how the scheduler uses resource requests (not limits) to place Pods, and derive a Pod's QoS class.
• Create LimitRange and ResourceQuota objects and predict which requests they reject or default.
• Place Pods with nodeSelector, node affinity, Pod affinity and anti-affinity, and topology spread constraints.
• Use taints and tolerations (NoSchedule, PreferNoSchedule, NoExecute) correctly.
• Create and read a Horizontal Pod Autoscaler and know its prerequisites.
• Apply Pod Security Standards with namespace labels.

Resource requests and limits

Every container may declare resources.requests and resources.limits for CPU and memory.

• For CPU and memory capacity checks, the default scheduler compares effective requests with node allocatable capacity and existing allocations, not live usage. Taints, affinity, volumes and other constraints also affect placement.
• The kubelet and container runtime enforce limits. CPU limits cause throttling; memory limits are enforced reactively and can result in an OOM kill. Inspect the termination reason (OOMKilled) as well as the exit code: 137 alone also occurs for other SIGKILL terminations.

CPU is expressed in cores or millicores (500m = half a core); memory in bytes with suffixes (256Mi, 1Gi). If a limit is set without a request and no admission default supplies one, Kubernetes copies the limit into the request. The following examples use container-level resources; clusters supporting Pod-level resources also require checking spec.resources, init-container requests and Pod overhead when calculating effective demand.

apiVersion: v1
kind: Pod
metadata:
  name: api
  namespace: shop
spec:
  containers:
  - name: api
    image: registry.example.com/api:3.2
    resources:
      requests: { cpu: 250m, memory: 256Mi }
      limits:   { cpu: 500m, memory: 512Mi }

A Pod that requests more than any node can offer stays Pending, and kubectl describe pod shows an event like 0/3 nodes are available: 3 Insufficient cpu. Check capacity with kubectl describe node <node> (sections Allocatable and Allocated resources).

QoS classes

Kubernetes derives a Quality of Service class at Pod creation. For these container-level examples without Pod-level resource settings:

See it with kubectl get pod api -n shop -o jsonpath='{.status.qosClass}'. QoS is not a strict eviction queue: the kubelet ranks usage exceeding requests, Pod priority and relative excess usage. Guaranteed Pods are not immune to node failure, system pressure or their own memory limits.

Container-level in-place Pod resize has been stable since v1.35. The resize subresource can adjust CPU/memory without replacing the Pod, subject to node capacity and resize policies; a container restart can still be necessary. For controller-managed workloads, also update the template so replacements retain the intended settings. Use the version-specific resize documentation rather than assuming that every resource or QoS change is allowed.

Worked example: why is my Pod Pending?

Scenario. A three-node cluster has 4 allocatable CPUs per node. Existing Pods already request 3.5 CPUs on each node. You create a Pod requesting 1 CPU with a limit of 2 CPUs.

Reasoning.

1. The scheduler filters nodes by requests only: free CPU per node is 4 - 3.5 = 0.5 CPU, which is less than the 1 CPU requested.
2. No node passes the filter, so the Pod remains Pending with Insufficient cpu.
3. Actual usage is irrelevant: the nodes may be nearly idle, but requests are reservations.

Solution. Add capacity or free reservations by safely resizing or relocating other workloads. Lower this Pod's request to 500m only if measurements show that amount satisfies its needs; reducing a declared request does not create real capacity.

Why other choices are worse. Raising the limit does nothing for scheduling. Deleting the Pod's limit changes nothing either, because the request is what failed. Cordoning nodes would reduce capacity further.

LimitRange and ResourceQuota

Two namespace-scoped objects control allocations through admission control. Changing these policies does not retroactively resize or evict existing Pods; admission checks can reject later create or relevant update requests. Quotas track declared allocations and object counts, not measured CPU/memory usage.

A LimitRange sets per-container (or per-Pod, per-PVC) minimums, maximums and defaults:

apiVersion: v1
kind: LimitRange
metadata:
  name: container-limits
  namespace: shop
spec:
  limits:
  - type: Container
    default:        { cpu: 500m, memory: 256Mi }   # default limit
    defaultRequest: { cpu: 100m, memory: 128Mi }   # default request
    max:            { cpu: "1",  memory: 1Gi }
    min:            { cpu: 50m,  memory: 64Mi }

Containers that omit requests and limits receive the defaults; containers above max or below min are rejected with a Forbidden error.

A ResourceQuota caps the aggregate use of a namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: shop-quota
  namespace: shop
spec:
  hard:
    pods: "10"
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    persistentvolumeclaims: "5"

Imperative creation is quick: kubectl create quota shop-quota --hard=pods=10,requests.cpu=4,requests.memory=8Gi -n shop. Inspect consumption with kubectl describe quota -n shop.

An important interaction: CPU/memory quotas require effective values for the resources being counted. Each relevant container must supply them explicitly or receive them through defaulting; otherwise Pod admission can fail with an error such as must specify requests.cpu. A LimitRange can fill in omitted requests and limits. When a ReplicaSet cannot create a Pod because quota would be exceeded, inspect its admission failure with kubectl describe rs.

Comparison

Placing Pods on nodes

nodeName and nodeSelector

spec.nodeName bypasses the scheduler and assigns the Pod to that node. It bypasses scheduler checks, including NoSchedule filtering; kubelet admission and NoExecute taints can still prevent continued execution. Static Pods are separately managed by the kubelet from local manifests. The standard simple method is a nodeSelector matching node labels:

kubectl label node worker-2 disk=ssd
kubectl get nodes -L disk

spec:
  nodeSelector:
    disk: ssd

If no node carries the label the Pod stays Pending with didn't match Pod's node affinity/selector.

Node affinity

Node affinity expresses richer rules with set-based operators (In, NotIn, Exists, DoesNotExist, Gt, Lt). Two flavours exist:

• requiredDuringSchedulingIgnoredDuringExecution: a hard rule, the Pod is not scheduled unless it matches.
• preferredDuringSchedulingIgnoredDuringExecution: a soft preference with a weight from 1 to 100.

"IgnoredDuringExecution" means that if the node's labels later change, running Pods are not evicted.

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["eu-west-1a", "eu-west-1b"]
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
          - key: disk
            operator: In
            values: ["ssd"]

Multiple nodeSelectorTerms are ORed; multiple matchExpressions inside a term are ANDed.

Pod affinity and anti-affinity

These rules place a Pod relative to other Pods, within a topology domain defined by topologyKey (for example kubernetes.io/hostname for per-node or topology.kubernetes.io/zone for per-zone).

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels: { app: web }
        topologyKey: kubernetes.io/hostname

Applied to every app=web replica, this prevents two matching Pods in the same namespace from sharing a node. The default namespace scope is the incoming Pod's namespace; use namespaces or namespaceSelector when a rule must span namespaces. With three replicas and two nodes, the third Pod stays Pending because the rule is required; use preferred for best-effort spreading. Pod affinity (podAffinity) does the opposite, for example co-locating a cache Pod with the web Pod that uses it.

Topology spread constraints (topologySpreadConstraints with maxSkew, topologyKey, whenUnsatisfiable) give even distribution across zones or nodes with a tolerance and are usually a better fit than anti-affinity for large replica counts.

Taints and tolerations

Affinity attracts a Pod to nodes. A taint does the reverse: it repels Pods from a node unless they carry a matching toleration.

kubectl taint nodes worker-3 dedicated=batch:NoSchedule
kubectl taint nodes worker-3 dedicated=batch:NoSchedule-   # trailing dash removes it
kubectl describe node worker-3 | grep -i taint

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: batch
    effect: NoSchedule

Kubeadm normally taints control-plane nodes with node-role.kubernetes.io/control-plane:NoSchedule. The control-plane node controller adds health taints such as node.kubernetes.io/not-ready and node.kubernetes.io/unreachable. Most Pods receive automatic 300-second NoExecute tolerations unless explicitly overridden; DaemonSet Pods get indefinite tolerations for these two health taints. A separate taint-eviction controller handles taint-based eviction in current Kubernetes.

A toleration only permits, it does not attract. To dedicate a node to batch Pods you need both a taint (keeps others out) and a nodeSelector or affinity (keeps batch Pods on it).

Worked example: dedicate a node to the batch team

Task. Reserve node worker-3 for new reports Pods in namespace batch. Assume only the approved workload may use the matching toleration, no other node has team=batch, and existing non-batch workloads have been safely relocated.

Steps.

1. Taint to repel others: kubectl taint nodes worker-3 team=batch:NoSchedule.
2. Label to attract: kubectl label node worker-3 team=batch.
3. Edit the Deployment template to add a toleration for team=batch:NoSchedule and nodeSelector: {team: batch}.
4. Verify with kubectl get pods -n batch -o wide and kubectl get pods -A -o wide --field-selector spec.nodeName=worker-3. NoSchedule does not remove existing Pods. Taints are placement controls, not tenant security boundaries: another authorised workload could request the same toleration. Restrict permissions and enforce allowed tolerations with admission policy where necessary.

Why not the alternatives. Only a taint: the batch Pods may run elsewhere. Only a nodeSelector: other teams' Pods can still land on worker-3. Cordoning blocks ordinary scheduling of these batch Pods too; special tolerations or direct node assignment can bypass that behaviour.

Horizontal Pod Autoscaler

A HorizontalPodAutoscaler (HPA, autoscaling/v2) adjusts the replica count of a Deployment, StatefulSet or ReplicaSet based on observed metrics. Prerequisites:

1. For this CPU-resource example, the resource metrics API (metrics.k8s.io) is available, usually supplied by metrics-server; check kubectl top pods -n shop. HPA custom/external metrics require their own adapters.
2. Every relevant container in the target Pods declares a CPU request, because CPU utilisation is measured against requests. CPU limits are not the denominator.

Readiness and startup probes from Kubernetes Deployments and application configuration help the controller avoid misleading startup metrics.

kubectl autoscale deployment web --cpu-percent=60 --min=2 --max=8 -n shop
kubectl get hpa -n shop
kubectl describe hpa web -n shop

The equivalent manifest:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
  namespace: shop
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
  minReplicas: 2
  maxReplicas: 8
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 60 }
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

The controller's sync period defaults to 15 seconds and is configurable. For complete, valid metrics, the basic recommendation is:

desiredReplicas = ceil( currentReplicas × currentMetric / targetMetric )

For example, 4 replicas averaging 90% CPU against a 60% target gives ceil(4 × 90 / 60) = 6. Actual action also depends on tolerance, readiness, missing metrics, min/max bounds and scaling policies. For downscaling, the controller keeps the highest recommendation in the stabilisation window (default 300 seconds); it is not simply a mandatory timer started by each low metric.

If kubectl get hpa shows <unknown>/60% in TARGETS, the usual causes are a missing metrics-server, no CPU request on the containers, or Pods that are not yet ready. Do not manage replicas manually with a fixed replicas in a manifest that you re-apply, because it fights the HPA.

The Vertical Pod Autoscaler and Cluster Autoscaler are separate add-ons: the first adjusts requests and the second adds nodes when Pods are unschedulable. Distinguish their roles and follow the task requirements rather than assuming a particular autoscaler will be requested.

Worked example: calculating HPA replicas

Scenario. The web Deployment has requests of 200m CPU per Pod and an HPA with target 50% utilisation, min 2, max 10. There are currently 3 Pods using 180m, 160m and 140m.

Reasoning.

1. Average usage = (180 + 160 + 140) / 3 = 160m, which is 160 / 200 = 80% of the request.
2. desired = ceil(3 × 80 / 50) = ceil(4.8) = 5.
3. The basic recommendation is 5, within min 2 and max 10. With all three Pods ready, complete metrics and no policy restriction, the HPA can scale to 5.

Why other answers are wrong. Using limits instead of requests changes the percentage and is not how the HPA computes it. Dividing by Pod count without rounding up (4) would under-provision, as the formula uses ceil.

Pod admission and Pod Security Standards

Admission controllers are plugins in the API server that validate or mutate requests after authentication and authorisation. LimitRanger, ResourceQuota, ServiceAccount and PodSecurity are examples; the enabled list is controlled by the API server flags (--enable-admission-plugins, in the static Pod manifest on kubeadm clusters).

Pod Security Admission enforces the three Pod Security Standards through namespace labels:

Each level can be applied in three modes: enforce (reject), audit (record in the audit log) and warn (show a warning to the user).

kubectl label namespace payments \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  pod-security.kubernetes.io/warn=restricted --overwrite

latest follows the policy version of the API server; pin a supported minor version when you need a stable policy during upgrades. Linux restricted policy permits adding back only NET_BIND_SERVICE; an approved Localhost seccomp profile is also valid. These fields are only part of the standard: image user, volumes and the full Pod spec must comply.

A Pod that violates the level is rejected with an error such as violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false. A compliant container spec contains:

securityContext:
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  capabilities: { drop: ["ALL"] }
  seccompProfile: { type: RuntimeDefault }

Applying enforce to a namespace does not evict existing Pods; it only affects creation and update of Pods. Workload resources such as Deployments are accepted, and the failure shows up in ReplicaSet events when Pod creation is refused. Use warn first when you are not sure, then switch to enforce. Kubernetes RBAC and ServiceAccount permissions govern who can submit or change workloads; Pod Security checks what those workloads may request.

Common pitfalls and exam tips

• Limits do not schedule. Check effective requests, taints, selectors, affinity and storage. A limit can indirectly increase a request when the request is defaulted from it.
• A toleration is not an attraction. Combine with a nodeSelector or affinity for dedicated nodes.
• NoExecute evicts. Use NoSchedule unless you really want running Pods removed.
• Quota without defaults. If a ResourceQuota tracks CPU or memory, Pods without values are rejected; add a LimitRange or set the resources.
• CPU-utilisation HPA needs CPU requests and resource metrics. Inspect HPA conditions and events when targets are <unknown>. Pod connectivity and Services must also be healthy for scaled replicas to carry useful traffic.
• Selectors are case sensitive and exact; confirm labels with kubectl get nodes --show-labels.
• Anti-affinity required with too few nodes leaves replicas Pending.
• Exam tip: read the question for the context and namespace, make the change imperatively where possible (kubectl taint, kubectl label, kubectl autoscale, kubectl create quota), and always verify with kubectl get ... -o wide or describe. kubectl explain pod.spec.affinity --recursive prints the exact field names when you forget the YAML structure.

Mini case study: the shared cluster at Northwind Traders

Northwind Traders runs a three-node cluster for two teams. Requirements:

1. Namespace analytics may use at most 6 CPUs of requests and 12 Gi of memory requests, and no more than 20 Pods. Containers without resources should get 200m CPU and 256Mi memory as requests.
2. Reserve worker-3 (GPU) for newly scheduled approved analytics workloads; relocate existing incompatible Pods safely.
3. The ingest Deployment in analytics should scale between 2 and 6 replicas at 70% CPU.

Solution.

1. Create the quota: kubectl create quota analytics-quota --hard=pods=20,requests.cpu=6,requests.memory=12Gi -n analytics. Because the quota tracks requests, add a LimitRange with defaultRequest: {cpu: 200m, memory: 256Mi} (and a default limit if desired) so Pods that omit resources are admitted instead of rejected.
2. Taint and label: kubectl taint nodes worker-3 gpu=true:NoSchedule and kubectl label node worker-3 gpu=true. In the analytics Pod templates that need a GPU, add the matching toleration and nodeSelector: {gpu: "true"}. Ordinary Pods lacking the toleration will not newly schedule there. This assumes users cannot bypass the placement policy with their own tolerations or direct assignment; use admission controls for enforceable tenant isolation.
3. Verify that ingest requests CPU, then run kubectl autoscale deployment ingest --cpu-percent=70 --min=2 --max=6 -n analytics. Confirm with kubectl get hpa -n analytics that TARGETS shows a percentage and not <unknown>.

Commentary. The three controls act at different layers: quota and LimitRange at admission, taints and selectors at scheduling, and the HPA at runtime. Missing any one of the pairings (taint + selector, quota + defaults) is the classic cause of lost points.

Key terms

• Request / limit: reserved amount used by the scheduler / ceiling enforced at runtime.
• QoS class: Guaranteed, Burstable or BestEffort; useful context for resource pressure, not a strict eviction order.
• LimitRange / ResourceQuota: per-object constraints and defaults / namespace totals.
• Node affinity / Pod (anti-)affinity: placement rules based on node labels / on other Pods.
• Taint / toleration: node repels Pods / Pod accepts a taint.
• HPA: Horizontal Pod Autoscaler adjusting replica counts from metrics.
• Pod Security Admission: namespace-label based enforcement of privileged, baseline and restricted standards.

Summary checklist

• The scheduler uses requests; limits are enforced at runtime (throttling, OOMKilled).
• QoS: Guaranteed (requests = limits), Burstable, BestEffort.
• LimitRange defaults and bounds; ResourceQuota allocation totals; policy changes do not retroactively resize existing Pods.
• nodeSelector, node affinity (required or preferred), Pod affinity and anti-affinity with topologyKey.
• Taint effects: NoSchedule, PreferNoSchedule, NoExecute; tolerations only permit.
• CPU-utilisation HPA needs resource metrics and requests; its basic formula uses ceil, with readiness, tolerance and behaviour adjustments.
• Pod Security labels: enforce, audit, warn with privileged, baseline or restricted.

Official sources

• Resource management for Pods and containers: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ (opens in a new tab)
• Limit Ranges: https://kubernetes.io/docs/concepts/policy/limit-range/ (opens in a new tab)
• Resource Quotas: https://kubernetes.io/docs/concepts/policy/resource-quotas/ (opens in a new tab)
• Assigning Pods to Nodes: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ (opens in a new tab)
• Taints and Tolerations: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ (opens in a new tab)
• Horizontal Pod Autoscaling: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ (opens in a new tab)
• Pod Security Admission: https://kubernetes.io/docs/concepts/security/pod-security-admission/ (opens in a new tab)
• In-place container resize: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/ (opens in a new tab)
• Node-pressure eviction: https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/ (opens in a new tab)
• Pod Security Standards: https://kubernetes.io/docs/concepts/security/pod-security-standards/ (opens in a new tab)
• CKA domains and current exam version: https://training.linuxfoundation.org/certification/certified-kubernetes-administrator-cka/ (opens in a new tab)
• CKA curriculum: https://github.com/cncf/curriculum (opens in a new tab)