Skip to content
FR

Kubernetes Network Troubleshooting: Services, DNS, and Routes

Step 23 of 24

CKA Kubernetes Network Troubleshooting: Services, DNS, and Gateway API

Introduction

Kubernetes network troubleshooting traces a request through Pods, Services, EndpointSlices, DNS, policy and external routing. It completes this course's Troubleshooting domain (30% of the CKA). Original scenarios below isolate selector, port, readiness and route-attachment faults. A symptom is evidence to investigate, not proof of a single cause.

Build on Pod connectivity, Services, and CoreDNS and NetworkPolicy, Ingress, and Gateway API routing. Examples assume a Linux IPv4 lab with domain cluster.local; adjust for the real DNS domain, CNI and Service dataplane.

After this lesson you will be able to trace a failing request from client to Pod, repair Service selector and port errors, read Endpoints and EndpointSlices, debug CoreDNS and DNS resolution from a Pod, identify NetworkPolicies and CNI faults, and diagnose Ingress and Gateway routing problems.

Learning objectives

  • Follow a systematic path: Pod, Service, Endpoints, DNS, policy, node networking, external entry.
  • Diagnose Services with no endpoints (selector, label, readiness, port mismatches).
  • Test connectivity and name resolution from inside the cluster with a temporary Pod.
  • Troubleshoot CoreDNS, resolv.conf and ndots behaviour.
  • Recognise a NetworkPolicy block and a missing or broken CNI.
  • Diagnose Ingress and Gateway API routes (class, backend, status conditions).

The path of a request

Think of the traffic as a chain; test each link from the inside out.

Step Question How to test
1. Pod Is the application running and listening on the port? kubectl get pod, kubectl logs, kubectl exec ... -- ss -ltn
2. Direct Pod IP Can another Pod reach the Pod IP and port? kubectl run tmp --rm -it --image=busybox -- wget -qO- http://<podIP>:<port>
3. Service Does the Service select the Pod and map the right port? kubectl get svc, kubectl describe svc, kubectl get endpointslices -o yaml
4. DNS Does the name resolve to the ClusterIP? nslookup web.shop.svc.cluster.local from a Pod
5. Policy Does a NetworkPolicy allow the flow? kubectl get netpol -n <ns>, describe
6. Node and CNI Do Pods on different nodes communicate? kube-proxy rules present? kubectl get pods -n kube-system, CNI Pod logs
7. External entry Does NodePort, LoadBalancer, Ingress or Gateway route to the Service? kubectl describe ingress, controller logs

If the same application works directly by Pod IP but not Service IP, inspect selectors, ports, endpoint conditions, traffic policies and the actual Service dataplane. This narrows the search without proving a kube-proxy failure. A name-resolution error with a working Service IP suggests DNS or the requested name/namespace; different HTTP Host headers or TLS names can also change application behaviour.

Services and EndpointSlice conditions

A Service of type ClusterIP selects Pods by label and forwards the Service port to the Pod targetPort. The control plane maintains EndpointSlices containing matching addresses, ports and conditions, including unready endpoints. Normal Service routing avoids endpoints with ready: false; publishNotReadyAddresses and terminating-endpoint handling are exceptions. The legacy Endpoints API is deprecated; prefer EndpointSlices. A Service with an empty endpoint list sends traffic nowhere, and clients see timeouts or connection refused.

Bash
kubectl get svc web -n shop -o wide
kubectl describe svc web -n shop
kubectl get endpointslices -n shop -l kubernetes.io/service-name=web
kubectl get endpointslices -n shop -l kubernetes.io/service-name=web -o yaml
kubectl get pods -n shop --show-labels
kubectl get pods -n shop -l app=web

The usual faults

Observation Cause Fix
Endpoints: <none> Selector matches no Pod (typo, wrong label) Compare Selector: with --show-labels; fix the Service or the Pod template labels
Slice addresses exist but conditions.ready is false Matching Pods are not Ready or are terminating Inspect readiness, serving and terminating conditions; fix the cause rather than just the listing
Endpoints exist, connection refused targetPort is wrong or the app listens on another port or on 127.0.0.1 only Check ss -ltn in the container; correct targetPort
Works by Pod IP, not through Service Selector or port mismatch, or kube-proxy problem describe svc, kube-proxy logs
Named targetPort not found Container port name differs Align the names
Service in another namespace not found Short name used across namespaces Use web.shop or web.shop.svc.cluster.local
NodePort unreachable externally Firewall, wrong node port, NetworkPolicy Test from a node, check security rules
LoadBalancer stays pending Missing/unready provider integration or failed provisioning Inspect the intended load-balancer controller and events; bare-metal controllers can fulfil Services without a cloud provider

The Service selector must match Pod labels, not the Deployment's labels. A frequent injected fault is a Deployment where spec.selector.matchLabels and the Pod template labels agree, but the Service selects app=webapp while the Pods carry app=web.

Worked example: Service with no endpoints

Scenario. In namespace shop, clients of Service web (port 80) time out. The Deployment web has 3 Pods Running.

Steps.

  1. EndpointSlices selected by kubernetes.io/service-name=web have no addresses.
  2. kubectl describe svc web -n shop shows Selector: app=webapp, Port: 80/TCP, TargetPort: 8080/TCP.
  3. kubectl get pods -n shop --show-labels shows the Pods labelled app=web, pod-template-hash=.... The selector is wrong.
  4. Patch the Service: kubectl patch svc web -n shop -p '{"spec":{"selector":{"app":"web"}}}'.
  5. kubectl get endpointslices -n shop -l kubernetes.io/service-name=web -o yaml now shows three IPs on 8080 with ready conditions. Assume the Pods are ready and the application listens on 8080.
  6. Test: kubectl run tmp -n shop --rm -it --image=busybox --restart=Never -- wget -qO- -T 3 http://web. The page returns.

Why this order? Missing addresses narrow the selector/Pod-discovery problem; present but unready addresses require a different readiness investigation. Fix the intended Service selector without changing an immutable Deployment selector. Direct Pod-label edits do not change the controller template and can disrupt membership; verify connectivity after the selector fix because DNS or policy faults may still coexist.

Testing from inside the cluster

Your laptop or the control plane host usually cannot reach Pod or Service IPs the same way a Pod can, so test from a Pod:

Bash
kubectl run tmp --rm -it --image=busybox:1.36 --restart=Never -- sh
# inside:
wget -qO- -T 3 http://web.shop
nslookup web.shop.svc.cluster.local

For richer tools, use an approved diagnostic image that supplies curl/dig, pinned according to the lab's image policy. With kubectl debug you can attach an ephemeral container to an existing Pod to test from the same network namespace and with the same NetworkPolicy identity. A temporary Pod in another namespace has a different label set and namespace, which matters when policies exist.

DNS and CoreDNS

A typical kubeadm lab runs CoreDNS as a Deployment in kube-system behind the kube-dns Service. Replica count is configurable. Ordinary ClusterFirst Pods receive cluster DNS settings from kubelet; NodeLocal DNSCache, custom dnsPolicy/dnsConfig and hostNetwork change the resolver path. The Corefile lives in the ConfigMap coredns in kube-system.

Pod resolv.conf typically contains:

Text
search shop.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

ndots:5 means that a name with fewer than five dots is first tried with each search suffix. So web resolves to web.shop.svc.cluster.local first. A name such as api.example.com (two dots) is tried with all suffixes before the absolute form, which creates extra queries but still works. Use a trailing dot (api.example.com.) to skip search domains.

Service names: <service>.<namespace>.svc.cluster.local. Headless Services return Pod IPs, and StatefulSet Pods get <pod>.<service>.<ns>.svc.cluster.local.

Diagnosing DNS

Bash
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns
kubectl get svc kube-dns -n kube-system
kubectl get endpoints kube-dns -n kube-system
kubectl get configmap coredns -n kube-system -o yaml
kubectl exec tmp -- cat /etc/resolv.conf
kubectl exec tmp -- nslookup kubernetes.default
Symptom Cause Fix
nslookup kubernetes.default times out CoreDNS not running, or Pod cannot reach the kube-dns ClusterIP (NetworkPolicy, CNI, kube-proxy) Check CoreDNS Pods, endpoints, policy
NXDOMAIN for a Service Wrong name or namespace, Service missing kubectl get svc -A; use FQDN
External names fail, internal work Forwarding in the Corefile (forward . /etc/resolv.conf) points to a bad upstream Fix upstream DNS on the node or Corefile
CoreDNS CrashLoopBackOff with Loop detected Node resolv.conf points to a local stub (127.0.0.53), creating a loop Point the kubelet resolvConf to the real file (/run/systemd/resolve/resolv.conf)
Pod uses node DNS, not cluster DNS dnsPolicy: Default Use ClusterFirst (the default for Pods)
Pods with hostNetwork: true miss cluster names They use node DNS Set dnsPolicy: ClusterFirstWithHostNet
CoreDNS change not effective Projection delay, DNS cache or reload error Inspect the projected Corefile, logs and reload; allow propagation. A failed reload normally retains the old config, whereas restarting with an invalid file can crash-loop

Worked example: DNS fails only for new Pods in namespace dev

Scenario. In namespace dev, nslookup web.shop times out from every Pod, while the same command works in shop. CoreDNS Pods are Running.

Reasoning.

  1. Resolution elsewhere demonstrates that some DNS paths work, not that every CoreDNS replica/path is healthy. Compare source labels, namespace policy and resolver settings.
  2. kubectl get netpol -n dev shows default-deny-egress, an egress policy with an empty podSelector and no rules.
  3. Egress to port 53 of the kube-system namespace is therefore blocked: Pods cannot query DNS.
  4. In this lab with ordinary CoreDNS endpoints, allow UDP and TCP 53 using one peer combining namespaceSelector kubernetes.io/metadata.name=kube-system and podSelector k8s-app=kube-dns. Confirm labels and the resolver path first; NodeLocal DNSCache needs rules appropriate to its actual address and dataplane.
  5. Test again: nslookup web.shop works. Remember to allow the actual application egress as well.

Why not restart CoreDNS? It would change nothing; the packets never reach it.

NetworkPolicies as a cause

NetworkPolicies combine allowed connections per direction. Check both source egress and destination ingress, plus node/hostNetwork exceptions. Timeouts and label-dependent results are useful clues but not proof: rejection behaviour varies by implementation and routing/firewall failures can look similar.

Bash
kubectl get netpol -A
kubectl describe netpol -n shop
kubectl get pod -n shop --show-labels
kubectl get ns --show-labels

Check: which policies select the destination Pod (podSelector), which directions are listed in policyTypes, whether the from rule uses podSelector, namespaceSelector, or ipBlock, and whether the source Pod's labels match. A frequent mistake is a from element that combines namespaceSelector and podSelector in the same list item (AND) versus two separate items (OR). Policies only work if the CNI plugin enforces them: Flannel alone does not, while Calico and Cilium do. A policy that "does nothing" may simply be unenforced.

Testing policy quickly: run a Pod with the allowed labels and one without, and compare. Remove or edit the policy only when the task says so; otherwise add an allow rule.

CNI and node-level networking

If Pods on one node cannot reach Pods on another node, or new Pods stay ContainerCreating with failed to set up sandbox container ... network plugin, look at the CNI.

  1. Identify the installed network implementation and its namespace, agents and supported operating systems. On a DaemonSet-based setup, inspect the agent on the affected eligible node; do not assume every CNI has a Pod in kube-system.
  2. kubectl logs -n kube-system <cni-pod> : errors about the pod CIDR, IPAM or the datastore.
  3. Inspect the runtime's configured CNI binary and config directories, commonly /opt/cni/bin and /etc/cni/net.d. File selection/chaining is runtime configuration-dependent. Missing/invalid config can produce network-not-ready or sandbox errors; inspect actual conditions and logs.
  4. Check the configured Pod address pools and IPAM mode. When the plugin uses node podCIDR allocation, compare it with controller-manager and kubeadm settings. Other IPAM implementations can allocate addresses without spec.podCIDR; its absence alone is not a fault.
  5. Required ports and kernel settings: net.bridge.bridge-nf-call-iptables=1, net.ipv4.ip_forward=1 and the br_netfilter module for iptables-based setups.

Where used, kube-proxy programs iptables, nftables or legacy IPVS rules; IPVS is deprecated in current Kubernetes. Some CNI dataplanes replace kube-proxy. If Services work only on some nodes, check that its DaemonSet Pod runs on the failing node (kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide) and read its logs. Its configuration is the ConfigMap kube-proxy (mode and cluster CIDR). Direct Pod-IP success with Service-IP failure suggests checking Service configuration, policy and forwarding. Existing kernel rules can keep working after kube-proxy stops, so also inspect whether updates are propagating.

Ingress and Gateway API

An Ingress resource does nothing without an installed Ingress controller (use the supported controller supplied by the lab; community ingress-nginx maintenance ended in March 2026). A Gateway needs a controller that implements the GatewayClass named in the Gateway.

Ingress checklist

Here ingress-system and lab-ingress-controller are exercise-supplied names, not an installation recipe. Substitute the actual namespace/controller. HTTP status codes are controller/application-specific clues.

Bash
kubectl get ingress -A
kubectl describe ingress shop-ingress -n shop
kubectl get ingressclass
kubectl get pods -n ingress-system
kubectl logs -n ingress-system deploy/lab-ingress-controller
Symptom Cause Fix
No ADDRESS on the Ingress Missing controller/class, provisioning issue or status publishing not configured Inspect the intended controller and entry point; an empty field alone is not proof of absence
404 from the controller Host or path does not match any rule Align host, path and pathType
503 from the controller Backend Service has no endpoints Fix Service/endpoints first
502/connection errors Wrong backend port number or protocol Use the Service port; set backend protocol annotation if needed
Backend not found in describe Service name or port typo, wrong namespace (Ingress backends must be in the same namespace) Correct reference
TLS errors Secret missing, wrong namespace, hostname not in certificate Create the TLS Secret in the Ingress namespace

Gateway API checklist

Gateway status conditions narrow configuration faults; check their observedGeneration and test the real dataplane as well:

Bash
kubectl get gatewayclass
kubectl get gateway -A
kubectl describe gateway web-gw -n shop
kubectl get httproute -A
kubectl describe httproute web-route -n shop
  • Check GatewayClass Accepted, its reason and controller health. False can mean a controller explicitly rejected the configuration, not that no controller exists.
  • Gateway must be Accepted=True and Programmed=True, with an address assigned.
  • HTTPRoute parent conditions separate Accepted from ResolvedRefs. Missing/forbidden backend references can leave Accepted true while ResolvedRefs is false; inspect reasons such as RefNotPermitted or BackendNotFound. A cross-namespace backend reference needs a ReferenceGrant in the target namespace, while route-to-Gateway attachment uses allowedRoutes instead.
  • A route attaches only if parentRefs names the Gateway correctly and the Gateway listener allows routes from the route's namespace (allowedRoutes) and matches the hostname.

Worked example: HTTPRoute is not accepted

Scenario. HTTPRoute shop-route in namespace shop should send shop.example.com to Service web through Gateway public-gw in namespace infra. Requests return the gateway's 404.

Steps.

  1. kubectl describe httproute shop-route -n shop shows under Parents: Accepted: False, reason NotAllowedByListeners.
  2. kubectl get gateway public-gw -n infra -o yaml shows the listener with allowedRoutes.namespaces.from: Same, so only routes in infra can attach.
  3. Since the task keeps the route in shop, use from: Selector with a label targeting the authorised namespace and apply that namespace label. Avoid from: All unless all namespaces are intentionally authorised; moving the route is a separate design choice.
  4. describe httproute now shows Accepted: True and ResolvedRefs: True.
  5. Verify: curl -H 'Host: shop.example.com' http://<gateway-address>/ returns the application.

Why this order? Conditions in the status are written by the controller and name the reason; reading them before editing YAML avoids guessing among hostnames, ports and backends.

Comparison: symptoms and first checks

Symptom First check
Timeout to Service, Pods fine kubectl get endpoints
Connection refused through Service targetPort and listening address
Works by IP, fails by name CoreDNS, resolv.conf, NetworkPolicy to port 53
Works in one namespace only NetworkPolicy, namespace labels
Pods on different nodes cannot talk CNI Pods and configuration, pod CIDR
Service works on some nodes kube-proxy on the failing node
Ingress 404 / 503 / 502 Rules / endpoints / backend port
Route not effective Gateway and HTTPRoute status conditions

For log collection while testing these paths, use kubectl logs and Kubernetes events.

Common pitfalls and exam tips

  • Testing from the wrong place. Use a Pod inside the cluster, in the right namespace.
  • Short names across namespaces. web works only in the same namespace; use web.shop.
  • Confusing port, targetPort and nodePort. port is the Service port, targetPort the container port, nodePort the node port (30000 to 32767 by default).
  • Selector versus labels. Compare them literally with --show-labels.
  • Ignoring readiness. Unready addresses can remain in EndpointSlices with ready: false; inspect conditions rather than assume presence means routable.
  • Forgetting policies. Timeouts that depend on namespace or labels are policy signs.
  • Editing the wrong object. Changing the Pod label does not change the immutable Deployment selector.
  • Exam tip: write the Service, namespace and port from the task in your notes. Use EndpointSlice addresses and conditions as an early check, and a temporary BusyBox Pod for tests.
  • Exam tip: the documentation allowed in the exam includes the Services, DNS and NetworkPolicy pages; copy YAML from them and adapt.

Mini case study: the unreachable payments API at Northwind

Northwind runs payments (Deployment, 2 replicas, label app=payments, container port 8443) in namespace fin, exposed by a Service payments and an Ingress payments served by the supported lab controller. Users get HTTP 503 from the Ingress. The engineer collects:

  1. kubectl get pods -n fin shows both Pods 0/1 Running.
  2. EndpointSlices for payments include the two Pod IPs with conditions.ready: false.
  3. kubectl describe pod -n fin shows Readiness probe failed: Get "http://10.244.1.7:8080/healthz": connection refused.
  4. The Service has port: 443, targetPort: 8443. The container listens on 8443, but the readiness probe uses port 8080.
  5. A NetworkPolicy named fin-default allows ingress only from Pods labelled role=frontend; the ingress controller runs in namespace ingress-system.
  6. The lab controller logs report no healthy upstream for fin/payments.

Solution.

  1. The controller's no-healthy-upstream evidence points to backend eligibility; HTTP 503 alone does not uniquely establish that cause.
  2. The slice addresses are unready because the readiness probe uses the wrong port (8080 instead of 8443). Patch the Deployment (kubectl edit deploy payments -n fin) so the probe uses port: 8443 and the correct scheme; wait for the rollout.
  3. Addresses become ready, but the controller path may still be blocked. Add a narrow ingress allowance combining the actual controller Pod labels with namespaceSelector kubernetes.io/metadata.name=ingress-system on TCP 8443; account for hostNetwork/source-NAT behaviour in the installed dataplane.
  4. Verify slice conditions, then test from a source actually allowed by policy. Assume this lab backend serves HTTPS on 8443: use an approved curl Pod with the trusted lab CA and the Service name/port https://payments.fin:443 (opens in a new tab), with matching certificate names. Finally test the external Ingress hostname. Configure the controller's upstream HTTPS/TLS trust according to its implementation; do not send plain HTTP to a TLS-only backend.

Commentary. There were two independent faults, hidden one behind the other. Fixing the first (probe) exposed the second (policy). Verifying each layer after each fix prevented the conclusion that the task was finished too early.

Key terms

  • EndpointSlice: backend addresses, ports and ready/serving/terminating conditions used for Service discovery and forwarding.
  • Selector: label query that picks Pods for a Service.
  • targetPort: container port that receives the Service traffic.
  • CoreDNS: cluster DNS server, exposed as the kube-dns Service.
  • ndots: resolver option controlling when search domains are used.
  • NetworkPolicy: namespaced allow rules for Pod traffic; needs an enforcing CNI.
  • CNI: plugin that configures Pod networking.
  • kube-proxy: Service-rule programmer where used; some dataplanes replace it.
  • Ingress controller / GatewayClass: implementation required for Ingress and Gateway objects to work.
  • ReferenceGrant: permission for cross-namespace references in Gateway API.

Summary checklist

  • Test in layers: Pod IP, Service, DNS, policy, CNI, external entry.
  • Missing slice addresses and present-but-unready addresses require different checks.
  • targetPort must match the port the application listens on.
  • Test from a temporary Pod and use FQDNs across namespaces.
  • Check CoreDNS Pods, the kube-dns endpoints, and resolv.conf for DNS failures.
  • Timeouts that depend on namespace or labels suggest NetworkPolicy; egress rules must allow DNS.
  • CNI Pods, /etc/cni/net.d and pod CIDR explain cross-node failures.
  • Ingress needs a controller and class; read Gateway and HTTPRoute status conditions.

Official sources