Kubernetes Course
DNS in Kubernetes
Every time a Pod calls another service by name — http://payment-svc or postgres.payments.svc.cluster.local — something resolves that name to an IP. That something is CoreDNS, and understanding exactly how it works is what separates engineers who debug network issues in two minutes from those who spend two hours.
CoreDNS: The Cluster's DNS Server
CoreDNS is a DNS server that runs as a Deployment in the kube-system namespace. Every Pod in the cluster has its /etc/resolv.conf configured to point at CoreDNS as its nameserver. When your application code calls payment-svc, the DNS resolution chain goes: process → kernel resolver → CoreDNS → Kubernetes Service record → ClusterIP.
CoreDNS is not just a passive DNS server — it actively watches the Kubernetes API. When you create a Service, CoreDNS immediately registers a DNS record for it. When you delete the Service, the record is removed. This happens in real time, without any manual DNS configuration. Creating a Service and calling it by name from any Pod in the cluster works immediately.
What CoreDNS creates for each Service
For every Service, CoreDNS creates an A record (or AAAA for IPv6) mapping the fully-qualified domain name to the Service's ClusterIP. For headless Services (clusterIP: None), it creates individual A records for each Pod IP instead. For ExternalName Services, it creates a CNAME record. For StatefulSet Pods, it creates individual A records per Pod with stable hostnames.
The DNS Name Format
Every Service in Kubernetes gets a DNS name following a predictable pattern. Understanding the pattern lets you construct the correct DNS name for any service without guessing.
DNS Name Components
metadata.name of the Service objectpayment-svc.payments.svc.cluster.localShort forms (within same namespace):
payment-svc or payment-svc.payments
The DNS Search Path — Why Short Names Work
In the previous lesson you called services by their short name — just payment-svc instead of the full FQDN. This works because Kubernetes configures each Pod's /etc/resolv.conf with a search path. When you query an unqualified name, the resolver automatically appends each domain in the search list and tries them in order until one resolves.
kubectl exec -it checkout-api-6f8b9d-2xkpj -n production -- cat /etc/resolv.conf
# Show the DNS configuration inside any running container
# This file is injected by Kubernetes when the Pod startsnameserver 10.96.0.10 search production.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal options ndots:5
What just happened?
nameserver 10.96.0.10 — This is the ClusterIP of the kube-dns Service in kube-system — the stable IP that CoreDNS is reachable at. Every Pod in the cluster sends DNS queries to this IP, regardless of which namespace they're in.
search path explained — When a Pod in the production namespace queries payment-svc, the resolver tries these names in order until one answers:
1. payment-svc.production.svc.cluster.local ← hits immediately (same namespace)
2. payment-svc.svc.cluster.local (would try if #1 fails)
3. payment-svc.cluster.local (would try if #2 fails)
4. payment-svc.eu-west-1.compute.internal (external DNS, last resort)
options ndots:5 — A name with fewer than 5 dots is treated as a relative name and the search path is applied. A name with 5 or more dots is treated as absolute — the search path is skipped. This means payment-svc (0 dots) goes through the search path, but payment-svc.payments.svc.cluster.local. (with trailing dot = absolute) does not.
DNS Lookup Behaviour in Practice
The scenario: You're a developer trying to understand exactly how your checkout service resolves the payment service name. You need to trace the full DNS resolution chain from inside the Pod to understand a latency issue you're seeing on the first request.
kubectl exec -it checkout-api-6f8b9d-2xkpj -n production -- nslookup payment-svc
# Basic DNS lookup — resolves the short name using the search path
# Shows which nameserver answered and what IP was returned
kubectl exec -it checkout-api-6f8b9d-2xkpj -n production -- \
nslookup payment-svc.production.svc.cluster.local
# Fully-qualified lookup — bypasses search path, direct query
# Returns same result but slightly faster (no search path iteration)
kubectl exec -it checkout-api-6f8b9d-2xkpj -n production -- \
nslookup payment-svc.payments.svc.cluster.local
# Cross-namespace lookup by FQDN — calling a service in the 'payments' namespace
# from a Pod in the 'production' namespace
kubectl exec -it checkout-api-6f8b9d-2xkpj -n production -- \
nslookup postgres-0.postgres-headless.payments.svc.cluster.local
# StatefulSet Pod DNS — stable per-Pod hostname
# Format: [pod-name].[headless-service-name].[namespace].svc.cluster.local
# postgres-0, postgres-1, postgres-2 are reachable individually via this pattern
kubectl run dns-debug --image=nicolaka/netshoot --rm -it --restart=Never -n production -- \
dig payment-svc.production.svc.cluster.local +short
# dig: lower-level DNS query tool — shows TTL, query time, answer section
# +short: just the IP address — good for scripting
# Useful when nslookup doesn't give enough detail$ kubectl exec -it checkout-api-6f8b9d-2xkpj -n production -- nslookup payment-svc Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: payment-svc Address 1: 10.96.214.88 payment-svc.production.svc.cluster.local $ kubectl exec -it checkout-api-6f8b9d-2xkpj -n production -- \ nslookup payment-svc.payments.svc.cluster.local Server: 10.96.0.10 Name: payment-svc.payments.svc.cluster.local Address 1: 10.96.77.31 payment-svc.payments.svc.cluster.local $ kubectl exec -it checkout-api-6f8b9d-2xkpj -n production -- \ nslookup postgres-0.postgres-headless.payments.svc.cluster.local Name: postgres-0.postgres-headless.payments.svc.cluster.local Address 1: 10.244.2.14
What just happened?
Two different payment-svc Services — The short name payment-svc resolved to 10.96.214.88 (the one in the production namespace). The FQDN payment-svc.payments.svc.cluster.local resolved to 10.96.77.31 (the one in the payments namespace). Two completely different Services with the same name in different namespaces — namespace isolation at work. Short names are local to the namespace; FQDNs are global.
StatefulSet Pod DNS — The headless Service creates individual A records for each StatefulSet Pod. postgres-0 resolves to a Pod IP (10.244.2.14), not a ClusterIP. This is what makes StatefulSets work for databases — each replica has a stable, addressable DNS name that applications can use to target specific replicas for reads.
The ndots:5 latency problem — When a Pod queries a short name like payment-svc, the resolver tries up to four search path suffixes sequentially until one answers. That's potentially 4 DNS queries instead of 1. For services that make thousands of DNS lookups per second, this overhead is real. The fix: use the FQDN in performance-critical code paths to skip the search path entirely.
CoreDNS Configuration
CoreDNS is configured via a ConfigMap called coredns in the kube-system namespace. The config uses CoreDNS's own DSL (called a Corefile). Most clusters use the default configuration, but two customisations are commonly needed: forwarding specific domains to internal DNS servers, and tweaking caching behaviour.
The scenario: Your company has an on-premises Active Directory domain at corp.internal with an internal DNS server at 10.0.0.53. Pods in Kubernetes need to resolve db.corp.internal to reach on-premises databases. You need to configure CoreDNS to forward queries for corp.internal to the on-premises DNS server while still handling all cluster.local queries internally.
kubectl get configmap coredns -n kube-system -o yaml
# View the current CoreDNS configurationapiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 { # Handle all DNS queries on port 53
errors # Log errors
health { # Health check endpoint at :8080/health
lameduck 5s # 5s grace period during shutdown
}
ready # Ready check endpoint at :8181/ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
# Handle cluster.local and reverse DNS queries
pods insecure # Create DNS records for Pod IPs (insecure = no pod name validation)
fallthrough in-addr.arpa ip6.arpa # Fall through to next plugin for reverse DNS
ttl 30 # TTL for cluster.local records (30 seconds)
}
prometheus :9153 # Expose Prometheus metrics on port 9153
forward . /etc/resolv.conf { # Forward all other queries to the node's DNS resolver
max_concurrent 1000 # Max concurrent upstream DNS queries
}
cache 30 # Cache DNS responses for 30 seconds
loop # Detect and break forwarding loops
reload # Hot-reload config changes without restart
loadbalance # Round-robin order of resolved IPs
}
corp.internal:53 { # Custom stanza: handle corp.internal queries specially
errors
cache 30
forward . 10.0.0.53 { # Forward all corp.internal queries to on-premises DNS
force_tcp # Use TCP (more reliable for large responses like AD)
}
}$ kubectl apply -f coredns-configmap.yaml configmap/coredns configured $ kubectl rollout restart deployment/coredns -n kube-system deployment.apps/coredns restarted $ kubectl exec -it checkout-api-6f8b9d-2xkpj -n production -- \ nslookup db.corp.internal Server: 10.96.0.10 Name: db.corp.internal Address 1: 10.0.1.45 ← resolved by on-premises DNS via CoreDNS forwarding $ kubectl exec -it checkout-api-6f8b9d-2xkpj -n production -- \ nslookup payment-svc.production.svc.cluster.local Server: 10.96.0.10 Name: payment-svc.production.svc.cluster.local Address 1: 10.96.214.88 ← still resolved locally by CoreDNS kubernetes plugin
What just happened?
Selective forwarding — The custom corp.internal:53 stanza intercepts all queries for that domain before they reach the .:53 catch-all. It forwards them to the on-premises DNS at 10.0.0.53. The cluster.local queries are still handled entirely by the Kubernetes plugin — they never reach the external DNS.
CoreDNS hot-reload — The reload plugin in the Corefile means CoreDNS watches its own ConfigMap for changes and reloads automatically. In practice, a kubectl rollout restart is still the cleanest way to ensure the new config takes effect immediately across all replicas.
Prometheus metrics — CoreDNS exposes rich metrics at port 9153: query rate, error rate, cache hit/miss ratio, upstream latency. If your cluster is experiencing DNS resolution problems at scale (a very real production issue — DNS is often the bottleneck on high-RPS clusters), these metrics are essential for diagnosing it.
Pod DNS Settings
The default DNS behaviour can be overridden per-Pod using the dnsPolicy and dnsConfig fields. This is rarely needed but important to know for edge cases.
apiVersion: v1
kind: Pod
metadata:
name: custom-dns-pod
namespace: production
spec:
dnsPolicy: ClusterFirst # Default: use CoreDNS, fall back to node DNS for external
# Other options:
# Default: use the node's DNS (not CoreDNS) — inherits node's resolv.conf
# ClusterFirstWithHostNet: use CoreDNS when hostNetwork: true
# None: DNS entirely controlled by dnsConfig below
dnsConfig:
nameservers:
- 10.96.0.10 # CoreDNS ClusterIP — usually already set by dnsPolicy
- 8.8.8.8 # Add Google DNS as a second nameserver (failover)
searches:
- production.svc.cluster.local # Custom search domains — appended to the defaults
- svc.cluster.local
- cluster.local
- company.internal # Add a custom corporate domain to the search path
options:
- name: ndots
value: "3" # Lower ndots from 5 to 3 — reduces unnecessary search lookups
# Names with fewer than 3 dots go through search path
# Names with 3+ dots are treated as absolute
containers:
- name: app
image: company/app:1.0.0$ kubectl apply -f custom-dns-pod.yaml pod/custom-dns-pod created $ kubectl exec -it custom-dns-pod -n production -- cat /etc/resolv.conf nameserver 10.96.0.10 nameserver 8.8.8.8 search production.svc.cluster.local svc.cluster.local cluster.local company.internal options ndots:3
What just happened?
ndots: 3 performance optimisation — Reducing ndots from 5 to 3 means names with 3 or more dots are treated as absolute (no search path applied). The full FQDN payment-svc.production.svc.cluster.local has 4 dots — with ndots:5 it would still go through the search path, with ndots:3 it's treated as absolute and resolved directly. This eliminates unnecessary DNS queries for services that already use FQDNs. High-RPS services that call external APIs by FQDN benefit noticeably from this change.
dnsPolicy: None — When set to None, the Pod gets no DNS configuration whatsoever unless you specify it entirely in dnsConfig. This is the escape hatch for highly specialised workloads (network appliances, DNS forwarders) that need complete control over their resolver configuration. For 99% of applications, leave dnsPolicy at the default ClusterFirst.
DNS Record Types Created by Kubernetes
Different Kubernetes objects create different DNS record types. Here's the full picture:
| Object type | DNS name pattern | Record type | Resolves to |
|---|---|---|---|
| ClusterIP Service | svc.ns.svc.cluster.local | A record | The Service's ClusterIP (stable virtual IP) |
| Headless Service | svc.ns.svc.cluster.local | Multiple A records | Individual Pod IPs (changes with Pod churn) |
| StatefulSet Pod | pod-N.svc.ns.svc.cluster.local | A record | That specific Pod's IP (stable name, IP may change on reschedule) |
| ExternalName Service | svc.ns.svc.cluster.local | CNAME record | The externalName value (e.g. api.stripe.com) |
| Pod (with hostname set) | hostname.subdomain.ns.svc.cluster.local | A record | The Pod's IP (requires subdomain + hostname fields set) |
Teacher's Note: DNS is where most "why can't my service reach X" bugs hide
In my experience, about 40% of "Service A can't reach Service B" issues are actually DNS issues. Either the name is wrong, the namespace suffix is missing for a cross-namespace call, the Service was recently deleted and CoreDNS cached a negative response, or CoreDNS itself is having a bad day. The first thing I check is always: can I resolve the name from inside the calling Pod?
The two-minute DNS debug runbook: (1) kubectl exec -it [pod] -- nslookup [service-name] — does the name resolve at all? (2) If not, try the FQDN: nslookup [svc].[ns].svc.cluster.local. (3) Check the CoreDNS pods: kubectl get pods -n kube-system -l k8s-app=kube-dns. (4) Check CoreDNS logs: kubectl logs -n kube-system -l k8s-app=kube-dns. Most DNS issues are found in one of these four steps.
A real-world gotcha: the DNS cache TTL is 30 seconds by default. If you delete and recreate a Service (maybe you changed the type), Pods that already resolved the old IP may keep using it for up to 30 seconds. During blue-green deployments or Service changes, this caching window is real. Build in a 30–60 second wait after Service changes before assuming DNS has propagated.
Practice Questions
1. A Pod in the checkout namespace needs to call a Service named payment-svc in the payments namespace. What is the correct fully-qualified DNS name to use?
2. What is the name of the ConfigMap in the kube-system namespace that contains the CoreDNS configuration (Corefile)?
3. Which /etc/resolv.conf option controls the threshold for when a DNS name is treated as absolute (skipping the search path) vs relative (going through the search path)? What is Kubernetes's default value?
Quiz
1. A Pod in the production namespace calls http://order-svc/api. There is a Service named order-svc in the same namespace. Does the short name resolve successfully?
2. What is the key difference in DNS behaviour between a regular ClusterIP Service and a headless Service (clusterIP: None)?
3. A high-throughput service makes thousands of DNS lookups per second using short names like payment-svc. Why might this cause unnecessary DNS load, and what is the fix?
Up Next · Lesson 36
Network Policies
By default every Pod can talk to every other Pod. Network Policies let you define firewall rules inside the cluster — restricting which Pods can communicate with which, and blocking lateral movement if a service is compromised.