Consider this post more of a summary or a cheat sheet of the important parts mentioned in the documentation about DNS records automatically created for Services and Pods in your K8S cluster’s DNS Server.

Kubelet configures each Pod’s DNS (/etc/resolv.conf) so that the running containers can make requests to/perform DNS lookups for Services and other Pods across namespaces by names instead of IP addresses that are not stable for Pods at least.

Services

So if you have a Service called my-service, you can send requests to it (or perform DNS queries) using a hostname in this form/syntax – <svc>.<namespace>.svc.<cluster-domain> where cluster-domain is generally cluster.local or whatever you see in your /etc/resolv.conf.

Let’s see an example:

# Namespace: test

# A service backing 3 nginx pods
$ k get svc
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)
my-service   ClusterIP        10.38.213.42    <none>        80/TCP

# Running pods
$ k get po
NAME
pod-0
pod-1
pod-2

# Get a shell to the running container inside the pod
$ k exec -it pod-0 -- /bin/bash
# Run nslookup for DNS query (can also try dig)
$ nslookup my-service.test.svc.cluster.local
Server:		10.38.0.10
Address:	10.38.0.10#53

Name:	my-service.test.svc.cluster.local
Address: 10.38.213.42 # IP address of the SVC

# Send an HTTP request to the SVC hostname
$ curl http://regular-service.test.svc.cluster.local
<h1>Welcome to nginx!</h1>

As you can see in the example above, we have a my-service SVC in test namespace that we hit using the following hostname – my-service.test.svc.cluster.local. We can also use just my-service or my-service.test or my-service.test.svc instead of the full hostname string like this:

$ nslookup my-service|my-service.test|my-service.test.svc
...
Name:	my-service.test.svc.cluster.local
Address: 10.38.213.42

This is possible because of the search line in /etc/resolv.conf that does the expansion:

search test.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.38.0.10
options ndots:5

The IP address that the Service hostname resolves to (10.38.213.42 in the example above), is the cluster IP of the SVC. For headless services though (clusterIP: None), we would see multiple A/AAAA records/addresses for each backing pod. Here’s an example:

# headless-svc is a Headless Service backing 3 pods
$ nslookup headless-svc
...

Name:	headless-svc.test.svc.cluster.local
Address: 10.68.23.5
Name:	headless-svc.test.svc.cluster.local
Address: 10.68.26.5
Name:	headless-svc.test.svc.cluster.local
Address: 10.68.23.4

Now just like K8S creates A/AAAA records automatically for both “normal” and headless Services with a specific hostname syntax, it also creates SRV records for each named port of these SVCs. For each named port, the SRV host follows this syntax – <_port-name>.<_port-protocol>.<svc>.<namespace>.svc.<cluster-domain>. Here’s an example lookup for a headless SVC with a named port http (mapped to 6333) and protocol tcp:

# Running inside a pod
$ nslookup -q=SRV _http._tcp.headless-svc
...
_http._tcp.headless-svc.test.svc.cluster.local	service = 10 33 6333 pod-0.headless-svc.test.svc.cluster.local.
_http._tcp.headless-svc.test.svc.cluster.local	service = 10 33 6333 pod-2.headless-svc.test.svc.cluster.local.
_http._tcp.headless-svc.test.svc.cluster.local	service = 10 33 6333 pod-1.headless-svc.test.svc.cluster.local.

# dig usage
$ dig +short SRV _http._tcp.headless-svc.test.svc.cluster.local
10 33 6333 pod-0.headless-svc.test.svc.cluster.local.
10 33 6333 pod-2.headless-svc.test.svc.cluster.local.
10 33 6333 pod-1.headless-svc.test.svc.cluster.local.

Pods

Pods automatically get A/AAAA records with the following hostname:

<pod-ip-address>.<namespace>.pod.<cluster-domain>

So for example:

# List of pods
$ k get pod -o wide
NAME        IP
pod-0       10.37.0.12
pod-1       10.37.1.6

# Get shell access to container inside pod-0
$ k exec -it pod-0 -- /bin/bash
# DNS query for pod-1 (syntax shown above)
$ nslookup 10-37-1-6.test.pod.cluster.local
...
Name:	10-37-1-6.test.pod.cluster.local
Address: 10.37.1.6

As you can see above, the Pod hostname starts with the IP address. We can instead use a Pod’s hostname (like pod-0 or pod-1) which can be helpful in some cases (like StatefulSets). This requires setting up a headless Service along with understanding two interesting fields in a Pod’s spec called hostname and subdomain. Let’s start with the latter first.

The hostname of a Pod comes from metadata.name but you can configure/override it by specifying a new name in spec.hostname. You can try the hostname or hostname --fqdn command inside a Pod container to see the value come from either metadata.name or spec.hostname.

Now there’s also spec.subdomain which can be set to indicate that a pod is part of a sub-group within that namespace – this is just a conceptual aspect. If spec.subdomain is set to foo then running hostname --fqdn (fully qualified domain name) inside a pod will yield <hostname>.foo.<namespace>.<cluster-domain>. As an example:

# Running from pod-0 inside test NS
$ hostname
pod-0
$ hostname --fqdn # with no spec.subdomain
pod-0
$ hostname --fqdn # with spec.subdomain=foo
pod-0.foo.test.svc.cluster.local

Now that you understand this way of configuring a Pod’s hostname and FQDN, it can be used in conjunction with a headless Service where if the SVC’s metadata.name is the same as the Pod’s spec.subdomain, then the cluster DNS server will start returning A/AAAA records for all the Pods being backed by the SVC. These records will be set for the same FQDN/DNS hostnames that we saw above (pod-0.foo.test.svc.cluster.local).

So to summarise this, for pod-0.foo.test.svc.cluster.local hostname to return the appropriate DNS records (IP address of pod-0 Pod in test namespace), you must have a headless SVC in test namespace with the name metadata.name: foo and the Pods being backed by it must also have spec.subdomain: foo and metadata.name: pod-0 or spec.hostname: pod-0. You can find an example setup manifest for it here.

This is precisely what happens when you set spec.serviceName in a StatefulSet and set up a headless SVC with the same name. The STS’s pods inherit spec.subdomain value from STS’s spec.serviceName and spec.hostname is auto-generated (sts-0, sts-1, sts-2, …) allowing you to directly look up the IP of a Pod by hitting sts-0.<spec.serviceName>.<namespace>.<cluster-domain>.