Kubernetes Services and Pods DNS Resolution (Hostnames)

Consider this post more of a summary or a cheat sheet of the important parts mentioned in the documentation about DNS records automatically created for Services and Pods in your K8S cluster’s DNS Server.

Kubelet configures each Pod’s DNS (/etc/resolv.conf) so that the running containers can make requests to/perform DNS lookups for Services and other Pods across namespaces by names instead of IP addresses that are not stable for Pods at least.


So if you have a Service called my-service, you can send requests to it (or perform DNS queries) using a hostname in this form/syntax – <svc>.<namespace>.svc.<cluster-domain> where cluster-domain is generally cluster.local or whatever you see in your /etc/resolv.conf.

Let’s see an example:

# Namespace: test

# A service backing 3 nginx pods
$ k get svc
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)
my-service   ClusterIP    <none>        80/TCP

# Running pods
$ k get po

# Get a shell to the running container inside the pod
$ k exec -it pod-0 -- /bin/bash
# Run nslookup for DNS query (can also try dig)
$ nslookup my-service.test.svc.cluster.local

Name:	my-service.test.svc.cluster.local
Address: # IP address of the SVC

# Send an HTTP request to the SVC hostname
$ curl http://regular-service.test.svc.cluster.local
<h1>Welcome to nginx!</h1>

As you can see in the example above, we have a my-service SVC in test namespace that we hit using the following hostname – my-service.test.svc.cluster.local. We can also use just my-service or my-service.test or my-service.test.svc instead of the full hostname string like this:

$ nslookup my-service|my-service.test|my-service.test.svc
Name:	my-service.test.svc.cluster.local

This is possible because of the search line in /etc/resolv.conf that does the expansion:

search test.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The IP address that the Service hostname resolves to ( in the example above), is the cluster IP of the SVC. For headless services though (clusterIP: None), we would see multiple A/AAAA records/addresses for each backing pod. Here’s an example:

# headless-svc is a Headless Service backing 3 pods
$ nslookup headless-svc

Name:	headless-svc.test.svc.cluster.local
Name:	headless-svc.test.svc.cluster.local
Name:	headless-svc.test.svc.cluster.local

Now just like K8S creates A/AAAA records automatically for both “normal” and headless Services with a specific hostname syntax, it also creates SRV records for each named port of these SVCs. For each named port, the SRV host follows this syntax – <_port-name>.<_port-protocol>.<svc>.<namespace>.svc.<cluster-domain>. Here’s an example lookup for a headless SVC with a named port http (mapped to 6333) and protocol tcp:

# Running inside a pod
$ nslookup -q=SRV _http._tcp.headless-svc
_http._tcp.headless-svc.test.svc.cluster.local	service = 10 33 6333 pod-0.headless-svc.test.svc.cluster.local.
_http._tcp.headless-svc.test.svc.cluster.local	service = 10 33 6333 pod-2.headless-svc.test.svc.cluster.local.
_http._tcp.headless-svc.test.svc.cluster.local	service = 10 33 6333 pod-1.headless-svc.test.svc.cluster.local.

# dig usage
$ dig +short SRV _http._tcp.headless-svc.test.svc.cluster.local
10 33 6333 pod-0.headless-svc.test.svc.cluster.local.
10 33 6333 pod-2.headless-svc.test.svc.cluster.local.
10 33 6333 pod-1.headless-svc.test.svc.cluster.local.


Pods automatically get A/AAAA records with the following hostname:


So for example:

# List of pods
$ k get pod -o wide
NAME        IP

# Get shell access to container inside pod-0
$ k exec -it pod-0 -- /bin/bash
# DNS query for pod-1 (syntax shown above)
$ nslookup 10-37-1-6.test.pod.cluster.local
Name:	10-37-1-6.test.pod.cluster.local

As you can see above, the Pod hostname starts with the IP address. We can instead use a Pod’s hostname (like pod-0 or pod-1) which can be helpful in some cases (like StatefulSets). This requires setting up a headless Service along with understanding two interesting fields in a Pod’s spec called hostname and subdomain. Let’s start with the latter first.

The hostname of a Pod comes from metadata.name but you can configure/override it by specifying a new name in spec.hostname. You can try the hostname or hostname --fqdn command inside a Pod container to see the value come from either metadata.name or spec.hostname.

Now there’s also spec.subdomain which can be set to indicate that a pod is part of a sub-group within that namespace – this is just a conceptual aspect. If spec.subdomain is set to foo then running hostname --fqdn (fully qualified domain name) inside a pod will yield <hostname>.foo.<namespace>.<cluster-domain>. As an example:

# Running from pod-0 inside test NS
$ hostname
$ hostname --fqdn # with no spec.subdomain
$ hostname --fqdn # with spec.subdomain=foo

Now that you understand this way of configuring a Pod’s hostname and FQDN, it can be used in conjunction with a headless Service where if the SVC’s metadata.name is the same as the Pod’s spec.subdomain, then the cluster DNS server will start returning A/AAAA records for all the Pods being backed by the SVC. These records will be set for the same FQDN/DNS hostnames that we saw above (pod-0.foo.test.svc.cluster.local).

So to summarise this, for pod-0.foo.test.svc.cluster.local hostname to return the appropriate DNS records (IP address of pod-0 Pod in test namespace), you must have a headless SVC in test namespace with the name metadata.name: foo and the Pods being backed by it must also have spec.subdomain: foo and metadata.name: pod-0 or spec.hostname: pod-0. You can find an example setup manifest for it here.

This is precisely what happens when you set spec.serviceName in a StatefulSet and set up a headless SVC with the same name. The STS’s pods inherit spec.subdomain value from STS’s spec.serviceName and spec.hostname is auto-generated (sts-0, sts-1, sts-2, …) allowing you to directly look up the IP of a Pod by hitting sts-0.<spec.serviceName>.<namespace>.<cluster-domain>.

Leave a Reply

Your email address will not be published. Required fields are marked *