Kubernetes Services and Pods DNS Resolution (Hostnames)
Consider this post more of a summary or a cheat sheet of the important parts mentioned in the documentation about DNS records automatically created for Services and Pods in your K8S cluster’s DNS Server.
Kubelet configures each Pod’s DNS (/etc/resolv.conf
) so that the running containers can make requests to/perform DNS lookups for Services and other Pods across namespaces by names instead of IP addresses that are not stable for Pods at least.
Services
So if you have a Service called my-service
, you can send requests to it (or perform DNS queries) using a hostname in this form/syntax – <svc>.<namespace>.svc.<cluster-domain>
where cluster-domain
is generally cluster.local
or whatever you see in your /etc/resolv.conf
.
Let’s see an example:
# Namespace: test
# A service backing 3 nginx pods
$ k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
my-service ClusterIP 10.38.213.42 <none> 80/TCP
# Running pods
$ k get po
NAME
pod-0
pod-1
pod-2
# Get a shell to the running container inside the pod
$ k exec -it pod-0 -- /bin/bash
# Run nslookup for DNS query (can also try dig)
$ nslookup my-service.test.svc.cluster.local
Server: 10.38.0.10
Address: 10.38.0.10#53
Name: my-service.test.svc.cluster.local
Address: 10.38.213.42 # IP address of the SVC
# Send an HTTP request to the SVC hostname
$ curl http://regular-service.test.svc.cluster.local
<h1>Welcome to nginx!</h1>
As you can see in the example above, we have a my-service
SVC in test
namespace that we hit using the following hostname – my-service.test.svc.cluster.local
. We can also use just my-service
or my-service.test
or my-service.test.svc
instead of the full hostname string like this:
$ nslookup my-service|my-service.test|my-service.test.svc
...
Name: my-service.test.svc.cluster.local
Address: 10.38.213.42
This is possible because of the search
line in /etc/resolv.conf
that does the expansion:
search test.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.38.0.10
options ndots:5
The IP address that the Service hostname resolves to (10.38.213.42
in the example above), is the cluster IP of the SVC. For headless services though (clusterIP: None
), we would see multiple A/AAAA records/addresses for each backing pod. Here’s an example:
# headless-svc is a Headless Service backing 3 pods
$ nslookup headless-svc
...
Name: headless-svc.test.svc.cluster.local
Address: 10.68.23.5
Name: headless-svc.test.svc.cluster.local
Address: 10.68.26.5
Name: headless-svc.test.svc.cluster.local
Address: 10.68.23.4
Now just like K8S creates A/AAAA records automatically for both “normal” and headless Services with a specific hostname syntax, it also creates SRV records for each named port of these SVCs. For each named port, the SRV
host follows this syntax – <_port-name>.<_port-protocol>.<svc>.<namespace>.svc.<cluster-domain>
. Here’s an example lookup for a headless SVC with a named port http
(mapped to 6333
) and protocol tcp
:
# Running inside a pod
$ nslookup -q=SRV _http._tcp.headless-svc
...
_http._tcp.headless-svc.test.svc.cluster.local service = 10 33 6333 pod-0.headless-svc.test.svc.cluster.local.
_http._tcp.headless-svc.test.svc.cluster.local service = 10 33 6333 pod-2.headless-svc.test.svc.cluster.local.
_http._tcp.headless-svc.test.svc.cluster.local service = 10 33 6333 pod-1.headless-svc.test.svc.cluster.local.
# dig usage
$ dig +short SRV _http._tcp.headless-svc.test.svc.cluster.local
10 33 6333 pod-0.headless-svc.test.svc.cluster.local.
10 33 6333 pod-2.headless-svc.test.svc.cluster.local.
10 33 6333 pod-1.headless-svc.test.svc.cluster.local.
Pods
Pods automatically get A/AAAA records with the following hostname:
<pod-ip-address>.<namespace>.pod.<cluster-domain>
So for example:
# List of pods
$ k get pod -o wide
NAME IP
pod-0 10.37.0.12
pod-1 10.37.1.6
# Get shell access to container inside pod-0
$ k exec -it pod-0 -- /bin/bash
# DNS query for pod-1 (syntax shown above)
$ nslookup 10-37-1-6.test.pod.cluster.local
...
Name: 10-37-1-6.test.pod.cluster.local
Address: 10.37.1.6
As you can see above, the Pod hostname starts with the IP address. We can instead use a Pod’s hostname
(like pod-0
or pod-1
) which can be helpful in some cases (like StatefulSets). This requires setting up a headless Service along with understanding two interesting fields in a Pod’s spec called hostname
and subdomain
. Let’s start with the latter first.
The hostname
of a Pod comes from metadata.name
but you can configure/override it by specifying a new name in spec.hostname
. You can try the hostname
or hostname --fqdn
command inside a Pod container to see the value come from either metadata.name
or spec.hostname
.
Now there’s also spec.subdomain
which can be set to indicate that a pod is part of a sub-group within that namespace – this is just a conceptual aspect. If spec.subdomain
is set to foo
then running hostname --fqdn
(fully qualified domain name) inside a pod will yield <hostname>.foo.<namespace>.<cluster-domain>
. As an example:
# Running from pod-0 inside test NS
$ hostname
pod-0
$ hostname --fqdn # with no spec.subdomain
pod-0
$ hostname --fqdn # with spec.subdomain=foo
pod-0.foo.test.svc.cluster.local
Now that you understand this way of configuring a Pod’s hostname and FQDN, it can be used in conjunction with a headless Service where if the SVC’s metadata.name
is the same as the Pod’s spec.subdomain
, then the cluster DNS server will start returning A/AAAA records for all the Pods being backed by the SVC. These records will be set for the same FQDN/DNS hostnames that we saw above (pod-0.foo.test.svc.cluster.local
).
So to summarise this, for pod-0.foo.test.svc.cluster.local
hostname to return the appropriate DNS records (IP address of pod-0
Pod in test
namespace), you must have a headless SVC in test
namespace with the name metadata.name: foo
and the Pods being backed by it must also have spec.subdomain: foo
and metadata.name: pod-0
or spec.hostname: pod-0
. You can find an example setup manifest for it here.
This is precisely what happens when you set spec.serviceName
in a StatefulSet and set up a headless SVC with the same name. The STS’s pods inherit spec.subdomain
value from STS’s spec.serviceName
and spec.hostname
is auto-generated (sts-0
, sts-1
, sts-2
, …) allowing you to directly look up the IP of a Pod by hitting sts-0.<spec.serviceName>.<namespace>.<cluster-domain>
.