Service Discovery & DNS

Imagine walking into a massive, bustling corporate office looking for “Alice in Accounting.” You don’t know her desk number, and even if you did yesterday, they reorganize the desks every night. If you memorize her desk number (her IP address), you will inevitably get lost.

In a dynamic microservices environment where containers are constantly created, scaled, and destroyed, IP addresses are ephemeral. You can never hardcode an IP like 172.17.0.2 in your application’s configuration.

To solve this, Docker provides a built-in Embedded DNS Server so containers can find each other reliably by name, regardless of what IP address they happen to have today.

1. The Anatomy of Docker DNS (`127.0.0.11`)

Every Docker container running in a user-defined network has its /etc/resolv.conf file automatically configured by Docker to point to a very specific, magic IP address: 127.0.0.11.

This isn’t an external server; it is a lightweight DNS server embedded directly into the Docker daemon itself.

How Resolution Works

When your application (let’s say an api container) attempts to connect to a database using the hostname db:

The Query: The application makes a DNS lookup for db. The OS routes this query to 127.0.0.11.
The Intercept: The Docker daemon intercepts this request.
The Internal Lookup: Docker checks its internal registry: “Is there a container named db or with the alias db running in the same network as the api container?”
Internal Response: If yes, Docker immediately returns the container’s private IP (e.g., 172.18.0.5).
External Fallback: If no, Docker acts as a proxy and forwards the query to the host machine’s configured DNS servers (e.g., 8.8.8.8 for Google DNS).

Crucial Detail: This embedded DNS is only fully functional on user-defined bridge networks (and overlay networks). On the default bridge network, automatic service discovery via container name is disabled to prevent accidental conflicts.

2. Interactive: DNS Resolver

Type a hostname to see how Docker intelligently resolves it. Try searching for db, redis, api, google.com, or a random fake name.

root@web:/# ping

Container Resolver (127.0.0.11)

Waiting...

Docker Network: my-net

db → 172.18.0.5

redis → 172.18.0.6

api → 172.18.0.7

3. Round-Robin Load Balancing

Docker DNS isn’t just for locating single containers; it also provides a primitive layer of client-side load balancing.

If you scale a service using Docker Compose or Docker Swarm (e.g., you have 3 replicas of your web service), Docker’s DNS handles this gracefully. When you query the web hostname, Docker returns the IPs of all the healthy replicas, but it shuffles the order (Round-Robin) with each request.

# First query from an adjacent container
nslookup web
# -> 172.18.0.5, 172.18.0.6, 172.18.0.7

# Second query
nslookup web
# -> 172.18.0.6, 172.18.0.7, 172.18.0.5

Most modern HTTP clients and libraries will take the first IP in the list. By rotating the list, Docker ensures that traffic is distributed roughly evenly across all instances of your service, without requiring a dedicated proxy like Nginx or HAProxy for basic internal communication.

4. War Story: The DNS Cache Trap

While Docker’s embedded DNS is powerful, it exposes a very common pitfall in production systems: Application-level DNS Caching.

Imagine this scenario:

Your Node.js api connects to a mongodb container at 172.18.0.5.
The mongodb container crashes and Docker restarts it. It receives a new IP: 172.18.0.9.
Docker immediately updates its internal DNS.
The Failure: Your api container keeps trying to connect to 172.18.0.5 and throws timeouts.

Why did this happen? Many runtimes (like older versions of Node.js, and notably the Java Virtual Machine) cache DNS lookups indefinitely by default for performance. Even though Docker’s DNS was updated, your application never asked Docker again; it just used its cached, stale IP.

The Solution: Always ensure your application’s DNS cache TTL (Time To Live) is configured properly.

Java: Adjust the networkaddress.cache.ttl property in your java.security file to a reasonable value (e.g., 5-30 seconds).
Node.js: Ensure you are using connection pooling libraries that gracefully handle connection drops and force re-resolution, or avoid caching at the HTTP layer unecessarily.

5. Summary

127.0.0.11: The magic IP inside every container that acts as the gateway to Docker’s embedded DNS server.
Automatic Registration: Containers are automatically registered by name and alias when they join a user-defined network.
Load Balancing: Scaled services receive multiple IPs, returned in a round-robin format for basic load distribution.
Beware the Cache: Always configure your application runtimes to respect DNS TTLs, otherwise ephemeral IPs will lead to broken connections.

Service Discovery