Local DNS for home Kubernetes: tracing resolver loops from the router to CoreDNS

Opening GitLab in a browser does not prove that Argo CD can clone its repository. Resolving a Registry name on the NUC does not prove that a Pod or the container runtime uses the same DNS path. Once browser-facing access and internal machine access are separated, a single claim that “DNS works” stops being useful.

My router’s dnsmasq owns the internal records. The NUC uses the router as an upstream, and RKE2 CoreDNS resolves names for Pods. This design separates public and internal names; it is not a split-horizon setup that returns different addresses for the same hostname inside and outside the network.

The examples use the following network. Replace the addresses with your own static addresses or fixed leases. The home.arpa domain is reserved for home networks by RFC 8375.

Identify the client before changing the resolver

An application Pod using ClusterFirst asks cluster DNS. CoreDNS gets its upstreams from its configuration and the resolver supplied by kubelet. Image pulls normally resolve names on the node through the container runtime, not through the application’s Pod DNS.

application Pod → CoreDNS → router dnsmasq → public upstream
                                └→ answers home.arpa locally
node / container runtime ──→ router dnsmasq

If the host succeeds and the Pod fails, changing application code is a poor starting point. If image pulls fail but a running Pod resolves the Registry, node DNS is still a suspect. Identify which client is failing before choosing where to inspect.

Keep internal records on the router

On Asuswrt-Merlin, enable JFFS custom scripts/configs and merge these records into /jffs/configs/dnsmasq.conf.add, preserving existing settings:

local=/home.arpa/
host-record=gitlab.home.arpa,192.168.50.10
host-record=registry.home.arpa,192.168.50.10
host-record=vault.home.arpa,192.168.50.10

host-record defines the individual names. local prevents queries for unknown names in this zone from going to a public upstream. This is easier to diagnose than sending every unknown internal name to one machine. See the dnsmasq manual for the directive semantics and Merlin’s custom config documentation for the persistent configuration path.

During a window when a brief DNS interruption is acceptable, reload dnsmasq on the router:

service restart_dnsmasq

Ask the router directly from a LAN client:

nslookup gitlab.home.arpa 192.168.50.1
nslookup registry.home.arpa 192.168.50.1
nslookup vault.home.arpa 192.168.50.1

All three should resolve to the example NUC address. If they do not, fix the router configuration, leases, and DNS service before changing CoreDNS. DHCP clients also need the router DNS configuration; clients using their own public DNS or DoH may not see these records.

Inspect the NUC’s effective resolver

On a Linux node managed by NetworkManager, start with:

nmcli connection show --active
resolvectl status
readlink -f /etc/resolv.conf
cat /etc/resolv.conf
cat /run/systemd/resolve/resolv.conf

If the relevant connection is named LAN, this example points IPv4 DNS at the router. It applies to connections managed by NetworkManager. For systemd-networkd or another backend, update that host’s corresponding configuration source instead:

sudo nmcli connection modify "LAN" \
  ipv4.ignore-auto-dns yes \
  ipv4.dns "192.168.50.1"

IPv6 DHCP or router advertisements can supply another DNS server. If IPv6 is in use, confirm that this resolver also knows the internal zone. Applying connection changes may interrupt SSH, so keep console access or another maintenance path before reactivating the connection using the method appropriate for that host.

Check resolvectl status again, then test both systemd-resolved and ordinary application resolution:

resolvectl query gitlab.home.arpa
getent hosts gitlab.home.arpa
getent hosts registry.home.arpa

A successful resolvectl query proves that systemd-resolved’s path works. It does not prove that the resolver file passed to Pods is appropriate.

Why 127.0.0.53 can break CoreDNS

On the host, 127.0.0.53 is systemd-resolved’s local stub. That can be a valid entry point for host programs. Inside an ordinary Pod network namespace, however, loopback refers to the Pod itself rather than the host.

If kubelet passes that loopback nameserver to CoreDNS and the Corefile uses forward . /etc/resolv.conf, a forwarding loop can result. A plugin/loop: Loop ... detected error calls for inspecting that forwarding relationship, not removing the loop plugin. The detector only covers certain simple loops at startup; its silence is not proof that the topology is correct. The CoreDNS loop documentation describes both the failure and the detector’s limits.

Before selecting a non-stub resolver file, inspect its contents. It must list upstreams reachable from the relevant node and Pod networks, and the router must not forward the same queries back to CoreDNS. /run/systemd/resolve/resolv.conf is a common candidate, not a guarantee.

Choose kubelet’s resolver explicitly in RKE2

My original host runbook changes the /etc/resolv.conf symlink. An alternative that preserves the host stub is to specify the resolver at the RKE2 layer. These are alternatives, not two steps that must both be applied.

Merge the following key into /etc/rancher/rke2/config.yaml on each relevant node, keeping its other settings:

resolv-conf: /run/systemd/resolve/resolv.conf

This is an RKE2 key, distinct from kubelet’s resolvConf field. The RKE2 agent reference documents it. A flat file containing DNS servers from several interfaces or a VPN may not preserve per-domain routing. In that case, use a configuration-managed resolver file containing suitable upstreams and point RKE2 at that file instead.

Apply node service changes in a maintenance window, choosing rke2-server or rke2-agent for the node’s role and proceeding one node at a time. Expect temporary API unavailability on a single-node control plane. Once the setting is effective, identify the actual CoreDNS Deployment and recreate the affected Pods:

kubectl -n kube-system get deployments
kubectl -n kube-system rollout restart deployment/<coredns-deployment>
kubectl -n kube-system rollout status deployment/<coredns-deployment>

Restarting CoreDNS without fixing the upstream simply gives a new Pod the same broken input.

Test cluster, internal, and public names separately

This creates a short-lived diagnostic Pod. Confirm that the cluster can obtain the test image so an image-pull failure is not mistaken for a DNS failure inside the test:

kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- sh

Inside the Pod:

cat /etc/resolv.conf
nslookup kubernetes.default.svc.cluster.local
nslookup gitlab.home.arpa
nslookup registry.home.arpa
nslookup example.com
exit

Testing only example.com misses the internal zone. Testing only GitLab misses Kubernetes Service discovery. Replace cluster.local if the cluster uses a different domain.

DNS locates a service; it does not grant access. A new internal hostname must also match TLS certificates, SSH known hosts, and registry credentials. Public browser access can continue through Cloudflare Tunnel. VPN access needs internal DNS as well as routing, rather than assuming names become available merely because the tunnel is connected.