Starting with gvproxy v0.9.0, changing DNS servers after starting Podman can break DNS lookups in Alpine containers due to DNS server list caching within gvproxy itself. This is commonly triggered by DISconnecting from a corporate VPN. Just changing internet connections won't do it by itself because you need to satisfy both:
- A change in host DNS servers
- The old DNS server is no longer accessible.
Disconnecting from a corporate VPN can reliably trigger this since the private DNS servers generally aren't accessible to the public internet. If you just swap internet connections, you can probably still access the old ISP's DNS servers on the public internet.
IPv4 queries still work, but AAAA queries (IPv6 addresses) time out, since they take different paths in gvproxy. The AAAA queries go to the cached server which is no longer routable/accessible, and the A queries go to the system resolver that follows the DNS server changes correctly.
Now as to why Alpine is relevant, it uses musl, which fails combined DNS lookups entirely if either A or AAAA times out, returning nothing for either IPv4 or IPv6, even though the IPv4 half succeeded just fine.
Reproduce
- Connect to a VPN that supplies privately routed DNS servers.
- Run
podman machine stop && podman machine start. - Disconnect the VPN, then run:
podman run --rm docker.io/library/python:3.12-alpine python -c '
import socket
for f in (socket.AF_INET, socket.AF_INET6, socket.AF_UNSPEC):
try:
print(f.name, socket.getaddrinfo("github.com", 443, f, socket.SOCK_STREAM)[0][4][0])
except socket.gaierror as e:
print(f.name, e)'IPv4 succeeds. IPv6 and the combined lookup both fail with Try again after about five seconds. Reconnecting the VPN fixes it without restarting Podman. The same test does not fail with gvproxy v0.8.9.
AAAA queries use raw forwarding added in #696. Its DNS server list is read only at startup, so it keeps using the VPN's server after disconnecting. IPv4 queries use the host resolver and follow the current configuration.
Expected: DNS should keep working after the host's DNS configuration changes, without restarting the VM.
Tested on macOS 26.5 arm64, Podman 6.1.4, gvproxy v0.9.0, using applehv.
Activity
5Thanks for the quick fix!
I think there's a second half to this fix though. Here is a quick attempt at what I think is missing. I haven't gone through this in detail, which is why it's not a PR. Essentially we also need to refresh the DNS server list as well to keep things fresh.
4. Thanks for the quick fix! I think there's a second half to this fix though. Here is a quick attempt at what I think is missing. I haven't gone through this in detail, which is why it's not a PR. Essentially we also need to refresh the DNS server list as well to keep things fresh.
Something like this will indeed also be needed. However, the AAAA behaviour is a regression from previous releases, while the generic fallback for SOA/… is something new, so it is imo more a bug than a regression that it does not work 100%.
The reason I’m making that distinction is that trying to get the DNS server list fresh has proven to be quite problematic in the past (we’ve reverted the code related to this :-/ ), and we’ll probably make a 0.9.1 release relatively soon, so I’d like to get the AAAA fix out in it, and to take more time to add the DNS server list refresh.
Makes total sense to me. I can even file a separate issue with the DNS refresh stuff if you want to get this issue closed ASAP.
Makes total sense to me. I can even file a separate issue with the DNS refresh stuff if you want to get this issue closed ASAP.
Would be useful so that we keep track of what is still missing!
Thanks for the report, I should have anticipated this... I’ve opened https://github.com/containers/gvisor-tap-vsock/pull/755 which should fix this.