gvproxy brakes under load: virtio_net virtio9 eth0: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out

We experience this issue when running ramen drenv stress test. When this happens, podman becomes inaccessible (e.g. podman ps fails to access the machine), and accessing registry containers in podman fails. The only way to recover is to stop and start the podman machine.

Reproduced with vfkit (applehv) and krunkit (libkrun) with podman 5.8.0.

This happens much more with libkrun - we get much faster networking (3 times faster iperf3 --reversin e), but the issue is reproduced almost in very run.

How we reproduce:

• Run drenv stress test with 100 runs (~8 hours)
  • vfkit: reproduce once in 15 runs
  • libkrun: reproduce twice in 3 runs

The workload:

• we start 3 minikube clusters in parallel and provision lot of stuff in parallel. Every cluster pulls about 3 GiB of images
• clusters are configured to use registry cache containers in podman, so gvproxy provide 3 GiB of image to 3 clusters
• For more info on how we setup the registry cache see https://github.com/RamenDR/ramen/tree/main/test/registry-cache

Example podman log with a broken podman machine: podman-machine-default.log.gz

Reproduced with and without #613 - the change does not affect this issue.

I tried to get a goroutine stacktrace:

• with kill -QUIT - but it seems that gvproxy stderr is redirected to /dev/null.
• with dlv - it fails to attach to gvproxy since we don't have debug symbols

There is pprof endpoint but it is gated behind -debug flag which is not enable by podamn, and podman also do not use --listen so we don't have access to the pprof endpoint.

So we have several issues:

• Stability issues in production
• No way wo debug in production
• No documentation on how to debug gvproxy in production