What happens
All four virtiofs shares of a podman machine created with the libkrun provider are dead inside the guest: every read through the mount returns Connection refused (ECONNREFUSED). The VM boots fine, the GPU device works, SSH works; only the host-shared filesystems are unusable.
Observed on an M3 Pro with macOS 26.7.1, podman 6.1.3 (client), krunkit 1.3.2 + libkrun 1.19.6 + virglrenderer-krun 0.10.4e from the libkrun homebrew tap (relocated from the arm64_tahoe bottles), podman machine-os 6.1 image (Fedora CoreOS 44.20260829.3.1). The shares are the four podman machine defaults: /Users, /private, /var/folders and the host ~/.config/containers to guest /etc/containers.
Reproducing
Standard podman machine init --provider libkrun plus podman machine start (krunkit receives the usual --device virtio-fs,sharedDir=...,mountTag=... options), then inside the guest:
$ mount | grep virtiofs
a2a0ee2c717462feb1de2f5afd59de5fd2d8 on /Users type virtiofs (rw,relatime,context=system_u:object_r:nfs_t:s0)
...
$ ls /Users
ls: cannot access '/Users': Connection refused
$ ls /etc/containers/
ls: cannot access '/etc/containers/': Connection refusedEvery share behaves the same, including fresh first boots, so it is not a remount artifact. The guest kernel logs no virtiofs errors; the failure surfaces as ECONNREFUSED on access, which suggests the host-side backend for the shares is not serving.
Consequences and workaround
This breaks the guest podman service on podman machines (Failed to obtain podman configuration: open /etc/containers/storage.conf: connection refused) and every host bind mount. I route around it by shadowing /etc/containers with a tmpfs populated from /usr/share/containers and keeping container workloads guest-local, but obviously the shares themselves remain unusable.
Happy to collect any host-side logs that would help: tell me which logging to enable on krunkit/libkrun for the virtiofs backend and I will attach them (I could not find a knob that surfaces backend errors for these devices).
Activity
5
paoloantinori17d agoThe guest kernel is 7.1.10-200.fc44.aarch64, from the stock image that podman 6.1.3 (brew, on macOS 26 with an Apple M3 Pro) pulls for machines created with podman machine init --provider libkrun.
The host runs brew podman only and has no system libkrun, so leftovers from a previous installation are unlikely. I reproduced the failure with two stacks anyway, the bundled pkg one and the krunkit 1.3.2 and libkrun 1.19.6 bottles from the slp/mesa-krunkit homebrew tap relocated into a private directory. Both showed the same ECONNREFUSED, so the relocation is not the trigger.
The symptom was that all four host-share mounts mounted, but the first actual use returned Connection refused. Reading /etc/containers/storage.conf from the guest failed the same way, which is why I worked around it by shadowing /etc/containers with a tmpfs.
One fresh data point from today: on the machine I have running now, created the same way on 2026-10-07 and up for days, the shares work, and ls /Users from the guest lists the host home. The failure does not look deterministic. If it helps I can create a machine from scratch and capture the full journal of a failing boot.
slp17d agoOne fresh data point from today: on the machine I have running now, created the same way on 2026-10-07 and up for days, the shares work, and ls /Users from the guest lists the host home.
Is that the same machine where you found the issue before? If so, is there any chance the previous guest image had a different kernel?
The failure does not look deterministic. If it helps I can create a machine from scratch and capture the full journal of a failing boot.
That could help, thanks.
paoloantinori17d agoIs that the same machine where you found the issue before?
No. The machines that showed the failure were created on Oct 6 and 7 and removed at the end of that debugging session; I filed the issue on Oct 7 at 18:20 CEST. The machine that works now was created later that same evening, 22:36 CEST, with the same init line. It has had exactly three boots since, with journal history complete from the first one (Oct 7 22:37:39 CEST). All three boots discover the four virtiofs devices cleanly, and the journals contain zero "connection refused" lines, so this machine never showed the failure.
One caveat. Between Oct 8 and this morning its workload never touched the shares, because I stage everything with podman cp due to this bug, so the first explicit re-verification happened today: ls /Users from the guest lists the host home, on the current boot and on the previous one.
If so, is there any chance the previous guest image had a different kernel?
No. During the investigation I recorded the kernel of a failing machine verbatim: 7.1.10-200.fc44.aarch64 #1 SMP PREEMPT_DYNAMIC Sun Aug 23 16:21:29 UTC 2026. The working machine, read just now, reports the identical string with the same build timestamp. The machine-os cache was deleted between the two eras, so the working machine pulled quay.io/podman/machine-os:6.1 fresh at creation and still got the same image build.
So the difference between failing and working is neither the machine image nor the kernel. One more environment note: the podman client on this host has since moved from 6.1.3 to 6.1.4, so the from-scratch attempts will run under 6.1.4 unless I pin the client back.
Next I will create a machine from scratch, capture the full journal of the boot together with the host-side krunkit output, and attach both here. If the failure does not show up on the first attempt I will retry a few times.
paoloantinori17d agoFollow-up from the from-scratch attempts. I created four fresh machines today with the same recipe as the original ones (sandboxed config dir, the official podman machine start choreography, krunkit 1.3.2 + libkrun 1.19.6 from the tap). I exercised the four virtiofs shares on the first boot of three of them (the fourth was replaced before testing due to a slip in my own test script): every one came up with the shares working (ls /Users lists the host home; /var/folders and /private read fine). No ECONNREFUSED on any boot, so there is no failing-boot journal to attach yet.
The attempts also add a late data point on your kernel question: the image cache on this host was refreshed today, and a fresh pull of quay.io/podman/machine-os:6.1 now ships kernel 7.2.5-200.fc44.aarch64, while both original observations (the failing machines and the working one) ran 7.1.10-200.fc44.aarch64. So the floating tag does drift across days; it just had not drifted between the two machines I compared in my previous comment. Since the original failures, two pieces have moved on this host: the podman client (6.1.3 to 6.1.4) and now the guest image (7.1.10 to 7.2.5).
With no reproduction on the current stack, the remaining way to chase the failure is to pin the old pieces back: the machine-os build the failing machines used (CoreOS 44.20260829.3.1) and the 6.1.3 client. Is that image build still obtainable on your side, or is the drift plus the current-stack non-reproduction enough to work with?
Hi, I've been trying to reproduce this issue on a couple of different machines, but wasn't able to do so. What kernel is that CoreOS image running? Is there any chance you have leftovers from a previous libkrun installation laying somewhere?