Hello,
I am currently running krun microVMs through rootless Podman/crun. I directly attach TAP interfaces using krun.tap_name (containers/crun#2200). This provides a transparent, direct networking required for specific tasks.
This is the behaviour I observe with libkrun ≤ 1.19.4:
The guest starts with a trusted bootstrap that runs as guest root, configures eth0 and explicit routes, then drops to uid 1000 with setpriv, no_new_privs, and only the requested capabilities.
pacman -Q crun libkrun libkrunfw
crun 1.30.1-1
libkrun 1.19.4-1
libkrunfw 5.6.2-1podman run --rm \
--runtime=/usr/bin/crun \
--annotation run.oci.handler=krun \
--userns=keep-id:uid=1000,gid=1000 \
--user=1000:1000 \
docker.io/library/alpine:3.22 \
id
uid=0(root) gid=0(root)Since 1.19.5, commit 20aa1ef ("init: Added GID and UID parsing"), if I understand it correctly, it applies the OCI uid/gid before execvp(). As a result, the bootstrap starts directly unprivileged and cannot perform the initial privileged security and network setup.
pacman -Q crun libkrun libkrunfw
crun 1.30.1-1
libkrun 1.19.6-1
libkrunfw 5.6.2-1podman run --rm \
--runtime=/usr/bin/crun \
--annotation run.oci.handler=krun \
--userns=keep-id:uid=1000,gid=1000 \
--user=1000:1000 \
docker.io/library/alpine:3.22 \
id
uid=1000(test) gid=1000(1000)I agree that honouring the OCI process user is the expected behaviour. The difficulty is that --user also sets the VMM's identity: I keep the VMM unprivileged (mapped host user, --cap-drop=ALL), and --user=0:0 would mean running it as container root, which I'd rather avoid.
So, I would like to ask: is there a supported way to run a privileged guest bootstrap first, while keeping the host-side VMM rootless, and only apply the OCI uid/gid afterwards?
I also noticed 20aa1ef is only on stable-1.19.x; the rewritten init on main doesn't seem to have an equivalent yet, so I wasn't sure which behaviour is intended for 2.0.
Many thanks in advance.
Thank you! jm
Activity
8
javimox29d ago@slp : hi. Thanks.
I am also thinking of small change on the crun side, an opt-in annotation such as krun.bootstrap_user=0:0, which would only override the uid/gid in the guest copy of the OCI config while keeping the host-side VMM running as the rootless mapped user.
The flow would then be roughly:
rootless VMM
-> guest bootstrap as root
-> configure networking/security
-> drop to the intended workload uid/gidBut if a privileged bootstrap phase is something that makes sense directly in libkrun, that would probably be cleaner and also useful outside crun.
Do you have a preference for where this should live? I’d be happy to explore an implementation once the intended interface is clear.
Ty, jm.
edit for context: my bootstrap is the guest's entry point itself. It starts as guest root and drops privileges on its own, because it also establishes the final per-process state (CAP_NET_RAW when requested, the capability bounding set, and no_new_privs) before execing the actual workload. A separate hook run by init before switching user couldn't pass that state on.
I think here would be enough being able to choose the guest's initial uid/gid independently of the host-side VMM identity.
slp28d agoBut if a privileged bootstrap phase is something that makes sense directly in libkrun, that would probably be cleaner and also useful outside crun.
Do you have a preference for where this should live? I’d be happy to explore an implementation once the intended interface is clear.
I think it'd be nicer extending the init-blob API to allow configuring a bootstrap script which will be run before switching uid/gid. cc/ @mtjhrc
mtjhrc28d agothe bootstrap starts directly unprivileged and cannot perform the initial privileged security and network setup.
Could you clarify what you mean by "bootstrap" here? Are you referring to the guest init or the user guest payload? The user payload (and only user payload) is supposed to run as the mapped user - that's the intended behavior.
I also noticed https://github.com/libkrun/libkrun/commit/20aa1ef06e31b66e6860a05512a99e40caf186fc is only on stable-1.19.x; the rewritten init on main doesn't seem to have an equivalent yet, so I wasn't sure which behaviour is intended for 2.0.
Yes we want to support it, we need to port this.
which would only override the uid/gid in the guest copy of the OCI config while keeping the host-side VMM running as the rootless mapped user.
Could you elaborate on the use case for this? We could potentially add it as an additional security mechanism, but I don't think it should be the default because it breaks expected privileges for virtiofs after elevating to root. If the payload starts as a normal user and then elevates to root (e.g., via sudo), the filesystem operations won't work correctly because, from the filesystem's perspective, the user isn't actually root.
So for 2.x, while that wasn't the case in 1.x, the plan is to make running the VMM as root the default, with UID/GID only affecting the user payload. We could still support a non-root VMM as an option, but it shouldn't be the default. This could potentially be configured by an OCI annotation (e.g., krun.vmm_user=) and entirely implemented in crun (it decides which user libkrun runs as). But like I said I am not so sure if it's a good idea.
I think it'd be nicer extending the
init-blobAPI to allow configuring a bootstrap script which will be run before switching uid/gid. cc/ @mtjhrc
Yes such APIs/hooks would be welcome in 2.x. In technical terms you can add functions to the InitConfig builder which passes it to the guest init via a virtual json file in the guest filesystem.
mtjhrc28d agoOh I missed your EDIT:
edit for context: my bootstrap is the guest's entry point itself. It starts as guest root and drops privileges on its own, because it also establishes the final per-process state (CAP_NET_RAW when requested, the capability bounding set, and no_new_privs) before execing the actual workload
In that case, couldn't you implement my proposed annotation (krun.vmm_user=) + change the entry point of the container to your bootstrap script, which manually drops the privileges? Or is changing the entry-point not an option in your setup?
slp28d agoOh I missed your EDIT:
edit for context: my bootstrap is the guest's entry point itself. It starts as guest root and drops privileges on its own, because it also establishes the final per-process state (CAP_NET_RAW when requested, the capability bounding set, and no_new_privs) before execing the actual workload
In that case, couldn't you implement my proposed annotation (
krun.vmm_user=) + change the entry point of the container to your bootstrap script, which manually drops the privileges? Or is changing the entry-point not an option in your setup?
I think extending init-blob to support custom scripts would be nicer, muvm already needs them. It could even support two scripts, privileged and non-privileged, one to be run before the uid/gid change.
javimox28d agoThanks both.
Could you clarify what you mean by "bootstrap" here? Are you referring to the guest init or the user guest payload? The user payload (and only user payload) is supposed to run as the mapped user - that's the intended behavior.
@mtjhrc: by "bootstrap" I meant the user payload, not the guest init; changing the entry point is an option. Actually, the entry point is already the bootstrap: it starts as guest root, configures a TAP interface and routes, sets the final capability state and no_new_privs, drops to uid/gid 1000, and then execs the workload.
In that case, couldn't you implement my proposed annotation (krun.vmm_user=) + change the entry point of the container to your bootstrap script, which manually drops the privileges? Or is changing the entry-point not an option in your setup?
The reason I keep the VMM as the mapped non-root user is mainly least privilege for an untrusted-workload sandbox. i want the host side VMM constrained to the invoking user's privileges. It's also practical for the TAP setup, with --user=0:0 and the existing keep-id mapping I got TUNSETIFF: EPERM, whereas the TAP works when the VMM runs as its owning host user.
So process.user=0:0 for the guest payload plus something like krun.vmm_user=1000:1000 for the host-side VMM looks like a very good fit for this use case.
I understand the virtiofs concern. The privileged phase is intentionally very small in my case; it only does network and capability setup, and then immediately drops to uid/gid 1000.
I think extending init-blob to support custom scripts would be nicer, muvm already needs them. It could even support two scripts, privileged and non-privileged, one to be run before the uid/gid change.
@slp: custom scripts in init-blob sound also useful beyond my case. For this particular setup, the entry point already owns the privilege transition and final process state, so if the VMM and payload identities can be separated I probably wouldn't need an additional init hook.
If you think krun.vmm_user is a reasonable optional interface, I could prototype it and test the crun side.
Ty, jm
javimox22d agoI opened the PR containers/crun#2318.
It adds the annotation krun.vmm_user=UID:GID. The VMM runs with that UID/GID, while process.user continues to apply to the workload inside the guest.
With --user=0:0 --annotation krun.vmm_user=1000:1000 on libkrun 1.19.6, the workload inside the guest starts as root while the VMM runs as UID/GID 1000 on the host.
With --cap-drop=ALL, the VMM also runs without capabilities and with NoNewPrivs=1.
Hi, the ability to run a custom privileged bootstrap is not supported by
initright now, but could be an interesting addition to both 2.x (which will also get the uid/gid parsing behavior) and 1.x.