#905

init: optional vsock exec server so OCI runtimes can implement exec and signal delivery

OpenAlphaJackopened 24d ago

krun containers can't do podman exec or health checks (containers/crun#2090) and podman stop can't reach the workload (#10, containers/crun#2158), because nothing on the host can talk to a process inside the VM.

Proposal: an opt-in server in the init binary, exec_server: {port} in the init config, exposed through Builder::exec_server(port) / krun_init_builder_exec_server(). The VMM maps the port to a Unix socket with VsockDevice::add_unix_port(port, path, true). Small framed protocol (type u8, len u32 LE), host-CID only: spawn a process with argv/env/cwd/uid/gid/pty, relay stdio, winsize and signals, report the exit status; plus a request that signals the workload. Also covers the GitLab runner case in #667.

I've tested this approach end-to-end with podman 6.1 on a forked libkrun2 and crun.

Questions before I open the PR:

  • is a forked server process from init acceptable (thread can't survive KRUN_INIT_PID1=1)
  • should the port be fixed or configurable
  • should the exec'd process also get the OCI rlimits/caps or is uid/gid enough?

Activity

2
mtjhrc23d ago

This feature is very much welcome! In fact it's something I've been meaning to implement as well. This https://github.com/libkrun/libkrun/pull/502 unfinished virtio-console work was motivated by a use case like this.

My original idea was to use a separate virtio-console control port, while exec -it would use dynamically-added virtio console ports (shows up as hvcN in the guest):

host PTY <-> virtio-console <-> /dev/hvcN in guest <-> exec'd process

Upon more investigation today, I think vsock is probably better currently. The main issue with the console approach is the current HVC tty allocation limit in the guest kernel and the fact it needs to preallocate many virtio queues for the ports upfront. But still we could possibly design the API so this is just an implementation detail for the user of this.

Regarding your questions:

  • I'd say we can have the server as a thread in krun-init for now. I'd be fine with making the control server incompatible with that mode initially and revisiting it if needed. On the other hand a process is fine too, it's just weird for this to not be handled by init process, when we already typically have an init process anyway. (btw KRUN_INIT_PID1 is supposed to move to a proper InitBuilder api method instead of env var.)

  • I think we can default to a fixed/private guest vsock port, but make it overridable. This also mean we need to pass the vsock device as an argument to InitBuilder::apply.

  • should the exec'd process also get the OCI rlimits/caps or is uid/gid enough?

    Yes I suppose. We should try to mirror normal container semantics if possible. But it's not important to handle in the initial PR.

Also one thing to be careful about with the host control endpoint is AF_UNIX TSI. If KRUN_TSI_FLAGS_HIJACK_UNIX were enabled, a guest process could potentially ask TSI to connect to the host control socket, which would then appear to krun-init as a host-originated vsock connection. So the host-CID check alone isn't sufficient as a long-term security boundary; the control endpoint needs to be unreachable through TSI.

Also regarding your PoC one thing I would really change is is the host-side API/ABI. Right now crun knows the guest port, protocol version, frame IDs, framing, exec request encoding, TTY resize encoding, exit-status encoding, etc. I'd rather keep all of that private to krun-init and expose normal function calls to crun. It also means crun doesn't need to maintain another protocol parser. This is the same idea as with the init configuration json - In libkrun 2.0 the API contract between init-blob on the host and the guest init isn't supposed to be public and can freely change.

For example, I could imagine the API looking roughly like this:

// `state_dir` is crun's already-resolved per-container runtime state
// directory, e.g. $XDG_RUNTIME_DIR/crun/<container-id> with crun's
// rootless default, or under an explicitly configured state root.
let control_socket = state_dir.join("krun-init.sock");

let init = InitBuilder::new()
    // Enabling this also arranges the private guest-vsock <-> host
    // control-endpoint mapping.
    .control_socket(&control_socket)
    // .workload_is_pid1(true)  // incompatible with control service for now
    .build()?;

Then a later crun exec resolves the same per-container state directory and connects to it:

let control_socket = state_dir.join("krun-init.sock");

// Represents the running krun-init, not an individual exec session.
let controller = InitController::open(&control_socket)?;

// NOTE: Initial PR doesn't need to implement all of these,
// this is just illustrative
let request = ExecRequest::builder("/bin/bash")
    .arg("-l")
    .env("FOO", "bar")
    .cwd("/workspace")
    .uid(uid)
    .gid(gid)
    .additional_gids(groups)
    .rlimit(...)
    .build()?;


let process = controller.exec(
    request,
    ExecIO::Pipes(stdout_fd, stdin_fd, stderr_fd),
    // or:
    // ExecIO::Tty(tty_fd)
)?;

process.signal(SIGTERM)?;

let exit_code = process.wait();

// We can also have control commands that are not specific to the payload
controller.signal_entrypoint(SIGTERM)?;

Also for the guest <-> host protocol, since both sides are Rust we should have a shared crate with structs containing serde annotations and use some already existing serialization format instead of implementing it ourselves. (the currently duplicated serde structs for the init config json should also eventually go to the proposed crate).

AlphaJack22d ago

Thanks, I reworked the branch along those lines:

  • Server is now a thread in krun-init (refused with KRUN_INIT_PID1=1 for now, prints a message).
  • New crate init/common (krun-init-common) holds the serde protocol types plus the host client and the guest server; wire format is bincode (already in the lock). Nothing of it is public.
  • Host API in libkrun_init: Builder::control_socket(path) (+ control_vsock_port() to override the private default), Config::apply_with_vsock(overlay, payload, vsock) which adds the unix port mapping itself, and Controller::open(path, timeout) → exec_pipes()/exec_tty() → Process::wait(), plus Controller::signal_entrypoint(). ffier can't express an optional foreign &mut VsockDevice, so apply() keeps its signature and errors if a control socket is configured.
  • TSI: the muxer now rejects guest connect()s whose AF_UNIX path matches a listening unix-port mapping, so the control endpoint is unreachable with HIJACK_UNIX.
  • Processes are spawned through a double fork in the server, so init's waitpid(-1) reaper stays untouched and the same code is tested in-process without a VM; the helper hands back a pidfd so signals can't hit a reused pid.
  • rlimits/caps for the exec'd process: not in this PR, as agreed.
  • Tests: client vs server in one process (krun-init-common), muxer path check, and a new init-control integration test (make test TEST=init-control). crun side is on AlphaJack/crun krun-exec and uses only the C API; podman exec/healthcheck/stop work end to end with it.

PR here: #912 How do you plan to handle the crun migration to libkrun 2.0?