Lab 3: Virtio-Vsock

Background

This is a trace-it-and-integrate lab. vsock is a host↔guest communication channel that needs no network: no IP, no MAC, no routing, no firewall. It is a socket family (AF_VSOCK) addressed by a (CID, port) pair — a context ID identifying the endpoint plus a port number. The guest talks to the host with ordinary socket(AF_VSOCK, ...) / connect() / bind() calls; Firecracker bridges that to Unix domain sockets on the host. This is the control channel that firecracker-containerd and Kata Containers run their in-VM agents over — the host orchestrator talks to a containerd shim agent or a Kata agent inside the guest without giving the guest a network. You will configure a vsock device, run traffic host→guest and guest→host, trace the CID/port multiplexing and the Unix-socket bridge, and connect it to how the real systems use it.

Vsock is the most structured of the three devices: where block is one stateless request queue and net is two frame queues, vsock carries multiplexed, connection-oriented streams over its queues, with a per-connection state machine (connect, data, credit-update, shutdown). You will not trace every state here; you will trace enough to understand the bridge and the addressing.

Why This Lab Matters for Contributors

• vsock is the integration seam between Firecracker and its orchestrators. The firecracker-containerd integration lab and Kata both depend on it; understanding it makes those systems legible instead of magic.
• vsock connection-state bugs (a stuck connection, a credit miscalculation, a leaked host socket) are a real and tricky issue class — they live in a per-connection state machine, not a stateless request loop. See the issue-roadmap virtio stage.
• It cements the virtio-vsock deep dive and shows a device whose "host I/O" is another socket, not a file or a TAP.

Prerequisites

• Lab 1 and Lab 2 complete.
• A guest with python3 or socat/nc that speaks AF_VSOCK (recent socat and python3 do; nc usually does not). The guest kernel needs vmw_vsock_virtio_transport (most Firecracker guest kernels build it in — zcat /proc/config.gz | grep VSOCK to check).
• Verify the vsock device, its queues, and the host-socket bridge exist on your branch:

# All must return hits. If empty, the layout moved — find by role.
rg -l "impl VirtioDevice for"        src/vmm/src/devices/virtio/vsock/
rg -n "RXQ_INDEX|TXQ_INDEX|EVQ_INDEX|guest_cid|VSOCK_HOST_CID|HOST_CID" \
   src/vmm/src/devices/virtio/vsock/
rg -n "UnixListener|UnixStream|uds_path|unix_socket|Muxer|VsockMuxer|connect|listen" \
   src/vmm/src/devices/virtio/vsock/

The Addressing Model You Are Tracing

            CID 2 (the host, well-known)            CID = guest_cid (e.g. 3)
        ┌─────────────────────────────┐         ┌──────────────────────────┐
        │ HOST                        │         │ GUEST                    │
        │  Firecracker vsock device   │ vsock   │  AF_VSOCK sockets        │
        │   ┌── uds_path  (the bridge)│◄──tx/rx─►│   connect()/bind()       │
        │   │   /tmp/v.sock           │  queues │                          │
        │   │   /tmp/v.sock_PORT      │         │                          │
        └───┼─────────────────────────┘         └──────────────────────────┘
            │
   host process connects/listens here (a Unix socket), NOT AF_VSOCK

Two well-known facts to fix:

• The host is always CID 2 (VMADDR_CID_HOST). The guest is whatever guest_cid you configure (must be ≥ 3; CID 0 and 1 are reserved). Confirm the constant: rg -n "VSOCK_HOST_CID|HOST_CID|CID.*2|VMADDR_CID_HOST" src/vmm/src/devices/virtio/vsock/.
• The host side is bridged to Unix sockets, not real AF_VSOCK. Firecracker translates between the guest's AF_VSOCK and host Unix domain sockets rooted at the uds_path you configure. This is deliberate: the host process talks plain Unix sockets and never needs AF_VSOCK privileges or a vsock-capable host kernel.

Step-by-Step Tasks

Step 1: Configure a vsock device

vsock is a single pre-boot device (PUT /vsock). Give it a guest_cid and a host uds_path — the filesystem path that becomes the host end of the bridge.

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/debug/firecracker
API=/tmp/fc-vsock.sock
LOG=/tmp/fc-vsock.log
UDS=/tmp/v.sock
rm -f "$API" "$UDS"*; : > "$LOG"
sudo "$BIN" --api-sock "$API" &

curl -X PUT --unix-socket "$API" --data \
  '{"log_path":"'"$LOG"'","level":"Warning","show_level":true,"show_log_origin":true}' \
  http://localhost/logger

curl -X PUT --unix-socket "$API" --data \
  '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1"}' \
  http://localhost/boot-source

curl -X PUT --unix-socket "$API" --data \
  '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs

# The vsock device: guest is CID 3, host bridge socket is /tmp/v.sock.
curl -X PUT --unix-socket "$API" --data \
  '{"vsock_id":"vsock0","guest_cid":3,"uds_path":"'"$UDS"'"}' \
  http://localhost/vsock

curl -X PUT --unix-socket "$API" --data \
  '{"vcpu_count":2,"mem_size_mib":1024}' http://localhost/machine-config

curl -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' http://localhost/actions

After boot, the host bridge socket exists:

ls -l /tmp/v.sock      # a Unix socket Firecracker created and listens on

Note: vsock_id is required by the schema even though there is only one vsock device; verify the exact field names on your branch: rg -n "guest_cid|uds_path|vsock_id" src/firecracker/swagger/firecracker.yaml.

Step 2: The bridge protocol — host connects to the guest

This is the part that trips everyone up. To reach a guest listener from the host, you connect to the uds_path Unix socket and then send a single text line CONNECT <port>\n. Firecracker reads that line, opens a vsock connection to (guest_cid, port) inside the guest, and from then on the Unix socket is a transparent byte pipe.

Start a listener in the guest first:

# Guest serial console: listen on AF_VSOCK port 5000 with socat (or a tiny python server).
socat - VSOCK-LISTEN:5000,fork
#   (leave it running; it will echo whatever it receives to its stdout)

Now connect from the host through the bridge:

# Host: connect to the Unix socket, speak the CONNECT handshake, then send data.
{ printf 'CONNECT 5000\n'; sleep 0.2; printf 'hello-from-host\n'; sleep 0.5; } \
  | socat - UNIX-CONNECT:/tmp/v.sock

Firecracker replies OK <assigned_host_port>\n on success, then pipes bytes through. You should see hello-from-host arrive at the guest's socat. You just sent host→guest traffic with no network at all.

# Confirm the handshake constant in the bridge code:
rg -n "CONNECT|\"OK \"|OK_LINE|fn send_response|handshake" src/vmm/src/devices/virtio/vsock/

Step 3: The other direction — guest connects to the host

To reach a host listener from the guest, the host listens on a Unix socket named <uds_path>_<port> (the bridge appends _PORT), and the guest does a plain AF_VSOCK connect to (CID 2, port).

# Host: listen on the per-port bridge socket for guest connections to port 6000.
socat - UNIX-LISTEN:/tmp/v.sock_6000,fork
#   (leave it running)

# Guest: connect to the host (CID 2) on port 6000 and send data.
{ printf 'hello-from-guest\n'; sleep 0.3; } | socat - VSOCK-CONNECT:2:6000

hello-from-guest appears at the host's socat. Note the asymmetry: host→guest uses the single uds_path plus a CONNECT <port> line; guest→host uses a per-port uds_path_<port> listener and a direct AF_VSOCK connect to CID 2. Confirm the suffix logic:

rg -n "uds_path|_|format!.*port|local_port|host_sock_path|push_str" \
   src/vmm/src/devices/virtio/vsock/

Step 4: Trace the CID/port multiplexing through the muxer

vsock multiplexes many connections over one device and its queues. The component that tracks them is the muxer — a map from (local_port, peer_port) (and CID) to a connection object with its own state and a host-side socket. Find it.

rg -n "struct VsockMuxer|MuxerConnection|struct VsockConnection|ConnMap|HashMap|local_port|peer_port|key" \
   src/vmm/src/devices/virtio/vsock/

Add trace points to the connection lifecycle (adjust names to what rg shows):

#![allow(unused)]
fn main() {
// When a new connection is created in the muxer:
log::warn!("[trace] vsock new conn: local_port={} peer_port={}", local_port, peer_port);
// When a data packet is moved guest→host or host→guest:
log::warn!("[trace] vsock data: dir={} len={} local_port={}", dir, len, local_port);
// When a connection is torn down:
log::warn!("[trace] vsock close: local_port={}", local_port);
}

Rebuild and re-run Steps 2–3 with two simultaneous connections (different ports). The trace shows each (local_port, peer_port) tracked independently — that is the multiplexing. The single vsock device, with its RX/TX/event queues, carries all of them; the muxer demultiplexes by port.

Note: vsock has three virtqueues — RX, TX, and an event queue (used for things like the guest CID changing on migration). Confirm: rg -n "RXQ_INDEX|TXQ_INDEX|EVQ_INDEX|NUM_QUEUES" src/vmm/src/devices/virtio/vsock/. You traced RX and TX on net; vsock packets carry an virtio_vsock_hdr with src_cid/dst_cid/src_port/ dst_port/op/len — that header is the addressing and the per-connection protocol.

Step 5: Watch the host-side sockets and packet ops

Observe the bridge from the host side without touching code. List the bridge sockets and watch connections come and go.

# The host bridge sockets (the main one plus per-port listeners you created):
ls -l /tmp/v.sock*
ss -x | grep v.sock           # Unix sockets and their peers

Then read the vsock packet op codes the protocol uses — request, response, data, credit-update, shutdown — so the trace lines make sense:

rg -n "VSOCK_OP_REQUEST|VSOCK_OP_RESPONSE|VSOCK_OP_RW|VSOCK_OP_CREDIT|VSOCK_OP_SHUTDOWN|VSOCK_OP_RST|enum.*Op" \
   src/vmm/src/devices/virtio/vsock/

Each connection is a small state machine over these ops: a connect is REQUEST→RESPONSE, data is RW, flow control is CREDIT_UPDATE/CREDIT_REQUEST, teardown is SHUTDOWN/RST. A bug in this machine shows up as a hung or half-open connection — far nastier than a block read returning wrong bytes.

Step 6: Connect it to firecracker-containerd / Kata

Now the integration payoff. In firecracker-containerd, the host runtime shim starts a microVM and talks to an agent inside the guest over vsock — the agent runs runc to launch the actual OCI container, and the shim sends it create/start/exec/IO requests over a vsock connection bridged through exactly the uds_path mechanism you just used. Kata Containers does the same with its kata-agent. The control plane (ttRPC/gRPC over vsock) never touches the guest's network, which is why a Kata/firecracker-containerd guest can have no network interface and still be fully managed.

Map the lab to the real system:

Read the integration lab for the full picture: firecracker-containerd.

Implementation Requirements / Deliverables

• A booted microVM with a vsock device (guest_cid=3, a uds_path), and the host bridge socket present on disk.
• A host→guest transfer: a guest AF_VSOCK listener, a host CONNECT <port> handshake through uds_path, and data delivered.
• A guest→host transfer: a host uds_path_<port> listener and a guest AF_VSOCK connect to CID 2, with data delivered.
• A reading log naming: the host-CID constant, the muxer struct, the CONNECT/OK handshake code, and the _<port> suffix logic.
• Trace output from two simultaneous connections showing distinct (local_port, peer_port) tracked by the muxer.
• A short written explanation of how firecracker-containerd or Kata uses this exact mechanism for its in-guest agent.
• All trace instrumentation removed.

Troubleshooting

The host bridge socket /tmp/v.sock never appears

The PUT /vsock must happen before InstanceStart (vsock is a pre-boot device), and the uds_path directory must be writable by the firecracker process. Read the curl response to /vsock; a 400 means a bad field name (rg the swagger). Stale sockets from a previous run can also collide — rm /tmp/v.sock* before booting.

CONNECT <port> gets no OK back

Either nothing is listening on that vsock port inside the guest (start the guest listener first), or the guest kernel lacks the virtio vsock transport (zcat /proc/config.gz | grep VSOCK in the guest; you need CONFIG_VIRTIO_VSOCKETS). Firecracker returns OK <port> only after the guest accepts; a silent hang means no guest acceptor.

socat says "Address family not supported"

Your socat/python3 build doesn't have AF_VSOCK. Use a newer socat, or a small python program that imports socket.AF_VSOCK. nc typically cannot do vsock at all.

guest→host works but host→guest doesn't (or vice versa)

You mixed up the two mechanisms. host→guest: connect to the single uds_path and send CONNECT <guest_port>\n. guest→host: the host listens on uds_path_<host_port> and the guest connects to CID 2. The asymmetry is the most common mistake — re-read Steps 2 and 3.

Connections hang half-open under load

That is the credit/flow-control or shutdown state machine — the hard part of vsock. Trace the VSOCK_OP_CREDIT_* and VSOCK_OP_SHUTDOWN/RST ops and which side is waiting. This is exactly the bug class the muxer's per-connection state exists to get right; it is real contributor territory.

Expected Output

# Host bridge sockets after boot and two listeners:
srwxr-xr-x ... /tmp/v.sock
srwxr-xr-x ... /tmp/v.sock_6000

# host→guest: guest socat receives
hello-from-host

# Firecracker's handshake reply on the host CONNECT:
OK 1024

# Device trace for two concurrent connections:
[Warning] [trace] vsock new conn: local_port=5000 peer_port=1024
[Warning] [trace] vsock new conn: local_port=6000 peer_port=1025
[Warning] [trace] vsock data: dir=h2g len=15 local_port=5000
[Warning] [trace] vsock data: dir=g2h len=16 local_port=6000

Stretch Goals

1. Throughput and a big transfer. Pipe a multi-megabyte file through a vsock connection in each direction and watch the VSOCK_OP_CREDIT_UPDATE ops pace it. Explain vsock flow control vs TCP's.
2. Many connections at once. Open dozens of simultaneous vsock connections and watch the muxer's map grow; find where a connection is removed on close and check for leaks.
3. Run a real agent shape. Write a tiny guest "agent" that listens on a fixed vsock port and answers a request/response protocol, and a host "shim" that drives it through uds_path — a minimal model of the firecracker-containerd control channel.
4. Snapshot a live vsock. Snapshot a microVM with an active vsock connection and restore it. Read how vsock's Persist handles in-flight connection state — or whether connections are dropped across snapshot. Tie to the snapshotting masterclass.
5. Read the spec ops. Map every VSOCK_OP_* you found to the connection state transition it drives, and draw the connection state machine from REQUEST to RST.

Validation / Self-check

Answer without notes; these gate completion.

1. How is a vsock endpoint addressed, what CID is the host always, and what is the valid range for a guest_cid?
2. Why does Firecracker bridge vsock to Unix domain sockets on the host rather than exposing real AF_VSOCK? What does that buy the host process?
3. Describe both connection directions precisely: the CONNECT <port> handshake for host→guest and the uds_path_<port> listener for guest→host.
4. What does the muxer do, what key does it map connections by, and why is its per-connection state harder to get right than block's stateless request loop?
5. How many virtqueues does vsock have, and what is the third (event) queue for?
6. Explain how firecracker-containerd or Kata uses vsock for its in-guest agent, and why such a guest can have no network interface.
7. Name two vsock bug classes that live in the connection state machine and how they manifest.

Cross-references: virtio-vsock, virtqueues, the-event-manager, firecracker-containerd integration lab.

Next: Lab 4 — Build a Virtio Device: implement a complete custom virtio-MMIO device from scratch — trait, activate, queue processing, config space, feature negotiation, registration, and a guest program that exercises it.