The virtio Vsock Device

Vsock is a socket family — AF_VSOCK — that lets a process on the host and a process inside the guest talk over a stream connection without any network stack, IP address, or NIC. There are no Ethernet frames, no routing, no firewall: just an endpoint identified by a pair, (CID, port). CID is a Context ID, a small integer naming a VM (or the host). The host is always CID 2 (VMADDR_CID_HOST); the guest's CID is whatever you configured (guest_cid). Inside the guest, vsock looks like an ordinary socket family. On the host side, though, Firecracker is the vsock device — it does not open a kernel AF_VSOCK socket. Instead it bridges the guest's vsock traffic to host Unix domain sockets through an internal multiplexer (the "muxer").

This chapter dissects that device: the three virtqueues (RX, TX, EVENT) that carry vsock packets, the (CID, port) addressing and the muxer that maps connections to host Unix sockets, the per-connection state machine driven by the op field in each packet header, and the credit-based flow control that keeps a fast sender from drowning a slow receiver. After this chapter you will be able to: configure a vsock device; open a connection in both directions (guest→host and host→guest) from the shell; trace a byte from a guest write() through a TX descriptor chain to a host Unix-socket write(); and explain why the agent control channel in firecracker-containerd and Kata rides on vsock.

Note: Keep three layers separate in your head. The transport (virtio-MMIO) tells the guest the device exists and wires up kicks and interrupts. The queues (virtqueues) are the split-ring data structures whose descriptor chains carry the bytes. This chapter is the device-type layer on top: what a vsock packet means and what the muxer does with it. Vsock is virtio device type ID 19.

Where the device lives

# The vsock device module — confirm the directory exists on your branch.
rg -n "TYPE_VSOCK|VIRTIO_VSOCK|device type|= 19" src/vmm/src/devices/virtio/vsock/
find src/vmm/src/devices/virtio/vsock -name '*.rs' | sort
# The top-level device struct and its VirtioDevice impl.
rg -n "struct Vsock|impl.*VirtioDevice for Vsock|fn device_type" src/vmm/src/devices/virtio/vsock/

The vsock device is one virtio device per microVM — you cannot configure two. Its top-level type (call it Vsock<B>, generic over a backend B; verify the name on your branch) implements the same VirtioDevice trait as block and net: device_type() returns 19, queues() returns three, activate() wires it live. The interesting machinery — the muxer, the per-connection state, the host Unix-socket backend — lives behind the backend type.

# The host-Unix-socket backend and the multiplexer.
rg -n "VsockUnixBackend|VsockMuxer|struct .*Muxer|uds_path|fn new" src/vmm/src/devices/virtio/vsock/

The three virtqueues

A vsock device exposes three virtqueues, not the two you might expect. Find the index constants:

rg -n "RXQ|TXQ|EVQ|RX_INDEX|TX_INDEX|EVENT|NUM_QUEUES|QUEUE_SIZES" src/vmm/src/devices/virtio/vsock/

The direction convention is from the driver's point of view, the standard virtio convention: "RX" is what the guest receives. So the device's job on the RX queue is to produce packets, and on the TX queue to consume them. Each descriptor chain in TX or RX carries exactly one vsock packet: a fixed header followed by an optional data payload. The mechanics of how a descriptor chain is walked — DescriptorChain, reading the header out of guest memory, the available/used rings — are entirely in virtqueues; the vsock code reuses that machinery and only adds meaning to the bytes.

   guest                                   Firecracker (VMM thread)                 host
 ┌────────┐    TX queue (driver→device)   ┌───────────────────────────┐
 │ guest  │ ───[hdr|data]──► descriptor ──► │  parse hdr → muxer route  │ ──► ┌──────────────┐
 │ process│                  chain          │  (dst_port → conn)        │     │ Unix socket  │
 │  write │                                 │  write payload to host fd │     │  *_<port>    │
 └────────┘                                 │                           │     └──────────────┘
     ▲       RX queue (device→driver)       │  read host fd → build hdr │            │
     └──────[hdr|data]◄── descriptor ◄───── │  fill guest RX buffer     │ ◄──────────┘
                          chain             └───────────────────────────┘
                                                EVENT queue: reset notifications

# How the device picks apart a TX chain and assembles an RX chain.
rg -n "fn process_rx|fn process_tx|RxOps|TxBuf|VsockPacket|fn read_from|fn write_to" \
  src/vmm/src/devices/virtio/vsock/

The vsock packet header

Every packet — TX or RX — begins with a fixed virtio_vsock_hdr carried in the first part of the descriptor chain. This is the contract; learn its fields:

rg -n "virtio_vsock_hdr|struct VsockPacket|src_cid|dst_cid|src_port|dst_port|buf_alloc|fwd_cnt|fn op|fn type_" \
  src/vmm/src/devices/virtio/vsock/

The (src_cid, src_port, dst_cid, dst_port) four-tuple uniquely identifies a connection. The muxer keys its connection table on the (local_port, peer_port) pair (the CIDs are fixed for a given device, so they fall out).

The op field and the connection state machine

The op field is the verb. The muxer reads it off each TX packet and drives a per-connection state machine; it writes op values into RX packets it originates.

rg -n "VSOCK_OP_REQUEST|VSOCK_OP_RESPONSE|VSOCK_OP_RW|VSOCK_OP_CREDIT_UPDATE|VSOCK_OP_CREDIT_REQUEST|VSOCK_OP_SHUTDOWN|VSOCK_OP_RST|ConnState|enum .*State" \
  src/vmm/src/devices/virtio/vsock/

stateDiagram-v2
    [*] --> Closed
    Closed --> LocalInit: guest TX VSOCK_OP_REQUEST (guest→host)
    Closed --> PeerInit: host connect → device RX VSOCK_OP_REQUEST (host→guest)
    LocalInit --> Established: RX VSOCK_OP_RESPONSE from host backend
    PeerInit --> Established: guest TX VSOCK_OP_RESPONSE
    Established --> Established: VSOCK_OP_RW (data) / CREDIT_UPDATE
    Established --> Closing: VSOCK_OP_SHUTDOWN (one half)
    Closing --> Closed: both halves shut, or VSOCK_OP_RST
    Established --> Closed: VSOCK_OP_RST
    LocalInit --> Closed: VSOCK_OP_RST (refused)

Note: A RST can arrive in almost any state — it is how a refused or broken connection is reported. When you find a guest that connects and is immediately reset, you are looking for where the muxer emits RST: usually "no host process is listening on the expected Unix socket."

# Where the muxer emits RST and how it maps a packet to a connection.
rg -n "VSOCK_OP_RST|send_bytes|recv_pkt|send_pkt|conn_map|insert|remove|fn process" \
  src/vmm/src/devices/virtio/vsock/

The muxer: bridging to host Unix sockets

This is the part with no analogue in block or net. The muxer (VsockMuxer) holds a table mapping each live connection (local_port, peer_port) to a host Unix-domain-socket file descriptor, and shuttles bytes between virtqueue packets and those fds. It is registered in the EventManager's epoll set so that both a guest kick (queue eventfd) and host socket readiness (a Unix-socket fd becoming readable) wake the device.

rg -n "VsockMuxer|HashMap|epoll|EventSet|register|fn handle_event|LocalListener|fn notify" \
  src/vmm/src/devices/virtio/vsock/

The two connection directions use the host Unix socket differently. This is the single most important operational fact about Firecracker vsock — internalize it:

Guest → host. A guest process connects to (CID 2, port P). The device receives a VSOCK_OP_REQUEST on TX. The muxer then tries to connect, on the host, to a Unix socket whose path is the configured uds_path with _<P> appended — "<uds_path>_<P>". Some host process must already be listening there. If the connect succeeds, the muxer sends VSOCK_OP_RESPONSE back on RX; if nothing is listening, it sends VSOCK_OP_RST.

Host → guest. A host process connects to the bare uds_path Unix socket, then writes a text command "CONNECT <port>\n". The muxer parses it, opens a vsock connection to the guest by emitting a VSOCK_OP_REQUEST on the RX queue toward (guest_cid, port), and on success replies on the Unix socket with "OK <assigned_port>\n". After that line, the Unix socket is a transparent byte pipe to the guest process.

rg -n "CONNECT|fn parse|OK |b\"OK|host_sock|uds_path|UnixListener|UnixStream" \
  src/vmm/src/devices/virtio/vsock/

sequenceDiagram
    participant H as host process
    participant U as Unix socket(s)
    participant M as VsockMuxer
    participant Q as RX/TX queues
    participant G as guest process
    Note over H,G: Guest → host
    G->>Q: TX VSOCK_OP_REQUEST (dst_cid=2, dst_port=P)
    M->>U: connect to "<uds_path>_P"
    U-->>M: connected (a process was listening)
    M->>Q: RX VSOCK_OP_RESPONSE
    G->>Q: TX VSOCK_OP_RW (data)
    M->>U: write(data) to host fd
    Note over H,G: Host → guest
    H->>U: connect "<uds_path>", send "CONNECT P\n"
    M->>Q: RX VSOCK_OP_REQUEST (dst_cid=guest_cid, dst_port=P)
    G->>Q: TX VSOCK_OP_RESPONSE
    M->>U: reply "OK <port>\n"
    H->>U: write(data); M->>Q: RX VSOCK_OP_RW (data)

Configuration and activation

The vsock device is created via the API before boot — it is pre-boot only, and there is exactly one per microVM.

# The API handler and config struct.
rg -n "VsockDeviceConfig|guest_cid|uds_path|vsock_id|PUT.*vsock|fn set_vsock" \
  src/vmm/src/ src/firecracker/src/

# Create the device: guest CID 3, bridged to /tmp/v.sock.
curl --unix-socket /tmp/fc.sock -i -X PUT 'http://localhost/vsock' \
  -H 'Content-Type: application/json' \
  -d '{ "vsock_id": "vsock0", "guest_cid": 3, "uds_path": "/tmp/v.sock" }'

guest_cid must be ≥ 3 (0, 1, 2 are reserved). When the guest driver finishes the virtio handshake and writes DRIVER_OK, the transport calls the device's activate:

rg -n "fn activate|register.*queue|EventManager|epoll|ioeventfd|self\.queues" \
  src/vmm/src/devices/virtio/vsock/

At activate, the device registers with the EventManager: the RX, TX, and EVENT queue eventfds (each fired by a guest QueueNotify via KVM_IOEVENTFD — see virtio-MMIO) and the muxer's epoll fd, which itself watches the host uds_path listener plus every per-connection Unix-socket fd. From that point a single VMM-thread epoll wakeup can mean "guest kicked a queue" or "a host socket has bytes."

Real-world use: the agent control channel

Vsock exists in Firecracker for one dominant reason: a control channel between an in-guest agent and a host-side shim, without a network. In firecracker-containerd and Kata Containers, the host runtime shim and the in-VM agent speak a gRPC/ttrpc protocol over vsock — the shim connects to the guest agent's well-known port to start containers, stream stdio, and report exits. No tap device, no bridge, no IP to assign or firewall. See firecracker-containerd integration for the real consumer.

Tip: When debugging a firecracker-containerd VM that "boots but the shim can't reach the agent," the vsock muxer is the prime suspect. Check the uds_path, confirm the agent is listening on its port inside the guest, and confirm the host side is connecting to <uds_path> (host→guest) or listening on <uds_path>_<port> (guest→host) with the right port suffix.

The device also implements Persist so its connection-bearing state participates in snapshot/restore; after restore the guest is told via the EVENT queue that the transport reset.

rg -n "impl Persist|fn save|fn restore|VsockState|TYPE_VSOCK|Persist for Vsock" \
  src/vmm/src/devices/virtio/vsock/

Reading exercise

# 1. Confirm the device type ID and the three queues.
rg -n "= 19|TYPE_VSOCK|NUM_QUEUES|RXQ|TXQ|EVQ|QUEUE_SIZES" src/vmm/src/devices/virtio/vsock/

# 2. The packet header and its fields.
rg -n "virtio_vsock_hdr|struct VsockPacket|buf_alloc|fwd_cnt|fn op|fn len_" src/vmm/src/devices/virtio/vsock/

# 3. Every opcode and the state machine that consumes it.
rg -n "VSOCK_OP_|ConnState|enum .*State|fn state" src/vmm/src/devices/virtio/vsock/

# 4. The muxer, the connection table, and the CONNECT text protocol.
rg -n "VsockMuxer|conn_map|HashMap|CONNECT|uds_path|UnixListener|UnixStream" src/vmm/src/devices/virtio/vsock/

# 5. Credit-based flow control.
rg -n "buf_alloc|fwd_cnt|credit|peer_avail|tx_cnt|rx_cnt" src/vmm/src/devices/virtio/vsock/

# 6. activate(): queue eventfds + muxer epoll fd into the EventManager.
rg -n "fn activate|register|EventManager|epoll" src/vmm/src/devices/virtio/vsock/

# 7. On a running microVM (guest CID 3, uds_path /tmp/v.sock):
#    GUEST→HOST: on the host, listen on the suffixed socket:
#       socat - UNIX-LISTEN:/tmp/v.sock_5234
#    then in the guest connect to (CID 2, port 5234) and type.
#    HOST→GUEST: in the guest, listen on port 5234; on the host:
#       socat - UNIX-CONNECT:/tmp/v.sock   then send:  CONNECT 5234<newline>

Answer:

1. Why three virtqueues and not two? What does the EVENT queue carry, and from whose point of view are "RX" and "TX" named?
2. Walk a guest→host connection from the guest's connect((2, 5234)) to host bytes arriving: which queue carries the REQUEST, what Unix-socket path does the muxer dial, and what op confirms it?
3. Do the same for host→guest: what does the host process write to <uds_path>, and what does the muxer emit on which queue?
4. Pick three VSOCK_OP_* values and place them on the state machine. Which op can appear in (almost) any state, and what real failure does it usually signal?
5. What do buf_alloc and fwd_cnt mean, and which op refreshes them?
6. What gets registered with the EventManager at activate, and why must host socket fds be in the same epoll set as the guest queue eventfds?

Common bugs and symptoms

Validation: prove you understand this

1. Draw the vsock data path end to end for guest→host: guest write() → which queue → which descriptor-chain contents → which muxer step → which host syscall. Name the op on each leg.
2. Explain (CID, port) addressing. What is the host CID, what constrains guest_cid, and why does Firecracker not open a host AF_VSOCK socket?
3. List the three virtqueues, their directions (from the driver's view), and what each carries. Why is the EVENT queue necessary?
4. Take REQUEST, RESPONSE, RW, SHUTDOWN, and RST and walk the connection state machine, saying which side emits each and on which queue it lands.
5. Explain credit-based flow control using buf_alloc and fwd_cnt: what does the sender compute, and which op closes the loop when the window opens?
6. A firecracker-containerd VM boots but the shim cannot reach the in-VM agent. Give the ordered list of vsock-specific checks — config, direction, socket path/suffix, port, listener — you would run to localize the break.

Next: The virtio Balloon Device — reclaiming guest memory back to the host on demand: the inflate/deflate queues, the page-hint protocol, and why a balloon is a cooperative, not coercive, memory control.