The virtio Net Device and the Host TAP

The virtio net device is how a guest gets a network card. To the guest it is a perfectly ordinary virtio NIC (device type ID 1) with two virtqueues; to the host it is two halves bolted together inside the VMM thread: the virtio device logic that talks the virtio wire protocol to the guest driver, and a backend — a host TAP device, i.e. a /dev/net/tun file descriptor opened in TAP mode through which Firecracker reads and writes raw Ethernet frames. The guest's kernel thinks it is DMA-ing packets to a NIC; in reality the VMM is copying frames out of guest memory and write()-ing them to a file descriptor that the host kernel forwards into a bridge or NAT that an operator wired up. Nothing about routing, firewalling, or NAT lives in Firecracker — it owns exactly one hop: guest ↔ TAP fd.

This chapter dissects both halves and traces a packet end to end in both directions: the TX path (guest → host) where the device drains the TX virtqueue and writes to the TAP, and the RX path (host → guest) where a readable TAP fd drives the device to fill the RX virtqueue and inject an interrupt. Along the way you will meet the virtio_net_hdr prefix that rides in front of every frame, the offload/GSO feature negotiation that must match between guest, device, and TAP, the per-device rate limiters, and the one genuinely surprising hook: the MMDS detour, where the TX path quietly steals frames addressed to 169.254.169.254 and feeds them to an in-VMM TCP/IP stack instead of the TAP. After this you will be able to read net/device.rs, draw the symmetric RX/TX diagram from memory, and debug why a guest can ping the host but not the internet.

Note: The net device is two virtqueues plus a file descriptor. Everything else — offloads, rate limiting, MMDS, MTU — is policy layered on top of that core. If you can trace one frame each way through the queues and the TAP fd, you understand the device; the rest is feature bits. Keep the virtqueue mechanics and the MMIO transport in a separate layer of your head — this chapter assumes both.

Where the net device lives

# The whole device. Names by ROLE; confirm exact paths on your branch.
rg -n "struct Net\b|impl VirtioDevice for Net|fn activate" src/vmm/src/devices/virtio/net/
find src/vmm/src/devices/virtio/net -name '*.rs'
# The two queues, by convention RX=0, TX=1.
rg -n "RX_INDEX|TX_INDEX|RX_QUEUE|TX_QUEUE|queues\[" src/vmm/src/devices/virtio/net/
# Device type ID 1 and the feature bits offered.
rg -n "TYPE_NET|fn device_type|avail_features|VIRTIO_NET_F" src/vmm/src/devices/virtio/net/

The device is one struct — call it Net (verify on your branch) — implementing the VirtioDevice trait. It owns: the two Queues, the TAP backend, the negotiated feature bits, the device-config space (which holds the MAC and MTU), the RX and TX rate limiters, two eventfds (one per queue) that the guest "kicks" by writing to, an interrupt object, and — if MMDS is enabled for this interface — an MmdsNetworkStack. By convention queue 0 is RX (device → guest, receive) and queue 1 is TX (guest → device, transmit). Get the directions straight now: the guest drives both, but data flows opposite ways, and the descriptor write-flags reflect that.

Tip: The WRITE flag is the device's perspective. RX buffers are device-writable because the device writes the received frame into them. TX buffers are device-readable because the device reads the outgoing frame out of them. If you ever see RX descriptors without the write flag, the guest driver is broken — and Firecracker should reject the chain, not crash.

The TAP backend

# The TAP abstraction: open /dev/net/tun, TUNSETIFF with IFF_TAP|IFF_NO_PI.
rg -n "struct Tap|fn open_tap|/dev/net/tun|TUNSETIFF|IFF_TAP|IFF_NO_PI" src/vmm/src/devices/virtio/net/
find src/vmm/src/devices/virtio/net -name 'tap.rs'
# Reading and writing raw frames on the fd.
rg -n "fn read_tap|fn write_to_tap|fn read\b|fn write\b" src/vmm/src/devices/virtio/net/tap.rs

A TAP device is a virtual Ethernet interface backed by a file descriptor. Firecracker opens /dev/net/tun, then issues TUNSETIFF with the flags IFF_TAP | IFF_NO_PI:

• IFF_TAP — operate at layer 2: each read()/write() carries one full Ethernet frame (with the 14-byte header), not a layer-3 IP packet (that would be IFF_TUN).
• IFF_NO_PI — no packet info: do not prepend the 4-byte tun_pi struct. Firecracker manages its own framing via virtio_net_hdr (below), so it wants the raw frame and nothing else.

The host side of this TAP appears as a normal NIC named by host_dev_name (e.g. tap0). Firecracker never touches routing — an operator brings tap0 up and attaches it to a bridge or an iptables NAT so frames go somewhere. From Firecracker's side, the TAP is just a bidirectional pipe of Ethernet frames. See the TAP networking masterclass for the host plumbing.

   GUEST                       FIRECRACKER (VMM thread)                    HOST KERNEL
 ┌────────┐   virtqueues   ┌──────────────────────────────┐   /dev/net/tun  ┌─────────┐
 │ virtio │◄══════════════►│  Net device  ◄── TAP fd ──►   │◄═══════════════►│  tap0   │──► bridge
 │  NIC   │  RX(0) TX(1)   │  (Ethernet frames + net_hdr)  │  raw frames     │ (IFF_TAP)│    / NAT
 └────────┘                └──────────────────────────────┘                 └─────────┘

Configuration. Each interface is created via the API before boot:

curl -X PUT --unix-socket "$API" \
  --data '{"iface_id":"net1","guest_mac":"06:00:AC:10:00:02","host_dev_name":"tap0"}' \
  http://localhost/network-interfaces/net1

A PATCH to the same endpoint can retune the rate limiters at runtime; the device binding itself is fixed at PUT time.

Activation: wiring the device to the EventManager

rg -n "fn activate" src/vmm/src/devices/virtio/net/device.rs
rg -n "register|EventSet::IN|as_raw_fd|RateLimiter|timer" src/vmm/src/devices/virtio/net/event_handler.rs

Nothing flows until the guest driver completes the virtio handshake and writes DRIVER_OK (see the MMIO transport). At that point the device's activate runs, and its job is to register every fd it cares about with the EventManager's epoll set:

flowchart LR
  A[guest writes DRIVER_OK] --> B[Net::activate]
  B --> C[register TX queue eventfd]
  B --> D[register RX queue eventfd]
  B --> E[register TAP fd EPOLLIN]
  B --> F[register RX/TX rate-limiter timer fds]
  C & D & E & F --> G[EventManager epoll loop]

After activation, the VMM thread's single epoll loop dispatches four kinds of readiness:

The TX path: guest → host

rg -n "fn process_tx|fn handle_tx|write_to_tap|TX_INDEX|read_from_desc|frame_buf" \
  src/vmm/src/devices/virtio/net/device.rs

The guest driver builds an outgoing Ethernet frame in guest memory, links it into a TX descriptor chain (device-readable), bumps the available ring, and kicks the TX queue eventfd. The VMM thread wakes and runs process_tx, which loops over available TX chains:

sequenceDiagram
  participant G as Guest driver
  participant Q as TX virtqueue
  participant N as Net::process_tx
  participant M as guest memory
  participant T as TAP fd
  G->>Q: fill descriptor chain + bump avail idx
  G->>N: write TX queue eventfd (kick)
  N->>Q: pop() head of available chain
  N->>M: read frame bytes out of the chain
  Note over N: strip/inspect virtio_net_hdr; rate-limiter check
  alt destined for MMDS 169.254.169.254
    N->>N: detour into MmdsNetworkStack (no TAP)
  else normal traffic
    N->>T: write() the frame to the TAP fd
  end
  N->>Q: add_used(head) — return the chain to the guest
  N->>G: inject IRQ if the guest asked to be notified

Step by step, the things worth knowing:

1. Pop the chain. Queue::pop yields a descriptor chain head. The device reads the frame bytes sequentially across the chain via bounds-checked vm-memory accessors — the guest is untrusted, so a bad addr/len must fail the chain, not panic the VMM.
2. The virtio_net_hdr prefix. The first bytes of the chain are not the Ethernet frame — they are a virtio_net_hdr (see below) carrying offload/GSO metadata. The device must account for that header when locating the actual Ethernet payload.
3. Rate-limiter gate. Before the write, the TX rate limiter is consulted. If a token bucket (ops/s or bandwidth) is dry, the frame is not dropped — processing pauses and resumes when the timer fd fires. See the rate limiter.
4. MMDS detour (the surprising one). If the frame's destination is the MMDS link-local address, it is not written to the TAP; it is handed to the in-VMM stack instead (next section).
5. Write to TAP. Otherwise write_to_tap issues a write() of the raw frame on the TAP fd. The host kernel takes it from there.
6. Return the buffer. add_used puts the head index on the used ring so the guest can recycle the descriptor; an IRQ is injected if EVENT_IDX suppression hasn't told the device to stay quiet.

Warning: A frame larger than what one TAP write() can take, or a malformed virtio_net_hdr, must be handled as a per-frame error — drop that frame and move on. A single bad guest frame must never stall the queue or take down the VMM thread that is shared with every other device.

The RX path: host → guest

rg -n "fn process_rx|read_tap|rx_deferred|RX_INDEX|rx_frame_buf|fn try_read" \
  src/vmm/src/devices/virtio/net/device.rs

RX is the mirror image, driven by the host rather than the guest. When a frame arrives on tap0, the host kernel makes the TAP fd readable; epoll wakes the VMM thread, which runs process_rx:

sequenceDiagram
  participant T as TAP fd
  participant N as Net::process_rx
  participant Q as RX virtqueue
  participant M as guest memory
  participant G as Guest driver
  T-->>N: TAP fd readable (EPOLLIN)
  N->>T: read() one frame into an internal buffer
  Note over N: prepend virtio_net_hdr; rate-limiter check
  N->>Q: pop() a free RX (device-writable) chain
  alt a free RX chain exists
    N->>M: write header + frame into the chain
    N->>Q: add_used(head, bytes_written)
    N->>G: inject IRQ
  else no RX buffers available
    N->>N: set rx_deferred — wait for guest to post buffers
  end

The subtlety is backpressure. The host can deliver frames faster than the guest posts empty RX buffers. When process_rx reads a frame but finds no free RX descriptor chain, it cannot drop the frame on the floor casually and it cannot block — so it stashes the frame and sets a deferred flag (rx_deferred or similar). The device then stops reading the TAP and waits. When the guest later posts fresh RX buffers and kicks the RX queue eventfd, that event resumes RX, drains the held frame, and continues reading the TAP. This is why the RX queue eventfd is a registered event source: it is the "buffers available again" signal, not a request to transmit.

Each RX frame written into guest memory is prefixed with a virtio_net_hdr, then the Ethernet frame, then add_used records the total bytes written so the guest knows how much of its buffer is valid.

The virtio_net_hdr and offloads

rg -n "virtio_net_hdr|VIRTIO_NET_F_CSUM|TSO|UFO|set_offload|TUNSETOFFLOAD|num_buffers" \
  src/vmm/src/devices/virtio/net/
rg -n "mtu|VIRTIO_NET_F_MTU|VIRTIO_NET_F_MAC|VIRTIO_F_VERSION_1|VIRTIO_RING_F_EVENT_IDX" \
  src/vmm/src/devices/virtio/net/

Every frame on both queues is preceded by a fixed header that carries offload metadata:

struct virtio_net_hdr {
    u8  flags;        // e.g. NEEDS_CSUM: checksum not yet computed
    u8  gso_type;     // NONE / TCPv4 / TCPv6 / UDP — segmentation type
    u16 hdr_len;      // length of the protocol headers
    u16 gso_size;     // segment size for GSO/TSO
    u16 csum_start;   // where the checksum region begins
    u16 csum_offset;  // where in the region to store the checksum
    u16 num_buffers;  // (mergeable-rx-buffers) frames spanning multiple buffers
};

This header is the contract that lets a NIC and a stack skip work. With TSO (TCP Segmentation Offload) the guest can hand the device one large TCP "frame" (tens of KB) plus a gso_size, and the segmentation into MTU-sized packets happens downstream. With checksum offload (VIRTIO_NET_F_CSUM), the guest skips computing the TCP/UDP checksum and the device/host fills it. These are huge throughput wins — but they only work if all three parties agree.

Warning: If the device offers HOST_TSO4 but the TAP was not configured with the matching TUNSETOFFLOAD, the host kernel will receive an oversized "frame" it cannot segment, and you get silent drops or EINVAL on write(). Offload feature bits and TUNSETOFFLOAD must be kept in lockstep — grep set_offload to see how Firecracker derives one from the other.

Other feature bits in play: VIRTIO_NET_F_MAC (MAC supplied in config space), VIRTIO_NET_F_MTU (advertise an MTU in config space; configurable per interface), plus the transport bits VIRTIO_F_VERSION_1 (bit 32) and VIRTIO_RING_F_EVENT_IDX (bit 29) for interrupt suppression.

The MMDS detour

rg -n "mmds|MmdsNetworkStack|169\.254|detour_frame|ns\.detour|mmds_ns" \
  src/vmm/src/devices/virtio/net/

The guest reaches the metadata service at 169.254.169.254 — but there is no real network there. Firecracker intercepts it. On the TX path, before the frame is written to the TAP, the device checks whether it is destined for the MMDS link-local address. If so, it detours the frame into an in-VMM TCP/IP stack — MmdsNetworkStack, the "dumbo" stack — which answers HTTP requests for metadata and synthesizes reply frames that get injected back on the RX path as if they came from the network. The TAP never sees MMDS traffic.

  guest frame ──► process_tx ──┬── dst == 169.254.169.254 ? ──► MmdsNetworkStack (dumbo TCP/IP)
                               │                                      │ synthesized reply
                               └── else ──► write_to_tap              ▼
                                                              injected on RX queue ──► guest

This is why MMDS "just works" without any host networking, and why a misconfigured TAP can break internet access while metadata still resolves: the two paths diverge inside process_tx. See the MMDS metadata service for the dumbo stack internals.

Reading exercise

Run these against a Firecracker checkout (substitute your branch's paths if the rg output is empty):

# 1. Map the net device's files and find the two queue indices.
find src/vmm/src/devices/virtio/net -name '*.rs'
rg -n "RX_INDEX|TX_INDEX|RX_QUEUE|TX_QUEUE" src/vmm/src/devices/virtio/net/

# 2. Read both processing functions back to back.
rg -n "fn process_tx|fn process_rx|fn read_tap|fn write_to_tap" src/vmm/src/devices/virtio/net/device.rs

# 3. Find where the TAP fd is opened and which TUNSETIFF flags are used.
rg -n "/dev/net/tun|TUNSETIFF|IFF_TAP|IFF_NO_PI|TUNSETOFFLOAD" src/vmm/src/devices/virtio/net/tap.rs

# 4. Find the deferred-RX backpressure flag and where it is set and cleared.
rg -n "rx_deferred|deferred" src/vmm/src/devices/virtio/net/

# 5. Find the MMDS detour decision in the TX path.
rg -n "mmds|169\.254|MmdsNetworkStack|detour" src/vmm/src/devices/virtio/net/

# 6. List the offered feature bits.
rg -n "VIRTIO_NET_F|avail_features|VIRTIO_F_VERSION_1|VIRTIO_RING_F_EVENT_IDX" src/vmm/src/devices/virtio/net/

Now answer:

1. Which queue index is RX and which is TX on your branch, and which queue holds device-writable descriptors? Tie your answer to the VIRTQ_DESC_F_WRITE flag from virtqueues.
2. In process_tx, at exactly which point relative to the TAP write() is the MMDS destination checked — before or after? Why must it be where it is?
3. What does the device do when process_rx reads a frame but no free RX buffer exists? Name the flag and explain what event later resumes RX.
4. Where does the virtio_net_hdr come from on the RX path, and who consumes it — the host or the guest?
5. Why does an offload feature bit offered by the device require a matching TUNSETOFFLOAD on the TAP? What breaks if they disagree?
6. Trace what the four event sources registered in activate are, and which handler each one drives.

Common bugs and symptoms

Validation: prove you understand this

1. The net device is virtio device type ID 1 and exposes two virtqueues. State which index is RX and which is TX, the data-flow direction of each, and which one carries device-writable descriptors.
2. Trace a single outbound packet from the moment the guest kicks the TX eventfd to the moment the host kernel has it: name every step, including where guest memory is read and where the TAP write() happens.
3. Trace a single inbound packet from "TAP fd becomes readable" to "guest receives an interrupt," and explain the deferred-RX backpressure case where no RX buffer is available.
4. What is the virtio_net_hdr, where does it sit relative to the Ethernet frame on each queue, and name two offload features it enables and why they need matching TUNSETOFFLOAD on the TAP.
5. Explain the MMDS detour: where in the TX path the interception happens, what frames trigger it, where they go instead of the TAP, and how the reply reaches the guest.
6. List the four fd-like event sources the device registers in activate, and for each say what readiness on it means and which handler runs.

Next: The virtio Vsock Device — same virtqueue machinery, no TAP: a host-guest socket bridge instead of an Ethernet pipe, and a very different backend.