Lab 2: Virtio-Net and TAP

Background

This is a trace-it-and-build-infra lab. Firecracker has no virtual switch and no NAT of its own: a guest network interface is a virtio-net device in the VMM bound to a host TAP device (a kernel /dev/net/tun endpoint that looks like an Ethernet NIC to the host). Frames the guest transmits come out of the TAP fd on the host; frames you write to the TAP go into the guest. Routing, bridging, NAT, and DNS are the host's job, set up with ordinary Linux networking. You will build all of that, attach a virtio-net interface, SSH into the guest, and then trace an RX frame and a TX frame through the TAP fd and the two virtqueues — watching the same frames on the wire with tcpdump. Finally you will apply a token-bucket rate limiter and watch it throttle.

Net is the natural next device after block: it adds a second virtqueue (RX and TX are separate queues) and an external file descriptor (the TAP fd, also registered with the EventManager — so the device wakes both on a guest kick and on host frames arriving). If you understood block's one-queue, one-fd shape, net is that shape doubled with a direction to keep straight.

Why This Lab Matters for Contributors

Networking issues — dropped frames, MTU mismatches, rate-limiter accounting, TAP error handling, RX-buffer exhaustion — are debugged exactly here, by correlating a guest-side capture, a TAP-side tcpdump, and the device's queue handlers. The issue-roadmap virtio stage and the networking masterclass live in this territory.
The TAP-plus-routing setup is the same plumbing firecracker-containerd, the Go SDK, and every production deployment use; understanding it by hand demystifies all of them.
It makes the two-queue, two-fd structure of a real device concrete, and connects virtio-net-and-tap, rate-limiting-token-bucket, and the-event-manager.

Prerequisites

Lab 1 complete — you can trace a virtio device end to end.
Root on a Linux host with iproute2 (ip), iptables (or nftables), tcpdump, and IP-forwarding allowed. A real (non-containerized) host is easiest.
A guest rootfs with an SSH server and a known login (most Firecracker example rootfs images ship openssh-server; if yours doesn't, install it or use the serial console for the parts that don't need SSH).
Verify the net device, its two queues, and the TAP plumbing exist on your branch:

# All must return hits. If empty, the layout moved — find by role.
rg -l "impl VirtioDevice for"          src/vmm/src/devices/virtio/net/
rg -n "RX_INDEX|TX_INDEX|rx_queue|tx_queue|NUM_QUEUES|fn process_rx|fn process_tx" \
   src/vmm/src/devices/virtio/net/
rg -n "Tap|/dev/net/tun|tap_fd|read_tap|write|IFF_TAP|TUNSETIFF" \
   src/vmm/src/devices/virtio/net/

The Path You Are Tracing

flowchart LR
    subgraph host[HOST]
      Inet[(Internet / LAN)] <--> NAT["iptables MASQUERADE + ip_forward"]
      NAT <--> TAP["tap0 (host TAP, /dev/net/tun fd)"]
    end
    subgraph fc[FIRECRACKER VMM thread]
      TAP <-->|TAP fd, EventManager| NET["virtio-net device"]
      NET <-->|RX queue / TX queue| GM["guest memory virtqueues"]
    end
    subgraph guest[GUEST]
      GM <-->|virtio_net driver| ETH["eth0 in the microVM"]
    end

TX (guest → host): guest writes a frame into the TX virtqueue, kicks → device pops the chain, write()s the frame to the TAP fd → kernel routes/NATs it out.
RX (host → guest): a frame arrives on the TAP → the TAP fd becomes readable → EventManager wakes the device → device read()s the frame and copies it into a buffer the guest pre-posted on the RX virtqueue → add_used, raise IRQ → guest driver receives it.

Step-by-Step Tasks

Step 1: Create and route a host TAP

Create a TAP device, give it a host-side IP, enable forwarding, and NAT the guest out through your uplink. Pick a subnet that doesn't collide with your LAN (here 172.16.0.0/24).

# Pick your uplink (the interface with the default route):
UPLINK=$(ip route | awk '/default/ {print $5; exit}'); echo "UPLINK=$UPLINK"

TAP=tap0
HOST_IP=172.16.0.1
GUEST_IP=172.16.0.2
MASK=24

# Create the TAP, owned so the firecracker process can open it (run FC as the same user, or as root):
sudo ip tuntap add dev "$TAP" mode tap
sudo ip addr add "$HOST_IP/$MASK" dev "$TAP"
sudo ip link set "$TAP" up

# Enable forwarding and NAT the guest subnet out the uplink:
sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -t nat -A POSTROUTING -o "$UPLINK" -j MASQUERADE
sudo iptables -A FORWARD -i "$TAP" -o "$UPLINK" -j ACCEPT
sudo iptables -A FORWARD -i "$UPLINK" -o "$TAP" -m state --state RELATED,ESTABLISHED -j ACCEPT

ip addr show "$TAP"

Note: A TAP is an L2 device — it carries Ethernet frames, which is exactly what virtio-net moves. (A TUN is L3/packets; virtio-net wants TAP.) Confirm Firecracker opens it as a TAP:
rg -n "IFF_TAP|IFF_NO_PI|TUNSETIFF|/dev/net/tun" src/vmm/src/devices/virtio/net/

Step 2: Attach a virtio-net interface and boot

Boot a microVM as before, and add a network interface bound to tap0. The guest_mac is optional but makes the guest NIC deterministic.

ARCH=$(uname -m)
BIN=build/cargo_target/${ARCH}-unknown-linux-musl/debug/firecracker
API=/tmp/fc-net.sock
LOG=/tmp/fc-net.log
rm -f "$API"; : > "$LOG"
sudo "$BIN" --api-sock "$API" &

curl -X PUT --unix-socket "$API" --data \
  '{"log_path":"'"$LOG"'","level":"Warning","show_level":true,"show_log_origin":true}' \
  http://localhost/logger

curl -X PUT --unix-socket "$API" --data \
  '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1"}' \
  http://localhost/boot-source

curl -X PUT --unix-socket "$API" --data \
  '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
  http://localhost/drives/rootfs

# The network interface, bound to the host TAP:
curl -X PUT --unix-socket "$API" --data \
  '{"iface_id":"eth0","guest_mac":"06:00:AC:10:00:02","host_dev_name":"tap0"}' \
  http://localhost/network-interfaces/eth0

curl -X PUT --unix-socket "$API" --data \
  '{"vcpu_count":2,"mem_size_mib":1024}' http://localhost/machine-config

curl -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' http://localhost/actions

Step 3: Bring up the guest side and get connectivity

The host configured routing; now give the guest an IP, a route, and DNS on the serial console.

# Guest serial console:
ip addr add 172.16.0.2/24 dev eth0
ip link set eth0 up
ip route add default via 172.16.0.1
echo 'nameserver 1.1.1.1' > /etc/resolv.conf

# Prove it:
ping -c 3 172.16.0.1      # the host TAP
ping -c 3 1.1.1.1         # out through NAT

If ping 1.1.1.1 works but DNS does not, that is a resolv.conf problem, not a Firecracker problem.

Step 4: SSH into the guest

With routing up, SSH from the host straight to the guest IP. (Ensure the guest's sshd is running and permits your login — for a lab, password or an authorized key baked into the rootfs.)

# Host:
ssh root@172.16.0.2
#   If sshd isn't running in the guest, start it on the serial console: `systemctl start ssh`
#   or `/usr/sbin/sshd`. For a key, append your pubkey to the guest /root/.ssh/authorized_keys.

You now have a real shell into the microVM over its virtio-net path. Every keystroke and every byte of scp traffic crosses the TAP fd and the virtqueues you are about to trace.

Step 5: Watch the frames on the TAP with tcpdump

Before instrumenting code, observe the path non-invasively. tcpdump on tap0 sees exactly the frames crossing between host and guest.

# Host, in its own terminal:
sudo tcpdump -ni tap0 -e

# From the host (another terminal), generate traffic to the guest:
ping -c 3 172.16.0.2
ssh root@172.16.0.2 'cat /etc/hostname'

You will see ARP, then ICMP echo/reply, then the SSH TCP handshake and data — each as an Ethernet frame with the guest's MAC (06:00:ac:10:00:02) on one side and the TAP's MAC on the other. A frame to the guest is an RX from the device's perspective; a frame from the guest is a TX. Keep this running; you will correlate it with the device trace next.

Step 6: Find and instrument the RX and TX handlers

Net has two queue handlers (RX and TX) plus a TAP-fd handler. Find them by role.

# The RX and TX processing functions and the queue indices.
rg -n "fn process_rx|fn process_tx|fn read_from_tap|fn write_to_tap|RX_INDEX|TX_INDEX|RX_QUEUE|TX_QUEUE" \
   src/vmm/src/devices/virtio/net/
# The EventManager subscription: the device subscribes BOTH the queue eventfds AND the TAP fd.
rg -n "fn activate|register|Subscriber|EventSet|tap.*as_raw_fd|queue_evt" \
   src/vmm/src/devices/virtio/net/

Add three trace points (adjust names to what rg shows):

#![allow(unused)]
fn main() {
// 1. TX: guest kicked the TX queue — we are about to drain frames to the TAP.
log::warn!("[trace] net TX kick");

// 2. After writing a frame to the TAP fd (the guest→host hop):
log::warn!("[trace] net TX→tap: {} bytes", frame_len);

// 3. RX: a frame arrived on the TAP fd; we are copying it into a guest RX buffer.
log::warn!("[trace] net RX←tap: {} bytes", frame_len);
}

Warning: virtio-net prepends a virtio_net_hdr to each frame; frame_len may or may not include it depending on where you log. Read how the header is handled (rg -n "virtio_net_hdr|vnet_hdr|hdr_len|num_buffers" src/vmm/src/devices/virtio/net/) so your byte counts line up with tcpdump's.

Rebuild (tools/devtool build) and re-boot (Steps 2–3).

Step 7: Trace a TX frame (guest → host)

With the device trace and the TAP tcpdump both running, generate guest-originated traffic.

# Guest (over SSH or serial): send something outbound.
ping -c 2 1.1.1.1

The sequence you should see, in order:

# In /tmp/fc-net.log (the device):
[Warning] [trace] net TX kick
[Warning] [trace] net TX→tap: 98 bytes      # an ICMP echo request frame

# In tcpdump -ni tap0 (the wire), the same frame leaving:
06:00:ac:10:00:02 > ..., ethertype IPv4, 172.16.0.2 > 1.1.1.1: ICMP echo request

That is the full TX path: guest driver built a chain on the TX queue, kicked, the handler popped it, read the frame out of guest memory (the data descriptor is device-readable on TX), and write()-d it to the TAP fd — where tcpdump caught it. Trace it through the code: TX descriptors are device-readable; the device does not set WRITE on them.

Step 8: Trace an RX frame (host → guest)

Now the reverse. Generate traffic into the guest and watch the RX path.

# Host: ping the guest, or just let the SSH session's return traffic flow.
ping -c 2 172.16.0.2

# tcpdump shows the frame arriving on tap0 toward the guest:
... > 06:00:ac:10:00:02, ethertype IPv4, 172.16.0.1 > 172.16.0.2: ICMP echo request

# The device log shows it being delivered:
[Warning] [trace] net RX←tap: 98 bytes

The RX path is woken differently from TX: there is no guest kick. A frame arriving makes the TAP fd readable, the EventManager epoll loop wakes, and the device read()s the frame and copies it into a buffer the guest pre-posted on the RX queue (RX descriptors are device-writable — they carry the WRITE flag). Confirm in code that the TAP fd is registered as a subscriber alongside the queue eventfds:

rg -n "activate" -A 40 src/vmm/src/devices/virtio/net/ | rg -n "tap|register|EventSet::IN|as_raw_fd|queue_evt"

Note: If the guest has not posted enough RX buffers (RX exhaustion), incoming frames have nowhere to land and the device must drop or backpressure. This is a real bug class — trace what your branch does when the RX queue is empty (rg -n "rx.*empty|no.*buffer|RX_RATE|deferred|rx_deferred" src/vmm/src/devices/virtio/net/).

Step 9: Apply a rate limiter and watch it throttle

Firecracker rate-limits net and block with a token bucket — two buckets per direction (operations and bandwidth). PATCH the interface with a bandwidth limit and measure the cap.

# Find the rate-limiter shape the API accepts (size = bucket capacity in bytes, refill_time in ms):
rg -n "rate_limiter|bandwidth|ops|one_time_burst|refill_time|size" \
   src/firecracker/swagger/firecracker.yaml

# Cap RX+TX bandwidth to ~1 MB/s: a 1,000,000-byte bucket refilled every 1000 ms.
curl -X PATCH --unix-socket "$API" --data '{
  "iface_id": "eth0",
  "rx_rate_limiter": { "bandwidth": { "size": 1000000, "one_time_burst": 0, "refill_time": 1000 } },
  "tx_rate_limiter": { "bandwidth": { "size": 1000000, "one_time_burst": 0, "refill_time": 1000 } }
}' http://localhost/network-interfaces/eth0

Measure before and after with iperf3 (run a server on the host, client in the guest) or a large scp:

# Host: iperf3 -s
# Guest: iperf3 -c 172.16.0.1 -t 10
#   Expect throughput to settle near the bucket's refill rate (~1 MB/s ≈ 8 Mbit/s) once the
#   one_time_burst is spent.

Read how the limiter gates the queue handler — when the bucket is empty, the device stops processing and re-arms a timer fd to resume when tokens refill:

rg -n "TokenBucket|consume|budget|refill|Timer|deferred|rate_limiter" \
   src/vmm/src/rate_limiter/ src/vmm/src/devices/virtio/net/

This is the rate-limiting-token-bucket deep dive made visible, and the same mechanism Lab 1's stretch goal applied to block.

Implementation Requirements / Deliverables

A working host TAP (tap0) with an IP, IP-forwarding, and NAT — and the exact ip/iptables commands you used.
A microVM with a virtio-net interface bound to tap0, reachable by ping and SSH from the host.
tcpdump -ni tap0 output showing ARP, ICMP, and SSH frames with the guest MAC.
A reading log naming the RX handler, the TX handler, the TAP-fd subscription, and where the WRITE flag is set/checked for RX vs TX descriptors.
A traced TX frame: device log TX→tap correlated with the same frame in tcpdump.
A traced RX frame: device log RX←tap correlated with tcpdump, plus a one-sentence explanation of why RX is woken by the TAP fd, not a guest kick.
A rate-limiter PATCH and an iperf3/scp measurement showing throughput capped near the bucket's refill rate.
All trace instrumentation removed.

Troubleshooting

Guest can ping the host TAP but not the internet

NAT or forwarding is off. Re-check sysctl net.ipv4.ip_forward (must be 1), the MASQUERADE rule on the uplink, and that $UPLINK is your real default-route interface. sudo iptables -t nat -L -n -v should show the MASQUERADE rule with packet counts climbing.

SSH connection refused / times out

Confirm the guest sshd is running (systemctl status ssh on the serial console), that the guest IP and route are set (Step 3), and that no host firewall blocks port 22 to 172.16.0.2. A timeout usually means routing; "connection refused" means routing works but sshd isn't listening.

`tcpdump` on tap0 shows nothing

The TAP must be up and the guest interface configured. If tcpdump sees outbound ARP from the guest but no replies, the host-side IP/route is wrong. If it sees nothing at all, the guest eth0 is down or the virtio-net interface didn't attach — re-check the /network-interfaces/eth0 PUT response.

`[trace]` byte counts don't match tcpdump lengths

virtio-net frames carry a virtio_net_hdr (often 10 or 12 bytes) that tcpdump does not show. Log the frame length excluding the header, or account for it. rg -n "virtio_net_hdr|hdr_len|vnet_hdr" src/vmm/src/devices/virtio/net/.

Rate limiter has no visible effect

The one_time_burst lets an initial burst through at full speed; measure over a longer window (10s+) so the steady-state refill rate dominates. Also confirm the PATCH succeeded (read the curl response) and that you limited the direction you're testing (TX from the guest's perspective is outbound).

`ip tuntap add` fails with permission denied

You need CAP_NET_ADMIN — run the ip/iptables commands with sudo. The firecracker process must be able to open the existing TAP; running it as root for this lab avoids TAP-ownership friction.

Expected Output

# tcpdump on tap0 during a guest `ping 1.1.1.1` (TX) and host `ping guest` (RX):
06:00:ac:10:00:02 > ..., IPv4, 172.16.0.2 > 1.1.1.1: ICMP echo request     # TX, guest→host
... > 06:00:ac:10:00:02, IPv4, 172.16.0.1 > 172.16.0.2: ICMP echo request  # RX, host→guest

# Device trace, correlated:
[Warning] [trace] net TX kick
[Warning] [trace] net TX→tap: 98 bytes
[Warning] [trace] net RX←tap: 98 bytes

# iperf3 after a ~1 MB/s rate limiter (steady state):
[  5]  9.00-10.00  sec  ... ~8.0 Mbits/sec

Stretch Goals

Use a bridge instead of routing. Create a br0, enslave tap0, and give the bridge the host IP. Compare the topology to the routed setup and explain when each is used (multiple microVMs on one L2 segment vs per-VM routes).
Trace RX-buffer exhaustion. Flood the guest faster than it posts RX buffers and find the device's drop/backpressure path. Read what your branch does and whether it's a known issue.
Two interfaces, two TAPs. Attach a second /network-interfaces/eth1 on tap1 and route a second subnet. Confirm each device is a separate VirtioDevice with its own queues and IRQ.
Ops rate limiter. Add an ops-bucket limit (packets/s, not bytes/s) and observe small-packet floods being throttled by packet rate rather than bandwidth.
MMDS over the link. Configure /mmds/config on this interface and curl the metadata service from the guest at 169.254.169.254. That traffic is handled in-VMM by dumbo, not sent to the TAP — trace where the device diverts MMDS-destined frames. See the networking masterclass MMDS lab.

Validation / Self-check

Answer without notes; these gate completion.

What is a TAP device, why does virtio-net want a TAP (not a TUN), and what does the host have to do that Firecracker does not (routing/NAT)?
Net has two virtqueues. Which is RX and which is TX, and for each, do the data descriptors carry the WRITE flag? Why?
A guest kick wakes the TX path. What wakes the RX path instead, and what is registered with the EventManager to make that happen?
Trace a guest ping outbound: name each hop from the guest driver to the wire, and say where tcpdump on tap0 sits in that sequence.
What is "RX-buffer exhaustion", and why is it a net-specific concern that block does not have?
Explain the token-bucket rate limiter: what are the two buckets, what is one_time_burst, and what does the device do when a bucket is empty?
Why does the device prepend a virtio_net_hdr, and how does it affect your byte-count correlation with tcpdump?

Cross-references: virtio-net-and-tap, rate-limiting-token-bucket, the-event-manager, networking masterclass, mmds-metadata-service.

Next: Lab 3 — Virtio-Vsock: a host↔guest socket channel with CID/port multiplexing and a host-Unix-socket bridge — the channel firecracker-containerd and Kata run their agents over.

Open-Source Engineer & Contributor