Networking — Intensive

Every other masterclass treats the network as a wire that "just works." This one is about the wire itself — and the surprising amount of policy Firecracker layers on top of a single file descriptor. A microVM has no NIC of its own; it has a virtio-net device whose backend is a host TAP file descriptor, and nothing about routing, NAT, firewalling, bandwidth fairness, or metadata lives inside the guest's idea of "the network." It lives on the host, in iptables and bridges you wire up by hand, in token buckets the VMM enforces on the device fast path, and in a tiny in-VMM TCP/IP stack that fabricates replies for 169.254.169.254. This intensive makes all three concrete.

The reason a serious contributor needs this is that "guest networking is broken" is one of the most common, most under-diagnosed classes of Firecracker issue — and almost none of it is a Firecracker bug. The guest can ping the host but not the internet (a missing NAT rule). Metadata resolves but real traffic doesn't (the MMDS detour works, the TAP path doesn't). Throughput is a tenth of the configured cap (a too-small token bucket, or a timer fd that never got registered). To triage any of these you must hold the entire path in your head — guest virtqueue → VMM device → TAP fd → host bridge/NAT → uplink — and know which hop owns which failure. That is what these three labs build.

   GUEST                FIRECRACKER (VMM thread)              HOST KERNEL                 WORLD
 ┌────────┐  virtqueues ┌───────────────────────────┐  /dev/net/tun ┌──────────┐  NAT  ┌────────┐
 │ virtio │◄═══════════►│  Net device               │◄═════════════►│  tapN    │──────►│ bridge │──► internet
 │  NIC   │  RX(0)TX(1) │  • rate limiters (Lab 2)  │  raw frames   │ (IFF_TAP)│ iptables│ / route│
 │        │             │  • MMDS detour (Lab 3) ───┼──► dumbo      └──────────┘       └────────┘
 └────────┘             └───────────────────────────┘   169.254.169.254
            ▲ Lab 1: the host plumbing that makes tapN reach the internet ▲

Note: Firecracker owns exactly one hop — guest ↔ TAP fd. Everything to the right of the TAP is the host operator's job, and everything inside the device (rate limiting, MMDS) is policy the VMM enforces on that one hop. If you can draw this diagram from memory and name which component owns each arrow, you can triage almost any guest-networking issue. The labs make you build, throttle, and intercept each arrow in turn.

What you will be able to do

By the end of this intensive you will be able to:

1. Create a TAP device and wire a guest to the internet two ways — a bridge and an iptables NAT/SNAT — and explain the host-side packet path frame by frame, from tapN to the uplink.
2. Run a microVM with multiple network interfaces and reason about routing, source-based routing, and why the guest's default route matters.
3. Troubleshoot a broken guest network methodically with ip, ip route, tcpdump, iptables -L -v, and /proc/sys/net/ipv4/ip_forward — narrowing the failure to a single hop instead of guessing.
4. Configure token-bucket rate limiters on a net interface (rx_rate_limiter/tx_rate_limiter) and a drive (rate_limiter) with size/one_time_burst/refill_time, on both the ops and bandwidth buckets.
5. Measure the limiters with iperf3 (network) and fio (disk), see the cap take effect, and PATCH the limits at runtime without rebooting the guest.
6. Read the rate_limiter implementation — the TokenBucket, the lazy refill, the two-bucket model, and the non-blocking pause/resume cycle — and connect the code to the numbers you measured.
7. Configure and use MMDS — PUT /mmds/config (version V2), PUT /mmds data, the guest-side 169.254.169.254 route, and the IMDSv2 token flow — and trace how the net device intercepts MMDS traffic and the dumbo stack answers it.

The three labs

Do them in order. Lab 1 builds the host plumbing every later lab assumes; Lab 2 throttles the path you just built; Lab 3 adds the one piece of traffic that never reaches the TAP at all.

• Lab 1: TAP devices and host networking — build-it / trace-it. Create a TAP device, give the guest internet access two ways (a Linux bridge, and an iptables MASQUERADE NAT), add a second interface, and walk the host-side packet path. Then break it and fix it with ip, tcpdump, and iptables. The full host networking story, grounded in docs/network-setup.md.

• Lab 2: Rate limiting — build-it / measure-it. Put token-bucket limiters on a net interface and a drive, measure the effect with iperf3 and fio, PATCH the limits live, and read the rate_limiter code (rg) so the math you measured matches the math in the source. The multi-tenancy/fairness motivation, made measurable.

• Lab 3: MMDS — build-it / trace-it. Configure MMDS V2, push a metadata document, set up the guest route to 169.254.169.254, fetch metadata through the IMDSv2 token flow, and trace the interception: where the net device's TX path checks the destination and detours the frame into the dumbo in-VMM TCP/IP stack instead of the TAP.

Prerequisites

This is a masterclass, not an introduction. You must have completed all of Level 7: The Virtio Device Model — in particular Lab 7.1: Trace a virtio-block I/O and Lab 7.2: Virtqueues and MMIO — so the virtqueue/MMIO machinery underneath the net device is already second nature. You must also have read, and be able to draw from memory, the deep dive this intensive expands on:

You need a working Firecracker build, the ability to boot a microVM by hand exactly as in Level 1, Lab 1.3, and root on the host (creating TAP devices, bridges, and iptables rules all require it). Confirm your environment now:

# 1. You have a firecracker binary (path is arch/libc-specific — verify on your branch).
ls build/cargo_target/$(uname -m)-unknown-linux-musl/release/firecracker

# 2. You can create TAP devices and write iptables/bridge rules (root, and the modules present).
id -u                                  # 0, or you can sudo
ls /dev/net/tun                        # the TUN/TAP char device must exist
modprobe tun 2>/dev/null; lsmod | grep -E "^tun|bridge" || echo "load tun/bridge if missing"

# 3. The host-networking guidance the labs build on.
ls docs/network-setup.md docs/mmds/ 2>/dev/null

# 4. A kernel + rootfs to boot, and a guest with curl/iperf3/fio in it for the measurement labs.
ls vmlinux-* *.ext4 2>/dev/null || echo "fetch a kernel + rootfs per docs/getting-started.md"

Warning: Run these labs on a disposable host or VM you own. You will create TAP interfaces, bridges, and iptables rules, enable IP forwarding, and (in Lab 2) saturate a NIC and a disk with iperf3/fio. None of it should touch a shared or production machine. Tear the box down and it all disappears; the labs note the cleanup commands as you go.

How this intensive connects to the rest of the curriculum

The through-line: Firecracker owns one hop, and everything else is policy. The TAP is a dumb pipe; the bridge/NAT is the operator's; the rate limiter is a throttle the VMM puts on the pipe; MMDS is the one exception where the VMM originates traffic to the guest. Hold that division clearly and the whole subsystem stops being mysterious.

Common mistakes contributors make in this area

How to verify you are ready to start

# Answer each from the deep dives BEFORE Lab 1.
# 1. Which queue index is RX and which is TX on the net device?
rg -n "RX_INDEX|TX_INDEX|RX_QUEUE|TX_QUEUE" src/vmm/src/devices/virtio/net/
# 2. Which TUNSETIFF flags does Firecracker set on the TAP, and what do they mean?
rg -n "TUNSETIFF|IFF_TAP|IFF_NO_PI" src/vmm/src/devices/virtio/net/tap.rs
# 3. Where are the two token buckets, and what does each throttle?
rg -n "struct TokenBucket|TokenType::Ops|TokenType::Bytes" src/vmm/src/rate_limiter/
# 4. Where does the net device decide a frame is MMDS-bound and divert it?
rg -n "mmds|MmdsNetworkStack|169\.254" src/vmm/src/devices/virtio/net/

If you cannot say, from memory, that Firecracker owns only the guest↔TAP hop, that the rate is size/refill_time, and that the MMDS detour happens in the net device's TX path before the TAP write, re-read the three deep dives above before continuing. The labs assume all three are second nature.

Next: Lab 1: TAP devices and host networking — build the host plumbing that turns a bare TAP into a guest with real internet access, two ways, then break it and fix it with ip, tcpdump, and iptables.