Lab 1: TAP devices and host networking

Background

A microVM does not have a network card; it has a virtio-net device whose entire backend is one host file descriptor — a TAP device, opened on /dev/net/tun in layer-2 mode. Firecracker reads raw Ethernet frames out of that fd and writes them in, and that is all it does. It does not route, it does not NAT, it does not bridge, it does not firewall. The instant a frame leaves the TAP and enters the host kernel, you — the operator — own it. If you want the guest to reach the internet, you must build that path on the host with the same tools you would use for any Linux interface: a bridge, or IP forwarding plus an iptables NAT.

This lab is the full host-networking story. You will create a TAP device by hand, attach it to a running microVM, and then give that guest real internet access two different ways: bridged, and NAT'd. You will add a second interface and reason about routing. And you will deliberately break the path at each hop and diagnose it with ip, tcpdump, and iptables — because "the guest can't reach the internet" is the single most common networking question, and the answer is almost never inside Firecracker. The canonical reference is docs/network-setup.md; this lab makes it runnable.

Why This Matters for Contributors

When a Type: Bug issue says "guest has no internet," the maintainers expect you to prove where the break is before claiming it's a Firecracker bug — and 90% of the time it is host misconfiguration, not Firecracker. A contributor who can say "the frame leaves the TAP fine (tcpdump -i tap0 shows it), the host forwards it (ip_forward=1, the FORWARD chain accepts), but the MASQUERADE rule is missing so the reply has no return path" has done real triage. A contributor who says "it doesn't work" has not. This lab builds the muscle to walk the packet path hop by hop. It also grounds the virtio-net deep dive: you will see the host side of the TAP fd that the device's read_tap/write_to_tap talk to.

Prerequisites

You completed the networking intensive overview readiness check.
A working firecracker binary, a vmlinux-* kernel, and an ext4 rootfs that boots (per Level 1, Lab 1.3). The rootfs should have iproute2, ping, and curl inside it.
Root on a disposable host/VM you own, with /dev/net/tun present and an uplink with real internet (you will note its name, e.g. eth0).
Read The virtio Net Device and the Host TAP.

# Verify, and capture your uplink interface name into a variable you'll reuse.
ls /dev/net/tun && modprobe tun
ip -o route get 8.8.8.8 | awk '{print "uplink:",$5; print "gw:",$3}'   # note the dev name
ls vmlinux-* *.ext4

Step-by-Step Tasks

Step 1 — Create a TAP device and bring it up

A TAP is a virtual layer-2 interface. Firecracker will bind to it by name (host_dev_name); you create it, give it an address, and bring it up. We give the host side of the TAP an address in a small subnet and the guest will take the other address in that subnet.

TAP_DEV="tap0"
TAP_IP="172.16.0.1"        # host side of the TAP
MASK_SHORT="/30"           # a /30 = exactly 2 usable addresses: host + guest
GUEST_IP="172.16.0.2"      # guest side

# Create the TAP, owned by this user, and address it.
ip tuntap add "$TAP_DEV" mode tap
ip addr add "${TAP_IP}${MASK_SHORT}" dev "$TAP_DEV"
ip link set "$TAP_DEV" up

# Confirm it exists and is UP with the address.
ip -br addr show "$TAP_DEV"
# tap0  UNKNOWN  172.16.0.1/30 ...

Note: A /30 gives you exactly two host addresses — perfect for the point-to-point host↔guest link of one microVM. Each microVM gets its own TAP and its own tiny subnet; do not share a TAP between two microVMs (their frames would collide on one fd). docs/network-setup.md uses exactly this one-TAP-per-microVM /30 pattern.

Step 2 — Boot the microVM bound to the TAP

Start Firecracker, configure the network interface to use tap0, and boot. The guest_mac is advertised to the guest via the virtio config space; pick any locally-administered MAC.

API=/tmp/fc.sock
rm -f "$API"
sudo ./firecracker --api-sock "$API" &

curl -X PUT --unix-socket "$API" --data \
 '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1 pci=off"}' \
 http://localhost/boot-source

curl -X PUT --unix-socket "$API" --data \
 '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
 http://localhost/drives/rootfs

# The network interface: bind the guest's virtio-net to host tap0.
curl -X PUT --unix-socket "$API" --data \
 '{"iface_id":"net1","guest_mac":"06:00:AC:10:00:02","host_dev_name":"tap0"}' \
 http://localhost/network-interfaces/net1

curl -X PUT --unix-socket "$API" --data '{"vcpu_count":2,"mem_size_mib":1024}' \
 http://localhost/machine-config

curl -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' http://localhost/actions

The guest boots; you have a serial console. Inside the guest, configure its side of the /30 and a default route through the host side of the TAP:

# --- run INSIDE the guest (serial console) ---
ip addr add 172.16.0.2/30 dev eth0
ip link set eth0 up
ip route add default via 172.16.0.1 dev eth0

# Sanity: the guest can reach the host side of the TAP.
ping -c2 172.16.0.1        # this must work; it's the directly-connected /30 peer

If ping 172.16.0.1 works, the Firecracker hop is correct end to end: guest virtqueue → net device → TAP fd → host tap0. Everything from here is pure host networking. If it does not work, the break is in the device/TAP wiring — jump to Troubleshooting.

Step 3 — Path A: NAT the guest to the internet with iptables

The guest can reach the host but not the world: its packets to, say, 8.8.8.8 arrive at the host on tap0, but the host has no rule to forward and masquerade them out the uplink, and the reply has no return path. Fix that with IP forwarding + a MASQUERADE (SNAT) rule.

UPLINK="eth0"              # your real internet-facing interface from the prereqs

# 1. Turn on IP forwarding (the host must act as a router). THIS IS THE #1 FORGOTTEN STEP.
sysctl -w net.ipv4.ip_forward=1

# 2. SNAT/MASQUERADE: rewrite the guest's source IP to the host's as packets leave the uplink.
iptables -t nat -A POSTROUTING -o "$UPLINK" -j MASQUERADE

# 3. Allow forwarding between the TAP and the uplink in both directions.
iptables -A FORWARD -i tap0 -o "$UPLINK" -j ACCEPT
iptables -A FORWARD -i "$UPLINK" -o tap0 -m state --state RELATED,ESTABLISHED -j ACCEPT

The guest also needs DNS. Put a resolver in the guest's /etc/resolv.conf:

# --- INSIDE the guest ---
echo 'nameserver 8.8.8.8' > /etc/resolv.conf
ping -c2 8.8.8.8                 # numeric: tests routing + NAT
curl -sS https://example.com -o /dev/null -w '%{http_code}\n'   # name + TLS: tests DNS too

flowchart LR
  G["guest 172.16.0.2"] -->|"frame"| NET["Net device"]
  NET -->|"write_to_tap"| TAP["host tap0 172.16.0.1"]
  TAP -->|"routed, ip_forward=1"| FW["FORWARD chain ACCEPT"]
  FW -->|"POSTROUTING MASQUERADE"| UP["uplink eth0 (src → host IP)"]
  UP --> NETW["internet"]
  NETW -.->|"reply to host IP"| UP -.->|"conntrack un-NAT"| TAP -.-> NET -.-> G

The MASQUERADE rule is what makes the reply find its way back: Linux conntrack remembers the translation, so a packet returning to the host's address is un-NAT'd and routed back down tap0. This is the standard "host as a router for the microVM" setup, and it's what docs/network-setup.md recommends for a single host.

Step 4 — Path B: bridge the guest onto a real L2 network

NAT hides the guest behind the host's IP. A bridge instead puts the guest directly on a layer-2 segment — it gets its own address on the same network as the host's uplink (e.g. via DHCP), and other machines can reach it directly. You attach the TAP to a Linux bridge and put the uplink on the same bridge.

# Tear down the NAT path first if you did Step 3 (so the two don't fight):
iptables -t nat -F; iptables -F FORWARD

# Create a bridge, enslave the TAP and the uplink to it.
ip link add br0 type bridge
ip link set tap0 master br0
ip link set "$UPLINK" master br0
ip link set br0 up

# Move the host's IP from the uplink onto the bridge (the bridge is now the host's interface).
# WARNING: if you are SSH'd in over $UPLINK this will drop your connection — do it on console.
ip addr flush dev "$UPLINK"
dhclient br0 2>/dev/null || ip addr add 192.168.1.50/24 dev br0   # adjust to your LAN

Now the guest can take an address on the real LAN (statically or via DHCP if a DHCP server is on the segment):

# --- INSIDE the guest ---
ip addr flush dev eth0
ip addr add 192.168.1.77/24 dev eth0        # an address on the host's real LAN
ip route add default via 192.168.1.1        # the LAN's real gateway
ping -c2 192.168.1.1 && ping -c2 8.8.8.8

Tip: Bridged is the right model when the guest must be reachable from the LAN (a server). NAT is the right model when the guest only needs outbound access and should stay hidden (the common serverless case). NAT also needs no changes to the host's own addressing, which is why docs/network-setup.md leads with it. Pick deliberately; do not run both at once.

Step 5 — A second interface, and routing

A microVM can have multiple interfaces. Add a second TAP and a second network-interfaces entry (before boot — interface bindings are pre-boot; only rate limiters can be PATCHed live):

# Host: a second TAP in a different /30.
ip tuntap add tap1 mode tap
ip addr add 172.16.0.5/30 dev tap1
ip link set tap1 up

# In the API config, a second iface BEFORE InstanceStart:
curl -X PUT --unix-socket "$API" --data \
 '{"iface_id":"net2","guest_mac":"06:00:AC:10:00:06","host_dev_name":"tap1"}' \
 http://localhost/network-interfaces/net2

# --- INSIDE the guest, after boot ---
ip addr add 172.16.0.6/30 dev eth1
ip link set eth1 up
# Now the guest has two uplinks. Which is the default route? Only ONE default wins;
# the rest need explicit routes or source-based routing (ip rule + a routing table).
ip route                       # inspect; the guest picks based on metric/order

The lesson: a second interface does not "just" add bandwidth — the guest's routing table decides which interface a given destination uses, and a misconfigured default route is a classic "interface up but no traffic" symptom. Multi-interface microVMs are common in orchestrated setups (one for tenant traffic, one for a control/metadata plane); the routing is the guest's responsibility, not Firecracker's.

Step 6 — Walk the host-side packet path with tcpdump

Now prove the path is what you think it is. Re-enable the NAT path (Step 3), then watch the same packet at three points as the guest pings 8.8.8.8:

# Terminal 1: at the TAP — frames leaving the microVM.
tcpdump -ni tap0 icmp
# Terminal 2: at the uplink — the SAME pings, but NAT'd to the host's source IP.
tcpdump -ni "$UPLINK" icmp
# Terminal 3 (guest): generate the traffic.
#   ping -c5 8.8.8.8

You should see, on tap0, 172.16.0.2 > 8.8.8.8; and on the uplink, <host-ip> > 8.8.8.8 — the source address changed, which is the MASQUERADE doing its job. The reply appears on the uplink as 8.8.8.8 > <host-ip> and on tap0 as 8.8.8.8 > 172.16.0.2. Seeing the address rewrite happen is the whole point: you have observed the NAT, not assumed it.

Implementation Requirements / Deliverables

A TAP device created, addressed (/30), and brought up; a microVM booted bound to it.
The guest can ping the host side of the TAP (proves the Firecracker hop is correct).
Path A: guest reaches the internet via ip_forward + MASQUERADE + FORWARD rules; both numeric ping and a curl https:// (DNS+TLS) succeed.
Path B: guest reaches the LAN via a bridge with the TAP and uplink enslaved.
A second interface added pre-boot; the guest's routing table inspected and explained.
A three-point tcpdump capture showing the source-address rewrite at the NAT.
A written diagnosis, for a broken path, naming the exact hop that failed and the command that proved it.

Troubleshooting

Guest cannot ping the host side of the TAP (`172.16.0.1`)

The break is inside the Firecracker hop, not the host network. Check, in order:

ip -br addr show tap0                 # is tap0 UP and addressed on the host?
rg -n "host_dev_name|tap0" <(curl -s --unix-socket "$API" http://localhost/network-interfaces/net1) 2>/dev/null
tcpdump -ni tap0                       # do the guest's ARP/ICMP frames even reach the host TAP?

If tcpdump -ni tap0 shows nothing when the guest pings, the frame is not leaving the device — the guest's eth0 is down, has the wrong address, or the host_dev_name in the PUT didn't match the TAP name. If it shows the guest's ARP requests but no replies, the host side of the TAP isn't up/addressed in the same subnet. This is the boundary between "Firecracker's problem" (rare) and "host config" (common); tcpdump -ni tap0 is the call that draws the line.

Guest pings the host but not `8.8.8.8`

Routing/NAT on the host. Walk it:

cat /proc/sys/net/ipv4/ip_forward            # MUST be 1 — the most common miss
iptables -t nat -L POSTROUTING -v -n         # is the MASQUERADE rule present, with hits?
iptables -L FORWARD -v -n                    # is FORWARD allowing tap0 <-> uplink?
ip route get 8.8.8.8                          # does the HOST itself have a route out?

A MASQUERADE rule with zero packet counts while the guest is pinging means traffic isn't reaching POSTROUTING — usually ip_forward=0 or a FORWARD-chain DROP. A rule with hits but no reply means a return-path problem (a RELATED,ESTABLISHED FORWARD rule missing, or a default DROP policy).

Name resolution fails but numeric IPs work

DNS, not networking. The guest's /etc/resolv.conf has no (reachable) nameserver. Set nameserver 8.8.8.8 in the guest. ping 8.8.8.8 working while curl https://example.com hangs is the textbook signature.

`iptables` rules vanish after reboot, or conflict with Docker

iptables rules are not persistent and Docker installs its own FORWARD policies. On a dev box, just re-apply the rules each session. If Docker's FORWARD policy is DROP, your -A FORWARD ... ACCEPT must come before Docker's catch-all, or insert with -I FORWARD 1.

Bridged path drops your SSH session

Moving the host IP off the uplink onto the bridge (Step 4) tears down any connection that was using the uplink's address. Do bridge setup from the console, not over SSH on the interface you're enslaving.

Expected Output

# Guest, NAT path working:
$ ping -c2 8.8.8.8
64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=12.3 ms
$ curl -sS https://example.com -o /dev/null -w '%{http_code}\n'
200

# Host, three-point tcpdump showing the NAT rewrite:
# on tap0:    172.16.0.2 > 8.8.8.8: ICMP echo request
# on eth0:    203.0.113.7 > 8.8.8.8: ICMP echo request   <-- source rewritten by MASQUERADE

Stretch Goals

Source-based routing with two interfaces. Give the guest two TAPs in two subnets and configure ip rule + a second routing table so traffic from each guest source IP egresses its own interface. Prove it with tcpdump on both TAPs.
A MTU mismatch. Set the guest's eth0 MTU above what the path supports and watch large transfers stall (PMTU black hole). Then advertise a correct MTU via the interface config and confirm VIRTIO_NET_F_MTU carries it (rg -n "VIRTIO_NET_F_MTU|mtu" src/vmm/src/devices/virtio/net/).
Offload mismatch. Read where Firecracker derives TUNSETOFFLOAD from the negotiated features (rg -n "set_offload|TUNSETOFFLOAD" src/vmm/src/devices/virtio/net/), then reason about what breaks if the TAP's offloads and the device's feature bits disagree (the deep dive's EINVAL-on-large-frame bug).
Reproduce a real issue. Find an open networking issue (gh issue list --repo firecracker-microvm/firecracker --search "network OR tap OR connectivity in:title state:open") and reproduce its host-side setup precisely.

Validation / Self-check

Answer without notes; these gate completion:

What is the only network hop Firecracker owns, and where does host responsibility begin?
Why does ping 172.16.0.1 (the host side of the TAP) succeeding prove the Firecracker hop is correct, regardless of whether the guest can reach the internet?
Name the three host-side things the NAT path needs (forwarding, SNAT, forward-chain accept) and what each one does. Which is the most commonly forgotten?
In the three-point tcpdump, which field changes between tap0 and the uplink, and which component changed it?
Contrast bridged vs NAT'd: when do you want each, and which one changes the host's own addressing?
A guest has two interfaces, both up, but traffic only uses one. Where is the bug — Firecracker or the guest? What do you inspect?
Why must each microVM have its own TAP rather than sharing one?

Next: Lab 2: Rate limiting — now that the path works, put a token-bucket throttle on it, measure the cap with iperf3 and fio, and read the rate_limiter code so the numbers you measure match the math in the source.

Open-Source Engineer & Contributor