Lab 3: MMDS

Background

There is one piece of guest network traffic that never reaches the TAP: a request to 169.254.169.254. That link-local address is the microVM Metadata Service (MMDS) — Firecracker's in-VMM clone of the AWS EC2 Instance Metadata Service (IMDS). The orchestrator pushes a JSON document into the VMM over the REST API; the guest reads it back over its own NIC by curl-ing http://169.254.169.254/... as if a real metadata server were listening. But there is no server, and no host networking is involved. Firecracker's virtio-net device intercepts frames bound for the MMDS address on the TX path — before they would be written to the TAP — and hands them to a tiny in-VMM TCP/IP stack called dumbo, which fabricates the HTTP reply and injects it back on the RX queue. The guest cannot tell the difference.

This lab makes that whole loop concrete. You will configure MMDS with version V2 (the token/session model that mirrors IMDSv2), push a metadata document, set up the guest's route so 169.254.169.254 resolves to its NIC, fetch metadata through the full IMDSv2 token flow, and then trace the interception in the source: where process_tx checks the destination, where the frame detours into MmdsNetworkStack/dumbo, and where the synthesized reply is injected. By the end you understand the one deliberate place Firecracker originates traffic to the guest — and why that makes dumbo part of the attack surface.

Why This Matters for Contributors

MMDS is small but load-bearing and security-relevant. dumbo parses guest-supplied Ethernet/IPv4/TCP frames inside the host process — it is attack surface, which is exactly why it is minimal and why production hosts are told to block guest egress to 169.254.169.254. A contributor who can trace a metadata request from the guest's curl through the net-device hook into dumbo and back understands both a real feature and a real trust boundary; that is the kind of person who can safely touch the parser or the V2 token logic. This lab is also the prerequisite for the security masterclass threat-model audit, which counts dumbo among the parsers in the attack surface.

Prerequisites

You completed Lab 1: a microVM with a working TAP and a configured guest NIC. (MMDS does not need internet — it needs the guest's NIC to exist and be up.)
A guest rootfs with curl (or wget) installed.
Read The microVM Metadata Service (MMDS) and the MMDS detour section of virtio-net & TAP.

# The MMDS and dumbo modules, and the net-device hook, exist on your branch.
find src/vmm/src/mmds src/vmm/src/dumbo -name "*.rs" | head
rg -n "mmds|MmdsNetworkStack|169\.254" src/vmm/src/devices/virtio/net/ | head
ls docs/mmds/ 2>/dev/null

Step-by-Step Tasks

Step 1 — Configure MMDS (version V2) and push data, pre-boot

There are two distinct endpoints, and the distinction matters: PUT /mmds/config sets the settings (version, which interfaces may reach MMDS, the IPv4 address) and is pre-boot; PUT /mmds sets the contents (the JSON the guest reads) and can be updated while the VM runs.

API=/tmp/fc.sock
rm -f "$API"; sudo ./firecracker --api-sock "$API" &

# Boot source + rootfs (as in Lab 1).
curl -X PUT --unix-socket "$API" --data \
 '{"kernel_image_path":"./vmlinux-6.1.x","boot_args":"console=ttyS0 reboot=k panic=1 pci=off"}' \
 http://localhost/boot-source
curl -X PUT --unix-socket "$API" --data \
 '{"drive_id":"rootfs","path_on_host":"./ubuntu-24.04.ext4","is_root_device":true,"is_read_only":false}' \
 http://localhost/drives/rootfs

# The interface that will be GRANTED MMDS access (must match network_interfaces below).
curl -X PUT --unix-socket "$API" --data \
 '{"iface_id":"net1","guest_mac":"06:00:AC:10:00:02","host_dev_name":"tap0"}' \
 http://localhost/network-interfaces/net1

# 1. MMDS settings: V2 (token/session), grant net1, the standard link-local address. PRE-BOOT.
curl -X PUT --unix-socket "$API" --data \
 '{"version":"V2","network_interfaces":["net1"],"ipv4_address":"169.254.169.254"}' \
 http://localhost/mmds/config

# 2. MMDS contents: the metadata document the guest will read. (Can also be PATCH'd live later.)
curl -X PUT --unix-socket "$API" --data '{
  "latest": { "meta-data": {
    "instance-id": "i-0abc123",
    "local-hostname": "fc-guest",
    "placement": { "availability-zone": "us-east-1a" }
  } }
}' http://localhost/mmds

curl -X PUT --unix-socket "$API" --data '{"vcpu_count":2,"mem_size_mib":1024}' http://localhost/machine-config
curl -X PUT --unix-socket "$API" --data '{"action_type":"InstanceStart"}' http://localhost/actions

Note: If network_interfaces in /mmds/config does not list an interface, MMDS is unreachable over it — the net-device hook only diverts frames on a granted interface. "Connection refused at 169.254.169.254" is almost always a missing grant or the wrong interface, not a guest problem. This is the deliberate "explicitly allowed interface only" design.

Step 2 — Route `169.254.169.254` to the NIC inside the guest

The guest must know to send 169.254.169.254 out its eth0. The address is link-local, so a host route on the device is enough; no gateway is involved (the "next hop" is the in-VMM stack, which answers ARP for the address).

# --- INSIDE the guest --- bring up the NIC (from Lab 1) and add a route to the MMDS address.
ip addr add 172.16.0.2/30 dev eth0
ip link set eth0 up
ip route add 169.254.169.254 dev eth0     # send the metadata address out eth0

# Confirm the route is there.
ip route get 169.254.169.254
# 169.254.169.254 dev eth0 ...

Step 3 — Fetch metadata through the IMDSv2 token flow

V2 is a two-step flow, exactly like AWS IMDSv2: PUT /latest/api/token to obtain a short-lived session token, then present it in the X-metadata-token header on every GET. A bare GET without a token is refused — that is the whole point of V2 (it defends against confused-deputy/SSRF attacks where a vulnerable guest workload is tricked into fetching metadata).

# --- INSIDE the guest ---

# A bare GET with no token: V2 refuses it.
curl -s http://169.254.169.254/latest/meta-data/instance-id
# (401 / unauthorized — no token)

# 1. Get a session token (TTL in seconds via the X-metadata-token-ttl-seconds header).
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
            -H "X-metadata-token-ttl-seconds: 21600")
echo "token: $TOKEN"

# 2. Now GET with the token. This succeeds.
curl -s -H "X-metadata-token: $TOKEN" \
     http://169.254.169.254/latest/meta-data/instance-id
# i-0abc123
curl -s -H "X-metadata-token: $TOKEN" \
     http://169.254.169.254/latest/meta-data/placement/availability-zone
# us-east-1a

Tip: Confirm the exact header names on your branch — IMDS header spelling is a detail that drifts. rg -n "X-metadata-token|token-ttl|generate_token|api/token" src/vmm/src/mmds/ shows the server side. The flow (PUT-for-token, then GET-with-token) is stable; the precise header strings are the thing to verify.

Step 4 — Update metadata live, and read it back

PUT/PATCH /mmds works while the VM runs — this is how an orchestrator pushes fresh metadata to a live guest. Change a field on the host and re-read it in the guest with the same token.

# HOST: PATCH the contents (merge) while the guest runs.
curl -X PATCH --unix-socket "$API" --data \
 '{"latest":{"meta-data":{"local-hostname":"fc-guest-renamed"}}}' http://localhost/mmds

# GUEST: re-read — the new value is served immediately, no reboot.
curl -s -H "X-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-hostname
# fc-guest-renamed

This is the /mmds (runtime contents) vs /mmds/config (pre-boot settings) split made tangible: you just changed contents on a running VM, which /mmds/config would not let you do.

Step 5 — Prove the traffic never touches the TAP

The defining property of MMDS: these packets are intercepted inside the net device and never reach the host TAP. Prove it by sniffing the TAP while the guest hammers MMDS.

# HOST: sniff the TAP for MMDS traffic.
tcpdump -ni tap0 host 169.254.169.254
# --- GUEST: generate metadata traffic ---
for i in 1 2 3; do curl -s -H "X-metadata-token: $TOKEN" \
   http://169.254.169.254/latest/meta-data/instance-id >/dev/null; done

tcpdump on tap0 shows nothing for 169.254.169.254 — the frames were diverted in process_tx before write_to_tap. Contrast: a curl to any normal address does appear on tap0. Seeing the absence is the proof. (This is also why a broken TAP can leave internet dead while metadata still works — the two paths diverge inside the device.)

Step 6 — Trace the interception in the source

Now read the code path you just exercised. Three hops: the net-device hook (decide + divert), the dumbo stack (parse + answer), and the MMDS store/V2 logic (the data and the token check).

# 1. The net-device TX hook: where a frame's destination is checked and diverted to MMDS.
rg -n "mmds|MmdsNetworkStack|169\.254|detour|is_mmds|process_tx" src/vmm/src/devices/virtio/net/

# 2. dumbo: the tiny in-VMM TCP/IP stack that terminates the connection and parses HTTP.
find src/vmm/src/dumbo -name "*.rs"
rg -n "Tcp|Ipv4|EthernetFrame|arp|checksum|Connection" src/vmm/src/dumbo/

# 3. The MMDS store, the V2 token/session logic, and the HTTP handler.
find src/vmm/src/mmds -name "*.rs"
rg -n "struct Mmds|MmdsVersion|V1|V2|token|session|X-metadata-token|fn get|fn put|generate_token" \
  src/vmm/src/mmds/

# 4. Where /mmds and /mmds/config become VmmActions (the API → VMM wiring).
rg -n "mmds|MmdsConfig|PutMmds|GetMmds|PatchMmds" src/vmm/src/rpc_interface.rs src/firecracker/src/

Read it as one flow and you should be able to narrate: a guest frame to 169.254.169.254 arrives in process_tx; the device checks the destination (and that this interface is MMDS-granted); if so, the frame goes to MmdsNetworkStack instead of the TAP; dumbo parses Ethernet → IPv4 → a minimal TCP state machine, hands the HTTP bytes to the MMDS handler; the handler checks the V2 token, reads the JSON store, builds an HTTP response; dumbo frames it and the net device injects it on the RX queue as if it came off the wire. The guest's curl returns. No TAP, no host server, no real network.

flowchart TD
    Guest["guest: curl 169.254.169.254 (+ X-metadata-token)"] --> TX["Net::process_tx"]
    TX --> Check{"dst == MMDS addr<br/>AND iface granted?"}
    Check -->|no| TAP["write_to_tap → host network"]
    Check -->|yes| NS["MmdsNetworkStack (dumbo)"]
    NS --> Parse["parse Ethernet → IPv4 → minimal TCP"]
    Parse --> HTTP["MMDS HTTP handler"]
    HTTP --> Tok{"V2: valid token?"}
    Tok -->|no| Refuse["401 / unauthorized"]
    Tok -->|yes| Store["read JSON store"]
    Store --> Resp["build HTTP response"]
    Refuse --> Inject["inject reply frame on RX queue"]
    Resp --> Inject
    Inject --> Guest

Implementation Requirements / Deliverables

MMDS configured V2 via PUT /mmds/config, granting a specific interface; data pushed via PUT /mmds.
A guest route to 169.254.169.254; the IMDSv2 two-step token flow performed by hand (token PUT, then GET with X-metadata-token).
A demonstration that a tokenless GET is refused under V2.
Live metadata update via PATCH /mmds read back in the running guest.
A tcpdump on the TAP proving MMDS traffic never reaches it.
A written trace of the three hops (net hook → dumbo → MMDS store/V2), each anchored to the file you found it in.

Troubleshooting

Connection refused / no reply from `169.254.169.254`

The interface isn't granted, or you routed to the wrong device. Check network_interfaces in /mmds/config lists the interface the guest is using, and that the guest's ip route get 169.254.169.254 resolves to that NIC (eth0). Re-GET the config: curl -s --unix-socket "$API" http://localhost/mmds/config. A granted interface plus a correct host route are both required.

V2 GETs return 401 even with a token

Either the token expired (the TTL you set in X-metadata-token-ttl-seconds elapsed — fetch a new one), or the header name is wrong for your branch. rg -n "X-metadata-token|token" src/vmm/src/mmds/ shows the exact spelling the server expects. Re-fetching the token and copying the header string verbatim fixes the common case.

Metadata looks stale after a host push

/mmds updates the store immediately, but the guest may have cached the HTTP response, or you pushed before the interface was granted. Confirm with a fresh curl (no cache) and confirm ordering: grant the interface in /mmds/config (pre-boot), then PUT/PATCH /mmds contents.

The guest can reach the internet but not MMDS (or vice versa)

These are two different paths that diverge in process_tx. Internet works but MMDS doesn't → the MMDS grant/route is wrong (the TAP path is fine). MMDS works but internet doesn't → the detour is fine but the host NAT/bridge is broken (back to Lab 1). The tcpdump-on-TAP test from Step 5 tells you which side you're on.

A malformed request hangs or crashes the connection

That's a dumbo parser edge case — and a genuinely interesting find, because dumbo is attack surface. Reduce it to a minimal reproducing frame, identify which parser layer (Ethernet/IPv4/TCP) mishandles it (rg in src/vmm/src/dumbo/), and treat it as a potential security issue — report privately to AWS Security, not a public issue, per SECURITY.md.

Expected Output

# Tokenless GET under V2 (Step 3):
$ curl -s http://169.254.169.254/latest/meta-data/instance-id
# (no body / 401)

# Token flow (Step 3):
$ TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token -H "X-metadata-token-ttl-seconds: 21600")
$ curl -s -H "X-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id
i-0abc123

# TAP sniff during MMDS traffic (Step 5): NOTHING for 169.254.169.254
$ tcpdump -ni tap0 host 169.254.169.254
listening on tap0 ...
0 packets captured     <-- the detour: frames never reached the TAP

Stretch Goals

V1 vs V2, observed. Reconfigure with "version":"V1" (pre-boot) and show a bare GET now succeeds without a token. Then explain the SSRF/confused-deputy attack V2 closes, and why V1 is deprecated. (docs/prod-host-setup.md on egress blocking is relevant here.)
Why MMDS isn't snapshotted. Snapshot a microVM (per Level 9, Lab 9.2), restore it, and show MMDS contents are gone — then explain the "VM state vs orchestrator policy" reasoning (rg -n "Persist|mmds|snapshot" src/vmm/src/mmds/).
Read the dumbo TCP state machine. Find how dumbo handles a single short connection — SYN, data, FIN — and identify what it deliberately does not implement (it is not a general stack). Argue why each omission is a security win.
The token-store internals. Find how V2 tokens are generated and validated (rg -n "generate_token|token|session" src/vmm/src/mmds/). Is the token bound to the connection, the interface, time, or all three? What would weaken it?
Find an MMDS issue to own. gh issue list --repo firecracker-microvm/firecracker --search "mmds OR metadata OR imds in:title state:open" — pick one and reproduce it with the setup above.

Validation / Self-check

Answer without notes; these gate completion:

Where does MMDS run, what address does the guest use, and what is actually "listening" there?
Distinguish PUT /mmds from PUT /mmds/config: what does each set, and which is pre-boot vs runtime?
Describe the IMDSv2 (V2) token flow exactly — both steps — and the specific attack the token defends against.
Trace a metadata request from the guest's curl to the synthesized reply, naming the net-device hook, dumbo, the V2 token check, and the RX-queue injection.
Why does MMDS traffic never appear on the host TAP? At exactly which point in the TX path is it diverted, and why must it be there?
Why is dumbo deliberately minimal, and why is that minimalism a security property? Name the three protocol layers it must parse.
Why is MMDS content deliberately excluded from snapshots, and what is the orchestrator expected to do on restore?

This is the last lab of the networking intensive. You now own the whole guest-network story — the TAP and host plumbing (Lab 1), the token-bucket throttle on it (Lab 2), and the one detour that never touches the wire (this lab). Carry dumbo-as-attack-surface into the security threat-model audit, or move on to the Performance & Density intensive to make the fairness and overhead arguments behind rate limiting quantitative.

Open-Source Engineer & Contributor