Lab 5.3: Build It — A Multi-Step Integration Test

Background

This is the lab where you write the kind of test a feature PR requires. CONTRIBUTING is blunt: new functionality needs integration tests. An integration test boots a real microVM and asserts an observable, end-to-end behavior — not "the API accepted my JSON," but "the guest actually sees the device I configured." The canonical example, and the one you'll build here, is: configure a second block device, boot, and prove from inside the guest that it appears as /dev/vdb with the right size and contents.

This is a build-it lab. You will write a complete, plausible pytest using the Microvm helper and the SSH-into-guest utilities, run it under tools/devtool test, and make it pass on the real parametrized kernel matrix. You already traced an existing test in Lab 5.1 and learned the framework's verbs; now you produce one.

Note: The test below is written to match the framework conventions verified against the live repo (Microvm.spawn/basic_config/add_drive/add_net_iface/start/ssh, the microvm_factory, guest_kernel, and rootfs fixtures). Method signatures drift — when something doesn't match, rg the helper on your branch and adapt. The shape is what you're learning.

Why This Lab Matters for Contributors

• This is the deliverable that turns a feature PR from "rejected" to "reviewable." No test, no merge.
• It exercises the full stack you've studied: API → VmmAction → DeviceManager → virtio-block → guest. See the virtio-block deep dive for what's underneath.
• The SSH-into-guest pattern is how you assert guest-visible behavior, which is the only kind that matters for a user-facing feature.
• You'll reuse this exact structure for net, vsock, balloon, mmds, and metrics tests.

Prerequisites

• Completed Lab 5.1 (you can read and run an integration test).
• A built checkout; tools/devtool test runs.

# The model you'll copy from — read it before writing.
sed -n '1,120p' tests/integration_tests/functional/test_drive_virtio.py
rg -n "def add_drive|def basic_config|def add_net_iface|def ssh\b" tests/framework/microvm.py

Step 1: Decide the behavior and the assertion

A good integration test names one behavior and one guest-visible assertion. Pick from:

We build the second-drive test. The assertion is concrete and unambiguous: create a host file of known size, attach it as a second virtio-block device, boot, and from inside the guest confirm /dev/vdb exists and reports the matching number of 512-byte sectors.

sequenceDiagram
    participant T as Test
    participant H as Host
    participant FC as firecracker
    participant G as Guest
    T->>H: dd a 64 MiB scratch file
    T->>FC: spawn() + basic_config()  (rootfs = /dev/vda)
    T->>FC: add_drive("scratch", scratch_path)   (PUT /drives/scratch → /dev/vdb)
    T->>FC: add_net_iface()  (so we can SSH)
    T->>FC: start()  (InstanceStart → guest boots)
    T->>G: ssh.run("lsblk -bno NAME,SIZE")
    G-->>T: vdb present, size == 64 MiB
    T->>T: assert

Step 2: Confirm the helper signatures on your branch

Never trust a signature from prose. Pin the real ones:

# add_drive: how do you attach an extra block device, and what params does it take?
rg -n "def add_drive" -A8 tests/framework/microvm.py

# Where does it route — PUT /drives/{id}?
rg -n "drives|drive_id|path_on_host|is_root_device" tests/framework/microvm.py | head

# The ssh property and run() return shape (exit_code, stdout, stderr).
rg -n "def ssh\b|def ssh_iface|class .*Ssh|def run\b" tests/framework/*.py

You're confirming three things: the add_drive keyword arguments (drive_id, path_on_host, is_root_device, is_read_only, possibly io_engine), that it issues PUT /drives/{id}, and that ssh.run(cmd) returns a (exit_code, stdout, stderr)-style result.

Step 3: Write the test

Create the file under the functional directory. Name it so pytest collects it and a reader knows what it covers:

$EDITOR tests/integration_tests/functional/test_second_drive.py

# Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
"""Integration test: a second block device shows up in the guest as /dev/vdb."""

import os

# A size that's easy to verify and cheap to allocate.
SCRATCH_BYTES = 64 * 1024 * 1024  # 64 MiB


def _make_scratch_file(path, size_bytes):
    """Create a sparse file of an exact size to back a virtio-block device."""
    with open(path, "wb") as f:
        f.truncate(size_bytes)
    assert os.path.getsize(path) == size_bytes
    return path


def test_second_drive_appears_as_vdb(uvm_plain):
    """
    Attach a second drive before boot and assert the guest sees it as /dev/vdb
    with the expected size. Proves the virtio-block path end to end:
    PUT /drives -> DeviceManager -> virtio-mmio -> guest /dev/vdb.
    """
    vm = uvm_plain

    # 1. Bring the VMM up (process + API socket), but do not boot yet.
    vm.spawn()

    # 2. Root device + kernel + machine-config in one shot. rootfs is /dev/vda.
    vm.basic_config()

    # 3. Create the host backing file and attach it as a SECOND drive.
    scratch_host_path = _make_scratch_file(
        os.path.join(vm.path, "scratch.ext4"), SCRATCH_BYTES
    )
    vm.add_drive(
        drive_id="scratch",
        path_on_host=scratch_host_path,
        is_root_device=False,
        is_read_only=False,
    )

    # 4. A NIC so we can SSH in to assert from inside the guest.
    vm.add_net_iface()

    # 5. Boot.
    vm.start()

    # 6. Assert from INSIDE the guest. The second virtio-block device is vdb.
    exit_code, stdout, _ = vm.ssh.run("test -b /dev/vdb && echo OK")
    assert exit_code == 0, "guest does not see a second block device /dev/vdb"
    assert "OK" in stdout

    # 7. Size check: /sys reports the device size in 512-byte sectors.
    exit_code, stdout, _ = vm.ssh.run("cat /sys/block/vdb/size")
    assert exit_code == 0
    sectors = int(stdout.strip())
    assert sectors * 512 == SCRATCH_BYTES, (
        f"vdb size mismatch: {sectors} sectors "
        f"({sectors * 512} bytes) != {SCRATCH_BYTES} bytes"
    )


def test_second_drive_read_only_is_enforced(uvm_plain):
    """A drive added with is_read_only=True must be read-only in the guest."""
    vm = uvm_plain
    vm.spawn()
    vm.basic_config()

    ro_path = _make_scratch_file(os.path.join(vm.path, "ro.ext4"), SCRATCH_BYTES)
    vm.add_drive(
        drive_id="ro_scratch",
        path_on_host=ro_path,
        is_root_device=False,
        is_read_only=True,
    )
    vm.add_net_iface()
    vm.start()

    # /sys/block/<dev>/ro == 1 means the guest sees it read-only.
    exit_code, stdout, _ = vm.ssh.run("cat /sys/block/vdb/ro")
    assert exit_code == 0
    assert stdout.strip() == "1", "drive added is_read_only=True is writable in guest"

    # And a write must fail.
    exit_code, _, _ = vm.ssh.run("dd if=/dev/zero of=/dev/vdb bs=512 count=1 2>/dev/null")
    assert exit_code != 0, "write to a read-only drive unexpectedly succeeded"

A few deliberate choices, each defensible at review:

Tip: If your branch's uvm_plain is already booted, drop spawn()/basic_config()/start() and use the factory directly (vm = microvm_factory.build(guest_kernel, rootfs)). rg -n "def uvm_plain|def uvm\b" tests/conftest.py to see exactly what you're handed.

Step 4: Run it

# Collect first — confirm pytest sees both tests and the parametrization.
tools/devtool test -- integration_tests/functional/test_second_drive.py --collect-only -q

# Run it, verbose, stop on first failure.
tools/devtool test -- integration_tests/functional/test_second_drive.py -v -x

Expected (kernel tags depend on ALL_GUEST_KERNELS):

integration_tests/functional/test_second_drive.py::test_second_drive_appears_as_vdb[vmlinux-6.1.x] PASSED
integration_tests/functional/test_second_drive.py::test_second_drive_appears_as_vdb[vmlinux-5.10.x] PASSED
integration_tests/functional/test_second_drive.py::test_second_drive_read_only_is_enforced[vmlinux-6.1.x] PASSED
...

Step 5: Make it fail on purpose, then fix it

Prove the test detects the behavior, not just that it runs. Temporarily comment out the add_drive call and rerun:

tools/devtool test -- "integration_tests/functional/test_second_drive.py::test_second_drive_appears_as_vdb" -v -x

You should see test -b /dev/vdb fail — the guest has no second device. Read how the framework surfaces that: the assertion message, plus the attached Firecracker log and serial console output in the test's results dir. Restore add_drive and confirm green again.

# Find where the failure artifacts (log/serial/metrics) are saved for inspection.
rg -n "results_dir|log_file|console|serial|metrics" tests/framework/microvm.py | head

Step 6: Style gate and prepare the PR

# Python style for the test (black/isort/mdformat are part of the gate).
tools/devtool fmt
tools/devtool checkstyle

Then a signed-off commit. A feature PR would pair this with the production change; here the test stands alone:

git checkout -b test/second-drive-vdb-integration
git add tests/integration_tests/functional/test_second_drive.py
git commit -s -m "test: assert a second drive appears as /dev/vdb with correct size"

Deliverables

• A new pytest under integration_tests/functional/ that boots a microVM and asserts a guest-visible behavior.
• It uses the Microvm helper (spawn/basic_config/add_drive/add_net_iface/start) and ssh.run(...) to assert from inside the guest.
• It passes under tools/devtool test across the parametrized kernel matrix.
• You made it fail (removed add_drive), read the failure from the captured log/serial, and restored it to green.
• tools/devtool checkstyle is clean; the commit is signed off.

Troubleshooting

ssh.run times out — the guest never came up

Read the serial output (rg -n "serial\|console" tests/framework/microvm.py for where it's saved). If the kernel didn't boot, the problem is config order — add_drive/add_net_iface must precede start(). If it booted but SSH fails, the NIC isn't up.

/dev/vdb exists but the size is wrong

Block-device size is reported in 512-byte sectors at /sys/block/vdb/size. Multiply by 512 before comparing to bytes. If it's off by the rootfs, you asserted on vda, not vdb.

add_drive raises / API returns 400

rg -n "def add_drive" -A8 tests/framework/microvm.py and match the exact kwargs your branch expects. A second is_root_device=True drive, or a duplicate drive_id, is rejected by the API.

Passes on one kernel, fails on another

Older guests enumerate or name devices differently, or lack a driver. Read which kernel's case is red in the node id, then SSH-debug that one (ssh.run("dmesg | grep -i virtio")).

"address already in use" / leaked process on rerun

A prior failed run leaked a firecracker. pgrep -af firecracker and kill it; the factory normally reaps, but a hard crash can leak. This is the topic of Lab 5.4.

Expected Output

$ tools/devtool test -- integration_tests/functional/test_second_drive.py -v
collected N items
...::test_second_drive_appears_as_vdb[vmlinux-6.1.x] PASSED
...::test_second_drive_read_only_is_enforced[vmlinux-6.1.x] PASSED
N passed in Xs

Stretch Goals

1. Write known bytes into the host scratch file before attaching it, then dd them back out inside the guest and assert they match — proving the data path, not just enumeration.
2. Add a @pytest.mark.parametrize("io_engine", ["Sync", "Async"]) (or use the existing io_engine fixture) and confirm both block I/O engines pass.
3. Extend to a PATCH /drives/{id} (rate-limiter) update at runtime and assert the new limit takes effect (rg -n "patch_drive" tests/framework/microvm.py).
4. Convert the size assertion to a helper in tests/framework/ so the next drive test reuses it — a small, real test-refactor PR.

Validation / Self-check

Answer without notes; these gate completion:

1. Why does this test assert from inside the guest instead of trusting the API's 2xx response?
2. In what order must add_drive, add_net_iface, and start be called, and why?
3. How does the guest report a block device's size, and what unit is it in?
4. Why is the scratch backing file created under vm.path rather than a fixed /tmp name?
5. What did making the test fail (Step 5) prove that a passing run alone does not?
6. Why is "one behavior per test" worth splitting the read-only check into its own function?
7. Which deep dive explains what actually happens between PUT /drives and /dev/vdb appearing?

Next: Lab 5.4 — Fix It: A Flaky Test, where you take a test that passes most of the time and make it pass every time — for the right reason.