Lab 8.3: Improve Error Messages and Diagnostics

Background

When a Firecracker microVM fails to start, an operator on the other side of an orchestrator sees only what Firecracker chose to tell them. If that is Error: InvalidState or a bare errno 22, the operator cannot tell whether they passed a bad memory size, the host ran out of a KVM resource, the TAP device was missing, or a seccomp filter denied a syscall — and Firecracker, by design, has no interactive debugger and a deliberately tiny surface to inspect. The error message is the diagnostic interface. In a VMM that runs untrusted, multi-tenant workloads at scale, a good failure message is not a nicety; it is the difference between a five-minute fix and a multi-team incident.

Improving error messages and diagnostics is one of the highest-value, most accessible contribution classes in Firecracker. It is accessible because it rarely changes behavior — you are making an existing failure legible, not changing when it fails. It is high-value because every operator who ever hits that path benefits. And it teaches you the codebase's error architecture — the per-subsystem error enums, how they compose into VmmActionError, and how they reach the operator over the API and the log. This lab takes a place where Firecracker fails opaquely, improves the error type/message/ context, and adds a test that asserts the new message — because an unasserted message will silently rot back to uselessness on the next refactor.

Why This Lab Matters for Contributors

• It is a real, mergeable contribution that does not require touching the threat model, the device model, or snapshot compatibility — the three things that make first PRs spin.
• It forces you to learn Firecracker's error architecture: thiserror-style enums per subsystem, ?/From composition up to VmmActionError, and the API/log boundary where the message is rendered.
• It builds the maintainer's instinct for what an operator actually needs at the moment of failure — the same instinct you will apply to every future fix.
• It corresponds to issue-roadmap Stage 3: Error Messages, a deliberately early rung on the contribution ladder.

Prerequisites

• The fix/test/PR loop from Lab 8.2 — this lab produces the same PR artifacts (minimal diff, asserting test, CHANGELOG, sign-off).

• The API → action path from Level 3: you need to know how an error becomes a VmmActionError and then an HTTP response.

• Optionally the KVM and device material from Level 4 and Level 7, since the best targets are ioctl and device-setup failures.

• Confirm your toolchain:

  git switch -c diag/improve-error-message origin/main
  git config user.name; git config user.email     # for DCO sign-off later
  

What Makes a Good Operator-Facing Error in a VMM

Before you touch code, internalize the rubric. Every change in this lab is judged against it.

Note: The last two are in tension and both matter. The operator needs detail; the guest must never learn host internals through an error channel (the guest is untrusted — see the threat model framing). The API caller (the orchestrator) is trusted and may see more than the guest; the host log may see more than the API. Put the most sensitive context where only the host operator reads it.

The single most common defect: an error that drops the underlying cause. A handler catches a rich OS or KVM error and returns a flat enum variant with no payload. The fix is almost always to carry the source through with #[source]/#[from] and to interpolate the offending value into the message.

Firecracker's Error Architecture (orient first)

Errors in Firecracker are per-subsystem enums, usually thiserror-style, that compose upward. Map it on your branch before editing — names and paths are version-sensitive (verify):

# The top-level action error the API caller ultimately sees.
rg -n "enum VmmActionError" src/vmm/src/rpc_interface.rs

# Per-subsystem error enums (config, devices, builder, vstate).
rg -n "enum .*Error" src/vmm/src/vmm_config/
rg -n "enum .*Error" src/vmm/src/devices/virtio/ | head
rg -n "enum .*Error" src/vmm/src/builder.rs src/vmm/src/vstate/

# The thiserror idiom: #[error("...")] messages and #[from]/#[source] composition.
rg -n '#\[error\(' src/vmm/src/ | head -20
rg -n '#\[from\]|#\[source\]' src/vmm/src/ | head -20

# Where errors become the HTTP response body (the api_server side).
rg -n "fault_message\|impl .*for VmmActionError\|fn to_response\|ErrorResponse" src/firecracker/src/

flowchart LR
    OS["OS / KVM error<br/>(errno, ioctl failure)"] --> SE["subsystem Error enum<br/>(#[source] carries the cause)"]
    SE --> AE["VmmActionError<br/>(#[from] composes it up)"]
    AE --> H["api_server renders<br/>{ fault_message }"]
    AE --> L["the log<br/>(richer host-side context)"]
    H --> O["operator / orchestrator<br/>sees the message"]

The lesson: a good message is built at the subsystem layer (where the context exists) and must survive the trip up to VmmActionError and out to the operator. If a layer flattens it, that is your bug.

Step-by-Step Tasks

Step 1 — Find an opaque failure (20 min)

Pick one of the high-value target classes. Each is a real place Firecracker can fail unhelpfully:

Grep for the candidates — flat variants with no payload, and discarded sources:

# Error variants that carry NO context (no fields) — prime candidates.
rg -n 'enum .*Error' -A30 src/vmm/src/vmm_config/ | rg -n '^\s+[A-Z][A-Za-z]+,'

# Places that map a rich error into a flat one (losing the cause).
rg -n "map_err\(\|_\| \|\.map_err\(|\.ok\(\)|let _ =" src/vmm/src/ | rg -i "error\|kvm\|ioctl" | head

# Bare debug-formatted errors reaching the user (a smell).
rg -n 'format!\("\{:\?\}"|{:?}", .*err' src/vmm/src/ | head

Pick one concrete failure and reproduce its current message the way an operator would. For a config example, drive the API and capture the body (as in Lab 8.1); for a KVM or device example, trigger it by hand (e.g. point a drive at a missing path, or a network iface at a non-existent host_dev_name) and capture the log:

API=/tmp/fc-diag.socket; rm -f "$API"
sudo ./build/cargo_target/x86_64-unknown-linux-musl/debug/firecracker \
  --api-sock "$API" --level Debug --log-path /tmp/fc-diag.log &
# Trigger the failure (example: a TAP device that does not exist).
curl -s -i -X PUT --unix-socket "$API" --data \
  '{"iface_id":"net1","guest_mac":"06:00:AC:10:00:02","host_dev_name":"does-not-exist0"}' \
  http://localhost/network-interfaces/net1
rg -n "error\|fail\|net1\|does-not-exist0" /tmp/fc-diag.log

Record the current message verbatim. That is your "before."

Step 2 — Diagnose what context is missing (10 min)

Hold the current message against the rubric. For each missing property, note where the context exists in the code:

The most common finding: the offending value and the underlying #[source] are both in scope at the failure site but neither reaches the message. That is exactly what you will fix.

Step 3 — Improve the error type, message, and context (30 min)

Edit at the subsystem layer where the context lives. Three idiomatic moves, in order of value:

(a) Carry the offending value in the variant and message.

#![allow(unused)]
fn main() {
// Before: a flat variant, no context.
#[derive(Debug, thiserror::Error)]
pub enum NetworkInterfaceError {
    #[error("Could not create the network device.")]
    CreateNetworkDevice,
}

// After: name the device, the host resource, and the cause.
#[derive(Debug, thiserror::Error)]
pub enum NetworkInterfaceError {
    #[error("Failed to open host TAP device \"{host_dev_name}\" for iface \"{iface_id}\": {source}")]
    OpenTap {
        iface_id: String,
        host_dev_name: String,
        #[source]
        source: TapError,   // the real cause, preserved
    },
}
}

(b) Preserve the underlying cause instead of discarding it.

# Find the spot that throws away the real error.
rg -n "map_err\(|_| \|\.map_err\(\|_e\|" src/vmm/src/devices/ | head

#![allow(unused)]
fn main() {
// Before: the cause is erased.
Tap::open_named(name).map_err(|_| NetworkInterfaceError::CreateNetworkDevice)?;

// After: the cause is carried, and the message names the inputs.
Tap::open_named(name).map_err(|source| NetworkInterfaceError::OpenTap {
    iface_id: cfg.iface_id.clone(),
    host_dev_name: name.to_string(),
    source,
})?;
}

(c) Make sure it survives up to VmmActionError. If VmmActionError composes subsystem errors with #[from], your richer message rides along automatically; confirm there is no lossy to_string() or {:?} flattening on the way out:

rg -n "impl From<.*Error> for VmmActionError\|#\[from\]" src/vmm/src/rpc_interface.rs
rg -n "fault_message\|to_string\(\)|{:?}" src/firecracker/src/api_server/ | head

Warning: Keep host-sensitive detail (absolute host paths, raw fds, host process internals) out of the API response body — that channel can be observed by a less-trusted party. Put the full detail in the log (the host-only channel) and a safe, useful summary in the API fault_message. Read how the response is rendered before deciding where each piece of context goes.

Note (version-sensitive): TapError, NetworkInterfaceError, and the exact #[error] strings in these snippets are illustrative — the real type and variant names drift between branches. Use the rg from the architecture section to find the actual enum and adapt. The technique — carry the value, carry the source, render safely — is what you are learning.

Step 4 — Add a test that asserts the new message (20 min)

An unasserted message rots. Pin it with a test so a future refactor that flattens it fails CI. Match the project's idiom — most subsystem errors are tested with a small #[cfg(test)] unit test that formats the error and asserts on the string:

rg -n "#\[cfg\(test\)\]\|fn test_.*error\|\.to_string\(\).*contains\|assert.*Display" \
  src/vmm/src/devices/virtio/net/ | head

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_open_tap_error_names_device_and_cause() {
        let err = NetworkInterfaceError::OpenTap {
            iface_id: "net1".to_string(),
            host_dev_name: "does-not-exist0".to_string(),
            source: TapError::OpenTun(std::io::Error::from_raw_os_error(libc::ENODEV)),
        };
        let msg = err.to_string();
        // The operator must see the iface, the host device, and that there IS a cause.
        assert!(msg.contains("net1"), "missing iface id: {msg}");
        assert!(msg.contains("does-not-exist0"), "missing host_dev_name: {msg}");
        assert!(msg.to_lowercase().contains("tap"), "missing operation context: {msg}");
    }
}
}

For an API-level guarantee (the operator-visible body), add a small pytest integration test asserting the fault_message contains the actionable phrase — this is the contract that actually matters to an orchestrator, and CONTRIBUTING wants integration coverage for behavior:

# tests/integration_tests/functional/test_net_error_message.py
def test_missing_tap_error_is_actionable(uvm_plain):
    vm = uvm_plain
    vm.spawn()
    resp = vm.api.network.put(
        iface_id="net1", guest_mac="06:00:AC:10:00:02", host_dev_name="does-not-exist0"
    )
    assert resp.status_code == 400, resp.text
    body = resp.text.lower()
    assert "tap" in body and "does-not-exist0" in body, resp.text

Confirm fails-then-passes: with your change reverted the assertions on the new phrasing fail; with it applied they pass.

cargo test -p vmm test_open_tap_error_names_device_and_cause
tools/devtool test -- integration_tests/functional/test_net_error_message.py

Step 5 — Run the gate, CHANGELOG, sign-off, PR (15 min)

Same closing discipline as Lab 8.2:

tools/devtool fmt
tools/devtool checkstyle
tools/devtool checkbuild --all      # clippy is warnings-as-errors
tools/devtool test

CHANGELOG — a diagnostics improvement is usually ### Changed (the message changed) or ### Fixed if it corrected a misleading one:

### Changed

- [#NNNN](https://github.com/firecracker-microvm/firecracker/pull/NNNN): Network interface setup now
  reports the failing iface id, the host TAP device name, and the underlying cause when opening the
  host device fails, instead of a generic "could not create the network device" error.

Commit with sign-off and open the PR (template, license-acceptance line, linked issue) exactly as in Lab 8.2 Step 7–8:

git add -A
git commit -s -m "diag: include iface id, tap name, and cause in net setup error" \
  -m "Network setup previously returned a generic error on TAP open failure. Carry the iface id, host device name, and source error so operators can act. Adds a unit test and an integration test asserting the message. Fixes #NNNNN"
git push -u origin diag/improve-error-message
gh pr create --repo firecracker-microvm/firecracker --base main --fill

Tip: In the PR description, paste the before and after messages side by side. Reviewers approve diagnostics PRs fast when they can see the improvement in two lines without checking out the branch.

Implementation Requirements

• A chosen opaque failure with its current message captured verbatim (the "before").
• A rubric-driven note of which properties were missing and where the context lived.
• An improved error: the offending value in the message, the underlying cause preserved via #[source]/#[from], host-sensitive detail kept out of the operator-visible channel.
• A test asserting the new message (unit), plus an integration test for the API-visible contract.
• Fails-then-passes confirmed.
• Green tools/devtool checkstyle + checkbuild --all + test.
• CHANGELOG entry, DCO-signed commit, PR with before/after messages.

Expected Output

Before (captured in Step 1):

{ "fault_message": "Could not create the network device." }

After (rendered to the operator):

{ "fault_message": "Failed to open host TAP device \"does-not-exist0\" for iface \"net1\": No such device (os error 19)" }

And the asserting test green, red when the fix is reverted:

test devices::virtio::net::tests::test_open_tap_error_names_device_and_cause ... ok

Troubleshooting

The richer message doesn't reach the API caller

A layer between the subsystem error and VmmActionError is flattening it (a to_string(), a {:?}, or a map_err(|_| ...)). Trace the path with the rg from the architecture section and find where the context is dropped; that lossy hop is part of your fix.

Clippy complains about the new error fields

Often clippy::result_large_err (the enum grew) or an unused-field lint. Box a large source if needed, or restructure the variant. Justify any #[allow] in the PR; prefer fixing the lint.

The test asserts a substring that changes when someone reformats the message

Assert the stable, load-bearing parts — the iface id, the host device name, the word "tap" — not the exact punctuation or wording. The test guards that the context is present, not the prose.

The error now leaks a host path into the API response

You put host-sensitive detail in the operator-visible channel. Move it to the log (host-only) and keep a safe summary in fault_message. Re-read how the response body is built before deciding the split.

#[from] causes an ambiguous conversion

Two variants both #[from] the same source type. Drop #[from] on one and convert explicitly with map_err, or wrap one source in a distinct newtype.

Stretch Goals

1. Audit a whole subsystem's error enum: list every variant that carries no context, and propose a minimal PR that enriches the worst three. (Keep it to one logical change — one enum.)
2. Find a place where an error is logged and returned, with the two messages out of sync, and unify them.
3. Add a Debug-vs-Display review: ensure operator-facing rendering uses Display ({}/to_string()), and that no {:?} of an error reaches the API body.
4. Trace one ioctl failure (e.g. KVM_SET_USER_MEMORY_REGION returning EINVAL) from the vCPU run loop deep dive and make its message name the ioctl and the errno.
5. Compare your before/after against how a recently merged diagnostics PR did it: gh pr list --repo firecracker-microvm/firecracker --state merged --search "error message".

Validation / Self-check

Answer without notes. These gate completion:

1. Which rubric properties did the original message lack, and where in the code did the missing context already exist?
2. Did you preserve the underlying cause with #[source]/#[from], or did you reword a flat variant? (The former is the real fix.)
3. How did you keep host-sensitive detail out of the operator-visible channel while still making the message actionable?
4. Does your test fail when the fix is reverted, and does it assert the stable parts of the message rather than the punctuation?
5. Why is improving an error message a behavior-preserving change, and why does that make it a strong first-PR class?
6. Could an on-call operator act on your new message without reading the source?

Next: you have completed Level 8 — return to the Level 8 index to confirm the deliverables, then advance to Level 9. This contribution class maps onto issue-roadmap Stage 3: Error Messages; the full graded contribution cycle is the Capstone.