Stage 2 — Build, Tooling, and Logging

What class of issue this is

Stage 2 is the second on-ramp rung: changes to the build/test tooling, the style gates, and the logging/metrics surface. None of it changes how a microVM behaves at runtime, but all of it is real, reviewed, merged work that teaches you the machinery every later PR runs through.

Concretely, a Stage 2 PR is one of:

• A tools/devtool tweak, a CI/Buildkite/GitHub-Actions fix, or a build-script (tools/release.sh, tools/*.py) correction.
• A clippy or rustfmt nit — silencing a lint properly (not blanket-#[allow]), fixing a formatting drift, or correcting a cargo sort / import-ordering issue that the gate flags.
• A logging improvement: a log message at the wrong level, missing context, a {:?} where a human-readable message belongs, or a new metric counter for an event that is currently invisible.

Why it's at this difficulty

The blast radius is the developer experience and observability, not guest correctness. A wrong log level annoys an operator; it does not corrupt state. So the bar is can you make the change without breaking the gate that enforces it — which means you have to understand the gates themselves, the thing Stage 1 only made you run. Maps to Level 1 and Level 2; pairs with the logging & metrics deep dive.

What you must already understand

• The whole Stage 1 workflow (it is assumed from here on and never re-explained).
• The style gates and what each one checks:

tools/devtool fmt          # cargo fmt + clippy --fix + cargo sort + black/isort/mdformat
tools/devtool checkstyle   # the read-only style verification CI runs
tools/devtool checkbuild --all   # build all targets/arches/features the way CI does

• Clippy is warnings-as-errors. The exact invocation CI uses:

cargo clippy --all --all-targets --all-features -- -D warnings

• Firecracker's logging is its own in-tree subsystem, not env_logger. Find it before you touch it:

rg -n "macro_rules! (error|warn|info|debug)|struct Logger|impl Log " src/vmm/src/logger/
rg -n "METRICS\.|IncMetric|SharedIncMetric|struct .*Metrics" src/vmm/src/logger/metrics.rs

The metrics are a fixed set of counters (METRICS.<subsystem>.<event>.inc()), serialized to the metrics FIFO. Adding a counter means adding a field to the right metrics struct, not inventing an ad-hoc log line.

Representative tasks

How to approach a clippy nit

Illustrative of the pattern.

Symptom: a new compiler/clippy version flags an idiom CI now rejects. First reproduce exactly what CI sees — the toolchain is pinned in rust-toolchain.toml, so use it:

rg -n 'channel' rust-toolchain.toml           # the effective MSRV / pinned channel
cargo clippy --all --all-targets --all-features -- -D warnings 2>&1 | head -40

Fix the cause, not the symptom. tools/devtool fmt runs clippy --fix for the mechanical ones; for a real lint, change the code idiomatically (e.g. Vec::new() → Vec::with_capacity, &Vec<T> → &[T], redundant clone removed). A blanket #[allow(clippy::...)] is almost never the right fix and reviewers will push back — reserve #[allow] for cases with a // reason: comment.

How to approach a logging improvement

Illustrative. Run the grep to find a real candidate.

Symptom: when a device action fails, the log line is error!("{:?}", e) — no context about which device or what the operator should do. Logging exists to make a failing microVM diagnosable.

rg -n 'error!\("\{:\?\}"|warn!\("\{:\?\}"' src/vmm/src/devices/ | head
git log --oneline -n 5 -- src/vmm/src/logger/

--- a/src/vmm/src/devices/virtio/block/device.rs
+++ b/src/vmm/src/devices/virtio/block/device.rs
@@
-        if let Err(e) = self.process_queue(0) {
-            error!("{:?}", e);
-        }
+        if let Err(e) = self.process_queue(0) {
+            // Include the device id and queue so an operator can attribute the failure.
+            error!("block: {}: failed to process request queue: {e}", self.id());
+        }

Two rules for logging diffs:

1. Level reflects severity. error! = the microVM is degraded or an operation failed; warn! = recoverable / unexpected-but-handled; info! = lifecycle milestones; debug! = developer tracing. Do not log at error! for something the code handles cleanly.
2. No guest-controlled data at scale, no PII, no secrets. A guest can drive a code path thousands of times per second — logging guest-influenced strings on a hot path is both a log-spam DoS and an information leak. When in doubt, increment a metric instead of logging.

For the metric path, add the counter where the metrics live and .inc() at the call site:

rg -n "pub struct BlockDeviceMetrics|block:" src/vmm/src/logger/metrics.rs | head

#![allow(unused)]
fn main() {
// at the failure site, in addition to (or instead of) a log line:
METRICS.block.event_fails.inc();
}

Confirm the counter serializes by flushing metrics on a running microVM (PUT /actions {"action_type":"FlushMetrics"}) and reading the metrics FIFO.

What a good PR looks like

• One concern, and it is honest about its category. A clippy fix is a clippy fix; do not smuggle a behaviour change into it.
• All gates green locally before push: tools/devtool fmt, then tools/devtool checkstyle, then tools/devtool checkbuild --all. A Stage 2 PR failing the very gate it touches is an instant bounce.
• Logging changes justify the level and the content in the PR description, and never log guest-controlled data on a hot path.
• Metric additions add a field to the correct metrics struct and are reset/serialized like their neighbours — rg the struct and copy the existing pattern exactly.
• CHANGELOG entry when the change is operator-visible (a new metric, a changed log level a tool parses, a new devtool flag). Pure internal lint fixes usually need none — follow the PR template.

Graduation criteria — ready for Stage 3 when

• You have two merged PRs in this space (any mix of build/tooling, clippy/fmt, logging, metrics) that passed CI without a maintainer re-asking you to run the gates.
• You can explain, without looking, what tools/devtool fmt vs checkstyle vs checkbuild --all each do and which one CI fails on.
• You can locate the logger and metrics subsystems with rg and add a counter following the existing pattern.
• You know the rule of thumb: on a hot or guest-driven path, prefer a metric to a log line. This is the seam into Stage 3, where you make the errors themselves carry the context.

Next: Stage 3 — Error Messages and Diagnostics.