Project 3: A Custom Rate-Limiting Policy

Firecracker throttles guest I/O with a token-bucket rate limiter. Each network and block device can carry two buckets — one for operations per second (IOPS / packets) and one for bandwidth (bytes) — each parameterized by size (bucket capacity), one_time_burst (an initial allowance), and refill_time (how long to refill the bucket). When a device wants to do I/O, it asks its limiter for budget; if the bucket is empty, the operation is deferred until a timer refills it. This is the mechanism that lets a single host run thousands of mutually-untrusting microVMs without one noisy guest starving the others.

The shipped policy is deliberately simple: independent per-device buckets, a fixed refill, no notion of fairness across devices or across microVMs, no dynamic adjustment to load. That simplicity is a feature — but it is also a place where a sharp contributor can add real value. This project asks you to design, implement, and benchmark a custom or extended rate-limiting policy: a new bucket scheme, a fairness mechanism across devices, or a load-adaptive (dynamic) limiter — and to prove with numbers that it does what you claim without regressing the simple case.

Note: Read the rate-limiting token-bucket deep dive and do the networking masterclass, especially Lab 2: rate limiting. This brief assumes you know the TokenBucket math, the two-bucket (ops + bandwidth) structure, how a device consumes and is throttled, and how the limiter is driven by a timer fd on the EventManager loop. It will not re-derive the token bucket.

Problem & motivation

The token bucket is the right primitive, but the policy layered on it is minimal, and three gaps are real and reachable:

• No cross-device fairness. A microVM with two block devices and a net device has three independent limiters. There is no way to say "this microVM gets X total IOPS, shared fairly across its devices." Under contention, allocation between a guest's own devices is whatever the arrival order happens to produce, not a fairness policy.
• The limiter is static. size/refill_time are set at configure time (and can be PATCHed). There is no mechanism for the limiter to adapt — to give a bursty device more headroom when the host is idle and clamp it when the host is busy. Every serverless platform eventually wants some form of this (a "burst credit" or a load-aware ceiling).
• The bucket scheme is one shape. A single bucket with a linear refill is the classic shape, but there are others — hierarchical buckets (a parent ceiling over child devices), a hierarchical-token-bucket-style borrow scheme, a deficit-round-robin fairness layer — each with different burst and fairness properties.

The motivation is leverage at density: the limiter is the contract that makes oversubscription safe (see oversubscription and density). A better policy directly improves how many guests you can pack without tail-latency collapse — and it is a self-contained, well-tested module (src/vmm/src/rate_limiter/) that you can extend without touching the device internals much.

What you'll build

Pick one policy direction and build it to maintainer quality with benchmarks.

Whichever you choose, the artifact is: the new policy as a clean module behind the existing limiter interface, unit tests proving the token math, an integration test proving end-to-end throttling, and a benchmark comparing it to the shipped limiter on a contention workload.

Prerequisites

• Level 7 (the device model) and Level 9 Lab 3: perf regression.
• Rate-limiting token-bucket deep dive.
• Networking masterclass and its rate-limiting lab.
• Deep dives: virtio-net-and-tap, virtio-block, the-event-manager (the timer-fd that drives refills).
• Engineering essays oversubscription and density and io-engines.

Phased plan

Phase 0 — Build it and map the limiter (1 day)

tools/devtool build --release
# The limiter module — read it cover to cover:
rg -n "struct TokenBucket|fn reduce|fn auto_replenish|RateLimiter|BucketReduction|refill_time" src/vmm/src/rate_limiter/
# How a device consumes budget and is deferred when empty:
rg -n "rate_limiter|RateLimiter|consume|throttled|process_rx|process_tx" src/vmm/src/devices/virtio/net/
rg -n "rate_limiter|consume|deferred|process_queue" src/vmm/src/devices/virtio/block/
# How the refill timer fd is wired into the event loop:
rg -n "TimerFd|timerfd|EventManager|process|register" src/vmm/src/rate_limiter/
# The API surface that configures/patches limiters:
rg -n "rate_limiter|RateLimiterConfig|TokenBucketConfig|PATCH" src/firecracker/src/api_server/ src/vmm/src/vmm_config/

Produce capstone-work/limiter-path.md: a trace from PUT/PATCH config → RateLimiterConfig → TokenBucket construction → a device asking for budget → reduce/throttle → timer-fd refill → resume. With file references found by rg.

Anti-staleness: the limiter has been refactored (auto-replenish vs. timer-driven refill, the exact reduce/BucketReduction API). Confirm the current shape on your branch before designing against it, and check git log --oneline -- src/vmm/src/rate_limiter/ for recent changes.

Phase 1 — Reproduce the gap as a test (the framing)

Before building, demonstrate the gap. Write a test (unit or integration) that shows the shipped limiter's limitation for your chosen direction:

• Direction A: two block devices on one microVM, a shared ceiling you want — show the current limiters cannot express it (one device can consume the budget you meant to share).
• Direction B: a bursty workload against a static limiter — show it either wastes headroom when idle or clamps too hard under burst, with no adaptation.
• Direction C: a child that needs to borrow idle parent budget — show the flat scheme cannot.

This failing/illustrative test is your problem-articulation artifact and the baseline your benchmark improves on.

Phase 2 — Implement the policy behind the existing interface

The key design rule: the device code should not need to know which policy it uses. Devices call a limiter interface (consume/reduce/"may I do N bytes / M ops?"). Your new policy implements that interface; the simple bucket remains the default.

#![allow(unused)]
fn main() {
// Sketch — a fairness layer that sits over per-device buckets. rg the real interface first.
struct FairLimiter {
    parent: TokenBucket,             // the shared ceiling for the microVM
    children: Vec<DeviceShare>,      // weighted shares per device
    deficits: Vec<u64>,              // deficit-round-robin counters
}

impl FairLimiter {
    fn consume(&mut self, device: usize, tokens: u64) -> Consumption {
        // 1. check the parent ceiling
        // 2. apply the per-device deficit/weight so no device starves others
        // 3. return Granted | Throttled(retry_after)
    }
}
}

For Direction B (dynamic), the adaptation belongs in how/when the bucket replenishes — e.g. accrue burst credit while idle, or scale refill_time from a utilization EWMA. Keep the signal and the response separable so you can tune them.

Tip: Resist putting policy in the device. The reason the current design is clean is that the device just asks for budget. A fairness or adaptive policy that leaks into process_rx/process_queue is a code smell reviewers will reject. Keep it in the limiter module.

Milestone: the new policy passes the Phase-1 illustrative test, and the default (simple bucket) behavior is byte-for-byte unchanged when the new policy is not configured.

Phase 3 — Wire it through config and snapshots

Expose the new policy through the API (a new config field on the device's rate-limiter config, or a new /machine-config-level knob — decide and justify), and make sure it survives snapshot/restore. Limiter state (current bucket level, deficits) is part of the device's persisted state in some versions; confirm and handle it.

rg -n "RateLimiterConfig|TokenBucketConfig|deserialize|impl Persist" src/vmm/src/rate_limiter/ src/vmm/src/vmm_config/
rg -n "rate_limiter.*persist|save_state|restore" src/vmm/src/devices/virtio/net/ src/vmm/src/devices/virtio/block/

If the policy adds config, it adds API surface — which is a backward-compatibility commitment. Design it as optional and defaulting to today's behavior.

Phase 4 — Benchmark under contention

The headline. The whole claim ("fairer" / "adapts" / "borrows") is meaningless without a contention benchmark. Construct a workload where the limitation bites and measure the shipped limiter vs. yours:

scenario: 1 microVM, 2 block devices, shared 10k IOPS ceiling, dev A greedy / dev B steady
limiter            dev A IOPS   dev B IOPS   fairness (Jain)   aggregate IOPS
shipped (independent)  ~10k         ~10k          —              ~20k (over ceiling)
shipped (split 5k/5k)  ~5k (idle B wastes 5k)                     ~5k–10k
yours (fair shared)    ~adaptive    ~protected   ~0.9x           ~10k (at ceiling)

For Direction B, measure burst absorption and idle-headroom reclaim over time (a time series, not a single number). Run on a real microVM with fio (block) or iperf3 (net) in the guest, ≥10 runs, host/kernel/CPU stated, and characterize the noise.

Key code areas

Design considerations & trade-offs

• Don't break the simple case. The default, single-bucket behavior must be bit-identical when your policy is not configured. Add, don't replace. This is the difference between a mergeable PR and a rejected one.
• Fairness has a definition — pick one. Max-min fairness, weighted shares, deficit round-robin — they behave differently under bursts. State which you implement and measure it (Jain's fairness index is a clean metric).
• Adaptation can oscillate. A dynamic limiter that reacts too fast to a noisy signal will hunt. Use an EWMA or a hysteresis band, and show stability in the time series.
• The limiter is on the data path. It runs per I/O on the VMM thread. A fairness computation that is O(devices) per packet is fine; anything heavier needs care. Measure the per-op overhead and show the simple case did not regress.
• API and snapshot are forever. Any config you add is a compatibility commitment; any state you persist is a snapshot-format commitment. Optional, defaulted, and versioned.
• Cross-microVM fairness is out of scope for Firecracker. One process is one microVM; the host (cgroups, the jailer's --resource-limit) handles inter-microVM fairness. Keep your policy within a microVM unless you have a very good reason — and if you don't, say so in the design note.

How to test & validate

• Unit tests (cargo test): the token math is the most testable thing in the codebase. Test refill timing, burst exhaustion, the fairness distribution under a fixed arrival sequence, and the adaptation response to a synthetic signal. Use a controllable clock so the tests are deterministic.

  rg -n "#\[test\]|fn test_|fake.*clock|advance" src/vmm/src/rate_limiter/
  tools/devtool test -- -k rate_limiter   # or cargo test in the module
  

• Integration test (pytest): boot a microVM with two devices, drive fio/iperf3 in the guest, and assert the achieved rates match the policy (and that the default limiter still produces today's numbers). New functionality requires this.

• Benchmark: scripted, repeatable, ≥10 runs, fairness index and aggregate throughput reported, host characterized.

• The gates: tools/devtool fmt, checkstyle, checkbuild --all (clippy is warnings-as-errors).

Stretch goals

• A PATCH-able adaptive policy where the target rate can change at runtime (Firecracker already supports PATCH /drives/{id} and PATCH /network-interfaces/{id} for limiters — extend it).
• A hierarchical bucket where a microVM-level ceiling sits over per-device buckets, with borrowing, and show idle-headroom reclaim.
• Expose limiter telemetry as a metric (faults-deferred, budget-utilization) via the metrics subsystem.
• Tie into Project 6: show the fairness policy's effect on tail latency across many microVMs under contention.

What a strong deliverable looks like

A strong deliverable is a new rate-limiting policy implemented cleanly behind the existing limiter interface, with the simple case provably unchanged, unit tests for the token/fairness math, a pytest integration test, and a contention benchmark with a fairness metric and aggregate throughput, host characterized — plus a design note that names the fairness definition or adaptation signal and the alternatives rejected.

The upstreaming path:

1. RFC first for anything non-trivial. A new fairness or adaptive policy changes behavior under contention and adds API/snapshot surface — open the design note as a GitHub issue and negotiate scope. Search the live state: gh issue list --repo firecracker-microvm/firecracker --search "rate limit OR token bucket OR fairness".
2. A scoped bucket-scheme improvement (e.g. a documented new burst behavior, or a bug in the refill math) may be mergeable directly — find a real issue first.
3. Bring the benchmark. The maintainers will not accept a fairness claim without numbers. Arrive with the table in the PR description and the harness in the repo.

Even if the policy stays local, a benchmarked fairness/adaptive limiter with a write-up is a portfolio-grade artifact in the subsystem that makes oversubscription safe. A finished version at 90+ on the rubric is maintainer-grade I/O-policy work.

Next: Project 4 — a rust-vmm upstream contribution for the cleanest path to a real merge, or Project 6 — the boot/density benchmark harness to build the measurement muscle this project leans on.