The Rate Limiter and Token Bucket

A microVM that can read its disk and flood its network at full host speed is a problem when thousands of them share one machine: one noisy tenant starves the rest. Firecracker's answer is a rate limiter — a token-bucket throttle wired into the virtio block and net devices that bounds both operations per second and bandwidth. It is small, self-contained code (src/vmm/src/rate_limiter/), but it sits directly on the device fast path, so understanding it is non-negotiable if you touch block or net. This chapter dissects the token-bucket math, the two-bucket-per-direction model, the size / one_time_burst / refill_time parameters, how an empty bucket pauses virtqueue processing, and how a timer eventfd resumes it — all without ever blocking a thread.

After this chapter you will be able to: compute the token state of a bucket at any instant; explain why there are two buckets per limiter and what each throttles; trace how an exhausted limiter stops and later restarts queue processing through the EventManager; and reason about why this is a non-blocking design.

Note: The rate limiter is a throttle on processing, not a queue or a buffer. When out of tokens it does not drop or store I/O — it stops pulling from the virtqueue and arms a timer. The guest's requests pile up in guest memory (the virtqueue) until tokens refill. This is the key mental model: backpressure, not buffering.

Where the rate limiter lives

# The rate limiter module — token bucket + the RateLimiter that owns two of them.
rg -n "struct TokenBucket|struct RateLimiter|fn consume|fn replenish|fn new\b|fn auto_replenish" \
  src/vmm/src/rate_limiter/
# Where block and net construct and consult their limiters.
rg -n "RateLimiter|rate_limiter|TokenType|consume|is_blocked|BUCKET" \
  src/vmm/src/devices/virtio/block/ src/vmm/src/devices/virtio/net/

A RateLimiter (verify the name on your branch) wraps two TokenBuckets — one counting operations, one counting bytes — plus a TimerFd (an EventFd-backed timer) used to wake the device when tokens are due. The device asks the limiter "may I do this operation of N bytes?"; the limiter consults both buckets and answers yes or no.

The token bucket

A token bucket is the classic throughput regulator. Picture a bucket that holds up to size tokens and refills at a constant rate. Every unit of work removes tokens; if the bucket is empty, the work must wait until enough tokens have dripped back in.

        refill at  size / refill_time  tokens per unit time
                         │
                         ▼
                 ┌───────────────┐  capacity = size (+ one_time_burst initially)
                 │ ● ● ● ● ●     │
                 │ ● ● ●         │  ◄── current token count
                 └───────┬───────┘
                         │ consume(n): if tokens >= n, take n → allow
                         ▼          else → block, arm timer for the deficit
                    work proceeds

Firecracker's bucket does not run a background thread topping itself up. It refills lazily: every time the device calls consume, the bucket computes how many tokens should have arrived since it was last touched — elapsed_time * size / refill_time — adds them (capped at size), then checks whether the request fits.

# The lazy refill math — read it; this is the heart of the device.
rg -n "fn consume|fn auto_replenish|last_update|Instant|elapsed|refill_time|processed_capacity" \
  src/vmm/src/rate_limiter/

#![allow(unused)]
fn main() {
// Conceptual shape (read the real consume()/replenish() with the rg above):
//   now            = Instant::now();
//   elapsed        = now - self.last_update;
//   new_tokens     = elapsed.as_nanos() * size / refill_time_ns;
//   self.budget    = min(self.size, self.budget + new_tokens);
//   self.last_update = now;     // (only advanced by whole-token increments — verify)
//   if self.budget >= n { self.budget -= n; Ok }   else { Err(deficit) }
}

Tip: The lazy-refill trick is why the limiter costs nothing when idle and needs no timer thread. A timer is armed only when a request is denied — to fire exactly when the deficit will be covered. When you read consume, watch how last_update advances; a subtle bug there (advancing it past the point you actually credited tokens) silently changes the effective rate. This is the kind of math a reviewer reads line by line.

The parameters

Param	Meaning	Units
`size`	Bucket capacity — the steady-state burst and the refill target	tokens (operations, or bytes)
`refill_time`	Time to refill `size` tokens from empty → sets the rate = `size / refill_time`	milliseconds
`one_time_burst`	Extra tokens available once, at startup, on top of `size` — absorbs an initial spike	tokens

The rate is size / refill_time. size alone is the burst you can spend instantly before the rate kicks in. one_time_burst is a one-shot allowance — useful when, say, a freshly booted guest does a large initial read and you do not want that to count against steady-state throughput.

rg -n "size|refill_time|one_time_burst|RateLimiterConfig|TokenBucketConfig" \
  src/vmm/src/rate_limiter/ src/vmm/src/vmm_config/

Two buckets, two dimensions

A single rate limiter contains two independent token buckets because there are two distinct things you throttle:

Bucket	Token =	Limits	TokenType
ops	one operation/request	IOPS (requests per second)	`TokenType::Ops`
bandwidth	one byte	throughput (bytes per second)	`TokenType::Bytes`

An operation must satisfy both: a 1-byte read still costs one ops-token, and a 1 MiB read costs one ops-token and a million byte-tokens. Either empty bucket blocks the operation. This lets you cap, independently, "how many I/Os per second" and "how many bytes per second" — a workload of many tiny reads is bounded by the ops bucket, a workload of few huge reads by the bandwidth bucket.

rg -n "TokenType::Ops|TokenType::Bytes|enum TokenType|consume.*Ops|consume.*Bytes" \
  src/vmm/src/rate_limiter/ src/vmm/src/devices/virtio/

Note: The entropy device is the exception — it carries a single bandwidth bucket only (random bytes/second), no ops bucket. Block and net carry the full two-bucket limiters, and net carries two of them (one for RX, one for TX). Confirm on your branch.

How a device uses it: the pause/resume cycle

This is the part that matters on the fast path. When the device pops work off a virtqueue (virtqueues), it asks the limiter to consume before doing the I/O. Two outcomes:

Tokens available — consume succeeds, the device does the host pread/write (block) or read/write on the TAP fd (net), completes the descriptor chain, and moves on.
Tokens exhausted — consume fails. The device stops processing the queue, marks itself rate-limited, and the limiter's TimerFd is armed to fire when the deficit is covered. The device returns to the EventManager loop. Crucially, the thread is not blocked — it goes back to servicing other devices.

When the timer fires, its eventfd becomes readable in the VMM thread's epoll set. The EventManager wakes the device's timer handler, which calls the limiter's refill, then resumes processing the queue from where it stopped.

sequenceDiagram
    participant Q as virtqueue (guest)
    participant D as device (process_queue)
    participant RL as RateLimiter (2 buckets)
    participant T as TimerFd
    participant EM as VMM thread (EventManager)
    Q->>D: kick (ioeventfd) — buffers available
    D->>RL: consume(ops=1, bytes=N)
    alt tokens available
        RL-->>D: Ok
        D->>D: do host I/O, add_used, inject IRQ
    else exhausted
        RL-->>D: Blocked(deficit)
        D->>T: arm timer for deficit
        D->>EM: return (thread free for other devices)
        T-->>EM: timer fires (eventfd readable)
        EM->>D: timer handler
        D->>RL: replenish; resume process_queue
    end

# The device side of the cycle: the "blocked" state and the timer handler.
rg -n "is_blocked|RATE_LIMITER|rate_limited|process_rate_limiter|TimerFd|fn activate|register" \
  src/vmm/src/devices/virtio/block/ src/vmm/src/devices/virtio/net/

At activate (the transport's DRIVER_OK), the device registers not only its queue eventfd but also the rate limiter's timer fd with the EventManager. That registration is what lets an exhausted limiter resume itself later. If you ever see a device that throttles correctly but then never resumes, the timer-fd registration in activate is the first place to look.

   ┌──────────── VMM thread: EventManager epoll set ────────────┐
   │  block queue eventfd   ─┐                                   │
   │  block RL timer fd      ├─► one device, multiple sources    │
   │  net TX queue eventfd   │                                   │
   │  net RX queue eventfd   │                                   │
   │  net RX RL timer fd      ┘                                  │
   │  net TX RL timer fd                                         │
   └────────────────────────────────────────────────────────────┘
      An empty bucket arms a timer fd → epoll wakes → resume.

Configuration and runtime updates

Rate limiters are set per device. For a block drive, in the PUT /drives/{id} body; for a net interface, separate rx_rate_limiter and tx_rate_limiter in PUT /network-interfaces/{id}. The JSON shape is the bucket parameters:

# A drive with ~1000 IOPS and ~10 MiB/s, plus a one-time burst on bandwidth.
curl -X PUT --unix-socket $API --data '{
  "drive_id":"rootfs","path_on_host":"./rootfs.ext4","is_root_device":true,"is_read_only":false,
  "rate_limiter":{
    "ops":      {"size":1000,    "refill_time":1000},
    "bandwidth":{"size":10485760,"one_time_burst":52428800,"refill_time":1000}
  }
}' http://localhost/drives/rootfs

refill_time is in milliseconds, so {"size":1000,"refill_time":1000} = 1000 tokens per 1000 ms = 1000 ops/s; {"size":10485760,"refill_time":1000} = 10 MiB/s. Either bucket may be omitted to leave that dimension unlimited.

Tip: Rate limiters can be updated at runtime via PATCH — PATCH /drives/{id} and PATCH /network-interfaces/{id} accept a new rate_limiter/rx_rate_limiter/tx_rate_limiter without rebooting the guest. This is how an orchestrator re-throttles a misbehaving tenant live.
rg -n "PATCH|update_rate_limiter|patch|RateLimiterUpdate" src/vmm/src/rpc_interface.rs src/vmm/src/devices/virtio/

Why non-blocking matters

The entire design exists to avoid one thing: a slow tenant blocking the VMM thread. The VMM thread services every device for the microVM. If a throttled block device blocked the thread waiting for tokens, it would freeze the net device, the vsock device, and the rest of the guest's I/O along with it. So the limiter never sleeps a thread — it returns control to the EventManager and uses a timer fd to be woken precisely when there is work it can legally do. This is the same discipline as the rest of Firecracker's threading model: the fast path must never block on something a guest controls.

Reading exercise

# 1. The token bucket and its lazy refill.
rg -n "struct TokenBucket|fn consume|fn auto_replenish|last_update|refill_time|one_time_burst" \
  src/vmm/src/rate_limiter/

# 2. The RateLimiter wrapping two buckets + a timer.
rg -n "struct RateLimiter|TokenType::Ops|TokenType::Bytes|TimerFd|fn new\b" src/vmm/src/rate_limiter/

# 3. The device side: consume on the hot path, and the blocked state.
rg -n "consume|is_blocked|rate_limited|process_rate_limiter" \
  src/vmm/src/devices/virtio/block/ src/vmm/src/devices/virtio/net/

# 4. Timer-fd registration in activate (what lets a throttled device resume).
rg -n "fn activate|TimerFd|register|RATE_LIMITER" src/vmm/src/devices/virtio/

# 5. Config + runtime PATCH wiring.
rg -n "RateLimiterConfig|rate_limiter|rx_rate_limiter|tx_rate_limiter|PATCH" \
  src/vmm/src/vmm_config/ src/vmm/src/rpc_interface.rs

# 6. The unit tests pin the math — read them to confirm your mental model.
rg -n "#\[test\]|fn test_.*bucket|consume|replenish" src/vmm/src/rate_limiter/

Answer:

Write the formula for how many tokens a bucket holds at time t, given size, refill_time, and the time of the last consume. What caps it?
Why are there two buckets per limiter? Give a workload bounded by each, and explain why an operation must satisfy both.
Distinguish size, refill_time, and one_time_burst. What is the steady-state rate in terms of the first two?
Trace what happens, step by step, when a block device's bandwidth bucket is empty: what stops, what gets armed, which thread does what, and what makes it resume.
Why does the limiter refill lazily on consume instead of running a background refill thread? What does that buy you?
The entropy device's limiter differs from block/net's. How, and why does that make sense?

Common bugs and symptoms

Symptom	Root cause	Where to look
Device throttles, then never resumes	Rate-limiter timer fd not registered with the `EventManager` in `activate`	`fn activate`; timer-fd registration; `EventManager`
Effective rate is wrong (too fast/slow)	`last_update` advanced incorrectly, or refill math rounds the wrong way	`consume`/`auto_replenish`; the token accounting; the unit tests
Configured limit ignored	Limiter built but device never calls `consume` on that path (e.g. only TX checked, not RX)	the `consume` call sites in block/net `process_*`
`one_time_burst` spent every refill, not once	Burst tokens folded into the refillable pool instead of a one-shot reserve	the bucket constructor; how `one_time_burst` is tracked
VMM thread stalls under throttling	Device blocks the thread waiting for tokens instead of returning + arming a timer	the blocked-state return path; never sleep on the fast path
Runtime `PATCH` of rate limiter has no effect	New config not applied to the live limiter, or applied to a copy	`update_rate_limiter`/PATCH dispatch in `rpc_interface.rs`

Validation: prove you understand this

Draw a token bucket and label size, the refill rate (size/refill_time), and one_time_burst. Compute the bucket state after 250 ms of idle for size=1000, refill_time=1000.
Explain the two-bucket (ops + bandwidth) model and why a single operation can be blocked by either.
Explain lazy refill: what consume computes, why there is no background thread, and when a timer is armed.
Walk the full pause→resume cycle for an exhausted limiter, naming the timer fd, the EventManager, and the thread that is not blocked.
Give the rate_limiter JSON for a 2000 IOPS, 20 MiB/s drive and explain each number and its unit.
A PR makes the device sleep the VMM thread until tokens refill "to keep the code simple." Explain, on both correctness and isolation grounds, why a maintainer rejects it.

Next: The Jailer — leaving the device model for the security barrier: chroot, cgroups, namespaces, and the privilege drop that wraps every Firecracker process. (Or return to the deep-dives index to pick your next thread.)

Open-Source Engineer & Contributor