The RX and TX Paths

Three concepts in the six-part treatment: NAPI and the receive path, the transmit path and qdiscs, and multiqueue and scaling.

The receive and transmit paths are not mirror images. They run in different contexts, queue in different places, and fail in different ways — and treating one as the reverse of the other is the most common source of confusion in this subsystem.

Concept 1: NAPI and the Receive Path

1. What problem it solves

One interrupt per packet is fine at 100 Mb/s and catastrophic at 10 Gb/s: at 800,000 packets per second the machine does nothing but take interrupts, and under a flood it can reach receive livelock — spending 100% of its time in interrupt handlers, making no forward progress, and never running the code that would consume the packets.

NAPI switches between interrupt-driven and polled operation automatically: the first packet raises an interrupt, the handler disables further interrupts and schedules a poll, and the poll drains the ring in batches until it is empty — at which point interrupts are re-enabled.

Under load, a NAPI driver takes almost no interrupts at all.

2. Where it exists in the kernel

rg -n "napi_schedule\b|napi_complete_done\b|napi_poll\b" -A 20 net/core/dev.c | head -40
rg -n "struct napi_struct \{" -A 25 include/linux/netdevice.h
rg -n "netif_napi_add" -A 10 include/linux/netdevice.h | head
$EDITOR Documentation/networking/napi.rst

3. The path

   PACKET ARRIVES
     └── the NIC DMAs it into a pre-posted receive buffer in host memory
     └── raises an interrupt (MSI-X, usually per-queue, pinned to a CPU)
           │
     ┌─────▼──────────────────────────────────────────────────────┐
     │ HARDIRQ (the driver's handler). Atomic. Microseconds.      │
     │   - disable this queue's interrupts                        │
     │   - napi_schedule() -> raise NET_RX_SOFTIRQ                │
     │   - return IRQ_HANDLED                                     │
     └─────┬──────────────────────────────────────────────────────┘
           │
     ┌─────▼──────────────────────────────────────────────────────┐
     │ SOFTIRQ (NET_RX). Still ATOMIC -- cannot sleep.            │
     │   net_rx_action() -> driver's ->poll(napi, budget)         │
     │     for each descriptor, up to BUDGET (default 64):        │
     │       - build an sk_buff around the received buffer        │
     │       - eth_type_trans(): set skb->protocol, pull the      │
     │         ethernet header                                    │
     │       - napi_gro_receive(): try to MERGE with an in-flight │
     │         flow (see the GRO section)                         │
     │     if the ring emptied: napi_complete_done() and RE-ENABLE │
     │       interrupts                                            │
     │     if the budget ran out: stay polled, come back          │
     └─────┬──────────────────────────────────────────────────────┘
           │
     netif_receive_skb()
       ├── XDP already ran, much earlier (see the next chapter)
       ├── tc ingress / BPF classifiers
       ├── packet taps: tcpdump gets a CLONE here
       └── deliver by skb->protocol -> ip_rcv()
                 ├── netfilter NF_INET_PRE_ROUTING
                 ├── routing decision: local, or forward?
                 ├── netfilter NF_INET_LOCAL_IN
                 └── ip_local_deliver_finish -> tcp_v4_rcv()
                       ├── find the socket (a hash lookup)
                       ├── the TCP state machine
                       ├── queue on sk->sk_receive_queue
                       └── sk->sk_data_ready() -> WAKE THE READER
                                                   │
   ...the reader's process is now runnable. It is scheduled, returns from
      recv(), and copies the data out. THAT is when your process runs.

Two things about this are worth stating plainly. The entire path from interrupt to sk_data_ready runs in softirq context on whatever CPU took the interrupt, not in your process. And your process does not appear until the very last step, which is why "the receive path is slow" is usually a scheduling or wakeup problem measured with the tools from Lab 10.

4. The budget, and ksoftirqd

cat /proc/sys/net/core/netdev_budget          # packets per net_rx_action, default 300
cat /proc/sys/net/core/netdev_budget_usecs    # ...or a time limit
cat /proc/net/softnet_stat

/proc/net/softnet_stat has one row per CPU. The columns to know:

A rising time_squeeze means the receive softirq is not keeping up, and the work is being handed to ksoftirqd — which is when you see that thread at 100% CPU.

# Watch it move under load:
watch -n1 'cat /proc/net/softnet_stat; ps -eo comm,pcpu | grep ksoftirqd'

5. Experiment

CLAIM. NAPI really does stop taking interrupts under load, and you can watch the transition.

METHOD.

IF=$(ls /sys/class/net | grep -v lo | head -1)
IRQ=$(grep -i "$IF" /proc/interrupts | head -1 | cut -d: -f1 | tr -d ' ')

snapshot() { grep "^ *$IRQ:" /proc/interrupts; grep -c . /dev/null; }

echo "== light load (a few pings)"
before=$(grep "^ *$IRQ:" /proc/interrupts | awk '{s=0; for(i=2;i<=NF-2;i++)s+=$i; print s}')
ping -c 100 -i 0.05 -q <peer> >/dev/null 2>&1
after=$(grep "^ *$IRQ:" /proc/interrupts | awk '{s=0; for(i=2;i<=NF-2;i++)s+=$i; print s}')
echo "interrupts: $((after-before)) for ~200 packets"

echo "== heavy load"
before=$after
iperf3 -c <peer> -t 10 >/dev/null 2>&1
after=$(grep "^ *$IRQ:" /proc/interrupts | awk '{s=0; for(i=2;i<=NF-2;i++)s+=$i; print s}')
echo "interrupts: $((after-before)) for millions of packets"

sudo bpftrace -e '
  tracepoint:napi:napi_poll { @polled = hist(args.work); }
  tracepoint:irq:softirq_entry /args.vec == 3/ { @net_rx_softirqs = count(); }'

PREDICT FIRST: the ratio of packets to interrupts under light load, and under heavy load. The second number should be startling — that is the whole point of NAPI.

6. Failure mode

Concept 2: The Transmit Path and Queueing Disciplines

1. What problem it solves

Transmit has a problem receive does not: the device can be busy. A packet must be held somewhere until the NIC can take it, and that "somewhere" is a decision about fairness, latency, and how much buffering to allow.

Too little buffering wastes the link. Too much creates bufferbloat: a full queue means every packet waits behind megabytes of someone else's bulk transfer, and interactive traffic becomes unusable while throughput looks perfect.

2. Where it exists in the kernel

rg -n "__dev_queue_xmit\b" -A 50 net/core/dev.c | head -60
ls net/sched/ | head -30
rg -n "struct Qdisc_ops \{" -A 25 include/net/sch_generic.h
tc qdisc show
$EDITOR Documentation/networking/ 2>/dev/null; ls Documentation/networking/ | grep -i sch

3. The path

   send() ──▶ tcp_sendmsg()
                ├── copy from user into skb page frags
                ├── the congestion window decides how much may go NOW
                └── tcp_write_xmit -> ip_queue_xmit()
                      ├── route lookup (cached in the socket)
                      ├── build the IP header: skb_push()
                      ├── netfilter NF_INET_LOCAL_OUT, then POST_ROUTING
                      └── dev_queue_xmit()
                            ├── select a TX queue (skb_tx_hash / XPS)
                            ├── tc egress / BPF
                            ├── THE QDISC -- enqueue()
                            │     may DROP here (queue full, AQM decision)
                            └── qdisc_run() -> dequeue() -> the driver
                                  └── ndo_start_xmit()
                                        - map the frags for DMA
                                        - write descriptors
                                        - ring the doorbell
                                        - return NETDEV_TX_OK
   ...later, a TX completion interrupt frees the skb ("TX cleanup").

The qdisc is the part with no receive equivalent. It is where queueing policy lives, and the default matters enormously:

tc qdisc show
tc -s qdisc show dev eth0            # statistics, including DROPS and backlog
sysctl net.core.default_qdisc

4. Backpressure

The other thing transmit has that receive does not: a way to say stop.

   The driver's TX ring fills.
     └── netif_stop_queue() -- the qdisc stops dequeuing
           └── the qdisc's queue fills
                 └── enqueue() starts DROPPING (or the socket blocks)
                       └── TCP sees the loss/backpressure and slows down

   Later, TX completions free descriptors:
     └── netif_wake_queue() -- and it flows again.

   BQL (Byte Queue Limits) tunes how much the driver ring is allowed to
   hold, so the QDISC does the queueing (where the smart AQM is) rather
   than the dumb FIFO in the driver.

ls /sys/class/net/eth0/queues/tx-0/byte_queue_limits/ 2>/dev/null
cat /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit 2>/dev/null

5. Experiment

CLAIM. Bufferbloat is real, measurable in a VM, and fq_codel fixes it.

METHOD. Between two network namespaces connected by veth, with an artificial bottleneck:

# Set up two namespaces with a rate-limited link.
sudo ip netns add a; sudo ip netns add b
sudo ip link add va type veth peer name vb
sudo ip link set va netns a; sudo ip link set vb netns b
sudo ip -n a addr add 10.9.0.1/24 dev va; sudo ip -n a link set va up
sudo ip -n b addr add 10.9.0.2/24 dev vb; sudo ip -n b link set vb up
sudo ip -n a link set lo up; sudo ip -n b link set lo up

for q in pfifo_fast fq_codel; do
  sudo ip netns exec a tc qdisc replace dev va root handle 1: tbf rate 10mbit burst 32kb latency 400ms
  sudo ip netns exec a tc qdisc replace dev va parent 1:1 handle 10: $q 2>/dev/null || true
  echo "== $q"
  sudo ip netns exec b iperf3 -s -D 2>/dev/null
  sudo ip netns exec a iperf3 -c 10.9.0.2 -t 15 >/dev/null 2>&1 &
  sleep 3
  sudo ip netns exec a ping -c 10 -q 10.9.0.2 | tail -2
  wait; sudo ip netns exec b pkill iperf3
done
sudo ip netns del a; sudo ip netns del b

PREDICT FIRST: with a bulk transfer saturating a 10 Mb/s link, what is the ping RTT under pfifo_fast? Under fq_codel? The difference is usually one to two orders of magnitude, and throughput is nearly identical.

6. Failure mode

Concept 3: Multiqueue and Scaling

1. What problem it solves

One CPU cannot drive a 100 Gb/s NIC. Modern NICs have many hardware queues, and the question is how to map packets to them so that the work spreads across CPUs without reordering a flow — because TCP treats reordering as loss.

2. Where it exists in the kernel

$EDITOR Documentation/networking/scaling.rst      # RSS, RPS, RFS, XPS, all in one doc
ls /sys/class/net/eth0/queues/
cat /proc/interrupts | grep -i eth
ethtool -l eth0 2>/dev/null; ethtool -x eth0 2>/dev/null | head

3. The mechanisms

   THE INVARIANT: all packets of one flow go to ONE queue, hence ONE CPU.
     - preserves ordering (TCP reordering looks like loss)
     - keeps the flow's socket and data in one CPU's cache

   THE COST: one enormous flow cannot use more than one CPU. A single
   TCP stream is limited by one core's receive processing, which is why
   benchmarks use many parallel streams.

# Which CPUs handle which queue:
for q in /sys/class/net/eth0/queues/rx-*; do
  echo "$q: rps_cpus=$(cat $q/rps_cpus 2>/dev/null)"
done
grep -i eth /proc/interrupts        # per-queue interrupts, per CPU
cat /proc/sys/net/core/rps_sock_flow_entries

4. Experiment

CLAIM. A single flow uses one CPU; parallel flows spread; and you can see the mapping.

METHOD.

# Where does receive processing happen?
sudo bpftrace -e '
  tracepoint:net:netif_receive_skb { @by_cpu[cpu] = count(); }' &

iperf3 -c <peer> -P 1 -t 10 >/dev/null 2>&1        # ONE stream
kill %1

sudo bpftrace -e 'tracepoint:net:netif_receive_skb { @by_cpu[cpu] = count(); }' &
iperf3 -c <peer> -P 8 -t 10 >/dev/null 2>&1        # EIGHT streams
kill %1

PREDICT FIRST: with one stream, how many CPUs show receive processing? With eight? And does the answer change if the NIC has only one hardware queue?

Then measure the cache effect:

# Pin the application to the CPU handling its queue vs. a different one:
taskset -c 0 iperf3 -c <peer> -t 10 2>/dev/null | tail -3
taskset -c 3 iperf3 -c <peer> -t 10 2>/dev/null | tail -3
sudo perf stat -e cache-misses,LLC-load-misses -a -- sleep 10

5. Failure mode

Validation / Self-check

1. What problem does NAPI solve, and what is receive livelock?
2. In what context does the receive path run, from interrupt to sk_data_ready? What does that forbid?
3. What does time_squeeze in /proc/net/softnet_stat mean, and what should you do about it?
4. Walk the receive path from the NIC interrupt to your process returning from recv(), naming every function.
5. Why is the transmit path not a mirror image of receive? Name three differences.
6. What is a qdisc, and what does fq_codel do that pfifo_fast does not?
7. Describe bufferbloat, and explain why throughput can look perfect while latency is terrible.
8. What is BQL for, and which layer does it want to do the queueing?
9. What does netif_stop_queue/netif_wake_queue implement, and what is the failure if you get the pairing wrong?
10. Compare RSS, RPS, RFS, and XPS. Which is free?
11. Why do all packets of one flow go to one CPU, and what does that cost?
12. Your 100 Gb/s NIC gets 8 Gb/s on a benchmark. Give three things to check, in order.

Next: Sockets, Filtering, and XDP — who is allowed to intercept a packet, and where.