Lab 13: An XDP Packet Counter

Background

You will write a BPF program that runs in the network driver, before the kernel has allocated an sk_buff — on the raw bytes the NIC just wrote into memory. It will count packets by protocol, then drop some of them, and you will measure exactly how much work that avoids.

This is also your first real BPF program, and BPF is worth learning on its own terms: it is how a great deal of modern kernel extension happens, it is verified rather than trusted, and it is loadable at runtime with no rebuild.

Why This Lab Matters

  • XDP is the fastest packet-processing path in Linux, and understanding why means understanding everything the normal path does that XDP skips.
  • The BPF verifier teaches you, forcefully, what safe kernel code looks like.
  • The whole thing runs on veth in a VM — no special hardware.
  • "Where did my packet go?" is answered with the same tooling.

Prerequisites

  • The sk_buff, The RX and TX Paths, and Sockets, Filtering, and XDP read.
  • A guest kernel with CONFIG_BPF_SYSCALL, CONFIG_BPF_EVENTS, CONFIG_XDP_SOCKETS, and CONFIG_DEBUG_INFO_BTF — all in lab-fast.config except XDP_SOCKETS.
  • clang, llvm, libbpf, and bpftool in the guest (or built on the host and copied in).
# In the guest:
grep -E 'CONFIG_(BPF_SYSCALL|BPF_EVENTS|XDP_SOCKETS|DEBUG_INFO_BTF)=' /proc/config.gz 2>/dev/null \
  || zcat /proc/config.gz | grep -E 'BPF_SYSCALL|XDP_SOCKETS|DEBUG_INFO_BTF'
ls -l /sys/kernel/btf/vmlinux
bpftool version
clang --version

Predict First

  1. How many lines is a "count packets by IP protocol" XDP program?
  2. What happens if you read one byte past data_end? At load time or at run time?
  3. With an XDP program returning XDP_DROP for all traffic, what does netif_receive_skb count?
  4. What do the interface's rx_packets statistics show in that case? Why do the two answers differ?
  5. Native versus generic mode: how much difference, and in which direction?
  6. Can an XDP program allocate memory? Loop? Call a kernel function?

The Target

   NIC DMAs a frame into a page
     │
     ▼
   driver's NAPI poll
     │
     ├──▶ XDP PROGRAM  ◀── YOUR CODE. No sk_buff exists.
     │      ctx = { data, data_end, data_meta, ingress_ifindex, rx_queue_index }
     │      │
     │      ├── XDP_PASS     → continue: build an sk_buff, up the stack
     │      ├── XDP_DROP     → free the buffer. THE STACK NEVER SEES IT.
     │      ├── XDP_TX       → send it back out the same interface
     │      ├── XDP_REDIRECT → another interface, or an AF_XDP socket
     │      └── XDP_ABORTED  → drop + a tracepoint (this means a bug)
     │
     ▼ (only on XDP_PASS)
   build sk_buff → netif_receive_skb → ip_rcv → ... → your socket

Step-by-Step Tasks

Step 1: A test network you can flood safely

Two namespaces joined by a veth pair. veth supports native XDP, which matters.

sudo ip netns add xdplab
sudo ip link add veth0 type veth peer name veth1
sudo ip link set veth1 netns xdplab

sudo ip addr add 10.7.0.1/24 dev veth0
sudo ip link set veth0 up
sudo ip -n xdplab addr add 10.7.0.2/24 dev veth1
sudo ip -n xdplab link set veth1 up
sudo ip -n xdplab link set lo up

ping -c 2 10.7.0.2

Step 2: The BPF program

xdp_count.bpf.c:

// SPDX-License-Identifier: GPL-2.0
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/in.h>
#include <linux/ip.h>
#include <bpf/bpf_endian.h>
#include <bpf/bpf_helpers.h>

/* A per-CPU array keyed by IP protocol number. Per-CPU means no atomics
 * and no cache-line contention on the counter -- at 10 Mpps that matters,
 * and it is why almost every real XDP program uses PERCPU maps.        */
struct {
	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
	__uint(max_entries, 256);
	__type(key, __u32);
	__type(value, __u64);
} proto_count SEC(".maps");

/* A single flag we can flip from user space to turn dropping on and off. */
struct {
	__uint(type, BPF_MAP_TYPE_ARRAY);
	__uint(max_entries, 1);
	__type(key, __u32);
	__type(value, __u32);
} drop_proto SEC(".maps");

SEC("xdp")
int xdp_count_prog(struct xdp_md *ctx)
{
	void *data     = (void *)(long)ctx->data;
	void *data_end = (void *)(long)ctx->data_end;
	struct ethhdr *eth = data;
	struct iphdr *iph;
	__u32 key, *want_drop, zero = 0;
	__u64 *cnt;

	/* EVERY access must be bounds-checked against data_end BEFORE the
	 * dereference. The verifier rejects the program otherwise -- it is
	 * not a runtime check you can forget, it is a LOAD-TIME proof
	 * obligation. This is the single biggest difference from writing C. */
	if ((void *)(eth + 1) > data_end)
		return XDP_PASS;

	if (eth->h_proto != bpf_htons(ETH_P_IP))
		return XDP_PASS;

	iph = (void *)(eth + 1);
	if ((void *)(iph + 1) > data_end)		/* check again for iph */
		return XDP_PASS;

	key = iph->protocol;
	cnt = bpf_map_lookup_elem(&proto_count, &key);
	if (cnt)					/* the verifier requires
							 * this NULL check too */
		__sync_fetch_and_add(cnt, 1);

	want_drop = bpf_map_lookup_elem(&drop_proto, &zero);
	if (want_drop && *want_drop == key)
		return XDP_DROP;

	return XDP_PASS;
}

char _license[] SEC("license") = "GPL";

Step 3: Build and load it

clang -O2 -g -target bpf -c xdp_count.bpf.c -o xdp_count.bpf.o
llvm-objdump -S xdp_count.bpf.o | head -40      # the BPF instructions

# Load and attach, native mode:
sudo ip link set dev veth0 xdp obj xdp_count.bpf.o sec xdp

# Confirm WHICH mode you got. This is the step people skip.
ip -d link show dev veth0 | grep -i xdp
#   "prog/xdp"        -> NATIVE
#   "prog/xdp generic" -> GENERIC. Your benchmark will lie.

sudo bpftool prog show
sudo bpftool map show

Step 4: Watch it count

MAPID=$(sudo bpftool map show | awk '/proto_count/ {print $1}' | tr -d :)

sudo ip netns exec xdplab ping -c 20 -q 10.7.0.1 >/dev/null

# Protocol 1 = ICMP. Per-CPU map: one value per CPU, so sum them.
sudo bpftool map dump id "$MAPID" | head -20
sudo bpftool map lookup id "$MAPID" key 1 0 0 0

PREDICT FIRST: you sent 20 pings. What will the ICMP counter read — 20, 40, or something else? (Think about which direction veth0's XDP hook sees.)

Step 5: Drop, and measure what it avoided

DROPID=$(sudo bpftool map show | awk '/drop_proto/ {print $1}' | tr -d :)

# Measure the normal path first:
sudo bpftrace -e '
  tracepoint:net:netif_receive_skb /str(args.name) == "veth0"/ { @stack_saw = count(); }
  tracepoint:xdp:xdp_exception { @exceptions = count(); }' &
sudo ip netns exec xdplab ping -f -c 5000 10.7.0.1 >/dev/null 2>&1
sleep 1; kill %1

# Now drop ICMP (protocol 1) in XDP:
sudo bpftool map update id "$DROPID" key 0 0 0 0 value 1 0 0 0

sudo bpftrace -e '
  tracepoint:net:netif_receive_skb /str(args.name) == "veth0"/ { @stack_saw = count(); }' &
sudo ip netns exec xdplab ping -f -c 5000 -W 1 10.7.0.1 >/dev/null 2>&1
sleep 1; kill %1

ip -s link show veth0 | head -6         # the INTERFACE counters

PREDICT FIRST: with the drop enabled, what does @stack_saw read? And what does ip -s link's RX packets read? Explain, precisely, why they disagree.

Step 6: Native versus generic

sudo ip link set dev veth0 xdp off

# Force generic mode:
sudo ip link set dev veth0 xdpgeneric obj xdp_count.bpf.o sec xdp
ip -d link show dev veth0 | grep -i xdp

sudo bpftrace -e 'tracepoint:net:netif_receive_skb { @ = count(); }' &
time sudo ip netns exec xdplab ping -f -c 20000 10.7.0.1 >/dev/null 2>&1
kill %1

sudo ip link set dev veth0 xdpgeneric off
sudo ip link set dev veth0 xdp obj xdp_count.bpf.o sec xdp
sudo bpftrace -e 'tracepoint:net:netif_receive_skb { @ = count(); }' &
time sudo ip netns exec xdplab ping -f -c 20000 10.7.0.1 >/dev/null 2>&1
kill %1

PREDICT FIRST: which is faster, and by how much? And in generic mode, does netif_receive_skb still fire for dropped packets? (The answer tells you exactly where the generic hook sits.)

Step 7: Fight the verifier, deliberately

The verifier is a teacher. Break the program on purpose and read what it says.

/* 1. Remove a bounds check: */
	iph = (void *)(eth + 1);
	/* if ((void *)(iph + 1) > data_end) return XDP_PASS;   ← delete this */
	key = iph->protocol;

/* 2. Remove the NULL check on a map lookup: */
	cnt = bpf_map_lookup_elem(&proto_count, &key);
	__sync_fetch_and_add(cnt, 1);          /* no `if (cnt)` */

/* 3. An unbounded loop: */
	for (int i = 0; i < ctx->data_end - ctx->data; i++) { ... }

clang -O2 -g -target bpf -c xdp_count.bpf.c -o bad.o
sudo ip link set dev veth0 xdp obj bad.o sec xdp    # read the ENTIRE error

PREDICT FIRST, for each of the three: does it fail at compile time or at load time? And what does the verifier's message actually tell you?

The output is long and worth reading in full — it prints the instruction-by-instruction state it inferred, and the line where its proof failed.

Step 8: Clean up

sudo ip link set dev veth0 xdp off
sudo ip link del veth0
sudo ip netns del xdplab

Implementation Requirements / Deliverables

  • A veth test network, with ping working both ways.
  • xdp_count.bpf.c compiled, loaded, and verified to be in native mode with ip -d link.
  • Per-protocol counts read back with bpftool, and the per-CPU values summed correctly.
  • The drop flag flipped from user space at runtime, with no reload.
  • A measurement showing netif_receive_skb reaches zero for dropped traffic.
  • An explanation of why interface rx_packets disagrees with the stack's count.
  • Native versus generic measured, with the direction and magnitude recorded.
  • All three verifier failures triggered, with the messages saved and explained in your own words.
  • All six predictions recorded, with results and a one-sentence note on each miss.

Expected Output

$ ip -d link show dev veth0 | grep -i xdp
    prog/xdp id 42 tag 8f3c1a2b4d5e6f70 jited        ← NATIVE (no "generic")

$ sudo bpftool map dump id 17
[{
    "key": 1,
    "values": [{ "cpu": 0, "value": 5000 },
               { "cpu": 1, "value": 0 }, ...]
}]

$ # before the drop
@stack_saw: 5000
$ # after enabling the drop
@stack_saw: 0                        ← the stack never saw a single one

$ ip -s link show veth0
    RX: bytes  packets  errors  dropped
        490000    5000       0        0    ← the INTERFACE counted them:
                                             they arrived, then XDP dropped
                                             them before the stack.

And the verifier, when you remove a bounds check:

; key = iph->protocol;
23: (71) r1 = *(u8 *)(r2 +9)
 invalid access to packet, off=23 size=1, R2(id=0,off=23,r=14)
 R2 offset is outside of the packet
processed 24 insns (limit 1000000) ...

It tells you the instruction, the register, the offset, and what it could not prove. That is the verifier doing its job, and reading these messages is the fastest way to learn what it requires.

Debugging Steps

Error fetching program/map!

bpftool needs root and a BTF-enabled kernel. ls -l /sys/kernel/btf/vmlinux; if it is missing, CONFIG_DEBUG_INFO_BTF was off, usually because pahole was not installed at build time.

libbpf: failed to find BTF for extern

Your vmlinux.h or bpf_helpers.h does not match the kernel. Regenerate:

sudo bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

The program loads but counts nothing

Check the direction: XDP is ingress only. veth0's XDP sees packets arriving at veth0, which are the ones the peer sent. Ping from the namespace to 10.7.0.1.

prog/xdp generic when you asked for native

The driver does not support native XDP. veth and virtio_net do; check ethtool -i for the driver and look for ndo_bpf in its source:

rg -n "ndo_bpf|ndo_xdp_xmit" drivers/net/veth.c drivers/net/virtio_net.c | head

The verifier rejects something you are sure is correct

Read the whole message, then simplify: fewer branches, explicit bounds variables, #pragma unroll for small loops. The verifier is conservative and cannot always prove what you know.

XDP_TX produces nothing on the wire

You must fix the ethernet header (swap the MACs) before returning XDP_TX; otherwise the frame is addressed to yourself.

Everything works but performance is unimpressive

You are measuring veth in a VM, and ping -f from a namespace is not a load generator. The ratios here are informative; absolute packet rates need real hardware and pktgen or trafgen.

Experiment

CLAIM. XDP's advantage is precisely the work it skips, and you can attribute the saving layer by layer by dropping the same traffic at four different hooks.

METHOD. Drop identical traffic at each of four places and measure CPU cost per packet:

HookHow
XDPThe program above, XDP_DROP
tc ingress BPFtc qdisc add dev veth0 clsact; tc filter add dev veth0 ingress bpf da obj drop.o
netfilternft add rule inet f c ip protocol icmp drop
socketLet it arrive and have the application discard it
for hook in xdp tc nft app; do
  echo "== $hook"
  # ...attach the relevant one...
  sudo perf stat -a -e cycles,instructions -- \
    sudo ip netns exec xdplab ping -f -c 20000 10.7.0.1 >/dev/null 2>&1
  sudo bpftrace -e 'tracepoint:net:netif_receive_skb { @ = count(); }' -c \
    "ip netns exec xdplab ping -f -c 5000 10.7.0.1"
done

PREDICTION. Rank the four by cycles per packet before measuring, and predict the ratio between the cheapest and the most expensive.

RESULT. Then answer the design question: for each hook, name a job for which it is the right choice and XDP is the wrong one. There are good answers for all three — stateful NAT, per-cgroup policy, and anything needing to see a reassembled stream — and knowing them is what separates "XDP is fast" from actually understanding the stack.

Test

cat > ~/kernel-labs/userspace/xdp-test.sh <<'EOF'
#!/usr/bin/env bash
# Verify the XDP program counts and drops correctly. Exit 0 on success.
set -euo pipefail
IF=${IF:-veth0}; NS=${NS:-xdplab}; PEER=${PEER:-10.7.0.1}
OBJ=${OBJ:-xdp_count.bpf.o}
fail() { echo "FAIL: $*"; exit 1; }

ip link show "$IF" >/dev/null 2>&1 || { echo "SKIP: no $IF"; exit 4; }

ip link set dev "$IF" xdp off 2>/dev/null || true
ip link set dev "$IF" xdp obj "$OBJ" sec xdp

# 1. It must be NATIVE, or every later measurement is meaningless.
ip -d link show dev "$IF" | grep -q 'prog/xdp generic' && fail "loaded in GENERIC mode"

MAPID=$(bpftool map show | awk '/proto_count/ {print $1}' | tr -d :)
DROPID=$(bpftool map show | awk '/drop_proto/ {print $1}' | tr -d :)
[ -n "$MAPID" ] && [ -n "$DROPID" ] || fail "maps not found"

# 2. It counts.
before=$(bpftool map lookup id "$MAPID" key 1 0 0 0 -j 2>/dev/null | grep -o '[0-9]*' | paste -sd+ | bc)
ip netns exec "$NS" ping -c 10 -q "$PEER" >/dev/null 2>&1
after=$(bpftool map lookup id "$MAPID" key 1 0 0 0 -j 2>/dev/null | grep -o '[0-9]*' | paste -sd+ | bc)
[ "$after" -gt "$before" ] || fail "ICMP counter did not increase"

# 3. It drops, and the stack never sees them.
bpftool map update id "$DROPID" key 0 0 0 0 value 1 0 0 0
seen=$(bpftrace -e "tracepoint:net:netif_receive_skb /str(args.name) == \"$IF\"/ { @ = count(); }" \
        -c "ip netns exec $NS ping -f -c 500 -W 1 $PEER" 2>/dev/null | grep -o '@: [0-9]*' | awk '{print $2}')
[ "${seen:-0}" -eq 0 ] || fail "stack saw $seen packets that XDP should have dropped"

bpftool map update id "$DROPID" key 0 0 0 0 value 0 0 0 0
ip link set dev "$IF" xdp off
echo PASS
EOF
chmod +x ~/kernel-labs/userspace/xdp-test.sh
sudo ~/kernel-labs/userspace/xdp-test.sh

Verify it can fail: change XDP_DROP to XDP_PASS in the drop branch and re-run — check 3 must fail. Load it in generic mode — check 1 must fail.

Challenge Extensions

  1. XDP_TX: a one-line load balancer. Swap the source and destination MACs and return XDP_TX to reflect packets. Then extend it to rewrite the destination IP and act as a trivial DSR load balancer. This is, in outline, how Katran and Cilium work.

  2. AF_XDP. Add an XSK map and redirect selected packets to a user-space AF_XDP socket, bypassing the stack entirely. Measure the throughput against a normal socket. This is the kernel's answer to DPDK.

  3. Add a drop reason. Find a bare kfree_skb() in net/ or a driver, work out the correct SKB_DROP_REASON_* (or add one), and convert it. This is a genuinely good upstream patch — small, mechanical to review, and it makes someone's future debugging session shorter.

  4. Compare with tc-BPF. Write the same counter as a tc ingress program and diff them. Note what tc-BPF can do that XDP cannot (full skb access, egress, metadata) and what it costs.

  5. Trace xdp_exception. Make your program return XDP_ABORTED and watch the xdp:xdp_exception tracepoint fire. Then explain why a production program returning XDP_ABORTED is always a bug.

  6. Read a real one. tools/testing/selftests/bpf/progs/ has dozens of XDP programs, and Cilium and Katran are open source. Pick one, read it, and identify every bounds check and why the verifier needed it.

Validation / Self-check

  1. Where does an XDP program run relative to sk_buff allocation, and what follows from that about what it can do?
  2. Name the five XDP_* return actions and when each is right.
  3. Why must every packet access be bounds-checked against data_end? When is that enforced?
  4. Why do XDP programs use per-CPU maps for counters?
  5. Why do the interface's rx_packets and the stack's netif_receive_skb count disagree when dropping?
  6. Name the three XDP modes. Which makes a benchmark lie, and how do you check which you are in?
  7. Give the three verifier failures you triggered and, for each, what the verifier could not prove.
  8. Can an XDP program loop? Allocate? Call an arbitrary kernel function? What are the alternatives?
  9. From the experiment: rank XDP, tc-BPF, netfilter, and socket-level dropping by cost, and give a job for which each of the latter three beats XDP.
  10. What must you fix before returning XDP_TX, and what happens if you do not?
  11. Why is XDP_ABORTED in production always a bug?
  12. You want stateful connection tracking at 10 Mpps. What is the tension, and what would you actually build?

Phase Complete

You have now read four of the eight domains at depth: the scheduler, memory, storage, and networking. Those four touch everything else in the kernel — a graphics driver allocates memory, sleeps, takes interrupts, and DMAs, and it is readable because of what you now know.

Next: Graphics, and the four remaining domains.