The sk_buff

Three concepts in the six-part treatment: the four pointers, fragments and sharing, and GRO and GSO — the two optimizations that make the whole design pay.

If you understand only one structure in net/, make it this one. Every function you will read takes an skb and does something to those pointers.

Concept 1: The Four Pointers

1. What problem it solves

A packet gains and loses headers as it moves through the stack. Received, it arrives as [eth][ip][tcp][payload] and each layer must strip its header. Transmitted, it starts as payload and each layer must prepend one.

Doing that by copying would mean four copies per packet in each direction — at ten million packets per second, that is the entire machine.

Instead, one buffer is allocated with headroom reserved at the front, and layers move a pointer.

2. Where it exists in the kernel

rg -n "struct sk_buff \{" -A 90 include/linux/skbuff.h | head -100
rg -n "^static inline.*skb_put\b" -A 12 include/linux/skbuff.h
rg -n "^static inline.*skb_push\b|^static inline.*skb_pull\b|skb_reserve" -A 10 include/linux/skbuff.h | head -40
$EDITOR Documentation/networking/skbuff.rst 2>/dev/null || ls Documentation/networking/

3. The layout

   ONE ALLOCATION:

   head                data            tail              end
     │                   │               │                 │
     ▼                   ▼               ▼                 ▼
     ┌───────────────────┬───────────────┬─────────────────┐
     │    HEADROOM       │   THE DATA    │    TAILROOM     │
     └───────────────────┴───────────────┴─────────────────┘
       room to PUSH        skb->len         room to PUT
       headers onto        bytes here       payload into

   skb_reserve(skb, n)  move data AND tail forward -> create headroom
   skb_put(skb, n)      move tail  forward -> the data grew at the END
   skb_push(skb, n)     move data  BACKWARD -> a header was prepended
   skb_pull(skb, n)     move data  forward  -> a header was consumed
   skb_trim(skb, n)     move tail  backward -> the data shrank

Receive and transmit use them in opposite directions, and that symmetry is the whole design:

   RECEIVE                              TRANSMIT
   ───────                              ────────
   driver: skb_put(frame_len)           tcp: build payload, skb_put()
   eth_type_trans: skb_pull(14)         tcp: skb_push(sizeof(tcphdr))
   ip_rcv: skb_pull(ip_hdr_len)         ip:  skb_push(sizeof(iphdr))
   tcp_v4_rcv: skb_pull(tcp_hdr_len)    eth: skb_push(ETH_HLEN)
        │                                    │
   data now points at the PAYLOAD       data now points at the ETH HEADER

This is why headroom exists. A driver allocating a receive buffer reserves NET_SKB_PAD bytes at the front so that if the packet is later encapsulated — pushed through a tunnel, given a VLAN tag — there is room to prepend without reallocating.

The header offsets are stored separately, because after skb_pull the header is no longer at skb->data:

skb_reset_network_header(skb);          /* record where the IP header is  */
skb_set_transport_header(skb, ihl * 4); /* and the TCP header             */

struct iphdr  *iph = ip_hdr(skb);       /* head + network_header          */
struct tcphdr *th  = tcp_hdr(skb);      /* head + transport_header        */

rg -n "skb_network_header|skb_transport_header|skb_mac_header" -A 5 include/linux/skbuff.h | head -30

4. Experiment

CLAIM. The pointers move; nothing is copied. You can watch a packet's headroom shrink as headers are pushed.

METHOD. In a module or a kprobe, print the geometry at several points:

sudo bpftrace -e '
kprobe:ip_rcv {
	$skb = (struct sk_buff *)arg0;
	@rx_headroom = hist($skb->data - $skb->head);
	@rx_len      = hist($skb->len);
	@rx_datalen  = hist($skb->data_len);
}
kprobe:ip_output {
	$skb = (struct sk_buff *)arg1;
	@tx_headroom = hist($skb->data - $skb->head);
}' &
ping -c 20 -q "$(ip route | awk '/^default/{print $3;exit}')" >/dev/null 2>&1
curl -s -o /dev/null https://example.com 2>/dev/null || true
kill %1

PREDICT FIRST: how much headroom does a received packet have at ip_rcv? How much does a transmitted one have at ip_output? Which is larger, and why?

Then watch a single packet's len change as it climbs:

sudo bpftrace -e '
kprobe:netif_receive_skb { $s = (struct sk_buff *)arg0; @at_dev = hist($s->len); }
kprobe:ip_rcv            { $s = (struct sk_buff *)arg0; @at_ip  = hist($s->len); }
kprobe:tcp_v4_rcv        { $s = (struct sk_buff *)arg0; @at_tcp = hist($s->len); }'

PREDICT FIRST: by how much does len decrease between each pair? (Hint: 14, then 20, then 20 or 32 with options.)

5. Failure mode

Concept 2: Fragments and Sharing

1. What problem it solves

Two things break the simple "one linear buffer" picture.

Large packets should not need a large contiguous allocation. A 64 KB TSO segment as one kmalloc would be an order-4 allocation that fails under fragmentation. Instead the payload lives in page fragments and only the headers are linear.

A packet often needs to go to more than one place — a raw socket, tcpdump, and the real protocol handler. Copying it three times is wasteful when nobody is going to modify it.

2. Where it exists in the kernel

rg -n "struct skb_shared_info \{" -A 25 include/linux/skbuff.h
rg -n "skb_clone\b|pskb_copy\b|skb_copy\b" -A 15 net/core/skbuff.c | head -40
rg -n "skb_shared\b|skb_cloned\b|skb_header_cloned" -A 6 include/linux/skbuff.h | head -30

3. Fragments

   struct sk_buff
     head ─────────────────────────────────┐
      ┌──────────┬─────────────┬───────────┴──────────────┐
      │ headroom │   LINEAR    │ tailroom │ skb_shared_info│
      │          │  (headers)  │          │  ┌───────────┐ │
      └──────────┴─────────────┴──────────┤  │ nr_frags  │ │
                                          │  │ frags[0] ─┼─┼──▶ page, offset, size
   skb->len      = linear + data_len      │  │ frags[1] ─┼─┼──▶ page, offset, size
   skb->data_len = the frags only         │  │ frag_list │ │
   linear part   = len - data_len         │  └───────────┘ │
                                          └────────────────┘

   A "nonlinear" skb has data_len > 0. You CANNOT just read skb->data and
   expect the payload -- most of it is in pages.

The consequence for anyone touching packet data:

/* WRONG for a nonlinear skb: the payload is not at skb->data. */
memcpy(dst, skb->data + offset, len);

/* Right: either linearize (a copy, and it can fail), */
if (skb_linearize(skb))
        return -ENOMEM;

/* ...or pull just the header you need into the linear part, */
if (!pskb_may_pull(skb, sizeof(struct tcphdr)))
        goto drop;                  /* not enough data at all */

/* ...or use the accessor that walks frags for you. */
skb_copy_bits(skb, offset, dst, len);

pskb_may_pull() is the idiom you will see everywhere in protocol code, and it does two jobs at once: it checks the packet is long enough, and it makes the bytes contiguous. Forgetting it is a classic remote out-of-bounds read.

4. Sharing and cloning

   skb_clone(skb)      a NEW sk_buff header pointing at the SAME data.
                       Cheap. The data is now SHARED and READ-ONLY.
                       skb_cloned() becomes true for both.

   pskb_copy(skb)      copies the linear part; still shares the frags.

   skb_copy(skb)       copies everything. Expensive.

   THE RULE: before writing to packet data, ensure you own it.
       if (skb_shared(skb))            skb = skb_clone(...)
       if (skb_cloned(skb))            pskb_expand_head(...) or skb_copy()
   The helper that does the right thing is skb_ensure_writable().

This is where tcpdump fits: a packet capture is a clone, so running tcpdump costs almost nothing per packet — until something downstream needs to modify the packet and now has to un-share it, which is why capture can change performance measurably.

rg -n "skb_ensure_writable|skb_cow\b|skb_cow_head" -A 12 net/core/skbuff.c include/linux/skbuff.h | head -30

5. Experiment

CLAIM. Most large packets are nonlinear, and cloning is what makes capture affordable.

METHOD.

sudo bpftrace -e '
kprobe:netif_receive_skb {
	$s = (struct sk_buff *)arg0;
	@nonlinear[$s->data_len > 0] = count();
	@frags = hist($s->data_len ? 1 : 0);
	@linear_bytes = hist($s->len - $s->data_len);
}' &
# Generate some large received packets:
curl -s -o /dev/null http://speed.hetzner.de/100MB.bin 2>/dev/null &
sleep 5; kill %1 %2 2>/dev/null

PREDICT FIRST: for a bulk TCP download, what fraction of received skbs are nonlinear? And how many bytes are in the linear part of a typical one?

Then measure cloning:

sudo bpftrace -e '
kretprobe:skb_clone   { @clones = count(); }
kprobe:skb_copy       { @copies = count(); }
kprobe:__pskb_pull_tail { @pulls = count(); }' &
# ...with and without tcpdump running:
sudo timeout 10 tcpdump -i any -w /dev/null 2>/dev/null &
ping -c 100 -i 0.05 -q "$(ip route | awk '/^default/{print $3;exit}')" >/dev/null 2>&1
kill %1 2>/dev/null

PREDICT FIRST: does tcpdump cause clones or copies? What would make it cause copies instead?

6. Failure mode

Concept 3: GRO and GSO

1. What problem it solves

At 10 Gb/s with 1500-byte frames, a machine receives about 800,000 packets per second. Every one costs a trip through the entire stack: routing, netfilter, TCP state machine, socket lock. That per-packet cost, not the bytes, is the bottleneck.

The fix is to make the stack see fewer, larger packets than the wire carries.

2. Where it exists in the kernel

rg -n "napi_gro_receive\b|dev_gro_receive" -A 20 net/core/gro.c net/core/dev.c 2>/dev/null | head -40
rg -n "skb_gso_segment|netif_needs_gso" net/core/dev.c | head
rg -n "SKB_GSO_TCPV4|gso_size|gso_segs" include/linux/skbuff.h | head -20
ethtool -k eth0 2>/dev/null | grep -E 'segmentation|offload|gro|gso|tso|lro'

3. The two directions

   GRO -- Generic Receive Offload (INBOUND)
     The driver's NAPI poll receives several small packets. GRO merges
     adjacent TCP segments of the same flow into ONE large skb (payload in
     frags) before handing it up.
       80 packets of 1500 B ──▶ 1 skb of 64 KB
     The stack is traversed ONCE instead of 80 times.
     Reversible: the skb remembers gso_size, so it can be re-segmented if
     it is forwarded.

   GSO -- Generic Segmentation Offload (OUTBOUND)
     TCP builds ONE large skb (up to 64 KB) and hands it down. Segmentation
     into MTU-sized frames happens as LATE as possible:
       - the NIC does it (TSO), if it can
       - otherwise the kernel does it just before ndo_start_xmit
     Everything above the device -- routing, netfilter, qdisc -- sees ONE
     packet instead of 45.

The performance argument is simply arithmetic: per-packet cost divided by 45.

ethtool -k eth0 | grep -E 'tcp-segmentation-offload|generic-receive-offload|generic-segmentation'
# Turn them off and measure. This is the experiment below.

4. Experiment

CLAIM. GRO and GSO are worth an order of magnitude in per-packet cost, and turning them off shows exactly how much.

METHOD. In the guest, between two veth peers or against another host:

IF=eth0
ethtool -k $IF | grep -E 'gro|gso|tso'

sudo bpftrace -e '
  kprobe:netif_receive_skb { $s = (struct sk_buff *)arg0;
                             @rx_skbs = count(); @rx_size = hist($s->len); }
  kprobe:dev_queue_xmit    { $s = (struct sk_buff *)arg0;
                             @tx_skbs = count(); @tx_size = hist($s->len); }' &

for setting in on off; do
  sudo ethtool -K $IF gro $setting gso $setting tso $setting 2>/dev/null
  echo "== offloads $setting"
  iperf3 -c <peer> -t 10 2>/dev/null | tail -4 || \
    curl -s -o /dev/null -w '%{speed_download}\n' http://<peer>/big
done
sudo ethtool -K $IF gro on gso on tso on
kill %1

PREDICT FIRST: with the offloads off, how many more netif_receive_skb calls for the same bytes? And what happens to the size histogram — where does it move?

Then look at the CPU cost:

sudo perf stat -e cycles,instructions -a -- sleep 10   # during each run
cat /proc/net/softnet_stat                             # time_squeeze column

5. Failure mode

Tip: "Turn off GRO/GSO/TSO and see if the problem persists" is the first diagnostic step for a large class of networking bugs, because it makes what the stack sees match what the wire carries. Remember to turn them back on.

Validation / Self-check

1. Name the four pointers in an sk_buff and what each delimits.
2. Which four functions move them, and in which direction does each move which pointer?
3. Why does a driver reserve headroom on a receive buffer?
4. After skb_pull past the IP header, how do you find the IP header again?
5. What does skb_under_panic mean, and what mistake causes it?
6. What is the difference between skb->len, skb->data_len, and the linear length?
7. What does pskb_may_pull do — both jobs — and what bug class does forgetting it create?
8. What does skb_clone share and what does it not? What must you check before writing to packet data?
9. Why is tcpdump cheap, and under what circumstances does it stop being cheap?
10. Explain GRO and GSO in one sentence each, and state the arithmetic that makes them worth it.
11. Why does tcpdump not show you wire packets when GRO is on?
12. What is the first thing to try when diagnosing an MTU or fragmentation bug, and why?

Next: The RX and TX Paths — from the wire to a socket, and back.