Level 7: Runtime and Shuffle

You have spent Level 6 above the seam, in Hive's world, watching queries become DAGs. Now you drop below the seam, into the code that runs inside the container: the runtime and the shuffle. This is the most performance-critical and most bug-prone code in Tez. It is where slow queries are actually slow, where data skew actually bites, and where a fetch failure actually originates. It is also where, as a contributor, you can finally write the fix rather than just route the ticket.

By the end of this level you can turn up shuffle logging on a real DAG, read the fetch/merge/spill log lines against the source that emits them, name the shuffle counters and reason about what moves them, induce and diagnose a fetch failure, and — the capstone — write a new Processor from scratch, wire it into a DAG, run it in local mode, and verify it with counters, the way a real runtime-library PR is built and tested.

Learning Objectives

By the end of Level 7 you must be able to:

1. Trace the full shuffle path — OrderedPartitionedKVOutput → sorter → spill → ShuffleHandler → FetcherOrderedGrouped → MergeManager → OrderedGroupedKVInput → processor — naming the class at every hop.
2. Turn up the real log4j categories for FetcherOrderedGrouped, ShuffleScheduler, MergeManager, and the sorters, and read their actual log lines back to the source that prints them.
3. Name the shuffle-related TaskCounter members and predict which way each moves when you change tez.runtime.io.sort.mb or induce a fetch failure.
4. Induce a fetch failure deterministically (Tez has a built-in fault injector) and follow the InputReadErrorEvent back up to the AM.
5. Distinguish the two halves of the runtime — the framework code in tez-runtime-library/tez-runtime-internals versus the user Processor — and know exactly what contract the framework offers the processor.
6. Write, wire, run, and test a new SimpleProcessor subclass the way a contributor would, including the counters and the test a PR must ship with.

How Shuffle Works in Tez

Unlike Hadoop MapReduce's monolithic ShuffleConsumerPlugin, Tez splits shuffle into framework code (tez-runtime-library) and user code (the Processor). The processor never sees unsorted records — partitioning, sorting, spilling, fetching, and merging all happen in the runtime layer before the processor's reader hands it a key.

PRODUCER (map-side)                          CONSUMER (reduce-side)
──────────────────                           ──────────────────────
Processor
  └─ KeyValueWriter
       └─ OrderedPartitionedKVOutput
            └─ sorter: PipelinedSorter /      OrderedGroupedKVInput
               DefaultSorter                    └─ ShuffleScheduler
                 └─ spill → IFile (data+idx)         └─ FetcherOrderedGrouped ──HTTP──┐
                      │                                    (parallel copies)          │
                      ▼                                        │                      │
                 ShuffleHandler (aux-service) ◀────────────────┘                      │
                 serves partitions over HTTP                                          ▼
                                                             MergeManager (mem + disk merge)
                                                               └─ KeyValuesReader → Processor

The two structural facts that matter for every shuffle bug:

• The sorter runs on the producer. By default tez.runtime.sorter.class is PIPELINED (PipelinedSorter); DefaultSorter is the alternative. Spills and the io.sort.mb buffer live here.
• The fetcher and merge run on the consumer. ShuffleScheduler runs a pool of FetcherOrderedGrouped threads that pull partitions over HTTP from the ShuffleHandler; MergeManager merges fetched segments in memory and on disk; the processor reads the merged, grouped result via KeyValuesReader.

Required Reading

This level is the hands-on companion to three deep dives — read them alongside the labs:

• Deep dive: Tez runtime internals — how the container boots a task and drives the processor.
• Deep dive: Shuffle and sort — the producer and consumer halves in detail.
• Deep dive: IPO abstractions — the Input/Processor/Output contracts your Lab 7.2 processor implements.
• This level's labs:
  • Lab 7.1 — Debug Shuffle Behavior
  • Lab 7.2 — Modify a Processor: Add Deduplication

Source Areas

tez-runtime-library — the shuffle and IPO implementations

All paths under tez-runtime-library/src/main/java/org/apache/tez/runtime/library/.

# Confirm the ordered-grouped shuffle classes exist:
ls tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/

tez-runtime-internals — the in-container task driver

All paths under tez-runtime-internals/src/main/java/org/apache/tez/runtime/.

# The class that runs one task inside the container:
find tez-runtime-internals/src/main/java -name LogicalIOProcessorRuntimeTask.java

Note: tez-runtime-library is where you add features (new outputs, inputs, processors, sorter tweaks); tez-runtime-internals is the driver that calls them. Lab 7.2 adds to the library; the driver runs your processor unchanged.

Key Classes — Quick Reference

Deliverables

Before advancing to Level 8, you must produce:

• A shuffle-log reading: a real shuffle-heavy DAG run with fetch, merge, and spill log lines captured and each mapped to the class/method that emitted it (Lab 7.1).
• A counter table showing how SHUFFLE_BYTES, SPILLED_RECORDS, NUM_SHUFFLED_INPUTS, and ADDITIONAL_SPILL_COUNT moved when you changed tez.runtime.io.sort.mb (Lab 7.1).
• A diagnosed, induced fetch failure: the injector config you used, the failure log line, and the InputReadErrorEvent path up to the AM (Lab 7.1).
• A working DedupProcessor: full Java, wired into a two-vertex DAG, run in local mode, output verified by a counter — plus the unit test a PR would ship (Lab 7.2).
• From memory: the difference between the ordered-grouped and unordered reader, and why deduplication is free on one and needs a Set on the other.

Common Mistakes

How to Verify Success

# 1) You can name the shuffle counters from the source of truth.
grep -nE "SHUFFLE_BYTES|SPILLED_RECORDS|NUM_SHUFFLED_INPUTS|ADDITIONAL_SPILL_COUNT" \
  tez-api/src/main/java/org/apache/tez/common/counters/TaskCounter.java

# 2) You can point at the fault injector you'll use in Lab 7.1.
grep -rn "shuffle.fetch.testing.errors" \
  tez-runtime-library/src/main/java/org/apache/tez/runtime/library/api/TezRuntimeConfiguration.java

# 3) You can show the processor base class Lab 7.2 extends.
grep -rn "abstract class SimpleProcessor" \
  tez-runtime-library/src/main/java/org/apache/tez/runtime/library/processor/SimpleProcessor.java

When each command lands and you can explain what it proves, you understand the runtime well enough to modify it.

PR Profile: Level 7 Graduate

A contributor who completes this level can credibly:

• Diagnose a shuffle bug from logs and counters and file a report that names the class (FetcherOrderedGrouped, MergeManager) and the counter evidence, not just "shuffle is slow."
• Tune and defend a shuffle setting (io.sort.mb, shuffle.parallel.copies, shuffle.fetch.failures.limit) with counter deltas as evidence.
• Add a runtime-library feature — a new Processor, a counter, a small output/input tweak — with the unit test and local-mode validation that Tez reviewers require.
• Review a shuffle PR for the right things: does it move a counter it claims to? does it have a test that simulates the fetch/merge path rather than mocking it away?

You can now change Tez's runtime, not just read it. That is the last skill before Level 8, where you take a real reported issue from reproduction to a merged fix.

Next: Level 8 — Reproduce and Fix a Real Issue →