Open-Source Engineer & Contributor

A collection of deep, implementation-level curricula for engineers who want to contribute seriously to major open-source projects — not just fix typos, but build the kind of sustained understanding that leads to committer status.

Each curriculum is designed around how the project is actually developed, tested, reviewed, and maintained by its core contributors. Labs reference real source code, real issue trackers, and real contribution workflows.

Curricula

How to Use This Book

Each curriculum is self-contained. Start at the curriculum's Introduction page and work through its levels sequentially. Levels build on each other — skipping levels skips foundations that later labs depend on.

What you will need for any curriculum:

• 3+ years of Java (or the project's primary language) on production-grade codebases
• Comfort reading large, unfamiliar codebases without a guide
• Git, a build tool (Maven / Gradle / sbt), and an IDE (IntelliJ recommended)
• Patience: the path from contributor to committer is measured in months to years

Select a curriculum from the table above or from the sidebar to begin.

Apache Tez Open-Source Contributor Curriculum

Welcome to the Apache Tez Open-Source Contributor Curriculum — a complete, implementation-heavy roadmap for engineers who want to become serious Apache Tez contributors and eventually operate at the level of a core contributor, committer, or PMC-aware engineer.

What This Curriculum Is

This is not a tutorial. It is a structured engineering apprenticeship built around how Apache Tez is actually developed, tested, reviewed, and maintained by its committers and PMC members.

Every level is tied to real Apache Tez source code, real JIRA issue patterns, real test infrastructure, and real contribution workflows. The labs mirror the work an Apache Tez committer actually does — reading state machine code, tracing DAG execution paths, debugging shuffle failures, reproducing reported issues, and preparing patches for community review.

The curriculum will not hold your hand. It will point you at the right parts of the codebase, give you the right questions to ask, and push you to develop the muscle memory of someone who works at this level habitually.

Who This Is For

This curriculum is designed for strong backend and distributed systems engineers who:

• Have 3+ years of Java development experience (Maven-based projects)
• Are familiar with Hadoop, YARN, or MapReduce at a conceptual level
• Understand distributed systems fundamentals: scheduling, fault tolerance, partitioning, shuffle
• Want to contribute to Apache open-source at a serious level — not just fix typos

You should be comfortable with:

• Reading large, unfamiliar Java codebases without a guide
• git workflows, reading diffs, working with patch-based reviews
• The Hadoop ecosystem at a high level: YARN, HDFS, MapReduce, Hive
• Distributed execution concepts: task graphs, data movement, speculative execution

What You Will Be Able to Do

After completing this curriculum, you will be able to:

How to Use This Curriculum

Work through the 9 levels sequentially. Do not skip levels. Each level builds directly on the previous one, and the labs depend on the conceptual foundations laid earlier.

Beyond the 9 levels, the curriculum includes five additional sections:

The curriculum closes with a Capstone Project — a full contribution cycle from issue reproduction to merged patch and engineering write-up.

Required Tools

Before starting Level 1, ensure you have the following installed and working:

Java 8 or Java 11 (OpenJDK recommended — match the Tez branch target)
Apache Maven 3.6.3 or newer
Git 2.x
IntelliJ IDEA (strongly recommended) or Eclipse with M2E
Docker (optional — useful for containerized mini-cluster environments)

You will also need:

• A clone of the Apache Tez repository (GitHub mirror of the Apache GitBox repo)
• A clone of the Apache Hadoop repository (for YARN API context and integration reference)
• An account on Apache JIRA (free to create)
• Subscription to the Apache Tez mailing lists:
  • dev@tez.apache.org — development discussion (required)
  • issues@tez.apache.org — JIRA notifications (optional but useful)

Note on Java version: Apache Tez's master branch targets Java 8 as the minimum. Some newer branches may require Java 11. Always check the pom.xml at the root of the branch you are working on.

Apache Tez at a Glance

Apache Tez is a general-purpose DAG execution engine built on top of Apache YARN. It is the primary execution engine for Apache Hive since Hive 0.13, and is used by other Hadoop ecosystem projects including Pig, Cascading, and Spark (historically).

Why Tez Exists

MapReduce forces every computation into a Map → Shuffle → Reduce pattern. Complex analytical queries (like multi-join SQL) require chaining many MapReduce jobs, with intermediate results written to HDFS between each stage. This is slow and wasteful.

Tez allows arbitrary directed acyclic graphs (DAGs) of computation where:

• Vertices represent computation stages
• Edges represent data movement between stages
• Container reuse eliminates JVM startup overhead between tasks
• Data can be pipelined between tasks without HDFS materialization
• The same container can run multiple task types

This makes Tez significantly faster than MapReduce for multi-stage queries.

Key Modules

You will spend the majority of your time in these modules:

Key Classes (High-Level Preview)

Apache Tez Community

Apache Tez is a mature project with an active but selective community. The codebase reflects years of careful design decisions, many of which are documented in JIRA issues, design documents, and mailing list threads rather than in code comments.

What the community values:

• Patches that include tests
• Issues that include a clear reproduction case
• Comments that demonstrate you have read the existing code
• Contributors who engage respectfully and patiently
• Sustained contribution over time, not one-off patches

The path from contributor to committer is measured in years, not weeks. That is intentional. The Apache meritocracy rewards sustained, high-quality contribution — not volume of patches.

This curriculum will help you build the habits and depth of understanding that make that path realistic.

Begin with Level 1: Hadoop and Tez Foundation.

Overview & Prerequisites

Status: Full content coming in Phase 11.

This section covers the complete prerequisites checklist, environment setup guide, and how to navigate the curriculum effectively.

Topics covered:

• Detailed environment setup (Java, Maven, Git, IDE configuration)
• Cloning and verifying the Apache Tez and Hadoop repositories
• Subscribing to Apache Tez mailing lists
• Setting up an Apache JIRA account
• How to navigate each curriculum level
• How to use the labs

Tez Warm-Up: From Data Engineer to Source Contributor

Before you read a single line of VertexImpl.java, you need to have sat in the seat of the person whose workload Tez is serving. The engineers who built Tez's state machines, container reuse logic, and shuffle pipelines were solving specific, painful problems that showed up in production Hive and Pig workloads every day. If you skip that context and go straight to the source code, you will memorize class names without understanding why the design exists.

This chapter is the missing first mile. You will run Tez from the outside — as a data engineer would — across a series of practical scenarios covering different data shapes, query patterns, and ecosystem integrations. After each scenario, the chapter maps what you observed back to the source code structures that own it. By the end, you will have a mental model that makes every internal class feel like an old acquaintance rather than an alien term.

What Tez Actually Is (Two Sentences)

Apache Tez is a general-purpose DAG execution engine that runs on Apache YARN. It does not execute SQL or process files itself — it provides a runtime that other systems (Hive, Pig, custom applications) compile their work into, and then Tez runs that compiled work as a directed acyclic graph of parallel tasks.

Everything else — SQL parsing, query planning, physical operators, file format codecs — belongs to the caller. Tez sees vertex descriptors, edge properties, and processor classes. That boundary is what you need to hold clearly in mind throughout the curriculum.

Where Tez Sits in the Data Engineering Spectrum

┌─────────────────────────────────────────────────────────────────────────────┐
│                     Data Engineering Tool Spectrum                          │
│                                                                             │
│  Batch ◄───────────────────────────────────────────────► Streaming         │
│                                                                             │
│  MapReduce    Tez      Spark         Flink         Kafka Streams            │
│  (2004)       (2013)   (2014)        (2014)        (2016)                   │
│  pure batch   batch+   micro-batch   true stream   native stream            │
│               pipelined & batch      & batch                                │
│                                                                             │
│  ──────────────────────────────────────────────────────────────────────     │
│  Ingest Layer:  Flume  →  Kafka  →  Flink/Kafka Streams                    │
│  Storage Layer: HDFS / S3 / ORC / Parquet / Iceberg / Delta Lake           │
│  Query Layer:   Hive (Tez), Presto, Trino, Spark SQL, Flink SQL            │
└─────────────────────────────────────────────────────────────────────────────┘

Tez vs. MapReduce

MapReduce forces every computation into map → shuffle → reduce. A five-join SQL query becomes five chained MapReduce jobs with HDFS materializations between each. Tez expresses that same query as one DAG, pipelines intermediate data between vertices without HDFS writes, and reuses JVMs across tasks. Typical improvement: 2–5x on complex queries, 10x+ on workflows that would have been five MR jobs.

Tez vs. Spark

When you are on a Hadoop/YARN cluster where Hive is the primary SQL layer, Tez is the right choice. Spark is a better fit for Python/Scala workloads, ML pipelines, or when you need in-memory caching across multiple queries.

Tez vs. Flink

Flink is a streaming-first engine that also handles batch. Tez is a batch-first engine that handles simple pipelines. The key structural difference: Flink maintains persistent operator state across windows and checkpoints; Tez vertices are stateless per-task (state is external: HDFS, HBase). If you are building event-time windowed aggregations or exactly-once stream processing, you want Flink. If you are running nightly ETL on HDFS data via Hive, Tez is the right tool and Flink would be overengineered for the job.

Tez vs. Flume (Ingest)

Flume is not a computation engine — it is a log/event ingestion agent that moves data from sources (web servers, syslog, Kafka) to sinks (HDFS, Kafka, HBase). The typical pipeline is:

Application Logs → Flume Agent → HDFS (ORC/Parquet files) → Hive table → Tez query

Flume and Tez are not competitors; they are peers in the same pipeline. Tez reads the data that Flume (or Kafka, or Sqoop) landed on HDFS. Knowing this boundary matters when you encounter a data quality bug: is it in the ingest (Flume), the storage format (ORC serialization), or the compute layer (Tez/Hive)?

Data Formats in the Tez Ecosystem

Tez itself is format-agnostic. It does not read or write ORC, Parquet, or Iceberg directly. Tez sees InputDescriptor and OutputDescriptor objects — the actual codec lives in the class pointed to by those descriptors. The format lives in the tez-mapreduce compatibility layer (MRInput, MROutput) or in Hive's vectorized readers.

ORC (Optimized Row Columnar)

ORC is Hive's native format. When you INSERT INTO an ORC table and query it via Hive-on-Tez:

• The input split is an OrcSplit generated by OrcInputFormat in the Hive ORC library.
• Tez receives that split as a DataSourceDescriptor in the DAGPlan.
• MRInput wraps OrcInputFormat.createRecordReader(), feeding vectorized row batches to Hive's MapOperator.
• The key Tez entry point is MRInputLegacy.createReaderInternal() in tez-mapreduce/src/main/java/org/apache/tez/mapreduce/input/MRInputLegacy.java.

ORC's predicate pushdown (column pruning, row group skipping) happens before Tez sees the data — entirely inside OrcInputFormat. If a Hive-on-Tez query reads 10 billion rows instead of the 100K it should (wrong predicate pushdown), the bug is in ORC/Hive, not in Tez.

Parquet

Parquet is the other dominant columnar format, more common in cross-ecosystem pipelines (Spark + Hive interop). With Hive-on-Tez reading Parquet:

• ParquetInputFormat generates ParquetInputSplit objects.
• Tez receives those as DataSourceDescriptor entries.
• Vectorization depth varies: ORC vectorization is deeper in Hive; Parquet goes through an additional row-column translation layer.

From a Tez contributor's standpoint, Parquet vs. ORC differences show up mainly in:

1. Split size calculations affecting vertex parallelism (how many map tasks Tez schedules)
2. Record skew when one Parquet file is much larger than others

Iceberg

Apache Iceberg is a table format (not a file format). It stores data in Parquet or ORC files but adds a metadata layer for ACID semantics, time travel, and hidden partitioning. Hive + Tez reads Iceberg via IcebergInputFormat (from the Iceberg Hive runtime JAR).

From Tez's view, Iceberg is yet another InputFormat. The novel behavior is:

• Iceberg's snapshot-based read means splits can come from multiple physical locations.
• Iceberg's PlanningUtil generates splits that can be much more numerous than traditional partition-based splits — this affects Tez vertex parallelism significantly.
• Time-travel queries (SELECT ... FOR SYSTEM_TIME AS OF ...) generate a different split list at query compile time, which Hive encodes into the DAGPlan before Tez sees it.

Key insight for contributors: Tez bugs triggered by Iceberg tables are almost always about parallelism (too many small tasks, too few tasks for large snapshots) or about the DataSourceDescriptor encoding. The actual file reading is not Tez's responsibility.

Scenario 1: Classic Batch ETL — Aggregation Over a Large Table

What the data engineer does:

-- Run in Hive CLI connected to a cluster with Hive-on-Tez enabled
SET hive.execution.engine=tez;

CREATE TABLE daily_sales (
  event_date STRING,
  product_id BIGINT,
  region     STRING,
  revenue    DOUBLE
)
STORED AS ORC
TBLPROPERTIES ("orc.compress"="ZSTD");

-- Query: daily revenue by region, last 90 days
SELECT
  event_date,
  region,
  SUM(revenue)     AS total_revenue,
  COUNT(*)         AS transaction_count
FROM daily_sales
WHERE event_date >= '2026-03-01'
GROUP BY event_date, region
ORDER BY event_date, region;

What Tez does under the hood:

1. Hive compiles the query to MapWork (map-side partial aggregation) + ReduceWork (global aggregation + sort).
2. DagUtils.createVertex() in Hive creates two Tez Vertex objects: Map 1 and Reducer 2.
3. The edge between them is SCATTER_GATHER (partitioned shuffle by GROUP BY key hash).
4. ShuffleVertexManager auto-parallelism kicks in: it monitors how much data map tasks produce, then dynamically reduces the reducer count if data is smaller than expected (config: tez.shuffle-vertex-manager.desired-task-input-size).
5. Map tasks run MapProcessor → HashTableContainer (partial agg) → OrderedPartitionedKVOutput (partitioned, sorted).
6. Reducer tasks run ReduceProcessor → OrderedGroupedKVInput (merge shuffle inputs) → PTFOperator (for ORDER BY) → FileSinkOperator → ORC writer.

Dataset characteristics and edge behaviors:

Bridge to source code:

cd ~/tez-src

# ShuffleVertexManager — the most important vertex manager for map-reduce style DAGs
find . -name "ShuffleVertexManager.java" -path "*/tez-dag/*"
# tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/ShuffleVertexManager.java

# Auto-parallelism: how ShuffleVertexManager decides to reduce the number of reducers
grep -n "computeParallelism\|desiredTaskInputSize\|onSourceTaskCompleted" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/ShuffleVertexManager.java | head -30

# The edge between Map 1 and Reducer 2 is SCATTER_GATHER — EdgeProperty documentation
grep -n "SCATTER_GATHER" \
  tez-api/src/main/java/org/apache/tez/dag/api/EdgeProperty.java

Scenario 2: Multi-Table Join — The Real Workload Tez Was Built For

What the data engineer does:

SET hive.execution.engine=tez;
SET hive.auto.convert.join=true;      -- enable map-side (broadcast) joins
SET hive.mapjoin.smalltable.filesize=25000000;  -- 25 MB threshold

SELECT
  o.order_id,
  c.customer_name,
  p.product_name,
  o.quantity * p.unit_price AS line_total
FROM orders          o
JOIN customers       c ON o.customer_id = c.customer_id
JOIN products        p ON o.product_id  = p.product_id
WHERE o.order_date = '2026-05-31'
AND   c.region = 'US-WEST';

What Tez does:

Hive's query planner analyzes table sizes:

• customers and products are small (< 25 MB) → broadcast join (MapJoin)
• orders is large (> 25 MB) → the probe side, goes through SCATTER_GATHER shuffle

The resulting DAG has:

1. Map 1 — reads orders, builds hash table from customers and products small tables, emits matching rows. Small tables arrive via a BROADCAST edge (ONE_TO_ONE semantics: every map task gets the full small table).
2. Optionally a Reducer 2 if there's a DISTINCT or ORDER BY.

VertexGroup for broadcast joins: Hive uses VertexGroup to express that one physical vertex's output goes to both Map 1 and any other map-side consumer. This is expressed via DAG.addVertex() with a VertexGroup wrapper.

Dataset edge cases for joins:

Bridge to source code:

# ONE_TO_ONE edge (broadcast) — how every map task gets all small-table data
grep -n "ONE_TO_ONE\|BroadcastEdgeManager" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/ -r | grep -v Test | head -20

# VertexGroup — Hive's mechanism for fan-out to multiple consumers
grep -n "class VertexGroup\|addVertexGroup" \
  tez-api/src/main/java/org/apache/tez/dag/api/DAG.java | head -15

# How the DAGAppMaster sees both edges from the same vertex
grep -n "vertexGroup\|groupInput" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java | head -20

Scenario 3: Direct Tez API — No Hive

Not all Tez workloads go through Hive. Custom data pipelines, internal batch frameworks, and migration tools often build Tez DAGs directly. The canonical example is OrderedWordCount in tez-examples/.

// Simplified from tez-examples/src/main/java/org/apache/tez/examples/OrderedWordCount.java
TezClient tezClient = TezClient.create("OrderedWordCount", tezConf);
tezClient.start();

DAG dag = DAG.create("OrderedWordCount");

// Vertex 1: Tokenize words from input files
Vertex tokenizerVertex = Vertex.create(
    "Tokenizer",
    ProcessorDescriptor.create(TokenProcessor.class.getName()),
    numMapTasks,
    MRHelpers.getMapResource(conf));
tokenizerVertex.addDataSource(
    "Input",
    MRInput.createConfigBuilder(conf, TextInputFormat.class, inputPath).build());

// Vertex 2: Sort and deduplicate
Vertex sumVertex = Vertex.create(
    "Sorter",
    ProcessorDescriptor.create(SumProcessor.class.getName()),
    numReduceTasks,
    MRHelpers.getReduceResource(conf));
sumVertex.addDataSink(
    "Output",
    MROutput.createConfigBuilder(conf, TextOutputFormat.class, outputPath).build());

// Edge: SCATTER_GATHER, sorted by word key
dag.addVertex(tokenizerVertex)
   .addVertex(sumVertex)
   .addEdge(Edge.create(tokenizerVertex, sumVertex, EdgeProperty.create(
       DataMovementType.SCATTER_GATHER,
       DataSourceType.PERSISTED,
       SchedulingType.SEQUENTIAL,
       OrderedPartitionedKVOutput.createConfigBuilder(conf, HashPartitioner.class).build(),
       OrderedGroupedKVInput.createConfigBuilder(conf).build())));

DAGClient dagClient = tezClient.submitDAG(dag);
DAGStatus status = dagClient.waitForCompletion();

What this teaches about Tez's structure:

• Vertex.create(name, processorDescriptor, parallelism, resource) — the four primitives of a vertex: name, code to run, how many copies, how much resource.
• EdgeProperty.create(movementType, sourceType, schedulingType, outputDesc, inputDesc) — edge properties completely specify how data moves.
• MRInput/MROutput bridge the gap between legacy Hadoop InputFormat/OutputFormat and Tez's native I/O descriptors.

Bridge to source code:

# Read OrderedWordCount to understand the complete DAG lifecycle from a client
cat tez-examples/src/main/java/org/apache/tez/examples/OrderedWordCount.java

# Follow TezClient.submitDAG() into the AM
grep -n "public.*submitDAG" \
  tez-api/src/main/java/org/apache/tez/dag/api/client/TezClient.java

# EdgeProperty — the central struct that determines routing
cat tez-api/src/main/java/org/apache/tez/dag/api/EdgeProperty.java

Scenario 4: Pipelined Execution — Where Tez Approaches Flink

Tez supports PIPELINED edge scheduling (vs. SEQUENTIAL). With pipelined edges, downstream tasks can start before all upstream tasks complete — the data flows like a stream within the DAG.

EdgeProperty pipelinedEdge = EdgeProperty.create(
    DataMovementType.SCATTER_GATHER,
    DataSourceType.PERSISTED_PIPELINED,    // <-- pipelined
    SchedulingType.CONCURRENT,             // <-- downstream starts immediately
    outputDescriptor,
    inputDescriptor);

This is used by Hive for query pipelining in long-running SELECT ... INSERT chains. The downstream vertex starts consuming partial output from the upstream before it finishes, reducing end-to-end latency for multi-stage queries.

When pipelining causes problems:

Bridge to source code:

grep -n "PERSISTENT_PIPELINED\|PIPELINED\|CONCURRENT" \
  tez-api/src/main/java/org/apache/tez/dag/api/EdgeProperty.java

grep -n "SchedulingType.CONCURRENT" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head -10

Dataset Scenarios for Testing Edge Cases

When you are writing a repro test case or validating a fix, the dataset you choose determines which code paths you exercise. Use these as your starting templates.

Dataset 1: The Empty Partition

// Generate test data where one reduce partition has 0 records
// Triggers ShuffleVertexManager.onSourceTaskCompleted() with 0-byte output
private static final int NUM_PARTITIONS = 10;
private static final int RECORDS_PER_PARTITION = 100;

// Force all records into partitions 0–8, leave partition 9 empty
int partition = key.hashCode() % (NUM_PARTITIONS - 1);  // never 9

What this tests: ShuffleVertexManager must handle a vertex where some reducer partitions receive zero input. Before TEZ-3247, this caused reducers to hang waiting for shuffle data that would never arrive.

# Test class that covers empty-partition behavior
grep -rn "emptyPartition\|zeroInput\|emptyInput" tez-tests/src/test/ | head -10

Dataset 2: Extreme Key Skew

// One key accounts for 95% of records
for (int i = 0; i < 1_000_000; i++) {
    String key = (i < 950_000) ? "hot_key" : "key_" + i;
    writer.write(new Text(key), new IntWritable(1));
}

What this tests: The reducer that receives hot_key gets ~950,000 records while other reducers get ~50 each. This exposes:

• Speculative execution decisions in LegacySpeculator
• Container reuse after the skewed reducer finishes last
• Per-vertex timing in VertexImpl.checkTasksForCompletion()

Dataset 3: Zero-Row Input

// Empty input — 0 files, 0 records
// The DAG should complete SUCCEEDED with 0 output, not hang
String inputPath = "/tmp/empty_dir_" + UUID.randomUUID();
fs.mkdirs(new Path(inputPath));  // create directory but put no files in it

What this tests: VertexImpl must handle the case where MRInput generates 0 splits. A vertex with 0 input splits sets its parallelism to 0, transitions immediately to V_SUCCEEDED without scheduling any tasks. This has historically been a source of NullPointerException bugs when downstream vertices assume at least one upstream task ran.

grep -n "setParallelism.*0\|numTasks.*0\|zeroTasks\|numSourceTasks.*0" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head -15

Dataset 4: Very Wide Rows (Many Columns)

// 1,000 columns per row — stresses IFile serialization and spill logic
StringBuilder sb = new StringBuilder();
for (int col = 0; col < 1000; col++) {
    sb.append("column_").append(col).append("=").append("value_").append(col).append("\t");
}
writer.write(new Text("key"), new Text(sb.toString()));  // ~30 KB per record

What this tests: PipelinedSorter and DefaultSorter spill thresholds. With 30 KB per record, even a modest sort buffer fills quickly. This exercises the spill path in tez-runtime-library/src/main/java/org/apache/tez/runtime/library/output/OrderedPartitionedKVOutput.java and exposes off-by-one bugs in the IFile index writer.

Dataset 5: Many Small Files (HDFS Small-File Problem)

# Generate 50,000 files of 1 KB each — a classic HDFS anti-pattern
for i in $(seq 1 50000); do
  echo "record_$i value_$i" > /tmp/smallfiles/file_$i.txt
done
hadoop fs -put /tmp/smallfiles /data/input/smallfiles/

What this tests: Split generation produces 50,000 map tasks. This is a realistic workload that stresses:

• TaskSchedulerManager task queue management
• Container reuse logic (50,000 containers → reuse is essential for performance)
• DAGAppMaster AMRM heartbeat frequency under high task count

# Container reuse configuration
grep -n "heldContainer\|releaseTimeout\|IDLE_TIMEOUT" \
  tez-dag/src/main/java/org/apache/tez/dag/app/launcher/ContainerLauncherImpl.java \
  | head -20

Dataset 6: Nested Structs (Complex Types)

-- ORC table with nested complex types
CREATE TABLE events (
  event_id  BIGINT,
  metadata  STRUCT<
    user_id:       BIGINT,
    session_id:    STRING,
    properties:    MAP<STRING, STRING>,
    tags:          ARRAY<STRING>
  >,
  timestamp BIGINT
) STORED AS ORC;

What this tests: ORC vectorized reader deserialization of STRUCT, MAP, and ARRAY types. These types are serialized into Hive's OrcStruct/OrcMap/OrcList classes before being passed through MRInput to the MapOperator. If the column count or type tree changes between what the ORC file was written with and what the Hive schema says, you get schema evolution behavior — which can generate bugs that look like Tez data corruption but are actually ORC schema evolution issues.

Dataset 7: Partitioned Iceberg Table (Snapshot Isolation)

# Using PyIceberg or Spark to create an Iceberg table with multiple snapshots
from pyiceberg.catalog import load_catalog
catalog = load_catalog("hive_catalog", **{"uri": "thrift://hive-metastore:9083"})
table = catalog.load_table("db.events_iceberg")

# Write 3 snapshots representing 3 days of appends
for day in range(3):
    df = generate_day_data(day)
    table.append(df)

# Now query with time travel — Hive generates a DAGPlan that reads snapshot 1
hive_execute("""
    SELECT COUNT(*) FROM db.events_iceberg
    FOR SYSTEM_TIME AS OF '2026-05-29 00:00:00'
""")

What this tests: Iceberg's IcebergInputFormat generates a split list that differs per snapshot. The DataSourceDescriptor passed to Tez encodes the snapshot ID. If Hive resolves the wrong snapshot, Tez faithfully executes it — the bug is in the DagUtils snapshot resolution in Hive, not in Tez. But the symptom (wrong row count) looks like a Tez data bug.

Running Tez End-to-End: The Local Developer Loop

Before writing source code, every Tez contributor should be able to do this loop in under 10 minutes:

# 1. Clone and build
git clone https://github.com/apache/tez.git ~/tez-src
cd ~/tez-src
mvn clean install -DskipTests -Pdist -q   # ~8–12 min cold, 3–4 min warm

# 2. Run the canonical integration test that exercises the full stack
mvn test -pl tez-tests \
    -Dtest=TestOrderedWordCount \
    -DfailIfNoTests=false 2>&1 | tail -30

# 3. Run a single unit test (fast feedback loop — use this constantly)
mvn test -pl tez-dag \
    -Dtest=TestVertexImpl#testVertexSucceededSpeculation \
    -DfailIfNoTests=false 2>&1 | tail -20

# 4. Run OrderedWordCount in local mode (no YARN cluster required)
hadoop jar tez-examples/target/tez-examples-*.jar orderedwordcount \
    -D tez.local.mode=true \
    /path/to/input /tmp/tez-output-$(date +%s)

# 5. Verify output
hadoop fs -cat /tmp/tez-output-*/part-* | sort | head -20

The TestOrderedWordCount test is your baseline health check. If it passes, the full end-to-end stack (TezClient → DAGAppMaster → VertexImpl → shuffle → MRInput/MROutput) is working. If it fails, something fundamental is broken and you need to fix that before touching anything else.

The Bridge: User Scenario → Source Code

Every scenario above maps to a specific source subsystem. Use this table whenever you see a runtime behavior and want to find the code responsible:

What to Verify Before Starting Level 1

Run through this checklist once. It takes 30–45 minutes and proves your environment is solid.

# Environment check
java -version    # must be Java 8 or Java 11
mvn -version     # must be 3.6.3+
git --version    # must be 2.x

# Clone and build
git clone https://github.com/apache/tez.git ~/tez-src
cd ~/tez-src
mvn clean install -DskipTests -Pdist 2>&1 | tail -10

# Confirm build artifacts exist
ls tez-dist/target/tez-*.tar.gz  # should exist
ls tez-examples/target/tez-examples-*.jar

# Run the unit test suite in the two most important modules
mvn test -pl tez-dag -DfailIfNoTests=false 2>&1 | grep -E "Tests run:|FAIL|ERROR" | tail -5
mvn test -pl tez-api -DfailIfNoTests=false 2>&1 | grep -E "Tests run:|FAIL|ERROR" | tail -5

# Run the critical end-to-end test
mvn test -pl tez-tests -Dtest=TestOrderedWordCount -DfailIfNoTests=false 2>&1 | tail -10

# All lines should read "Tests run: N, Failures: 0, Errors: 0"

If any of these fail before you have modified a single line of code, stop and fix your environment. Do not proceed into Level 1 with a broken baseline. A broken baseline means every subsequent mvn test will produce false failures that obscure the real work.

Continue to Overview & Prerequisites or jump directly to Level 1: Hadoop and Tez Foundation.

16-Week Plan: From Curious Reader to Tez Committer Candidate

This is a 16-week, ~10-hour-per-week plan that maps the curriculum (Levels 1–9 plus a 2-week capstone) onto a calendar. Each week states:

• Reading — concrete Tez source files. Open them; do not just skim diagrams.
• Hands-on — what you must build/run on your machine.
• JIRA practice queries — searches that surface real, beginner-appropriate issues.
• Labs — the curriculum labs you must complete.
• Exit checkpoint — concrete deliverables. If you cannot produce them, repeat the week.

The plan assumes you have ~/tez-src checked out, tez-tests/ building with mvn -DskipTests install, and a working Java 8+/Maven 3.6+ environment.

Weeks 1–2: Level 1 — Orientation and First DAG

Week 1 — The DAG model and the client API

Reading

• tez-api/src/main/java/org/apache/tez/dag/api/DAG.java (entire file; ~600 lines)
• tez-api/src/main/java/org/apache/tez/dag/api/Vertex.java
• tez-api/src/main/java/org/apache/tez/dag/api/Edge.java
• tez-api/src/main/java/org/apache/tez/dag/api/EdgeProperty.java
• tez-api/src/main/proto/DAGApiRecords.proto — focus on DAGPlan, VertexPlan, EdgePlan, EdgeProperty.

Hands-on

• Build Tez from source: mvn clean install -DskipTests -Phadoop28.
• Run OrderedWordCount against a local file using MiniTezCluster (see tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java).
• Inspect the generated DAGPlan: print it with dag.createDag(...).toString().

JIRA practice queries

project = TEZ AND status in (Open, "In Progress") AND labels = newbie
project = TEZ AND component = tez-api AND fixVersion is empty AND priority in (Trivial, Minor)

Labs

• Lab 1.1 — Trace a WordCount end-to-end.
• Lab 1.2 — Modify the DAG: add a second mapper vertex.

Exit checkpoint

• You can name every required argument to DAG.create(), Vertex.create(), Edge.create(), and EdgeProperty.create().
• You can diagram the WordCount DAG without looking.
• You have one JIRA ticket open in a browser tab that you've read end-to-end (description + every comment).

Week 2 — Edges in depth

Reading

• tez-api/src/main/java/org/apache/tez/dag/api/EdgeProperty.java — all three enums (DataMovementType, DataSourceType, SchedulingType).
• tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/EdgeManager*.java — five built-in edge managers.
• tez-api/src/main/java/org/apache/tez/dag/api/InputDescriptor.java, OutputDescriptor.java, ProcessorDescriptor.java.

Hands-on

• Build the same WordCount with BROADCAST instead of SCATTER_GATHER for the edge. Observe the failure mode and explain it.
• Write a 3-vertex DAG (A -> B -> C) where A->B is ONE_TO_ONE and B->C is SCATTER_GATHER. Run it; confirm parallelism rules from the source.

JIRA practice queries

project = TEZ AND text ~ "EdgeManager" AND resolution = Unresolved
project = TEZ AND text ~ "broadcast" AND status = Resolved ORDER BY created DESC

Labs

• Lab 1.3 — Edge type matrix experiment.

Exit checkpoint

• Edge type matrix (movement × scheduling × source) drawn from memory.
• You can predict, given edge properties, which EdgeManager impl will be picked.
• One short forum/dev-list email you drafted (do not send) summarizing your reading of an EdgeManager file.

Weeks 3–4: Level 2 — Build, run, and read tests

Week 3 — Tez build system and module layout

Reading

• pom.xml (root), tez-api/pom.xml, tez-dag/pom.xml.
• BUILDING.txt.
• tez-tests/src/test/java/org/apache/tez/test/MiniTezCluster.java — entry-point for nearly every integration test.

Hands-on

• Run mvn -pl tez-dag test -Dtest=TestVertexImpl#testBasicVertexCompletion.
• Run mvn -pl tez-tests test -Dtest=TestTezJobs#testWordCount.
• Profile a build: mvn -DskipTests install -X 2>&1 | grep "Building\|BUILD".

JIRA practice queries

project = TEZ AND component = build AND status = Open
project = TEZ AND text ~ "MiniTezCluster" AND resolution = Unresolved

Labs

• Lab 2.1 — Build Tez and run all tez-api tests.
• Lab 2.2 — Add a no-op test to tez-dag and run it via Maven.

Exit checkpoint

• You can explain why tez-dag depends on tez-api but not vice versa.
• You know the difference between tez-runtime-internals and tez-runtime-library.
• You can run a single test via Maven without consulting any docs.

Week 4 — Tests as documentation

Reading

• tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java (~5000 lines; pick the top 10 test methods).
• tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestDAGImpl.java.
• tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestTaskImpl.java.

Hands-on

• Pick one test method in TestVertexImpl; rewrite it from scratch in your notebook, then diff against the original.
• Add an assertion that fails; observe the message; fix it.

JIRA practice queries

project = TEZ AND text ~ "flaky" AND status in (Open, "In Progress")
project = TEZ AND text ~ "TestVertexImpl" AND resolution = Unresolved

Labs

• Lab 2.3 — Read TestVertexImpl#testKilledTasksHandling and explain every line.

Exit checkpoint

• You can write a test that constructs a VertexImpl directly (without MiniTezCluster).
• You understand the DrainDispatcher pattern (see state-machines.md).

Weeks 5–6: Level 3 — Submission and AM lifecycle

Week 5 — TezClient and submission

Reading

• tez-api/src/main/java/org/apache/tez/client/TezClient.java.
• tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java.
• tez-api/src/main/java/org/apache/tez/client/TezSessionImpl.java.

Hands-on

• Write a small Java program that uses TezClient directly (no MR shim) to submit a DAG to MiniTezCluster.
• Use both session and non-session modes; measure the second-DAG latency difference.

JIRA practice queries

project = TEZ AND component = "tez-api" AND text ~ "TezClient" AND status = Open

Labs

• Lab 3.1 — Build a custom client that submits two DAGs in one session.

Exit checkpoint

• You can list every method that talks to the AM over RPC (grep for dagAMProtocol in TezClient.java).
• You can name the three local resources that TezClientUtils uploads.

Week 6 — DAGAppMaster bring-up

Reading

• tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java — focus on serviceInit, serviceStart, dispatcher registration.
• tez-dag/src/main/java/org/apache/tez/dag/app/TaskCommunicatorManager.java.
• tez-dag/src/main/java/org/apache/tez/dag/app/launcher/ContainerLauncher*.java.

Hands-on

• Run a DAG against MiniTezCluster with AM logs at DEBUG. Identify the line in DAGAppMaster.java that emits the first "Created DAG" log line.

Labs

• Lab 3.2 — Map an AM log line to source code (Lab in Level 3).

Exit checkpoint

• You can list the AsyncDispatcher event-handler registrations in DAGAppMaster in order.
• You can walk the path from TezClient.submitDAG() to DAGImpl being instantiated inside the AM.

Weeks 7–9: Level 4 — Vertex internals and state machines

Week 7 — State machine library

Reading

• hadoop-yarn-common StateMachineFactory source (you'll need to fetch Hadoop source separately).
• tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java — read only the stateMachineFactory block first (~200 lines near the top).

Hands-on

• Write a toy StateMachineFactory for a Light (OFF, ON, BROKEN) in a scratch project.

Labs

• Lab 4.1 — State-machine introduction.

Exit checkpoint

• You can explain SingleArcTransition vs MultipleArcTransition without notes.

Week 8 — VertexManager plugins

Reading

• tez-api/src/main/java/org/apache/tez/dag/api/VertexManagerPlugin.java, VertexManagerPluginContext.java.
• tez-dag/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java.

Labs

• Lab 4.2 — VertexManager deep dive (the depth-bar lab).

Exit checkpoint

• A working CountingVertexManager with passing unit test, as specified in Lab 4.2.

Week 9 — Task and TaskAttempt

Reading

• tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java.
• tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java.

Labs

• Lab 4.3 — Task lifecycle walk.
• Lab 4.4 — TaskAttempt termination causes.

Exit checkpoint

• You can draw the TaskAttempt state machine from memory.
• You can list every TaskAttemptTerminationCause and what produces it.

Weeks 10–11: Level 5 — Runtime, IPO, and shuffle

Week 10 — Runtime task execution

Reading

• tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezTaskRunner2.java.
• tez-runtime-internals/src/main/java/org/apache/tez/runtime/LogicalIOProcessorRuntimeTask.java.

Labs

• Lab 5.1 — Trace a task from container start to processor exit.

Exit checkpoint

• You can list every umbilical call a task makes during its lifetime (grep umbilical in tez-runtime-internals).

Week 11 — Shuffle and merge

Reading

• tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/ShuffleManager.java.
• tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/Fetcher.java.
• tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/sort/impl/PipelinedSorter.java.

Labs

• Lab 5.2 — Spilled output inspection on MiniTezCluster.
• Lab 5.3 — Force a fetch failure.

Exit checkpoint

• You can explain IFile framing in two paragraphs.
• You can name the three sorter implementations and when each is used.

Week 12: Level 6 — Scheduling and container reuse

Reading

• tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java.
• tez-dag/src/main/java/org/apache/tez/dag/app/rm/TaskSchedulerManager.java.
• tez-dag/src/main/java/org/apache/tez/dag/app/rm/container/AMContainerImpl.java.

JIRA practice queries

project = TEZ AND text ~ "container reuse" AND status in (Open, "In Progress")

Labs

• Lab 6.1 — Disable container reuse; measure latency cost.
• Lab 6.2 — Read and explain tez.am.container.reuse.* configs.

Exit checkpoint

• You can list the four conditions under which a container is not reused.

Week 13: Level 7 — MapReduce compatibility and integrations

Reading

• tez-mapreduce/src/main/java/org/apache/tez/mapreduce/input/MRInput.java.
• tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java.
• tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java.

Labs

• Lab 7.1 — Submit a vanilla MR job via Tez (tez.lib.uris mode).

Exit checkpoint

• You can write a one-page essay on "what MRInput does that a plain LogicalInput does not."

Week 14: Level 8 — Production diagnostics

Reading

• tez-api/src/main/java/org/apache/tez/common/counters/TezCounters.java.
• tez-dag/src/main/java/org/apache/tez/dag/history/HistoryEventHandler.java.
• tez-plugins/tez-yarn-timeline-history/.

Labs

• Lab 8.1 — Read a real ATS event dump.
• Lab 8.2 — Trace a failure through the AM log + ATS + counters.

Exit checkpoint

• You can answer: "Why did vertex X fail?" given only an AM log and ATS dump.

Weeks 15–16: Capstone

Follow capstone/index.md start-to-finish:

1. Issue selection (week 15, day 1–2).
2. Reproduction → root cause (week 15, day 3–7).
3. Implementation + tests (week 16, day 1–4).
4. Patch submission + write-up (week 16, day 5–7).

Exit checkpoint

• A real patch attached to a real JIRA, with passing tests and a clear summary.
• A 1500–3000 word public write-up of the experience.

How to use this plan when you fall behind

• If you finish a week's reading but cannot pass the exit checkpoint, repeat the week. Do not advance.
• If a JIRA query returns no results, change the query. The dev community moves; labels and components shift.
• Skip a Level only if you can pass all exit checkpoints from previous Levels in one sitting.

Milestones: M1 Through M9

Milestones are the "what does mastery look like at this stage" checkpoints. Each milestone has:

• Expected completion — a calendar guideline.
• Skills you must demonstrate — 5–8 concrete abilities.
• Self-check questions — answer them out loud, without notes.
• 20-point rubric — five criteria, four points each.
• Pass threshold — minimum total to advance.
• Move to the next level when — the binary gate.

Pass thresholds are deliberately high. The point is competence, not throughput.

M1 — Orientation (end of Week 2)

You can read the Tez DAG API and explain what every method on DAG, Vertex, and Edge does.

Skills

1. Write a 3-vertex DAG end-to-end without consulting docs.
2. Explain the three enums on EdgeProperty and pick the correct one for a given problem.
3. Name the protobuf message that represents a DAG on the wire.
4. Predict which built-in EdgeManager implementation will be selected for a given edge.
5. Locate any class in the tez-api module by name within 30 seconds.

Self-check questions

• What is the difference between DataSourceDescriptor and a runtime Input?
• Why is DAG.verify() called before submission?
• Which class produces the protobuf DAGPlan?

Rubric

Pass threshold: 14/20, with no criterion below 2.

Move to Level 2 when: you can draft a new DAG class in 10 minutes from a verbal problem statement, on a whiteboard.

M2 — Build and Test Literacy (end of Week 4)

You can navigate the codebase, build it, and run any test by name.

Skills

1. Run a single test in any module via mvn -pl <module> test -Dtest=Class#method.
2. Add a new test file to tez-dag and have it picked up by Maven.
3. Read TestVertexImpl and explain at least 10 individual test methods.
4. Identify the module of a class given just its FQN (e.g., o.a.t.dag.app... → tez-dag).
5. Build Tez from a clean checkout in under 5 minutes (with cached deps).
6. Distinguish unit tests from MiniTezCluster-backed integration tests.

Self-check questions

• Why does tez-dag depend on tez-api and not the reverse?
• What is DrainDispatcher and why do tests use it?
• Where do MiniTezCluster tests live and what classpath do they need?

Rubric

Pass threshold: 14/20.

Move to Level 3 when: you can clone Tez on a fresh laptop, build it, and run a TestVertexImpl method by name within 15 minutes.

M3 — Submission and AM Bring-up (end of Week 6)

You can trace a DAG from TezClient.submitDAG() to DAGImpl.handle(...) inside the AM.

Skills

1. List the three local resources TezClientUtils uploads.
2. Explain session vs non-session mode and the AM keep-alive mechanism.
3. Name every AsyncDispatcher event-handler registered in DAGAppMaster.
4. Locate the line of code where DAGImpl is constructed inside the AM.
5. Read AM logs at DEBUG and map lines to source positions.
6. Run MiniTezCluster in your tests and inspect AM logs.

Self-check questions

• What RPC does TezClient use to submit a DAG? Which protocol class?
• How does the AM stay alive between DAGs in a session?
• What happens if the AM dies during a DAG run with recovery disabled?

Rubric

Pass threshold: 14/20.

Move to Level 4 when: you can answer "where in the AM does my DAG show up?" with a file:line citation.

M4 — State Machines and VertexManager (end of Week 9)

You can read and modify the vertex/task/attempt state machines.

Skills

1. Write a small StateMachineFactory-based state machine from scratch.
2. Add a transition to VertexImpl.stateMachineFactory and update tests in the same patch.
3. Implement a custom VertexManagerPlugin with a unit test.
4. Diagnose an InvalidStateTransitonException from a stack trace.
5. Distinguish SingleArcTransition from MultipleArcTransition.
6. Explain the dispatcher single-threading invariant.

Self-check questions

• Why must state-machine code be single-threaded? What breaks if not?
• What happens if you forget to register a transition for an event in a state?
• How does ShuffleVertexManager implement slow-start?

Rubric

Pass threshold: 16/20 — this is the first hard gate.

Move to Level 5 when: you have submitted (or at minimum drafted) a state machine change that compiles, with a passing test.

M5 — Runtime and Shuffle (end of Week 11)

You can read the runtime data path and explain spill, merge, and fetch.

Skills

1. Walk a single task's lifecycle: container start → processor.run() → output close.
2. Explain IFile framing and the difference between V1 and V2.
3. Distinguish DefaultSorter, PipelinedSorter, and unordered output.
4. Diagnose a fetcher failure from logs.
5. Read ShuffleManager and explain its scheduling of fetchers.
6. Explain combiners and where they run in the pipeline.

Self-check questions

• What umbilical RPCs does a task make during its run?
• Where is the spill threshold checked?
• What triggers a FAILED_FETCH event upstream?

Rubric

Pass threshold: 15/20.

Move to Level 6 when: you can intentionally produce a fetcher failure on MiniTezCluster and explain every log line.

M6 — Scheduling and Container Reuse (end of Week 12)

You understand how Tez decides where tasks run.

Skills

1. Read YarnTaskSchedulerService and explain its scheduling loop.
2. List the conditions under which a container is/is not reused.
3. Explain affinity, locality, and racks.
4. Tune tez.am.container.reuse.* for a given workload.
5. Diagnose "stuck" scheduling.

Self-check questions

• Why does Tez prefer to reuse containers over requesting new ones?
• What happens if tez.am.container.idle-release-timeout-min.millis is too low?

Rubric

Pass threshold: 14/20.

Move to Level 7 when: you can explain why container reuse is on by default and pick five workloads where you would tune it.

M7 — Integrations (end of Week 13)

You can read and modify the MapReduce shim and explain Hive-on-Tez at a high level.

Skills

1. Write a DAG that uses MRInput reading from HDFS.
2. Explain MROutput commit semantics.
3. Sketch how Hive's TezTask builds a DAG.
4. Identify which features Hive uses (custom edges, manager plugins, dynamic reconfig).

Self-check questions

• What does MROutput.commit() do, and what guarantees does it offer?
• Why does Hive use ROOT_INPUT_INITIALIZER_FAILED heavily in its bug fixes?

Rubric

Pass threshold: 12/16 (only 4 criteria here).

Move to Level 8 when: you can read a Hive query plan and predict its DAG.

M8 — Production Diagnostics (end of Week 14)

You can debug a real Tez job failure given logs and an ATS dump.

Skills

1. Read a Tez counters dump and find a bottleneck.
2. Find a VertexImpl failure cause from AM logs in <5 minutes.
3. Read ATS events and reconstruct a DAG timeline.
4. Identify a stuck task vs a slow task vs a failed task from counters.
5. Build a one-pager triage runbook for your team.

Rubric

Pass threshold: 16/20.

Move to capstone when: you've helped someone (on chat, dev list, or internally) debug a real Tez issue successfully.

M9 — Capstone (end of Week 16)

You've shipped a patch.

Skills

1. Selected an appropriate issue.
2. Reproduced and root-caused.
3. Implemented a fix with tests.
4. Submitted a patch in the project's accepted format.
5. Responded to at least one round of review feedback.

Rubric (20 points)

Pass threshold: 16/20, and the patch must compile and pass mvn verify on the affected module.

Global Rubric (committer-readiness)

Use this every quarter, regardless of level, to self-assess.

A committer-track contributor should be at level 3 on all four dimensions and level 4 on at least one. Aim for 3/3/3/3 → 4/3/3/4 by month 12 of focused contribution.

Level 1: Hadoop and Tez Foundation

This level establishes the technical baseline every subsequent level depends on. You will understand where Tez fits in the Hadoop ecosystem, successfully build the project from source, run the test suite, and execute your first Tez DAG in local mode.

Learning Objectives

By the end of Level 1 you must be able to:

1. Explain where Apache Tez sits in the Hadoop ecosystem and why it exists
2. Build Apache Tez from source using Maven, with and without tests
3. Execute unit tests scoped to a single module and interpret the results
4. Run a simple Tez DAG in local mode without a YARN cluster
5. Locate any class mentioned in Levels 2–9 without using a search engine
6. Articulate the difference between a MapReduce job and a Tez DAG at the execution model level
7. Read TezConfiguration.java and find any configuration key by category

The Hadoop Ecosystem Context

Apache Tez lives inside the Hadoop ecosystem. Before touching a line of Tez code, build an accurate mental model of the stack:

┌─────────────────────────────────────────────────────┐
│         Apache Hive / Apache Pig / Cascading        │  ← Query / scripting layer
├─────────────────────────────────────────────────────┤
│                  Apache Tez                         │  ← DAG execution engine
├─────────────────────────────────────────────────────┤
│                  Apache YARN                        │  ← Cluster resource management
├─────────────────────────────────────────────────────┤
│                  Apache HDFS                        │  ← Distributed storage
└─────────────────────────────────────────────────────┘

YARN (Yet Another Resource Negotiator) manages cluster resources. It runs an ApplicationMaster (AM) per application, allocates containers, and monitors health. Tez's DAGAppMaster IS a YARN ApplicationMaster.

HDFS stores input, output, and sometimes intermediate data. Tez prefers to keep intermediate data on local disk or in memory, but falls back to HDFS for recovery and large-scale shuffles.

Tez submits a DAGAppMaster to YARN, which requests containers for task execution. Tasks read inputs, execute processors, and write outputs — either directly to downstream tasks via shuffle or to HDFS for final output.

MapReduce vs. Tez

For a 10-stage Hive aggregation query, MapReduce requires 10 separate MR jobs with HDFS writes between every stage. Tez runs the same query as a single DAG — no HDFS round-trips between stages, containers reused across task waves, and pipeline-style data movement between compatible vertices.

Required Reading

Complete in this order before starting the labs:

Note on reading strategy: In a mature Apache codebase, Javadoc is often the best documentation that exists. Class-level Javadoc on public API classes reflects decisions debated and agreed upon by committers. Read it seriously.

Source Code Areas to Inspect

Read these files before and after the labs. You are not modifying anything yet.

tez-api — Public API

tez-dag — Core Execution Engine

tez-runtime-library — I/O Implementations

tez-examples — Reference Implementations

Key Classes Quick Reference

JIRA Issue Categories for Level 1 Contributors

At this stage, focus exclusively on:

• Documentation — Javadoc typos, outdated parameter descriptions, missing @param or @return annotations, broken links in comments
• Test improvements — Adding missing assertions to existing tests, improving test method naming, removing dead code from test classes
• Checkstyle violations — Unused imports, line length violations, missing final keywords

How to find these:

1. Go to Apache Tez JIRA
2. Search: project = TEZ AND labels = "newbie" AND resolution = Unresolved
3. Also scan: project = TEZ AND component = "Documentation" AND resolution = Unresolved
4. Look at recently closed "trivial" issues to understand the standard for accepted patches

Warning: Do not pick up a JIRA issue and immediately upload a patch. Read all existing comments. If there is an active discussion or existing assignee, move on. Leave a comment saying you are investigating before you claim an issue.

Deliverables

You must demonstrate all of the following before advancing to Level 2:

• Successful mvn install -DskipTests output — no build failures
• At least one unit test class run successfully (e.g., TestDAGImpl)
• Successful local DAG execution showing DAG completed: SUCCEEDED
• Ability to locate DAGAppMaster, TezClient, and OrderedGroupedKVInput by memory
• Written explanation (2–3 sentences) of why a Tez DAG is faster than chained MapReduce
• Written explanation of the difference between a YARN container and a Tez task

Common Mistakes

How to Verify Success

# 1. Full build without tests
cd /path/to/tez
mvn install -DskipTests -q && echo "BUILD OK"

# 2. Unit test from tez-dag
mvn test -pl tez-dag -am -Dtest=TestDAGImpl -q

# 3. Local DAG run (from Lab 1.3)
# Expected final output line:
#   DAG: [OrderedWordCount] finished with status: [SUCCEEDED]

Patch Profile: Level 1 Graduate

You are not ready to submit: bug fixes in state machines, new features, performance patches, or changes to the shuffle path. Those require Levels 3–7.

Lab 1.1: Build Apache Tez from Source

Background

Apache Tez is a multi-module Maven project. Building from source is the mandatory first step for any contributor — you need the ability to make code changes, rebuild specific modules, and run tests against your local changes. This lab walks through the full build, from cloning to verifying artifacts.

Why This Lab Matters for Contributors

• You cannot submit a credible patch without first verifying it builds cleanly
• Knowing which Maven flags control which modules saves hours during development
• Understanding the build structure helps you scope test runs efficiently
• Build failures are sometimes real bugs — knowing a clean build baseline lets you detect regressions

Prerequisites

Verify before starting:

java -version    # Must be Java 8 or Java 11
mvn -version     # Must be Maven 3.6.3 or newer
git --version    # Must be 2.x

Disk space: at least 10 GB free. The full build with tests generates large artifacts. Memory: at least 8 GB RAM. The tez-dag unit tests can spike to 4 GB during parallel runs.

Step-by-Step Tasks

Step 1: Clone the Repository

git clone https://github.com/apache/tez.git
cd tez

The GitHub repository at https://github.com/apache/tez is a mirror of the canonical Apache GitBox repository. For contribution purposes (submitting patches via JIRA), the GitHub mirror is acceptable for development. The patch will be attached to the JIRA issue rather than sent as a GitHub PR — this is Apache's traditional workflow.

Verify the remote:

git remote -v
# origin  https://github.com/apache/tez.git (fetch)
# origin  https://github.com/apache/tez.git (push)

Step 2: Inspect the Branch Structure

git branch -r | grep -v HEAD | sort

You will see branches like:

• origin/master — development trunk
• origin/branch-0.10 — stable release branch
• origin/branch-0.9 — older stable branch

For contributor work, use master unless you are reproducing an issue specific to a release branch. Bug fixes for release branches are typically backported from master.

Check the current Hadoop dependency in pom.xml:

grep -m1 "hadoop.version" pom.xml

This tells you which Hadoop version Tez is built against. The default Hadoop version target controls which APIs are available.

Step 3: Full Build (Skip Tests)

mvn install -DskipTests -q

Expected duration: 5–15 minutes depending on hardware and Maven cache state.

The first run downloads all dependencies. With a warm Maven cache (~/.m2/repository), subsequent builds of unchanged modules are near-instant due to incremental compilation.

What -DskipTests does:
Skips compilation and execution of test classes. Use this for iterative development when you are not changing test code.

What -q does:
Suppresses INFO-level Maven output. Remove -q if you need to debug build failures.

When the build completes, you will see:

[INFO] BUILD SUCCESS
[INFO] Total time:  X min Y s

If you see BUILD FAILURE, go to the Troubleshooting section below.

Step 4: Verify Build Artifacts

After a successful build, key JARs exist in each module's target/ directory:

find . -name "tez-dag-*.jar" -not -path "*/test-*" | grep -v sources
# Expected: ./tez-dag/target/tez-dag-<version>.jar

find . -name "tez-api-*.jar" -not -path "*/test-*" | grep -v sources
# Expected: ./tez-api/target/tez-api-<version>.jar

The assembled distribution tarball is built by a separate command:

mvn package -DskipTests -Pdist -q
ls tez-dist/target/*.tar.gz

This produces the full binary distribution used by HDP and other distributions.

Step 5: Build a Single Module

During development you will almost always build a single module to save time:

# Build only tez-dag and its dependencies
mvn install -DskipTests -pl tez-dag -am -q

# Build only tez-api (no dependencies needed — it has none in Tez)
mvn install -DskipTests -pl tez-api -q

-pl specifies the module path. -am (also-make) builds all upstream dependencies first. This is the command you will run hundreds of times during contributor work.

Step 6: Configure IntelliJ IDEA

IntelliJ handles Maven multi-module projects natively.

1. File → Open → select the tez/ directory (the one containing pom.xml)
2. IntelliJ detects the Maven project and imports all modules
3. When prompted, select the JDK that matches the build (Java 8 or Java 11)
4. Wait for the initial index build to complete (2–5 minutes)

Verify the import worked:

• Open tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java
• Ctrl+Click on any class reference — it should navigate correctly
• Open Find Class (Cmd+O / Ctrl+N) and search TestDAGImpl — it should find the test

Enable checkstyle integration:

1. Install the CheckStyle-IDEA plugin (Settings → Plugins)
2. Configure it to use src/config/checkstyle.xml in the Tez repo root
3. This gives you real-time checkstyle feedback as you edit

Implementation Requirements

This lab has no code to implement. Deliverables are:

1. A successful mvn install -DskipTests run (screenshot or terminal output)
2. Identification of the Hadoop version Tez is built against
3. Location of the tez-dag-<version>.jar artifact
4. A working IntelliJ project that resolves all imports

Troubleshooting Common Build Failures

"Source/Target Java version mismatch"

error: Source option X is no longer supported. Use Y or later.

Cause: Your JAVA_HOME or java in PATH is the wrong version.
Fix:

export JAVA_HOME=$(/usr/libexec/java_home -v 11)   # macOS
export PATH=$JAVA_HOME/bin:$PATH
java -version   # verify
mvn install -DskipTests -q

"Cannot resolve dependency: org.apache.hadoop:..."

Cause: The required Hadoop version is not in Maven Central or your local cache.
Fix: Ensure Maven Central is reachable. If building offline, use an internal repository mirror. On a clean machine with network access this should not occur.

"Killed" or "Out of Memory"

Cause: Maven forked JVM runs out of heap.
Fix:

export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=512m"
mvn install -DskipTests -q

"ERROR: Failed to execute goal ... tez-tests"

Cause: The tez-tests module requires specific integration test infrastructure.
Fix: Build only the modules you need:

mvn install -DskipTests -pl tez-api,tez-dag,tez-runtime-library,tez-examples -am -q

Expected Output

[INFO] Reactor Summary:
[INFO] Apache Tez ......................................... SUCCESS [  2.345 s]
[INFO] tez-api ............................................ SUCCESS [ 15.678 s]
[INFO] tez-dag ............................................ SUCCESS [ 45.123 s]
[INFO] tez-runtime-internals .............................. SUCCESS [ 12.456 s]
[INFO] tez-runtime-library ................................ SUCCESS [ 18.789 s]
[INFO] tez-mapreduce ...................................... SUCCESS [  8.012 s]
[INFO] tez-examples ....................................... SUCCESS [  5.234 s]
...
[INFO] BUILD SUCCESS

Stretch Goals

1. Build against a specific Hadoop version by overriding the hadoop.version property:

   mvn install -DskipTests -Dhadoop.version=3.3.6 -q
   

2. Inspect the generated effective-pom.xml for tez-dag to see all inherited dependency versions:

   mvn help:effective-pom -pl tez-dag | grep -A3 "dependency>"
   

3. Identify which modules depend on tez-api by inspecting all pom.xml files:

   grep -r "tez-api" */pom.xml | grep "artifactId"
   

Related Real-World Issue Types

• Build breakage issues (e.g., dependency version conflicts) — you can observe but not fix at Level 1
• Java version compatibility issues — important context when reading bug reports

Lab 1.2: Run Unit and Integration Tests

Background

Apache Tez has a well-structured test suite that spans unit tests, module-level integration tests, and full cluster integration tests using MiniTezCluster. Understanding how to run specific tests, read failures, and scope test execution is essential for contributor work — your patch must include a passing test run before upload.

Why This Lab Matters for Contributors

• You must run tests before submitting any patch
• Being able to run a single test class in seconds makes iteration fast
• Understanding test failure output is the first step to debugging
• Many flaky tests are contributor opportunities once you understand how tests work

How Tez Tests Are Organized

Tez tests fall into three categories:

For Level 1–3 work, focus exclusively on unit tests.

Key unit test classes in tez-dag (path: tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/):

Supporting test infrastructure in tez-dag/src/test/java/org/apache/tez/dag/app/:

Step-by-Step Tasks

Step 1: Run All Unit Tests in tez-dag

cd /path/to/tez
mvn test -pl tez-dag -am -q 2>&1 | tail -30

Expected duration: 3–8 minutes depending on hardware.

Expected completion:

[INFO] Tests run: NNNN, Failures: 0, Errors: 0, Skipped: NN
[INFO] BUILD SUCCESS

Some tests are marked @Ignore or skipped due to environment constraints — a non-zero Skipped count is normal.

Step 2: Run a Single Test Class

mvn test -pl tez-dag -am -Dtest=TestDAGImpl -q

Expected output (last few lines):

[INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: X.XXX s
[INFO] BUILD SUCCESS

If a test fails, you will see:

[ERROR] Tests run: 42, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: X.XXX s
[ERROR] testDAGCreation(org.apache.tez.dag.app.dag.impl.TestDAGImpl): expected:<...> but was:<...>

Step 3: Run a Single Test Method

mvn test -pl tez-dag -am -Dtest=TestDAGImpl#testDAGCreation -q

This is the command you will use most often: run exactly one test after a code change to verify your fix.

Step 4: Read the Surefire Report

Maven writes detailed test results to:

tez-dag/target/surefire-reports/

For a failing test, read the .txt file for the test class:

cat tez-dag/target/surefire-reports/org.apache.tez.dag.app.dag.impl.TestDAGImpl.txt

This contains the full stack trace, which is often more informative than the Maven console output.

Step 5: Run Tests in tez-api

mvn test -pl tez-api -q

tez-api tests are faster and simpler. Key test classes:

Step 6: Run Tests in tez-runtime-library

mvn test -pl tez-runtime-library -am -q

This includes shuffle and I/O tests. Expected duration: 5–10 minutes.

Key test classes:

Step 7: Understand a Test Failure

Intentionally break a test to understand failure output:

1. Open tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java
2. Find the getTotalVertices() method
3. Add return 0; as the first line
4. Run mvn test -pl tez-dag -am -Dtest=TestDAGImpl -q
5. Read the failure output in both the console and the surefire report
6. Revert the change with git checkout tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java

This exercise makes test failure output familiar before you encounter a real failure.

Debugging Test Failures

Adding Log Output

Tez uses SLF4J + Log4j. To enable debug-level logging during a test run:

mvn test -pl tez-dag -am -Dtest=TestDAGImpl \
  -Dlog4j.configuration=file:src/test/resources/log4j.properties \
  -Dlog4j.logger.org.apache.tez=DEBUG

Running Tests with Remote Debug (IntelliJ)

To attach a debugger to a Maven test run:

mvn test -pl tez-dag -am -Dtest=TestDAGImpl \
  -Dmaven.surefire.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005"

In IntelliJ: Run → Attach to Process → port 5005. The test JVM pauses until IntelliJ connects.

Testing Checklist

Before submitting any patch:

• Run mvn test -pl <changed-module> -am — zero failures
• If adding a new test: mvn test -pl <module> -am -Dtest=<YourNewTest> passes
• Run mvn checkstyle:check -pl <changed-module> — zero violations
• If the change touches shuffle or I/O: run mvn test -pl tez-runtime-library -am

Expected Output

A clean test run for TestDAGImpl:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.tez.dag.app.dag.impl.TestDAGImpl
[INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.345 s
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESS

Stretch Goals

1. Find all test classes in tez-dag that test the VertexImpl state machine:

   find tez-dag/src/test -name "*.java" | xargs grep -l "VertexImpl"
   

2. Count the total number of test methods in TestVertexImpl:

   grep -c "@Test" tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java
   

3. Identify which test classes take the longest to run by examining surefire report timestamps:

   grep "Time elapsed" tez-dag/target/surefire-reports/*.txt | sort -t= -k2 -rn | head -10
   

4. Find tests that use MockDAGAppMaster to understand the test infrastructure pattern:

   grep -rl "MockDAGAppMaster" tez-dag/src/test/
   

Related Real-World Issue Types

• Flaky tests (timing-dependent, environment-dependent) — a major contributor opportunity
• Tests that don't assert anything meaningful — test quality improvements
• Missing test coverage for error paths — discoverable by reading state machine code

Lab 1.3: Run a Simple Tez DAG Locally

Background

Apache Tez supports a local mode that runs the entire DAG execution inside a single JVM without YARN or HDFS. This is the primary environment for rapid development and testing. Understanding how to run a DAG in local mode is essential before attempting cluster testing.

The tez-examples module contains reference DAG implementations. OrderedWordCount is the canonical example: it reads text, counts word occurrences, and sorts by frequency. It demonstrates the complete Tez DAG API: TezClient, DAG, Vertex, Edge, and I/O processors.

Why This Lab Matters for Contributors

• Local mode is how you verify behavior changes without a cluster
• All integration test work in tez-tests builds on the same local mode infrastructure
• Understanding how a real DAG is constructed gives concrete context for reading state machine code
• Every DAG execution produces log output that teaches you about the AM lifecycle

Understanding Tez Local Mode

Tez local mode is enabled by setting tez.local.mode=true in the TezConfiguration. When this is set:

• No YARN cluster is contacted
• No containers are launched — task execution happens in threads within the same JVM
• LocalMode.java replaces the full DAGAppMaster with a lightweight local executor
• HDFS is replaced by the local filesystem (configurable)

Key configuration for local mode:

TezConfiguration tezConf = new TezConfiguration();
tezConf.setBoolean(TezConfiguration.TEZ_LOCAL_MODE, true);
// Use local filesystem instead of HDFS
tezConf.set("fs.defaultFS", "file:///");
tezConf.setBoolean("tez.local.mode.without.network", true);

Anatomy of OrderedWordCount

Before running the example, read tez-examples/src/main/java/org/apache/tez/examples/OrderedWordCount.java.

The DAG structure:

[Tokenizer Vertex]
      |
      | (SCATTER_GATHER edge — partitioned by hash, sorted)
      v
[SumReducer Vertex]
      |
      | (SCATTER_GATHER edge — partitioned by value for sort)
      v
[Sorter Vertex] → HDFS output

Tokenizer: Reads input text lines, splits into words, emits (word, 1) pairs.
Processor class: TokenProcessor (inner class in OrderedWordCount)

SumReducer: Receives (word, [1, 1, 1, ...]) groups, sums counts, emits (word, count).
Processor class: SumProcessor (inner class in OrderedWordCount)

Sorter: Receives by (count, word) key (reversed), emits sorted output.
Processor class: NoOpSorter — uses OrderedGroupedKVInput to do the sort during shuffle

The key insight: Tez uses edge properties and I/O processor configuration to control the sort and partition behavior. The Sorter vertex does not sort — the shuffle/merge into OrderedGroupedKVInput does the sorting.

Step-by-Step Tasks

Step 1: Prepare Sample Input

mkdir -p /tmp/tez-lab/input
cat > /tmp/tez-lab/input/words.txt << 'EOF'
the quick brown fox jumps over the lazy dog
the dog barked at the fox
quick brown dog
EOF

Step 2: Build tez-examples

cd /path/to/tez
mvn package -DskipTests -pl tez-examples -am -q

Locate the examples JAR:

ls tez-examples/target/tez-examples-*.jar | grep -v sources | grep -v tests

Step 3: Run OrderedWordCount in Local Mode

The example is run as a standard Java main class:

# Set classpath to include Tez JARs
TEZ_HOME=/path/to/tez

CLASSPATH=\
$TEZ_HOME/tez-examples/target/tez-examples-*.jar:\
$TEZ_HOME/tez-api/target/tez-api-*.jar:\
$TEZ_HOME/tez-dag/target/tez-dag-*.jar:\
$TEZ_HOME/tez-runtime-library/target/tez-runtime-library-*.jar:\
$TEZ_HOME/tez-runtime-internals/target/tez-runtime-internals-*.jar:\
$TEZ_HOME/tez-mapreduce/target/tez-mapreduce-*.jar:\
$TEZ_HOME/tez-common/target/tez-common-*.jar

# Add Hadoop JARs (required for FileSystem, Configuration, etc.)
# If Hadoop is installed:
CLASSPATH=$CLASSPATH:$(hadoop classpath)
# If not, add from Maven local cache manually

java -cp "$CLASSPATH" \
  org.apache.tez.examples.OrderedWordCount \
  /tmp/tez-lab/input \
  /tmp/tez-lab/output \
  1

Tip: The easiest way to handle classpaths during development is to use Maven's exec:java goal or to build a fat JAR using the shade plugin. The tez-dist assembly includes all JARs and the bin/ scripts handle classpath setup.

Step 4: Run with Maven exec plugin (simpler)

If you have Hadoop installed and HADOOP_HOME set, use the Tez distributed shell script:

cd $TEZ_HOME
bin/tez-examples.sh OrderedWordCount \
  /tmp/tez-lab/input \
  /tmp/tez-lab/output \
  1

Or, add local mode flags to the Hadoop conf:

java -Dtez.local.mode=true \
     -Dfs.defaultFS=file:/// \
     -cp "$CLASSPATH" \
     org.apache.tez.examples.OrderedWordCount \
     /tmp/tez-lab/input \
     /tmp/tez-lab/output \
     1

Step 5: Verify Output

cat /tmp/tez-lab/output/part-*

Expected output (sorted by frequency descending):

the	4
dog	3
fox	2
quick	2
brown	2
...

Step 6: Read the Execution Log

Examine the log output from the run. Key lines to understand:

INFO  TezClient: Submitting DAG to YARN, queueName=...
INFO  DAGAppMaster: Running DAG: [OrderedWordCount]
INFO  VertexImpl: Vertex: [Tokenizer] initialized
INFO  VertexImpl: Vertex: [Tokenizer] started
INFO  DAGImpl: DAG: [OrderedWordCount] finished with status: [SUCCEEDED]

These lines correspond directly to state machine transitions you will study in Level 4. For each log line, identify the state transition it represents.

Implementation Requirements

Modify OrderedWordCount to add a fourth vertex that filters out words with count < 2:

1. Add a new Vertex named "Filter" after SumReducer and before Sorter
2. Write a minimal FilterProcessor extends AbstractProcessor:
   • In run(): iterate the input, skip pairs where the count value < 2, forward the rest
3. Add an edge SumReducer → Filter and Filter → Sorter
4. Run the modified DAG and verify that single-occurrence words are removed from output

This exercise teaches you:

• How to add a vertex to an existing DAG
• How to write a minimal Processor implementation
• How edges connect processors

Do not overthink the implementation — the processor body is ~20 lines.

Debugging Checklist

If the DAG fails with DAG status: FAILED:

1. Read the log for ERROR lines — they contain the failure reason and task attempt ID
2. Check DAGAppMaster log for VertexImpl: Vertex [...] failed
3. The error message will include the class and method where the exception occurred
4. Common causes:
   • Classpath missing a required JAR (NoClassDefFoundError)
   • Output directory already exists (FileAlreadyExistsException)
   • Wrong input path (FileNotFoundException)

Clean output directory before re-running:

rm -rf /tmp/tez-lab/output

Expected Output

A successful run ends with:

INFO  DAGImpl: DAG: [OrderedWordCount] finished with status: [SUCCEEDED]
INFO  TezClient: Shutting down TezSession...

Stretch Goals

1. Enable INFO-level logging for org.apache.tez.dag.app.dag.impl and observe vertex state transitions in the console output during the DAG run.

2. Modify the DAG to use UnorderedKVInput/UnorderedKVOutput instead of the ordered pair for the first edge. Observe the difference in output ordering.

3. Change the parallelism of the Sorter vertex to 2 and observe the output directory structure (2 part files instead of 1).

4. Add a timer around the TezClient.submitDAG() → DAGClient.waitForCompletion() block and measure execution time for different input sizes.

Related Real-World Issue Types

• Local mode-specific bugs (different from cluster mode) — contributor opportunity
• DAG API usability issues — often exposed by example code
• Local mode configuration issues — often reported by new users

Lab 1.4: Project — Number Pipeline DAG

What You Are Building

A self-contained, runnable Java project that builds and executes a 3-vertex Tez DAG entirely in local mode — no YARN cluster, no HDFS, no Docker required.

Generator (2 tasks)
    │  SCATTER_GATHER shuffle
    ▼
Multiplier (2 tasks)   [value * 2]
    │  SCATTER_GATHER shuffle
    ▼
Sink (1 task)          [sum → counter]

Numbers 0–99 flow through the pipeline. The expected final sum is: sum(0..99) * 2 = 4950 * 2 = 9900.

This pipeline intentionally mirrors the structure of Apache Tez's own OrderedWordCount example but with an integer domain so the math is verifiable without a corpus.

Project Location

book/projects/
├── pom.xml                              ← parent; sets Tez + Hadoop versions
└── level-1-number-pipeline/
    ├── pom.xml
    └── src/main/java/org/apache/tez/learning/l1/
        ├── GeneratorProcessor.java      ← no inputs; emits integers
        ├── MultiplierProcessor.java     ← one input, one output; value * 2
        ├── SinkProcessor.java           ← sums values; publishes counter
        ├── FilterProcessor.java         ← exercise stub (incomplete)
        └── NumberPipelineDAG.java       ← main class; configures + submits DAG

Prerequisites

• Completed Lab 1.1 (Apache Tez built from source with mvn install -DskipTests)
• Java 8+ on $PATH
• Maven 3.6+ on $PATH

Step 1: Set the Tez Version

The parent pom.xml needs to reference the exact version that mvn install installed into your local ~/.m2 repository. Find it:

# Inside your apache/tez clone:
grep -m1 '<version>' pom.xml

Open book/projects/pom.xml and set <tez.version> to match:

<tez.version>0.10.3-SNAPSHOT</tez.version>   <!-- adjust to your build -->

Step 2: Compile

cd /path/to/opensource-engineer-and-contributor/apache-tez/book/projects

# Build only the level-1 module (fast; skips the other modules)
mvn -pl level-1-number-pipeline package -q

You should see no errors. The fat JAR is at:

level-1-number-pipeline/target/level-1-number-pipeline-1.0-SNAPSHOT-jar-with-dependencies.jar

If you see Could not resolve dependency org.apache.tez:tez-api:

1. Verify that tez.version matches the version in ~/.m2/repository/org/apache/tez/tez-api/
2. Re-run mvn install -DskipTests in your Tez clone

Step 3: Run

java -jar level-1-number-pipeline/target/level-1-number-pipeline-1.0-SNAPSHOT-jar-with-dependencies.jar

Expected output (log lines abbreviated):

TezClient started (local mode).
Submitting DAG...
[SinkProcessor] task=0  partialSum=9900

=== NumberPipeline Result ===
  Expected : 9900
  Actual   : 9900
  Result   : PASS

Note: You will see a large number of INFO log lines from the Tez framework. This is normal for local mode. The important lines are the ones from [SinkProcessor] and the final === Result === block.

Step 4: Read Every Source File

Before modifying anything, read each Java file carefully.

GeneratorProcessor.java

Key questions:

1. Which Tez interface does it implement?
2. Why is output.start() called before getWriter()? What happens if you remove it?
3. How does the processor know which range of numbers to generate? What Tez API provides this?
4. The key and value written are both the same integer n. Why? When would you want them to differ?

MultiplierProcessor.java

Key questions:

1. OrderedGroupedKVInput vs OrderedPartitionedKVOutput — which side is the input and which is the output? Why are they named differently?
2. Both input.start() and output.start() are called. What does input.start() actually trigger? (Hint: look at OrderedGroupedKVInput.start() in the Tez source.)
3. FACTOR = 2 is hardcoded. The Javadoc explains how to pass it via UserPayload. What is the size in bytes of an int encoded in a ByteBuffer?

SinkProcessor.java

Key questions:

1. What is the type of getContext().getCounters()?
2. findCounter(group, name) — what happens if the counter doesn't exist yet when first called?
3. There is only one Sink task (parallelism=1). If you changed it to 2, would the counter still be correct? Why?

NumberPipelineDAG.java

Key questions:

1. What does tez.local.mode=true actually change about task execution?
2. OrderedPartitionedKVEdgeConfig.newBuilder(keyClass, valueClass, partitionerClass) — what is HashPartitioner doing here, and where does the partition count come from?
3. dagClient.waitForCompletion() — does this block on the calling thread, or is it async?
4. EnumSet.of(StatusGetOpts.GET_COUNTERS) — why is this extra call needed? Why aren't counters always included in DAGStatus?

Step 5: Break It and Understand It

Make each change, run the JAR, observe the failure, then revert.

Break 1: Remove output.start()

In GeneratorProcessor.run(), comment out logicalOutput.start().

Expected: NullPointerException or IllegalStateException from the Tez runtime when getWriter() is called on an uninitialized output.

Why this matters: Tez I/O objects are lazily initialized. The start() method triggers buffer allocation, sort buffer setup, and (for inputs) the shuffle fetch. Forgetting start() is a common first patch mistake.

Break 2: Set the wrong parallelism

Change sink parallelism from 1 to 3, run again.

Observe: does the result change? Is it still 9900? Why or why not?

Expected: the total counter is still 9900, because each Sink task emits a partial sum and the AM aggregates counters across all tasks automatically.

Break 3: Swap key and value in the Generator

Change writer.write(new IntWritable(n), new IntWritable(n)) to writer.write(new IntWritable(0), new IntWritable(n)) (fixed key = 0).

Expected: all values route to the same Multiplier task (the one that owns partition 0). The other Multiplier task gets no work. The result is still 9900 (correct) but the work distribution is skewed. You can verify this by adding a counter in MultiplierProcessor that tracks how many records each task processed.

Why this matters: key-skew (many records with the same key) is one of the most common Tez/MapReduce performance problems. This exercise makes it visible.

Step 6: Add a FilterProcessor (Exercise)

Open FilterProcessor.java. This is the skeleton for your exercise.

Your task: Insert a FilterProcessor between Multiplier and Sink that drops all values not divisible by 4, then verify the new expected sum.

Step 6a: Implement FilterProcessor

1. Add a private int threshold field.
2. In initialize(), read the threshold from UserPayload:

   byte[] bytes = getContext().getUserPayload().deepCopyAsArray();
   this.threshold = ByteBuffer.wrap(bytes).getInt();
   

3. In run(), replace if (true) with if (value.get() % threshold == 0).

Step 6b: Update NumberPipelineDAG.buildDAG()

Vertex filter = Vertex.create("filter",
    ProcessorDescriptor.create(FilterProcessor.class.getName())
        .setUserPayload(UserPayload.create(
            ByteBuffer.allocate(4).putInt(4).flip())),  // threshold=4
    2);  // same parallelism as multiplier

// New edge chain: generator → multiplier → filter → sink
.addEdge(Edge.create(generator,  multiplier, edgeConf.createDefaultEdgeProperty()))
.addEdge(Edge.create(multiplier, filter,     edgeConf.createDefaultEdgeProperty()))
.addEdge(Edge.create(filter,     sink,       edgeConf.createDefaultEdgeProperty()));

Step 6c: Calculate the new expected sum

After multiplying by 2, the values are: 0, 2, 4, 6, 8, …, 198. After filtering (keep only values divisible by 4): 0, 4, 8, 12, …, 196. Sum of {0, 4, 8, …, 196} = 4 * sum(0, 1, 2, …, 49) = 4 * (49*50/2) = 4 * 1225 = 4900.

Update NumberPipelineDAG.expectedSum() to return 4900 and verify PASS.

Step 7: Connect This to the Tez Source

Every class you used in this project maps to a real Tez module.

After running the pipeline successfully, open each source file above. For each one:

1. Find the method you called
2. Read its implementation — what does it actually do?
3. Find the unit test class for that file (usually in src/test/java/ under the same package)

Step 8: Find Related JIRA Issues

This pipeline uses OrderedPartitionedKVOutput. Search the Tez JIRA for issues in this component to find real bugs and improvements you could work on:

project = TEZ AND component = "runtime-library" AND status in (Open, Patch Available)
ORDER BY priority DESC

Also search specifically:

text ~ "OrderedPartitionedKVOutput" AND status in (Open, "Patch Available")

For each open issue you find, ask yourself:

1. Do you understand what the bug description is saying?
2. Can you locate the relevant code in the source?
3. Is there a failing test, or do you need to write one?

Expected Deliverables

• Project compiles without errors
• Running the JAR prints PASS with result 9900
• You can answer all questions in Step 4 (with file:line references to the source)
• You have run all three "Break It" experiments and understand each failure
• FilterProcessor is implemented and the pipeline prints PASS with result 4900
• You have opened all 5 source files from the "Connect to Source" table
• You have found at least 2 open JIRA issues in the runtime-library component

Level 2: Apache Contributor Onboarding

This level teaches you how the Apache open-source contribution machine works — not in the abstract, but in the specific context of Apache Tez. You will set up your tooling, understand the community structure, learn the patch workflow, and submit your first meaningful change.

Learning Objectives

By the end of Level 2 you must be able to:

1. Subscribe to dev@tez.apache.org and read a week's worth of threads
2. Navigate Apache Tez JIRA to find and evaluate open issues
3. Describe the full lifecycle of a patch: from JIRA issue to committed code
4. Generate a unified diff patch from a Git branch
5. Run Apache checkstyle and resolve all violations before submitting a patch
6. Write a JIRA comment that adds technical value
7. Find any class in the Tez repository in under 30 seconds

Apache Open-Source Contribution Fundamentals

Apache projects operate differently from GitHub-native open-source projects. The primary communication channels are mailing lists, not GitHub issues or Slack. Patches are attached to JIRA issues, not submitted as GitHub pull requests (though GitHub PRs may be used as a convenience in some projects — Tez still prefers JIRA-based workflow).

The Contribution Hierarchy

PMC (Project Management Committee)
  └─ Committers (can commit directly)
       └─ Contributors (submit patches via JIRA)
            └─ Everyone else (can file issues, ask questions)

Becoming a contributor means submitting patches. Becoming a committer means sustained, high-quality contributions over time that earn the trust of existing committers.

The Patch Lifecycle

1. Find or file a JIRA issue
2. Leave a comment: "I'm looking into this"
3. Make changes on a local branch
4. Run: mvn test -pl <module> -am  (must pass)
5. Run: mvn checkstyle:check -pl <module>  (must pass)
6. Generate a patch: git diff origin/master > TEZ-NNNN.patch
7. Attach the patch to the JIRA issue
8. Set JIRA status to "Patch Available"
9. Wait for review — a committer will comment or set "Reviewed" or "Not a bug"
10. Address feedback → upload v2 patch → repeat
11. Committer commits the patch (you cannot commit yourself until you are a committer)

Required Reading

Source Code Areas to Inspect

Apache Tez JIRA Structure

Issue Types You Will Encounter

Priority Levels

For Level 2 contributors: Only work on Minor and Trivial issues. Do not pick up Major or higher issues until you have at least 3 accepted patches in the project.

Component Labels

JIRA issues are labeled by component. The most relevant for early contributors:

Mailing List Etiquette

How to Subscribe

# Send an empty email to:
dev-subscribe@tez.apache.org
# You will receive a confirmation email — reply to it

What to Read First

Do not post until you have read at least two weeks of threads. Understand:

• What issues are currently being discussed
• How committers respond to patches
• The tone and technical depth expected
• What questions get quick responses vs. what gets ignored

How to Ask a Question

Good question format:

Subject: [QUESTION] Understanding VertexImpl initialization flow

Hi dev@,

I'm trying to understand the initialization sequence in VertexImpl.
Specifically, I'm looking at the transition from INITIALIZING to INITED
in VertexImpl.java around line 1234.

The code calls rootInputInitializer() before transitioning, but I'm unclear
on what happens if an initializer throws an unchecked exception.

I've read the JIRA issue TEZ-XXXX and the associated commit, but I still
have this question. Can anyone point me to the relevant code path?

Thanks,
[Your name]

What makes this question good:

• Specific class and approximate line number
• State machine terminology used correctly
• References prior research
• Concrete question, not "how does Tez work?"

What makes a question bad:

• "How do I contribute?" — this is answered in the contributing guide
• "Can you explain how shuffle works?" — too broad; you should read the code first
• Posting before subscribing and reading archives

Apache Checkstyle

Tez enforces checkstyle on every patch. A patch that fails checkstyle will not be committed.

Running Checkstyle

# Check a specific module
mvn checkstyle:check -pl tez-dag

# Check all modules (slow)
mvn checkstyle:check

# Check and see violations inline
mvn checkstyle:checkstyle -pl tez-dag
open tez-dag/target/checkstyle-result.xml

Common Violations

JIRA Issue Categories for Level 2 Contributors

In addition to Level 1 categories, you can now attempt:

• Test improvements — adding tests for uncovered paths you identify from reading the code
• Logging improvements — adding LOG.debug() statements that would help diagnose issues
• Checkstyle fixes — especially in modules you have been reading

Discipline: The quality of your first 5 patches determines how quickly you build credibility in the community. A patch with a checkstyle violation, compilation error, or test failure will be rejected immediately. Every patch must be verified locally before upload.

Deliverables

• Subscribed to dev@tez.apache.org and can describe two active discussions
• Apache JIRA account created
• One JIRA issue identified, studied, and commented on (even if not yet working on it)
• Lab 2.1 completed: module-by-module walkthrough documented
• Lab 2.2 completed: patch generated, checkstyle passing, JIRA description written
• Understanding of the difference between a Minor and a Trivial issue

Common Mistakes

How to Verify Success

# Your patch generates cleanly
git diff origin/master > /tmp/TEZ-NNNN.001.patch
cat /tmp/TEZ-NNNN.001.patch | head -20   # should show only your intended changes

# Checkstyle passes on the module you changed
mvn checkstyle:check -pl <changed-module>

# Tests pass
mvn test -pl <changed-module> -am -Dtest=<RelevantTestClass>

Patch Profile: Level 2 Graduate

You are not ready to submit: behavioral code changes, new features, bug fixes in state machines or shuffle. Continue to Level 3.

Lab 2.1: Navigate the Repository Structure

Background

Before writing a single line of code, a new contributor must be able to navigate the repository with the same fluency as a committer. This lab builds that fluency by walking you through every module, understanding the Maven multi-module structure, and being able to locate any class in under 30 seconds.

Repository Root Layout

apache/tez/
├── pom.xml                     # Root POM — module declarations, dep management
├── tez-api/                    # Public client API
├── tez-common/                 # Utilities shared across modules
├── tez-dag/                    # DAG AppMaster — the core of Tez
├── tez-examples/               # Example DAG implementations
├── tez-ext-service-tests/      # External service integration tests
├── tez-mapreduce/              # MapReduce compatibility layer
├── tez-plugins/                # Optional plugins (ATSv2, etc.)
├── tez-runtime-internals/      # Internal runtime interfaces
├── tez-runtime-library/        # I/O processors, shuffle
├── tez-tests/                  # Integration test suite
├── tez-tools/                  # Performance analysis utilities
├── src/
│   └── config/
│       ├── checkstyle.xml      # Style enforcement rules
│       └── checkstyle-suppressions.xml
└── CHANGES.txt                 # Release changelog

Module-by-Module Walkthrough

tez-api — The Public Contract

Everything in tez-api is part of the public API that application developers use. Changes here must be backward-compatible or explicitly versioned. This is the highest-stability module.

Key packages:

Exercise:

# Count public classes in tez-api (the API surface)
find tez-api/src/main/java -name "*.java" | wc -l

# Find all classes that implement or extend AbstractProcessor
grep -rl "extends AbstractProcessor" tez-runtime-library/src/

tez-dag — The Application Master

This is the largest and most complex module. It implements the DAG AppMaster that runs in a YARN container and orchestrates vertex and task execution.

Key packages:

Exercise:

# Count lines in DAGImpl (the most complex class)
wc -l tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java

# Count state machine transitions in VertexImpl
grep "addTransition" tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | wc -l

tez-runtime-library — I/O and Shuffle

The I/O module implements the actual data reading/writing done inside task containers. Shuffle happens here.

Key packages:

Exercise:

# Find all Input implementations
find tez-runtime-library/src/main/java -name "*Input*.java" | grep -v test

# Find the shuffle Fetcher
find tez-runtime-library/src/main/java -name "Fetcher.java"
wc -l $(find tez-runtime-library/src/main/java -name "Fetcher.java")

tez-common — Shared Utilities

Contains utilities used by multiple modules that do not fit in tez-api:

• TezUtils — configuration serialization/deserialization
• TezTaskID, TezVertexID, TezDAGID — ID types
• ReflectionUtils — Tez-specific reflection helpers
• VersionUtils — version compatibility checks

tez-mapreduce — MapReduce Compatibility

Allows MapReduce jobs to run on Tez without code changes. Contains MRInput, MROutput, and the mapper/reducer wrapping infrastructure.

tez-examples — Reference Implementations

Four example DAGs:

tez-tests — Integration Test Suite

Contains tests that run against MiniTezCluster — a full in-process Tez + YARN + HDFS cluster. These tests are slow (minutes each) but provide end-to-end coverage.

Key test class: TestMiniTezSessionWithLocalMode — runs example DAGs in local mode.

Maven Structure Deep Dive

Root pom.xml

Read the root pom.xml to understand:

1. Module declarations (<modules> section) — the build order
2. Dependency management (<dependencyManagement>) — canonical versions for all deps
3. Plugin management (<pluginManagement>) — canonical plugin configurations
4. Build profiles — hadoop-2 vs hadoop-3, dist profile for assembly

Exercise:

# What Hadoop version does Tez build against by default?
grep -A2 "hadoop.version" pom.xml | head -5

# What Java version is required?
grep "maven.compiler" pom.xml

# How many external dependencies does the root pom manage?
grep "<artifactId>" pom.xml | wc -l

Module pom.xml Structure

Each module follows the same pattern:

<parent>
  <groupId>org.apache.tez</groupId>
  <artifactId>tez</artifactId>
  <version>0.10.x-SNAPSHOT</version>
</parent>

<artifactId>tez-dag</artifactId>
<name>Tez DAG</name>

<dependencies>
  <!-- Module-specific dependencies -->
</dependencies>

Modules declare their inter-dependencies explicitly. This is how Maven knows the build order.

Exercise:

# What modules does tez-dag depend on?
grep -A3 "<dependency>" tez-dag/pom.xml | grep "tez-" | grep "artifactId"

# What does tez-runtime-library depend on?
grep -A3 "<dependency>" tez-runtime-library/pom.xml | grep "tez-" | grep "artifactId"

Finding Classes Quickly

By Name

find . -name "VertexImpl.java"
find . -name "Fetcher.java"
find . -name "TestDAGImpl.java"

By Content

# Find the class that defines TEZ_LOCAL_MODE
grep -rl "TEZ_LOCAL_MODE" --include="*.java" .

# Find all state machine StateMachine declarations
grep -rl "StateMachineFactory" --include="*.java" . | grep -v test

In IntelliJ

• Navigate to class: ⌘ O (macOS) — type class name, supports wildcards
• Navigate to file: ⌘ ⇧ O — type file name
• Find usages: ⌥ F7 — shows all places a class/method is used
• Go to implementation: ⌘ ⌥ B — jumps from interface to implementation

Navigation Checklist

After completing this lab, time yourself on each:

If any take longer, repeat the exercises in this lab.

Expected Output

By end of this lab you should have notes documenting:

1. The line count of VertexImpl.java and DAGImpl.java
2. The number of state machine transitions in VertexImpl
3. The names of all 4 example DAG classes
4. The Hadoop version Tez builds against
5. Which module handles shuffle (your own words, not copy-pasted)

Stretch Goals

1. Generate the full module dependency graph:

   mvn dependency:tree -pl tez-dag -am | grep "\\-\\-" | head -30
   

2. Find all Protocol Buffer definition files (.proto):

   find . -name "*.proto" | sort
   

   For each, identify which module it belongs to and what messages it defines.

3. Read tez-api/src/main/proto/DAGApiRecords.proto completely. Identify which messages correspond to Java classes you have already read.

Lab 2.2: Prepare a Patch Using Apache Practices

Background

A "patch" in Apache open-source culture means a unified diff file attached to a JIRA issue. This lab walks you through the complete workflow: finding a safe change to make, preparing the patch, verifying it, and writing the JIRA description.

This lab uses a real but trivial change as the vehicle — a Javadoc improvement in tez-api. Trivial changes are intentional: the goal is to master the workflow, not to write impressive code.

The Apache Git Patch Workflow

Apache Tez development uses a linear history on master (now trunk in some Apache projects, master in Tez). The standard contributor workflow:

origin/master  (read-only for non-committers)
      |
      ↓ checkout
local/master
      |
      ↓ branch
local/TEZ-NNNN
      |
      ↓ make changes
      ↓ mvn test (pass)
      ↓ mvn checkstyle:check (pass)
      ↓ git diff origin/master > TEZ-NNNN.001.patch
      |
      → Attach to JIRA

You never push your branch to Apache. You generate a diff and attach it.

Step-by-Step Tasks

Step 1: Set Up Your Working Branch

cd /path/to/tez

# Always start from a clean, up-to-date master
git fetch origin
git checkout master
git merge origin/master

# Create a branch named after the JIRA issue you are working on
# Use TEZ-0000 as a placeholder for this lab
git checkout -b TEZ-0000-javadoc-tezvertex

Verify you are on the new branch:

git branch
# * TEZ-0000-javadoc-tezvertex
#   master

Step 2: Find a Target for Your Change

Open tez-api/src/main/java/org/apache/tez/dag/api/Vertex.java.

Look for public methods that:

• Have no Javadoc, or
• Have a @param tag with a non-descriptive name like // TODO, or
• Have a @return tag missing from a non-void method

A useful starting point:

# Find methods with empty or missing Javadoc in tez-api
javadoc -private -sourcepath tez-api/src/main/java \
  org.apache.tez.dag.api 2>&1 | grep "no comment"

Or manually: open Vertex.java in IntelliJ, look at the addDataSink() method. If it lacks a @param description for dataSink, that is your target.

Step 3: Make the Change

Add or improve the Javadoc for the method you identified. Follow this format exactly:

/**
 * Adds a {@link DataSink} to this vertex. The sink will receive the output
 * of this vertex after all tasks complete.
 *
 * @param outputName
 *          the name used to identify this sink in the DAG; must be unique
 *          within this vertex
 * @param dataSink
 *          the {@link DataSink} descriptor defining the sink type and
 *          configuration
 * @return this {@link Vertex} instance (for method chaining)
 * @throws IllegalStateException if the vertex has already been added to a {@link DAG}
 */
public Vertex addDataSink(String outputName, DataSinkDescriptor dataSink) {

Rules for Apache Javadoc style:

• First sentence is a brief imperative description (no subject: "Adds a…" not "This method adds a…")
• Multi-line @param descriptions indent the continuation by 10 spaces (2 more than @param)
• Use {@link ClassName} for all class references
• Use {@code value} for code literals and parameter names in prose

Step 4: Verify Compilation

mvn compile -pl tez-api -q

Expected: BUILD SUCCESS with no errors.

Step 5: Run Checkstyle

mvn checkstyle:check -pl tez-api

Expected: BUILD SUCCESS. If there are violations, fix them before continuing.

Common Javadoc-specific violations:

• JavadocStyle — Javadoc comment does not end with a period
• JavadocMethod — @param or @return tag is missing
• JavadocVariable — public field missing Javadoc

Step 6: Run the Relevant Tests

mvn test -pl tez-api -q

Expected: BUILD SUCCESS. Even a pure Javadoc change requires a test run — checkstyle runs as part of the test phase in some configurations.

Step 7: Generate the Patch

# Verify what you changed
git diff

# The diff should show only the lines you intentionally changed
# No whitespace changes, no unrelated files

# Generate the patch file
git diff origin/master > /tmp/TEZ-0000.001.patch

# Inspect it
cat /tmp/TEZ-0000.001.patch

The patch file should:

• Start with diff --git a/tez-api/...
• Show exactly the lines you added/removed (prefixed with +/-)
• Contain no changes to files you did not intend to modify

If the patch is longer than expected, run git status to find unexpected changes and use git checkout -- <file> to revert them.

Step 8: Write the JIRA Description

For the JIRA issue you would create for this patch, write:

Summary line format:

TEZ-0000. Improve Javadoc for Vertex.addDataSink()

Description format:

Problem:
The addDataSink() method in Vertex.java has no @param documentation for the
'dataSink' parameter. This makes it harder for new users to understand the
expected input without reading the implementation.

Fix:
Add complete @param, @return, and @throws Javadoc for addDataSink().

Testing:
mvn test -pl tez-api  (all existing tests pass)
mvn checkstyle:check -pl tez-api  (no violations)

Step 9: Review the Patch as a Committer Would

Before attaching a patch, ask yourself:

1. Does the patch contain only the changes described in the JIRA description?
2. Does it pass mvn test -pl <module> locally?
3. Does it pass mvn checkstyle:check -pl <module>?
4. Is the commit message format correct? (TEZ-NNNN. Short description.)
5. Is there a clear explanation in the JIRA description of what was wrong and what was fixed?

If any answer is "no", fix it before uploading.

Common Mistakes

JIRA Status Workflow

After attaching your patch:

1. Set the JIRA status to "Patch Available"
2. Add a comment: "Patch attached. Tested with mvn test -pl tez-api and mvn checkstyle:check -pl tez-api, both pass."
3. Wait for a committer to review — do not ping on the mailing list immediately
4. If no response in 2 weeks, it is acceptable to send one polite reminder to dev@tez.apache.org:

   Subject: [REMINDER] TEZ-NNNN patch available for review
   
   Hi dev@,
   
   Friendly reminder that TEZ-NNNN has a patch attached. Any feedback welcome.
   https://issues.apache.org/jira/browse/TEZ-NNNN
   
   Thanks
   

Expected Output

At the end of this lab you have:

1. A local branch TEZ-0000-javadoc-tezvertex with a Javadoc change
2. A passing test run: mvn test -pl tez-api
3. A passing checkstyle run: mvn checkstyle:check -pl tez-api
4. A patch file at /tmp/TEZ-0000.001.patch with only the intended diff
5. A written JIRA description (even if not submitted) in the format above

Stretch Goals

1. Find a real Minor or Trivial open issue in Apache Tez JIRA that has been open for more than 6 months with no patch. Leave a JIRA comment expressing interest.

2. Attempt the same patch workflow with a real issue:

   • Use git checkout -b TEZ-<real-number>-<short-description> for the branch name
   • Use the real JIRA number in the patch filename: TEZ-NNNN.001.patch

3. Read three recently committed Tez patches by browsing JIRA issues with status "Resolved". For each, read the complete comment thread to understand the feedback cycle and how many patch revisions were required.

4. Generate a git log view that shows only your branch's commits:

   git log origin/master..HEAD --oneline
   

   This is what a committer sees when reviewing your work.

Lab 2.3 — Fix It: NullPointerException in TezTaskAttemptID.fromString

Lab type: Fix-It — reproduce → locate → write failing test → patch → verify → format patch
Estimated time: 90–120 min
Tez component: tez-common → org.apache.tez.common.TezTaskAttemptID

Background

TezTaskAttemptID is the primary key that links a task attempt to its vertex, DAG, and application. Its static fromString method parses a serialised ID like:

attempt_1609459200000_0001_1_00_000000_0

In the Tez codebase the parse path for certain malformed inputs has historically thrown an unguarded NullPointerException rather than a descriptive IllegalArgumentException. Null returns from String.split() or Integer.parseInt() call sites that skip validation are the common culprit.

This lab walks the complete Apache contribution workflow for such a bug:

1. Reproduce the crash in a test
2. Read the source to understand why it crashes
3. Apply the minimal fix
4. Verify tests pass and checkstyle is clean
5. Produce a .patch file ready for JIRA upload

Step 1 — Locate the Source File

cd ~/tez-src         # your local Tez clone from Lab 1.1
find . -name "TezTaskAttemptID.java" | head -5

Expected path:

./tez-common/src/main/java/org/apache/tez/common/TezTaskAttemptID.java

Open the file and read the fromString method in full.

Questions to answer before continuing

Step 2 — Find the Existing Tests

find . -name "TestTezTaskAttemptID.java" | head -5

Expected path:

./tez-common/src/test/java/org/apache/tez/common/TestTezTaskAttemptID.java

Open it.

Questions

Step 3 — Reproduce the Bug

Add the following test to TestTezTaskAttemptID.java inside the existing test class. Do not modify the test — the goal is to make it pass, not work around it.

@Test(expected = IllegalArgumentException.class)
public void testFromStringNullInput() {
    TezTaskAttemptID.fromString(null);
}

@Test(expected = IllegalArgumentException.class)
public void testFromStringTooFewParts() {
    // Fewer underscore-separated tokens than the format requires
    TezTaskAttemptID.fromString("attempt_1609459200000_0001_1");
}

Run the tests:

cd tez-common
mvn test -pl . -Dtest=TestTezTaskAttemptID -q 2>&1 | tail -30

Expected result: Both new tests FAIL (the method throws NullPointerException or ArrayIndexOutOfBoundsException, not IllegalArgumentException).

Record the exact exception and stack-trace line. You will need this for the JIRA description later.

Step 4 — Apply the Fix

Open TezTaskAttemptID.java and apply a minimal patch to the fromString method.

Rules for a minimal patch

• Add a null-check at the very top of fromString; throw IllegalArgumentException with a clear message
• Add a length-check on the parsed tokens before subscripting the array
• Do not reformat unrelated lines (this produces noisy diffs that fail checkstyle review)
• Do not change method signatures or visibility

Hint — guard pattern used elsewhere in the same class

Search the file for how other fromString variants guard their input:

grep -n "IllegalArgumentException" TezTaskAttemptID.java

Use the same pattern and message style.

Step 5 — Verify the Fix

# All TezTaskAttemptID tests must pass
mvn test -pl tez-common -Dtest=TestTezTaskAttemptID -q

# Full tez-common test suite (regression guard)
mvn test -pl tez-common -q 2>&1 | tail -20

# Checkstyle must be clean
mvn checkstyle:check -pl tez-common -q 2>&1 | grep -E "ERROR|WARNING|violation" | head -20

All three commands must produce zero errors.

Step 6 — Understand the Checkstyle Rules

cat tez-common/src/main/checkstyle/tez-checkstyle.xml | grep -A2 "LineLength"
cat tez-common/src/main/checkstyle/tez-checkstyle.xml | grep -A2 "Javadoc"

Questions

Step 7 — Format the Patch File

Apache Tez uses the unified diff format. From the repo root:

cd ~/tez-src
git diff > /tmp/TEZ-XXXX.001.patch

Inspect the patch:

cat /tmp/TEZ-XXXX.001.patch

Checklist before uploading to JIRA

• Patch header shows the correct file path relative to the repo root
• Only TezTaskAttemptID.java and TestTezTaskAttemptID.java are modified
• No trailing whitespace on any changed line (grep -P "\s+$" /tmp/TEZ-XXXX.001.patch)
• Patch applies cleanly to a fresh checkout: git apply --check /tmp/TEZ-XXXX.001.patch
• mvn test -pl tez-common still passes after git apply

Step 8 — Write the JIRA Description

Draft a JIRA ticket description following the Apache Tez convention:

Summary: TezTaskAttemptID.fromString throws NPE/AIOOBE on malformed input
         instead of IllegalArgumentException

Description:
  TezTaskAttemptID.fromString does not validate its input before parsing.
  Passing null or a string with fewer than N underscore-separated parts
  causes an unhandled NullPointerException (null path) or
  ArrayIndexOutOfBoundsException (short-string path) instead of
  the expected IllegalArgumentException.

  Steps to reproduce:
    TezTaskAttemptID.fromString(null);
    → NullPointerException at TezTaskAttemptID.java:NN

    TezTaskAttemptID.fromString("attempt_1609459200000_0001_1");
    → ArrayIndexOutOfBoundsException at TezTaskAttemptID.java:NN

  Fix: add explicit null guard + array-length guard at the top of fromString.

Priority: Minor
Component: tez-common

Replace NN with the actual line numbers from your stack traces in Step 3.

Step 9 — Connect the Concepts

Reflection

1. Why should library code throw IllegalArgumentException rather than letting a NullPointerException propagate?
2. What does the Apache contribution guide say about test coverage for bug fixes?
   (Hint: CONTRIBUTING.md or the Apache Tez wiki — every bug fix must include a reproducing test.)
3. How does the Tez fromString guard pattern compare to the one in Hadoop's TaskAttemptID.forName?
4. Could this same class of bug exist in TezDAGID.fromString or TezVertexID.fromString?
   Check both files and note your findings.

Lab 2.4 — Review It: Spot the Flaws in TEZ-FAKE001.001.patch

Lab type: Review-It — read a synthetic patch, find every flaw, explain the impact, propose fixes
Estimated time: 60–90 min
Tez component: tez-dag → org.apache.tez.dag.app.dag.impl.TaskImpl

Context

You are a Tez committer reviewing a patch uploaded to JIRA. The contributor claims the patch fixes a race condition where TaskImpl.getCounters() returns null when called before any task attempt has completed.

Your job is to review the patch before it merges. There are exactly 5 intentional flaws hidden in the diff below. Find them all.

The Synthetic Patch

diff --git a/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java b/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java
index a1b2c3d..e4f5a6b 100644
--- a/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java
+++ b/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java
@@ -214,6 +214,8 @@ public class TaskImpl implements Task, EventHandler<TaskEvent> {
 
+  import org.apache.tez.common.counters.TezCounters;
+
   public synchronized TezCounters getCounters() {
     TezCounters counters = null;
     if (successfulAttempt != null) {
@@ -221,7 +223,7 @@ public class TaskImpl implements Task, EventHandler<TaskEvent> {
       counters = successfulAttempt.getCounters();
     } else {
       counters = attemptList.stream()
-          .filter(a -> a.getState() == TaskAttemptState.SUCCEEDED)
+          .filter(a -> a.getState() == TaskAttemptState.RUNNING)
           .findFirst()
           .map(TaskAttemptImpl::getCounters)
           .orElse(null);
@@ -231,6 +233,14 @@ public class TaskImpl implements Task, EventHandler<TaskEvent> {
     return counters;
   }
 
+  /**
+   * Returns the counter for this task, or a new empty TezCounters object
+   * if no counters are available yet.
+   *
+   * @return counters, never null
+   */
+  public synchronized TezCounters getCountersOrEmpty() {
+    TezCounters c = getCounters();
+    return c == null ? new TezCounters() : c;
+  }
+
diff --git a/tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestTaskImpl.java b/tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestTaskImpl.java
index b7c8d9e..f0a1b2c 100644
--- a/tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestTaskImpl.java
+++ b/tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestTaskImpl.java
@@ -891,6 +891,18 @@ public class TestTaskImpl {
 
+  @Test
+  public void testGetCountersBeforeAnyAttempt() {
+    // No attempts started; counters should not be null
+    initTask();
+    TezCounters result = task.getCounters();
+    assertNotNull("getCounters() must not return null", result);
+  }
+
+  @Test
+  public void testGetCountersOrEmptyReturnsSameObjectEachTime() {
+    initTask();
+    TezCounters first  = task.getCountersOrEmpty();
+    TezCounters second = task.getCountersOrEmpty();
+    assertSame("Must return same instance", first, second);
+  }
+

Your Task

For each flaw you find, fill in the table:

Guided Questions

Work through these questions one by one. Each one points at a different flaw.

Question 1 — Import placement

Look at where the import statement was added:

+  import org.apache.tez.common.counters.TezCounters;
+
   public synchronized TezCounters getCounters() {

• Is this a valid location for a Java import declaration?
• What would happen at compile time if this diff were applied as-is?
• Where should imports go in a Java file?
• Lookup: does TaskImpl.java already import TezCounters at the top?
  (grep "import.*TezCounters" tez-dag/src/main/java/.../TaskImpl.java)

What is the flaw?

Question 2 — The filter predicate

The patch changes the fallback stream filter from:

.filter(a -> a.getState() == TaskAttemptState.SUCCEEDED)

to:

.filter(a -> a.getState() == TaskAttemptState.RUNNING)

• Re-read the JIRA description: the reporter says getCounters() returns null when called before any attempt has completed.
• Does filtering for RUNNING attempts fix that?
• What does it mean to read counters from a RUNNING attempt vs a SUCCEEDED one?
• Are the counters of a still-running attempt considered final/reliable?

What is the flaw? What should the filter be?

Question 3 — The new test testGetCountersBeforeAnyAttempt

Read the test body carefully:

TezCounters result = task.getCounters();
assertNotNull("getCounters() must not return null", result);

• The test asserts that getCounters() is not null when no attempt has started.
• But the patch does not change getCounters() to return an empty object — it adds a separate method getCountersOrEmpty() for that.
• When successfulAttempt is null and attemptList is empty, what does getCounters() actually return?
• Will this test pass or fail against the patched code?

What is the flaw?

Question 4 — The new test testGetCountersOrEmptyReturnsSameObjectEachTime

assertSame("Must return same instance", first, second);

• getCountersOrEmpty() is implemented as:
  return c == null ? new TezCounters() : c;
• Each call creates a new TezCounters() when c is null.
• Does the assertSame assertion match the implementation?
• Is assertSame testing a documented contract, or is it over-specifying an implementation detail?
• What assertion would actually verify the intended contract ("not null")?

What is the flaw?

Question 5 — The JIRA description says the fix is needed, but…

Re-read the patch one final time. The root cause (as stated in the JIRA) is that getCounters() can return null. The correct caller-safe fix for most Tez callers would be to make getCounters() itself never return null (return empty TezCounters as the contract).

Instead the patch adds getCountersOrEmpty() as a new method — but leaves the old getCounters() method returning null.

• Every existing caller of getCounters() still gets null.
• The Tez codebase uses getCounters() in aggregation loops that iterate counters: counters.incrAllCounters(taskCounters) — passing null there throws NPE.
• How many callers of getCounters() exist in tez-dag?

grep -rn "\.getCounters()" tez-dag/src/main/ | grep -v "//.*getCounters" | wc -l

• Does the patch actually fix the original bug?

What is the flaw?

Answer Key (Read After You've Filled the Table)

Reflection

1. A patch that adds a new method instead of fixing the old one is sometimes called an "additive workaround." When is that acceptable? When is it wrong?

2. The Apache Tez review process requires that every patch include a test that would have failed before the fix and passes after. Does this patch satisfy that requirement? Why or why not?

3. If you were the committer, what feedback would you leave on JIRA? Write two or three sentences in the style of a real review comment (constructive, specific, pointing at the line).

4. Look up a real Tez JIRA review thread (search issues.apache.org/jira for project = TEZ AND labels = patch-available AND resolution = Fixed). Find one comment where a committer asked for a test change. What did they say?

Level 3: Tez Architecture

This level gives you a working mental model of how all Tez components fit together. After completing it you will be able to trace any execution path — from API call to task output — through the code without getting lost. Architecture knowledge is what separates a contributor who fixes isolated bugs from one who can design improvements.

Learning Objectives

By the end of Level 3 you must be able to:

1. Draw the Tez component topology from memory (Client → AM → RM → NM → Container)
2. Trace a DAG.submit() call through four class boundaries to the first vertex start
3. Explain the role of each of the four state machines and how they interact
4. Describe what happens on each of the three communication channels between components
5. Explain the Input-Processor-Output (IPO) model and how it relates to DAG edges
6. Identify which Protocol Buffer message type carries a given piece of information

Component Topology

┌─────────────────────────────────────────────────────────────────────┐
│  Client JVM                                                         │
│  ┌─────────────┐                                                    │
│  │  TezClient  │──── submitDAG() ────────────────────────────────┐  │
│  └─────────────┘                                                 │  │
└──────────────────────────────────────────────────────────────────┼──┘
                                                                   │ DAGPlan (protobuf)
                                                                   ▼
┌─────────────────────────────────────────────────────────────────────┐
│  YARN ResourceManager                                               │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  ApplicationMaster container (DAGAppMaster)                  │   │
│  │                                                              │   │
│  │  ┌───────────┐  ┌────────────┐  ┌──────────┐  ┌─────────┐  │   │
│  │  │  DAGImpl  │→ │ VertexImpl │→ │ TaskImpl │→ │ TaskAttemptImpl│  │
│  │  └───────────┘  └────────────┘  └──────────┘  └─────────┘  │   │
│  │         │              │                                     │   │
│  │         └──── events ──┘                                    │   │
│  │                                                              │   │
│  │  ContainerLauncher ─── launches ──────────────────────────┐ │   │
│  └─────────────────────────────────────────────────────────┬─┘ │   │
└────────────────────────────────────────────────────────────┼───┼───┘
                                                             │   │
                                              container req  │   │ container
                                                             ▼   ▼
┌─────────────────────────────────────────────────────────────────────┐
│  YARN NodeManagers (one per worker node)                            │
│  ┌───────────────────────────────────────────────────────────────┐  │
│  │  TezChild (task container JVM)                                │  │
│  │  ┌────────────────────────────────────────────────────────┐  │  │
│  │  │  LogicalIOProcessorRuntimeTask                         │  │  │
│  │  │   Input(s) ─── Processor ─── Output(s)                │  │  │
│  │  └────────────────────────────────────────────────────────┘  │  │
│  └───────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘

Communication Channels

The Four State Machines

Tez execution is modeled as four nested state machines. Each tracks a specific level of granularity and sends events to the others.

DAGImpl State Machine

Key transition: NEW → INITED triggers VertexInitializedEvent for each vertex.

VertexImpl State Machine

The most complex state machine in Tez. Has ~30 states and 80+ transitions.

Core states (simplified):

The VertexImpl state machine is defined by the StateMachineFactory at the top of VertexImpl.java. Reading the factory definition gives you the complete transition table.

TaskImpl State Machine

Each vertex has N tasks (parallelism = N). TaskImpl tracks one task across its attempts.

TaskImpl manages the attempt retry logic: if attempt 1 fails, TaskImpl decides whether to launch attempt 2 based on the failure mode and retry count configuration.

TaskAttemptImpl State Machine

One actual container execution of a task.

Event System

State machine transitions are driven by events. The event bus (AsyncDispatcher) routes events from producers to the correct state machine.

Key Event Types

The event flow for a normal task success:

Container reports TA_DONE
  → TaskAttemptImpl: RUNNING → SUCCEEDED
  → sends T_ATTEMPT_SUCCEEDED to TaskImpl
    → TaskImpl: RUNNING → SUCCEEDED
    → sends V_TASK_COMPLETED to VertexImpl
      → VertexImpl checks: all tasks done?
        → if yes: sends DAG_VERTEX_COMPLETED to DAGImpl
          → DAGImpl checks: all vertices done?
            → if yes: DAG transitions to SUCCEEDED

Every state transition in this chain corresponds to a log line you will see in the AM logs.

Protocol Buffers

All cross-process data in Tez is serialized with Protocol Buffers (proto3 in newer versions).

The DAGPlan message is what TezClient sends to the AM. It contains the complete description of the DAG: vertices, edges, processor descriptors, I/O configurations, and edge properties. It is generated from the DAG API object.

// In DAGImpl.java, the plan is received and deserialized:
DAGPlan dagPlan = clientAMProtocol.submitDAG(submitDAGRequest).getDagId();
// Plan is then converted to DAGImpl state

Input-Processor-Output (IPO) Model

Each task runs a single AbstractProcessor. The processor has access to named Input and Output instances, which are determined by the edges in the DAG.

┌──────────────────────────────────────────────────────────────────┐
│  Task container                                                  │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  LogicalIOProcessorRuntimeTask                          │    │
│  │                                                         │    │
│  │  Inputs:                    Outputs:                    │    │
│  │  ┌──────────────────┐       ┌──────────────────────┐   │    │
│  │  │ OrderedGrouped   │       │ OrderedPartitioned   │   │    │
│  │  │ KVInput          │──┐ ┌──│ KVOutput             │   │    │
│  │  └──────────────────┘  │ │  └──────────────────────┘   │    │
│  │                        ▼ │                              │    │
│  │               ┌───────────────┐                         │    │
│  │               │  MyProcessor  │                         │    │
│  │               │  extends      │                         │    │
│  │               │  AbstractProcessor                      │    │
│  │               └───────────────┘                         │    │
│  └─────────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────────┘

Edge Property Types

The EdgeProperty in the DAG API determines what I/O classes are used between two vertices.

SCATTER_GATHER corresponds to the classic MapReduce shuffle. BROADCAST is used for joins where one side is small enough to replicate to all tasks.

DataMovementEvent

When a task output is ready, it sends a DataMovementEvent through the umbilical to the AM. The AM routes it to the downstream tasks so their input knows which partition to fetch.

This event routing is the mechanism by which OrderedGroupedKVInput discovers where each upstream partition is located — it receives DataMovementEvents from the AM containing the shuffle server address and partition index.

Required Reading

Key Classes Quick Reference

JIRA Categories for Level 3

Having read the architecture, you can now evaluate:

• Architecture improvement JIRAs — proposals to change how components interact
• State machine correctness bugs — transitions that lead to wrong states
• Event routing issues — events that are lost or sent to wrong consumers
• Container reuse improvements — how tasks are assigned to existing containers

You are still not ready to submit fixes for state machine bugs — those require Level 4. But you can now read these issues intelligently and leave informed comments.

Deliverables

• Draw the component topology diagram from memory (no looking)
• Trace TezClient.submitDAG() to VertexImpl V_START event through class names
• Identify the state machines and their event types from code (not from this page)
• Explain in your own words what DataMovementEvent does and why it exists
• Lab 3.1 completed: DAG submission trace documented
• Lab 3.2 completed: IPO abstraction walkthrough complete

Common Mistakes

Lab 3.1: Trace a DAG Submission End-to-End

Background

A DAG goes from a Java object constructed with the API to running tasks in containers through a sequence of method calls, IPC calls, and event posts that spans six class boundaries and three JVMs. This lab asks you to trace that path precisely — class name, method name, and the data that crosses each boundary.

Being able to reconstruct this trace from code (not from documentation) is the skill. That means reading DAGAppMaster.java, DAGImpl.java, VertexImpl.java, and TezChild.java and following the chain yourself.

The Six Class Boundaries

[1] TezClient.submitDAG(dag)
         │
         │  DAGClientAMProtocol (IPC) — carries: SubmitDAGRequest{DAGPlan}
         ▼
[2] DAGClientHandler.submitDAG(request)   [in DAGAppMaster]
         │
         │  posts: DAGAppMasterEvent(NEW_DAG_SUBMITTED)
         ▼
[3] DAGAppMaster.handle(event)
         │
         │  calls createDag(dagPlan) → new DAGImpl(...)
         │  posts: DAGEvent(DAG_INIT)
         ▼
[4] DAGImpl.handle(DAGEvent{DAG_INIT})
         │
         │  InitTransition: initializes all VertexImpl objects
         │  posts: VertexEvent(V_INIT) for each vertex
         ▼
[5] VertexImpl.handle(VertexEvent{V_INIT})
         │
         │  InitTransition: sets up tasks, calls VertexManager
         │  posts: VertexEvent(V_START) when ready
         │  posts: TaskEvent(T_SCHEDULE) for each task
         ▼
[6] TaskImpl → TaskAttemptImpl → ContainerLauncher → NM
         │
         │  NM starts container JVM: TezChild.main()
         ▼
[Container JVM] TezChild receives task assignment via TezTaskUmbilicalProtocol
         │
         ▼
LogicalIOProcessorRuntimeTask.run()  — Processor.run() called

Step-by-Step Tasks

Step 1: Find the Entry Point in TezClient

Open tez-api/src/main/java/org/apache/tez/dag/api/TezClient.java.

Find the submitDAG(DAG dag) method. Answer:

1. What is the name of the IPC protocol interface used to communicate with the AM?
2. What does TezClient do if it does not yet have an AM to talk to (session not started)?
3. What method on the DAG object serializes it to a DAGPlan protobuf?
4. What request object wraps the DAGPlan before it is sent over IPC?

# Find the IPC protocol interface
grep -n "Protocol" tez-api/src/main/java/org/apache/tez/dag/api/TezClient.java | head -10

# Find DAGPlan construction
grep -n "DAGPlan\|createDag\|getPlan" tez-api/src/main/java/org/apache/tez/dag/api/TezClient.java

Step 2: Find the AM-side IPC Handler

The AM exposes the DAGClientAMProtocol interface. The implementation is in DAGAppMaster.

# Find the implementation of submitDAG on the AM side
grep -rn "submitDAG" tez-dag/src/main/java/org/apache/tez/dag/app/ | grep -v test

Open the handler class. Answer:

1. What is the exact class name that implements DAGClientAMProtocol?
2. What event type does it post to the AsyncDispatcher after receiving the DAGPlan?
3. Does the submitDAG call on the AM side block until the DAG completes, or does it return immediately?

Step 3: Trace DAGAppMaster Initialization

Open tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java.

Find the serviceStart() method. Read the component initialization order:

1. List the components initialized in serviceStart() in order
2. Find where AsyncDispatcher is created and started
3. Find where the DAGEventDispatcher (the component that routes DAGEvents to DAGImpl) is registered

# Find component initialization
grep -n "addService\|serviceStart\|startService" \
  tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java | head -20

Step 4: Read the DAGImpl Init Transition

Open tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java.

Find the StateMachineFactory definition. Locate the transition for DAGEventType.DAG_INIT.

The transition handler class is InitTransition. Find it in the same file.

Answer:

1. What does InitTransition.transition() do with each vertex in the DAG?
2. After initializing vertices, what event does DAGImpl post?
3. Under what condition does the init transition immediately move to RUNNING vs waiting?

# Find the init transition
grep -n "InitTransition\|DAG_INIT" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java | head -20

Step 5: Read the VertexImpl Init Transition

Open tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java.

Find the transition from INITIALIZING on event V_INIT. The handler is InitTransition (a different class from the one in DAGImpl).

Answer:

1. What is the VertexManager and when is it invoked during initialization?
2. How does VertexImpl know how many tasks to create (the parallelism)?
3. What event does VertexImpl send to DAGImpl when initialization completes?

# Find vertex init transition
grep -n "V_INIT\|InitTransition" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head -20

Step 6: Trace the Container Launch

After tasks are scheduled, TaskAttemptImpl requests a container from the TaskScheduler. When a container is assigned, ContainerLauncher builds the launch context.

# Find the container launch command construction
grep -rn "containerLaunchContext\|getContainerLaunchContext\|vargs" \
  tez-dag/src/main/java/org/apache/tez/dag/app/launcher/ | grep -v test | head -10

Answer:

1. What is the main class of the container JVM? (The class with main() that YARN launches)
2. What information is passed to TezChild via system properties vs environment variables?
3. How does TezChild know which task to run when it starts?

Step 7: Read TezChild.main()

Open tez-dag/src/main/java/org/apache/tez/dag/app/TezChild.java.

Find the main() method and the run() loop.

Answer:

1. What IPC interface does TezChild use to communicate with the AM?
2. What does TezChild do when it receives a TaskSpec from the AM?
3. What class is instantiated to actually run the processor?

# Find TezChild
find tez-dag/src/main/java -name "TezChild.java"
wc -l $(find tez-dag/src/main/java -name "TezChild.java")

Complete the Trace Table

Fill in this table by reading the code (not from this page or any other documentation):

Fill in the ? cells from the actual code. Each cell should contain the real method name.

Expected Output

A completed trace table with all cells filled from code, not from documentation. Each answer should be verifiable by pointing to a specific line in a specific file.

Example format for your notes:

Step 2: DAGClientHandler.submitDAG()
  in: tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java
  line: ~1234
  posts: DAGAppMasterEvent(NEW_DAG_SUBMITTED)

Stretch Goals

1. Find the AsyncDispatcher queue size configuration. What happens if the queue fills up?

   grep -rn "AsyncDispatcher\|dispatcher.queue" \
     tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java | head -10
   

2. Find where the AM is told to exit when the DAG completes:

   grep -n "stop\|shutdown\|exit" \
     tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java | grep -i "succeeded\|complete"
   

3. Trace what happens to a TA_DONE event from TezChild back to DAGImpl:

   • TezChild calls a method on the umbilical
   • The AM receives it and posts a TaskAttemptEvent
   • TaskAttemptImpl transitions to SUCCEEDED
   • The chain continues up to DAGImpl Identify every class and event in this reverse chain.

Lab 3.2: Understand the IPO Abstraction

Background

Every task in Tez runs a Processor that reads from one or more Input objects and writes to one or more Output objects. This Input-Processor-Output (IPO) model is the fundamental abstraction for how data moves through a DAG. Edge properties in the API (EdgeProperty, DataMovementType) determine which I/O classes are instantiated in the container.

This lab walks through the IPO model from the API layer to the runtime, tracing how an ORDERED_PARTITIONED_KV_OUTPUT configuration becomes actual bytes in a shuffle buffer.

The IPO Interface Hierarchy

tez-runtime-api (in tez-api module):
  AbstractLogicalInput
      └── AbstractInput
  AbstractLogicalOutput
      └── AbstractOutput
  AbstractProcessor

tez-runtime-library (implementations):
  OrderedPartitionedKVOutput    extends AbstractLogicalOutput
  OrderedGroupedKVInput         extends AbstractLogicalInput
  UnorderedKVOutput             extends AbstractLogicalOutput
  UnorderedKVInput              extends AbstractLogicalInput
  UnorderedPartitionedKVOutput  extends AbstractLogicalOutput
  BroadcastKVInput              extends AbstractLogicalInput (alias for UnorderedKVInput)

The key interface chain:

AbstractLogicalOutput.initialize() → called by LogicalIOProcessorRuntimeTask
AbstractLogicalOutput.start()      → called when the processor is started
AbstractLogicalOutput.getWriter()  → returns KeyValueWriter for the processor to use
AbstractLogicalOutput.commit()     → called after processor.run() completes
AbstractLogicalOutput.close()      → cleanup

Step-by-Step Tasks

Step 1: Read the AbstractLogicalOutput Interface

Open tez-runtime-internals/src/main/java/org/apache/tez/runtime/api/AbstractLogicalOutput.java.

Answer:

1. What is the purpose of the initialize() method? What does it return?
2. What is the difference between start() and initialize()? Why are they separate?
3. What method does a Processor call to get a writer to write records?

Step 2: Trace OrderedPartitionedKVOutput.initialize()

Open tez-runtime-library/src/main/java/org/apache/tez/runtime/library/output/OrderedPartitionedKVOutput.java.

Find the initialize() method.

Answer:

1. What configuration key controls the buffer size for sorting?
2. What class is created in initialize() to handle the actual sort-and-spill?
3. How is the Partitioner class determined at runtime?

# Find sort buffer configuration
grep -n "SORT_MB\|sortmb\|buffer" \
  tez-runtime-library/src/main/java/org/apache/tez/runtime/library/output/OrderedPartitionedKVOutput.java \
  | head -10

# Find the writer/sorter creation
grep -n "new.*Writer\|new.*Sorter\|ExternalSorter" \
  tez-runtime-library/src/main/java/org/apache/tez/runtime/library/output/OrderedPartitionedKVOutput.java

Step 3: Trace the Write Path

When a processor calls writer.write(key, value), the data goes:

KeyValueWriter.write(key, value)
  → ExternalSorter.collect(key, value, partition)
    → SpillThread triggers when buffer is full
      → IFile.Writer writes sorted partition to local disk
        → On close(): merges all spills into final output file

Find the ExternalSorter class:

find tez-runtime-library/src/main/java -name "ExternalSorter.java"

Answer:

1. What data structure holds records before they are spilled?
2. What algorithm is used to sort records in the buffer?
3. How is the sort key computed for (K, V) pairs with a custom Comparator?

Step 4: Read OrderedGroupedKVInput.initialize()

Open tez-runtime-library/src/main/java/org/apache/tez/runtime/library/input/OrderedGroupedKVInput.java.

Find initialize().

Answer:

1. What class handles the shuffle (fetching data from remote nodes)?
2. How does the input know which upstream tasks it needs to fetch from?
3. What event type does the input consume to discover shuffle locations?

grep -n "Shuffle\|ShuffleManager\|DataMovementEvent" \
  tez-runtime-library/src/main/java/org/apache/tez/runtime/library/input/OrderedGroupedKVInput.java \
  | head -15

Step 5: Trace the Read Path

When a processor calls keyValueReader.next(), the data flow is:

KeyValueReader.next()
  → MergedKeyValueIterator.next()    [merging multiple sorted partitions]
    → TezRawKeyValueIterator         [from TezMerger]
      → IFile.Reader reads from local merged file

But before the merge can happen, the shuffle must fetch data:

DataMovementEvent arrives (from AM, routed from upstream task)
  → ShuffleManager records: "partition P is at host H:port/path"
  → Fetcher.fetch() downloads the partition file
    → stores locally
  → When all partitions fetched: MergeManager merges them
    → final sorted output available for KeyValueReader

Find the Fetcher class:

find tez-runtime-library/src/main/java -name "Fetcher.java"
wc -l $(find tez-runtime-library/src/main/java -name "Fetcher.java")

Answer:

1. What HTTP endpoint does Fetcher call to retrieve partition data?
2. What does Fetcher do if the HTTP request fails?
3. How many simultaneous fetch connections does Fetcher allow by default?

Step 6: Understand DataMovementEvent Routing

The DataMovementEvent is what connects output to input. When a task completes its output:

1. The ShuffleHandler (shuffle server) registers the output location
2. The task sends a DataMovementEvent via the TezTaskUmbilicalProtocol to the AM
3. The AM routes the event to the downstream tasks that need it
4. The downstream input receives it and knows to fetch from that location

Find the DataMovementEvent class:

find . -name "DataMovementEvent.java" | grep -v test

Answer:

1. What fields does DataMovementEvent carry?
2. Why is the payload (userPayload) a byte array and not a typed field?
3. How does the AM know which downstream tasks to route the event to?

# Find the AM-side routing logic
grep -rn "DataMovementEvent\|routeEvent" \
  tez-dag/src/main/java/org/apache/tez/dag/app/ --include="*.java" | grep -v test | head -15

Step 7: Edge Properties → I/O Classes

The EdgeProperty object in the API specifies which I/O classes to use. Trace how EdgeProperty becomes actual I/O class instantiation in the container.

Starting point:

# Find EdgeProperty
find tez-api/src/main/java -name "EdgeProperty.java"

Then trace:

# How does VertexImpl use EdgeProperty to configure I/O for a vertex?
grep -n "EdgeProperty\|getInputDescriptor\|getOutputDescriptor" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head -15

Answer:

1. What field in EdgeProperty specifies the Input class for the destination vertex?
2. What field specifies the Output class for the source vertex?
3. How is the class name passed to the container so it can instantiate the correct I/O class?

Build the IPO Map

For OrderedWordCount, fill in this table by reading the code:

Read tez-examples/src/main/java/org/apache/tez/examples/OrderedWordCount.java to fill in the ? cells.

grep -n "EdgeProperty\|KVOutput\|KVInput\|DataMovementType" \
  tez-examples/src/main/java/org/apache/tez/examples/OrderedWordCount.java

Expected Output

By end of this lab you have:

1. The IPO map table for OrderedWordCount completed
2. An answer for each step question (from code, not from documentation)
3. Understanding of what DataMovementEvent carries and why it exists
4. Knowledge of which configuration key controls sort buffer size

Stretch Goals

1. Find the shuffle HTTP server that serves partition data to Fetcher:

   find . -name "ShuffleHandler.java" | grep -v test
   

   What HTTP framework does it use? What is the URL pattern for fetching a partition?

2. Trace what happens when Fetcher receives corrupted data (a checksum mismatch). Does the task fail immediately? Or does it retry from a different source?

3. Find the EdgeManagerPlugin interface and read its contract:

   find tez-api/src/main/java -name "EdgeManagerPlugin.java"
   

   What three methods must a custom edge manager implement, and what do they do? Why would you use a custom edge manager instead of SCATTER_GATHER?

4. Look at IntersectExample.java in tez-examples. It uses BROADCAST for one edge. Explain why: what is the semantic meaning of broadcasting in a join operation?

Lab 3.3 — Build It: Multi-Input Union DAG

Lab type: Build It — real Maven project, compilable Java, run + break + fix cycle
Estimated time: 90–120 min
Maven module: book/projects/level-3-multi-input
Main class: org.apache.tez.learning.l3.MultiInputDAG

What You Will Build

EvenNumberSource(1) ──even-edge──┐
                                  ├─▶ MultiInputUnionProcessor(1) ──▶ UnionSinkProcessor(1)
OddNumberSource(1)  ──odd-edge───┘

Two source vertices emit separate streams of integers (even: 0,2,4,…,98 and odd: 1,3,5,…,99). A middle vertex receives both streams through two named input edges, unions them, and forwards everything to a terminal sink. The sink sums all values and publishes the result via a Tez counter.

Expected output: TotalSum=4950 PASS

This is the smallest possible Tez program with a multi-input vertex — the same structural pattern used by every Tez join, union, and co-group operation.

Step 1 — Set the Tez Version

Open book/projects/pom.xml and confirm <tez.version> matches your local build:

cd ~/tez-src         # your Tez clone from Lab 1.1
git log --oneline -1 | head -c 60
mvn help:evaluate -Dexpression=project.version -q -DforceStdout 2>/dev/null

If the version printed differs from what is in the POM, update <tez.version> before continuing.

Step 2 — Compile and Run the Unit Tests

cd /path/to/apache-tez/book/projects

# Compile and test the new module only
mvn -pl level-3-multi-input test

You should see:

Tests run: 10, Failures: 0, Errors: 0, Skipped: 0

Read every test in TestMultiInputProcessors.java before moving on.

Questions

Step 3 — Build the Fat JAR and Run the DAG

mvn -pl level-3-multi-input package -q

java -jar level-3-multi-input/target/level-3-multi-input-1.0-SNAPSHOT-jar-with-dependencies.jar

Expected final line:

[MultiInputDAG] TotalSum=4950  expected=4950  PASS

If you see FAIL, the counter value is wrong — note the actual value before proceeding to the debugging exercises.

Step 4 — Read Every Source File

Work through each file in src/main/java/org/apache/tez/learning/l3/.

EvenNumberSource.java

MultiInputUnionProcessor.java

MultiInputDAG.java

Step 5 — Break It: Three Experiments

Perform each experiment, observe the failure, then revert before the next one.

Experiment A — Swap the edge names

In MultiInputDAG.buildDAG(), swap EVEN_EDGE and ODD_EDGE:

.setDestinationEdgeName(MultiInputUnionProcessor.ODD_EDGE)   // was EVEN_EDGE
// ...
.setDestinationEdgeName(MultiInputUnionProcessor.EVEN_EDGE)  // was ODD_EDGE

Rebuild and run.

• Does the DAG succeed or fail?
• Is the sum still 4950?
• Why does swapping the names not cause a failure here, but would cause a failure in a join operation where the left and right inputs have different schemas?

Experiment B — Remove one start() call

In MultiInputUnionProcessor.run(), remove evenInput.start().

Rebuild and run.

• What exception is thrown? On which line?
• Search the Tez source for the method that throws this exception. What is the guard condition?

Experiment C — Make one source emit duplicates

In EvenNumberSource.run(), change int n = i * 2 to int n = 0 (every write uses key=0).

Rebuild and run.

• What is the counter value now?
• Is the DAG PASS or FAIL?
• What does this reveal about how OrderedGroupedKVInput handles duplicate keys when the value type is IntWritable?

Step 6 — Implement a FilterUnionProcessor

Create a new file in the same package: FilterUnionProcessor.java

Specification:

• Extends AbstractLogicalIOProcessor
• Has the same two named inputs as MultiInputUnionProcessor
• Accepts a threshold via UserPayload (key "threshold", default 50)
• Reads from both inputs; only forwards values >= threshold
• Increments counter UnionPipeline/FilteredCount for each record dropped

Wire it into the DAG as a replacement for MultiInputUnionProcessor:

Vertex filter = Vertex.create(
    "FilterUnion",
    ProcessorDescriptor.create(FilterUnionProcessor.class.getName())
        .setUserPayload(UserPayload.create(
            ByteBuffer.wrap("threshold=50".getBytes()))),
    1);

Expected result: With threshold=50, values 0–49 are dropped, values 50–99 are forwarded. Sum at sink = 50+51+…+99 = 3725. FilteredCount = 50.

Step 7 — Tez Source Connection Table

For each class below, locate the corresponding source file in your Tez clone and record the path:

Step 8 — Connect to Real Tez Data Flows

Open tez-examples/src/main/java/org/apache/tez/examples/JoinDataGen.java or OrderedWordCount.java in the Tez source tree.

1. Find a DAG in the examples that has more than 2 vertices.
2. Draw its topology as an ASCII diagram.
3. Identify which vertex is the "union-like" vertex (if any) that receives edges from multiple sources.
4. Compare its processor class to MultiInputUnionProcessor: what is similar, what is different?

Step 9 — JIRA Research

Search issues.apache.org/jira for:

project = TEZ AND text ~ "multi-input" AND resolution = Fixed

Find one resolved issue involving multiple inputs to a single vertex.

• What was the bug?
• Which class was modified?
• Was a test added? If so, what does it test?

Level 4: DAG State Machine Internals

This level takes you inside VertexImpl — the most complex class in the Tez codebase. You will read the full state machine, understand every major state and the conditions that drive transitions, learn how VertexManager plugs in to control scheduling, and understand how speculative execution works. After this level you are capable of diagnosing vertex-level failures from AM log output and writing patches to the state machine.

Learning Objectives

By the end of Level 4 you must be able to:

1. Read a StateMachineFactory definition and produce a transition table from code
2. Explain the full INITIALIZING → INITED → RUNNING → SUCCEEDED path with all preconditions
3. Describe what VertexManager does and when it is invoked
4. Explain the difference between ImmediateStartVertexManager and ShuffleVertexManager
5. Describe the speculative execution trigger conditions and what it causes
6. Trace a vertex failure from first task failure to DAGImpl receiving V_COMPLETED
7. Explain what vertex groups are and why they exist

Reading a StateMachineFactory

Tez uses Hadoop's StateMachineFactory (from hadoop-common). The pattern:

private static final StateMachineFactory<VertexImpl, VertexState, VertexEventType, VertexEvent>
    stateMachineFactory =
        new StateMachineFactory<>(VertexState.NEW)

        // From NEW
        .addTransition(VertexState.NEW,
            VertexState.INITIALIZING,
            VertexEventType.V_INIT,
            new InitTransition())

        // From INITIALIZING
        .addTransition(VertexState.INITIALIZING,
            EnumSet.of(VertexState.INITED, VertexState.FAILED),
            VertexEventType.V_INIT_DONE,
            new InitedTransition())
        ...
        .installTopology();

Reading rules

1. First argument — the source state (where we are now)
2. Second argument — the destination state(s). If an EnumSet, the transition handler decides which destination to return.
3. Third argument — the event type that triggers this transition
4. Fourth argument — the SingleArcTransition or MultipleArcTransition handler

A SingleArcTransition always goes to the same destination state. Its transition() method returns void.

A MultipleArcTransition can go to different states. Its transition() method returns the next VertexState.

When you see an EnumSet as the second argument, look for a MultipleArcTransition implementation — the logic inside that class decides which state to move to.

How to extract the full transition table

# List all addTransition calls in VertexImpl
grep -n "addTransition" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java \
  | wc -l

# Print them all
grep -n "addTransition" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

The Full Vertex State Machine

NEW → INITIALIZING (event: V_INIT)

Triggered by DAGImpl.InitTransition when the DAG is initializing.

Handler: InitTransition

What happens:

• VertexImpl sets up inputsWithInitializers — inputs that require RootInputInitializer
• Registers event handlers for root input initializer completion events
• If there are no root input initializers, immediately posts V_INIT_DONE (transitions to INITED in the same logical step)

Precondition for INITIALIZING → INITED:

• All RootInputInitializers have reported completion
• VertexManager.initialize() has completed

INITED → RUNNING (event: V_START)

Triggered by DAGImpl when all source vertices of this vertex have started or when the vertex has no source edges (it is a root vertex).

Handler: StartTransition

What happens:

• Calls vertexManager.onVertexStarted()
• The VertexManager decides when to schedule tasks

Important: the V_START event does not directly schedule tasks. The VertexManager does, via VertexManagerPlugin.scheduleVertexTasks().

RUNNING task completion handling

Each task completion (success or failure) generates a V_TASK_COMPLETED event.

The TaskCompletedTransition handler:

• Increments the succeeded/failed task counter
• Checks if all tasks are done → if yes, triggers V_COMPLETE_EVENT
• Checks speculative execution conditions
• Checks if failure count exceeds tolerable failures threshold

Key configuration: tez.vertex.failure-tasks-percent.to-fail-vertex — percentage of task failures that cause the entire vertex to fail. Default: 0 (any failure fails the vertex). Setting to > 0 enables partial failure tolerance.

RUNNING → COMMITTING (all tasks succeeded)

Before a vertex is marked SUCCEEDED, its output committers run.

Handler: VertexCommitCallback

What happens:

• OutputCommitter.commitOutput() is called for each output with a committer
• Commit is atomic: either all outputs commit or the vertex fails
• The AM must not fail between task completion and output commit (AM recovery handles this)

RUNNING → FAILED

Triggers:

1. A task exceeds the failure threshold (V_TASK_COMPLETED with failure)
2. A container dies without a task completion report
3. VertexManager reports an error
4. A downstream vertex fails and error propagation is configured

RECOVERING states

When the AM restarts (e.g., due to a node failure), VertexImpl enters RECOVERING states. Recovery reads history events from the timeline service to reconstruct which tasks completed before the AM died, avoiding re-running already-succeeded tasks.

This is the most complex part of VertexImpl. Recovery bugs are a major category of contributor-fixable issues.

VertexManager

VertexManager is the plugin interface that controls task scheduling within a vertex. It sits between the AM framework and the actual task scheduler.

Interface (simplified)

public abstract class VertexManagerPlugin {
    // Called when vertex is initialized; plugin configures itself
    public abstract void initialize() throws Exception;

    // Called when V_START event fires; plugin decides when to schedule tasks
    public abstract void onVertexStarted(List<TaskAttemptIdentifier> completions)
        throws Exception;

    // Called each time a source vertex task completes
    // Plugin uses this to update scheduling decisions (for slow-start)
    public abstract void onSourceTaskCompleted(TaskAttemptIdentifier completedSrcTaskAttempt)
        throws Exception;

    // Called when vertex configuration changes (e.g., auto-parallelism)
    public abstract void onVertexManagerEventReceived(VertexManagerEvent vmEvent)
        throws Exception;
}

ImmediateStartVertexManager

The default for root vertices and vertices with no special scheduling requirements.

Behavior:

• Schedules all tasks immediately when onVertexStarted() is called
• Does not wait for any source task completion
• Used by: Tokenizer vertex in OrderedWordCount

ShuffleVertexManager

Used for vertices that receive SCATTER_GATHER input from a source vertex.

Behavior:

• Implements slow start: waits until a configurable fraction of source tasks have completed before scheduling downstream tasks
• Configuration key: tez.shuffle-vertex-manager.min-src-fraction (default 0.25) and tez.shuffle-vertex-manager.max-src-fraction (default 0.75)
• Implements auto-parallelism: can reduce the downstream vertex's parallelism based on the actual size of shuffle data
• When auto-parallelism reduces parallelism, it calls context.reconfigureVertex() which posts a V_PARALLELISM_UPDATED event to VertexImpl

Why VertexManager Matters for Contributors

Auto-parallelism and slow-start bugs are a major category of Tez issues. The interaction between ShuffleVertexManager and VertexImpl involves:

• Parallelism changes after task scheduling
• Race conditions between task completion events and parallelism updates
• Recovery of vertices that had parallelism changed before AM death

Speculative Execution

Speculative execution launches a duplicate task attempt when the original attempt is slow.

Trigger conditions

VertexImpl checks speculation conditions in TaskCompletedTransition and on a periodic timer:

1. At least one task has completed (we have a baseline for "normal" task duration)
2. The running attempt has been running longer than speculative_threshold * median_time
3. The running attempt's progress is lower than expected for its elapsed time

Configuration:

tez.am.speculation.enabled = true   (default: false)
tez.am.speculation.interval-ms = 5000  (check interval)

What happens

1. VertexImpl posts a TaskEventType.T_ADD_SPEC_ATTEMPT event to TaskImpl
2. TaskImpl creates a new TaskAttemptImpl
3. Both attempts run concurrently
4. The first to succeed wins; the other is killed
5. The winning attempt's output is committed; the losing attempt's output is discarded

Interaction with ShuffleVertexManager

If a speculative attempt completes, ShuffleVertexManager receives an onSourceTaskCompleted callback for the winning attempt. It must de-duplicate: the task's output should only be counted once regardless of which attempt succeeded.

Vertex Groups

Vertex groups (VertexGroup in the API) allow multiple vertices to be treated as a single logical vertex for downstream consumption.

Use case: merging the output of multiple Map vertices before a single Reduce vertex, without an intermediate shuffle. This is used in the Hive UnionAll operator implementation.

Key classes:

• VertexGroup API: tez-api/src/main/java/org/apache/tez/dag/api/VertexGroup.java
• GroupInputEdge: an edge from a VertexGroup to a regular vertex
• The downstream vertex sees a single MergedLogicalInput that combines all group members

Key Classes for This Level

# Find the VertexManager implementations
find tez-dag/src/main/java -name "*VertexManager*.java" | grep -v test

JIRA Categories for Level 4 Contributors

You are now ready to investigate and submit patches for:

• Vertex failure handling bugs — incorrect state transitions, wrong error messages
• VertexManager logic bugs — slow-start fraction calculation, auto-parallelism edge cases
• Recovery bugs — vertices that fail to recover correctly after AM restart
• Speculation bugs — duplicate completions, wrong trigger conditions
• Test improvements — TestVertexImpl has hundreds of tests; adding coverage for edge cases

Approach:

1. Find a TestVertexImpl test that is @Ignored — read the comment explaining why
2. If the bug is fixed, the @Ignore can be removed (a trivial but real contribution)
3. Or find a state machine transition that has no test coverage (grep for the transition, then grep for the handler class name in test files)

Deliverables

• Extract the complete VertexImpl state transition table (all source states, event types, destination states) from the code
• Explain ShuffleVertexManager slow-start in your own words, with the relevant config keys
• Trace a vertex failure through TaskImpl → VertexImpl → DAGImpl using event type names
• Identify one @Ignored test in TestVertexImpl and read why it is ignored
• Lab 4.1 completed: full state machine map documented
• Lab 4.2 completed: VertexManager walkthrough complete

Common Mistakes

Lab 4.1: Read the VertexImpl State Machine

Background

VertexImpl.java is the most complex class in Apache Tez. It is approximately 6,000 lines long and contains the complete state machine for vertex execution, including initialization, scheduling, task completion handling, failure handling, speculative execution, and AM recovery. Reading it systematically — rather than linearly — is the skill this lab builds.

The output of this lab is a complete state transition table that you have produced from the source code, without reference to any external documentation.

How to Read a Large State Machine Class

Do not read VertexImpl.java from top to bottom. Instead:

1. Start with the StateMachineFactory declaration (search for stateMachineFactory =)
2. Extract all addTransition calls — this gives you the complete transition table
3. For each transition, find the handler class — the inner class that implements SingleArcTransition or MultipleArcTransition
4. Read each handler's transition() method — this is the actual state machine logic
5. Trace inter-state-machine events — where does the handler post events to other state machines?

Step-by-Step Tasks

Step 1: Find the StateMachineFactory

grep -n "stateMachineFactory" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head -5

Note the line number. The factory declaration starts there and continues for hundreds of lines. Read the entire factory definition — do not skip any transitions.

Step 2: Count All States and Transitions

# Count distinct source states referenced in addTransition
grep "addTransition(VertexState\." \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java \
  | sed 's/.*addTransition(VertexState\.\([A-Z_]*\).*/\1/' \
  | sort -u

# Count total transitions
grep -c "addTransition" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

Record your numbers. You should find more than 30 distinct source states and more than 80 transitions.

Step 3: Build the Transition Table

For each line in the StateMachineFactory, extract:

• Source state
• Event type
• Destination state(s)
• Handler class name

Begin with the transitions from NEW:

# Find all transitions FROM NEW
awk '/addTransition\(VertexState\.NEW/,/\.addTransition/' \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java \
  | head -20

Then from INITIALIZING:

grep -A4 "addTransition(VertexState\.INITIALIZING" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head -40

Build a table with columns: Source State | Event | Destination | Handler.

Step 4: Trace the Happy Path

The happy path for a vertex with no source edges (a root vertex, e.g., Tokenizer):

NEW
  V_INIT → INITIALIZING (InitTransition)
    V_INIT_DONE → INITED (InitedTransition — if no root input initializers)
  V_START → RUNNING (StartTransition)
    [VertexManager schedules tasks]
    [All tasks complete successfully]
    V_TASK_COMPLETED (final task) → COMMITTING (TaskCompletedTransition)
    V_COMMIT_COMPLETED → SUCCEEDED (CommitCompletedTransition)

For each transition in the happy path, find the handler class and answer:

InitTransition.transition():

grep -n "class InitTransition" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

• What does InitTransition do when there are no RootInputInitializers?
• Does it immediately post V_INIT_DONE, or is there an intermediate step?

InitedTransition.transition() (or whatever class handles V_INIT_DONE):

• When does INITIALIZING go to INITED vs going directly to RUNNING?
• What is the condition that allows immediate transition to RUNNING?

StartTransition.transition():

• What method on VertexManager is called here?
• Does this method block or is it asynchronous?

TaskCompletedTransition.transition():

• How does it track whether all tasks have completed?
• What is numSuccessSourceAttemptCompletions?
• At what point does it decide the vertex can move to COMMITTING?

Step 5: Trace the Failure Path

A task fails. The event chain:

TaskAttemptImpl: RUNNING → FAILED (sends T_ATTEMPT_FAILED to TaskImpl)
  TaskImpl: RUNNING → FAILED (if retry limit exceeded; sends V_TASK_COMPLETED{FAILED})
    VertexImpl: RUNNING → ?

Find the handler for V_TASK_COMPLETED when the task is FAILED:

# TaskCompletedTransition handles both success and failure
grep -n "TaskCompletedTransition" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head -10

Answer:

1. What field tracks the number of failed tasks?
2. What is the condition that causes the vertex to transition to FAILED?
3. What event does VertexImpl send to DAGImpl when it fails?
4. Does DAGImpl fail immediately when a vertex fails, or does it try to continue?

# Find how DAGImpl handles vertex failure
grep -n "DAG_VERTEX_COMPLETED\|vertexFailed\|VERTEX_FAILED" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java | head -15

Step 6: Find the RECOVERING States

grep "RECOVERING" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java \
  | grep "VertexState\." | head -20

Answer:

1. How many RECOVERING_* states exist?
2. What event exits the RECOVERING state?
3. What class handles recovery completion?

Step 7: Find All @Ignored Tests in TestVertexImpl

grep -n "@Ignore" \
  tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java

For each @Ignored test:

1. Read the comment explaining why it is ignored
2. Determine if the bug has been fixed (search JIRA for the referenced issue number)
3. If the fix exists, the test can likely be re-enabled — this is a contributor opportunity

Step 8: Find a Transition with No Test Coverage

Pick three transition handler classes from your transition table. For each, check if TestVertexImpl has a test that exercises that handler:

# Example: does TestVertexImpl test TaskCompletedTransition?
grep -n "TaskCompletedTransition\|taskCompletedTransition" \
  tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java | head -5

# If none found, search for tests that trigger V_TASK_COMPLETED
grep -n "V_TASK_COMPLETED\|VertexEventType.V_TASK_COMPLETED" \
  tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java | head -10

Identify one transition that appears to have insufficient test coverage and document it. This is a potential Test JIRA issue you could file and fix.

Deliverable: Your Transition Table

Produce a table in this format (populate all rows from code):

| Source State      | Event Type          | Destination       | Handler Class            |
|---|---|---|---|
| NEW               | V_INIT              | INITIALIZING      | InitTransition           |
| INITIALIZING      | V_INIT_DONE         | INITED / FAILED   | InitedTransition         |
| INITED            | V_START             | RUNNING           | StartTransition          |
| RUNNING           | V_TASK_COMPLETED    | RUNNING/SUCCEEDED | TaskCompletedTransition  |
| ...               | ...                 | ...               | ...                      |

Your table should have at least 30 rows (covering the main execution paths). Recovery states are optional for this level.

Expected Output

1. A complete (or near-complete) state transition table for VertexImpl
2. Answers to all questions in Steps 4–6 with file:line references
3. List of @Ignored tests with your assessment of whether they could be re-enabled
4. One transition identified as having insufficient test coverage

Stretch Goals

1. Produce the same transition table for TaskImpl and TaskAttemptImpl. Compare their complexity (number of states and transitions) to VertexImpl.

2. Find all places where VertexImpl calls eventHandler.handle() to post an event to another state machine. What are the target state machines and what event types are used?

   grep -n "eventHandler.handle" \
     tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java \
     | grep -v "VertexEvent" | head -20
   

3. Find the V_PARALLELISM_UPDATED transition — what does it do, and why is it one of the most bug-prone transitions in the state machine?

Lab 4.2: VertexManager Deep Dive

Background

VertexManager is the hook that makes Tez more than just a DAG scheduler. By plugging in a custom VertexManagerPlugin, applications can implement dynamic parallelism, slow start, skew handling, and custom task scheduling — without modifying the core AM.

This lab walks through the two built-in VertexManager implementations, explains their behaviors via code reading, and ends with a minimal custom VertexManagerPlugin that you write and unit-test.

The VertexManagerPlugin Contract

Full interface: tez-api/src/main/java/org/apache/tez/dag/api/VertexManagerPlugin.java

public abstract class VertexManagerPlugin {
    private VertexManagerPluginContext context;

    // Called once by the AM to provide the context object
    public final void setContext(VertexManagerPluginContext context) { ... }

    // The plugin implementation must implement these:
    public abstract void initialize() throws Exception;
    public abstract void onVertexStarted(List<TaskAttemptIdentifier> completions)
        throws Exception;
    public abstract void onSourceTaskCompleted(
        TaskAttemptIdentifier completedSrcTaskAttempt) throws Exception;
    public abstract void onVertexManagerEventReceived(
        VertexManagerEvent vmEvent) throws Exception;
    // Called when an input is initialized (root inputs only):
    public void onRootVertexInitialized(String inputName,
        InputDescriptor inputDescriptor, List<Event> events) throws Exception {}
}

VertexManagerPluginContext — what the plugin can call back into

find tez-api/src/main/java -name "VertexManagerPluginContext.java"
cat $(find tez-api/src/main/java -name "VertexManagerPluginContext.java")

Key methods on the context:

Reading ImmediateStartVertexManager

find tez-dag/src/main/java -name "ImmediateStartVertexManager.java"
cat $(find tez-dag/src/main/java -name "ImmediateStartVertexManager.java")

Answer these questions from the code:

1. In initialize(): does ImmediateStartVertexManager do anything? If not, why does it exist?
2. In onVertexStarted(): does it schedule tasks immediately or wait for anything?
3. What TaskWithLocation does it create for each task? Does it provide any location hints?
4. Does it implement onSourceTaskCompleted()? If so, what does it do?

Expected finding: ImmediateStartVertexManager is intentionally minimal. Its purpose is to provide a named, testable implementation that schedules all tasks immediately with no location hints. It is the baseline from which ShuffleVertexManager diverges.

Reading ShuffleVertexManager

find tez-dag/src/main/java -name "ShuffleVertexManager.java"
wc -l $(find tez-dag/src/main/java -name "ShuffleVertexManager.java")

Slow Start

Find the slow-start logic in onSourceTaskCompleted().

grep -n "minFraction\|maxFraction\|min-src-fraction\|completedSourceTasks\|pendingTasksToSchedule" \
  $(find tez-dag/src/main/java -name "ShuffleVertexManager.java") | head -20

Answer:

1. What is the variable that tracks how many source tasks have completed?
2. At what fraction does ShuffleVertexManager start scheduling tasks?
3. What is the formula: at fraction F between minFraction and maxFraction, what percentage of downstream tasks are scheduled?

Auto-Parallelism

Find the auto-parallelism logic:

grep -n "reconfigureVertex\|numBipartiteSourceTasks\|desiredTaskInputSize\|targetParallelism" \
  $(find tez-dag/src/main/java -name "ShuffleVertexManager.java") | head -20

Answer:

1. What configuration key enables auto-parallelism?
2. What information does ShuffleVertexManager use to compute the optimal parallelism?
3. When is context.reconfigureVertex() called?
4. What is the minimum parallelism ShuffleVertexManager will ever set (the floor)?

VertexManagerEvent handling

When auto-parallelism is enabled, each upstream task sends a VertexManagerEvent to the downstream VertexManagerPlugin containing statistics about its output (byte count, record count, partition sizes).

grep -n "VertexManagerEvent\|onVertexManagerEventReceived\|vmEvent" \
  $(find tez-dag/src/main/java -name "ShuffleVertexManager.java") | head -15

Answer:

1. What protobuf message is decoded from the event payload?
2. What statistic is accumulated across all events?
3. How does ShuffleVertexManager use the accumulated statistics to decide on new parallelism?

Write a Minimal Custom VertexManager

Create a CountingVertexManager that:

1. Schedules 50% of tasks immediately when onVertexStarted() is called
2. Schedules the remaining tasks when all source tasks have completed
3. Logs the number of scheduled tasks at each scheduling call

This is the core pattern of slow-start, stripped to its minimum.

Implementation skeleton

package org.apache.tez.dag.library.vertexmanager;

import org.apache.tez.dag.api.VertexManagerPlugin;
import org.apache.tez.dag.api.VertexManagerPluginContext;
import org.apache.tez.dag.api.TaskAttemptIdentifier;
import org.apache.tez.dag.api.event.VertexManagerEvent;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.ArrayList;
import java.util.List;

public class CountingVertexManager extends VertexManagerPlugin {

    private static final Logger LOG =
        LoggerFactory.getLogger(CountingVertexManager.class);

    private int totalSourceTasks = 0;
    private int completedSourceTasks = 0;
    private boolean secondBatchScheduled = false;
    private int totalTasksToSchedule = 0;

    @Override
    public void initialize() {
        totalTasksToSchedule = getContext().getCurrentParallelism();
        // Count source tasks across all input vertices
        for (String inputVertex : getContext().getInputVertexEdgeProperties().keySet()) {
            totalSourceTasks += getContext().getVertexNumTasks(inputVertex);
        }
    }

    @Override
    public void onVertexStarted(List<TaskAttemptIdentifier> completions) {
        // Schedule first 50%
        int firstBatch = totalTasksToSchedule / 2;
        List<VertexManagerPluginContext.ScheduleTaskRequest> toSchedule = new ArrayList<>();
        for (int i = 0; i < firstBatch; i++) {
            toSchedule.add(VertexManagerPluginContext.ScheduleTaskRequest.create(i, null));
        }
        LOG.info("CountingVertexManager: scheduling first batch of {} tasks", firstBatch);
        getContext().scheduleTasks(toSchedule);
    }

    @Override
    public void onSourceTaskCompleted(TaskAttemptIdentifier completedSrcTaskAttempt) {
        completedSourceTasks++;
        if (!secondBatchScheduled && completedSourceTasks >= totalSourceTasks) {
            // Schedule remaining 50%
            int firstBatch = totalTasksToSchedule / 2;
            List<VertexManagerPluginContext.ScheduleTaskRequest> toSchedule = new ArrayList<>();
            for (int i = firstBatch; i < totalTasksToSchedule; i++) {
                toSchedule.add(VertexManagerPluginContext.ScheduleTaskRequest.create(i, null));
            }
            LOG.info("CountingVertexManager: scheduling second batch of {} tasks",
                toSchedule.size());
            getContext().scheduleTasks(toSchedule);
            secondBatchScheduled = true;
        }
    }

    @Override
    public void onVertexManagerEventReceived(VertexManagerEvent vmEvent) {
        // No-op: we don't need statistics for this simple implementation
    }
}

Implementation tasks

1. Identify the correct API method: getContext().scheduleTasks() vs getContext().scheduleVertexTasks() — check which one exists in your version of the API.

2. Write a unit test using MockVertexManagerPluginContext (if it exists) or a mock:

   • Initialize the manager with parallelism = 10 and 4 source tasks
   • Call onVertexStarted() — verify 5 tasks are scheduled
   • Call onSourceTaskCompleted() 4 times — verify remaining 5 tasks are scheduled on the 4th call
   • Verify secondBatchScheduled is true after

3. Register the CountingVertexManager in a DAG:

   Vertex reduceVertex = Vertex.create("reducer",
       ProcessorDescriptor.create(MyReducer.class.getName()), 10);
   reduceVertex.setVertexManagerPlugin(
       VertexManagerPluginDescriptor.create(CountingVertexManager.class.getName()));
   

Finding the VertexManager Test Utilities

# Find mock context for testing
find tez-dag/src/test -name "*Mock*Vertex*" -o -name "*VertexManager*Test*" | grep -v ".class"

# Find TestShuffleVertexManager
find . -name "TestShuffleVertexManager.java" | grep test

Read TestShuffleVertexManager.java to understand how VertexManager tests are structured. The test creates a mock context, calls lifecycle methods in order, and asserts which tasks were scheduled.

Expected Output

1. Answers to all questions in the ImmediateStartVertexManager and ShuffleVertexManager sections, with file:line references
2. A working CountingVertexManager implementation that compiles
3. A unit test that passes for the two scheduling scenarios

Stretch Goals

1. Read CartesianProductVertexManager — the most complex VertexManager:

   find tez-dag/src/main/java -name "CartesianProductVertexManager.java"
   

   What computation does it coordinate? When is it used?

2. Find a ShuffleVertexManager related JIRA (search for "ShuffleVertexManager" in JIRA). Read the issue description and the patch. What invariant was violated?

3. Implement a NoOpVertexManager that schedules no tasks (for testing DAG failure paths). Use it in a test DAG and verify the vertex fails with FAILED status after the timeout.

Lab 4.3 — Build It: WavingVertexManager

Lab type: Build It — VertexManagerPlugin with full JUnit + Mockito test suite
Estimated time: 120–150 min
Maven module: book/projects/level-4-waving-manager
Key class: org.apache.tez.learning.l4.WavingVertexManager

What You Will Build

A VertexManagerPlugin that schedules tasks in configurable waves:

• Wave 0: tasks 0 to waveSize-1
• Wave 1: tasks waveSize to 2×waveSize-1
• Wave N: starts only when all tasks in wave N-1 have succeeded

Wave size is read from UserPayload as "waveSize=N".
Default: WavingVertexManager.DEFAULT_WAVE_SIZE = 2.

This is a minimal but complete VertexManagerPlugin — the same architectural pattern used by ImmediateStartVertexManager, ShuffleVertexManager, and the VertexManagerPlugin inside every Hive-on-Tez reduce vertex.

Step 1 — Understand the VertexManagerPlugin Contract

Before reading any code, open the Tez source:

find ~/tez-src -name "VertexManagerPlugin.java" | head -3
find ~/tez-src -name "VertexManagerPluginContext.java" | head -3
find ~/tez-src -name "ImmediateStartVertexManager.java" | head -3

Read all three files completely. Then answer:

Step 2 — Compile and Run the Tests

cd /path/to/apache-tez/book/projects
mvn -pl level-4-waving-manager test

Expected:

Tests run: 13, Failures: 0, Errors: 0, Skipped: 0

Step 3 — Read the Source Code

Open WavingVertexManager.java and work through every section.

initialize()

onVertexStarted()

onTaskAttemptCompleted()

scheduleNextWave()

Step 4 — Read the Test Suite

Open TestWavingVertexManager.java. For each test, before reading the assertions:

1. Read the test name
2. Predict what the test will assert
3. Then read the actual assertions and compare to your prediction

Pay particular attention to how Mockito is used:

Questions

Step 5 — Break It: Three Experiments

Experiment A — Remove the if (!successful) return guard

Delete the early-return in onTaskAttemptCompleted. Run:

mvn -pl level-4-waving-manager test -Dtest=TestWavingVertexManager#testFailedAttemptDoesNotAdvanceWave

• Which test fails?
• What is the actual vs. expected scheduleVertexTasks call count?
• Why does treating failures as successes cause premature wave advancement?

Experiment B — Remove the BitSet.clone() in checkAndScheduleNextWave

Change:

BitSet scheduledCopy = (BitSet) scheduled.clone();
scheduledCopy.andNot(waveFinished);

to:

scheduled.andNot(waveFinished);

Run the full test suite.

• Which tests fail?
• What data corruption does this mutation cause? Trace through testThreeWavesForSixTasks manually.

Experiment C — Change count < waveSize to count <= waveSize

In scheduleNextWave(), change the loop condition.

• How many tasks does wave 0 now schedule?
• Which test catches this?

Step 6 — Add a New Feature: onVertexManagerEventReceived

The real ShuffleVertexManager uses onVertexManagerEventReceived to receive partition statistics from map tasks. Add support for a simple variant:

Create a new callback method:

@Override
public void onVertexManagerEventReceived(
        List<VertexManagerEvent> vmEvents) throws Exception {
    // If any event's user payload contains "skip=true", mark
    // that task as finished so it does not block wave advancement.
    for (VertexManagerEvent event : vmEvents) {
        // TODO: parse UserPayload for "skip=true"; if present, call
        //       onTaskAttemptCompleted(taskIndex, true) to release the wave
    }
}

Write a test for this method:

@Test
public void testSkipEventReleasesWave() {
    // set up 4 tasks, wave size 2
    // trigger onVertexStarted (wave 0: tasks 0,1)
    // send a VertexManagerEvent for task 0 with payload "skip=true"
    // verify task 0 is treated as done for wave-completion purposes
}

Step 7 — Tez Source Connection Table

Step 8 — ShuffleVertexManager Deep Dive

Open ShuffleVertexManager.java in the Tez source:

find ~/tez-src -name "ShuffleVertexManager.java"

1. Read onVertexStarted(). Does it schedule tasks immediately like WavingVertexManager, or does it wait? What does it wait for?
2. Find the slowStartFraction field. How does it determine when to start scheduling?
3. Find where reconfigureVertex() is called. What does it change about the vertex?
4. How does ShuffleVertexManager prevent double-scheduling? Compare its guard to the scheduled BitSet in WavingVertexManager.
5. ShuffleVertexManager has ~700 lines. Identify the 5 most important methods (the ones that contain the core scheduling logic) and list them.

Step 9 — JIRA Research: VertexManager Bugs

Search:

project = TEZ AND component = "tez-dag" AND text ~ "VertexManager" AND resolution = Fixed

Find one resolved issue where a VertexManagerPlugin had a scheduling bug.

• What was the bug? (Race condition? Double scheduling? Wrong wave boundary?)
• What was the fix?
• Was a test added? What does it mock?

Lab 4.4 — Fix It: Null Dereference in ShuffleVertexManager on Zero-Partition Source

Lab type: Fix-It — reproduce → locate → write failing test → patch → verify → format patch
Estimated time: 120–150 min
Tez component: tez-dag → org.apache.tez.dag.app.dag.impl.ShuffleVertexManager

Background

ShuffleVertexManager uses partition statistics sent by map tasks to decide when to start reduce tasks (slow-start) and how many reducers to run (auto-parallelism). It processes these statistics via onVertexManagerEventReceived().

A long-standing bug category in this path: when a source vertex has zero output partitions (all records were filtered, or the vertex ran with zero tasks), the plugin can receive a ShuffleVertexManager.VertexManagerEvent whose payload encodes 0 partitions. In several versions of Tez, this caused a NullPointerException or ArithmeticException (divide by zero) deep in the statistics-processing path — the code assumed at least one partition existed.

This lab reproduces the bug pattern in a unit test, locates the exact guard that is missing, applies the fix, and submits a patch.

Step 1 — Locate the Source File

find ~/tez-src -name "ShuffleVertexManager.java" | head -5

Expected:

./tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/ShuffleVertexManager.java

Also locate the test file:

find ~/tez-src -name "TestShuffleVertexManager.java" | head -5

Step 2 — Read the Statistics Path

In ShuffleVertexManager.java, find the method that processes VertexManagerEvent payloads. It will have a call to ShuffleVertexManagerBase.parseStatsHeader() or similar, and will work with numPartitions or partitionCount.

Trace the complete call chain from onVertexManagerEventReceived() to the line that first uses the partition count arithmetically.

Questions

Step 3 — Find the Existing Test

find ~/tez-src -name "TestShuffleVertexManager.java"

Open it and search for any test that covers the zero-partition case:

grep -n "zero\|0.*partition\|partition.*0" TestShuffleVertexManager.java -i | head -20

Note: in most Tez versions there is no such test — that is the gap you will fill.

Step 4 — Write the Reproducing Test

Add the following test to TestShuffleVertexManager.java. The exact helper methods depend on the version you have; adapt the setup pattern from the nearest existing test (look for testAutoParallelism or testSlowStart).

@Test(expected = Exception.class)   // replace Exception with the specific type you observe
public void testZeroPartitionSourceDoesNotCrash() throws Exception {
    // TODO: set up a ShuffleVertexManager with auto-parallelism enabled
    // TODO: send a VertexManagerEvent with numPartitions = 0
    // TODO: call onVertexManagerEventReceived with that event
    //       The call should NOT throw — once fixed.
    //       Mark expected = Exception.class so the test initially *passes*
    //       when the bug exists (the code throws), then change to asserting
    //       no throw after the fix is applied.
}

Run:

cd ~/tez-src
mvn test -pl tez-dag -Dtest=TestShuffleVertexManager#testZeroPartitionSourceDoesNotCrash -q 2>&1 | tail -30

Record: which exception is thrown and on which line.

Step 5 — Apply the Fix

In ShuffleVertexManager.java, add a guard at the point identified in Step 2.

Rules

• The guard must be a minimum: either if (partitionCount == 0) { return; } to skip the event, or if (partitionCount == 0) { partitionCount = 1; } to normalise (choose the semantically correct one — which is safer for scheduling?)
• Do not reformat surrounding code
• Do not change method signatures

Step 6 — Update the Test

Now that the fix is applied, update the test:

@Test
public void testZeroPartitionSourceDoesNotCrash() throws Exception {
    // Same setup as before
    // This time assert NO exception is thrown
    // Optionally assert that scheduling state is unchanged
}

Run the full tez-dag test suite:

mvn test -pl tez-dag -q 2>&1 | tail -20

All tests must pass.

Step 7 — Checkstyle

mvn checkstyle:check -pl tez-dag -q 2>&1 | grep -E "ERROR|WARNING|violation" | head -20

Zero violations required.

Step 8 — Format the Patch

cd ~/tez-src
git diff > /tmp/TEZ-ZEROPART.001.patch
cat /tmp/TEZ-ZEROPART.001.patch

Checklist:

• Only ShuffleVertexManager.java and TestShuffleVertexManager.java modified
• No trailing whitespace: grep -P "\\s+$" /tmp/TEZ-ZEROPART.001.patch
• Patch applies cleanly: git apply --check /tmp/TEZ-ZEROPART.001.patch
• All tests pass after git apply

Step 9 — Write the JIRA Description

Summary: ShuffleVertexManager throws [ExceptionType] when source vertex
         has zero output partitions

Description:
  When a source vertex completes with zero output partitions (all records
  filtered or vertex ran zero tasks), ShuffleVertexManager.onVertexManagerEventReceived
  receives a VertexManagerEvent with partitionCount=0.  The statistics
  processing path performs arithmetic on this value without a zero guard,
  causing [ExceptionType] at [ClassName].java:[line].

  Steps to reproduce:
    See attached TestShuffleVertexManager#testZeroPartitionSourceDoesNotCrash.

  Fix:
    Add a zero-partition guard at [method name], line [N].
    Skip or normalise the event when partitionCount == 0.

Priority: Major
Component: tez-dag
Affects Version: 0.10.x

Step 10 — Deeper Understanding

After completing the fix, answer these questions by reading ShuffleVertexManager.java:

Level 5: Testing Infrastructure

Apache Tez has one of the most complete test suites in the Hadoop ecosystem: thousands of unit tests, a MiniTezCluster integration harness, and a TestOrderedWordCount end-to-end reference. At this level you will move from reading tests to writing them — adding missing coverage to TestVertexImpl, submitting a real DAG against MiniTezCluster, and finding and fixing a flaky test.

Why testing matters for contributors

Every Tez patch must include either (a) a new test that fails without the patch and passes with it, or (b) a clear justification in the JIRA for why a test is not needed. Committers will block patches that regress existing tests or that add unverified logic.

What this level covers

Prerequisites

• Level 4 complete (you understand VertexImpl state machine and VertexManagerPlugin)
• Tez source checked out and mvn install -DskipTests succeeded

Test categories and Maven commands

Key test classes

Expected outcome

By the end of this level you will have:

1. Run a DAG against MiniTezCluster inside a JUnit test
2. Added a missing state-machine transition test to TestVertexImpl
3. Identified and fixed a flaky test (or documented why it flakes)

Lab 5.1 — Explore MiniTezCluster and TestOrderedWordCount

Lab type: Read & Run
Estimated time: 90 min
Tez module: tez-tests
Key class: org.apache.tez.test.TestOrderedWordCount

Overview

MiniTezCluster spins up an in-process YARN ResourceManager, NodeManager, HDFS NameNode, and DataNode, plus the Tez ApplicationMaster — all inside a single JVM. This lets you submit real DAGs in a JUnit test with no external infrastructure.

TestOrderedWordCount is the canonical example: it submits a multi-stage word-count DAG (tokenize → partition → sort → count) and asserts correct output.

Step 1 — Locate the Files

find ~/tez-src -name "MiniTezCluster.java" | head -5
find ~/tez-src -name "TestOrderedWordCount.java" | head -5
find ~/tez-src -name "MiniTezClusterWithTez.java" | head -5

Step 2 — Read MiniTezCluster.java

Open MiniTezCluster.java and answer:

Step 3 — Read TestOrderedWordCount.java

Work through the test lifecycle:

3a — @BeforeClass setUpClass()

3b — @Test testOrderedWordCount()