Level 1: Hadoop and Tez Foundation

This level establishes the technical baseline every subsequent level depends on. You will understand where Tez fits in the Hadoop ecosystem, successfully build the project from source, run the test suite, and execute your first Tez DAG in local mode.

Learning Objectives

By the end of Level 1 you must be able to:

1. Explain where Apache Tez sits in the Hadoop ecosystem and why it exists
2. Build Apache Tez from source using Maven, with and without tests
3. Execute unit tests scoped to a single module and interpret the results
4. Run a simple Tez DAG in local mode without a YARN cluster
5. Locate any class mentioned in Levels 2–9 without using a search engine
6. Articulate the difference between a MapReduce job and a Tez DAG at the execution model level
7. Read TezConfiguration.java and find any configuration key by category

The Hadoop Ecosystem Context

Apache Tez lives inside the Hadoop ecosystem. Before touching a line of Tez code, build an accurate mental model of the stack:

┌─────────────────────────────────────────────────────┐
│         Apache Hive / Apache Pig / Cascading        │  ← Query / scripting layer
├─────────────────────────────────────────────────────┤
│                  Apache Tez                         │  ← DAG execution engine
├─────────────────────────────────────────────────────┤
│                  Apache YARN                        │  ← Cluster resource management
├─────────────────────────────────────────────────────┤
│                  Apache HDFS                        │  ← Distributed storage
└─────────────────────────────────────────────────────┘

YARN (Yet Another Resource Negotiator) manages cluster resources. It runs an ApplicationMaster (AM) per application, allocates containers, and monitors health. Tez's DAGAppMaster IS a YARN ApplicationMaster.

HDFS stores input, output, and sometimes intermediate data. Tez prefers to keep intermediate data on local disk or in memory, but falls back to HDFS for recovery and large-scale shuffles.

Tez submits a DAGAppMaster to YARN, which requests containers for task execution. Tasks read inputs, execute processors, and write outputs — either directly to downstream tasks via shuffle or to HDFS for final output.

MapReduce vs. Tez

For a 10-stage Hive aggregation query, MapReduce requires 10 separate MR jobs with HDFS writes between every stage. Tez runs the same query as a single DAG — no HDFS round-trips between stages, containers reused across task waves, and pipeline-style data movement between compatible vertices.

Required Reading

Complete in this order before starting the labs:

Note on reading strategy: In a mature Apache codebase, Javadoc is often the best documentation that exists. Class-level Javadoc on public API classes reflects decisions debated and agreed upon by committers. Read it seriously.

Source Code Areas to Inspect

Read these files before and after the labs. You are not modifying anything yet.

tez-api — Public API

tez-dag — Core Execution Engine

tez-runtime-library — I/O Implementations

tez-examples — Reference Implementations

Key Classes Quick Reference

JIRA Issue Categories for Level 1 Contributors

At this stage, focus exclusively on:

• Documentation — Javadoc typos, outdated parameter descriptions, missing @param or @return annotations, broken links in comments
• Test improvements — Adding missing assertions to existing tests, improving test method naming, removing dead code from test classes
• Checkstyle violations — Unused imports, line length violations, missing final keywords

How to find these:

1. Go to Apache Tez JIRA
2. Search: project = TEZ AND labels = "newbie" AND resolution = Unresolved
3. Also scan: project = TEZ AND component = "Documentation" AND resolution = Unresolved
4. Look at recently closed "trivial" issues to understand the standard for accepted patches

Warning: Do not pick up a JIRA issue and immediately upload a patch. Read all existing comments. If there is an active discussion or existing assignee, move on. Leave a comment saying you are investigating before you claim an issue.

Deliverables

You must demonstrate all of the following before advancing to Level 2:

• Successful mvn install -DskipTests output — no build failures
• At least one unit test class run successfully (e.g., TestDAGImpl)
• Successful local DAG execution showing DAG completed: SUCCEEDED
• Ability to locate DAGAppMaster, TezClient, and OrderedGroupedKVInput by memory
• Written explanation (2–3 sentences) of why a Tez DAG is faster than chained MapReduce
• Written explanation of the difference between a YARN container and a Tez task

Common Mistakes

How to Verify Success

# 1. Full build without tests
cd /path/to/tez
mvn install -DskipTests -q && echo "BUILD OK"

# 2. Unit test from tez-dag
mvn test -pl tez-dag -am -Dtest=TestDAGImpl -q

# 3. Local DAG run (from Lab 1.3)
# Expected final output line:
#   DAG: [OrderedWordCount] finished with status: [SUCCEEDED]

Patch Profile: Level 1 Graduate

You are not ready to submit: bug fixes in state machines, new features, performance patches, or changes to the shuffle path. Those require Levels 3–7.