Lab 1.1: Build Apache Tez from Source

Background

Apache Tez is a large, multi-module Maven project. Building it from source is the mandatory first step for any contributor: you need the ability to compile, rebuild a single module, run tests against your local changes, and assemble a runnable distribution. Unlike the OpenSearch curriculum's Gradle build, Tez does not bundle its own JDK or provision its own toolchain — you supply a correct JDK, a modern Maven, and a Protocol Buffers compiler that matches the pinned version. Get those three right and the build is smooth; get them wrong and you will burn an afternoon on cryptic protoc and javac errors that have nothing to do with your code.

This lab walks the full build: clone, verify prerequisites against the real root pom.xml, run the reactor build with the skip flags that actually work, assemble the tez-dist tarballs, build a single module incrementally, and import into IntelliJ with the protobuf generated-sources gotcha handled. Everything here is verified against the current master checkout — but versions drift, so every prerequisite is paired with a grep you run yourself. Never trust a number in this book; trust the command that prints it.

Why This Lab Matters for Contributors

• You cannot submit a credible patch without first proving the project builds cleanly on your machine. A "build is broken" comment on your JIRA is an instant credibility hit.
• Knowing which Maven flag touches which module saves hours of needless full reactor builds. The inner loop is -pl <module> -am, not mvn install.
• A clean build baseline is what lets you tell a real regression from a local mistake. When a test fails, you must know whether it was already failing on pristine master.
• The same apache-rat:check, checkstyle:check, and spotbugs goals that gate CI run locally. Running them before you upload a patch is the difference between a one-round review and five.

Prerequisites

Verify each of these before you start. Run them from an empty directory first, then again from inside the clone once you have it.

java -version     # JDK 21 or newer (see Step 2 for why)
mvn -version      # Maven 3.9.14 or newer
git --version     # 2.x
protoc --version  # OPTIONAL — see Step 3; the build can supply its own

• A JDK whose major version satisfies the root pom's maven.compiler.release.
• Maven 3.9.14 or newer.
• At least 10 GB free disk. Dependencies, target/ outputs, and the two dist tarballs add up. The ~/.m2/repository cache alone runs to several GB after a first build.
• At least 8 GB RAM (16 GB comfortable). The tez-dag test JVMs are memory-hungry.
• Network access for the first build (downloads all dependencies and, by default, an embedded protoc).

Note: The single most common wasted hour in Tez onboarding is a JDK or protoc version mismatch. Do Step 2 and Step 3 carefully; they are the whole ballgame.

Step-by-Step Tasks

Step 1: Clone the Repository

git clone https://github.com/apache/tez.git
cd tez

The GitHub repository is a mirror of the canonical Apache GitBox repo (scm:git:https://gitbox.apache.org/repos/asf/tez.git — you can confirm the scm.url property in the root pom.xml). Historically Tez used a JIRA + patch workflow; the project has been migrating toward GitHub pull requests, but the JIRA issue is still the system of record — every change needs a TEZ-xxxx ticket. For now you only need a buildable local checkout.

git remote -v
git branch -r | grep -v HEAD | sort | head

You will see origin/master (the development trunk) plus release branches like origin/branch-0.10 and origin/branch-0.9. Use master for contributor work unless you are reproducing an issue specific to a release branch; fixes generally land on master and are backported.

Step 2: Confirm the Required JDK

Tez pins its compiler level in the root pom.xml. Read it — do not assume:

grep -n "maven.compiler.release\|maven.compiler.source\|maven.compiler.target" pom.xml

On current master this prints <maven.compiler.release>21</maven.compiler.release>, and the project README.md lists JDK 21+ and Maven 3.9.14 or later as hard requirements. Older release branches targeted Java 8/11 — which is exactly why you verify per-branch instead of trusting your memory.

Point JAVA_HOME at a matching JDK:

# macOS:
export JAVA_HOME=$(/usr/libexec/java_home -v 21)
export PATH="$JAVA_HOME/bin:$PATH"
java -version   # must report 21.x

# Linux (adjust path to your distro's JDK):
export JAVA_HOME=/usr/lib/jvm/java-21-openjdk
export PATH="$JAVA_HOME/bin:$PATH"

Step 3: Handle Protocol Buffers (the pinned protoc)

Tez serializes its DAG plan, events, and history with Protocol Buffers, so the build must run protoc to generate Java sources. The version is pinned — a mismatched protoc produces sources that fail to compile. Read the pin:

grep -n "protobuf.version\|protoc-jar-maven-plugin.version\|protoc.path" pom.xml

On master this is protobuf.version = 3.25.5. The good news: Tez uses the com.github.os72:protoc-jar-maven-plugin (you can see it wired into tez-api/pom.xml under generate-sources with <protocArtifact>com.google.protobuf:protoc:${protobuf.version}</protocArtifact>), which downloads an embedded protoc matching protobuf.version. On most machines — including Apple Silicon, for which Maven Central publishes an osx-aarch_64 protoc 3.25.5 artifact — you do not need to install protoc at all.

If the embedded binary cannot run on your platform (older CPUs, restricted networks, PowerPC), the README.md documents two escape hatches. Install a matching protoc and point the build at it:

# macOS (Homebrew installs the current major line; verify it prints 3.25.x, else pin it):
brew install protobuf
protoc --version                       # want: libprotoc 3.25.5

# Debian/Ubuntu:
sudo apt-get install -y protobuf-compiler
protoc --version

# Then tell the build to use your binary instead of the embedded one:
export PROTOC_PATH=$(which protoc)     # the plugin reads ${env.PROTOC_PATH}
# ...or per-invocation:
mvn install -DskipTests -Dprotoc.path=$(which protoc)

Warning: If your system protoc is a different version than protobuf.version, do not point the build at it — you will get subtle codegen mismatches. Either install exactly 3.25.5 or let the embedded protoc do its job. You can override the version for a one-off build with -Dprotobuf.version=3.25.5 per the README.

Step 4: Inspect the Module List

Before you build, know what you are building. The reactor modules are declared in the root pom:

grep -n "<module>" pom.xml

You will see (order matters — it is dependency order): hadoop-shim, tez-api, tez-build-tools, tez-common, tez-runtime-library, tez-runtime-internals, tez-mapreduce, tez-examples, tez-tests, tez-dag, tez-ext-service-tests, tez-ui, tez-plugins, tez-tools, hadoop-shim-impls, tez-dist, docs. Note two heavyweights you will want to skip during fast inner loops: tez-ui (a Node/Ember frontend) and docs (a Maven site).

Also read the Hadoop version you are building against:

grep -n "<hadoop.version>" pom.xml    # master: 3.4.2

Step 5: Full Build, Tests Skipped

Your "does it build?" command:

mvn clean install -DskipTests -Pnoui -q

Expected duration: 10–25 minutes cold (dependency download dominates), then a few minutes warm. Flag by flag:

On success the reactor summary ends with:

[INFO] Reactor Summary for tez ...:
[INFO] tez-api ............................................ SUCCESS
[INFO] tez-common ......................................... SUCCESS
[INFO] tez-runtime-library ................................ SUCCESS
[INFO] tez-runtime-internals .............................. SUCCESS
[INFO] tez-mapreduce ...................................... SUCCESS
[INFO] tez-examples ....................................... SUCCESS
[INFO] tez-dag ............................................ SUCCESS
[INFO] ...
[INFO] BUILD SUCCESS

Step 6: Verify Build Artifacts

Confirm the JARs exist. Do not hard-code the version — glob it:

find . -path "*/target/*.jar" -name "tez-api-*.jar"  | grep -v sources
find . -path "*/target/*.jar" -name "tez-dag-*.jar"  | grep -v sources

Read the project version so you know what the glob resolved to (you will reuse this exact string in Lab 1.4):

mvn help:evaluate -Dexpression=project.version -q -DforceStdout
# master currently prints: 1.0.0-SNAPSHOT

Step 7: Assemble the Distribution Tarballs

The runnable distribution is produced by the tez-dist module, not by a plain install. Read tez-dist/pom.xml and you will see it drives the maven-assembly-plugin with two descriptors (tez-dist.xml and tez-dist-minimal.xml), each emitting a dir and a tar.gz, with finalName tez-${project.version} and tez-${project.version}-minimal:

mvn clean package -DskipTests -Pnoui -q
ls -1 tez-dist/target/*.tar.gz
# tez-dist/target/tez-<version>.tar.gz          (full)
# tez-dist/target/tez-<version>-minimal.tar.gz  (minimal — Hadoop libs excluded)

The full tarball bundles Tez plus its runtime dependencies under lib/; the minimal tarball deliberately excludes the Hadoop jars (see the <useTransitiveDependencies>false and the hadoop-common/hadoop-hdfs/hadoop-yarn-* excludes in tez-dist-minimal.xml) for clusters that already provide Hadoop on the classpath. Both assemblies exclude tez-ui and tez-aux-services. Build against a specific Hadoop line by overriding the property, exactly as the README shows:

mvn package -DskipTests -Pnoui -Dhadoop.version=3.4.2 -q

Step 8: Incremental Single-Module Builds

You will run this shape hundreds of times. -pl selects a module; -am ("also make") builds its upstream dependencies first:

# Rebuild tez-dag and everything it depends on:
mvn install -DskipTests -pl tez-dag -am -q

# Rebuild only tez-api (leaf-ish; nothing upstream to make):
mvn install -DskipTests -pl tez-api -q

# Rebuild the runtime library plus deps (shuffle/sort code lives here):
mvn install -DskipTests -pl tez-runtime-library -am -q

Tip: -o (offline) skips remote repository checks once your ~/.m2 cache is warm — a real speedup on the inner loop. -am matters: without it, Maven assumes your upstream changes are already installed and you will debug a stale artifact.

Step 9: Import into IntelliJ IDEA

IntelliJ understands Maven multi-module reactors natively. Import the build, not the files.

1. File → Open → select the tez/ root (the directory containing the top-level pom.xml).
2. Let IntelliJ import it as a Maven project; it reads every <module> and materializes them.
3. Set the Project SDK to a JDK matching maven.compiler.release (Step 2). A mismatched SDK is the number-one cause of red squiggles in an otherwise-green build.
4. Wait for indexing (2–5 minutes).

The protobuf generated-sources gotcha. Tez's *.proto files compile to Java under each module's target/generated-sources/ (the protoc-jar-maven-plugin writes to ${project.build.directory}/generated-sources/java). If you open the project before a command-line build has generated those sources, IntelliJ will flag classes such as DAGProtos.* as unresolved. Fix: run Step 5 on the command line first, then in IntelliJ right-click the project → Maven → Generate Sources and Update Folders, and confirm target/generated-sources/java is marked as a Sources Root (blue folder). Never edit generated *.java under target/ — edit the .proto and regenerate.

Verify the import:

• Open tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java; Ctrl/Cmd+Click a class reference — it should navigate.
• Find Class (Cmd+O / Ctrl+N) → TestVertexImpl should resolve into tez-dag's test tree.

Deliverables

• A successful mvn clean install -DskipTests -Pnoui run (terminal output showing BUILD SUCCESS).
• The exact JDK major version and protobuf.version your branch requires, obtained via grep.
• The project version string from mvn help:evaluate (you will reuse it in Lab 1.4).
• Both tez-dist tarballs produced, and you can state the difference between full and minimal.
• A successful single-module build: mvn install -DskipTests -pl tez-dag -am.
• A working IntelliJ import that resolves a generated protobuf class (e.g. DAGProtos).

Troubleshooting

Stretch Goals

1. Map the dependency graph of tez-dag. Which upstream modules does it pull?

   mvn dependency:tree -pl tez-dag -Dincludes=org.apache.tez | head -40
   

   Confirm tez-api, tez-common, and tez-runtime-library appear.

2. Prove the incremental build works. Run mvn install -DskipTests -pl tez-api -o twice; the second run should reuse compiled output. Then touch one .java under tez-api/src/main/java and confirm the next run recompiles it.

3. Run the quality gates a reviewer will run. Before ever uploading a patch:

   mvn apache-rat:check   -pl tez-dag
   mvn checkstyle:check   -pl tez-dag
   mvn compile spotbugs:spotbugs -pl tez-dag   # spotbugs 4.9.3 per the pom
   

4. Visualize a state machine. Tez ships a visualize profile that renders its state machines to GraphViz — a genuinely useful way to prepare for Level 4:

   mvn compile -Pvisualize -DskipTests -pl tez-dag
   

Validation / Self-check

You are done when you can answer these without notes:

1. What exact JDK major version and protobuf.version does your branch require, and which command in the root pom.xml did you read them from?
2. Why does the build need to run protoc at all, and what are the two ways to control which protoc binary is used?
3. What does -Pnoui do, and why is it the right default for a fast contributor build?
4. What is the difference between the full and minimal tez-dist tarballs, and where does each one get assembled?
5. What does -pl tez-dag -am mean, and what breaks if you drop the -am?
6. Which three quality gates (name the Maven goals) will a reviewer expect to pass, and where does the checkstyle configuration physically live?
7. Why might IntelliJ show DAGProtos as unresolved right after import, and how do you fix it without editing anything under target/?

Where to go next: the Level 1 index frames how these four labs fit together, and Lab 1.2 — Run Unit and Integration Tests builds directly on this clean build. For the architecture behind what you just compiled, read the DAG model deep-dive; everything you build toward in Level 2 assumes you can produce a clean build on demand.