Lab 4.1: Read the VertexImpl State Machine

Background

VertexImpl.java (tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java) is the most complex class in Apache Tez — roughly 6,000 lines. It holds the complete state machine for vertex execution: initialization, root-input handling, scheduling, task-completion accounting, failure and kill cascades, commit, and AM recovery. Everything the Application Master knows about "what a vertex is doing right now" lives in one Hadoop StateMachineFactory and a family of transition-handler inner classes.

This is a code-reading lab, the Tez analog of reading the OpenSearch Coordinator. You will not change code. You will end with a transition table you built yourself from the source, plus answers that prove you understood the machine — not just the file names. The skill being trained is reading a large state machine structurally (factory → states → events → transitions → handlers) rather than linearly top-to-bottom.

Keep three deep-dives open beside this lab: state-machines.md (how Hadoop's StateMachineFactory works), vertex-lifecycle.md (the narrative walk-through), and event-routing.md (how events reach the vertex). The Level 4 overview has the class table you should keep pinned.

A note on line numbers. This lab never cites line numbers — they drift every release. It cites class names, method names, states, and events, all of which are stable. Every step gives you a grep to find the code on your checkout. Learn the names, not the offsets.

Why This Lab Matters for Contributors

The scariest issues in the Tez tracker are vertex-lifecycle bugs: "DAG hangs with one vertex stuck in RUNNING", "vertex went to SUCCEEDED then back to RUNNING", "kill during initialization leaks containers", "recovery replays a vertex into the wrong state". A maintainer triaging any of these opens VertexImpl, finds the relevant addTransition entry, and reads the handler. If you cannot navigate this file you cannot reproduce, review, or fix those issues — and you cannot review anyone else's patch that touches the state machine, which is where the highest-value review happens.

This lab is also the foundation for Lab 4.2 (the VertexManager hook that StartTransition invokes) and both build/fix labs (4.3, 4.4).

Prerequisites

• A Tez checkout that builds (mvn install -DskipTests succeeds at least once).
• VertexImpl.java open in an editor with "go to symbol" / outline support.
• grep / awk on the command line, run from the repo root.
• You have read state-machines.md and understand the difference between SingleArcTransition (one fixed destination) and MultipleArcTransition (destination computed at runtime).
• A reading log to append to as you go:

mkdir -p ~/tez-notes && : > ~/tez-notes/reading-log-4.1.md

How to Read a Large State Machine Class

Do not read VertexImpl.java from top to bottom. Read it in this order:

1. The StateMachineFactory declaration — the single source of truth for every transition.
2. The two state enums — the internal VertexState (the states the machine moves through) and the event enum VertexEventType (the inputs).
3. The transition matrix — every addTransition(...) call, grouped by source state.
4. The handler classes — the inner classes implementing SingleArcTransition / MultipleArcTransition, whose transition() methods contain the actual logic.
5. Inter-machine events — where a handler calls eventHandler.handle(...) to post an event to another state machine (TaskImpl, DAGImpl, target vertices).

Step-by-Step Tasks

Step 1 — Find the factory and read its opening

grep -n "stateMachineFactory\|StateMachineFactory" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head

The declaration opens like this (quoted from module tez-dag, class VertexImpl):

StateMachineFactory<VertexImpl, VertexState, VertexEventType, VertexEvent>
   stateMachineFactory
 = new StateMachineFactory<VertexImpl, VertexState, VertexEventType, VertexEvent>
          (VertexState.NEW)

      // Transitions from NEW state
      .addTransition
          (VertexState.NEW,
              EnumSet.of(VertexState.NEW, VertexState.INITED,
                  VertexState.INITIALIZING, VertexState.FAILED, VertexState.KILLED),
              VertexEventType.V_INIT,
              new InitTransition())

Read the four type parameters: operand (VertexImpl), state (VertexState), event type (VertexEventType), event (VertexEvent). The single constructor argument, VertexState.NEW, is the initial state of every vertex. The chain of .addTransition(...) calls that follows is the entire machine.

Log it: the factory's four type parameters and the initial state.

Step 2 — Read the state enum (and notice the naming trap)

There are two VertexState enums, and confusing them is a classic beginner mistake. Confirm which one the state machine uses:

grep -n "import.*VertexState;" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

You will find import org.apache.tez.dag.app.dag.VertexState; — the internal enum. Read it:

cat tez-dag/src/main/java/org/apache/tez/dag/app/dag/VertexState.java

It has exactly ten states:

public enum VertexState {
  NEW, INITIALIZING, INITED, RUNNING, SUCCEEDED,
  FAILED, KILLED, ERROR, TERMINATING, COMMITTING,
}

The other one — org.apache.tez.dag.api.event.VertexState — is a public API enum (SUCCEEDED, RUNNING, FAILED, KILLED, PARALLELISM_UPDATED, CONFIGURED, INITIALIZING) used only to notify VertexManager plugins and listeners via VertexStateUpdate. The state machine never transitions through CONFIGURED or PARALLELISM_UPDATED — those are notifications, not machine states. Record both enums in your log and note that the machine uses the internal one.

Step 3 — Count states, events, and transitions

# Total transitions
grep -c "addTransition" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

# Distinct source states
grep "addTransition(VertexState\." \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java \
  | sed -E 's/.*addTransition\(VertexState\.([A-Z_]+).*/\1/' | sort -u

Record your numbers. On current master this is on the order of 60 transitions across the ten states. (Some addTransition calls register a set of ignored events in one call, so "distinct source states" is a cleaner count than raw transitions.)

Step 4 — List the handler classes

grep -n "implements SingleArcTransition\|implements MultipleArcTransition\|static class .*Transition" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

You will find the real handler set. The load-bearing ones for this lab:

There are also five shared constant transitions instantiated once and reused (search for INTERNAL_ERROR_TRANSITION =, ROUTE_EVENT_TRANSITION =, TASK_ATTEMPT_COMPLETED_EVENT_TRANSITION =, SOURCE_TASK_ATTEMPT_COMPLETED_EVENT_TRANSITION =, COMMIT_COMPLETED_TRANSITION =). These map to InternalErrorTransition, RouteEventTransition, TaskAttemptCompletedEventTransition, SourceTaskAttemptCompletedEventTransition, and CommitCompletedTransition respectively.

Note on RECOVERING. There is no RECOVERING state in the internal enum. Recovery is not a separate state — it is the RecoverTransition registered on NEW for the V_RECOVER event, which replays persisted history and lands the vertex directly in INITED, SUCCEEDED, FAILED, KILLED, or ERROR. If you expected recovery states from other schedulers, this is where Tez differs. Note it in your log.

Step 5 — Build the transition matrix for THREE real flows

For each flow, derive every row from the real addTransition entries and read the handler's transition() method. Produce a table Source | Event | Destination(s) | Handler.

Flow A — Happy path (root vertex: init → running → succeeded)

Derived from the NEW, INITIALIZING, INITED, RUNNING, COMMITTING blocks:

Now answer from the code:

• InitTransition.transition() returns VertexState.NEW initially and only calls handleInitEvent(vertex) once all source vertices have inited (numInitedSourceVertices == sourceVertices.size()). A root vertex has no source vertices, so it inits immediately. Confirm: what does it do to target vertices when init succeeds? (It posts a V_INIT VertexEvent to each of them — the init wavefront propagates down the DAG.)
• StartTransition.transition() asserts the state is INITED, then returns vertex.startVertex(). Read startVertex(): the crucial line is vertexManager.onVertexStarted(...) — this is the hook Lab 4.2 dissects. Is it blocking? (It is a synchronous call into user plugin code; a throw drives the vertex to TERMINATING.)
• TaskCompletedTransition — read Flow B below; the same handler covers success and failure.
• The transition to COMMITTING vs SUCCEEDED is decided in checkTasksForCompletion(vertex) (a static VertexState helper). It only fires when completedTaskCount == tasks.size(), then returns COMMITTING if the vertex has outputs to commit, else finishes as SUCCEEDED.

Flow B — A task failure cascading to vertex failure

TaskCompletedTransition is a MultipleArcTransition — its destination is computed. Quote the core of its transition() (module tez-dag, class VertexImpl, inner class TaskCompletedTransition):

} else if (taskEvent.getState() == TaskState.FAILED) {
  taskFailed(vertex, task);
  if (vertex.failedTaskCount * 100 > vertex.maxFailuresPercent * vertex.numTasks) {
    LOG.info("Failing vertex: " + vertex.logIdentifier +
            " because task failed: " + taskEvent.getTaskID());
    vertex.tryEnactKill(VertexTerminationCause.OWN_TASK_FAILURE,
        TaskTerminationCause.OTHER_TASK_FAILURE);
    forceTransitionToKillWait = true;
  }
}
...
VertexState state = VertexImpl.checkTasksForCompletion(vertex);
if(state == VertexState.RUNNING && forceTransitionToKillWait){
  return VertexState.TERMINATING;
}
return state;

Matrix rows:

Answer:

1. Which field counts failures? (failedTaskCount.) What is the threshold? (maxFailuresPercent, compared as failedTaskCount * 100 > maxFailuresPercent * numTasks.)
2. What does tryEnactKill(...) do? (Fans task-kill/terminate events out so the rest of the running tasks are torn down; the vertex sits in TERMINATING until every task is accounted for.)
3. When the vertex finally lands in FAILED, what does it post upward? Find it:

   grep -n "DAG_VERTEX_COMPLETED\|new DAGEvent\|eventHandler.handle" \
     tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head
   

   Then read how DAGImpl reacts:

   grep -n "VERTEX_COMPLETED\|vertexFailed\|VertexCompleted" \
     tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java | head
   

   Does the DAG fail immediately, or continue as far as it can? (It records the vertex as failed and fails the DAG when the failure makes completion impossible — read the handler to state the exact rule.)

Flow C — A kill during INITIALIZING

Derived from the INITIALIZING block's V_TERMINATE entry:

Read the handler. It is deliberately tiny:

private static class TerminateInitingVertexTransition extends TerminateInitedVertexTransition {
  @Override
  public void transition(VertexImpl vertex, VertexEvent event) {
    super.transition(vertex, event);
  }
}

and the parent it delegates to:

private static class TerminateInitedVertexTransition
implements SingleArcTransition<VertexImpl, VertexEvent> {
  @Override
  public void transition(VertexImpl vertex, VertexEvent event) {
    VertexEventTermination vet = (VertexEventTermination) event;
    vertex.trySetTerminationCause(vet.getTerminationCause());
    vertex.addDiagnostic("Vertex received Kill in INITED state.");
    vertex.finished(VertexState.KILLED);
  }
}

Answer: why does killing in INITIALIZING go straight to KILLED (a SingleArc transition) while killing in RUNNING goes to TERMINATING first (VertexKilledTransition)? (A vertex that has not started running has no tasks to tear down, so there is nothing to wait for; a RUNNING vertex must drain in-flight tasks, hence the intermediate TERMINATING state.)

Step 6 — Identify the multi-arc (dynamic-destination) transitions

MultipleArcTransition.transition() returns the destination state. List every handler in VertexImpl that does so, and for one of them quote the return logic. The cleanest example is VertexNoTasksCompletedTransition, whose whole body is:

@Override
public VertexState transition(VertexImpl vertex, VertexEvent event) {
  return VertexImpl.checkTasksForCompletion(vertex);
}

The destination is entirely determined by checkTasksForCompletion, which weighs succeededTaskCount, failedTaskCount, numTasks, terminationCause, and whether outputs need committing. This is why the same event from the same state can land in SUCCEEDED, FAILED, or COMMITTING — the EnumSet in the addTransition call enumerates all legal outcomes, and the handler picks one.

Step 7 — Visualize the machine (the real tool exists)

Tez ships a GraphViz generator wired into the tez-dag POM. Confirm it:

grep -n "VisualizeStateMachine\|tez.dag.state.classes\|visualize" tez-dag/pom.xml

The visualize profile runs org.apache.hadoop.yarn.state.VisualizeStateMachine (from hadoop-yarn-common) over the classes listed in tez.dag.state.classes — DAGImpl, VertexImpl, TaskImpl, TaskAttemptImpl, AMNodeImpl, AMContainerImpl. Generate the graph:

mvn compile -Pvisualize -pl tez-dag
#   -> writes tez-dag/target/Tez.gv
dot -Tpng tez-dag/target/Tez.gv -o /tmp/tez-states.png   # if graphviz 'dot' installed

To render only VertexImpl, override the property:

mvn compile -Pvisualize -pl tez-dag \
  -Dtez.dag.state.classes=org.apache.tez.dag.app.dag.impl.VertexImpl \
  -Dtez.graphviz.title=VertexImpl

Compare the generated diagram to the three flows you built by hand. Every arc in your tables must appear in the graph; every arc in the graph you did not derive is a flow you have not yet read.

Step 8 — Find the under-tested transitions and the @Ignored test

grep -n "@Ignore" \
  tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java

For each @Ignored test: read the comment, note the JIRA it references (if any), and judge whether the underlying issue is fixed and the test could be re-enabled — a genuine, mergeable contributor task. Then pick three handler classes from your matrix and check whether TestVertexImpl exercises each:

grep -n "V_TASK_COMPLETED\|V_TERMINATE\|V_START" \
  tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java | head

Document one transition that looks under-covered — a candidate Test JIRA.

Deliverables

• The factory's four type parameters + initial state, in your log.
• Both VertexState enums recorded, with a note on which the machine uses and why.
• Transition matrices for all three flows (A happy, B failure cascade, C kill-in-INITIALIZING), every row attributed to a real handler class.
• The list of MultipleArcTransition handlers, with one quoted return-logic example.
• A rendered Tez.gv (or VertexImpl-only graph) checked against your matrices.
• The @Ignored test in TestVertexImpl assessed, plus one under-covered transition named.

Troubleshooting

Stretch Goals

1. Compare complexity. Build the same matrix (states, events, transitions) for TaskImpl and TaskAttemptImpl. Rank the three by number of states and transitions and explain why VertexImpl is the largest (it coordinates all its tasks plus reconfiguration and commit).
2. Trace inter-machine events. Find every place VertexImpl posts to another machine:

   grep -n "eventHandler.handle" \
     tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head -30
   

   Classify the targets (TaskImpl, DAGImpl, target/source VertexImpls) and the event types. Cross-reference event-routing.md.
3. The reconfiguration path. StartTransition/startVertex() call maybeSendConfiguredEvent() and interact with vertexToBeReconfiguredByManager. Trace how a VertexManager that changes parallelism (Lab 4.2/4.4) delays the CONFIGURED notification. Why is this one of the most bug-prone areas of the class?

Validation / Self-check

Answer in your own words, citing the class (never a line number) for each:

1. What are the four type parameters of the VertexImpl StateMachineFactory, and what is the initial state?
2. There are two VertexState enums. Which does the state machine use, which is for notifications, and name one state that exists in one but not the other.
3. TaskCompletedTransition is a MultipleArcTransition. Explain how the same event (V_TASK_COMPLETED) from RUNNING can end in RUNNING, SUCCEEDED, or TERMINATING.
4. Trace a task failure that fails the vertex: which counter and threshold decide it, which method tears down the surviving tasks, and which intermediate state the vertex passes through.
5. Why does a kill in INITIALIZING use a SingleArcTransition straight to KILLED, while a kill in RUNNING uses a transition to TERMINATING first?
6. There is no RECOVERING state. How does Tez recover a vertex after an AM restart, and which transition/event implements it?
7. Which real command generates a GraphViz diagram of VertexImpl, and where does it write the output?

When you can reproduce the three flows from memory and explain each dynamic destination as logic in a transition() method — not a fixed arrow — you have completed Lab 4.1. Continue to Lab 4.2: VertexManager Deep Dive.