Stage 4 — State Machine Transitions

What this stage teaches

Stage 4 is the first stage that requires you to understand the Tez AppMaster, not just navigate it. You learn:

The StateMachineFactory DSL used in Hadoop / Tez to declare finite state machines. The two canonical instances are VertexImpl.stateMachineFactory and TaskImpl.stateMachineFactory.
The InvalidStateTransitonException (note the historical typo — "Transiton", not "Transition" — preserved for API compatibility) that the state machine throws when an event arrives in a state with no registered transition.
How to add a transition with the right guard, without widening the surface area of the state machine accidentally.
The hard rule: never widen a transition without a dev@ design discussion. Adding a transition from RUNNING to KILLED on a new event class is a semantic change that may cascade to ATS, the client, and the speculator.
The TestVertexImpl and TestTaskImpl patterns for asserting that an event in a state produces an expected next state.

Patches are typically 30–250 lines: a transition table entry, a small guard helper, a fired event, and a deterministic regression test.

Reading order before you touch any code

tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java — read the static stateMachineFactory block end to end. It is several hundred lines of .addTransition(...) calls. Diagram it on paper.
tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java — same exercise for tasks.
tez-common/src/main/java/org/apache/tez/state/StateMachineTez.java — the wrapper Tez puts around the Hadoop state machine.
The deep dives state-machines and vertex-lifecycle. Do not skip these.

Then, and only then, file a JIRA.

JIRA filter to find candidates

The most fruitful filter:

project = TEZ
  AND resolution = Unresolved
  AND (text ~ "InvalidStateTransitonException" OR text ~ "Invalid event"
       OR text ~ "missing transition" OR description ~ "stateMachineFactory")
ORDER BY updated DESC

A second filter for postmortem-style tickets:

project = TEZ AND status = Open AND component in ("tez-dag")
  AND priority in (Major, Critical) AND text ~ "VertexState\\|TaskState"

Most real Stage 4 work comes from operator reports of an AM that crashed with InvalidStateTransitonException: Invalid event X on Y in state Z. That stack trace is the smoking gun: state Z received event X and had no registered handler. The fix is one of:

Add the transition with a guard (most common).
Suppress the event in that state because it is a benign late delivery (use addTransition(state, state, event) — a self-loop).
Fix the sender not to emit the event in that state (sometimes the bug is upstream).

Choosing wrong is the most common Stage 4 mistake. Pick option 3 only if you can prove the event should never have been emitted.

Walked example — missing `V_INIT` transition in `VertexState.NEW`

Symptom: an operator reports a recurring AM crash:

InvalidStateTransitonException: Invalid event: V_INIT at NEW
  at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(...)
  at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:NNNN)

V_INIT arriving while the vertex is in NEW is suspicious — NEW is supposed to accept V_INIT. Investigation reveals the transition is registered for the common path, but a recently-added early-error path emits V_INIT from a different thread before the main scheduler does, and the second V_INIT arrives while the vertex is back in NEW after a re-init.

Step 1 — Read the existing transitions

cd ~/tez-src
grep -n "addTransition(VertexState.NEW" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | head -20

You will see something like (simplified):

.addTransition(VertexState.NEW, VertexState.INITED,
    VertexEventType.V_INIT, new InitTransition())
.addTransition(VertexState.NEW, VertexState.FAILED,
    VertexEventType.V_TERMINATE, new TerminateNewVertexTransition())

V_INIT on NEW is registered. So the crash means the vertex was not in NEW when the second V_INIT arrived — it was somewhere else, perhaps INITED. Re-grep:

grep -n "addTransition(VertexState.INITED" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java | grep "V_INIT"

No hit. That is the bug: V_INIT arriving in INITED is unhandled.

Step 2 — Decide: add, ignore, or fix upstream

V_INIT in INITED is a duplicate event. It is benign (the vertex is already initialised; the second message is redundant). The correct fix is to ignore the duplicate — a self-loop. This is the safe, narrow change.

We are not widening behaviour. We are saying: "in INITED, a redundant V_INIT is a no-op, not a crash."

Step 3 — The diff

--- a/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java
+++ b/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java
@@
        .addTransition(VertexState.INITED, VertexState.RUNNING,
            VertexEventType.V_START, new StartTransition())
+
+       // A duplicate V_INIT can arrive when an early error path fires V_INIT
+       // concurrently with the scheduler. The vertex is already initialised;
+       // ignore the duplicate rather than crashing the AM. See TEZ-XXXX.
+       .addTransition(VertexState.INITED, VertexState.INITED,
+           VertexEventType.V_INIT, VERTEX_STATE_CHANGED_CALLBACK_NOOP)

Where VERTEX_STATE_CHANGED_CALLBACK_NOOP is either a constant MultipleArcTransition that does nothing, or, more idiomatically, a small inner class:

private static class IgnoreEventTransition
    implements SingleArcTransition<VertexImpl, VertexEvent> {
  @Override
  public void transition(VertexImpl vertex, VertexEvent event) {
    LOG.debug("Ignoring duplicate {} on vertex {} in state {}",
        event.getType(), vertex.getVertexId(), vertex.getState());
  }
}

Two rules in this diff:

The transition has a comment with the JIRA ID explaining why the self-loop exists. State-machine entries without comments are hard to remove safely two years later.
The transition logs at DEBUG, not INFO. If the duplicate event is actually a symptom of a larger bug upstream, the debug log is what tells the operator.

Step 4 — Regression test in `TestVertexImpl`

@Test(timeout = 10000)
public void testDuplicateVInitInInitedIsNoOp() throws Exception {
  initAllVertices(VertexState.INITED);                  // existing helper
  VertexImpl v = vertices.get("vertex1");
  assertEquals(VertexState.INITED, v.getState());

  // Fire a second V_INIT — must not throw, must not change state
  v.handle(new VertexEvent(v.getVertexId(), VertexEventType.V_INIT));
  dispatcher.await();

  assertEquals("duplicate V_INIT should leave INITED unchanged",
      VertexState.INITED, v.getState());
}

The test pattern:

Use the existing initAllVertices(VertexState.INITED) helper. Do not invent your own bootstrap.
Always call dispatcher.await() after v.handle(...). TestVertexImpl uses DrainDispatcher, which is the only way to make event-driven tests deterministic.
Assert the post state. Never assert on internal counters unless the transition is supposed to change them.

Run it:

cd ~/tez-src
mvn -pl tez-dag test -Dtest=TestVertexImpl#testDuplicateVInitInInitedIsNoOp -q 2>&1 | tail -30

Then run the whole TestVertexImpl suite. A single transition addition has broken a sibling test more than once in Tez history.

Step 5 — dev@ notification

Before you post the patch:

Subject: [DISCUSS] TEZ-XXXX — add INITED -> INITED self-loop for V_INIT

I have a repro for a recurring AM crash where V_INIT arrives twice. The state
machine currently has no INITED+V_INIT entry. Proposed fix: self-loop with a
debug log. Sender side (early-error path) is left unchanged on the grounds
that defensive handling in the state machine is cheaper than chasing every
sender. Would appreciate a sanity check before I post the patch.

If a committer replies "actually the sender is the bug, fix that instead," you revise your approach. If silence for 48 hours, post the patch.

The "never widen without dev@" rule

What counts as widening:

Adding a transition from a non-terminal state to a terminal state on a new event. Example: RUNNING -> KILLED on V_USER_REQUEST_FORCE_KILL.
Adding a transition that changes a previously-rejected event into an accepted one with side effects (counters updated, downstream events emitted).
Removing a transition.

What is not widening:

Adding a self-loop that ignores a duplicate event (as above).
Adding a transition that converts an InvalidStateTransitonException into a controlled ERROR transition, when the event was clearly a fatal-bug signal.

The dev@ rule exists because state machines are observed externally: the AM emits state-changed events to ATS, the client poll loop watches them, the speculator reads them. Adding a transition is an API change for those observers, even if no Java type signature changes.

Pitfalls

Don't add transitions to fix symptoms. If you see InvalidStateTransitonException and the cause is "the sender shouldn't have emitted that event," fix the sender. Adding a transition to silence the exception hides the real bug.
Don't forget the regression test. Every transition patch must have a test that fires the event in the state and asserts the result. Tests using DrainDispatcher are the only ones reviewers accept.
Don't use Mockito.spy on VertexImpl. The state machine has private internal state that spies cannot reach reliably. Use the production class with the test helpers in TestVertexImpl and MockDAGAppMaster.
Don't change the transition() callback signature. Existing transitions use SingleArcTransition or MultipleArcTransition. Pick the matching one; do not introduce a new interface.
Don't ignore the typo. InvalidStateTransitonException (no second "i") is the canonical name in Hadoop. If you "fix" the typo in Tez code, you break binary compatibility with downstream callers that catch the exception by name.
Don't bundle a transition fix with an unrelated cleanup. Reviewers will ask you to split.

Exit criteria — when you're ready for the next stage

Move to Stage 5 when:

You have shipped one transition fix in VertexImpl or TaskImpl with a passing regression test in the corresponding Test* class.
You can draw the VertexImpl state diagram from memory (8 states, the main transitions, the terminal set).
You have read TaskAttemptImpl.stateMachineFactory in full and recognise the similarities and differences to VertexImpl.
A committer has reviewed your transition patch and accepted the addition without asking for a dev@ design thread — meaning your choice of "ignore vs add vs fix sender" was correct.

Stage 5 takes you out of the AM event loop and into the scheduler.

Apache Tez Open-Source Contributor Curriculum