Failure Handling

A Tez DAG dies for a hundred reasons — a corrupt input split, a flaky NodeManager, an OOM in the user processor, a token expiry, an RM blip, the AM itself crashing. Tez does not treat these uniformly. It runs a layered escalation model: a single attempt fails cheaply and is retried; a task fails only after it burns a retry budget; a vertex fails only when its failed-task fraction crosses a threshold; a DAG fails when a vertex fails or the client kills it; and the AM persists enough state to survive its own death and resume. Every layer has its own termination-cause enum, its own config keys, and its own state-machine transition that decides "counted" vs "not counted."

After this chapter you can:

• Read a TaskAttemptTerminationCause off an AM log and predict whether it burns the task's retry budget or is absorbed silently.
• Trace a fetch failure from a consumer's InputReadErrorEvent all the way to the producer being re-run, and name the four thresholds that gate it.
• Explain why a blacklisted node's running containers are killed, and when blacklisting is ignored to keep the cluster schedulable.
• Say exactly what AM attempt 2 replays from the recovery log, and which in-flight DAGs are declared non-recoverable.
• Follow a diagnostic string from an attempt up to DAGStatus.getDiagnostics() on the client.

All paths below are relative to a current-master checkout (/Users/s0x/src/oss-repos/tez). Run each grep yourself — code moves between branches, so anchor on symbols, never line numbers.

The escalation ladder

 TaskAttempt  ── FAILED ──► TaskImpl.failedAttempts++
     │                          │
     │  KILLED (not counted)    │  failedAttempts >= maxFailedAttempts  (default 4)
     │                          ▼    OR TaskFailureType.FATAL
     └───────────────────►  Task FAILED
                                │
                                ▼  failedTaskCount*100 > maxFailuresPercent*numTasks  (default 0.0)
                            Vertex FAILED  (VertexTerminationCause.OWN_TASK_FAILURE)
                                │
                                ▼  DAGEventVertexCompleted(FAILED)
                            DAG FAILED  (DAGTerminationCause.VERTEX_FAILURE)
                                │
                                ▼  diagnostics accumulate at every rung
                            DAGStatus.getDiagnostics()  ──► client

The whole ladder is data-driven by termination-cause enums and a handful of config keys. We walk it bottom-up. For the state machines that host these transitions, keep ./task-attempt-lifecycle.md, ./task-lifecycle.md, and ./state-machines.md open in the next tab; for how the events physically move between the AM and the running task, see ./event-routing.md.

TaskAttemptTerminationCause — the root debugging signal

rg -n "enum TaskAttemptTerminationCause" \
  tez-common/src/main/java/org/apache/tez/dag/records/TaskAttemptTerminationCause.java

Every non-SUCCEEDED attempt carries one of these. This is the single most important field in an AM log when a job misbehaves. Quoted in full from org.apache.tez.dag.records.TaskAttemptTerminationCause (comments trimmed):

public enum TaskAttemptTerminationCause {
  UNKNOWN_ERROR,                        // gap in error propagation
  TERMINATED_BY_CLIENT,                 // killed by client command
  TERMINATED_AT_SHUTDOWN,               // killed due to execution shutdown
  TERMINATED_AT_RECOVERY,               // could not recover a running attempt
  INTERNAL_PREEMPTION,                  // Tez preempted for higher-pri work
  EXTERNAL_PREEMPTION,                  // cluster preempted the container
  TERMINATED_INEFFECTIVE_SPECULATION,   // speculative copy lost the race
  TERMINATED_EFFECTIVE_SPECULATION,     // original lost to the speculative copy
  TERMINATED_ORPHANED,                  // attempt no longer needed by the task
  APPLICATION_ERROR,                    // user code threw
  FRAMEWORK_ERROR,                      // Tez code threw
  INPUT_READ_ERROR,                     // failed reading inputs
  OUTPUT_WRITE_ERROR,                   // failed writing outputs
  OUTPUT_LOST,                          // attempt's output reported lost
  NO_PROGRESS,                          // no progress being made
  TASK_HEARTBEAT_ERROR,                 // AM lost the umbilical to the task
  CONTAINER_LAUNCH_FAILED,              // NM rejected the launch
  CONTAINER_EXITED,                     // container exited, cause unpropagated
  CONTAINER_STOPPED,                    // container stopped/released by Tez
  NODE_FAILED,                          // node for the container failed
  NODE_DISK_ERROR,                      // disk failed on the node
  COMMUNICATION_ERROR,                  // ~ launch failure
  SERVICE_BUSY,                         // service rejected the task
  INTERRUPTED_BY_SYSTEM,                // e.g. pre-emption
  INTERRUPTED_BY_USER,
}

Counted vs not counted — it's the FAILED/KILLED split, not the enum

There is a common misconception that specific causes are "not counted." The truth is more mechanical: an attempt that ends in FAILED increments the task's failedAttempts; an attempt that ends in KILLED does not. The termination cause only decides which terminal helper the attempt uses.

rg -n "KILLED_HELPER|FAILED_HELPER|NodeFailedBeforeRunningTransition" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java

In TaskAttemptImpl, node loss routes through KILLED_HELPER (NodeFailedBeforeRunningTransition extends TerminatedBeforeRunningTransition with super(KILLED_HELPER)), as do client kills, preemption, and losing a speculative race. User/framework exceptions and output/read errors route through FAILED_HELPER. So the practical classification is:

Warning: OUTPUT_LOST does count against the source task's retry budget — it drives the retroactive-failure path (below) which increments failedAttempts. NODE_FAILED does not, because a node dying is infrastructure, not the task's fault, and it terminates as KILLED. Older lore that lumps the two together is wrong; verify with the KILLED/FAILED helper wiring above.

The task retry budget

rg -n "failedAttempts\+\+|maxFailedAttempts|TaskFailureType.NON_FATAL" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java

TaskImpl.AttemptFailedTransition is where the budget is spent. The core decision, from org.apache.tez.dag.app.dag.impl.TaskImpl:

task.failedAttempts++;
task.getVertex().incrementFailedTaskAttemptCount();
// ...
if (task.failedAttempts < task.maxFailedAttempts &&
    castEvent.getTaskFailureType() == TaskFailureType.NON_FATAL) {
  // reschedule a fresh attempt if none is already pending
  if (task.shouldScheduleNewAttempt()) {
    if (!task.addAndScheduleAttempt(getSchedulingCausalTA())) {
      return task.finished(TaskStateInternal.FAILED);
    }
  }
} else {
  // too many failures, or a FATAL failure -> task is dead
  task.eventHandler.handle(new VertexEventTaskCompleted(task.taskId, TaskState.FAILED));
  return task.finished(TaskStateInternal.FAILED);
}

maxFailedAttempts comes from the per-vertex VertexConfig, which reads it in VertexImpl:

rg -n "TEZ_AM_TASK_MAX_FAILED_ATTEMPTS|getMaxFailedTaskAttempts" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

The killed path is separate: TaskImpl.AttemptKilledTransition calls incrementKilledTaskAttemptCount() and reschedules without touching failedAttempts. That is the mechanism behind "killed attempts don't count."

Both keys and defaults are quoted from tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java (TEZ_AM_TASK_MAX_FAILED_ATTEMPTS_DEFAULT = 4, TEZ_AM_TASK_MAX_ATTEMPTS_DEFAULT = 0). The max.failed.attempts Javadoc says verbatim: "This does not count killed attempts."

Node failures and blacklisting

rg -n "qualifiesForBlacklisting|numFailedTAs|blacklistSelf|registerBadNodeAndShouldBlacklist" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/node/AMNodeImpl.java

The AM keeps a per-node failure tally in AMNodeImpl, driven by AMNodeEventTaskAttemptEnded events. The state enum is small — quoted in full from org.apache.tez.dag.app.rm.node.AMNodeState:

public enum AMNodeState {
  ACTIVE,
  FORCED_ACTIVE,
  BLACKLISTED,
  UNHEALTHY,
}

Blacklisting fires when a node accumulates too many failed (not killed) task attempts:

protected boolean qualifiesForBlacklisting() {
  return blacklistingEnabled && (numFailedTAs >= maxTaskFailuresPerNode);
}

protected void blacklistSelf() {
  for (ContainerId c : containers) {
    sendEvent(new AMContainerEventNodeFailed(c, "Node blacklisted"));
  }
  containers.clear();
  sendEvent(new AMSchedulerEventNodeBlacklistUpdate(getNodeId(), true, schedulerId));
}

Warning: blacklistSelf sends AMContainerEventNodeFailed to every container currently on the node and clears the list. Blacklisting a node therefore kills its running attempts — it does not let them drain. Those attempts terminate as KILLED (node-failed helper), so they do not burn task budgets, but any output they had produced is lost.

The ignore-threshold safety valve

Blacklisting is not unconditional. AMNodeImpl.registerBadNodeAndShouldBlacklist delegates to PerSourceNodeTracker, which refuses to blacklist if doing so would remove too much of the cluster:

rg -n "computeIgnoreBlacklisting|blacklistDisablePercent|ignoreBlacklisting" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/node/PerSourceNodeTracker.java

boolean registerBadNodeAndShouldBlacklist(AMNode amNode) {
  if (nodeBlacklistingEnabled) {
    addToBlackList(amNode.getNodeId());
    computeIgnoreBlacklisting();          // flips ignoreBlacklisting when too many are down
    return !ignoreBlacklisting;
  }
  return false;
}

computeIgnoreBlacklisting sets ignoreBlacklisting = true once blacklistMap.size() >= numClusterNodes * blacklistDisablePercent / 100. When ignored, the node stays ACTIVE (or is forced back via FORCED_ACTIVE) and keeps taking work — Tez would rather run on a suspect node than have nowhere to schedule.

Note: The per-node threshold default is 10, not 3. Verify with rg -n "TEZ_AM_MAX_TASK_FAILURES_PER_NODE_DEFAULT" tez-api/.../TezConfiguration.java before you quote a number in a design doc.

stateDiagram-v2
    [*] --> ACTIVE
    ACTIVE --> BLACKLISTED: N_TA_ENDED (failed)\nnumFailedTAs >= maxTaskFailuresPerNode\n&& !ignoreBlacklisting
    ACTIVE --> UNHEALTHY: N_TURNED_UNHEALTHY
    BLACKLISTED --> FORCED_ACTIVE: N_IGNORE_BLACKLISTING_ENABLED
    FORCED_ACTIVE --> ACTIVE: N_IGNORE_BLACKLISTING_DISABLED
    BLACKLISTED --> UNHEALTHY: N_TURNED_UNHEALTHY
    UNHEALTHY --> ACTIVE: node healthy again

Node accounting is per task-scheduler source; see ./scheduler.md for how the scheduler consumes the resulting AMSchedulerEventNodeBlacklistUpdate and stops routing container requests to the node.

Fetch failure: the consumer re-runs the producer

This is the subtlest path in the whole system and the source of the worst production incidents. A downstream (consumer) attempt fetches shuffle output from an upstream (producer) attempt, the fetch fails, and — counter-intuitively — the producer is what gets re-run.

rg -n "OutputReportedFailedTransition|uniquefailedOutputReports|sendInputFailedToConsumers" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java

The path

1. The consumer's fetcher (in tez-runtime-library, e.g. ShuffleManager / ShuffleScheduler) gives up on a source and emits an InputReadErrorEvent (tez-api/.../runtime/api/events/InputReadErrorEvent.java) over the umbilical.

2. In the AM, Edge.sendTezEventToSourceTasks handles INPUT_READ_ERROR_EVENT: it calls the edge manager's routeInputErrorEventToSource to map the failed input index back to the source task, then delivers a TaskAttemptEventOutputFailed (TA_OUTPUT_FAILED) to the blamed producer attempt.

   rg -n "INPUT_READ_ERROR_EVENT|routeInputErrorEventToSource" \
     tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/Edge.java
   

3. The producer attempt's OutputReportedFailedTransition tallies the report and checks four independent guards before it will fail the producer. From org.apache.tez.dag.app.dag.impl.TaskAttemptImpl:

   boolean withinFailureFractionLimits =
       (failureFraction <= maxAllowedOutputFailuresFraction);
   boolean withinOutputFailureLimits =
       (sourceAttempt.uniquefailedOutputReports.size() < maxAllowedOutputFailures);
   // stay alive only if ALL guards are still satisfied
   if (!crossTimeDeadline && withinFailureFractionLimits && withinOutputFailureLimits
       && !(readErrorEvent.isLocalFetch() || readErrorEvent.isDiskErrorAtSource())
       && !tooManyDownstreamHostsBlamedTheSameUpstreamHost) {
     return sourceAttempt.getInternalState();      // absorb the report
   }
   

4. Once any guard trips, the producer attempt is marked FAILED with cause OUTPUT_LOST (TaskAttemptEventOutputFailed.getTerminationCause() returns OUTPUT_LOST). If the producer had already SUCCEEDED, it is un-succeeded via TerminatedAfterSuccessHelper; if still running it goes to FAIL_IN_PROGRESS. Either way it calls sendInputFailedToConsumers, pushing InputFailedEvent to every downstream so they stop trusting the stale output.

5. Back in TaskImpl, TaskRetroactiveFailureTransition (extends AttemptFailedTransition) calls unSucceed(task), then super.transition which does failedAttempts++ and reschedules the producer. It returns SCHEDULED and fires VertexEventTaskReschedule, so the producer's vertex decrements its succeeded/completed counts (VertexImpl.TaskRescheduledTransition) and the whole downstream chain reschedules to consume the regenerated output.

The four guards and their config

rg -n "TEZ_TASK_MAX_ALLOWED_OUTPUT_FAILURES|MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR|DOWNSTREAM_HOST_FAILURES" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

Additionally, isLocalFetch or isDiskErrorAtSource on the read-error event short-circuits all heuristics and fails the producer at once — a local disk error at the source is not going to fix itself.

Warning: This is the cascading-rerun engine. A single bad disk on one NM can generate read errors from many consumers; once the count/fraction/host guards trip, the producer re-runs, but if it re-runs on the same bad node the cascade repeats and can eat the task's retry budget. The host-spread guard and node blacklisting (above) are the brakes — tune them together, not in isolation. See ./shuffle-sort.md for the fetch side.

sequenceDiagram
    participant C as Consumer TA (fetcher)
    participant Edge as Edge (AM)
    participant P as Producer TA
    participant PT as Producer TaskImpl
    participant V as Producer VertexImpl
    C->>Edge: InputReadErrorEvent (umbilical)
    Edge->>P: TA_OUTPUT_FAILED (routeInputErrorEventToSource)
    Note over P: OutputReportedFailedTransition<br/>tally + 4 guards
    alt within all limits
        P-->>Edge: absorb (no state change)
    else a guard trips
        P->>P: FAILED, cause = OUTPUT_LOST
        P->>C: InputFailedEvent (sendInputFailedToConsumers)
        P->>PT: TA_FAILED (causal = output failed)
        Note over PT: TaskRetroactiveFailureTransition<br/>failedAttempts++, unSucceed, reschedule
        PT->>V: VertexEventTaskReschedule
    end

Vertex-level termination

rg -n "OWN_TASK_FAILURE|maxFailuresPercent|vertexFailuresBelowThreshold" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

A single failed task does not automatically fail its vertex — the vertex tolerates a configurable fraction. From org.apache.tez.dag.app.dag.impl.VertexImpl:

if (vertex.failedTaskCount * 100 > vertex.maxFailuresPercent * vertex.numTasks) {
  // exceeded tolerated failures -> fail the vertex
  vertex.tryEnactKill(VertexTerminationCause.OWN_TASK_FAILURE,
      TaskTerminationCause.OTHER_TASK_FAILURE);
}

maxFailuresPercent reads tez.vertex.failures.maxpercent, whose default is 0.0f (TEZ_VERTEX_FAILURES_MAXPERCENT_DEFAULT) — so out of the box a vertex fails on its first dead task. The VertexTerminationCause enum, quoted in full from org.apache.tez.dag.app.dag.VertexTerminationCause with its target VertexState:

public enum VertexTerminationCause {
  DAG_TERMINATED(VertexState.KILLED),          // DAG was killed
  OTHER_VERTEX_FAILURE(VertexState.KILLED),    // a sibling vertex failed the DAG
  ROOT_INPUT_INIT_FAILURE(VertexState.FAILED), // root Input init threw
  AM_USERCODE_FAILURE(VertexState.FAILED),     // VertexManager/EdgeManager/InputInitializer threw
  OWN_TASK_FAILURE(VertexState.FAILED),        // one of its tasks failed
  COMMIT_FAILURE(VertexState.FAILED),
  VERTEX_RERUN_AFTER_COMMIT(VertexState.FAILED),// output already committed, cannot rerun
  VERTEX_RERUN_IN_COMMITTING(VertexState.FAILED),
  INVALID_NUM_OF_TASKS(VertexState.FAILED),
  INIT_FAILURE(VertexState.FAILED),
  INTERNAL_ERROR(VertexState.ERROR),
  RECOVERY_ERROR(VertexState.FAILED),          // error writing recovery log
  COUNTER_LIMITS_EXCEEDED(VertexState.FAILED);
}

Note the two VERTEX_RERUN_* causes: once a vertex's output has been committed (as a shared vertex-group output, or during commit), it cannot be safely re-run — an OUTPUT_LOST that would normally reschedule turns into an outright vertex failure. This is why commit-mode (tez.am.commit-all-outputs-on-dag-success, default true) interacts with failure handling: committing early trades recoverability for earlier visibility. See ./vertex-lifecycle.md.

DAG-level termination

rg -n "VERTEX_FAILURE|enactKill|vertexFailed|checkVerticesForCompletion" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java

When a vertex reaches FAILED, it sends DAGEventVertexCompleted(FAILED). DAGImpl.VertexCompletedTransition reacts:

else if (vertexEvent.getVertexState() == VertexState.FAILED) {
  job.enactKill(DAGTerminationCause.VERTEX_FAILURE,
      VertexTerminationCause.OTHER_VERTEX_FAILURE);
  job.cancelCommits();
  job.vertexFailed(vertex);      // addDiagnostic("Vertex failed, ... diagnostics=" + vertex.getDiagnostics())
  forceTransitionToKillWait = true;
}

enactKill propagates DAG_TERMINATED to every other vertex, so a fail-fast DAG unwinds cleanly. The DAGTerminationCause enum, quoted in full from org.apache.tez.dag.app.dag.DAGTerminationCause:

public enum DAGTerminationCause {
  DAG_KILL(DAGState.KILLED),                 // client tryKillDAG()
  SERVICE_PLUGIN_ERROR(DAGState.FAILED),     // a service plugin errored
  VERTEX_FAILURE(DAGState.FAILED),           // a vertex failed
  ZERO_VERTICES(DAGState.FAILED),
  INIT_FAILURE(DAGState.FAILED),
  COMMIT_FAILURE(DAGState.FAILED),
  VERTEX_RERUN_AFTER_COMMIT(DAGState.FAILED),
  VERTEX_RERUN_IN_COMMITTING(DAGState.FAILED),
  RECOVERY_FAILURE(DAGState.FAILED),         // could not write/replay recovery
  INTERNAL_ERROR(DAGState.ERROR);
}

AM crash recovery

The AM is a single JVM; if it dies, YARN gives it another container and Tez must reconstruct the DAG from an append-only log on HDFS.

rg -n "maxUnflushedEvents|summaryStream|handleSummaryEvent|writeToRecoveryImmediately" \
  tez-dag/src/main/java/org/apache/tez/dag/history/recovery/RecoveryService.java

Summary events vs full events

RecoveryService writes two streams per DAG. The full event log (<dagId>.recovery) mirrors every state transition; the summary log (summary) holds only SummaryEvents — the coarse milestones that let a restart decide, quickly and safely, whether a DAG can be recovered at all.

rg -n "interface SummaryEvent|toSummaryProtoStream|writeToRecoveryImmediately" \
  tez-dag/src/main/java/org/apache/tez/dag/history/SummaryEvent.java

public interface SummaryEvent {
  boolean writeToRecoveryImmediately();               // fsync now, don't buffer
  void toSummaryProtoStream(OutputStream outputStream) throws IOException;
  // ...
}

Implementers include DAGSubmittedEvent, DAGFinishedEvent, DAGCommitStartedEvent, VertexCommitStartedEvent, VertexGroupCommitStartedEvent, VertexGroupCommitFinishedEvent, VertexFinishedEvent, and DAGKillRequestEvent. Summary events whose writeToRecoveryImmediately() returns true are flushed synchronously — you never want to lose the record that a commit started. Everything else is batched: the writer flushes when it accumulates maxUnflushedEvents events or the periodic timer fires.

Where the log lives

Paths come from TezCommonUtils (getSummaryRecoveryPath, getDAGRecoveryPath) and TezConstants:

<staging>/<appId>/recovery/<appAttemptId>/
    summary                    # DAG_RECOVERY_SUMMARY_FILE_SUFFIX
    <dagId>.recovery           # DAG_RECOVERY_RECOVER_FILE_SUFFIX

recovery is TezConstants.DAG_RECOVERY_DATA_DIR_NAME; the recovery dir is resolved via appContext.getCurrentRecoveryDir() and is per-app-attempt, so attempt 2 reads attempt 1's directory.

What replay reconstructs, and what it declares dead

rg -n "nonRecoverable|reason|skipAllOtherEvents" \
  tez-dag/src/main/java/org/apache/tez/dag/app/RecoveryParser.java

RecoveryParser reads the summary first to compute nonRecoverable + a human reason, then replays the full log. A DAG is declared non-recoverable when a commit was in flight without a clean closure — replaying it could double-commit or resurrect an output that was already made visible. Real reasons from the parser:

• "DAG Commit was in progress, not recoverable"
• "Vertex Commit was in progress, not recoverable"
• "Vertex Group Commit was in progress, not recoverable"
• "Vertex has been committed, but its full recovery events are not seen ..."

DAGAppMaster acts on the verdict: a non-recoverable DAG is driven straight to FAILED with addDiagnostic("DAG <id> can not be recovered due to <reason>") and a DAGRecoveredEvent(... DAGState.FAILED ...). A completed DAG is restored to its terminal state; anything else resumes from the last consistent snapshot.

rg -n "isNonRecoverable|can not be recovered|DAGRecoveredEvent" \
  tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java

Note: tez.am.max.app.attempts defaults to 2, but Tez requests min(tez.am.max.app.attempts, yarn.resourcemanager.am.max-attempts) from YARN. If YARN caps AM attempts at 1, recovery never runs no matter what the Tez key says. Verify with rg -n "TEZ_AM_MAX_APP_ATTEMPTS" tez-api/.../TezClientUtils.java.

For the AM's own startup and recovery entry points, cross-read ./dag-app-master.md.

Where diagnostics surface

Every rung of the ladder appends a human string, and they chain upward so the client sees the whole story.

rg -n "addDiagnostic|getDiagnostics" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java

• TaskAttemptImpl.addDiagnosticInfo records the raw cause + message on the attempt (e.g. the full OutputReportedFailedTransition message with all four guard values).
• TaskImpl.AttemptFailedTransition folds the attempt's diagnostics into the task: "TaskAttempt N failed, info=" + attempt.getDiagnostics().
• VertexImpl.addDiagnostic and DAGImpl.vertexFailed fold the vertex's diagnostics up: "Vertex failed, vertexName=..., diagnostics=...".
• DAGImpl exposes them through getDiagnostics(), which the RPC layer copies into DAGStatus; the client reads DAGStatus.getDiagnostics() (proxy.getDiagnosticsList()).

So a user staring at a failed job on the client sees the terminal TaskAttemptTerminationCause and its message, wrapped by the task, vertex, and DAG context — the entire escalation ladder in one string. The counters and event-timeline side of observability lives in ./counters-diagnostics.md.

Reading exercise

Work these against the checkout, in order:

# 1. The full attempt-cause enum and its comments
rg -n "enum TaskAttemptTerminationCause" -A 40 \
  tez-common/src/main/java/org/apache/tez/dag/records/TaskAttemptTerminationCause.java

# 2. Budget enforcement + the NON_FATAL guard
rg -n "failedAttempts\+\+|maxFailedAttempts|TaskFailureType" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java

# 3. The retroactive-failure path that re-runs a producer
rg -n "TaskRetroactiveFailureTransition|VertexEventTaskReschedule|unSucceed" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java

# 4. The four output-failure guards
rg -n "withinOutputFailureLimits|crossTimeDeadline|maxAllowedOutputFailures" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java

# 5. Blacklist decision + ignore threshold
rg -n "qualifiesForBlacklisting|computeIgnoreBlacklisting|blacklistSelf" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/node/AMNodeImpl.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/node/PerSourceNodeTracker.java

# 6. Non-recoverable DAG reasons
rg -n "nonRecoverable|reason =" \
  tez-dag/src/main/java/org/apache/tez/dag/app/RecoveryParser.java

Then answer: in step 3, why does TaskRetroactiveFailureTransition have to call unSucceed(task) before super.transition? (Hint: the task was already SUCCEEDED; AttemptFailedTransition assumes an uncompleted task, so the redundancy bookkeeping must be reset first.)

Common bugs and symptoms

Validation: prove you understand this

1. A TaskAttempt ends with cause NODE_FAILED and another with cause OUTPUT_LOST. Which one increments TaskImpl.failedAttempts, and why? Cite the helper wiring that makes one KILLED and the other FAILED.
2. In two sentences, explain how an InputReadErrorEvent from a consumer ends up re-running the producer task, not just the consumer. Name the AM class that routes the error back to the source.
3. List the four independent guards in OutputReportedFailedTransition and the config key + default behind each. Which one is not per-task but per-host?
4. tez.am.maxtaskfailures.per.node = 10 on a 6-node cluster with ignore-threshold-node-percent = 33. How many nodes can be blacklisted before blacklisting is ignored, and what happens to the 3rd node that qualifies? Show the arithmetic (6 * 33 / 100).
5. Give the exact HDFS path components for the summary and full recovery logs of DAG dag_1 under app-attempt 2, and say which events land in the summary log versus the full log.
6. Your DAG failed mid-commit and AM attempt 2 marks it non-recoverable. Quote the RecoveryParser reason string you'd expect, and explain why re-running would be unsafe rather than merely wasteful.
7. Trace one diagnostic string from TaskAttemptImpl.addDiagnosticInfo to DAGStatus.getDiagnostics() on the client, naming the method at each rung.