Counters and Diagnostics

When a Tez DAG misbehaves you get exactly two machine-readable signals out of the Application Master: counters (numeric aggregates rolled up from every task attempt) and diagnostics (free-text cause strings attached at every level of the DAG hierarchy). Everything else — the Tez UI, _history files on HDFS, the YARN Timeline entities, the job-analyzer plugins in tez-tools — is just a different rendering of those two data structures. This chapter is the maintainer's reference for how both are modeled, aggregated, size-limited, persisted, and surfaced. It runs almost entirely inside the AM (see dag-app-master.md) and is fed by the runtime heartbeat path described in event-routing.md.

After this chapter you can:

• Trace a single counter increment from user code in a task JVM up through TaskAttemptImpl → TaskImpl → VertexImpl → DAGImpl to the DAGStatus the client polls.
• Explain why counters have a hard cap, which config keys set it, and what LimitExceededException does to a running DAG.
• Read the real TaskCounter / DAGCounter enums and know which member means what — no guessing SHUFFLE_BYTES_TO_MEM vs SHUFFLE_BYTES_DISK_DIRECT.
• Follow a diagnostic string from a failed attempt up to what the client sees.
• Name the real history-logging services on master and pick the config key that selects one.
• Find the .dot visualization the AM writes and render it with Graphviz.

Every class, enum, and config key below is quoted from the checkout at /Users/s0x/src/oss-repos/tez. Code moves between branches, so each section opens with the grep/rg that locates the symbol rather than a line number — run it yourself before you trust anything here.

The TezCounters model

cd /Users/s0x/src/oss-repos/tez
ls tez-api/src/main/java/org/apache/tez/common/counters/
rg -n "class TezCounters|extends AbstractCounters" \
  tez-api/src/main/java/org/apache/tez/common/counters/TezCounters.java

A TezCounters is a two-level map: groupName → CounterGroup → counterName → TezCounter. The whole thing is generic machinery lifted from Hadoop's org.apache.hadoop.mapreduce.counters package. In tez-api, module org.apache.tez.common.counters, the class hierarchy is:

TezCounters mixes three concrete group implementations, visible in the class:

// tez-api  org.apache.tez.common.counters.TezCounters
public class TezCounters extends AbstractCounters<TezCounter, CounterGroup> {
  private static class FrameworkGroupImpl<T extends Enum<T>> // TaskCounter, DAGCounter
      extends FrameworkCounterGroup<T, TezCounter> implements CounterGroup { /* ... */ }
  private static class GenericGroup                          // user-defined groups
      extends AbstractCounterGroup<TezCounter> implements CounterGroup { /* ... */ }
  private static class FileSystemGroup                       // FileSystemCounter
      extends FileSystemCounterGroup<TezCounter> implements CounterGroup { /* ... */ }
  // ...
}

The distinction matters: framework groups (TaskCounter, DAGCounter) are backed by a dense array indexed by enum ordinal — cheap, fixed-size. Generic groups (your app counters) are backed by a HashMap<String, TezCounter> and are what blow the limits below. findCounter(Enum) and findCounter(String group, String name) on AbstractCounters are the accessors:

rg -n "public synchronized C findCounter|public synchronized G getGroup|incrAllCounters|aggrAllCounters" \
  tez-api/src/main/java/org/apache/tez/common/counters/AbstractCounters.java

Standard groups

Note: Both enums carry the comment // Keep in sync with tez-ui/src/main/webapp/config/default-app-conf.js. The Tez UI hard-codes the counter names it knows how to display. If you add a TaskCounter member and want it shown in the UI, you must add it there too — verify with rg -n "SHUFFLE_BYTES" tez-ui/src/main/webapp/config/default-app-conf.js.

Counter limits (and how they kill DAGs)

rg -n "TEZ_COUNTERS_MAX|TEZ_COUNTERS_MAX_GROUPS|COUNTER_NAME_MAX_LENGTH|GROUP_NAME_MAX_LENGTH" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

Counters are unbounded by construction — nothing stops user code from calling findCounter("perFile", path) in a hot loop — so Tez imposes hard caps. The keys and defaults are defined in TezConfiguration (tez-api) and enforced by Limits (tez-api, org.apache.tez.common.counters.Limits):

// tez-api  org.apache.tez.common.counters.Limits
public synchronized void checkCounters(int size) {
  if (firstViolation != null) {
    throw new LimitExceededException(firstViolation);
  }
  if (size > COUNTERS_MAX) {
    firstViolation = new LimitExceededException("Too many counters: " + size +
                                                " max=" + COUNTERS_MAX);
    throw firstViolation;
  }
}
public synchronized void incrCounters() {
  checkCounters(totalCounters + 1);
  ++totalCounters;
}

Two subtleties every contributor trips over:

1. Name-length limits truncate silently; count limits throw. filterName() in Limits chops an over-long name to maxLen - 1 characters — no exception. But adding one counter past COUNTERS_MAX throws LimitExceededException (tez-api, same package). Truncation is why two counters with long, near-identical names can collide into one.
2. The violation is sticky. Once firstViolation is set, every subsequent checkCounters on that Limits re-throws it. A single overflow poisons the instance.

The static Limits config is seeded from a default TezConfiguration and can be overridden once via Limits.setConfiguration(conf). Because it is process-wide static state, the AM and each task JVM each hold their own Limits.

Where the exception lands

The dangerous moment is aggregation on completion, not the increment itself. When a vertex or DAG finishes, it materializes a full counter set by summing children (next section). If that sum exceeds tez.counters.max, the aggregation throws and the entity fails:

rg -n "LimitExceededException|constructFinalFullcounters" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

// tez-dag  DAGImpl.finished(...)
try {
  counters = constructFinalFullcounters();
} catch (LimitExceededException e) {
  addDiagnostic("Counters limit exceeded: " + e.getMessage());
  finalState = DAGState.FAILED;
}

So a DAG that ran every task successfully can still end in FAILED purely because the aggregate counter count crossed 1200. VertexImpl does the same in logJobHistoryVertexFailedEvent. The tell-tale in the AM log is the diagnostic string Counters limit exceeded: followed by Too many counters: N max=1200.

Warning: The classic production cause is a VertexManagerPlugin or processor that creates one counter per input file, per partition, or per user key. At small scale it works; at 2,000 partitions it fails the DAG at the finish line with zero task failures. Treat counter names as a bounded, static vocabulary — never derive them from data.

The TaskCounter enum

rg -n "SHUFFLE_BYTES|OUTPUT_BYTES|SPILLED_RECORDS|INPUT_RECORDS_PROCESSED|MERGE_PHASE_TIME|GC_TIME_MILLIS" \
  tez-api/src/main/java/org/apache/tez/common/counters/TaskCounter.java

TaskCounter (tez-api, org.apache.tez.common.counters.TaskCounter) is the enum every runtime input/output populates. The members that actually matter when you are debugging a slow or skewed vertex:

Tip: The three OUTPUT_BYTES* counters are the single most useful triad for diagnosing a shuffle. OUTPUT_BYTES is what your serializer produced, OUTPUT_BYTES_WITH_OVERHEAD adds the IFile/format framing, and OUTPUT_BYTES_PHYSICAL is what hit the disk after compression. A large gap between WITH_OVERHEAD and PHYSICAL means compression is earning its keep; a tiny gap means your codec is doing nothing and you are paying CPU for it.

Note there is no NUM_SPILLS member and no MR-style MERGED_MAP_OUTPUTS semantics: the real spill members are ADDITIONAL_SPILL_COUNT and ADDITIONAL_SPILLS_BYTES_*, and MERGED_MAP_OUTPUTS exists but counts source-side merges. Read the enum before citing a counter name — the MR-era names you have muscle memory for often do not exist here.

DAGCounter

rg -n "NUM_FAILED_TASKS|NUM_KILLED_TASKS|TOTAL_LAUNCHED_TASKS|DATA_LOCAL_TASKS|WALL_CLOCK_MILLIS|AM_CPU_MILLISECONDS" \
  tez-api/src/main/java/org/apache/tez/common/counters/DAGCounter.java

DAGCounter (tez-api) holds metrics the AM computes about the DAG as a whole — these are not summed from tasks, they are incremented by the AM state machines:

The locality and outcome counters are incremented from TaskAttemptImpl, not DAGImpl. createDAGCounterUpdateEventTAFinished builds a DAGEventCounterUpdate on every attempt finish:

// tez-dag  TaskAttemptImpl.createDAGCounterUpdateEventTAFinished(...)
jce.addCounterUpdate(DAGCounter.WALL_CLOCK_MILLIS, amSideWallClockTimeMs);
if (taState == TaskAttemptState.FAILED) {
  jce.addCounterUpdate(DAGCounter.NUM_FAILED_TASKS, 1);
  jce.addCounterUpdate(DAGCounter.DURATION_FAILED_TASKS_MILLIS, amSideWallClockTimeMs);
} else if (taState == TaskAttemptState.KILLED) {
  jce.addCounterUpdate(DAGCounter.NUM_KILLED_TASKS, 1);
  // ...
} else if (taState == TaskAttemptState.SUCCEEDED) {
  jce.addCounterUpdate(DAGCounter.NUM_SUCCEEDED_TASKS, 1);
  // ...
}

That event is routed to DAGImpl.CounterUpdateTransition, which applies each increment to dagCounters.findCounter(key). The locality counter is chosen when the attempt is scheduled — TaskAttemptImpl sets its localityCounter field to DATA_LOCAL_TASKS, RACK_LOCAL_TASKS, or OTHER_LOCAL_TASKS.

Warning: NUM_FAILED_TASKS counts failed attempts, not distinct tasks. A DAG that retries one poison task five times before it succeeds reports NUM_FAILED_TASKS=4, NUM_SUCCEEDED_TASKS=1. Do not read it as "4 tasks were lost." See failure-handling.md for how attempts and tasks differ.

Aggregation: attempt → task → vertex → DAG

This is the spine of the whole subsystem. Follow it in code:

rg -n "getCounters|reportedStatus.counters|selectBestAttempt" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java
rg -n "aggrAllCounters|incrAllCounters|constructFinalFullcounters|aggrTaskCounters" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java

1. In the task JVM. LogicalIOProcessorRuntimeTask owns the live TezCounters. Each input/output increments its own TaskCounter members plus any user counters. TaskReporter (tez-runtime-internals) snapshots them into a TaskStatusUpdateEvent and heartbeats it to the AM — but only every tez.task.am.heartbeat.counter.interval-ms.max (default 4000 ms), because counter objects are large:

// tez-runtime-internals  TaskReporter.HeartbeatCallable
if ((nonOobHeartbeatCounter.get() - prevCounterSendHeartbeatNum) * pollInterval
        >= sendCounterInterval) {
  sendCounters = true;
  prevCounterSendHeartbeatNum = nonOobHeartbeatCounter.get();
}

2. At the attempt. TaskAttemptImpl.StatusUpdaterTransition stores the latest snapshot in reportedStatus.counters; getCounters() returns it (or EMPTY_COUNTERS). This is the only place raw task counters live in the AM.

3. At the task. TaskImpl.getCounters() returns the winning attempt's counters via selectBestAttempt() — losers are discarded:

// tez-dag  TaskImpl.getCounters()
TaskAttempt bestAttempt = selectBestAttempt();
TezCounters taskCounters = (bestAttempt != null)
    ? bestAttempt.getCounters() : TaskAttemptImpl.EMPTY_COUNTERS;
return taskCounters;

4. At the vertex and DAG. On completion each entity builds an AggregateTezCounters by summing children. This is constructFinalFullcounters, guarded by mayBeConstructFinalFullCounters so it happens once:

// tez-dag  DAGImpl.constructFinalFullcounters()
final AggregateTezCounters aggregateTezCounters = new AggregateTezCounters();
aggregateTezCounters.aggrAllCounters(dagCounters);   // the AM's own DAG_* counters
for (Vertex v : this.vertices.values()) {
  aggregateTezCounters.aggrAllCounters(v.getAllCounters());
}
return aggregateTezCounters;

VertexImpl.constructFinalFullcounters is symmetric: it starts from the vertex's own counters, then aggrAllCounters(task.getCounters()) for every task.

Note: incrAllCounters and aggrAllCounters differ in intent. On a plain TezCounters, incrAllCounters(other) delegates to aggrAllCounters (simple addition). But constructFinalFullcounters uses an AggregateTezCounters, whose framework counters track min / max / count in addition to the sum — that is how the Tez UI shows per-vertex min/avg/max for a counter across tasks. Verify with rg -n "min|max|aggregate" tez-api/src/main/java/org/apache/tez/common/counters/AggregateFrameworkCounter.java.

When are full counters computed?

getAllCounters() short-circuits: if the entity is in a terminal state (SUCCEEDED/FAILED/KILLED/ERROR) it returns the cached fullCounters; otherwise it computes a live aggregate on the fly. There is also getCachedCounters(), which serves a 10-second cached snapshot to avoid re-summing every status poll of a running DAG:

// tez-dag  DAGImpl.getCachedCounters()
if (fullCounters == null && cachedCounters != null
    && ((cachedCountersTimestamp + 10000) - System.currentTimeMillis() > 0)) {
  return cachedCounters;   // reuse for up to 10s
}

Counters only reach the client when the caller asks for them. DAGImpl.getDAGStatus(Set<StatusGetOpts>) attaches counters only if StatusGetOpts.GET_COUNTERS is present:

// tez-dag  DAGImpl.getDAGStatus(...)
if (statusOptions.contains(StatusGetOpts.GET_COUNTERS)) {
  status.setDAGCounters(getAllCounters());
}

So a client that polls DAGStatus without GET_COUNTERS never pays the aggregation cost — and never sees counters. StatusGetOpts (tez-api, org.apache.tez.dag.api.client) has exactly two members: GET_COUNTERS and GET_MEMORY_USAGE.

 in-process (task JVM)                         Application Master
 ┌───────────────────────────┐   heartbeat    ┌────────────────────────────────┐
 │ LogicalIOProcessorRuntime  │  ≤ 4000 ms     │ TaskAttemptImpl                 │
 │   TaskCounter + user cntrs │──TaskStatus───▶│   reportedStatus.counters       │
 └───────────────────────────┘   UpdateEvent   └───────────────┬────────────────┘
                                                                │ selectBestAttempt()
                                                                ▼  (losers discarded)
                                             TaskImpl.getCounters()  → winning attempt
                                                                │ VERTEX terminal
                                                                ▼  aggrAllCounters(task)
                                             VertexImpl.constructFinalFullcounters()
                                                                │ DAG terminal
                                                                ▼  aggrAllCounters(vertex)
                                             DAGImpl.constructFinalFullcounters()
                                                                │ getDAGStatus(GET_COUNTERS)
                                                                ▼
                                             DAGStatus.getDAGCounters()  → client

flowchart TD
  RT["LogicalIOProcessorRuntimeTask<br/>live TezCounters"] -->|TaskReporter heartbeat, every ~4s| TA["TaskAttemptImpl.reportedStatus.counters"]
  TA -->|TASK_SUCCEEDED / selectBestAttempt| T["TaskImpl.getCounters<br/>(winning attempt only)"]
  T -->|VertexImpl.constructFinalFullcounters<br/>aggrAllCounters| V["VertexImpl fullCounters<br/>(AggregateTezCounters: sum/min/max)"]
  V -->|DAGImpl.constructFinalFullcounters<br/>aggrAllCounters| D["DAGImpl fullCounters"]
  DAGSELF["dagCounters<br/>DAGCounter.* from AM state machines"] --> D
  D -->|getDAGStatus GET_COUNTERS| CS["DAGStatus.getDAGCounters → client"]
  D -->|DAGFinishedEvent| H["HistoryEventHandler → history log / ATS / proto"]

Diagnostics strings

rg -n "addDiagnostic|addDiagnosticInfo|getDiagnostics" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java

Every level keeps its own List<String> of diagnostics. The API is almost uniform — note the name difference, which trips up grep:

Propagation is not a full union — each level appends a summary of the child failure plus its own context. A failed attempt's stack goes into its TaskFinishedEvent/VertexFinishedEvent diagnostics field; the vertex records "Vertex ... killed/failed due to: <terminationCause>" plus the aggregated child diag; the DAG records "DAG did not succeed due to <cause>. failedVertices:N killedVertices:M":

// tez-dag  VertexImpl (FAILED/KILLED path)
addDiagnostic("Vertex " + logIdentifier + " killed/failed due to:" + terminationCause);
if (!StringUtils.isEmpty(diag)) {
  addDiagnostic(diag);   // aggregated child diagnostics
}

// tez-dag  DAGImpl.finishWithTerminationCause(...)
String diagnosticMsg = "DAG did not succeed due to " + dag.terminationCause
    + ". failedVertices:" + dag.numFailedVertices
    + " killedVertices:" + dag.numKilledVertices;
dag.addDiagnostic(diagnosticMsg);

What the client sees

The client polls DAGStatus (tez-api, org.apache.tez.dag.api.client.DAGStatus). Its getDiagnostics() returns the DAG-level list; VertexStatus.getDiagnostics() returns the vertex-level list. Both are simple pass-throughs to the protobuf getDiagnosticsList():

// tez-api  DAGStatus.getDiagnostics()
public List<String> getDiagnostics() {
  return proxy.getDiagnosticsList();
}

To reach the attempt-level detail for a specific failed task you must go to the history log or Tez UI, not the top-level DAGStatus — the DAG string only summarizes the cause. This is the number-one confusion when triaging: the client prints "DAG did not succeed due to VERTEX_FAILURE" and the actual Java exception is three levels down in the attempt's diagnostics.

Tip: In the AM log, diagnostic strings ride along in the *FinishedEvent toString output, and the critical-event stream is tagged [HISTORY]. Grep the syslog for [HISTORY][DAG: to get the ordered stream of DAG/vertex/task lifecycle events with their diagnostics inline — that logger is HistoryEventHandler.criticalEvents.

History and ATS logging

rg -n "TEZ_HISTORY_LOGGING_SERVICE_CLASS|SimpleHistoryLoggingService|history.logging.log.level" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
ls tez-dag/src/main/java/org/apache/tez/dag/history/logging/impl/
ls tez-plugins/

Every lifecycle transition emits a HistoryEvent (there are ~24 types — see HistoryEventType in tez-dag, e.g. DAG_SUBMITTED, VERTEX_FINISHED, TASK_ATTEMPT_FINISHED). HistoryEventHandler (tez-dag, org.apache.tez.dag.history) is the fan-out point: it routes recovery events to the RecoveryService and history events to the configured HistoryLoggingService.

The service is selected by one key:

The real implementations on master:

There is also tez-yarn-timeline-history-with-acls (ACL-aware ATS variant) and tez-yarn-timeline-cache-plugin (the Timeline cache-plugin the UI uses to group DAGs). Confirm the full set with ls tez-plugins/.

Filtering by log level

HistoryEventHandler gates each event through shouldLogEvent, which compares the DAG's HistoryLogLevel against the event type's level:

// tez-dag  HistoryEventHandler.shouldLogEvent(...)
if (dagLogLevel.shouldLog(historyEvent.getEventType().getHistoryLogLevel())) {
  return shouldLogTaskAttemptEvents(event, dagLogLevel);
}
return false;

HistoryLogLevel (tez-api, org.apache.tez.dag.api) is an ordered enum: NONE < AM < DAG < VERTEX < TASK < TASK_ATTEMPT < ALL, with DEFAULT = ALL. Set tez.history.logging.log.level=VERTEX to suppress the per-attempt firehose on a DAG with millions of tasks — a real scalability lever, since TASK_ATTEMPT_FINISHED events dominate the volume. There is a finer filter, tez.history.logging.taskattempt-filters, that drops attempts with specified TaskAttemptTerminationCause values when the level is TASK_ATTEMPT.

Warning: Dropping to AM or DAG level makes the Tez UI's per-task drill- down go blank — the events simply were never emitted. That is a debugging trade-off, not a bug. If a user reports "the UI shows the DAG but no tasks," check tez.history.logging.log.level before anything else.

Tez UI data flow

ls tez-ui/src/main/webapp/app/          # Ember app: adapters, serializers, models, routes
ls tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/

The Tez UI is an Ember.js single-page app (tez-ui/). It does not read the AM directly. The flow is: AM state machines → HistoryEventHandler → HistoryLoggingService → sink (Timeline Server, or proto/simple files) → the UI reads via REST. With ATS, the UI's adapters query the Timeline Server's entity REST API; the tez-yarn-timeline-cache-plugin supplies the DAG-grouping and domain logic the Timeline reader needs. For proto/simple logs there is tez-history-parser (tez-plugins), whose parsers — ATSFileParser, ProtoHistoryParser, SimpleHistoryParser — reconstruct the DAG object model from persisted events so offline tools can consume it.

Those offline tools are the job-analyzer plugins in tez-tools/analyzers/job-analyzer — real, runnable analyzers such as SkewAnalyzer, SpillAnalyzerImpl, SlowNodeAnalyzer, CriticalPathAnalyzer, and ShuffleTimeAnalyzer. They read the parsed history and emit CSV. That is the supported path for "diff two runs' counters" — see also tez-tools/counter-diff.

Note: Don't over-model the UI in your head. From a contributor's angle the only invariant that matters is: the UI can only show what a HistoryLoggingService persisted. If a field is missing in the UI, trace it back to the event that should carry it and check whether it was emitted at the active log level.

DAG visualization: the .dot file

rg -n "generateDAGVizFile|\.dot|TEZ_GENERATE_DEBUG_ARTIFACTS" \
  tez-dag/src/main/java/org/apache/tez/Utils.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java
rg -n "TEZ_GENERATE_DEBUG_ARTIFACTS" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

Yes — the AM emits a Graphviz .dot file, but only when debug artifacts are enabled. The gate is:

When true, DAGAppMaster.writeDebugArtifacts and DAGImpl call Utils.generateDAGVizFile (tez-dag, org.apache.tez.Utils). It walks the DAGPlan protobuf, emitting a node per vertex (labeled with the processor's short class name, and — when a scheduler is present — a priority), boxes for each root input/leaf output, and an edge per EdgePlan labeled with its dataMovementType and schedulingType:

// tez-dag  org.apache.tez.Utils.generateDAGVizFile(...)
String outputFile = "";
if (logDirs != null && logDirs.length != 0) {
  outputFile += logDirs[0] + File.separator;   // → AM container log dir
}
outputFile += dagId.toString();
if (scheduler != null) {
  outputFile += "_priority";                    // second pass, once priorities exist
}
outputFile += ".dot";
graph.save(outputFile);

So the file lands in the AM's first log directory as <dagId>.dot (and <dagId>_priority.dot once vertex priorities are known — the second write overrides the first). Render it locally:

# after copying the .dot out of the AM container log dir
dot -Tpng dag_1700000000000_0001_1.dot -o dag.png
dot -Tsvg dag_1700000000000_0001_1_priority.dot -o dag.svg

This is the DAG plan only — topology, edge types, priorities. It carries no counters or diagnostics; for those, use the UI or the history log. Enabling tez.generate.debug.artifacts also writes the DAG plan as text (writePBTextFile), which is the fastest way to confirm what plan the client actually submitted (see dag-client.md and dag-model.md).

Reading exercise

Work these against the checkout; each is a one-liner that returns hits.

cd /Users/s0x/src/oss-repos/tez

# 1. Read the whole TaskCounter enum — know every member before you cite one.
sed -n '1,240p' tez-api/src/main/java/org/apache/tez/common/counters/TaskCounter.java

# 2. Every place the runtime library increments a counter.
rg -n "\.increment\(|findCounter\(" tez-runtime-library/src/main/java | head -30

# 3. Trace the LimitExceededException kill path end to end.
rg -rn "LimitExceededException" tez-api/src/main/java tez-dag/src/main/java

# 4. Confirm aggregation uses AggregateTezCounters (min/max), not plain sums.
rg -n "AggregateTezCounters|aggrAllCounters" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

# 5. Count diagnostic append sites per level — build a mental model of the flow.
rg -c "addDiagnostic|addDiagnosticInfo" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java

# 6. See how a status poll decides whether to attach counters at all.
rg -n "GET_COUNTERS|getDAGStatus" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java

Then, on a real run: enable tez.generate.debug.artifacts=true, submit a small DAG, pull <dagId>.dot from the AM log dir, render it, and cross-check the edge dataMovementType labels against the plan you built in logical-physical.md. Finally open the same DAG in the Tez UI and confirm the per-vertex counter min/avg/max matches what AggregateTezCounters would compute from the tasks you see.

Common bugs and symptoms

Validation: prove you understand this

1. Name the four standard framework/FS counter groups and the exact class that defines each. Which two are backed by a dense enum-indexed array and which by a HashMap, and why does that distinction matter for tez.counters.max?
2. A DAG runs 2,000 tasks, all succeed, and it still ends FAILED. Give the precise mechanism — which method throws, what it was doing, and the config key and default that set the ceiling.
3. Two attempts of the same task report different counter values. Explain which attempt's counters survive to the task level, the method that chooses, and what happens to the loser's counters.
4. Trace one counters.findCounter("MyApp","ROWS").increment(1) call in a processor all the way to DAGStatus.getDAGCounters() on the client, naming every class it passes through and the event that carries it off the task JVM.
5. Distinguish SHUFFLE_BYTES, SHUFFLE_BYTES_TO_MEM, SHUFFLE_BYTES_TO_DISK, and SHUFFLE_BYTES_DISK_DIRECT using the enum's own definitions, and give the arithmetic relationship the Javadoc states among the first three.
6. Given the AM diagnostic DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, name the four levels whose diagnostics lists you would read to reach the underlying Java stack, and the exact class/field that stores each copy.
7. You set tez.history.logging.log.level=DAG. Predict exactly what disappears from the Tez UI and why, citing the method in HistoryEventHandler that makes the decision and the ordering of the HistoryLogLevel enum.