Container Reuse

Container reuse is the single biggest reason Tez runs short-task DAGs faster than MapReduce. MapReduce pays the full YARN allocation + JVM launch tax for every task; Tez pays it once per container and then keeps handing new work to the same warm JVM. The policy that decides whether a finished container gets another task, which task it gets, and when an idle container finally goes back to YARN lives entirely in the AM's task scheduler — the container itself just keeps asking "what next?". Get this wrong as a contributor and you produce idle clusters hoarding containers, DAGs that starve their own high-priority vertices, or tasks silently poisoned by a previous task's static state.

This chapter dissects both scheduler implementations in tez-dag/src/main/java/org/apache/tez/dag/app/rm/ — the classic YarnTaskSchedulerService with its DelayedContainerManager, and the DagAwareYarnTaskScheduler that is the default on master — plus the runtime half of the contract in TezChild. It sits between the task attempt lifecycle (which produces the terminal events), the scheduler (which owns priorities and requests), the runtime (which executes tasks in the reused JVM), and YARN integration (which supplies and reclaims the containers).

After this chapter you can:

• Trace a task-attempt completion from TaskAttemptImpl through TaskSchedulerManager into the scheduler's held-container logic.
• Explain the node → rack → any locality relaxation in both schedulers, and which config key gates each step.
• Quote every tez.am.container.reuse.* key with its real default.
• Use ObjectRegistry correctly and enumerate the JVM-reuse hazards it does not protect you from.
• Read an AM log and answer "why was this container released?"

All commands below run from the root of a Tez checkout. Line numbers are deliberately absent — code moves between branches; the grep commands are the stable coordinates.

The cast

Two scheduler implementations coexist because the DAG-aware one was written later to fix deadlock-shaped problems (a held container being matched to a vertex that transitively depends on the very tasks waiting for capacity). Which one runs is itself a config:

grep -n "TEZ_AM_YARN_SCHEDULER_CLASS" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

tez.am.yarn.scheduler.class defaults to "org.apache.tez.dag.app.rm.DagAwareYarnTaskScheduler" on master. Both implementations read the same tez.am.container.reuse.* keys, so the configuration section below applies to either.

Why reuse pays

Container allocation has three sequential costs: the RM round-trip (addContainerRequest → RM scheduling cycle → onContainersAllocated), the NM launch (ContainerLaunchContext setup, resource localization, forking the JVM), and JVM warmup (classloading, JIT). For a 5-second task:

That is 25–60% overhead per task, paid once per container with reuse instead of once per task. For a Hive query with ten thousand 2-second tasks, reuse is the difference between an interactive query and a batch job. Session mode (see tez-client.md and hive-integration.md) extends the same machinery across DAGs: the containers warmed by query 1 execute query 2.

The runtime half: TezChild's task loop

grep -n "shouldDie\|getTaskFuture\|cleanupOnTaskChanged\|while (!executor.isTerminated" \
  tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java

The container-side contract is one loop. TezChild.run() does not exit after a task completes — it polls the AM umbilical for the next task, forever, until the AM says die:

// tez-runtime-internals — org.apache.tez.runtime.task.TezChild#run
while (!executor.isTerminated() && !isShutdown.get()) {
  // ...
  ListenableFuture<ContainerTask> getTaskFuture = executor.submit(containerReporter);
  ContainerTask containerTask = getTaskFuture.get();
  // ...
  if (containerTask.shouldDie()) {
    LOG.info("ContainerTask returned shouldDie=true for container {}, Exiting", containerIdString);
    shutdown();
    return new ContainerExecutionResult(ContainerExecutionResult.ExitStatus.SUCCESS, null,
        "Asked to die by the AM");
  } else {
    // ...
    childUGI = handleNewTaskCredentials(containerTask, childUGI);
    handleNewTaskLocalResources(containerTask, childUGI);
    cleanupOnTaskChanged(containerTask);
    // Execute the Actual Task
    TezTaskRunner2 taskRunner = new TezTaskRunner2(defaultConf, childUGI,
        localDirs, containerTask.getTaskSpec(), appAttemptNumber, /* ... */);

The containerReporter is a ContainerReporter that calls umbilical.getTask(containerContext) in a backoff loop; the AM side of that RPC is TezTaskCommunicatorImpl.getTask. Every reuse decision therefore happens on the AM: the JVM is a dumb worker asking for its next assignment. Note the three per-task resets — credentials, local resources, and cleanupOnTaskChanged (which clears the ObjectRegistry, more below). Everything not on that list survives into the next task. See tez-runtime.md for what TezTaskRunner2 does with each task.

The AM half: what happens when an attempt ends

grep -n "S_TA_ENDED\|handleTASucceeded\|handleTAUnsuccessfulEnd\|deallocateTask" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/TaskSchedulerManager.java
grep -rn "new AMSchedulerEventTAEnded" \
  tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java

When a TaskAttemptImpl hits a terminal state (see task-attempt-lifecycle.md), it fires AMSchedulerEventTAEnded at the TaskSchedulerManager, whose S_TA_ENDED handler branches on the final state: SUCCEEDED goes to handleTASucceeded, FAILED/KILLED to handleTAUnsuccessfulEnd. Both end in a deallocateTask downcall into the scheduler, but with an inverted flag:

// tez-dag — org.apache.tez.dag.app.rm.TaskSchedulerManager#handleTASucceeded
if (event.getUsedContainerId() != null) {
  sendEvent(new AMContainerEventTASucceeded(usedContainerId,
      event.getAttemptID()));
  // ...
}
// ...
wasContainerAllocated = taskSchedulers[event.getSchedulerId()].deallocateTask(attempt,
    true, null, event.getDiagnostics());

// tez-dag — org.apache.tez.dag.app.rm.TaskSchedulerManager#handleTAUnsuccessfulEnd
wasContainerAllocated = taskSchedulers[event.getSchedulerId()]
    .deallocateTask(attempt, false, event.getTaskAttemptEndReason(), event.getDiagnostics());
// ...
if (attemptContainerId != null) {
  // Ask the container to stop.
  sendEvent(new AMContainerEventStopRequest(attemptContainerId));

This asymmetry is the first rule of reuse: only a SUCCEEDED attempt returns its container to the reuse pool. A failed or killed attempt's container gets an explicit AMContainerEventStopRequest — the JVM might be poisoned (OOM-adjacent, corrupted static state, half-written spill files), so Tez does not risk it. The corresponding AMContainerImpl state machine (states ALLOCATED, LAUNCHING, IDLE, RUNNING, STOP_REQUESTED, STOPPING, COMPLETED in AMContainerState — see state-machines.md) tracks the container's own lifecycle; the scheduler tracks its assignability.

sequenceDiagram
    participant TA as TaskAttemptImpl
    participant TSM as TaskSchedulerManager
    participant SCH as DagAwareYarnTaskScheduler
    participant AMC as AMContainerImpl
    participant TC as TezChild (in container)

    TA->>TSM: AMSchedulerEventTAEnded (SUCCEEDED)
    TSM->>AMC: AMContainerEventTASucceeded (RUNNING -> IDLE)
    TSM->>SCH: deallocateTask(attempt, taskSucceeded=true)
    SCH->>SCH: idleTracker.add(heldContainer)
    SCH->>SCH: tryAssignReuseContainer(hc, appState, isSession)
    alt matching pending request
        SCH->>TSM: taskAllocated(newAttempt, container)
        TSM->>AMC: AMContainerEventAssignTA (IDLE -> RUNNING)
        TC->>AMC: umbilical.getTask() returns new TaskSpec
    else no match yet
        SCH->>SCH: scheduleForReuse(localitySchedulingDelay)
        Note over SCH: relax locality per retry, release on idle expiry
    else idle expired / preempted
        SCH->>TSM: containerBeingReleased(cid)
        TC->>AMC: getTask() returns shouldDie=true
    end

When the scheduler does find a match, the upcall is TaskSchedulerManager.taskAllocated, which sends AMContainerEventAssignTA carrying the new TaskSpec — that is the payload the blocked umbilical.getTask() in TezChild eventually receives.

DagAwareYarnTaskScheduler: the default path

grep -n "enum HeldContainerState\|tryAssignReuseContainer\|tryAssignTaskToIdleContainer\|moveToNextMatchingLevel" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java

Its deallocateTask is the decision point you traced above:

// tez-dag — org.apache.tez.dag.app.rm.DagAwareYarnTaskScheduler#deallocateTask
hc = removeTaskAssignment(task);
if (hc != null) {
  if (taskSucceeded && shouldReuseContainers) {
    idleTracker.add(hc);
    newAssignment = tryAssignReuseContainer(hc, appState, isSession);
    // ...
  } else {
    if (releaseContainer(hc)) {
      releasedLaunchedContainer = hc.getId();
    }
  }
}

Each held container carries an explicit matching state:

// tez-dag — org.apache.tez.dag.app.rm.DagAwareYarnTaskScheduler.HeldContainerState
private enum HeldContainerState {
  MATCHING_LOCAL(true),
  MATCHING_RACK(true),
  MATCHING_ANY(true),
  ASSIGNED(false),
  RELEASED(false);

Matching runs from both directions. When a task request arrives, the scheduler scans idle containers best-locality-first; when a container goes idle, tryAssignReuseContainer scans pending requests. The request-side cascade:

// tez-dag — org.apache.tez.dag.app.rm.DagAwareYarnTaskScheduler#tryAssignTaskToIdleContainer(request)
if (request.hasLocality()) {
  hc = tryAssignTaskToIdleContainer(request, request.getNodes(), HeldContainerState.MATCHES_LOCAL_STATES);
  if (hc == null) {
    hc = tryAssignTaskToIdleContainer(request, request.getRacks(), HeldContainerState.MATCHES_RACK_STATES);
    if (hc == null) {
      hc = tryAssignTaskToIdleContainer(request, ResourceRequest.ANY, HeldContainerState.MATCHES_ANY_STATES);
    }
  }
}

Note the EnumSet gating: a request may only match a container at ANY if that container has already relaxed itself to MATCHING_ANY — the container's own clock (one moveToNextMatchingLevel() per localitySchedulingDelay tick, gated by reuseRackLocal / reuseNonLocal) controls how fast locality decays. Inside the innermost loop, compatibility is a container-signature superset check:

// tez-dag — org.apache.tez.dag.app.rm.DagAwareYarnTaskScheduler#tryAssignTaskToIdleContainer(location)
Object csig = hc.getSignature();
if (csig == null || signatureMatcher.isSuperSet(csig, request.getContainerSignature())) {
  // ... pick as bestMatch, prefer containers without affinity claims
} else {
  LOG.debug("Unable to assign task {} to container {} due to signature mismatch", ...);
}

The signature is a ContainerContext; isSuperSet (in tez-dag/src/main/java/org/apache/tez/dag/app/ContainerContext.java) demands identical javaOpts, an environment superset, and compatible local resources. Different vertex-level taskLocalFiles or JVM options silently kill reuse — one of the most common "0% reuse" root causes in Hive deployments that set per-vertex options.

Two more behaviors distinguish this scheduler:

• Descendant blocking. tryAssignTaskToIdleContainer refuses requests whose vertex is a descendant of vertices with pending tasks (requestTracker.isRequestBlocked): giving a downstream vertex the container that an upstream vertex is queued for would invert DAG scheduling order and can livelock a resource-tight queue.
• Idle handling is state-driven, not a sweeper thread. An idle container schedules itself (each HeldContainer implements Callable<Void>) for re-evaluation. When the app is idle, handleReuseContainerWhenIdle either retains it (session container within tez.am.session.min.held-containers), releases it ("Releasing expired idle container {}"), or re-schedules it for its remaining idle budget. When requests are pending but nothing matches even at max relaxation, it is released immediately: "Releasing idle container {} due to pending requests".

YarnTaskSchedulerService: the classic path

grep -n "static class HeldContainer\|enum LocalityMatchLevel\|class DelayedContainerManager" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java
grep -n "assignDelayedContainer\|assignReUsedContainerWithLocation\|tryAssignReUsedContainers" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java

The older scheduler expresses the same policy with a dedicated sweeper thread. Its HeldContainer tracks a locality level (enum LocalityMatchLevel { NEW, NODE, RACK, NON_LOCAL }), a nextScheduleTime, and a containerExpiryTime. The DelayedContainerManager thread holds a PriorityBlockingQueue<HeldContainer> ordered by nextScheduleTime and repeatedly pops the next due container:

// tez-dag — org.apache.tez.dag.app.rm.YarnTaskSchedulerService.DelayedContainerManager#mainLoop
HeldContainer delayedContainer = delayedContainers.peek();
// ...
if (currentTs - nextScheduleTs >= 0) {
  synchronized(YarnTaskSchedulerService.this) {
    delayedContainer = delayedContainers.poll();
    // ...
    assignedContainers = assignDelayedContainer(delayedContainer);
  }
  // Inform App should be done outside of the lock
  informAppAboutAssignments(assignedContainers);
}

assignDelayedContainer is the whole policy in one method. If the AM is idle (no DAG running, or no pending requests) it decides hold-vs-release based on session state and expiry; if a DAG is running it walks the locality ladder:

// tez-dag — org.apache.tez.dag.app.rm.YarnTaskSchedulerService#assignDelayedContainer
// if match level is NEW or NODE, match only at node-local
if (isNew || localityMatchLevel.equals(HeldContainer.LocalityMatchLevel.NEW)
    || localityMatchLevel.equals(HeldContainer.LocalityMatchLevel.NODE)
    // ...
    ) {
  assignReUsedContainerWithLocation(containerToAssign, NODE_LOCAL_ASSIGNER, assignedContainers, true);
}
// if re-use allowed at rack ...
if (assignedContainers.isEmpty()) {
  if ((reuseRackLocal || isNew) && (localitySchedulingDelay == 0 ||
    (localityMatchLevel.equals(HeldContainer.LocalityMatchLevel.RACK) /* ... */))) {
    assignReUsedContainerWithLocation(containerToAssign, RACK_LOCAL_ASSIGNER, assignedContainers, false);
  }
}
// if re-use allowed at non-local ...
if (assignedContainers.isEmpty()) {
  if ((reuseNonLocal || isNew) && (localitySchedulingDelay == 0
      || localityMatchLevel.equals(HeldContainer.LocalityMatchLevel.NON_LOCAL))) {
    assignReUsedContainerWithLocation(containerToAssign, NON_LOCAL_ASSIGNER, assignedContainers, false);
  }
}

No match? The container either has its LocalityMatchLevel incremented and is re-queued localitySchedulingDelay ms out, or — once past containerExpiryTime — is released with "Container's idle timeout expired. Releasing container...". A timeline for one held container with defaults:

t=0        task N SUCCEEDED; container wrapped as HeldContainer
           localityMatchLevel=NODE, expiry = now + rand(5000..10000) ms
  |
t=0..d     try NODE-local pending requests        (d = 250ms delay steps)
t=d        no match -> level=RACK   (only if rack-fallback enabled: default yes)
t=2d       no match -> level=NON_LOCAL (only if non-local-fallback: default NO)
  |            |
  |            +-- match at any step -> assign task N+1, level resets, expiry cleared
  v
t=5..10s   still unmatched past containerExpiryTime
           -> releaseUnassignedContainers() back to YARN, TezChild gets shouldDie

Note: With the default non-local-fallback.enabled=false, a held container whose node and rack have no pending work is not handed to an arbitrary task — it idles until expiry. That is deliberate: shipping a data-heavy task off-rack usually costs more than a fresh, well-placed allocation. isNew containers (freshly allocated, never ran a task — only held at all when new-containers.enabled=true) bypass the fallback flags and match at every level.

Configuration reference

grep -n "TEZ_AM_CONTAINER_REUSE\|CONTAINER_IDLE_RELEASE_TIMEOUT\|SESSION_MIN_HELD_CONTAINERS" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

Every key below is quoted from tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java; all are Scope.AM.

Tip: Session-held containers are exempt from idle expiry while the AM is idle — in assignDelayedContainer, containers in sessionMinHeldContainers get their expiry pushed forward instead of being released. If your idle HiveServer2 session "leaks" N containers, check this knob before filing a bug: holding them is the configured behavior.

ObjectRegistry: carrying warm state across tasks

grep -n "interface ObjectRegistry\|cacheForVertex\|cacheForDAG\|cacheForSession" \
  tez-api/src/main/java/org/apache/tez/runtime/api/ObjectRegistry.java
grep -n "enum ObjectLifeCycle\|clearCache" \
  tez-runtime-internals/src/main/java/org/apache/tez/runtime/common/objectregistry/ObjectRegistryImpl.java

Reuse gives you a warm JVM; ObjectRegistry is the sanctioned way to exploit it. It is exposed to processors/inputs/outputs via their context objects (see ipo-abstractions.md):

// tez-api — org.apache.tez.runtime.api.ObjectRegistry
public interface ObjectRegistry {
  public Object cacheForVertex(String key, Object value);
  public Object cacheForDAG(String key, Object value);
  public Object cacheForSession(String key, Object value);
  public Object get(String key);
  public boolean delete(String key);
}

The lifecycle enum lives in the implementation, org.apache.tez.runtime.common.objectregistry.ObjectRegistryImpl (tez-runtime-internals): ObjectLifeCycle { SESSION, DAG, VERTEX } — SESSION entries are "valid for the lifetime of the Tez JVM/Session". One instance per key regardless of lifecycle; re-caching under a different lifecycle replaces the old entry. Eviction is wired into the TezChild loop you already read:

// tez-runtime-internals — org.apache.tez.runtime.task.TezChild#cleanupOnTaskChanged
TezVertexID newVertexID = containerTask.getTaskSpec().getTaskAttemptID().getVertexID();
if (lastVertexID != null) {
  if (!lastVertexID.equals(newVertexID)) {
    objectRegistry.clearCache(ObjectRegistryImpl.ObjectLifeCycle.VERTEX);
  }
  if (!lastVertexID.getDAGID().equals(newVertexID.getDAGID())) {
    objectRegistry.clearCache(ObjectRegistryImpl.ObjectLifeCycle.DAG);
    startedInputsMap = HashMultimap.create();
  }
}

This is why Hive's map-join is fast under Tez: the deserialized broadcast hash table is cached in the registry, and the map tasks that run successively in the same container each skip rebuilding it. Pig does the equivalent for shared lookup structures. The discipline to copy: cache immutable or thread-confined objects only, keyed uniquely enough to never collide across vertices (get() searches all lifecycles), and never cache anything holding file handles or credentials at SESSION scope.

JVM reuse hazards: what does NOT get cleaned

TezChild resets a short, explicit list per task: credentials/UGI, added local resources, the object registry (per the boundaries above), FileSystem.clearStatistics(), and the logging MDC context. Everything else in the JVM persists. Real categories of poison, all grounded in how TezChild/TezTaskRunner2 execute successive tasks on pooled threads:

Warning: The single most reliable diagnostic question for "flaky task failures that never reproduce in tests" is: does it go away with tez.am.container.reuse.enabled=false? If yes, you are hunting static state, not logic. Bisect by disabling rack/non-local fallback first (narrows which tasks co-locate), then diff task-attempt ordering per container from the AM log's Assigning container to task lines.

Reuse under pressure: preemption and speculation

grep -n "preemptIfNeeded\|Preempting new container\|Preempting container" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java
grep -n "maybePreempt\|Avoiding preemption" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java

When a high-priority request cannot be satisfied (tez.am.preemption.percentage, default 10; wait deadline tez.am.preemption.max.wait-time-ms, default 60*1000; pacing tez.am.preemption.heartbeats-between-preemptions, default 3 — all in TezConfiguration), held containers are the first casualties, in a deliberate order. The classic scheduler's preemptIfNeeded releases new (never-used) held containers first, and explicitly defers to the assignment loop for idle reused ones:

// tez-dag — org.apache.tez.dag.app.rm.YarnTaskSchedulerService#preemptIfNeeded
for (HeldContainer heldContainer : delayedContainerManager.delayedContainers) {
  if (!heldContainer.isNew()) {
    LOG.debug("Reused container exists. Wait for assignment loop to release it. " ...);
    return true;
  }
  // ...
}
if (lowestPriNewContainer != null) {
  LOG.info("Preempting new container: " + lowestPriNewContainer.getId() + ...);
  releaseUnassignedContainers(Collections.singletonList(lowestPriNewContainer));
  // RM thinks it serviced our request; re-ask at this priority (TEZ-915)
  maybeRescheduleContainerAtPriority(lowestPriNewContainer.getPriority());

Only when no held containers remain does it kill running lower-priority tasks ("Preempting container: ... currently allocated to a task."). The re-request after releasing a new container matters: the RM already debited that allocation against your demand, so without maybeRescheduleContainerAtPriority (TEZ-915 — verify with git log --oneline --grep=TEZ-915) the AM would hang waiting for capacity it silently returned. The DAG-aware scheduler encodes the same idea as guards in maybePreempt: it refuses to preempt anything while idle containers exist ("Avoiding preemption since there are {} idle containers" — they will be matched or released by reuse logic) and only preempts containers assigned to descendants of the starved vertices, newest-assignment-first.

Speculation (see failure-handling.md) composes with reuse without special cases: a speculative attempt is an ordinary TaskRequest at the same priority, so it can itself be satisfied by a reused container. When one attempt wins, the loser is KILLED — and per the first rule above, a killed attempt's container is stopped, not reused. Aggressive speculation on a reuse-heavy DAG therefore quietly churns warm containers; if you see shouldDie=true exits paired with KILLED speculative attempts in the same container, that is the mechanism, not a bug.

Log lines to grep in AM logs

All strings below are verbatim from master; grep your AM's syslog for them.

Reading exercise

Run each, then answer the questions that follow.

# 1. The succeeded-vs-unsuccessful asymmetry
grep -n "deallocateTask\|AMContainerEventStopRequest\|AMContainerEventTASucceeded" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/TaskSchedulerManager.java

# 2. The whole classic policy in one method
grep -n "assignDelayedContainer" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java
# read the method top to bottom; annotate the IDLE / RUNNING_APP / else branches

# 3. Both locality ladders
grep -n "LocalityMatchLevel\|incrementLocalityMatchLevel\|resetLocalityMatchLevel" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java
grep -n "moveToNextMatchingLevel\|atMaxMatchLevel\|resetMatchingLevel" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java

# 4. What makes two containers "compatible"
grep -n "isSuperSet\|localResourcesCompatible\|javaOpts" \
  tez-dag/src/main/java/org/apache/tez/dag/app/ContainerContext.java

# 5. Everything the runtime resets between tasks — and by omission, what it doesn't
grep -n "handleNewTaskCredentials\|handleNewTaskLocalResources\|cleanupOnTaskChanged\|clearStatistics" \
  tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java

1. In assignDelayedContainer, under exactly which condition does an idle container in a session get its expiry extended instead of being released?
2. Why does assignReUsedContainerWithLocation pass honorLocality=true only for the NODE_LOCAL_ASSIGNER call?
3. In the DAG-aware scheduler, why can a request only match a container at ANY when the container is in MATCHING_ANY, rather than immediately?
4. List the three checks ContainerContext.isSuperSet performs and construct a Hive setting that would fail each one.
5. cleanupOnTaskChanged resets startedInputsMap only on a DAG boundary. What runtime feature does that map serve, and why is vertex-boundary reset unnecessary for it? (Cross-check shuffle-sort.md.)

Common bugs and symptoms

Validation: prove you understand this

1. Trace, class by class, what happens between a task attempt entering SUCCEEDED and the same container starting its next task. Name the event types on each hop and the two upcall/downcall methods on the scheduler boundary.
2. Why does Tez stop — rather than reuse — the container of a FAILED or KILLED attempt? Cite the exact call in handleTAUnsuccessfulEnd and give two concrete poisoning scenarios it defends against.
3. With defaults (delay-allocation-millis=250, rack fallback on, non-local fallback off, idle window 5–10 s), describe the full lifetime of a held container whose node and rack never see another pending request. When and why does the randomized expiry fire, and what does TezChild observe?
4. A user reports reuse stopped working after they set vertex-specific -Xmx options for one vertex. Explain the mechanism, citing the three conditions of ContainerContext.isSuperSet.
5. Contrast the two schedulers' idle-container machinery: sweeper thread + PriorityBlockingQueue vs self-scheduling HeldContainer state machine. Name one problem the DAG-aware descendant-blocking check solves that the classic scheduler cannot express.
6. Your session AM holds 40 idle containers and a new DAG's first vertex needs containers at a higher priority. Using only verified behavior, explain why preemption of running tasks will not occur while those 40 are held, and which log lines prove it.
7. Design a safe ObjectRegistry usage for a 2 GB broadcast dictionary shared by two vertices of the same DAG: which cacheFor* call, what key scheme, what happens to it at each boundary in cleanupOnTaskChanged, and one mistake that would OOM the container three DAGs later.