Glossary

Every term you will meet reading Tez source, a JIRA thread, a PR review, or an AM log line — defined in one to four sentences, with a link to the chapter that covers it in full. Use this as a fast lookup; follow the link when you need depth. Terms span the whole stack: the API and DAG model (org.apache.tez.dag.api.*), the AM state machines (org.apache.tez.dag.app.*), and the runtime data path (org.apache.tez.runtime.*).

Note: Tez reuses much MapReduce vocabulary (spill, IFile, combiner, shuffle, merge) but layers a general DAG model on top. Where a term means something subtly different in Tez than in classic MapReduce, the entry says so.

Jump to: A · B · C · D · E · F · G · H · I · J · K · L · M · N · O · P · R · S · T · U · V

A

AM (ApplicationMaster) — The per-application coordinator process, org.apache.tez.dag.app.DAGAppMaster, that runs one YARN container and drives every DAG: it builds the state machines, talks to the RM for containers, launches tasks, and routes events. One AM serves one application; in session mode it serves many DAGs sequentially. See ../deep-dives/dag-app-master.md.

AMContainer — The AM's model of one YARN container (AMContainerImpl), with its own state machine (AMContainerState: ALLOCATED, LAUNCHING, IDLE, RUNNING, …). Tracks which task attempt currently occupies the container and whether it can be reused. See ../deep-dives/container-reuse.md.

AMNode — The AM's model of one cluster node (AMNodeImpl), used for blacklisting: it counts task failures per node and can mark a node BLACKLISTED or UNHEALTHY. See ../deep-dives/failure-handling.md.

Attempt (task attempt) — One execution try of a task, identified by a TezTaskAttemptID. A task may have several attempts over its life: retries after failure, or concurrent speculative copies. The attempt is what actually runs in a container. See ../deep-dives/task-attempt-lifecycle.md.

ATS (YARN Application Timeline Server) — The historical event store Tez can log to via ATSHistoryLoggingService (in the tez-yarn-timeline-history plugin); the Tez UI reads DAG history from it. See ../deep-dives/counters-diagnostics.md.

Auto-parallelism — A vertex deciding its own task count at runtime instead of being fixed at plan time. ShuffleVertexManager collects VertexManagerEvent partition-size stats from upstream tasks and, before the vertex starts, reduces parallelism by merging partitions when the data is smaller than expected. See ../deep-dives/scheduler.md.

B

Blacklisting — Marking a node faulty after too many task failures so no more attempts are scheduled there; governed by tez.am.node-blacklisting.enabled and an ignore-threshold percentage so a cluster-wide outage doesn't blacklist everything. Implemented in AMNodeTracker / AMNodeImpl. See ../deep-dives/failure-handling.md.

Broadcast edge — An edge whose data-movement type sends every producer output to every consumer task (EdgeProperty.DataMovementType.BROADCAST), routed by BroadcastEdgeManager. Used for replicating a small side input (e.g. a Hive map-join hash table) to all downstream tasks. See ../deep-dives/logical-physical.md.

C

Combiner — An optional map-side reduce that pre-aggregates key/value pairs during spill and merge to shrink shuffle volume; set via tez.runtime.combiner.class, invoked once at least tez.runtime.combine.min.spills spills exist. Same idea as MapReduce's combiner. See ../deep-dives/shuffle-sort.md.

Committer (OutputCommitter) — The org.apache.tez.runtime.api.OutputCommitter that makes a data-sink output visible atomically (commitOutput) or discards it (abortOutput). Commit timing is controlled by tez.am.commit-all-outputs-on-dag-success: commit per-vertex on vertex success, or all at once on DAG success. See ../deep-dives/failure-handling.md.

Container reuse — Keeping a YARN container alive after its task finishes and assigning it the next matching task, avoiding JVM launch overhead. Enabled by default (tez.am.container.reuse.enabled); the scheduler honors locality with a delay before falling back to rack/non-local reuse. See ../deep-dives/container-reuse.md.

D

DAG (Directed Acyclic Graph) — The unit of work a client submits: a set of vertices connected by edges, described by org.apache.tez.dag.api.DAG on the client and executed by DAGImpl in the AM. Unlike MapReduce's fixed map→reduce, a Tez DAG can be arbitrarily deep and wide. See ../deep-dives/dag-model.md.

DAGAppMaster — See AM. The concrete class is org.apache.tez.dag.app.DAGAppMaster; its own lifecycle is DAGAppMasterState (NEW, INITED, RECOVERING, IDLE, RUNNING, …). See ../deep-dives/dag-app-master.md.

DAGClient — The client-side handle (org.apache.tez.dag.api.client.DAGClient, impl DAGClientImpl) returned by TezClient.submitDAG; you poll it for DAGStatus, wait for completion, and fetch counters. See ../deep-dives/dag-client.md.

DAGScheduler — The AM component that decides ordering of task scheduling across a DAG's vertices (not container allocation). Default DAGSchedulerNaturalOrder schedules in topological order; DAGSchedulerNaturalOrderControlled throttles downstream vertices. See ../deep-dives/scheduler.md.

DataMovementEvent — The runtime event (org.apache.tez.runtime.api.events.DataMovementEvent) a producer emits to tell a specific consumer "output partition N is ready at this location". The edge's EdgeManager routes it to the right consumer task(s). The heart of how outputs find inputs. See ../deep-dives/event-routing.md.

Dispatcher — The AM's central event bus (AsyncDispatcher / AsyncDispatcherConcurrent) that delivers each event type to its registered handler on a background thread. Every state machine is driven by events pulled off the dispatcher. See ../deep-dives/event-routing.md.

E

Edge — A connection between two vertices carrying data. Its EdgeProperty combines a data-movement type (one-to-one / broadcast / scatter-gather / custom), a data-source type (persisted / ephemeral), a scheduling type, and the input/output descriptors. Client class org.apache.tez.dag.api.Edge; AM runtime routing in org.apache.tez.dag.app.dag.impl.Edge. See ../deep-dives/logical-physical.md.

EdgeManager — The pluggable class that computes routing across an edge: given a producer task and partition, which consumer task(s) and input index receive it. Built-ins: ScatterGatherEdgeManager, BroadcastEdgeManager, OneToOneEdgeManager. Custom edge managers are how auto-parallelism rewrites routing at runtime. See ../deep-dives/logical-physical.md.

Event (runtime) — A message on the AM↔task data plane: DataMovementEvent, CompositeDataMovementEvent, InputReadErrorEvent, VertexManagerEvent, InputInitializerEvent, etc. (all in org.apache.tez.runtime.api.events). Distinct from the AM's internal state-machine events. See ../deep-dives/event-routing.md.

F

Fetcher — The consumer-side component that pulls producer output over HTTP (or directly from local disk). FetcherOrderedGrouped serves the sorted shuffle path; Fetcher (unordered) serves the unsorted path. It talks to the ShuffleHandler auxiliary service on each producer node. See ../deep-dives/shuffle-sort.md.

Final merge — In the ordered output path, the last on-disk merge of all spills into a single sorted output file plus index. Can be disabled (tez.runtime.enable.final-merge.in.output=false) so spills are shuffled individually, which pipelined shuffle relies on. See ../deep-dives/shuffle-sort.md.

G

Grouping (input split grouping) — Combining many small input splits into fewer, larger grouped splits so a source vertex launches a sensible number of tasks. Done by the input initializer (e.g. MRInputAMSplitGenerator) at the AM. See ../deep-dives/tez-runtime.md.

H

HistoryEvent — A record written to the history-logging service for every lifecycle milestone; the enum HistoryEventType lists them (DAG_SUBMITTED, VERTEX_STARTED, TASK_ATTEMPT_FINISHED, …). Backends: SimpleHistory (file), ATS, and Proto. See ../deep-dives/counters-diagnostics.md.

I

IFile — Tez's on-disk key/value file format (org.apache.tez.runtime.library.common.sort.impl.IFile) used for spills, merged outputs, and shuffle payloads: length-prefixed records, optional compression, a trailing checksum via IFileOutputStream/IFileInputStream. See ../deep-dives/shuffle-sort.md.

Input — The consumer half of the IPO model: a LogicalInput that reads data routed to a task and exposes a Reader. Runtime-library inputs include OrderedGroupedKVInput (sorted shuffle) and UnorderedKVInput (unsorted). See ../deep-dives/ipo-abstractions.md.

InputInitializer — AM-side plugin (InputInitializerDescriptor) that runs before a source vertex to compute its input splits/parallelism — e.g. generating grouped MR splits. See ../deep-dives/tez-runtime.md.

IPO (Input–Processor–Output) — Tez's core task abstraction: a task is a Processor fed by zero or more Inputs and writing to zero or more Outputs. Map and reduce become just processors with particular I/O; any DAG node is expressed this way. See ../deep-dives/ipo-abstractions.md.

J

JobToken / umbilical security — The shared secret (jobTokenSecretManager) the AM uses to authenticate containers on the umbilical protocol so only its own tasks can heartbeat in. See ../deep-dives/yarn-integration.md.

K

KV (key/value) — The record shape most Tez I/O moves: sorted (OrderedKV) or unsorted (UnorderedKV) partitioned outputs and their matching inputs, configured by tez.runtime.key.class / tez.runtime.value.class. See ../deep-dives/ipo-abstractions.md.

L

Local mode — Running the AM and all tasks in a single JVM with no YARN, for debugging and unit tests; enabled by tez.local.mode=true, with an optional no-network variant. Uses LocalContainerLauncher and LocalTaskSchedulerService. See ../deep-dives/local-mode.md.

LogicalInput / LogicalOutput / LogicalIOProcessor — The framework interfaces in org.apache.tez.runtime.api that a task's inputs, outputs, and processor implement; "logical" because they describe the task's view, independent of the physical routing the edge performs. See ../deep-dives/logical-physical.md.

Logical vs physical — The DAG the user writes (vertices and edges) is logical; at runtime it expands into physical tasks and the concrete data-movement routing the EdgeManager computes. Auto-parallelism is a physical rewrite of a logical plan. See ../deep-dives/logical-physical.md.

M

MergeManager — The consumer-side memory-and-disk merge engine for sorted shuffle (org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager): it accepts fetched map outputs into memory, spills and merges them under memory pressure, and feeds a single sorted iterator to the reduce processor. See ../deep-dives/shuffle-sort.md.

MRInput / MROutput — The tez-mapreduce bridge inputs/outputs (org.apache.tez.mapreduce.input.MRInput, .output.MROutput) that let a Tez task read via a Hadoop InputFormat and write via an OutputFormat, so existing MR formats work unchanged under Tez. See ../deep-dives/tez-runtime.md.

N

Non-session mode — The default (tez.am.mode.session=false): the AM runs one DAG then exits. Simpler and cheaper for one-off/batch jobs; contrast session mode. See ../deep-dives/dag-app-master.md.

O

One-to-one edge — A data-movement type (EdgeProperty.DataMovementType.ONE_TO_ONE) where producer task i sends to consumer task i; producer and consumer therefore have equal parallelism. Routed by OneToOneEdgeManager. Used when downstream should preserve upstream partitioning. See ../deep-dives/logical-physical.md.

Output — The producer half of the IPO model: a LogicalOutput a processor writes to via a Writer. Runtime-library outputs include OrderedPartitionedKVOutput (sort + partition), UnorderedKVOutput, and UnorderedPartitionedKVOutput. See ../deep-dives/ipo-abstractions.md.

P

Partitioner — The class (tez.runtime.partitioner.class, e.g. HashPartitioner) that maps each output key to a partition number in a scatter-gather output — i.e. which downstream task will consume it. See ../deep-dives/shuffle-sort.md.

Pipelined shuffle — Sending each spill to consumers as soon as it is written instead of waiting for final merge (tez.runtime.pipelined-shuffle.enabled); overlaps producer sort with consumer fetch to cut latency. Requires final merge off and speculation off. See ../deep-dives/shuffle-sort.md.

PipelinedSorter — The default sorter (org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter): a memory-block sorter that can sort on background threads and spill progressively; alternative to the older DefaultSorter. Selected via tez.runtime.sorter.class (PIPELINED vs LEGACY). See ../deep-dives/shuffle-sort.md.

Prewarm — Launching a PreWarmVertex of no-op tasks in a session before the real DAG so containers are already allocated and hot, cutting first-DAG latency. See ../deep-dives/container-reuse.md.

Processor — The compute of a task: a LogicalIOProcessor whose run method reads its inputs and writes its outputs. SimpleProcessor is the common base; tez-mapreduce provides map/reduce processors. See ../deep-dives/ipo-abstractions.md.

R

Recovery — Reconstructing DAG progress after an AM crash by replaying the recovery log written by RecoveryService and parsed by RecoveryParser, so a new AM attempt resumes rather than restarts. Enabled by tez.dag.recovery.enabled (default true); bounded by tez.am.max.app.attempts. See ../deep-dives/failure-handling.md.

Rerun (vertex/task rerun) — Re-executing an already-succeeded producer because a consumer reported its output unreadable (fetch failure). The AM sends V_TASK_RESCHEDULED / attempt-rerun events to regenerate the lost output. See ../deep-dives/failure-handling.md.

S

Scatter-gather edge — The classic shuffle edge (EdgeProperty.DataMovementType.SCATTER_GATHER): each producer partitions its output and each consumer gathers one partition from every producer — the generalized map→reduce shuffle. Routed by ScatterGatherEdgeManager. See ../deep-dives/shuffle-sort.md.

Session mode — tez.am.mode.session=true: one long-lived AM accepts multiple DAGs in succession, reusing containers between them. Ideal for interactive engines like Hive, where query latency dominates. Contrast non-session mode. See ../deep-dives/dag-app-master.md.

ShuffleHandler — The YARN NodeManager auxiliary service (in tez-aux-services, org.apache.tez.auxservices.ShuffleHandler) that serves producer output files to consumer fetchers over HTTP; identified by the tez.am.shuffle.auxiliary-service.id. See ../deep-dives/shuffle-sort.md.

ShuffleManager — The unsorted-shuffle consumer coordinator (...common.shuffle.impl.ShuffleManager): schedules fetchers, tracks completed inputs, and hands fetched data to an UnorderedKVInput. The ordered path's counterpart is ShuffleScheduler + MergeManager. See ../deep-dives/shuffle-sort.md.

ShuffleVertexManager — The VertexManagerPlugin that implements auto-parallelism and slow-start for scatter-gather vertices by consuming upstream VertexManagerEvent stats and deciding when and at what parallelism the vertex starts. See ../deep-dives/scheduler.md.

Slice — Informal term for one partition's worth of an output — the unit a single consumer task gathers across all producers in a scatter-gather edge. See ../deep-dives/logical-physical.md.

Slow-start — Delaying a consumer vertex's task launch until a fraction of its producer tasks have completed, so parallelism and locality decisions use real data. Governed by ShuffleVertexManager min/max source-fraction settings. See ../deep-dives/scheduler.md.

Speculation — Launching a second, concurrent attempt of a task that is running much slower than its peers, taking whichever finishes first; enabled by tez.am.speculation.enabled (off by default), driven by an estimator + speculator. See ../deep-dives/failure-handling.md.

Spill — Flushing an in-memory sort buffer to an on-disk IFile when it fills past tez.runtime.sort.spill.percent; a sorted output is the merge of one or more spills. See ../deep-dives/shuffle-sort.md.

Staging directory — The HDFS scratch dir (tez.staging-dir, default under /tmp/<user>/tez/staging) where the client uploads the DAG plan, configs, and localized resources for the AM to read. See ../deep-dives/tez-client.md.

State machine — The Hadoop StateMachineFactory-based finite-state machines that drive every AM entity: DAGImpl, VertexImpl, TaskImpl, TaskAttemptImpl, AMContainerImpl, AMNodeImpl. Events cause transitions and fire transition handlers. See ../deep-dives/state-machines.md.

T

Task — A logical unit of work within a vertex, identified by TezTaskID, modeled by TaskImpl with TaskStateInternal. A task owns one or more attempts; it succeeds when an attempt succeeds. See ../deep-dives/task-lifecycle.md.

TaskCommunicator — The AM-side plugin (org.apache.tez.serviceplugins.api.TaskCommunicator, default TezTaskCommunicatorImpl) that owns the umbilical RPC server tasks heartbeat into; managed by TaskCommunicatorManager. Pluggable so external launchers can supply their own. See ../deep-dives/task-attempt-lifecycle.md.

tez.lib.uris — The config key naming the HDFS location of the Tez binary tarball the AM and every container localize to get the framework on their classpath — the single most important deployment setting. See config reference and ../deep-dives/tez-client.md.

TezChild — The container-side entry point (org.apache.tez.runtime.task.TezChild) that starts in each task container, heartbeats the AM over the umbilical, and runs assigned attempts via TezTaskRunner2 / LogicalIOProcessorRuntimeTask. See ../deep-dives/tez-runtime.md.

TezClient — The client API (org.apache.tez.client.TezClient) that starts a session or submits a DAG: it stages resources, launches (or reuses) the AM, and returns a DAGClient. See ../deep-dives/tez-client.md.

TezCounters — The counter tree (org.apache.tez.common.counters.TezCounters) aggregated up attempt → task → vertex → DAG; built-in groups include TaskCounter, DAGCounter, and file-system counters. Bounded by tez.counters.max and related limits to protect AM heap. See ../deep-dives/counters-diagnostics.md.

U

Uber / inline mode — Running tasks inside the AM process rather than separate containers (tez.am.inline.task.execution.enabled), a debugging/tiny-job mode analogous to MapReduce uber mode; distinct from full local mode. See ../deep-dives/local-mode.md.

Umbilical — The RPC protocol (TezTaskUmbilicalProtocol) over which a container's tasks report status, heartbeat, fetch new attempts, and pull events from the AM. The task's lifeline; a missed heartbeat past tez.task.timeout-ms marks the attempt lost. See ../deep-dives/task-attempt-lifecycle.md.

Unordered output/input — The unsorted data path (UnorderedKVOutput / UnorderedKVInput, ShuffleManager) that moves records without sorting — cheaper when the consumer doesn't need sorted keys (e.g. a broadcast join build side). See ../deep-dives/shuffle-sort.md.

V

Vertex — A stage of the DAG: a set of tasks running the same processor with the same I/O, identified by TezVertexID, modeled by VertexImpl with VertexState. Edges connect vertices. See ../deep-dives/vertex-lifecycle.md.

VertexGroup — A named union of vertices treated as one logical output source, so a GroupInputEdge can fan several producers into one consumer input and commit them together (VertexGroupCommit). See ../deep-dives/dag-model.md.

VertexManager (VM plugin) — The pluggable per-vertex controller (VertexManagerPlugin) that decides when tasks start and can rewrite parallelism at runtime via its VertexManagerPluginContext. Built-ins: ShuffleVertexManager, RootInputVertexManager, InputReadyVertexManager, ImmediateStartVertexManager. The extension point behind slow-start and auto-parallelism. See ../deep-dives/scheduler.md.

VertexManagerEvent — A runtime event (org.apache.tez.runtime.api.events.VertexManagerEvent) a producer task sends to a downstream vertex's manager, carrying payload such as per-partition output sizes — the input to auto-parallelism decisions. See ../deep-dives/event-routing.md.