Tez vs Spark vs MapReduce

An honest engineering comparison for contributors. Tez did not appear in a vacuum: it was designed to fix specific, concrete limitations of MapReduce while making a different set of trade-offs than Spark. Understanding those trade-offs is not trivia — it explains why the Tez code is shaped the way it is, why Hive chose it, and which of your Tez instincts transfer to other engines and which don't.

No marketing here. Each system wins some axes and loses others; the goal is to know which, and why.

Note: This compares the three execution engines as frameworks. "Spark" here means Spark Core's execution engine (RDD/stage DAG + executors), not the whole ecosystem (SQL, MLlib, Structured Streaming), which is a different and much larger comparison.

The one-paragraph version

MapReduce runs a fixed two-stage map → shuffle → reduce pipeline, launching a fresh JVM per task and materializing every intermediate to HDFS. Tez generalizes that into an arbitrary DAG of Input–Processor–Output tasks, runs one long-lived ApplicationMaster per app, reuses containers across tasks, and can rewrite the plan at runtime — but keeps MapReduce's "no long-lived caching layer, YARN-native, disk-based shuffle" philosophy. Spark also runs a DAG, but on a fundamentally different substrate: long-lived executors holding an in-memory block cache (RDDs), a driver-centric scheduler, and its own cluster-manager abstraction. Tez sits between MapReduce and Spark: much more flexible than MR, much more YARN-integrated and memory-frugal than Spark.

Execution model

MapReduce gives you exactly two user functions and one shuffle between them. Anything more complex — a join of three tables, a multi-stage aggregation — is expressed as a chain of MapReduce jobs, each writing its output to HDFS and the next reading it back. That HDFS round-trip between stages is the defining cost: for an N-stage pipeline you pay N full materializations, with replication.

Tez replaces the two fixed stages with a DAG. A vertex is a stage; an edge carries data between stages; a task within a vertex is an Input–Processor–Output triple (org.apache.tez.runtime.api.LogicalIOProcessor fed by LogicalInputs, writing LogicalOutputs). Map and reduce become special cases: a "map" is a processor with an MRInput and a partitioned output; a "reduce" is a processor with a sorted-shuffle input. The three-way join that was three MR jobs becomes one Tez DAG with intermediate data staying on local disk between vertices — no HDFS round-trip, no re-read. This is the single largest source of Tez's speedup over MapReduce for Hive workloads.

Spark also builds a DAG, but the abstraction is the RDD (a partitioned, lineage-tracked dataset) and the scheduler cuts the DAG into stages at shuffle boundaries. The crucial difference from Tez is the RDD cache: a dataset can be persist()ed in executor memory and reused across many downstream operations and across jobs. That is what makes Spark strong for iterative algorithms (ML, graph) and interactive exploration — the data stays hot. Tez has no equivalent; each DAG is independent and intermediate data is not a first-class reusable object.

The engineering consequence: Tez's model is stateless between DAGs and its code reflects that — the DAGImpl state machine is born and dies with one DAG, and there is no block manager, no cache eviction policy, no lineage recomputation. That is less code and less memory, at the cost of no reuse.

Scheduling

MapReduce scheduling is slot-based and coarse: the JobTracker/AM asks YARN for containers, runs one task per container, and tears the container down when the task ends. Every task pays JVM startup.

Tez runs one DAGAppMaster per application that owns the whole DAG lifecycle and does its own two-level scheduling. The DAGScheduler (DAGSchedulerNaturalOrder) decides the order vertices are scheduled; the task scheduler (DagAwareYarnTaskScheduler) negotiates containers from YARN and — the key move — reuses them. When a task finishes, its container is not returned to YARN; it goes IDLE (AMContainerImpl state machine) and the scheduler assigns it the next matching task, honoring locality with a configurable delay (tez.am.container.reuse.locality.delay-allocation-millis) before falling back to rack- or non-local reuse. In a Hive session, the same containers serve query after query, so steady-state latency excludes JVM launch entirely. Container reuse is arguably Tez's second-biggest win over MapReduce after DAG execution.

Spark takes reuse further: executors are long-lived from application start and run many tasks concurrently as threads, with the driver's DAGScheduler and TaskScheduler dispatching tasks to them. There is no per-task container negotiation at all in steady state. This is lower-overhead than Tez's per-container YARN model, but it also means Spark holds a fixed executor footprint whether or not it is doing work (mitigated by dynamic allocation), whereas Tez's container set expands and contracts with the DAG and can be released back to YARN between DAGs. In a busy multi-tenant YARN cluster, Tez's willingness to give resources back is a real operational advantage.

Runtime replanning

This is where Tez is genuinely distinctive at the engine level. Tez vertices are controlled by pluggable VertexManagerPlugins that can change the plan while it runs:

• Auto-parallelism: ShuffleVertexManager collects VertexManagerEvent partition-size statistics from upstream tasks and, before a scatter-gather vertex starts, reduces its task count by merging small partitions — installing a custom EdgeManager to reroute accordingly. You get right-sized reducer counts without the user guessing.
• Slow-start: the same manager delays a consumer vertex's launch until a fraction of its producers finish, so parallelism and locality decisions use real data volume.

In stock MapReduce, reducer count is fixed at submit time and never changes. Spark's equivalent — Adaptive Query Execution — coalesces shuffle partitions and switches join strategies at runtime, but it lives in the Spark SQL layer, not Spark Core; the RDD engine itself does not replan. Tez put dynamic reconfiguration in the engine, reusable by any DAG builder (Hive, Pig, custom). That generality is why Hive can lean on it so heavily.

Shuffle architecture

All three shuffle by writing partitioned, optionally-sorted producer output and having consumers fetch the partition they own — but the plumbing differs.

MapReduce: map output is sorted and spilled to local disk as index+data files; reducers fetch over HTTP from the TaskTracker/NodeManager shuffle service; the reduce side merges fetched segments. Well-understood, disk-heavy, no pipelining.

Tez inherits and generalizes this. The producer side uses a pluggable sorter — the default PipelinedSorter (block-based, multi-threaded, progressive spill) or the legacy DefaultSorter — writing the IFile format (length-prefixed, optionally compressed, checksummed). The consumer side has two parallel stacks: ordered (ShuffleScheduler + MergeManager + FetcherOrderedGrouped) for sorted shuffle, and unordered (ShuffleManager + Fetcher) for when the consumer doesn't need sorted keys — a broadcast join build side, say. Data is served by the same NodeManager auxiliary service (org.apache.tez.auxservices.ShuffleHandler) MapReduce uses. Tez adds pipelined shuffle (tez.runtime.pipelined-shuffle.enabled): ship each spill as it is written instead of waiting for a final merge, overlapping producer sort with consumer fetch. It also signals empty partitions via events so consumers skip fetches that would return nothing. These are targeted latency optimizations MR never had.

Spark's shuffle writes map output to local disk too (sort-based shuffle writer), but the read side pulls blocks into executor memory managed by the block manager, and — critically — the shuffle output can feed a cached RDD reused downstream. Spark's shuffle is comparable to Tez's in mechanism but integrated with its memory/cache model rather than a standalone disk-based transfer.

The contributor takeaway: Tez's shuffle code (the orderedgrouped and impl packages in tez-runtime-library) is the most performance-sensitive, most frequently-patched part of the codebase, and it is recognizably MapReduce shuffle with better memory management and pipelining — not a from-scratch redesign.

Memory model

MapReduce has essentially no cross-task memory model: each task JVM is independent and short-lived, sort buffers (io.sort.mb) are per-task, and nothing persists.

Tez is frugal by design. Task memory is negotiated per-task (tez.task.resource.memory.mb), sort memory is tez.runtime.io.sort.mb, and Tez adds a weighted memory distributor that divides a task's heap across its inputs, outputs, and processor according to their declared needs (the tez.task.scale.memory.* keys). There is no long-lived cache to manage, no unified execution/storage memory pool — the AM holds control-plane state (counters bounded by tez.counters.max, container/node models) and tasks hold only what they need to sort and shuffle. This keeps AM heap small even for large DAGs, which matters because one AM must survive the whole application.

Spark's memory model is far more elaborate because it must be: the unified memory manager splits executor heap between execution memory (shuffle/sort/join) and storage memory (the RDD cache), with dynamic borrowing between them, plus off-heap options and spill. This complexity is the price of the cache. If you have worked on Spark's MemoryManager, Tez's memory code will feel spartan — because Tez deliberately declined the feature that forces that complexity.

Where Tez wins

• Hive integration and dynamic reconfiguration. VM plugins give Hive runtime auto-parallelism and dynamic join strategy support at the engine level. Hive-on-Tez is the reference deployment; the two co-evolved. See ../deep-dives/hive-integration.md.
• YARN-native multi-tenancy. Tez expands and contracts its container footprint per DAG and returns resources to YARN between DAGs, playing well in a shared cluster. It has no separate cluster-manager abstraction to run.
• Memory frugality. No cache means a small, predictable AM and task footprint — you can run many concurrent Tez apps where the same nodes would hold far fewer Spark executors.
• Pluggability. Nearly everything is a service plugin: task communicators, container launchers, task schedulers, vertex managers, edge managers, history-logging services. A contributor can extend behavior without forking the engine.
• MapReduce compatibility. tez-mapreduce runs existing Hadoop InputFormat/OutputFormat/committer code unchanged, so migrating an MR pipeline to a Tez DAG is incremental.

Where Tez doesn't

• No caching / iterative-algorithm story. Nothing like RDD.persist(). For ML or graph workloads that reread the same dataset dozens of times, Spark's cache is a structural advantage Tez cannot match.
• No interactive shell. Tez is a library you build a DAG against and submit; there is no tez-shell, no REPL, no notebook integration. Its interactivity is entirely mediated by an engine on top (Hive/LLAP).
• Smaller ecosystem and mindshare. Spark has orders of magnitude more users, connectors, docs, and Stack Overflow answers. Tez's user base is essentially "Hive and Pig", which means fewer eyes, slower feature velocity, and a steeper learning curve from sparse third-party material — part of why this curriculum exists.
• Lower-level API. Building a DAG directly with org.apache.tez.dag.api.DAG is far more verbose than a Spark DataFrame program. Tez expects an engine, not a human, to be its client.
• No built-in SQL / DataFrame layer. Spark ships one; Tez relies on Hive to provide it.

Why Hive chose Tez

Hive's workload is SQL over HDFS/S3: multi-stage joins and aggregations, batch and interactive, on shared YARN clusters. Every axis Tez wins on is one Hive needs:

• Multi-stage query plans map directly onto a DAG, eliminating the inter-stage HDFS materializations that made Hive-on-MapReduce slow.
• Runtime auto-parallelism and slow-start let Hive avoid guessing reducer counts — the planner emits a DAG and the engine right-sizes it from real data.
• Session mode + container reuse give interactive-query latency: a Hive session holds an AM and warm containers across queries.
• YARN-native resource behavior suits the multi-tenant clusters Hive runs on.
• Hive did not need a caching layer or a general-purpose API; it needed a fast, extensible, YARN-friendly DAG engine it could drive programmatically. That is precisely Tez's design center.

Spark could also back Hive (and does, via Hive-on-Spark), but Tez was built with Hive's requirements as its target, and the VM-plugin extension points exist largely to serve them.

What a Tez contributor learns that transfers

Even if you never touch Tez again, the concepts port widely:

• DAG execution and stage boundaries — the mental model behind Spark, Flink, Presto, and every modern distributed query engine.
• Event-driven state machines for distributed lifecycle management — the same pattern runs YARN itself and many control planes; see ../deep-dives/state-machines.md.
• Shuffle internals — sort/spill/merge, partitioning, fetch, and the memory-vs-disk trade-offs are near-universal across data engines; the specifics in ../deep-dives/shuffle-sort.md generalize.
• Runtime adaptivity — auto-parallelism is conceptually what Spark AQE and many cost-based re-optimizers do; you will recognize it everywhere.
• YARN application authorship — writing an ApplicationMaster, negotiating containers, and surviving restarts via recovery is transferable to any framework that runs on YARN.

Note: The comparisons above are architectural, not benchmarks. Relative performance depends entirely on workload, data size, cluster shape, and tuning. Use this to reason about design trade-offs and why the code is shaped this way — not to declare a winner. For the mechanisms behind every Tez claim here, follow the deep-dive links; every one is grounded in a class you can grep.