Configuration Keys Reference

The settings you will actually touch — tuning, reproducing a bug, or reading someone's tez-site.xml — grouped by area, each with its constant name, its key string, its real default, its scope, and what it does. Read the chapter a section points at before you change a key in a patch: a setting is the public face of a subsystem.

Every key below was extracted from two source files. Grep them to confirm the constant, key, and default in your checkout:

# AM / DAG / task keys:
grep -n "TEZ_AM_CONTAINER_REUSE_ENABLED" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
# Runtime (shuffle/sort/fetch) keys:
grep -n "TEZ_RUNTIME_IO_SORT_MB" \
  tez-runtime-library/src/main/java/org/apache/tez/runtime/library/api/TezRuntimeConfiguration.java

The scope story: `@ConfigurationScope`

Tez annotates each public config field with @ConfigurationScope(Scope.X) (the annotation is org.apache.tez.dag.api.ConfigurationScope; the enum is org.apache.tez.dag.api.Scope). The scope answers at what level you can override the key, which is the axis that matters most when you are reasoning about Hive or another engine setting values per-query or per-vertex:

Scope	Meaning (from `Scope.java`)
`AM`	Can only be set at the AM (application) level — one value for the whole app
`DAG`	Can be set at AM or per-DAG level
`VERTEX`	Can be set at AM, DAG, or per-vertex level — the most granular
`CLIENT`	Client-side only; never reaches the AM
`TEST`	Only meaningful in test harnesses (e.g. MiniTezCluster)

Most fields also carry @ConfigurationProperty(type="…"), which the ConfigStandardDoclet (in tez-tools/tez-javadoc-tools) reads at build time to generate tez-default-template.xml — the project's canonical, generated list of every public key and default. That file is the authority; this appendix is the curated subset a maintainer reaches for.

# The generated source of truth for keys + defaults:
find . -name tez-default-template.xml

Warning: Defaults drift across versions. The values below are current on master at extraction time, not eternal — a release can change one. When the table and the generated template disagree, the template wins. @Private / @Unstable keys (noted where relevant) can change or vanish without notice.

Jump to: AM & sessions · Task & retries · Scheduling & locality · Container reuse · Speculation · Deployment & classpath · Shuffle & sort buffers · Fetch · Local mode · Recovery · Counters limits · History & UI

AM & sessions

The application master's own resources and the session lifecycle. Read ../deep-dives/dag-app-master.md.

Constant · key	Default	Scope	What it does
`TEZ_AM_SESSION_MODE` · `tez.am.mode.session`	`false`	AM	Session mode: one AM serves many DAGs in succession. Hive turns this on
`TEZ_AM_RESOURCE_MEMORY_MB` · `tez.am.resource.memory.mb`	`1024`	AM	AM container memory (MB)
`TEZ_AM_RESOURCE_CPU_VCORES` · `tez.am.resource.cpu.vcores`	`1`	AM	AM container vcores
`TEZ_AM_LAUNCH_CMD_OPTS` · `tez.am.launch.cmd-opts`	JDK-dependent GC opts	AM	AM JVM opts; leave out `-Xmx` so Tez auto-sizes the heap
`TEZ_CONTAINER_MAX_JAVA_HEAP_FRACTION` · `tez.container.max.java.heap.fraction`	`0.8`	AM	Fraction of container memory used as `-Xmx` when Tez auto-sizes
`TEZ_AM_MAX_APP_ATTEMPTS` · `tez.am.max.app.attempts`	`2`	AM	How many times YARN may relaunch a lost AM (enables recovery)
`TEZ_AM_STAGING_DIR` · `tez.staging-dir`	`/tmp/<user>/tez/staging`	AM	HDFS scratch dir for DAG plan, configs, resources
`TEZ_SESSION_AM_DAG_SUBMIT_TIMEOUT_SECS` · `tez.session.am.dag.submit.timeout.secs`	`300`	AM	Idle session shuts down after this long with no DAG submitted (negative = never)
`TEZ_SESSION_CLIENT_TIMEOUT_SECS` · `tez.session.client.timeout.secs`	`120`	AM	Client wait for the AM to come up before submitting (session mode)
`TEZ_AM_SESSION_MIN_HELD_CONTAINERS` · `tez.am.session.min.held-containers`	`0`	AM	Containers an idle session holds for the next DAG's warm start
`TEZ_AM_CLIENT_THREAD_COUNT` · `tez.am.client.am.thread-count`	`2`	AM	Threads handling client RPC
`TEZ_QUEUE_NAME` · `tez.queue.name`	(none)	AM	YARN queue the application is submitted to
`TEZ_AM_ACLS_ENABLED` · `tez.am.acls.enabled`	`true`	AM	Enforce view/modify ACLs on the AM and its DAGs

Task & retries

Task container resources, heartbeats, and the retry limits that decide when a task (and thus the DAG) fails. Read ../deep-dives/task-attempt-lifecycle.md and ../deep-dives/failure-handling.md.

Constant · key	Default	Scope	What it does
`TEZ_TASK_RESOURCE_MEMORY_MB` · `tez.task.resource.memory.mb`	`1024`	DAG	Task container memory (MB); same value across tasks aids reuse
`TEZ_TASK_RESOURCE_CPU_VCORES` · `tez.task.resource.cpu.vcores`	`1`	DAG	Task container vcores
`TEZ_AM_TASK_MAX_FAILED_ATTEMPTS` · `tez.am.task.max.failed.attempts`	`4`	VERTEX	Failed attempts (not killed) before the task — and the DAG — fails
`TEZ_AM_TASK_MAX_ATTEMPTS` · `tez.am.task.max.attempts`	`0` (disabled)	VERTEX	Hard cap on all attempts (failed + killed); 0 disables the cap
`TEZ_AM_MAX_TASK_FAILURES_PER_NODE` · `tez.am.maxtaskfailures.per.node`	`10`	AM	Task failures on a node before it is considered faulty
`TASK_HEARTBEAT_TIMEOUT_MS` · `tez.task.timeout-ms`	`300000` (5 min)	AM	No heartbeat for this long → attempt declared lost
`TASK_HEARTBEAT_TIMEOUT_CHECK_MS` · `tez.task.heartbeat.timeout.check-ms`	`30000`	AM	Interval between lost-task sweeps
`TEZ_TASK_AM_HEARTBEAT_INTERVAL_MS` · `tez.task.am.heartbeat.interval-ms.max`	`100`	AM	Max task→AM heartbeat interval
`TEZ_TASK_MAX_EVENTS_PER_HEARTBEAT` · `tez.task.max-events-per-heartbeat`	`500`	AM	Events pulled from the AM per heartbeat
`TEZ_TASK_GET_TASK_SLEEP_INTERVAL_MS_MAX` · `tez.task.get-task.sleep.interval-ms.max`	`200`	AM	Max backoff a container waits before asking for another task
`TEZ_AM_MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC` · `tez.am.max.allowed.time-sec.for-read-error`	`300`	AM	Window in which a consumer's read-errors can trigger producer rerun
`TEZ_TASK_MAX_ALLOWED_OUTPUT_FAILURES` · `tez.task.max.allowed.output.failures`	`10`	AM	Unique downstream fetch failures before the producer attempt is failed
`TEZ_VERTEX_FAILURES_MAXPERCENT` · `tez.vertex.failures.maxpercent`	`0.0`	VERTEX	Percent of a vertex's tasks allowed to fail while the vertex still succeeds

Scheduling & locality

Cross-vertex ordering, the schedulers, and how long the AM waits for local containers. Read ../deep-dives/scheduler.md.

Constant · key	Default	Scope	What it does
`TEZ_AM_DAG_SCHEDULER_CLASS` · `tez.am.dag.scheduler.class`	`...dag.impl.DAGSchedulerNaturalOrder`	DAG	Cross-vertex scheduling order
`TEZ_AM_YARN_SCHEDULER_CLASS` · `tez.am.yarn.scheduler.class`	`...rm.DagAwareYarnTaskScheduler`	AM	YARN task-scheduler implementation
`TEZ_AM_NODE_BLACKLISTING_ENABLED` · `tez.am.node-blacklisting.enabled`	`true`	AM	Stop scheduling on nodes with too many failures
`TEZ_AM_NODE_BLACKLISTING_IGNORE_THRESHOLD` · `tez.am.node-blacklisting.ignore-threshold-node-percent`	`33`	AM	Cap on % of cluster that may be blacklisted (guards mass outages)
`TEZ_AM_RM_HEARTBEAT_INTERVAL_MS_MAX` · `tez.am.am-rm.heartbeat.interval-ms.max`	`1000`	AM	Max AM→RM heartbeat interval (raise to scale)
`TEZ_AM_CONTAINERLAUNCHER_THREAD_COUNT_LIMIT` · `tez.am.containerlauncher.thread-count-limit`	`500`	AM	Threads launching containers
`TEZ_AM_PREEMPTION_PERCENTAGE` · `tez.am.preemption.percentage`	`10`	AM	% of preemptable tasks preempted per round (0 = off)
`TEZ_AM_PREEMPTION_MAX_WAIT_TIME_MS` · `tez.am.preemption.max.wait-time-ms`	`60000`	AM	Deadline before an unsatisfied request preempts, to avoid hangs
`TEZ_AM_VERTEX_MAX_TASK_CONCURRENCY` · `tez.am.vertex.max-task-concurrency`	`-1` (unlimited)	VERTEX	Cap concurrent attempts per vertex

Container reuse

Keeping containers hot between tasks — the biggest single perf lever over MapReduce. Read ../deep-dives/container-reuse.md.

Constant · key	Default	Scope	What it does
`TEZ_AM_CONTAINER_REUSE_ENABLED` · `tez.am.container.reuse.enabled`	`true`	AM	Master switch for reuse
`TEZ_AM_CONTAINER_REUSE_RACK_FALLBACK_ENABLED` · `tez.am.container.reuse.rack-fallback.enabled`	`true`	AM	Reuse a container for a rack-local (not node-local) task
`TEZ_AM_CONTAINER_REUSE_NON_LOCAL_FALLBACK_ENABLED` · `tez.am.container.reuse.non-local-fallback.enabled`	`false`	AM	Reuse for non-local tasks (can hurt locality badly)
`TEZ_AM_CONTAINER_REUSE_NEW_CONTAINERS_ENABLED` · `tez.am.container.reuse.new-containers.enabled`	`false`	AM	Hold newly-allocated-but-unassigned containers for reuse
`TEZ_AM_CONTAINER_REUSE_LOCALITY_DELAY_ALLOCATION_MILLIS` · `tez.am.container.reuse.locality.delay-allocation-millis`	`250`	AM	Delay before falling to the next locality level (delay scheduling)
`TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MIN_MILLIS` · `tez.am.container.idle.release-timeout-min.millis`	`5000`	AM	Min time an idle container is held before it may be released
`TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS` · `tez.am.container.idle.release-timeout-max.millis`	`10000`	AM	Max idle-hold time; actual expiry is randomized in this band

Speculation

Re-running stragglers. Off by default. Read ../deep-dives/failure-handling.md.

Constant · key	Default	Scope	What it does
`TEZ_AM_SPECULATION_ENABLED` · `tez.am.speculation.enabled`	`false`	VERTEX	Enable speculative re-execution of slow tasks
`TEZ_AM_SPECULATOR_CLASS` · `tez.am.speculator.class`	(built-in default)	VERTEX	Speculator implementation
`TEZ_AM_TASK_ESTIMATOR_CLASS` · `tez.am.task.estimator.class`	(built-in default)	VERTEX	Runtime estimator feeding the speculator
`TEZ_AM_PROPORTION_RUNNING_TASKS_SPECULATABLE` · `tez.am.proportion.running.tasks.speculatable`	`0.1`	VERTEX	Max fraction of running tasks speculatable at once
`TEZ_AM_PROPORTION_TOTAL_TASKS_SPECULATABLE` · `tez.am.proportion.total.tasks.speculatable`	`0.01`	VERTEX	Max fraction of all tasks speculatable at once
`TEZ_AM_MINIMUM_ALLOWED_SPECULATIVE_TASKS` · `tez.am.minimum.allowed.speculative.tasks`	`10`	VERTEX	Floor on speculatable task count
`TEZ_AM_LEGACY_SPECULATIVE_SINGLE_TASK_VERTEX_TIMEOUT` · `tez.am.legacy.speculative.single.task.vertex.timeout`	`-1`	AM	Timeout after which a single-task vertex is speculated (neg = off)

Deployment & classpath

Where the framework and libraries come from — the settings that make or break a deployment. Read ../deep-dives/tez-client.md.

Constant · key	Default	Scope	What it does
`TEZ_LIB_URIS` · `tez.lib.uris`	(none)	AM	HDFS path(s) to the Tez tarball localized for AM + containers. The key deployment setting
`TEZ_LIB_URIS_CLASSPATH` · `tez.lib.uris.classpath`	(none)	AM	Relative classpath inside the `tez.lib.uris` archives
`TEZ_AUX_URIS` · `tez.aux.uris`	(none)	AM	Extra resources localized into AM/container working dirs
`TEZ_USE_CLUSTER_HADOOP_LIBS` · `tez.use.cluster.hadoop-libs`	`false`	AM	Use the cluster's Hadoop jars instead of bundling them in `tez.lib.uris`
`TEZ_IGNORE_LIB_URIS` · `tez.ignore.lib.uris`	(none)	AM	Ignore `tez.lib.uris` (dev/debug when classpath comes via `LocalResource`s)
`TEZ_USER_CLASSPATH_FIRST` · `tez.user.classpath.first`	`true`	CLIENT	Put user classpath before the framework classpath
`TEZ_CLUSTER_ADDITIONAL_CLASSPATH_PREFIX` · `tez.cluster.additional.classpath.prefix`	(none)	AM	Extra classpath entries for AM + containers
`TEZ_AM_SHUFFLE_AUXILIARY_SERVICE_ID` · `tez.am.shuffle.auxiliary-service.id`	`tez_shuffle`	AM	Name of the NodeManager shuffle aux-service to fetch from

Warning: tez.lib.uris is the single most common source of "AM won't start" and ClassNotFoundException failures. If it points at a tarball whose internal layout doesn't match tez.lib.uris.classpath, containers come up with a broken classpath. Verify the tarball exists in HDFS and its structure before chasing anything subtler.

Shuffle & sort buffers

Producer-side sort and the sorter choice. Runtime keys; scope is effectively per-vertex via the runtime config. Read ../deep-dives/shuffle-sort.md.

Constant · key	Default	What it does
`TEZ_RUNTIME_IO_SORT_MB` · `tez.runtime.io.sort.mb`	`100`	Sort buffer size (MB) per output; the primary sort-memory knob
`TEZ_RUNTIME_SORT_SPILL_PERCENT` · `tez.runtime.sort.spill.percent`	`0.8`	Buffer fullness that triggers a spill to disk
`TEZ_RUNTIME_IO_SORT_FACTOR` · `tez.runtime.io.sort.factor`	`100`	Streams merged at once during sort/merge
`TEZ_RUNTIME_SORTER_CLASS` · `tez.runtime.sorter.class`	`PIPELINED`	Sorter: `PIPELINED` (default) or `LEGACY` (DefaultSorter)
`TEZ_RUNTIME_PIPELINED_SORTER_SORT_THREADS` · `tez.runtime.pipelined.sorter.sort.threads`	`2`	Background sort threads for the pipelined sorter
`TEZ_RUNTIME_PIPELINED_SORTER_MIN_BLOCK_SIZE_IN_MB` · `tez.runtime.pipelined.sorter.min-block.size.in.mb`	`2000`	Chunk size the pipelined sorter allocates `io.sort.mb` in
`TEZ_RUNTIME_PIPELINED_SORTER_LAZY_ALLOCATE_MEMORY` · `tez.runtime.pipelined.sorter.lazy-allocate.memory`	`false`	Grow sort memory on demand vs allocate all upfront
`TEZ_RUNTIME_COMBINE_MIN_SPILLS` · `tez.runtime.combine.min.spills`	`3`	Spills that must exist before the combiner runs on merge
`TEZ_RUNTIME_INDEX_CACHE_MEMORY_LIMIT_BYTES` · `tez.runtime.index.cache.memory.limit.bytes`	`1048576`	Memory for caching spill index records
`TEZ_RUNTIME_UNORDERED_OUTPUT_BUFFER_SIZE_MB` · `tez.runtime.unordered.output.buffer.size-mb`	`100`	Buffer for unsorted output before spilling
`TEZ_RUNTIME_COMPRESS` · `tez.runtime.compress`	(unset)	Compress intermediate data
`TEZ_RUNTIME_COMPRESS_CODEC` · `tez.runtime.compress.codec`	(unset)	Codec class for intermediate compression
`TEZ_RUNTIME_PIPELINED_SHUFFLE_ENABLED` · `tez.runtime.pipelined-shuffle.enabled`	`false`	Ship each spill as it's written; needs final-merge off + speculation off
`TEZ_RUNTIME_ENABLE_FINAL_MERGE_IN_OUTPUT` · `tez.runtime.enable.final-merge.in.output`	`true`	Merge all spills into one sorted output; disable for pipelined shuffle

Fetch & transfer

Consumer-side fetch parallelism, buffers, and failure thresholds. Read ../deep-dives/shuffle-sort.md.

Constant · key	Default	What it does
`TEZ_RUNTIME_SHUFFLE_PARALLEL_COPIES` · `tez.runtime.shuffle.parallel.copies`	`20`	Concurrent fetchers per consumer
`TEZ_RUNTIME_SHUFFLE_FETCH_FAILURES_LIMIT` · `tez.runtime.shuffle.fetch.failures.limit`	`5`	Fetch failures for one source before reporting it
`TEZ_RUNTIME_SHUFFLE_FETCH_MAX_TASK_OUTPUT_AT_ONCE` · `tez.runtime.shuffle.fetch.max.task.output.at.once`	`20`	Max outputs fetched from one host per connection
`TEZ_RUNTIME_SHUFFLE_FETCH_BUFFER_PERCENT` · `tez.runtime.shuffle.fetch.buffer.percent`	`0.90`	Fraction of task memory for the fetch buffer
`TEZ_RUNTIME_SHUFFLE_MEMORY_LIMIT_PERCENT` · `tez.runtime.shuffle.memory.limit.percent`	`0.25`	Max fraction of shuffle memory a single fetched output may use
`TEZ_RUNTIME_SHUFFLE_MERGE_PERCENT` · `tez.runtime.shuffle.merge.percent`	`0.90`	Shuffle-memory fullness that triggers a memory-to-disk merge
`TEZ_RUNTIME_SHUFFLE_CONNECT_TIMEOUT` · `tez.runtime.shuffle.connect.timeout`	`180000`	Fetch connect timeout (ms)
`TEZ_RUNTIME_SHUFFLE_READ_TIMEOUT` · `tez.runtime.shuffle.read.timeout`	`180000`	Fetch read timeout (ms)
`TEZ_RUNTIME_SHUFFLE_KEEP_ALIVE_ENABLED` · `tez.runtime.shuffle.keep-alive.enabled`	`false`	HTTP keep-alive to the shuffle handler
`TEZ_RUNTIME_SHUFFLE_HOST_PENALTY_TIME_LIMIT_MS` · `tez.runtime.shuffle.host.penalty.time.limit`	`600000`	Max backoff before retrying a penalized host
`TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH` · `tez.runtime.optimize.local.fetch`	`true`	Read local outputs directly, bypassing HTTP
`TEZ_RUNTIME_EMPTY_PARTITION_INFO_VIA_EVENTS_ENABLED` · `tez.runtime.empty.partitions.info-via-events.enabled`	`true`	Signal empty partitions via events so no fetch is attempted

Local mode

Running everything in one JVM for debugging. Read ../deep-dives/local-mode.md.

Constant · key	Default	Scope	What it does
`TEZ_LOCAL_MODE` · `tez.local.mode`	`false`	AM	Run AM and tasks in a single process, no YARN
`TEZ_LOCAL_MODE_WITHOUT_NETWORK` · `tez.local.mode.without.network`	`false`	AM	Skip RPC; `LocalClient` calls the AM directly (only with local mode on)
`TEZ_AM_INLINE_TASK_EXECUTION_ENABLED` · `tez.am.inline.task.execution.enabled`	`false`	AM	Run tasks inside the AM process (uber-style; `@Private`)
`TEZ_AM_INLINE_TASK_EXECUTION_MAX_TASKS` · `tez.am.inline.task.execution.max-tasks`	`1`	AM	Parallel tasks in the AM process when inline execution is on

Recovery

Surviving an AM restart. Read ../deep-dives/failure-handling.md.

Constant · key	Default	Scope	What it does
`DAG_RECOVERY_ENABLED` · `tez.dag.recovery.enabled`	`true`	AM	Recover in-flight DAGs on a new AM attempt
`TEZ_AM_FAILURE_ON_MISSING_RECOVERY_DATA` · `tez.am.failure.on.missing.recovery.data`	`false`	AM	Fail if recovery is on but nothing was found to recover
`DAG_RECOVERY_MAX_UNFLUSHED_EVENTS` · `tez.dag.recovery.max.unflushed.events`	`100`	AM	Recovery events buffered before a forced flush
`DAG_RECOVERY_FLUSH_INTERVAL_SECS` · `tez.dag.recovery.flush.interval.secs`	`30`	AM	Interval between recovery-log flushes
`DAG_RECOVERY_FILE_IO_BUFFER_SIZE` · `tez.dag.recovery.io.buffer.size`	`8192`	AM	IO buffer (bytes) for the recovery file

Counters limits

Bounds that stop a runaway counter set from OOMing the AM. All @Unstable, AM-scope. Read ../deep-dives/counters-diagnostics.md.

Constant · key	Default	What it does
`TEZ_COUNTERS_MAX` · `tez.counters.max`	`1200`	Max counters per DAG
`TEZ_COUNTERS_MAX_GROUPS` · `tez.counters.max.groups`	`500`	Max counter groups per DAG
`TEZ_COUNTERS_COUNTER_NAME_MAX_LENGTH` · `tez.counters.counter-name.max-length`	`64`	Max counter-name length
`TEZ_COUNTERS_GROUP_NAME_MAX_LENGTH` · `tez.counters.group-name.max-length`	`256`	Max group-name length

Note: The older tez.am.counters.* keys are deprecated aliases of these, registered via Configuration.addDeprecation in TezConfiguration's static initializer. Grep: grep -n "addDeprecation" TezConfiguration.java.

History & UI

Where lifecycle history is logged and how the UI finds it. Read ../deep-dives/counters-diagnostics.md.

Constant · key	Default	Scope	What it does
`TEZ_HISTORY_LOGGING_SERVICE_CLASS` · `tez.history.logging.service.class`	`...impl.SimpleHistoryLoggingService`	AM	Which history backend to use (Simple / ATS / Proto / DevNull)
`TEZ_HISTORY_LOGGING_LOGLEVEL` · `tez.history.logging.log.level`	(HistoryLogLevel default)	DAG	Limit which event types are logged (AM / DAG / VERTEX / TASK / ALL)
`TEZ_SIMPLE_HISTORY_LOGGING_DIR` · `tez.simple.history.logging.dir`	container log dir	AM	Output dir for `SimpleHistoryLoggingService`
`TEZ_HISTORY_LOGGING_PROTO_BASE_DIR` · `tez.history.logging.proto-base-dir`	(none)	AM	Base dir for `ProtoHistoryLoggingService`; unset disables it
`TEZ_HISTORY_LOGGING_PROTO_QUEUE_SIZE` · `tez.history.logging.queue.size`	`100000`	AM	Max queue for the proto history logger
`YARN_ATS_EVENT_FLUSH_TIMEOUT_MILLIS` · `tez.yarn.ats.event.flush.timeout.millis`	`-1`	AM	Flush timeout for ATS on shutdown (-1 = compute)
`TEZ_GENERATE_DEBUG_ARTIFACTS` · `tez.generate.debug.artifacts`	`false`	DAG	Emit the DAG plan in text for debugging

Note: Hive typically configures ProtoHistoryLoggingService and reads it back with tez-history-parser, rather than relying on ATS. See the key classes map for the plugin classes.

Open-Source Engineer & Contributor