Local Mode

Tez ships two "no cluster" execution paths, and confusing them is one of the most common ways contributors ship a "passes locally, breaks on the cluster" patch:

• Local mode — tez.local.mode=true. The DAGAppMaster, the scheduler, the container launcher, and every TezChild run as threads inside the calling JVM. No ResourceManager, no NodeManager, and — with tez.local.mode.without.network=true — no RPC at all.
• MiniTezCluster — a real Hadoop MiniYARNCluster (RM + NMs as threads) running a real Tez AM submitted as a normal YARN application, with tasks forked into separate JVMs and a real ShuffleHandler auxiliary service.

Both let you run a DAG without a real cluster, but they exercise wildly different amounts of the stack. After this chapter you can: trace the exact code that switches TezClient to LocalClient; explain what the AM rewires when isLocal is true; say precisely which layers are faked, which are real, and which are skipped; predict the class of bug local mode will hide from you; and drive a Tez example under a debugger with breakpoints in the AM and the runtime hitting in the same JVM. This page is a sibling of tez-client.md, dag-app-master.md, shuffle-sort.md, and testing-framework.md; read it against those.

Note: Every class, config key, and default below is quoted from Apache Tez master. Verify with the rg/grep commands that open each section — run them yourself against your checkout at /Users/s0x/src/oss-repos/tez. Code moves between branches; line numbers rot, so nothing here is cited by line.

The cast

rg -n "TEZ_LOCAL_MODE|tez.local.mode" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java

1. Entry path: how TezClient switches to LocalClient

rg -n "isLocal|createFrameworkClient|LocalClient" \
  tez-api/src/main/java/org/apache/tez/client/FrameworkClient.java

TezClient never talks to YARN directly. It delegates all framework operations to a FrameworkClient, and the selection happens in one static factory. This is the single branch that decides your entire execution model:

// tez-api  org.apache.tez.client.FrameworkClient
public static FrameworkClient createFrameworkClient(TezConfiguration tezConf) {
  boolean isLocal = tezConf.getBoolean(
      TezConfiguration.TEZ_LOCAL_MODE, TezConfiguration.TEZ_LOCAL_MODE_DEFAULT);
  if (isLocal) {
    try {
      return ReflectionUtils.createClazzInstance("org.apache.tez.client.LocalClient");
    } catch (TezReflectionException e) {
      throw new TezUncheckedException("Fail to create LocalClient", e);
    }
  } else {
    // ... YarnClientFrameworkService.newFrameworkClient()
  }
}

Two things worth internalizing. First, LocalClient lives in tez-dag, not tez-api — it is loaded reflectively so tez-api need not depend on the AM code. Second, TezClient.createFrameworkClient() simply forwards amConfig.getTezConfiguration() to this factory, so the boolean you set on the TezConfiguration you hand to TezClient.create(...) is the only thing that matters.

LocalClient is a full FrameworkClient that impersonates YARN. Instead of submitting an application, it constructs and starts a DAGAppMaster on a thread in your JVM:

// tez-dag  org.apache.tez.client.LocalClient
protected void startDAGAppMaster(final ApplicationSubmissionContext appContext) {
  if (dagAmThread == null) {
    dagAmThread = createDAGAppMaster(appContext);
    dagAmThread.start();
    // spin until dagAppMaster leaves NEW/INITED, else time out
  }
}

createDAGAppMaster returns a Thread named "DAGAppMaster Thread" whose run() body copies the staging directory to a local working dir, builds an ApplicationAttemptId/ContainerId by hand, and calls DAGAppMaster.initAndStartAppMaster(dagAppMaster, conf). submitApplication is reduced to: start that thread, return the fabricated ApplicationId.

flowchart TD
    A["TezClient.create(name, tezConf)"] --> B["TezClient.createFrameworkClient()"]
    B --> C{"tezConf.getBoolean(TEZ_LOCAL_MODE)"}
    C -->|false| Y["YarnClientFrameworkService: submit YARN app"]
    C -->|true| L["ReflectionUtils.createClazzInstance('LocalClient')"]
    L --> S["LocalClient.submitApplication: startDAGAppMaster"]
    S --> T["new Thread('DAGAppMaster Thread')"]
    T --> M["DAGAppMaster.initAndStartAppMaster(...)"]
    M --> H["new DAGClientHandler(dagAppMaster)"]

Tip: LocalClient.init() also forces the DAG scheduler to org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled (via TEZ_AM_DAG_SCHEDULER_CLASS) and disables the AM web service (TEZ_AM_WEBSERVICE_ENABLE=false). If you are debugging scheduling order, know that local mode is not running the default scheduler.

See tez-client.md for the full TezClient state machine and dag-app-master.md for what initAndStartAppMaster sets up.

2. Config keys: tez.local.mode and tez.local.mode.without.network

rg -n "TEZ_LOCAL_MODE\b|TEZ_LOCAL_MODE_WITHOUT_NETWORK|_DEFAULT" \
  tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java | head

The distinction is subtle and important. Plain tez.local.mode=true still stands up the client-facing RPC server inside the AM, and DAGClient talks to it over loopback RPC — the AM and tasks share the JVM, but the client status/kill path is still real RPC. Turning on tez.local.mode.without.network=true removes even that:

rg -rn "isLocalWithoutNetwork|TEZ_LOCAL_MODE_WITHOUT_NETWORK" \
  tez-dag/src/main/java tez-api/src/main/java

// tez-dag  org.apache.tez.client.LocalClient
this.isLocalWithoutNetwork = tezConf.getBoolean(
    TezConfiguration.TEZ_LOCAL_MODE_WITHOUT_NETWORK,
    TezConfiguration.TEZ_LOCAL_MODE_WITHOUT_NETWORK_DEFAULT);
// ...
return isLocalWithoutNetwork
    ? new LocalDAGAppMaster(applicationAttemptId, cId, ...)
    : new DAGAppMaster(applicationAttemptId, cId, ...);

When isLocalWithoutNetwork is set, LocalClient constructs a LocalDAGAppMaster and routes getAMStatus, submitDag, and getDAGClient through an in-process DAGClientHandler (clientHandler) and a DAGClientImplLocal, bypassing RPC serialization entirely:

// tez-dag  org.apache.tez.dag.app.LocalDAGAppMaster (whole class, trimmed)
public class LocalDAGAppMaster extends DAGAppMaster {
  @Override
  protected void initClientRpcServer() {
    // nothing to do, clientRpcServer is not used by clients
  }
  public int getRpcPort() { return 0; }
}

Warning: without.network is the fastest configuration and the biggest liar. It removes the last piece of serialization in the client path, so a bug in DAGStatus/VertexStatus protobuf round-tripping (or in anything that only manifests when objects cross the RPC boundary) becomes invisible. Use it for speed; do not use it to validate client-facing behavior.

3. LocalContainerLauncher: TezChild on a thread pool

rg -n "numExecutors|newFixedThreadPool|TEZ_AM_INLINE_TASK_EXECUTION_MAX_TASKS|createSubTask|newTezChild" \
  tez-dag/src/main/java/org/apache/tez/dag/app/launcher/LocalContainerLauncher.java

In production, the ContainerLauncherManager routes to TezContainerLauncherImpl, which asks a NodeManager to start a container JVM. In local mode the AM's PluginManager selects the "uber" plugin, and ContainerLauncherManager.createUberContainerLauncher builds a LocalContainerLauncher. Its whole job is to run TezChild logic on a bounded thread pool instead of forking JVMs:

// tez-dag  org.apache.tez.dag.app.launcher.LocalContainerLauncher
numExecutors = conf.getInt(
    TezConfiguration.TEZ_AM_INLINE_TASK_EXECUTION_MAX_TASKS,
    TezConfiguration.TEZ_AM_INLINE_TASK_EXECUTION_MAX_TASKS_DEFAULT);
Preconditions.checkState(numExecutors >= 1, "Must have at least 1 executor");
ExecutorService rawExecutor = Executors.newFixedThreadPool(numExecutors,
    new ThreadFactoryBuilder().setDaemon(true)
        .setNameFormat("LocalTaskExecutionThread #%d").build());
this.taskExecutorService = MoreExecutors.listeningDecorator(rawExecutor);

The concurrency knob is tez.am.inline.task.execution.max-tasks (TEZ_AM_INLINE_TASK_EXECUTION_MAX_TASKS), default 1. That default is why local mode runs tasks single-file by default — a deliberate choice that makes runs deterministic and easy to step through, at the cost of hiding ordering bugs.

When the AM "launches a container," the launcher constructs a TezChild and submits it to the pool. Two details make it in-process:

// tez-dag  LocalContainerLauncher.createTezChild (trimmed)
long memAvailable = Runtime.getRuntime().maxMemory() / numExecutors;
TezChild tezChild = TezChild.newTezChild(defaultConf, null, 0,
    containerId.toString(), tokenIdentifier, attemptNumber, localDirs,
    workingDirectory, containerEnv, "", executionContext, credentials,
    memAvailable, context.getUser(), tezTaskUmbilicalProtocol, false,
    context.getHadoopShim());
// ...
ListenableFuture<TezChild.ContainerExecutionResult> f =
    taskExecutorService.submit(createSubTask(tezChild, event.getContainerId()));

First, memAvailable is not a real container size — it is this JVM's Runtime.maxMemory() divided by the executor count. Second, the umbilical handed to newTezChild is the AM's own umbilical object, obtained via ((TezTaskCommunicatorImpl) tal.getTaskCommunicator(taskCommId) .getTaskCommunicator()).getUmbilical(). TezChild then takes the in-process path instead of creating an RPC proxy:

// tez-runtime-internals  org.apache.tez.runtime.task.TezChild
if (umbilical == null) {
  // ... RPC.getProxy(TezTaskUmbilicalProtocol.class, ... address ...)
} else {
  this.umbilical = umbilical;   // in-process: heartbeats are method calls
  ownUmbilical = false;
}

The TezLocalTaskCommunicatorImpl on the AM side seals this: it overrides startRpcServer() to not bind a server (it sets address = InetSocketAddress(InetAddress.getLocalHost(), 0) and logs a debug line). So in local mode the "umbilical RPC" between a task and the AM is a Java method call on a shared object, and ContainerLaunchContext is never serialized.

See tez-runtime.md and task-attempt-lifecycle.md for what TezChild does once it is running, and scheduler.md for how requests reach the launcher.

4. What is faked, what is real, what is skipped

rg -n "getTezUberServicePluginName|LocalContainerLauncher|LocalTaskSchedulerService|TezLocalTaskCommunicatorImpl" \
  tez-dag/src/main/java/org/apache/tez/dag/app/launcher/ContainerLauncherManager.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/TaskSchedulerManager.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/TaskCommunicatorManager.java

The rewiring is not scattered if (isLocal) branches through the runtime — it is one decision in PluginManager.parseAllPlugins(boolean isLocal, ...):

// tez-dag  org.apache.tez.dag.app.PluginManager
if (!isLocal) {
  tezYarnEnabled = ...; uberEnabled = ...;
} else {
  tezYarnEnabled = false;
  uberEnabled = true;          // <-- everything routes to the "uber" plugin
}

Each of the three managers then maps the uber plugin name (TezConstants.getTezUberServicePluginName()) to its local implementation: TaskSchedulerManager to LocalTaskSchedulerService, ContainerLauncherManager to LocalContainerLauncher, TaskCommunicatorManager to TezLocalTaskCommunicatorImpl.

The scheduler is where "no YARN" is most visible. LocalTaskSchedulerService has no AMRMClient; its "cluster" is a LinkedBlockingQueue drained by an AsyncDelegateRequestHandler thread, and it fabricates Container objects:

// tez-dag  LocalTaskSchedulerService.LocalContainerFactory
public Container createContainer(Resource capability, Priority priority) {
  ContainerId containerId = ContainerId.newInstance(customAppAttemptId,
      nextId.getAndIncrement());
  NodeId nodeId = NodeId.newInstance("127.0.0.1", 0);
  return Container.newInstance(containerId, nodeId, "127.0.0.1:0",
      capability, priority, null);
}

Everything reports one node (getClusterNodeCount() returns 1), and "resources" come from Runtime.getRuntime().maxMemory() / availableProcessors(). There is no locality: every task lands on the single synthetic 127.0.0.1 node.

Shuffle is the other big fake. Local mode sets a shuffle port of 0 and relies on the local-disk-fetch optimization, tez.runtime.optimize.local.fetch (TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH, default true). When the fetcher sees that the source host/port equals its own, it reads segments straight off local disk rather than going over HTTP to a ShuffleHandler:

// tez-runtime-library  FetcherOrderedGrouped.callInternal (trimmed)
if (localDiskFetchEnabled
    && mapHost.getHost().equals(localShuffleHost)
    && mapHost.getPort() == localShufflePort) {
  setupLocalDiskFetch(mapHost);   // no HTTP, no ShuffleHandler
}

Because all tasks run in one JVM on 127.0.0.1, the equality always holds, so local mode never hits the ShuffleHandler HTTP path. See shuffle-sort.md for the fetch/merge machinery this bypasses.

The staging directory deserves a note. LocalClient copies the Tez system staging path to a local working directory (<staging>_wd) with FileUtil.copy "to simulate the resource localizing," then runs the AM against the local FS. The staging path itself may still be on HDFS (e.g. Hive against a pseudo-distributed cluster) — TestLocalMode is parameterized on exactly this (useDfs with a MiniDFSCluster) — but task execution always reads from the local copy.

5. When local mode lies to you

rg -n "TEZ_IGNORE_LIB_URIS|TEZ_LIB_URIS" \
  tez-dag/src/main/java/org/apache/tez/client/LocalClient.java \
  tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java

Every faked/skipped row above is a bug class that a green local-mode run cannot catch. Trace each claim to code before you trust a "works locally" result:

• Classpath / tez.lib.uris assembly. LocalClient.init() sets TEZ_IGNORE_LIB_URIS=true. TezClientUtils.setupTezJarsLocalResources then logs "Ignoring tez.lib.uris" and skips building any LocalResource for the Tez libraries — the client's own classpath already has them. A missing or malformed tez.lib.uris, a bad tarball, or a version-skewed Tez jar in HDFS will never surface locally; it only fails when a real NM tries to localize.

• ShuffleHandler / auxiliary service. Skipped entirely (port 0, local-disk fetch). Any bug in ShuffleHandler registration, the mapreduce_shuffle aux-service id (TEZ_AM_SHUFFLE_AUXILIARY_SERVICE_ID), or the HTTP fetch/SSL path is invisible.

• Security tokens / Kerberos. In-process umbilical means no TezTaskUmbilicalProtocol RPC to authenticate; LocalDAGAppMaster runs no client RPC server. AMRMToken, ClientToAMToken, and job-token-over-the-wire code paths are not exercised.

• Container memory limits and JVM opts. memAvailable is Runtime.maxMemory()/numExecutors, not the requested container size. tez.task.resource.memory.mb and tez.task.launch.cmd-opts shape a real JVM's heap and GC; in local mode there is no child JVM, so an OOM or a bad -Xmx/GC flag will not reproduce.

• Multiple JVMs vs one JVM: static state. This is the sharpest lie. In a cluster each task is a fresh JVM with fresh statics and fresh class loading. In local mode every task shares your JVM — and Tez even relies on the sharing:

  // tez-runtime-internals  LogicalIOProcessorRuntimeTask.cleanup (trimmed)
  // only clean up objects in non-local mode, because local mode shares the
  // same taskSpec in AM rather than getting it through RPC
  if (!tezConf.getBoolean(TezConfiguration.TEZ_LOCAL_MODE,
      TezConfiguration.TEZ_LOCAL_MODE_DEFAULT)) {
    inputSpecs.clear();
    outputSpecs.clear();
    // ...
  }
  

  A processor that stashes state in a static field, or that mutates a shared TaskSpec, can look correct locally and corrupt across attempts on a real cluster — or vice versa. Class-loading order and custom classloaders are also never tested.

Warning: The rule of thumb: local mode validates logic (DAG topology, VertexManager decisions, edge routing, sort/merge correctness). It does not validate deployment (localization, security, memory, isolation, shuffle transport). A patch that touches the second category needs a MiniTezCluster test or a real cluster run. See failure-handling.md.

6. IDE debugging recipe

rg -n "class OrderedWordCount|extends TezExampleBase" \
  tez-examples/src/main/java/org/apache/tez/examples/OrderedWordCount.java
rg -n "LOCAL_MODE|TEZ_LOCAL_MODE" \
  tez-examples/src/main/java/org/apache/tez/examples/TezExampleBase.java

OrderedWordCount extends TezExampleBase, which registers a -local CLI flag that sets tez.local.mode=true:

// tez-examples  org.apache.tez.examples.TezExampleBase.runDag (trimmed)
if (isLocalMode) {
  LOG.info("Running in local mode...");
  tezConf.setBoolean(TezConfiguration.TEZ_LOCAL_MODE, true);
}

Because the AM, scheduler, launcher, and every TezChild are threads in one JVM, a single debugger session hits breakpoints in both the AM and the runtime. That is the whole reason local mode exists (the TEZ_LOCAL_MODE javadoc says "Primarily used for debugging.").

A concrete recipe:

1. Set the smallest possible TezConfiguration in code or on the command line. The minimum is three lines:

   TezConfiguration conf = new TezConfiguration();
   conf.set("fs.defaultFS", "file:///");
   conf.setBoolean(TezConfiguration.TEZ_LOCAL_MODE, true);
   

2. To step across tasks deterministically, leave tez.am.inline.task.execution.max-tasks at its default 1. To expose concurrency, bump it and re-run.

3. Run an example under the debugger. From an IDE, launch org.apache.tez.examples.OrderedWordCount as the main class with program args -local <input> <output> 1, or drive it from a JUnit test in the shape of TestLocalMode. From the shell you can prove the path exists with:

   # ordered word count in local mode, no cluster, single JVM
   INPUT=/tmp/in.txt OUTPUT=/tmp/out
   echo "the quick brown fox the" > "$INPUT"
   $HADOOP_HOME/bin/hadoop jar tez-examples/target/tez-examples-*.jar \
     orderedwordcount -local "$INPUT" "$OUTPUT"
   

4. Set breakpoints and watch them all fire in one process:

   • DAGAppMaster.serviceInit — confirm isLocal is true and the plugin descriptors resolve to the uber plugin.
   • VertexImpl state transitions — see vertex-lifecycle.md and state-machines.md.
   • LocalContainerLauncher.createTezChild / launch — watch a task get submitted to the executor.
   • Your processor's run() — this is on a LocalTaskExecutionThread #N, the same JVM as the AM breakpoints above.

Tip: If without.network is off, you can also breakpoint the client path (DAGClientImpl to RPC to DAGClientHandler). If it is on, that path collapses to direct calls through DAGClientImplLocal; set breakpoints there instead.

  +---------------------------- one JVM (your debugger) -----------------------+
  |                                                                            |
  |   Thread "main"            Thread "DAGAppMaster Thread"                    |
  |   TezClient.submitDAG ---> DAGAppMaster / DAGImpl / VertexImpl             |
  |        ^                        |                                          |
  |        | in-process             | schedule                                 |
  |        | (or loopback RPC)      v                                          |
  |   DAGClientHandler        LocalTaskSchedulerService (thread pool)          |
  |                                 | submit                                   |
  |                                 v                                          |
  |              Thread "LocalTaskExecutionThread #0..N"                       |
  |              LocalContainerLauncher -> TezChild -> your Processor          |
  |                                 |  in-process umbilical (method call)      |
  |                                 +----------> TezLocalTaskCommunicatorImpl  |
  |                                                                            |
  +----------------------------------------------------------------------------+

7. MiniTezCluster contrast

rg -n "class MiniTezCluster|extends MiniYARNCluster|ShuffleHandler|TEZ_LIB_URIS|APPJAR" \
  tez-tests/src/test/java/org/apache/tez/test/MiniTezCluster.java

MiniTezCluster is the opposite end of the spectrum: a real cluster compressed onto one host.

// tez-tests  org.apache.tez.test.MiniTezCluster
public class MiniTezCluster extends MiniYARNCluster {
  public static final String APPJAR = JarFinder.getJar(DAGAppMaster.class);
  // constructors: (testName), (testName, noOfNMs),
  //               (testName, noOfNMs, numLocalDirs, numLogDirs)
}

It extends Hadoop's MiniYARNCluster, so serviceInit stands up a real RM and N NM threads, wires the mapreduce_shuffle ShuffleHandler as a YARN auxiliary service (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID), finds the Tez AM jar via JarFinder, copies it into the (Mini)DFS, and sets tez.lib.uris to that location. Tasks run in separate JVMs forked by the NM's ContainerExecutor. Note the real constructor signature is (testName, noOfNMs, numLocalDirs, numLogDirs) — the third and fourth args are directory counts, not DataNode/rack counts; HDFS comes from a separately created MiniDFSCluster (as in TestLocalMode) or a RawLocalFileSystem.

// typical usage (see tez-tests TestTezJobs / TestMRRJobs)
miniTezCluster = new MiniTezCluster(TestTezJobs.class.getName(), 1, 1, 1);
miniTezCluster.init(conf);
miniTezCluster.start();

TezConfiguration tezConf = new TezConfiguration(miniTezCluster.getConfig());
TezClient client = TezClient.create("test", tezConf);
client.start();

Local mode vs MiniTezCluster

Use MiniTezCluster when you are exercising RPC, security, localization, the ShuffleHandler HTTP path, container lifecycle (kill vs orderly shutdown), or HDFS-backed recovery (failure-handling.md). Use local mode for VertexManager/EdgeManager logic, sort/merge behavior, and single-JVM debugging. For pure unit tests of a single class, neither — see testing-framework.md.

Reading exercise

# 1. The one branch that picks your execution model
rg -n "isLocal|LocalClient|createFrameworkClient" \
  tez-api/src/main/java/org/apache/tez/client/FrameworkClient.java

# 2. How LocalClient starts the AM on a thread and picks the AM class
rg -n "startDAGAppMaster|createDAGAppMaster|isLocalWithoutNetwork|LocalDAGAppMaster" \
  tez-dag/src/main/java/org/apache/tez/client/LocalClient.java

# 3. The thread-pool "resource pool" and the config knob
rg -n "numExecutors|newFixedThreadPool|LocalContainerFactory|createContainer|getClusterNodeCount" \
  tez-dag/src/main/java/org/apache/tez/dag/app/rm/LocalTaskSchedulerService.java \
  tez-dag/src/main/java/org/apache/tez/dag/app/launcher/LocalContainerLauncher.java

# 4. The one decision that rewires the AM plugins
rg -n "parseAllPlugins|uberEnabled|tezYarnEnabled|getTezUberServicePluginName" \
  tez-dag/src/main/java/org/apache/tez/dag/app/PluginManager.java

# 5. Shuffle: how local-disk fetch replaces the ShuffleHandler
rg -n "localDiskFetchEnabled|setupLocalDiskFetch|TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH" \
  tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/FetcherOrderedGrouped.java \
  tez-runtime-library/src/main/java/org/apache/tez/runtime/library/api/TezRuntimeConfiguration.java

# 6. A local-mode test and a MiniCluster test, side by side
find tez-tests -name "TestLocalMode.java" -o -name "TestMRRJobs*.java" | grep -v target

Trace: for exercise 2, why does createDAGAppMaster return a Thread rather than a DAGAppMaster directly, and where does dagAppMaster actually get assigned? For exercise 5, why is the host/port equality always true in local mode?

Common bugs and symptoms

Validation: prove you understand this

1. Name the single method and boolean that decide whether a job runs in local mode, and cite the class that gets loaded reflectively as a result.
2. Distinguish tez.local.mode from tez.local.mode.without.network: what does each remove, which AM subclass does the latter select, and what method does that subclass no-op?
3. In local mode, where does the "umbilical RPC" between a task and the AM actually happen? Trace it from LocalContainerLauncher through TezChild.newTezChild to the else branch that assigns this.umbilical.
4. Give the config key, constant, and default that control local task concurrency, and explain what raising it exposes and why the default is 1.
5. List four layers local mode skips or fakes, and for each name the concrete class or config that is bypassed and one bug class it hides.
6. A patch changes tez.lib.uris handling / localization. Explain why a TestLocalMode-style test cannot validate it, and sketch the minimum MiniTezCluster setup that can — including why the AM jar has to reach DFS.
7. You want to step from VertexImpl scheduling a task into your processor's run() in a single debugger session. Explain why that works in local mode and would not with a real cluster, naming the threads involved.