Stage 11 — Backward Compatibility

What this stage teaches

Stage 11 is where every change is constrained by what shipped before. The code is often trivial; the reasoning is the work. Tez is a library that Hive and Pig compile against, whose AM deserialises plans written by other AM versions, and whose config keys live in production tez-site.xml files you will never see. Break any of those contracts and you break users who did nothing wrong. You learn:

• Tez's compatibility surface, concretely: the Apache @InterfaceAudience / @InterfaceStability annotations (from org.apache.hadoop.classification), the ABI that downstream projects link against, the protobuf wire format, and the config-key namespace.
• Why a source-compatible change can still be binary-incompatible, and why that distinction has bitten Tez in production.
• The deprecation cycle: alias, deprecate, keep the old path working, remove only in a later release.
• How the TezConfiguration deprecation machinery lets you rename a key without breaking anyone.

Patches are often 1–40 lines. The care is in the annotation, the Javadoc, and the CHANGES.txt line that flags the change as incompatible so it is not merged silently.

The mindset shift this stage demands: in earlier stages "does it work?" meant "does it pass the tests and behave correctly." Here it also means "does every existing caller, on every supported version, on both sides of a rolling upgrade, keep working." The reviewer you must satisfy is not thinking about your code — they are thinking about the Hive team six months from now upgrading a production cluster, and about the tez-site.xml some operator wrote three years ago. Compatibility is empathy for people you will never meet, encoded as discipline.

Prerequisite: Stage 10 plus the deep dives: tez-client, DAG model, and IPO abstractions. This stage assumes you know which classes live in tez-api (the public surface) versus tez-dag (the private impl).

The compatibility surface, verified

Everything below you can confirm in the checkout — do not take it on faith.

The annotations are Hadoop's, imported into tez-api:

cd /Users/s0x/src/oss-repos/tez
grep -rn "import org.apache.hadoop.classification" \
  tez-api/src/main/java/org/apache/tez/client/TezClient.java \
  tez-api/src/main/java/org/apache/tez/dag/api/DAG.java

You will see InterfaceAudience.Public, InterfaceAudience.Private, InterfaceStability.Evolving, InterfaceStability.Unstable. Audience × stability gives the cost matrix:

The config machinery lives in TezConfiguration. Tez uses Hadoop's Configuration.addDeprecation to alias old keys to new:

grep -n "addDeprecation" tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java | head

You will see real aliases like tez.am.counters.max.keys → TezConfiguration.TEZ_COUNTERS_MAX. That is the mechanism you use to rename a key compatibly: never delete the old name, alias it.

There is a subtler config hazard than renaming: changing a default value. A default change compiles, passes every test, and breaks nothing at build time — then silently alters the behaviour of every cluster that relied on the old default and never set the key explicitly. That makes it more dangerous than an API break, not less, because nothing warns anyone. Treat a default change as a behaviour change requiring a dev@ thread and a CHANGES.txt flag, and prefer to introduce new tuning behind a new key at a safe default rather than moving an existing one.

The wire format is proto2. The DAG plan messages are in tez-api/src/main/proto/DAGApiRecords.proto:

grep -n "message VertexPlan\|message DAGPlan\|optional\|required\|reserved" \
  tez-api/src/main/proto/DAGApiRecords.proto | head -30

Note it is proto2 with explicit required/optional fields — which is exactly why you must never add a required field (old writers won't set it) and never recycle a field number.

Finding Stage 11 issues today

project = TEZ AND resolution = Unresolved
  AND (summary ~ "compatibility" OR summary ~ "incompatible"
       OR summary ~ "deprecate" OR summary ~ "ABI" OR summary ~ "protobuf"
       OR summary ~ "rolling upgrade" OR labels = "incompatible")
ORDER BY priority DESC, updated DESC

And read the history — Tez has labelled incompatible changes in CHANGES.txt for a decade:

git log --oneline -i --grep=compat --grep=deprecat --grep=incompatible

Case study A — TEZ-3953: source-compatible is not binary-compatible

The single most important lesson in this stage. Read it:

git show 3f2373e2b   # Restore ABI-compat for DAGClient for TEZ-3951

The symptom. A prior change (TEZ-3951) added a method to the abstract class DAGClient. It compiled fine and Tez's own tests passed. But Hive, which is compiled against an older Tez jar and run against a newer one, broke at runtime — because adding an abstract method changes the class's binary contract.

The root cause. DAGClient.waitForCompletion(long) was declared abstract:

public abstract DAGStatus waitForCompletion(long timeMs)
    throws IOException, TezException, InterruptedException;

An abstract method forces every subclass to implement it. Downstream code that subclassed DAGClient against the old jar has no such implementation, so at load time against the new jar the class is abstract-incomplete — a binary incompatibility even though source against the new jar would compile.

The fix — give it a default body. Sergey Shelukhin made the method concrete with a throwing default, so old subclasses remain loadable:

public DAGStatus waitForCompletion(long timeMs)
    throws IOException, TezException, InterruptedException {
  // Make non-abstract to avoid compat issues in Hive.
  throw new UnsupportedOperationException();
}

Four lines. The comment names why. Behaviour for correct callers is unchanged; the binary contract is restored.

The lesson. For a Public class, adding an abstract method is a binary-incompatible change, even though it looks harmless and compiles. The safe way to extend a public abstract class is a concrete method with a default (throwing or sensible) body. Always ask: who subclasses this against an older jar? For tez-api types, the answer is usually Hive.

Case study B — TEZ-2740: alias first, deprecate, never break

git show a0c0727dd   # Create a reconfigureVertex alias for deprecated setVertexParallelism API

The context. Tez wanted to replace setVertexParallelism(...) on VertexManagerPluginContext with a better-named, better-documented reconfigureVertex(...). VertexManagerPluginContext is a Public API implemented by user vertex managers and called by Hive/Pig — you cannot just rename it.

The fix — add the new name, keep the old working. Bikas Saha added reconfigureVertex(...) as a fully-documented alias:

/**
 * API to reconfigure a {@link Vertex} that is reading root inputs based on the
 * data read from the root inputs. ...
 */
public void reconfigureVertex(int parallelism,
    @Nullable VertexLocationHint locationHint,
    @Nullable Map<String, EdgeManagerPluginDescriptor> sourceEdgeProperties,
    @Nullable Map<String, InputSpecUpdate> rootInputSpecUpdate);

and implemented it in VertexManager by delegating to the same underlying managedVertex.setParallelism(...) the old method used:

@Override
public synchronized void reconfigureVertex(int parallelism, VertexLocationHint vertexLocationHint,
    Map<String, EdgeManagerPluginDescriptor> sourceEdgeProperties,
    Map<String, InputSpecUpdate> rootInputSpecUpdate) {
  checkAndThrowIfDone();
  try {
    managedVertex.setParallelism(parallelism, vertexLocationHint, sourceEdgeProperties,
        rootInputSpecUpdate, true);
  } catch (AMUserCodeException e) {
    throw new TezUncheckedException(e);
  }
}

The old setVertexParallelism stayed. Both route to the same implementation. The CHANGES.txt entry flagged it under INCOMPATIBLE CHANGES so the release notes tell downstreams about the new preferred name.

The companion, TEZ-2287 (git show 11b584318), is even smaller — the pure deprecation step: a single @Deprecated annotation on VertexManagerPluginContext.getTaskContainer(...), no behaviour change, plus a CHANGES.txt line. That is the whole first half of a deprecation cycle.

The lesson — the cycle. Compatible evolution is three separate releases: (1) add the new name as an alias that delegates to the same code; (2) @Deprecated the old name, Javadoc pointing at the replacement, behaviour unchanged; (3) remove the old name only after downstreams (Hive, Pig) have shipped a release using the new one. Steps 1–2 can share a patch (TEZ-2740 did the alias); step 3 is a different release and needs dev@ sign-off.

Case study C — TEZ-2075: an accidental visibility change broke users

git show 8e8405b37   # Incompatible issue caused by TEZ-1233 that TezConfiguration.TEZ_SITE_XML is made private

The symptom. A refactor (TEZ-1233) changed TezConfiguration.TEZ_SITE_XML from public to private. Nothing in Tez broke. But downstream code that referenced the constant stopped compiling — an incompatible change slipped in as a side effect of an unrelated refactor.

The root cause and fix. One word:

-  private final static String TEZ_SITE_XML = "tez-site.xml";
+  public final static String TEZ_SITE_XML = "tez-site.xml";

Jeff Zhang restored public. The interesting part is the test: TestTezConfiguration reflectively iterates the public String constants to validate the config namespace, and TEZ_SITE_XML had to be excluded from that check (it is a filename, not a tez.-prefixed key):

if (!f.getName().endsWith("DEFAULT") && f.getType() == String.class
    && !f.getName().equals("TEZ_SITE_XML")) {

The lesson. Tightening a modifier — public → private, protected → package-private — is an incompatible change even when it is "just cleanup." A public constant is part of the ABI. Before you narrow any visibility in tez-api, grep for downstream usage and assume Hive/Pig depend on it. Incompatible changes must be deliberate and flagged in CHANGES.txt, never a refactoring side effect.

The protobuf rules (for when you evolve the wire format)

The DAG plan is serialised to YARN's RM and read by AMs that may be a different version, so DAGApiRecords.proto obeys strict rules:

1. New fields are optional, never required. An old writer won't set a new required field, and an old reader will reject a message missing one it expects.
2. Never recycle a field number. Removing a field? Mark it reserved. Reusing a number silently corrupts cross-version reads.
3. Never change a field's type. string → bytes looks identical on the wire but breaks at parse time. Add a new field with a new number instead.
4. The consumer must tolerate absence. Call hasFoo() before getFoo(); a proto2 unset field returns a default that is not the same as "absent."

Confirm a candidate number is unused before you claim it:

git log -p -S "= 13" -- tez-api/src/main/proto/DAGApiRecords.proto

Case study D — TEZ-1664: rolling upgrades need a version handshake

Compatibility isn't only about compiling — it's about a newer client talking to an older AM, or vice versa, during a rolling upgrade. Read how Tez made that safe:

git show d59b2318d   # Add checks to ensure that the client and AM are compatible

The problem. In a rolling upgrade, the submitting client and the running AM can be different Tez versions. Without a check, a subtle wire mismatch surfaces as a baffling runtime failure deep in the DAG instead of a clear "these versions are incompatible" at submit time.

The mechanism. Hitesh Shah added a build-time version stamp and a comparator:

• A filtered tez-api-version-info.properties (and a tez-dag twin) baked the version and build timestamp into the jars at build time via Maven resource filtering — so VersionInfo/TezApiVersionInfo/TezDagVersionInfo can report the exact version at runtime.
• A Simple2LevelVersionComparator compares only the major.minor of two versions (patch level is compatible), and the AM checks the client's reported version against its own on submit.

What the test did. TestSimple2LevelVersionComparator enumerates version pairs (equal, patch-differing, minor-differing, malformed) and asserts the compatibility verdict for each — a pure, fast unit test of the comparison rule, plus TestVersionInfo against checked-in fixture .properties files.

The lesson. "Backward compatible" includes the mixed-version cluster that exists for the duration of every rolling upgrade. When you change anything on the client↔AM boundary, ask what happens if the two ends are one minor version apart, and prefer a clear early failure over a mysterious late one. The major.minor granularity is deliberate: patch releases must stay wire-compatible.

The contribution playbook for this class

1. Classify the surface. Public or Private? Source or binary? A tez-api change is guilty until proven compatible.
2. For an API rename, alias-then-deprecate (TEZ-2740 / TEZ-2287). Never rename in place.
3. For a config rename, use addDeprecation — the existing machinery in TezConfiguration. Old key keeps working.
4. For a public abstract class, extend with a concrete default method, not an abstract one (TEZ-3953).
5. Never narrow visibility in tez-api without checking downstreams (TEZ-2075).
6. Flag every incompatible change in CHANGES.txt, and take anything genuinely breaking to a dev@ thread before coding.

Common mistakes

Exit criteria — when you're ready for the next stage

• You have shipped one compatibility-sensitive change — an alias+deprecation, a config-key deprecation via addDeprecation, or a proto field addition — with the correct annotation and a CHANGES.txt entry.
• You can explain the TEZ-3953 distinction (source- vs binary-compatible) and why adding an abstract method breaks Hive.
• You can recite the deprecation cycle and know that removal is a separate, later release requiring dev@ sign-off.
• You can state the four proto2 rules and confirm an unused field number with git log -S.
• You verified the annotation, config, and proto facts in the checkout yourself rather than trusting this page.

Stage 12 is the final stage: release-blocking issues and PMC-level work.