TRANSMISSION // 2611.23161v1

Why AI explanations should pass through the decoder

Source published

Attribution methods can highlight inputs that change a model’s raw coordinates but not its final prediction. A preprint tests decoder-aware alternatives for direction and rotation outputs.

AI-generated conceptual illustration for Why AI explanations should pass through the decoder
AI-generated conceptual illustration; not documentary evidence.

When an AI predicts a gaze direction or an object’s rotation, its final answer may be decoded from intermediate numbers. A new preprint argues that explanations should account for that decoding step—while cautioning that decoder-aware explanations are not automatically complete or reliable.

A model that predicts a direction may not output an angle directly. It can produce a vector, then use a fixed operation such as normalization to turn that vector into a direction. Changing the vector’s length alone does not change the direction. Yet an attribution method applied to the raw coordinates may still credit an input for causing that length change, even though the final answer stays the same.

That is the problem addressed by Attribution Through the Decoder: Prediction-Space Explanations for Structured Outputs, a preprint by Zaer Bin Zaqy and Brinnae Bent. The authors propose calculating attributions through the decoder: the operation that converts a model’s raw output into its usable prediction. For normalization, their approach uses a projection onto the direction-changing, or tangent, component of a change. In plain terms, it aims to disregard changes that only stretch or shrink the vector.

The abstract reports tests on synthetic tasks with known relevant inputs, MPIIGaze validation data, and PASCAL3D+ rotation predictions using a six-dimensional rotation decoder. On the synthetic tasks, standard coordinate-based attribution assigned as much as 72% of its attribution mass to an irrelevant region. For MPIIGaze, tangent Grad-CAM produced 44% greater prediction displacement after deleting 10% of image pixels. In the rotation tests, the full decoder differential improved mean deletion displacement by 14.1 degrees over an ambient-coordinate sum and 13.8 degrees over a decoder-invariant scalar.

Deletion tests remove selected input features or pixels and measure how much the prediction changes. Greater displacement can indicate that an attribution method selected inputs with more influence on the output. It does not, by itself, show that an explanation is complete, or that the selected inputs are meaningful to a person.

The authors also distinguish two possible problems: attribution to changes the decoder discards, and loss of information when a structured prediction is reduced to a single scalar for attribution. Results for a two-polarity tangent LayerCAM variant were mixed: it exceeded ambient-sum LayerCAM on circular regression, while the authors detected no difference on quaternion pose.

There are important qualifications. The abstract says that every fixed smooth scalar target on a sphere has points with zero tangential sensitivity. It also reports that an endpoint-projected version of Integrated Gradients can still credit inputs that only affect scale, while direct angular Integrated Gradients cannot. So accounting for decoder invariance does not guarantee that an attribution method captures every input that drives a prediction; the authors recommend checking both invariance and deletion performance.

This account is based on the paper’s abstract, not a full-text review. The reported figures therefore cannot be assessed here in detail, including the experimental setups, uncertainty estimates, or comparisons across conditions. The work is an arXiv preprint, and the supplied source does not establish whether it has undergone peer review. Its results describe specific benchmarks and methods, not a general guarantee that decoder-aware explanations will be better in every application.

Reviewed cross-references

These connections were reviewed by a model, not independently fact-checked by a human. Similarity does not imply agreement; consult the original papers and the evidence below.

  • Can a 3D body scan help predict cesarean delivery?

    Shared method: Both papers use Integrated Gradients for feature attribution: MvBody applies it to medical features and 2D projections in cesarean-risk prediction, while the structured-output study compares IG variants for decoded angles and rotations. The latter’s decoder-specific findings do not establish how MvBody’s explanations perform; the tasks and evaluation settings differ.

    Model review: · openai-codex/gpt-6-luna

    Evidence and provenance

    Selected excerpts, not complete papers. PDF extraction can omit figures or damage equations, tables and reading order; sources may be preprints.

    • 2611.23161v1 — Original paper

      Excerpt reference: static/papers/2611.23161v1.md:544-677
      Basis: Paper text
      Source SHA-256: c71f9f3cfd31203aecf5a25c0c12488f7b4d27ec7403011d4a7852d5646dad14

    • 2511.03212v1 — Original paper

      Excerpt reference: static/papers/2511.03212v1.md:531-574
      Basis: Paper text
      Source SHA-256: 11e1af13c26a084ac5c3927132293d4d259f0e2e0784b088ce946cf4880faa90

Automatically matched by shared topic keywords. These links do not imply that the sources agree or establish a consensus.