HiLRP: Toward One Trustworthy Explanation for Vision Transformer: Conservation-Valid Attribution via Attention Primitives
The paper introduces HiLRP, a unified attribution framework for Vision Transformers (ViTs). It decomposes attention and resolution-reduction operators into four operation types: linear maps, bilinear mixing, normalization or gating, and reindexing. Each operation admits a conservation-valid relevance rule. HiLRP supports new backbones by construction rather than architecture-specific derivation.
Vision Transformer designs have become diverse, incorporating convolutional stems, windowed, linear, or multi-axis attention, patch merging, and spatial reduction. Existing attribution methods often fail across these variants: Grad-CAM requires a terminal spatial feature map, attention rollout assumes global softmax attention, and layer-wise relevance propagation (LRP) requires module-specific rules. HiLRP addresses this by decomposing ViT operations into four primitive types, each with a conservation-valid relevance rule, enabling a unified attribution framework across architectural space.
HiLRP's decomposition into linear maps, bilinear mixing, normalization/gating, and reindexing provides a modular relevance propagation scheme that preserves conservation. This approach eliminates the need for architecture-specific LRP rules and can be applied to new ViT backbones by construction. The attribution maps decompose the prediction rather than relying on heuristic assumptions, potentially improving faithfulness across diverse ViT variants.
A unified, conservation-valid attribution method for ViTs could standardize interpretability across model variants, reducing the engineering burden for model developers and auditors. It may accelerate adoption of explainability tools in regulated industries where model transparency is required.
HiLRP could lower the cost of compliance and debugging for companies deploying ViT-based computer vision systems by providing a single, trustworthy explanation method across model variants. It may also enable new products in model auditing and explainability-as-a-service.
Observable next signals include: (1) release of open-source code or library implementing HiLRP; (2) benchmark comparisons against Grad-CAM, attention rollout, and LRP on multiple ViT backbones; (3) adoption by model interpretability platforms or integration into major frameworks; (4) follow-up papers extending HiLRP to other architectures such as MLP-Mixers or hybrid models.