Split the Labor: Separating Evidence Interpretation from Decision Aggregation
A paper proposes separating evidence interpretation from decision aggregation in language model systems, introducing a four-field evidence tuple (hypothesis, reliability bucket, rationale, provenance) and identifying a failure mode called count-scale drift.
The paper argues that systems using language models to reach conclusions from multiple sources conflate two operations: interpreting individual sources and combining interpretations. It proposes separating these operations and defines a four-field evidence tuple to interface between them. It also identifies count-scale drift, where thresholding a sum of unnormalized weights slides with the number of sources, and suggests pooling calibrated log-likelihood ratios as a fix.
The separation of evidence interpretation from decision aggregation allows each component to be optimized independently: interpretation benefits from high-capacity models and long context, while aggregation requires fixed arithmetic and comparability. The proposed four-field tuple standardizes the interface. Count-scale drift is shown to be equivalent to posterior thresholding at an operating point that shifts with source count, and pooling calibrated log-likelihood ratios corrects this drift.
This approach could improve the reliability and auditability of AI systems that synthesize information from multiple sources, such as research assistants, legal document review, and medical diagnosis support. By separating interpretation and aggregation, developers can more easily update or replace components and better control decision thresholds.
The proposed separation and arithmetic fix can reduce errors in multi-source decision systems, potentially lowering operational risk and improving trust in AI outputs. It may enable more modular system designs, reducing development and maintenance costs.
Next signals include empirical validation of the proposed tuple and pooling method on real-world multi-source tasks, adoption of the separation principle in agentic frameworks, and further analysis of count-scale drift in deployed systems. The paper's arithmetic fix may be incorporated into existing triage and diagnostic engines.