LEAP: Likelihood Elicitation and Aggregation for LLM-based Probabilistic Forecasting
LEAP (Likelihood Elicitation and Aggregation for Probabilistic forecasting) is proposed as a method that reorganizes how collected evidence is used in the prediction stage of LLM-based forecasting systems. It examines each evidence item separately, elicits likelihood parameters, and combines them with an explicit prior and deterministic probabilistic model into a posterior distribution. LEAP supports continuous, single-choice, and multi-choice forecasts. A benchmark covering forecasting, information-seeking, and browsing tasks was built, and LEAP was evaluated on the authors' own agent loop and several agent CLI frameworks. Given the same evidence, LEAP improves most prediction and calibration metrics.
LEAP is a new approach for LLM-based probabilistic forecasting that replaces monolithic prediction with per-evidence likelihood elicitation and aggregation. It supports multiple forecast types and was evaluated on a new benchmark, showing improvements in prediction and calibration metrics.
LEAP decomposes the prediction stage into per-evidence likelihood elicitation followed by deterministic probabilistic aggregation with an explicit prior. This design preserves reproducible evidence contributions and avoids collapsing uncertainty across competing outcomes. The benchmark includes forecasting, information-seeking, and browsing tasks, and evaluation spans multiple agent frameworks.
The method addresses a common limitation in LLM forecasting systems where all evidence is read together, obscuring individual evidence impact. By improving calibration and prediction metrics, LEAP could enhance reliability of LLM-based forecasting in domains like financial markets and sports outcomes.
Improved calibration and prediction accuracy in LLM forecasting can increase trust and adoption in decision-support systems for finance, sports analytics, and other probabilistic domains. The reproducible evidence contributions may also aid auditing and explainability.
Observable next signals include adoption of LEAP in agent frameworks, further benchmarks comparing monolithic vs. decomposed prediction, and extensions to other probabilistic tasks. Replication studies and open-source implementations would validate the reported improvements.