PredicateLongBench: Long-Context Evaluation Begins to Measure 'Task Difficulty' Instead of Just Length
The PredicateLongBench preprint, submitted on July 9, 2026, proposes two pipelines for controllable generation of long-context tasks and reports significant performance degradation of frontier models as predicate difficulty increases.
Long-context capability has long been dominated by 'how long is the context window,' but the window's capacity to hold information does not mean the model can reliably find and combine evidence under complex constraints. This study decouples length from task difficulty, providing a stress-test approach closer to real work for enterprise documents, research, and agent evaluations.
The paper controls retrieval conditions, combinatorial relationships, and reasoning complexity through predicate structures, and constructs reproducible data with two types of generation pipelines. It provides a difficulty axis, allowing teams to observe model degradation curves under the same context length but different constraint complexity, offering more information than a single average score.
Model vendors publishing only long-context windows and single benchmark scores will find it harder to support procurement decisions; application teams need to establish hierarchical evaluations based on their own document structures, constraint counts, and evidence combination methods.
When procuring long-context models, replace 'maximum window' with success rates stratified by task difficulty, retry rates, and manual verification costs to avoid paying for capacity that cannot translate into reliable results.
This is a preprint that has not yet been independently replicated. Next steps should check whether data generation introduces shortcuts, conduct replication experiments with different model versions, and examine the correlation of the difficulty axis with real legal, financial, and R&D document tasks.