Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes
Oilbird is a training-free speculative decoding method that adds a semantic draft source using hidden states the verifier already computes, re-keying an existing context pool. In three published drafters, at matched pool and budget, it lifts accepted length by 24-29%. Oilbird reaches 4.4x autoregressive decoding speed on API-Bank, against 3.9x for the strongest training-free baseline and 2.0x for EAGLE-3.
Training-free speculative decoding drafts by matching an exact suffix of the context against a pool of earlier context. That lookup misses correct drafts already in the pool, most visibly on tool-calling traffic, where a request repeats almost everything but the few values minted for it, and where one rejected token discards the correct continuation behind it. We diagnose the failure position by position across ten benchmarks and find it to be a problem of addressing rather than of coverage: on our densest tool-calling benchmark, about half of what the strongest exact-match drafter misses is present in the pool yet unreachable by exact matching. We therefore propose a second, semantic draft source: the same pool, re-keyed by the hidden state the verifier has already computed at each committed token, together with a merge that lets it ride inside an existing lexical drafter's tree. In three published drafters, at matched pool and budget, it lifts accepted length by 24-29%. Oilbird reaches 4.4x autoregressive decoding speed on API-Bank, against 3.9x for the strongest training-free baseline in our harness and 2.0x for EAGLE-3.
Oilbird introduces a semantic draft source that re-uses hidden states already computed by the verifier, enabling retrieval of correct continuations that exact-match methods miss. This addresses an addressing failure rather than a coverage failure, and the merge mechanism allows integration with existing lexical drafters without additional training.
Training-free speculative decoding methods are attractive for deployment because they avoid the cost and complexity of training a separate drafter model. Oilbird's approach could lower inference costs and latency for large language models, particularly in tool-calling and repetitive-context scenarios common in enterprise and consumer applications.
By improving decoding speed without training, Oilbird can reduce serving costs and improve user experience for LLM-powered products. This is especially relevant for high-volume, latency-sensitive applications such as chatbots, code assistants, and API-based tools.
If the method generalizes across model architectures and scales, it could become a standard component in LLM inference stacks. Next signals to watch include adoption by inference providers, integration into open-source serving frameworks, and benchmarks on additional workloads beyond API-Bank.