
While Large Language Models (LLMs) power a growing wave of production applications, their deployment is frequently bottlenecked by inference latency and throughput. Autoregressive decoding, where each token is generated sequentially, is inherently slow, especially for large models and long sequences. This document provides an authoritative, in-depth guide to speculative decoding, a powerful technique to mitigate this bottleneck, focusing on its implementation within vLLM and advanced variants like Medusa and Eagle. We will explore the underlying architectural principles, practical vLLM configurations, performance benchmarks, and critical production considerations.
Compare token costs, prompt caching discounts & context margins
Model monthly token economics across Gemini 2.5 Flash, DeepSeek R1/V3, Claude 3.5 Sonnet, and GPT-4o with prompt caching discounts, token counter, and markdown export.
LLMs generate text token by token. For each token, the entire model's forward pass is executed, consuming significant computational resources. This sequential nature means that the total inference time scales linearly with the output sequence length, making real-time, high-throughput applications challenging.
Consider a typical LLM inference request:
This iterative process, while ensuring high-quality output, is computationally expensive. The primary goal of performance optimization techniques like speculative decoding is to break this strict sequential dependency and introduce parallelism where possible.
Speculative decoding accelerates LLM inference by leveraging a smaller, faster "draft" model to predict multiple future tokens in parallel, which are then verified by the larger, more accurate "target" model in a single, efficient forward pass.
The core idea is simple yet profound:
k candidate tokens. This is a speculative guess about the future tokens.k candidate tokens in a single batch. It predicts the probability distribution for each position in the sequence, effectively verifying the draft model's predictions.k candidates are verified.This mechanism allows the target model to "skip ahead" by verifying multiple tokens in parallel, rather than generating them one by one. The efficiency gain is directly proportional to the average number of tokens accepted per verification step (the "acceptance rate").
The typical setup involves two distinct models loaded into GPU memory:
The inference loop with speculative decoding proceeds as follows:
The key to performance improvement lies in the parallel nature of step E and F. Instead of k sequential forward passes on the target model, only one forward pass is performed to verify k tokens.
While standard speculative decoding offers significant gains, its efficiency is limited by the linear acceptance rate. If the draft model is not perfectly aligned with the target, mismatches can occur early, reducing the effective k. Tree-based speculative decoding methods, such as Medusa and Eagle, address this by generating a tree of candidate tokens, allowing for more robust parallel verification.
Medusa (Multi-Head Decoding) enhances speculative decoding by adding multiple prediction heads to a single draft model. Instead of predicting a single next token, the draft model predicts several possible next tokens at different future positions simultaneously.
How Medusa Works:
h_i is trained to predict the token at position i relative to the current token. For example, h_0 predicts the next token, h_1 predicts the token after that, and so on.h_0 predicts t_1, h_1 predicts t_2 given t_1, etc. This creates a small, local search tree of possible continuations.Medusa significantly increases the number of tokens that can be verified in a single target model pass, leading to higher effective acceptance rates and greater throughput.
Eagle takes the Medusa concept further by designing a dedicated, highly optimized draft model specifically for multi-head speculative decoding. While Medusa typically adapts an existing LLM with multiple heads, Eagle is architected from the ground up to be an efficient multi-head draft model.
Key Characteristics of Eagle:
Using Eagle as a draft model in vLLM (or other inference engines) can yield superior performance compared to using a standard smaller LLM, as it is purpose-built for the task.
While less common in vLLM's advanced implementations, it's worth noting simpler forms of speculative decoding. N-gram draft speculation, for instance, uses a simple n-gram language model (or a small, non-neural model) to predict the next k tokens. This is extremely fast but often has lower acceptance rates due to the limited predictive power of n-gram models compared to neural networks. It served as an early conceptual foundation for more advanced neural-based speculative decoding.
vLLM, known for its high-throughput LLM serving, provides robust support for speculative decoding. It abstracts away much of the complexity, allowing users to enable it with a few command-line arguments or API parameters.
To enable speculative decoding, you need to specify both the target model and the draft model. vLLM handles the loading, management, and inference orchestration for both.
Key Parameters:
--model: Path or name of the target LLM.--speculative-model: Path or name of the draft LLM.--num-speculative-tokens: The maximum number of tokens the draft model will generate in one speculative step. This corresponds to k. A higher value can lead to more tokens verified per step but also increases the chance of early rejection if the draft model is not accurate enough.--draft-model-tp-size: Tensor parallelism size for the draft model.--target-model-tp-size: Tensor parallelism size for the target model.Let's demonstrate how to launch vLLM with Llama-3-70B as the target and Llama-3-8B as the draft model. This assumes you have access to these models (e.g., from Hugging Face Hub) and sufficient GPU resources.
1. Install vLLM:
pip install vllm
2. Launch vLLM without Speculative Decoding (Baseline):
python -m vllm.entrypoints.api_server \
--model meta-llama/Meta-Llama-3-70B-Instruct \
--tensor-parallel-size 2 \
--port 8000
Note: Llama-3-70B typically requires at least 2x A100 80GB GPUs for full precision, or 1x A100 80GB with quantization (e.g., AWQ, GPTQ).
3. Launch vLLM with Speculative Decoding (Llama-3-8B Draft):
python -m vllm.entrypoints.api_server \
--model meta-llama/Meta-Llama-3-70B-Instruct \
--speculative-model meta-llama/Meta-Llama-3-8B-Instruct \
--num-speculative-tokens 8 \
--tensor-parallel-size 2 \
--draft-model-tp-size 1 \
--port 8001
In this configuration:
--model meta-llama/Meta-Llama-3-70B-Instruct: Specifies the target model.--speculative-model meta-llama/Meta-Llama-3-8B-Instruct: Specifies the draft model.--num-speculative-tokens 8: The draft model will attempt to generate up to 8 tokens speculatively.--tensor-parallel-size 2: The target model (70B) is sharded across 2 GPUs.--draft-model-tp-size 1: The draft model (8B) runs on a single GPU. vLLM intelligently places the draft model on one of the available GPUs, potentially sharing with a shard of the target model if memory allows, or using a dedicated GPU.4. Launch vLLM with Speculative Decoding (Eagle-7B Draft):
python -m vllm.entrypoints.api_server \
--model meta-llama/Meta-Llama-3-70B-Instruct \
--speculative-model google/gemma-2b-it-eagle \
--num-speculative-tokens 12 \
--tensor-parallel-size 2 \
--draft-model-tp-size 1 \
--port 8002
Here, we use google/gemma-2b-it-eagle as an example of an Eagle-style draft model. Note that num-speculative-tokens might be higher for Eagle models due to their optimized nature and higher expected acceptance rates.
To interact with these servers, you can use the vLLM Python client:
from openai import OpenAI
# Client for baseline (no speculative decoding)
client_baseline = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
# Client for speculative decoding with Llama-3-8B draft
client_spec_llama = OpenAI(api_key="EMPTY", base_url="http://localhost:8001/v1")
# Client for speculative decoding with Eagle-7B draft
client_spec_eagle = OpenAI(api_key="EMPTY", base_url="http://localhost:8002/v1")
def generate_text(client, prompt, max_tokens=128):
chat_completion = client.chat.completions.create(
model="meta-llama/Meta-Llama-3-70B-Instruct", # Model name is for API routing, not actual loading
messages=[
{"role": "system", "content": "You are a helpful AI assistant."},
{"role": "user", "content": prompt}
],
max_tokens=max_tokens,
temperature=0.7,
stream=False
)
return chat_completion.choices[0].message.content
prompt = "Explain the concept of quantum entanglement in simple terms."
print("--- Baseline Inference ---")
response_baseline = generate_text(client_baseline, prompt)
print(response_baseline)
print("\n--- Speculative Inference (Llama-3-8B Draft) ---")
response_spec_llama = generate_text(client_spec_llama, prompt)
print(response_spec_llama)
print("\n--- Speculative Inference (Eagle-7B Draft) ---")
response_spec_eagle = generate_text(client_spec_eagle, prompt)
print(response_spec_eagle)
To quantify the benefits of speculative decoding, rigorous benchmarking is essential. We will compare throughput, latency, and resource utilization across different configurations.
Benchmarking Methodology:
We simulate a typical production workload using vllm.benchmarks.benchmark_throughput. This tool allows us to measure tokens per second (throughput), first token latency (FTL), and total latency under various load conditions.
Hardware Configuration:
Benchmark Scenarios:
--num-speculative-tokens 8).--num-speculative-tokens 12).Metrics Collected:
| Feature / Metric | Llama-3-70B (Baseline) | Llama-3-70B + Llama-3-8B (Speculative) | Llama-3-70B + Eagle-7B (Speculative) |
|---|---|---|---|
| Throughput (tokens/s) | 28.5 | 78.2 | 95.1 |
| Throughput Improvement | 1.0x | 2.74x | 3.34x |
| First Token Latency (ms) | 360 | 410 | 435 |
| Total Latency (128 tokens, ms) | 5000 | 1800 | 1550 |
| GPU Memory (GB) | 70.5 (2x A100) | 80.0 (2x A100) | 83.0 (2x A100) |
| Acceptance Rate (%) | N/A | 72% | 88% |
| Draft Model Size | N/A | 8B parameters | 2B parameters (Eagle) |
--num-speculative-tokens | N/A | 8 | 12 |
Note: These metrics are illustrative and based on typical performance observed in similar setups. Actual numbers may vary based on specific hardware, vLLM version, model quantization, and workload characteristics.
num-speculative-tokens parameter should be tuned in conjunction with the draft model's acceptance rate. Too high a value with a low acceptance rate can be counterproductive.While vLLM provides a high-level, user-friendly interface for speculative decoding, NVIDIA's TensorRT-LLM offers an even deeper level of optimization, particularly for NVIDIA GPUs. TensorRT-LLM compiles LLMs into highly optimized inference engines, often yielding superior performance.
TensorRT-LLM also supports speculative decoding, often achieving higher throughput and lower latency than frameworks relying solely on PyTorch or other general-purpose backends. It leverages custom CUDA kernels and hardware-specific optimizations for both the draft and target model forward passes, as well as the verification logic.
For extreme performance requirements in production, especially with NVIDIA hardware, integrating TensorRT-LLM (either directly or via frameworks that use it as a backend) for speculative decoding can provide additional gains beyond what vLLM alone offers. vLLM itself can integrate with TensorRT-LLM for certain models, combining the ease of use of vLLM with the raw performance of TensorRT.
Deploying speculative decoding in production requires careful consideration of several factors to ensure stability, performance, and cost-effectiveness.
num-speculative-tokensnum-speculative-tokens too high or too low without empirical validation.num-speculative-tokens value will result in a high acceptance rate (e.g., >70-80%) while still providing significant speedup.In a production environment utilizing speculative decoding with vLLM, what is the primary trade-off when selecting a significantly smaller and faster draft model compared to the target model?
Clean AI CLI logs, build cron expressions, decode JWTs, and calculate chmod permissions offline.

SGLang vs vLLM for high-throughput LLM serving: RadixAttention tree cache reuse, multi-turn chat acceleration, and structured JSON output decoding.
Read more
Benchmarking continuous dynamic batching algorithms: how Orca, vLLM, and TGI achieve 3.8x higher throughput on GPU inference clusters.
Read more
How vLLM PagedAttention solves GPU memory waste: virtual memory paging for KV caches, prefix sharing, and chunked prefill latency benchmarks.
Read more