Author: João Felipe De Souza

Real implementation of speculative decoding with prompt-dependent speedup analysis.

This project compares two draft→target configurations:

• GPT-2 → GPT-2-medium
• GPT-2 → GPT-2-large

and measures when speculative decoding actually beats baseline target-only decode.

For architecture and methodology, see DESIGN.md.

Speculative decoding is often described as a universal inference optimization:

1. a small draft model proposes K tokens
2. a larger target model verifies them in parallel
3. accepted tokens are committed
4. generation continues

In practice, whether this is faster depends on:

• how expensive the target model is
• how often draft and target agree
• how large K is
• runtime and cache-management overhead
• prompt structure

This project measures all of that directly.

• 12 prompt types
• K = 1, 2, 4, 8, 16
• output length: 128 tokens
• baseline target-only decode for each prompt

A more expensive target model gives speculative verification more room to amortize draft cost.

Structured, repetitive, and high-agreement prompts benefit most.

Prompts with strong speedup often have high acceptance:

• repetitive: 1.000
• dialogue: 0.896
• wikipedia_style: 0.821

But acceptance alone does not guarantee speedup:

• natural: 0.714 acceptance, still 0.950×
• technical: 0.671 acceptance, still 0.903×

This means the speedup condition is not just “high acceptance”, but:

high enough acceptance to overcome draft-model and runtime overhead

For successful prompts, large K usually wins:

• K=16 dominates for repetitive / dialogue / wikipedia / list_items
• K=8 dominates for code / repetitive_pattern / code_python / creative_fiction

For weaker prompts, large K often hurts because acceptance decays too fast.

Even in the stronger small -> large setup, some prompts still fail to beat baseline:

• natural
• technical
• code_json
• technical_ml

This confirms that speculative decoding is a conditional systems optimization, not a guaranteed speedup.

Speculative decoding works best when:

• the target model is much more expensive than the draft model
• draft-target agreement remains high over long draft horizons
• prompt structure is predictable enough to maintain acceptance

On RTX 2070:

• GPT-2 → GPT-2-medium gives only marginal average benefit
• GPT-2 → GPT-2-large gives strong benefit on many prompts
• best-case speedup reaches 1.846×

This makes speculative decoding a prompt-sensitive and model-gap-sensitive inference optimization.

speculative-decoding-impl/
├── speculative_decoding.py
├── extended_sweep.py
├── optimized_sweep.py
├── plot_speculative.py
├── plot_final.py
├── README.md
├── DESIGN.md
├── LICENSE
├── requirements.txt
├── results/
└── plots/

python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

python3 speculative_decoding.py

python3 extended_sweep.py

python3 optimized_sweep.py

python3 plot_speculative.py
python3 plot_final.py

• Single-request batch size only
• Python implementation, not fused kernels
• Consumer GPU only (RTX 2070)
• GPT-2 family only
• Prompt set is illustrative, not exhaustive

• Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2023)
• Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling (2023)