Feather

Feather is a high-performance emulation library that brings FP8 (E5M2 & E4M3) precision arithmetic to older GPU architectures (Ampere, Turing, Volta) that lack native hardware support.

By implementing custom bit-packing and optimized Triton kernels, Feather bypasses the memory bandwidth bottleneck, achieving 3x speedups over native PyTorch FP32 operations on memory-bound workloads.

Modern deep learning benefits enormously from low-precision arithmetic, but native hardware support is gated behind the latest GPUs:

• Native FP8: Exclusive to Ada Lovelace (RTX 40-series) and Hopper (H100).
• Older Hardware (RTX 3090, A100, V100): Forced to use FP16 or FP32, leaving huge bandwidth potential on the table.
• NOTE: Also supports FP16 packing.

• Software-Based FP8: Runs E5M2 and E4M3 on any CUDA GPU (RTX 20/30 series supported).
• 3x Speedup: Validated 3.37x speedup on large GEMV operations.
• 1.5x Speedup: Validated 1.5x speedup with naive kernel implementation compared to FP32 sdpa kernel with HF
• Correctness:
  • E5M2: Direct bit-manipulation for maximum speed.
  • E4M3: Full software emulation (rebias + denorm flushing) for stability.

Performance measured on NVIDIA RTX 3050 (Ampere), comparing PyTorch Native (FP32) vs. Feather (FP8).

• Task: Large Scale Matrix-Vector Multiplication (GEMV)

• Task: Flash Attention (FP8 vs FP16 vs FP32)

Performance comparison between torch.nn.functional.scaled_dot_product_attention (in FP32 and FP16) and Feather's custom Triton kernel (FP8).

More results are in results/

• TinyLlama/TinyLlama-1.1B-Chat-v1.0 is now available for Inference!

• Python 3.10+
• CUDA-capable GPU
• uv package manager

# Clone the repository
git clone https://github.com/SuriyaaMM/feather
cd feather

# Install dependencies
uv sync 
source .venv/bin/activate

# Run benchmarks
uv run pytest benchmark/bench_FILE_NAME.py --benchmark-histogram=benchmark_results
# check benchmark/bench_*.py for filenames

• GEMV operation

import torch
import feather

# 1. Create standard data
matrix = torch.randn(16384, 16384, dtype=torch.float16, device='cuda')
vector = torch.randn(16384, dtype=torch.float16, device='cuda')

# 2. Pack data (Offline / Pre-processing step)
# Compresses memory usage by 2x (vs FP16) or 4x (vs FP32)
m_packed = feather.pack_fp8_tensor(matrix, mode="E5M2").cuda()
v_packed = feather.pack_fp8_tensor(vector, mode="E5M2").cuda()

# 3. Run Fast GEMV
# Returns FP32 result with FP32 accumulation
result = feather.gemv(m_packed, v_packed, original_shape=matrix.shape)

• LLM Inference

from feather.models.tiny_llama import *
from feather.models.pack_weights import *
from transformers import AutoTokenizer
import torch
import time

model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

# offline packing
packed_weights = pack_llama_weights(
    hf_model=model_name,
    savefile="tinyllama_fp8.pt",
    dtype=torch.float16
)

# set configuration (TinyLlama-1.1B-Chat-v1.0)
cfg = FeatherTinyLlamaConfig(
    hidden_size=2048,
    num_hidden_layers=22,
    num_attention_heads=32,
    num_key_value_heads=4,
    intermediate_size=5632,
    vocab_size=32000,
    max_position_embeddings=2048,
    kv_block_size=16,
    max_num_blocks=256,
    num_heads_per_chunk=1,
)
# initialise the model
model = FeatherTinyLlama(cfg, packed_weights, device="cuda")
tokenizer = AutoTokenizer.from_pretrained(model_name)

# set prompt
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "tell me about united states of america"},
]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
ids = tokenizer.encode(prompt)

# generate tokens
with torch.inference_mode():
    out_ids = model.generate(
      ids, max_new_tokens=256, temperature=0.0, top_p=0.9, repetition_penalty=1.15
    )

• Tiny-Llama-1.1 Inference in FP8
• Cuda-Graph Optimisation
• Benchmark with VLLM
• Llama Family Implementation
• Quantisation Mechanics (Block Level etc)
• Improve Sampler

MIT License - see LICENSE for details.