Feather is a high-performance emulation library that brings FP8 (E5M2 & E4M3) precision arithmetic to older GPU architectures (Ampere, Turing, Volta) that lack native hardware support.
By implementing custom bit-packing and optimized Triton kernels, Feather bypasses the memory bandwidth bottleneck, achieving 3x speedups over native PyTorch FP32 operations on memory-bound workloads.
Modern deep learning benefits enormously from low-precision arithmetic, but native hardware support is gated behind the latest GPUs:
- Native FP8: Exclusive to Ada Lovelace (RTX 40-series) and Hopper (H100).
- Older Hardware (RTX 3090, A100, V100): Forced to use FP16 or FP32, leaving huge bandwidth potential on the table.
- NOTE: Also supports FP16 packing.
- Software-Based FP8: Runs
E5M2andE4M3on any CUDA GPU (RTX 20/30 series supported). - 3x Speedup: Validated 3.37x speedup on large GEMV operations.
- Correctness:
- E5M2: Direct bit-manipulation for maximum speed.
- E4M3: Full software emulation (rebias + denorm flushing) for stability.
Performance measured on NVIDIA RTX 3050 (Ampere), comparing PyTorch Native (FP32) vs. Feather (FP8).
- Task: Large Scale Matrix-Vector Multiplication (GEMV)
| Matrix Shape | PyTorch (FP32) | Feather (FP8-E5M2) | Feather (FP8-E4M3) |
|---|---|---|---|
| 8192 x 8192 | 1,389 |
431 |
720 |
| 16384 x 8192 | 2,862 |
841 |
1,368 |
| 16384 x 16384 | 5,635 |
1,679 |
2,703 |
- Task: Flash Attention (FP8 vs FP16 vs FP32)
Performance comparison between torch.nn.functional.scaled_dot_product_attention (in FP32 and FP16) and Feather's custom Triton kernel (FP8).
| Head Dim | Sequence Length | PyTorch (FP32) | PyTorch (FP16) | Feather (FP8) |
|---|---|---|---|---|
| 64 | 128 | 64.9 |
60.9 |
35.7 |
| 256 | 50.1 |
61.2 |
15.5 |
|
| 512 | 85.0 |
93.7 |
26.0 |
|
| 1024 | 252.4 |
307.5 |
50.6 |
|
| 4096 | 3,707 |
4,738 |
600.1 |
|
| 8192 | 15,604 |
18,827 |
2,056 |
|
| 256 | 128 | 44.7 |
61.2 |
13.8 |
| 256 | 46.8 |
62.7 |
57.2 |
|
| 512 | 148.7 |
185.6 |
95.3 |
|
| 1024 | 428.6 |
511.0 |
216.9 |
|
| 4096 | 5,560 |
6,223 |
3,047 |
|
| 8192 | 22,129 |
24,361 |
11,500 |
|
| 512 | 128 | 44.2 |
61.4 |
14.2 |
| 256 | 88.8 |
120.3 |
74.4 |
|
| 512 | 191.4 |
236.8 |
121.5 |
|
| 1024 | 702.1 |
813.3 |
269.1 |
|
| 4096 | 8,966 |
9,724 |
3,090 |
|
| 8192 | 33,290 |
35,965 |
9,887 |
More results are in results/
- Python 3.10+
- CUDA-capable GPU
- uv package manager
# Clone the repository
git clone https://github.com/SuriyaaMM/feather
cd feather
# Install dependencies
uv sync
source .venv/bin/activate
# Run benchmarks
uv run pytest benchmark/bench_FILE_NAME.py --benchmark-histogram=benchmark_results
# check benchmark/bench_*.py for filenamesimport torch
import feather
# 1. Create standard data
matrix = torch.randn(16384, 16384, dtype=torch.float16, device='cuda')
vector = torch.randn(16384, dtype=torch.float16, device='cuda')
# 2. Pack data (Offline / Pre-processing step)
# Compresses memory usage by 2x (vs FP16) or 4x (vs FP32)
m_packed = feather.pack_fp8_tensor(matrix, mode="E5M2").cuda()
v_packed = feather.pack_fp8_tensor(vector, mode="E5M2").cuda()
# 3. Run Fast GEMV
# Returns FP32 result with FP32 accumulation
result = feather.gemv(m_packed, v_packed, original_shape=matrix.shape)- SDPA Kernels
- Vulkan Support
- Integration with Torch (Functional Forward Propagation)
- Integration with Torch (Functional Backward Propagation)
MIT License - see LICENSE for details.