Repository navigation
Support 1 * 128 and 128 * 128 block-wise quant? #38
Description
In the CUDA 12.9 cuBLASLt documentation, I noticed support for 1×128 and 128×128 block-wise quantization methods. However, I found that nvmath-python currently lacks bindings for this type of quantize approach. I wonder do we have any plan for support this approach?
https://docs.nvidia.com/cuda/cublas/index.html#cublasltmatmulmatrixscale-t
Activity
szkarpinski commentedon Aug 6, 2025
Hi @zfan2356 , thank you for raising this issue. The lower-level
nvmath.bindingsfor those scaling types should be included in the next release of nvmath-python. I'll also start working on adding support to our higher-level host APIs innvmath.linalg.advanced.matmul.zfan2356 commentedon Aug 11, 2025
@szkarpinski Thanks for your reply! I also recommend supporting the pointer array mode for Group GEMM. Both of these features are commonly used in modern LLM pretrain. nvmath-python is a user-friendly Python library, and I hope it can support these two features as soon as possible. Thank you very much!