Skip to content

feat: packed int4 convrot model loading and dispatch - #2010

Draft
noctrex wants to merge 2 commits into
leejet:masterfrom
noctrex:convrot-w4-family
Draft

feat: packed int4 convrot model loading and dispatch#2010
noctrex wants to merge 2 commits into
leejet:masterfrom
noctrex:convrot-w4-family

Conversation

@noctrex

@noctrex noctrex commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds direct loading and execution of ComfyUI packed int4 convrot checkpoints (convrot_w4a4, asym_w4a8_int8). The packed nibble weights stream to the backend unconverted: no load-time reblock, no sidecar cache, roughly half a byte per weight parameter. Formats are documented in docs/int4_convrot.md (added here).

Loader/graph side:

  • packed-w4 tensor recognition in safetensors IO; packed payloads and companion tensors (row scales, codebook, s_channel, s_rel) pass through raw
  • block-builder support mirroring the int8 convrot chain: the activation rotation/quantize is shared with int8 tensorwise and cached per (input, group size)
  • activation quantize gated on unit scale for int4 inputs
  • docs: docs/int4_convrot.md

GGML dependency

The kernels live in the GGML submodule. The ggml-side changes are submitted as leejet/ggml#10 ("ggml : add packed int4 convrot mul_mat for CPU, CUDA, HIP and Vulkan"), which includes the HIP quantize-gate fix this PR's pin depends on.

This PR therefore bumps the ggml pin to a branch on my fork. Per the contribution guidelines, ggml updates are validated by the maintainer - keeping this PR in draft until that side is settled; the pin can be retargeted the moment the ggml content lands in leejet/ggml.

Backend notes (disclosure)

  • int4 (both kinds) runs on CPU, CUDA, HIP and Vulkan; convrot group sizes 64 and 256 verified.
  • On HIP, plain int8_tensorwise models and int8 convrot text encoders currently fall back to CPU; that enablement is the separate INT8 HIP/BLAS ggml PR's scope and does not affect int4 models.

Verification

All runs on RX 7900 XTX (gfx1100), Windows, ROCm 10 / Vulkan, 1024x1024, 8 steps, seed 42:

  • FLUX.2-klein-4b int4: Vulkan 11.9 s sampling (1.24 s/it, faster than the int8 baseline's 12.3 s), ROCm 9.56 s post-fix
  • FLUX.2-klein-4b w4a8: Vulkan 13.9 s, ROCm 12.40 s
  • FLUX.2-klein-4b int8 baseline: Vulkan 12.3 s (visual reference; images equivalent)
  • z_image-turbo int4 (gs=256 regression check): ROCm 14.38 s
  • Loader fixture against the merged loader: W4A4 bitwise vs CPU oracle, W4A8 within the int8-requant gate, raw packed passthrough intact
  • zero unsupported-node/CPU-fallback warnings for the int4 paths; images on-prompt and visually equivalent to int8

Linked context

Checklist

LLM Disclosure: Yes, used GLM-5.3-Flash

@coderabbitai

coderabbitai Bot commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@BBQ2077

BBQ2077 commented Oct 18, 2026

Copy link
Copy Markdown

Hi, does this PR also cover loading the MOSS-Image Qwen3.5-9B W4A8 GPTQ ConvRot text encoder through --llm?

Model repository:
https://huggingface.co/nif0/MOSS-Image-SFT-ComfyUI

Text encoder:
qwen3.5_9b_moss_image_sft_w4a8_gptq.safetensors

Environment

  • Windows 11
  • AMD Radeon RX 9070 (gfx1201), 16 GB VRAM
  • ROCm 10.1.0
  • Official prebuilt release: master-973-f0aac52

Observed failure

The current release fails metadata validation when loading this encoder. For example:

[E] Conditioner model tensor 'text_encoders.llm.model.layers.0.linear_attn.in_proj_qkv.weight' has wrong shape in model metadata: got [2048, 8192, 1, 1], expected [4096, 8192, 1, 1] --- model_manager.cpp:798

The physical input dimension appears to be half the expected logical dimension, consistent with packed 4-bit weights not being interpreted by the current loading path.

Replacing this encoder with Qwen3.5-9B-Q4_K_M.gguf, together with the Comfy-Org MOSS-SFT INT8 diffusion model and the external Qwen3.5 tokenizer, successfully generates images on the same machine.

I have not tested this PR branch yet. Could you confirm whether its packed W4A8 support also applies to this Qwen3.5-9B text encoder, including execution on HIP/ROCm?

Thank you!

@noctrex
feat: packed-w4 convrot model loading and dispatch
4ff20cd 
- loader: packed-w4 convrot tensor recognition; packed payloads stream
  to the backend unconverted (no load-time reblock, no sidecar cache)
- safetensors IO, tensor storage, block builder support for the
  packed-w4 payloads
- activation quantize gated on unit scale for w4 inputs
- ggml: pointer to the squashed w4 convrot kernel commit (6a7f9c31,
  includes group-size-64 quantize support as h64-PR overlap)
@noctrex
docs: add int4 convrot checkpoint documentation
2f4b8cf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

See More

Successfully merging this pull request may close these issues.

2 participants

@noctrex @BBQ2077