Nunchaku + SVDQuant: Running FLUX.1 at 4-Bit for Roughly 3x Faster Generation

FLUX.1 [dev] produces some of the best open-weight images available — and it is heavy. Twelve billion parameters means slow generations and a tight fit on consumer VRAM. Nunchaku, the inference engine built around the SVDQuant method from MIT's HAN Lab, attacks that problem at the numerical level: it runs FLUX with 4-bit weights and 4-bit activations while holding quality remarkably close to the 16-bit original, delivering large speedups and a big VRAM cut.

This guide explains how 4-bit inference stays sharp, and how to set it up in ComfyUI.

The Problem with Naive 4-Bit

Quantizing a model to 4 bits is easy; keeping it good is not. The killer issue is outliers — a small number of extreme values in the weights and activations that, when crushed into a 4-bit range, destroy quality. Most quantization schemes handle weights but choke on activation outliers, which is why aggressive image-model quantization historically looked washed-out or broke fine detail.

How SVDQuant Keeps Quality

SVDQuant's core trick is to absorb the outliers into a small, high-precision low-rank branch using a singular value decomposition. In plain terms: the bulky main path runs in 4-bit, while a lightweight, higher-precision side path carries the handful of extreme values that 4-bit can't represent. Nunchaku then fuses that low-rank branch into the kernels so it does not become its own performance tax. The result is W4A4 inference (4-bit weights, 4-bit activations) that stays close to the original model's output, not a degraded shadow of it.

For you, that means roughly 3–4x faster generation and a large VRAM reduction versus FP16 — enough to bring FLUX comfortably onto mid-range cards.

Installing Nunchaku in ComfyUI

1. Update ComfyUI.

2. Install ComfyUI-nunchaku via ComfyUI Manager (search "nunchaku") or clone it into custom_nodes/. It has a compiled backend, so the node pack includes an installation/wheel step — follow the repo's install notes for your CUDA and PyTorch versions. This is the one step where matching versions matters; read it carefully.

3. Download the SVDQuant FLUX model. These are pre-quantized svdq-int4 (or fp4 on newest GPUs) FLUX builds published by the Nunchaku team — place them where the node expects (its own models folder, per the repo).

4. Keep your normal text encoders and VAE; only the transformer is swapped for the quantized build.

The Workflow

The graph mirrors a standard FLUX text-to-image workflow, with the Nunchaku loader replacing the usual diffusion-model loader.

1. Nunchaku FLUX DiT Loader — points at the svdq-int4 model. It exposes options like cache and attention settings.

2. DualCLIPLoader (type flux) — clip_l + t5xxl as usual.

3. CLIP Text Encode — your prompt.

4. Empty Latent Image — 1024x1024 is the FLUX sweet spot.

5. KSampler — for FLUX [dev], ~20 steps, guidance ~3.5, euler + simple.

6. VAE Decode > Save Image.

Stacking with a Turbo LoRA

Nunchaku supports FLUX LoRAs, including step-distillation ones. Load a Turbo/Hyper FLUX LoRA through the Nunchaku LoRA node and drop steps to ~8. Now you are compounding two independent speedups — 4-bit compute and fewer steps — which is how people reach near-interactive FLUX generation on a single mid-range GPU.

Optimization Notes

• GPU generations matter. The newest architectures support a hardware FP4 path that is faster still; older cards use the INT4 path. Pick the model build that matches your card.

• First-run compilation. Expect a one-time warmup while kernels compile/cache; subsequent generations are the fast ones. Do not benchmark the first image.

• VRAM headroom freed by 4-bit is best spent on larger batches or a higher-res second pass, not on oversized base resolutions (FLUX still prefers ~1MP).

• Quality check: compare a 4-bit output against an FP8 render of the same seed/prompt. On most prompts the difference is minor; on very fine text or intricate patterns you may prefer FP8 for the final, using 4-bit for exploration.

Conclusion

SVDQuant is one of the more important practical advances for local generative AI because it makes the quality tier of models — not just the small ones — genuinely fast and VRAM-friendly. Set up Nunchaku once (mind the CUDA/PyTorch versions), keep a Turbo LoRA handy for previews, and reserve a higher-precision pass for hero images. It is the closest thing to a free lunch that FLUX users currently have.

Downloadable Workflow & References Nunchaku inference engine — https://github.com/mit-han-lab/nunchaku ComfyUI-nunchaku custom node — https://github.com/mit-han-lab/ComfyUI-nunchaku SVDQuant project page (MIT HAN Lab) — https://hanlab.mit.edu/projects/svdquant