FLUX vs SDXL: 4,000 Renders Later, Here's What Actually Broke

 

I spent two weeks rendering on a single RTX 4090. FLUX.1 dev on one side, SDXL base + refiner on the other. 4,237 images total, same prompts, same seeds where I could hold them steady. I wanted one answer and I did not get a clean one. FLUX is not "better." SDXL is not "safer." They fail at different points and the failures map to completely different briefs. That messiness is the actual review.

Why MMDiT Attention Hits a Memory Wall

The math explains every OOM crash you will eat on FLUX. So bear with me.

A UNet — the thing SDXL runs on — keeps text and image as separate streams. Image latent does its own self-attention. Text tokens get injected through cross-attention at fixed points in the architecture. Text never sits at the same table as image. It whispers from a side channel. This is why SDXL leans so hard on CFG: the conditioning is indirect, and the negative prompt exists to cancel out token bleed.

FLUX threw that out. The latent gets patchified — cut into 2×2 patches, flattened, projected into the same token space as the text. Then both get concatenated into one long sequence and shoved through joint attention. Every patch attends to every token and every other patch in a single operation. The MMDiT blocks share weights between streams. Later blocks drop text and let image patches self-attend alone.

And the cost? O((N_img + N_txt)²). At 1024×1024 with 2×2 patching, N_img is about 4,096 tokens. Square that number. Add text tokens on top. That quadratic is the wall. It is why FLUX at 1536² eats roughly 2.25× the VRAM of 1024². It is why batching FLUX on a 12 GB card is a joke unless you drop to Q4 GGUF.

Then T5. FLUX ships with T5-XXL — 4.7B parameters — alongside CLIP-L. T5 is the engine behind prompt fidelity. T5 is also why your encode step eats 12 GB+ at fp16 before a single sampler step runs. I have OOM'd on the encode, not the sample, more times than I care to admit. Nobody warns you about that failure mode. The encode is the bottleneck on long prompts, not the denoising.

One more structural difference that bites at render time. SDXL predicts noise (epsilon). FLUX predicts a velocity vector along a rectified-flow trajectory. Different sampler math. You cannot plug FLUX into a DDPM sampler and expect pixels — you get static. The ModelSamplingFlux node in ComfyUI remaps the sampler to that trajectory. Forget it and you burn a render.

Joint Attention vs Cross-Attention: What the Backbone Actually Decides

People ask me "which one understands prompts better" and the answer is not vibes. It is structural.

FLUX (MMDiT):

• Image and text tokens share one attention sequence. Shared weights in MMDiT blocks. Then DiT-only self-attention blocks for image refinement.

• RoPE in 2D for positional encoding. No fixed grid. This is why FLUX handles weird aspect ratios without retraining — RoPE extrapolates. I have pushed 1344×768 and 896×1152 and both held composition where SDXL would duplicate a subject.

• T5-XXL + CLIP-L. T5 does the heavy lifting. CLIP-L contributes pooled features for global conditioning (time embedding).

• Rectified flow prediction. Guidance-distilled variants (schnell, and dev as shipped) embed the guidance so you run CFG = 1.0. No negative prompt path. People fight this. They lose.

• 16-channel VAE. More spatial detail per token. Bigger memory footprint per pixel.

SDXL (UNet):

• Cross-attention. Image self-attention blocks, plus dedicated cross-attention layers where text gets injected. Text is a conditioning signal, not a peer in the room.

• Sinusoidal PE fixed to a ~1024 grid. Drift too far from 1024² and composition degrades. Duplicated subjects. Faces where there should be none. I have the screenshots. They are ugly.

• Dual CLIP — CLIP-L + OpenCLIP bigG. Both capped at 77 tokens. No transformer LM. Prompt parsing is shallow and literal next to T5. This is why SDXL wants tag-soup prompts and FLUX wants full sentences.

• Epsilon prediction. Standard DDPM/DDIM/UniPC samplers. Needs CFG 7–8 and a real negative prompt.

• 4-channel VAE. Two-stage pipeline with a refiner for high-frequency detail.

So: SDXL is a UNet with conditioning tricks bolted on. FLUX is one transformer treating text and image as a joint language. FLUX wins prompt adherence and multi-subject separation. SDXL wins on ecosystem — ControlNet, IP-Adapter, LoRA coverage are years ahead on the UNet path. That gap is closing. It is not closed.

Concrete Implementation: What I Actually Run

Drop these into ComfyUI. Both have shipped to clients.

SDXL text-to-image (base + refiner, my daily driver):

Positive:  a weathered fisherman mending a net on a misty wooden dock,
           golden hour, 85mm, shallow depth of field, film grain, kodak portra
Negative:  blurry, deformed hands, extra fingers, watermark, lowres, jpeg artifacts
CFG:       7.5
Sampler:   dpmpp_2m
Scheduler: karras
Steps:     30
Resolution:1024 x 1024  (or 1216 x 832 portrait)
Checkpoint:sd_xl_base_1.0.safetensors
Refiner:   sd_xl_refiner_1.0.safetensors @ denoise 0.3, steps 10

FLUX.1 [dev] text-to-image (guidance-distilled, CFG 1):

Prompt:   A weathered fisherman mending a net on a misty wooden dock at golden hour,
          85mm lens, shallow depth of field, subtle film grain, kodak portra 400,
          atmospheric volumetric fog rolling off the water
CFG:      1.0           # distilled — do not raise this
Sampler:   euler          # simple ODE solver, matches the flow trajectory
Scheduler: simple
Steps:    20             # 12-25 sweet spot; 8 for schnell
Resolution:1024 x 1024
Diffusion model: flux1-dev.safetensors

Three things I learned the hard way:

CFG 1.0 on FLUX dev is not a suggestion. Past ~1.1 you get color burn and oversaturation. The undistilled dev and pro variants accept real CFG (3.5–4.0) but you pay for it in steps and VRAM. I tried running dev at CFG 3.5 for a week thinking I was being clever. I was not being clever.

FLUX wants prose. SDXL wants tags. Swap grammar between them and both underperform. I ran the same 40-word prompt on both once. FLUX read it as a sentence and composed accordingly. SDXL read it as a bag of tokens, grabbed the loudest ones, and ignored the rest.

SDXL benefits from the refiner at high denoise. Base at denoise 1.0, chain a refiner KSampler at 0.3, steps 10. Crispness jumps noticeably. FLUX has no refiner — detail comes from the VAE and step count. There is no secret second pass to save you.

Running in ComfyUI

FLUX (guidance-distilled dev):

• Diffusion weights → ComfyUI/models/diffusion_models/flux1-dev.safetensors. GGUF variants (flux1-dev-Q4_K_S.gguf, Q5_K_M, Q8_0) go in the same folder.

• Text encoders → ComfyUI/models/text_encoders/. You need both t5xxl_fp16.safetensors and clip_l.safetensors. On 8 GB cards, grab t5xxl_fp8_e4m3fn.safetensors — halves T5 memory, roughly 1% fidelity loss, I cannot spot it in side-by-side tests.

• VAE → ComfyUI/models/vae/ae.safetensors. FLUX's 16-channel autoencoder. Do not point this at an SDXL VAE. It will not load. I tried. Wasted an afternoon.

• VRAM footprints I measured on a 4090, 1024², dev:

  • FP16 (full): ~23.5 GB. Fits on 24 GB. Tight. No batching.

  • FP8 (e4m3fn): ~12.4 GB. Comfortable on 12 GB.

  • GGUF Q8: ~11.2 GB. Near-lossless. My production default.

  • GGUF Q4_K_S: ~7.1 GB. Runs on 8 GB. Soft fine detail. Fine for thumbnails and client previews, not finals.

• Custom nodes: native ComfyUI handles FLUX T2I out of the box. For GGUF you need ComfyUI-GGUF from city96. That is it.

SDXL:

• Checkpoints → ComfyUI/models/checkpoints/sd_xl_base_1.0.safetensors and sd_xl_refiner_1.0.safetensors.

• LoRAs → ComfyUI/models/loras/. SDXL and FLUX LoRAs share the folder but use different loader nodes. Do not mix the loaders — a FLUX LoRA on an SDXL loader silently produces garbage.

• VRAM footprint, 1024²: ~8–10 GB FP16 base-only, ~12 GB with refiner staged. Roughly half of FLUX.

• No T5. Dual CLIP is bundled in the checkpoint or loads from ComfyUI/models/clip/.

The Graph Topology

SDXL base + refiner — the pipeline I actually ship:

Load Checkpoint (SDXL) → CLIPTextEncode (pos + neg) → EmptyLatentImage (1024×1024) → KSampler (base, denoise 1.0, steps 30, CFG 7.5, dpmpp_2m karras) → branch the latent into KSampler (refiner, denoise 0.3, steps 10, same CFG) → VAEDecode → SaveImage. The refiner reads the base latent and tightens high-frequency detail. Use a LatentPass reroute — the refiner needs the same latent shape, and a mismatch silently produces a black frame. I have shipped that black frame. Once.

FLUX [dev] — guidance distilled:

Load Diffusion Model (flux1-dev) → DualCLIPLoader (t5xxl + clip_l) → CLIPTextEncode (single positive — no negative node, do not wire one) → EmptyLatentImage → ModelSamplingFlux (sets the flow-matching shift) → KSampler (CFG 1.0, euler, steps 20) → VAEDecode (FLUX ae.safetensors) → SaveImage. ModelSamplingFlux is the node people forget. Without it FLUX outputs noise. Not bad images. Noise. GGUF loads via UnetLoaderGGUF instead of Load Diffusion Model.

Optimization & Production Tips

• Resolution is not free on either backbone. FLUX tolerates 768²–1536² natively thanks to RoPE, but joint attention cost scales with patch count. 1024² is the cost/composition sweet spot. 1536² roughly 2.25× the VRAM. SDXL wants 1024² ± 10%. Wider aspect ratios (1344×768) work but past 1.3× ratio I see duplicated subjects on SDXL — not on FLUX. That difference is the RoPE vs sinusoidal gap showing up in the output.

• T5 token budget is a silent killer. ComfyUI passes up to ~256 tokens to T5 by default. Long prompts cost VRAM on the encode step, not the sample step. If you OOM during encode, you hit the T5 wall. Switch to fp8 T5 — same fix as the 8 GB card path, and the fidelity hit is invisible in practice.

• Seed stability differs and it matters for iteration. FLUX is more seed-stable across step-count changes because flow matching is a smoother trajectory. I have changed FLUX from 20 to 28 steps on the same seed and composition held. SDXL can flip composition between 25 and 35 steps on the same seed. Lock seeds for A/B comparisons. Never compare across sampler changes — that is not a fair test and I see people do it constantly.

• Batching. FLUX batching is brutal. Joint attention × batch × quadratic patch count. Batch=2 at 1024² on FP8 ate ~18 GB in my tests. Batch FLUX only with GGUF Q4 on 12 GB cards, and even then keep batch=2 max. SDXL batches comfortably — batch=4 fits 12 GB. If your pipeline needs throughput over fidelity, SDXL wins this round hard.

• Offloading. --lowvram for FLUX on 8 GB cards pages the model to system RAM between steps. It runs. It is 2-3× slower. Fine for overnight batches, painful for iteration. SDXL on --lowvram is almost unusable for iteration speed — use --medvram with FP16 instead.

• Negative prompts on FLUX distilled are a no-op. Do not wire a negative node. It is ignored, wastes graph space, and confuses anyone reading your workflow later. On undistilled FLUX or SDXL the negative is load-bearing — keep it under 30 tokens or you dilute the positive.

Where This Lands in a Real Pipeline

Here is where I land after 4,237 renders and a lot of late nights. FLUX and SDXL are not competitors in a pipeline. They are different stages and different briefs. FLUX is the hero-shot engine. Prompt-heavy concept art, multi-subject editorial, anything where the brief is "make the image actually look like the description." Its weak ecosystem is real — ControlNet support exists but lags, IP-Adapter and LoRA coverage is thin and often finetuned on specific sub-trainers. FLUX struggles for pose-locked or identity-locked work today. SDXL is the control backbone. Pair it with ControlNet Union for pose, IP-Adapter Plus for face/subject identity, ADetailer-style inpainting for hands and faces. When a brief needs both fidelity and control, the pragmatic stack is FLUX for the hero render and SDXL + ControlNet for the iteration and refinement passes that need structural locking. Upscaling is backbone-agnostic — 4x-UltraSharp or ECM applied post-VAE works for both. Do not try to make one backbone do everything. I tried. I wasted a week on a graph that should have been two graphs.

Downloadable Workflow & References

• [Official ComfyUI FLUX Example Workflows (GitHub)] — https://github.com/comfyanonymous/ComfyUI_examples/tree/master/flux

• [Official ComfyUI SDXL Example Workflows (GitHub)] — https://github.com/comfyanonymous/ComfyUI_examples/tree/master/sdxl

• [FLUX.1 dev Weights (Black Forest Labs, HuggingFace)] — https://huggingface.co/black-forest-labs/FLUX.1-dev

• [SDXL Base + Refiner Weights (HuggingFace)] — https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0

• [ComfyUI GGUF Quantized Loader (city96, GitHub)] — https://github.com/city96/ComfyUI-GGUF

---