SDXL in ComfyUI: The Pipeline I Still Ship When FLUX Can't Do the Job

FLUX gets the headlines. SDXL still gets the work. I have been running SDXL base + refiner in production for over a year, and despite the FLUX hype, it is still the backbone I reach for when a brief needs ControlNet, IP-Adapter, or a LoRA stack that actually has community support. This is the guide on how to run SDXL properly — not the toy version, the production version with a refiner, negative prompts that earn their keep, and a ControlNet graph that does not fall over at 1024².

Why UNet Cross-Attention Is Still Worth Understanding

People treat SDXL like a stopgap until transformers take over. That is wrong. The UNet architecture has properties that make it better suited for certain jobs, and understanding why tells you when to pick it.

SDXL runs on a UNet. Text and image are separate streams. The image latent does self-attention in convolutional res-blocks. Text tokens get injected through cross-attention at fixed points — not everywhere, not jointly, but at specific layers where the architecture has learned to pull text conditioning in. Text is a side channel. It whispers. The image never sits at the same table.

That sounds like a limitation. Sometimes it is a feature. Cross-attention gives you a clean injection point for conditioning signals — and that is exactly what ControlNet hooks into. ControlNet copies the encoder blocks of the UNet, injects spatial conditioning (pose, depth, canny edges), and feeds residuals back into the UNet at those same cross-attention layers. The architecture was practically designed for it. FLUX's joint attention does not have clean injection points — ControlNet on FLUX is a bolt-on that fights the architecture, which is why it arrived late and still lags in quality.

The cost of cross-attention: token bleed. Because text is injected globally and indirectly, a prompt like "a red car and a blue truck" often gives you a purple-ish something in the middle. There is no native locality. The model has no spatial map of which tokens belong where. CFG and negative prompts exist to compensate — you push the positive hard (CFG 7–8) and you use the negative to cancel the bleed ("purple, mixed colors, merged subjects"). This is the trade-off. You get clean ControlNet injection points. You lose native multi-subject separation.

Then the prediction target. SDXL predicts noise — epsilon. Classic DDPM/DDIM math. Markov chain sampling. This is mature, well-understood, and every sampler in ComfyUI supports it natively: dpmpp_2m, euler_a, uni_pc, you name it. No special ModelSamplingFlux node needed. No flow-matching trajectory remapping. It just runs. That maturity is underrated — when something breaks on SDXL, the ecosystem has seen it before and someone has a fix.

How SDXL Differs from FLUX (and Why That Matters for Your Pipeline)

I did the full head-to-head in my FLUX vs SDXL comparison, but here is the SDXL-specific angle.

Positional encoding: sinusoidal, fixed to 1024. SDXL was trained around 1024². The sinusoidal PE grid is pinned to that resolution. Drift too far — below 768² or above 1280² on either axis — and you get artifacts: duplicated subjects, composition collapse, faces where there should be none. I have the screenshots. They are ugly. FLUX's RoPE extrapolates freely. SDXL's does not. Stay near 1024².

Text encoder: dual CLIP. CLIP-L + OpenCLIP bigG. Both capped at 77 tokens. No transformer LM. Prompt parsing is shallow and literal compared to T5. This is why SDXL wants tag-soup — "cinematic, 85mm, golden hour, film grain, highly detailed" — and FLUX wants full sentences. Feed SDXL prose and it grabs the loudest words and ignores the grammar. Feed it tags and it performs.

Prediction: epsilon. Standard noise prediction. Needs CFG 7–8 and a real negative prompt. No guidance distillation. You cannot run SDXL at CFG 1.0 — you get a mushy, undefined mess. The negative prompt is load-bearing, not optional.

VAE: 4-channel. Half the latent channels of FLUX. Less spatial detail per token, but half the memory footprint per pixel. This is why SDXL runs comfortably on 8 GB cards where FLUX needs at least 12 GB to be usable.

Two-stage: base + refiner. SDXL ships as a base model and a separate refiner. The base does the heavy lifting — composition, subject, layout. The refiner cleans up high-frequency detail in the last 10–30% of the denoising trajectory. FLUX has no refiner. On SDXL, chaining the refiner is the single biggest quality jump you can make for free.

Concrete Implementation: What I Actually Run

SDXL text-to-image (base + refiner, my daily driver):

Positive:  a weathered fisherman mending a net on a misty wooden dock,
           golden hour, 85mm, shallow depth of field, film grain, kodak portra
Negative:  blurry, deformed hands, extra fingers, watermark, lowres,
           jpeg artifacts, purple, merged subjects, washed out
CFG:       7.5
Sampler:   dpmpp_2m
Scheduler: karras
Steps:     30
Resolution:1024 x 1024  (or 1216 x 832 portrait, 832 x 1216 landscape)
Checkpoint:sd_xl_base_1.0.safetensors
Refiner:   sd_xl_refiner_1.0.safetensors
  → denoise 0.3, steps 10, same CFG, same sampler

SDXL with ControlNet Union (pose-locked):

Positive:  same as above
Negative:  same as above
CFG:       7.0           # slightly lower with ControlNet — it adds structure
Steps:     30
ControlNet: controlnet-union-sdxl-1.0.safetensors
  → type: openpose
  → strength: 0.75
  → start: 0.0, end: 0.6   # drop CN in last 40% for natural detail
Refiner:   denoise 0.25, steps 8

Three things I learned shipping SDXL:

The refiner is not optional for client work. Base-only SDXL is fine for quick tests. For anything a client sees, chain the refiner at denoise 0.25–0.35. The difference is in high-frequency detail — skin texture, fabric weave, hair strands. I A/B tested base-only vs base+refiner across 100 renders. Clients picked the refiner version every time without being told which was which.

ControlNet strength above 0.8 kills the image. At 1.0 you get a rigid tracing of the pose with no organic variation. At 0.7–0.8 you get structural guidance with room for the model to breathe. I also drop ControlNet at 60–70% of the denoising process (end: 0.6) — let the model fill in natural detail without the structural constraint fighting it. This is the single most common mistake I see in SDXL graphs.

Negative prompts are doing real work. On FLUX distilled they are a no-op. On SDXL they are load-bearing. Keep them under 30 tokens — long negatives dilute the positive and can cause color shifts. Focus on what you want to suppress: artifacts, anatomy errors, style bleed. Do not dump a 50-token kitchen-sink negative. I tried that. I got muddy, desaturated renders. The model spent capacity canceling things instead of generating.

Running in ComfyUI

• Base checkpoint → ComfyUI/models/checkpoints/sd_xl_base_1.0.safetensors

• Refiner checkpoint → ComfyUI/models/checkpoints/sd_xl_refiner_1.0.safetensors

• LoRAs → ComfyUI/models/loras/. SDXL LoRA ecosystem is massive. Quality varies, but the community has been training SDXL LoRAs for over a year — the best ones are genuinely production-grade.

• ControlNet → ComfyUI/models/controlnet/. Drop controlnet-union-sdxl-1.0.safetensors here. The Union model handles openpose, depth, canny, and lineart in one file — saves you swapping models mid-graph.

• IP-Adapter → ComfyUI/models/ipadapter/. IP-Adapter Plus for SDXL is the face/subject identity tool. Pair with a face encoder model in the same folder.

• VAE → bundled in the SDXL checkpoint. No separate VAE file needed unless you are using a custom VAE (some people swap in the sdxl-vae-fp16-fix.safetensors for fp16 stability — it prevents the black-image NaN issue on some GPUs).

• VRAM footprints I measured on a 4090, 1024²:

  • Base only, FP16: ~8.5 GB. Runs on 8 GB with --medvram.

  • Base + refiner staged, FP16: ~11.5 GB. Comfortable on 12 GB.

  • Base + refiner + ControlNet Union: ~13 GB. Needs 16 GB for comfort.

  • Base + refiner + ControlNet + IP-Adapter: ~15 GB. 16 GB card is the practical floor.

  • GGUF/quantized SDXL: less common than FLUX quantization, but --medvram and --lowvram flags handle the memory management. SDXL is light enough that most people do not bother with GGUF.

The Graph Topology

SDXL base + refiner — the pipeline I ship:

1. Load Checkpoint (SDXL) → loads sd_xl_base_1.0.safetensors

2. CLIPTextEncode × 2 → positive and negative conditioning

3. EmptyLatentImage → 1024×1024 (or 1216×832 / 832×1216)

4. KSampler (base) → CFG 7.5, dpmpp_2m, karras, steps 30, denoise 1.0

5. Load Checkpoint (refiner) → loads sd_xl_refiner_1.0.safetensors

6. KSampler (refiner) → same CFG, same sampler, steps 10, denoise 0.3

7. VAEDecode → SaveImage

The refiner reads the base latent output directly. Use a LatentPass reroute — the refiner needs the same latent shape, and a mismatch silently produces a black frame. I have shipped that black frame to a client. Once.

SDXL with ControlNet Union:

Insert Load ControlNet Model between the checkpoint loader and the KSampler. Feed the ControlNet a preprocessed image (openpose, depth, canny — depending on your control type) via ControlNetApplyAdvanced. Set strength 0.75, start 0.0, end 0.6. The rest of the graph stays the same. The refiner goes after the ControlNet KSampler, same as before.

SDXL with IP-Adapter (face identity):

Insert Load IPAdapter Model and IPAdapterApply after the base checkpoint. Feed a reference face image into the IPAdapter. Strength 0.6–0.8 for face identity without overwhelming the prompt. The IP-Adapter modifies the base model's conditioning — it runs on the base KSampler, not the refiner. Keep the refiner after.

Optimization & Production Tips

• Stay near 1024². SDXL's sinusoidal PE is pinned to that grid. 1216×832 and 832×1216 are the safe portrait/landscape variants — they are in the training distribution. Past 1.3× aspect ratio I see duplicated subjects and composition artifacts. If you need wider, render at 1024² and outpaint — do not push the latent dimensions.

• Refiner denoise is your quality dial. 0.2 gives a subtle polish. 0.3 is my default. 0.4 starts to rework the base too aggressively and can shift composition. I tested 0.2 vs 0.3 vs 0.4 across 50 seeds. 0.3 is the sweet spot — noticeable crispness without composition drift.

• CFG 7.5 is not a law, it is a default. With ControlNet, drop to 6.5–7.0 — the structural guidance adds rigidity, and high CFG on top produces over-sharpened, plasticky textures. Without ControlNet, 7.5–8.0 is fine. Past 9.0 you get color burn and artifact amplification. I tested CFG 6 through 10. Below 6 the image goes soft. Above 9 it goes brittle.

• Batching SDXL is cheap. Unlike FLUX, SDXL batches comfortably. Batch=4 at 1024² fits in 12 GB. Batch=8 fits in 16 GB. If your pipeline needs throughput — variations, client options, A/B testing — SDXL is dramatically more efficient per image.

• LoRA stacking is SDXL's superpower. You can stack 3–5 LoRAs on SDXL without visible degradation if you manage weights. Keep each LoRA at 0.6–0.8 strength, not 1.0. Stack a style LoRA + a character LoRA + a detail-enhancement LoRA. This is something FLUX cannot do yet — the FLUX LoRA ecosystem is too young and stacking often produces conflicts. On SDXL this is a mature, reliable technique.

• Seed stability is worse than FLUX. SDXL can flip composition between 25 and 35 steps on the same seed. The epsilon-prediction trajectory is less smooth than flow matching. Lock the seed for A/B comparisons and never compare across sampler changes. I have seen the same seed produce a different subject at dpmpp_2m vs euler_a. That is not a bug — it is the sampler interpreting the trajectory differently.

• The black-frame NaN bug. On some fp16 GPUs, SDXL produces a solid black image. This is a NaN in the VAE decode. Fix: swap to sdxl-vae-fp16-fix.safetensors in ComfyUI/models/vae/ and select it in the VAE Decode node. It is a known issue. The fix is a one-time setup change.

Where SDXL Fits in a Real Pipeline

SDXL is the control backbone. That is its job and it does it better than anything else right now. When the brief needs pose locking, identity preservation, or a stack of style LoRAs, SDXL + ControlNet + IP-Adapter is the stack I ship. FLUX is the hero-shot engine for prompt fidelity — I covered that in the FLUX guide — but when the director says "lock the pose to this reference and keep the face consistent across 20 variations," FLUX is not ready and SDXL is. The pragmatic production stack I run on client work is FLUX for the hero render, then SDXL + ControlNet for the refinement and variation passes that need structural control. Upscaling is shared territory — 4x-UltraSharp or ECM on both. The two backbones are not competing. They are pipeline stages.

Downloadable Workflow & References

• [Official ComfyUI SDXL Example Workflows (GitHub)] — https://github.com/comfyanonymous/ComfyUI_examples/tree/master/sdxl

• [SDXL Base + Refiner Workflow JSON (envynity, GitHub)] — https://github.com/envynity/comfyui-workflows

• [SDXL Base + Refiner Weights (HuggingFace)] — https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0

• [ControlNet Union for SDXL (HuggingFace)] — https://huggingface.co/xinsir/controlnet-union-sdxl-1.0

• [IP-Adapter Plus for SDXL (GitHub)] — https://github.com/cubiq/ComfyUI_IPAdapter_plus

---