ControlNet gives an SDXL workflow structural guidance from an image, such as edges, depth or pose. This guide shows the smallest useful ComfyUI graph and how to diagnose common connection errors. It is an editorial workflow reference, not a record of a hardware benchmark.
Visual guide: Trace an SDXL Canny control signal
How to use this map
- Start with an SDXL graph that already saves an image. Add the control branch as a separate change so you can distinguish base failures from control failures.
- Inspect the Canny preview directly. Missing subject edges or excessive background lines can explain a weak result before any strength tuning.
- Connect the processed control map and compatible model through Apply ControlNet to the conditioning used by the sampler. The map alone is not the complete control signal.
- Hold the prompt and seed steady for the first strength comparison. Check whether the required layout is preserved and whether unwanted edges are being copied.
Stop and check: Do not mix control weights from a different model family. Save the original reference, preprocessor settings and control preview with the output.
Next practical guides: SDXL in ComfyUI: First Image, Base and Refiner Workflow · ComfyUI Queue Errors: Stuck Prompts and Missing Nodes
Implementation reference: Official ControlNet examples. Use the matching version-specific example alongside this editorial map.
Before you start
Install a current ComfyUI release and confirm that a basic SDXL text-to-image graph completes first. Download an SDXL base checkpoint and an SDXL-compatible ControlNet model from its official distribution page. Match the conditioning type to the model: a Canny model expects an edge map, while a depth model expects a depth map. Record the exact file names and versions you use.
Minimal graph
- Load the SDXL checkpoint with Load Checkpoint. Connect its model, CLIP and VAE outputs to the appropriate sampler, text-encoder and decoder nodes.
- Encode positive and negative prompts using the CLIP output. Start with a simple subject and background so you can see whether the structural image is affecting the result.
- Load the SDXL ControlNet file using Load ControlNet. Load the reference image and, if your model requires one, run the matching preprocessor. Check that the preprocessor output is the image you intended to supply.
- Connect positive and negative conditioning plus the ControlNet and processed image to Apply ControlNet. Feed its conditioned outputs to the sampler.
- Create the latent image at a suitable SDXL resolution, sample, decode with VAE and save the output.
The official ComfyUI ControlNet example shows the current node connections and provides example workflows. Node names and sockets can change, so use that page as the source for a current release.
A concrete SDXL Canny starting point
For an edge-guided example, start with SDXL base 1.0 and the publisher’s SDXL Canny ControlNet. The latter was trained for SDXL base with Canny conditioning. Download a weight format supported by your ComfyUI loader; the Python Diffusers example on a model card is not itself a ComfyUI workflow.
Keep the checkpoint under ComfyUI/models/checkpoints/ and the compatible ControlNet weights under ComfyUI/models/controlnet/. A filename ending in “sd15” should not be treated as an SDXL substitute. The general official ControlNet tutorial includes an SD 1.5 scribble example: use it to understand connections, not as the SDXL model download list.
Trace each connection before running
| Source | Destination | Check |
|---|---|---|
| Load Checkpoint: MODEL | KSampler: model | The checkpoint is SDXL |
| Load Checkpoint: CLIP | Both text encoders | Positive and negative use the same checkpoint’s encoder |
| Text conditioning | Apply ControlNet: positive and negative | Do not bypass the control branch |
| Load ControlNet | Apply ControlNet: control_net | The selected file matches SDXL and Canny |
| Edge-map image | Apply ControlNet: image | Inspect the map before sampling |
| Conditioned positive and negative | KSampler conditioning | Both outputs reach the sampler |
| Empty Latent Image | KSampler: latent_image | Batch size one for diagnosis |
| KSampler latent and checkpoint VAE | VAE Decode → Save Image | Decode the sampled latent |
If your Apply ControlNet node exposes a required VAE input, supply the workflow’s compatible VAE. Follow the sockets on your installed node rather than copying an older screenshot.
Control strength and timing
The advanced Apply ControlNet node exposes strength and start/end percentages. Strength sets the influence of the control signal; the percentages define when it is active during denoising. They are separate controls. A high strength with a short interval is not equivalent to a low strength over the whole interval.
For a first diagnostic run, keep the interval unchanged from the compatible example. Compare an unconditioned baseline with one conditioned run before tuning timing. If you change strength, prompt, edge thresholds and seed together, you lose the ability to identify the cause of a difference.
A small experiment you can reproduce
This is a proposed exercise, not a FluxDraw test result. Photograph a simple object against an uncluttered background, such as a cup on a table. Choose the output aspect ratio before making the edge map, and preserve that framing through preprocessing and generation. Cropping the map differently from the output can move the structure you wanted to keep.
- Save the original reference and a visible edge-map preview. Confirm the cup rim, handle and table boundary remain readable.
- Use a short brief, for example: “A ceramic cup on a wooden table, side lighting, neutral background.” Keep it consistent for all runs.
- Save the base SDXL output with ControlNet removed from the conditioning path.
- Run the conditioned graph with the example’s settings, using the same prompt, dimensions and fixed seed.
- Change only ControlNet strength for a second conditioned output. Label each image with its actual settings.
- Check handle placement, object silhouette, background flexibility and artifacts separately. Select the result that meets the brief, rather than the one with the most rigid edge match.
After this works, repeat with a new reference. A setting that worked for a clean cup outline may be unsuitable for hair, foliage or a noisy photograph.
When control looks wrong, inspect the map first
A dense edge map can include shadows and surface texture that should not become structural boundaries. A sparse map may lose the subject entirely. Before adding more sampling steps, check whether the map itself contains the useful outline. If it does not, adjust preprocessing and save another preview while keeping the generation settings fixed.
For pose or depth, choose the matching SDXL control model and its documented preprocessing. A Canny map cannot be silently swapped for a pose skeleton or depth image. Combining multiple controls adds more possible conflicts; establish one working control first.
What to save when asking for help
Include the ComfyUI version, checkpoint and ControlNet filenames, preprocessing method, workflow JSON, reference map, output and full error text. State whether plain SDXL works. Remove credentials and private reference images before sharing. This lets someone distinguish a model-loading problem from bad conditioning or a memory failure.
Test one change at a time
Keep the checkpoint, seed, prompt, sampler and resolution fixed while you vary ControlNet strength. Save a baseline image with no ControlNet and at least two conditioned results. Check whether the intended edges or pose were preserved and whether the image became rigid or distorted. The useful strength depends on the model, reference image and prompt; a number from another workflow is only a starting point.
Common failures
Model will not load
Confirm the ControlNet file is in the expected model directory and matches SDXL. Restart or refresh model lists after adding files. Do not assume an SD 1.5 ControlNet is interchangeable with an SDXL model.
Validation fails before sampling
Inspect the highlighted node and the validation error. Check that Apply ControlNet receives a ControlNet object and an image, and that the conditioning outputs feed the sampler. Missing custom nodes may require their documented installation steps; avoid installing an unknown node solely to silence an error.
Output ignores or overfits the reference
View the preprocessor image first. If it is blank or noisy, fix the source or preprocessing before changing the prompt. Then adjust conditioning strength in small increments while holding the seed constant.
CUDA out of memory
Try batch size one, a smaller image, and the default memory settings. Peak use may happen during sampling or VAE decoding. Note the failing node and your GPU memory before changing models or adding tiled decoding.
Related guides
For a basic SDXL pipeline, see SDXL in ComfyUI. For application-level errors, see ComfyUI queue troubleshooting. If you plan to submit workflows via Python, use the API guide.
