Overview

FLUX.2-dev-Fun-Controlnet-Union is a ControlNet adapter for FLUX.2-dev that enables precise spatial control over image generation through multiple condition types including Canny edge detection, HED edge maps, depth maps, pose skeletons, MLSD line detection, scribble input, and grayscale images. Built by alibaba-pai, this model adds control mechanisms to 4 double blocks within the FLUX.2-dev architecture and includes inpainting capabilities. The critical distinction from FLUX.2-dev's native editing features is that this ControlNet reliably follows control instructions across multiple control types simultaneously without the speed degradation or similarity issues that plague the base model's editing mode. The model comes in two variants: the newer 2602 version with added Scribble and Gray controls plus CFG distillation for improved generation quality, and the original version supporting Canny, HED, Depth, Pose, and MLSD controls. Implementation requires the VideoX-Fun library and the base FLUX.2-dev model weights. The controlnet_conditioning_scale parameter operates optimally in the 0.65 to 0.80 range for balancing control strength with detail preservation, with detailed prompts recommended for stable results.

Best use cases

Character pose transfer and reference-guided generation. Apply human pose skeletons extracted from reference images to generate new characters in identical poses while maintaining style consistency through text prompts. This workflow proves essential for animation frame generation, fashion design exploration, and character animation where pose matching must be exact. The Pose control type specifically excels at this because the model receives explicit skeletal information that constrains the generation without requiring the base model to infer pose from natural images.

Edge-guided architectural and product visualization. Feed Canny or HED edge maps of architectural sketches or product outlines to generate photorealistic renders while preserving the input geometry exactly. Designers use this to convert rough structural sketches into finished visualizations without manual retouching. The edge control types work because FLUX.2-dev inherently respects structural boundaries, and explicit edge guidance eliminates the spatial drift common in pure text-to-image generation.

Depth-consistent 3D scene generation. Provide depth maps from existing scenes or 3D renders to generate variations that maintain consistent spatial layout and perspective. This supports applications like virtual staging, game asset variation, and architectural walk-through generation. Depth control is particularly valuable because it enforces volumetric consistency that text prompts alone cannot guarantee, preventing spatial contradictions between foreground and background elements.

Grayscale-to-color guided inpainting. Supply grayscale reference images and masks to inpaint regions while preserving luminance information from the grayscale reference. This handles scenarios like selectively recoloring black-and-white photographs or generating missing regions of vintage images with period-appropriate color palettes. Gray and inpainting controls combined allow fine-grained regional control without the edit-mode speed penalties of FLUX.2-dev's native approach.

Sketch-based refinement with scribble control. Draw loose scribbles to specify local modifications—reposition limbs, add accessories, adjust composition—without regenerating the entire image. Scribble control enables iterative creative workflows where artists maintain most of the generated content while making targeted adjustments, avoiding the computational waste of full regeneration.

Limitations

Generation speed degrades with multiple simultaneous controls. Although ControlNet architecture itself is efficient, applying multiple control types (e.g., Pose + Reference together) compounds inference latency compared to single-control or text-only FLUX.2-dev. Processing batch sizes larger than single images further reduces throughput, making this unsuitable for high-volume production pipelines requiring sub-second latency.

CFG distillation required for quality after control training. The base FLUX.2-dev model loses classifier-free guidance capability during ControlNet training, similar to the Z-Image-Turbo experience. While the 2602 variant includes post-training CFG distillation to recover quality, the original model variant performs poorly without this remediation. Users must explicitly use the 2602 weights or accept degraded output quality.

Controlnet_conditioning_scale hyperparameter requires tuning. The optimal range of 0.65 to 0.80 is narrow, and values outside this window produce either weak control adherence or artifacts. This mandates prompt engineering alongside control tuning—generic or vague prompts destabilize results even within the recommended scale range. No automatic parameter optimization exists; practitioners must manually iterate.

Inpainting mode adds computational overhead. Masking and selective regeneration require additional processing compared to unconditional generation, increasing inference time unpredictably depending on masked region size and complexity. The documentation provides no concrete timing benchmarks for inpainting inference.

No documented hardware requirements or inference speed metrics. The README omits VRAM requirements, inference time per image, batch processing capabilities, and optimal hardware specifications. Users must empirically determine whether their hardware suffices, creating unpredictable deployment planning.

Limited control resolution documentation. The README does not specify maximum input resolution for control maps, whether resizing occurs automatically, or how resolution mismatches between control and text prompts are handled. Users must infer these constraints through experimentation.

Model file size unspecified. Neither the 2602 nor original variant documentation states SAFETENSORS file size, preventing accurate disk space and download time estimation for production environments.

License terms unclear. The model lists "other" as its license without clarifying commercial use rights, modification permissions, or redistribution restrictions. This ambiguity creates legal risk for commercial deployments.

How it compares

Versus Z-Image-Fun-Controlnet-Union-2.1: Z-Image adds control to 15 layer blocks versus FLUX.2-dev's 4 double blocks, providing stronger fine-grained control at the cost of reduced speed. Choose FLUX.2-dev-Fun-Controlnet-Union for faster generation when moderate control strength suffices; choose Z-Image-2.1 when maximum spatial precision matters regardless of inference time. Z-Image-2.1 also offers a lite variant for weaker control with faster generation, a flexibility option FLUX.2-dev lacks.

Versus Qwen-Image-2512-Fun-Controlnet-Union: Qwen-Image adds control to 5 layer blocks, positioning it between FLUX.2-dev (4 blocks) and Z-Image (15 blocks) in control strength and speed tradeoff. FLUX.2-dev-Fun-Controlnet-Union generation quality typically exceeds Qwen-Image due to FLUX.2-dev's superior base model, but Qwen-Image may render faster if speed is the primary constraint. Both support similar control types; the choice hinges on your acceptance of quality-speed tradeoff.

Versus Z-Image-Turbo-Fun-Controlnet-Union: Z-Image-Turbo adds control to 6 blocks and trained on 1 million images at 1328 resolution using BFloat16 precision with 10,000 training steps. FLUX.2-dev-Fun-Controlnet-Union typically generates higher quality output because FLUX.2-dev itself surpasses Z-Image-Turbo as a base model, but Z-Image-Turbo may prioritize speed in your use case. Z-Image-Turbo's training data scale and resolution details provide more transparency than FLUX.2-dev's documentation.

Versus Z-Image-Turbo-Fun-Controlnet-Union-2.1: Z-Image-Turbo-2.1 underwent 8-step distillation to restore the acceleration capability lost during ControlNet integration, plus includes a tile variant for super-resolution. Choose FLUX.2-dev-Fun-Controlnet-Union for maximum output quality; choose Z-Image-Turbo-2.1 if you need guaranteed speed with acceptable quality degradation. Z-Image-Turbo-2.1 provides explicit acceleration benchmarks and super-resolution workflows absent from FLUX.2-dev-Fun-Controlnet-Union documentation.

Versus Z-Image-Turbo-Fun-Controlnet-Union-2.0: Z-Image-Turbo-2.0 contains a training bug (control_layers used instead of control_noise_refiner for refiner latents) that forces double forward passes and causes inference slowdown. Avoid Z-Image-Turbo-2.0 entirely in favor of either FLUX.2-dev-Fun-Controlnet-Union or Z-Image-Turbo-2.1, which fixed this issue.

Technical specifications

Architecture: ControlNet adapter integrated into FLUX.2-dev, with control mechanisms added to 4 double blocks of the base diffusion transformer. The adapter accepts spatial control signals (edge maps, depth maps, pose skeletons, grayscale images, scribble masks) and conditions generation without modifying the base model weights.

Training and data: The 2602 variant trained on an improved dataset with CFG (classifier-free guidance) distillation applied post-training to recover generation quality lost during ControlNet integration. Original variant lacks this distillation step. No explicit training step counts or dataset sizes disclosed.

Control types supported: Canny edge detection, HED edge maps, depth maps, OpenPose skeletons, MLSD line detection, scribble input, grayscale images. All control types function as standard ControlNet implementations.

Input format: Control maps and guidance images as image files; text prompts as strings; optional mask images for inpainting mode.

Output format: Generated images as standard image files.

Framework and library: VideoX-Fun library required for inference. Implementation uses standard Python with PyTorch or compatible tensor backend.

Model file format: SAFETENSORS format for both variants:

  • FLUX.2-dev-Fun-Controlnet-Union-2602.safetensors(newer with CFG distillation and Scribble/Gray controls)

  • FLUX.2-dev-Fun-Controlnet-Union.safetensors(original)

Required base model: FLUX.2-dev model weights must be separately downloaded and placed in models/Diffusion_Transformer/FLUX.2-dev/ directory structure.

Hyperparameter ranges: controlnet_conditioning_scale operates optimally between 0.65 and 0.80 for control strength and detail preservation balance.

Inference modes: Standard image generation from control + text prompt; inpainting mode with mask specification for selective region regeneration.

Model inputs and outputs

Inputs

  • Control image:Any of Canny, HED, depth map, pose skeleton, MLSD line map, scribble, or grayscale image (format and resolution constraints undocumented)

  • Text prompt:String description of desired output (detailed prompts recommended for stable results)

  • Optional mask image:Binary or grayscale mask for inpainting mode, specifying regions to regenerate

  • Controlnet_conditioning_scale:Float parameter between 0.65 and 0.80 controlling control strength (undocumented behavior outside this range)

Outputs

  • Generated image:Single image matching dimensions of control input (output resolution matching control map specification, assumed to match base FLUX.2-dev native resolution but not explicitly documented)

  • Optional batch outputs:Multiple images in batch mode (batch size limits and memory requirements undocumented)

Getting started

Clone the VideoX-Fun repository and organize model weights:

```
git clone https://github.com/aigc-apps/VideoX-Fun.git
cd VideoX-Fun
mkdir -p models/Diffusion_Transformer
mkdir -p models/Personalized_Model

Download FLUX.2-dev weights to models/Diffusion_Transformer/FLUX.2-dev/

Download ControlNet weights to models/Personalized_Model/

FLUX.2-dev-Fun-Controlnet-Union-2602.safetensors

FLUX.2-dev-Fun-Controlnet-Union.safetensors

Run inference

python examples/flux2_fun/predict_t2i_control.py
`` Thepredict_t2i_control.py` script handles loading both base model and ControlNet weights, accepting control images and text prompts as inputs. Consult the VideoX-Fun repository for parameter configuration and batch processing examples, as concrete inference code snippets do not appear in the provided README.

Frequently asked questions

Q: Can I use this model commercially?

A: The model lists "other" as its license without clarifying commercial use terms. Contact alibaba-pai directly to confirm licensing rights before commercial deployment.

Q: What GPU VRAM do I need to run this model?

A: The README does not specify VRAM requirements. You will need hardware sufficient for FLUX.2-dev base model inference plus ControlNet overhead, but exact specifications are undocumented. Empirical testing on your target hardware is necessary.

Q: Should I use the 2602 variant or the original variant?

A: Use the 2602 variant in nearly all cases. It includes CFG distillation post-training and adds Scribble and Gray control types, providing superior output quality and broader control options. The original variant lost CFG capability during training and performs poorly without the 2602 improvements.

Q: How does FLUX.2-dev-Fun-Controlnet-Union compare to FLUX.2-dev's native editing capabilities?

A: ControlNet reliably adheres to control instructions across multiple simultaneous control types without the speed degradation or similarity issues that plague FLUX.2-dev's native image-editing mode. ControlNet is the preferred approach when precise spatial control matters.

Q: What is the optimal controlnet_conditioning_scale value?

A: The recommended range is 0.65 to 0.80. Values outside this window produce either weak control adherence or generation artifacts. Optimal results require both tuning this parameter and using detailed text prompts; generic prompts destabilize output even within the recommended range.

Q: Does this model support batch processing or only single-image generation?

A: The documentation mentions generation speed slows when handling multiple images, implying batch support exists, but no guidance on batch size limits, practical throughput, or memory scaling is provided. Consult the VideoX-Fun repository code for batch configuration details.

Q: Can I fine-tune this ControlNet on custom control types or datasets?

A: The documentation does not discuss fine-tuning or custom training. The model is released as a pre-trained adapter intended for inference only; modification workflows are not documented.

Q: How does inference speed compare to standard FLUX.2-dev generation?

A: No concrete timing benchmarks appear in the documentation. Speed degrades with multiple simultaneous controls and batch processing compared to text-only generation, but quantitative metrics are absent. Empirical benchmarking on your hardware is required.