Introducing Modular Diffusers - Composable Building Blocks for Diffusion Pipelines

Modular Diffusers introduces a new way to build diffusion pipelines by composing reusable blocks. Instead of writing entire pipelines from scratch, you can mix and match blocks to create workflows tailored to your needs! This complements the existing

DiffusionPipeline

class with a more flexible, composable alternative. In this post, we'll walk through how Modular Diffusers works — from the familiar API to run a modular pipeline, to building fully custom blocks and composing them into your own workflow. We'll also show how it integrates with Mellon, a node-based visual workflow interface that you can use to wire Modular Diffusers blocks together.

Table of contents

Here is a simple example of how to run inference with FLUX.2 Klein 4B

using pre-built blocks:

import torch
from diffusers import ModularPipeline
# Create a modular pipeline - this only defines the workflow, model weights have not been loaded yet
pipe = ModularPipeline.from_pretrained(
"black-forest-labs/FLUX.2-klein-4B"
)
# Now load the model weights — configure dtype, quantization, etc in this step
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.to("cuda")
# Generate an image - API remains the same as DiffusionPipeline
image = pipe(
prompt="a serene landscape at sunset",
num_inference_steps=4,
).images[0]
image.save("output.png")

You get the same results as with a standard DiffusionPipeline

, but the pipeline is very different under the hood: it's composed of flexible blocks — text encoding, image encoding, denoising, and decoding — that you can inspect directly:

print(pipe.blocks)

Flux2KleinAutoBlocks(
...
Sub-Blocks:
[0] text_encoder (Flux2KleinTextEncoderStep)
[1] vae_encoder (Flux2KleinAutoVaeEncoderStep)
[2] denoise (Flux2KleinCoreDenoiseStep)
[3] decode (Flux2DecodeStep)
)

Each block is self-contained with its own inputs and outputs. You can run any block independently as its own pipeline, or add, remove, and swap blocks freely — they dynamically recompose to work with whatever blocks remain. Use .init_pipeline()

to convert blocks into a runnable pipeline, and .load_components()

to load the model weights.

# get a copy of the blocks
blocks = pipe.blocks
# pop out the text_encoder block
text_blocks = blocks.sub_blocks.pop("text_encoder")
# run it as its own pipeline
text_pipe = text_blocks.init_pipeline("black-forest-labs/FLUX.2-klein-4B")
# load the text_encoder, or reuse already loaded components: text_pipe.update_components(text_encoder=pipe.text_encoder)
text_pipe.load_components(torch_dtype=torch.bfloat16)
text_pipe.to("cuda")
prompt_embeds = text_pipe(prompt="a serene landscape at sunset").prompt_embeds
# create a new pipeline from the remaining blocks
# it now accepts prompt_embeds directly instead of prompt
remaining_pipe = blocks.init_pipeline("black-forest-labs/FLUX.2-klein-4B")
remaining_pipe.load_components(torch_dtype=torch.bfloat16)
remaining_pipe.to("cuda")
image = remaining_pipe(prompt_embeds=prompt_embeds, num_inference_steps=4).images[0]

For more on block types, composition patterns, lazy loading, and memory management with ComponentsManager

, check out the Modular Diffusers documentation.

Modular Diffusers really shines when creating your own blocks. A custom block is a Python class that defines its components, inputs, outputs, and computation logic — and once defined, you can plug it into any workflow.

Here's an example block that extracts depth maps from images using Depth Anything V2.

class DepthProcessorBlock(ModularPipelineBlocks):
@property
def expected_components(self):
return [
ComponentSpec("depth_processor", DepthPreprocessor,
pretrained_model_name_or_path="depth-anything/Depth-Anything-V2-Large-hf")
]
@property
def inputs(self):
return [
InputParam("image", required=True,
description="Image(s) to extract depth maps from"),
]
@property
def intermediate_outputs(self):
return [
OutputParam("control_image", type_hint=torch.Tensor,
description="Depth map(s) of input image(s)"),
]
@torch.no_grad()
def __call__(self, components, state):
block_state = self.get_block_state(state)
depth_map = components.depth_processor(block_state.image)
block_state.control_image = depth_map.to(block_state.device)
self.set_block_state(state, block_state)
return components, state

expected_components

defines what models the block needs — in this case, a depth estimation model. Thepretrained_model_name_or_path

parameter sets a default Hub repo to load from, soload_components

automatically fetches the depth model unless you override it inmodular_model_index.json

.inputs

andintermediate_outputs

define what goes in and comes out.__call__

is where the computation logic lives.

Let's use this block with Qwen's ControlNet workflow. Extract the ControlNet workflow and insert the depth block at the beginning:

# Create Qwen Image pipeline
pipe = ModularPipeline.from_pretrained("Qwen/Qwen-Image")
print(pipe.blocks.available_workflows)
# Supported workflows:
# - `text2image`: requires `prompt`
# - `image2image`: requires `prompt`, `image`
# - `inpainting`: requires `prompt`, `mask_image`, `image`
# - `controlnet_text2image`: requires `prompt`, `control_image`
# - `controlnet_image2image`: requires `prompt`, `image`, `control_image`
# Extract the ControlNet workflow — it expects a control_image input
blocks = pipe.blocks.get_workflow("controlnet_text2image")
# Show the blocks this workflow uses
print(blocks)
# Insert depth block at the beginning — its output (control_image)
# automatically flows to the ControlNet block that needs it
blocks.sub_blocks.insert("depth", DepthProcessorBlock(), 0)
# You can inspect any block's inputs and outputs with print(blocks.doc)
blocks.sub_blocks['depth'].doc

Blocks in a sequence share data automatically: the depth block's control_image

output flows to downstream blocks that need it, and its image

input becomes a pipeline input since no earlier block provides it.

from diffusers import ComponentsManager, AutoModel
from diffusers.utils import load_image
# ComponentsManager handles memory across multiple pipelines —
# it automatically offloads models to CPU when not in use
manager = ComponentsManager()
pipeline = blocks.init_pipeline("Qwen/Qwen-Image", components_manager=manager)
pipeline.load_components(torch_dtype=torch.bfloat16)
# The depth model loads automatically from the default path we set in expected_components —
# no need to load it manually even though it's not part of the Qwen repo.
# But controlnet is not included by default, so we do need to load it from a different repo
controlnet = AutoModel.from_pretrained("InstantX/Qwen-Image-ControlNet-Union", torch_dtype=torch.bfloat16)
pipeline.update_components(controlnet=controlnet)
# pipeline now takes image as input
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg")
output = pipeline(
prompt="an astronaut hatching from an egg, detailed, fantasy, Pixar, Disney",
image=image,
).images[0]

You can publish your custom block to the Hub so anyone can load it with trust_remote_code=True

. We've created a template to get you started — check out the Building Custom Blocks guide for the full walkthrough.

pipeline.save_pretrained(local_dir, repo_id="your-username/your-block-name", push_to_hub=True)

The DepthProcessorBlock

from this post is published at diffusers/depth-processor-custom-block — you can load and use it directly:

from diffusers import ModularPipelineBlocks
depth_block = ModularPipelineBlocks.from_pretrained(
"diffusers/depth-processor-custom-block", trust_remote_code=True
)

We've published a collection of ready-to-use custom blocks here.

ModularPipeline.from_pretrained

works with any existing Diffusers repo out of the box, but Modular Diffusers also introduces a new kind of repo: the Modular Repository.

A modular repository is able to reference components from their