2025-10-20 · 3 min read

Loading LoRA Explicitly in the Diffusers Pipeline

The Diffusers library provides fine-grained control over LoRA loading. You can load, combine, fuse, and unload adapters programmatically for inference and experimentation.

Basic loading

from diffusers import StableDiffusionXLPipeline
import torch

# Load base model
pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16"
)
pipe.to("cuda")

# Load a single LoRA
pipe.load_lora_weights("path/to/sdxl-watercolor-lora")

# Generate
prompt = "a mountain landscape in watercolor style"
image = pipe(prompt).images[0]

Simple and clean. The adapter is applied automatically.

Named adapters and hot-swap

Name each adapter for easy switching:

# Load first adapter as "watercolor"
pipe.load_lora_weights("path/to/sdxl-watercolor-lora", weight_name="watercolor")

# Load second adapter as "product"
pipe.load_lora_weights("path/to/sdxl-product-photo-lora", weight_name="product")

# Switch to first
pipe.set_adapter("watercolor")
image1 = pipe("a landscape").images[0]

# Switch to second
pipe.set_adapter("product")
image2 = pipe("a coffee mug").images[0]

# Unload
pipe.unload_lora_weights()

No restart, no reloading the base model. Adapters swap in microseconds.

Weighted fusion

Combine multiple adapters with scalar weights:

# Load both
pipe.load_lora_weights("path/to/sdxl-watercolor-lora", weight_name="watercolor")
pipe.load_lora_weights("path/to/sdxl-product-photo-lora", weight_name="product")

# Fuse them with weights
pipe.set_adapters(
    adapter_names=["watercolor", "product"],
    adapter_weights=[0.6, 0.4]  # 60% watercolor, 40% product
)

# Generate blended result
image = pipe("a ceramic mug on a table").images[0]

The result is a visual blend of both styles. Adjust weights to control the balance.

Full workflow: swapping and comparing

from diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
)
pipe.to("cuda")

# Load two adapters
pipe.load_lora_weights("path/to/sdxl-watercolor-lora", weight_name="watercolor")
pipe.load_lora_weights("path/to/sdxl-product-photo-lora", weight_name="product")

prompt = "a landscape with mountains and lake"

# Generate with each adapter
pipe.set_adapter("watercolor")
img_watercolor = pipe(prompt, guidance_scale=7.0).images[0]

pipe.set_adapter("product")
img_product = pipe(prompt, guidance_scale=7.0).images[0]

# Generate with blend
pipe.set_adapters(["watercolor", "product"], [0.7, 0.3])
img_blend = pipe(prompt, guidance_scale=7.0).images[0]

# Compare side-by-side
from PIL import Image
combined = Image.new("RGB", (img_watercolor.width * 3, img_watercolor.height))
combined.paste(img_watercolor, (0, 0))
combined.paste(img_product, (img_watercolor.width, 0))
combined.paste(img_blend, (img_watercolor.width * 2, 0))
combined.save("comparison.png")

Memory management

LoRA weights are loaded into VRAM alongside the base model:

# Check memory usage
import torch
print(f"VRAM allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")

A SDXL base (6 GB) + 3 LoRAs (150 MB each) ≈ 6.5 GB. Fits on a 12 GB card; on 8 GB, you must unload adapters:

# Unload to free VRAM
pipe.unload_lora_weights()
print(f"After unload: {torch.cuda.memory_allocated() / 1e9:.2f} GB")

Fused adapters

Merge a LoRA into the model weights for faster inference:

# Fuse current adapter
pipe.fuse_lora(unload_weights_after_fusing=False)

# Generate (faster, no LoRA overhead)
image = pipe(prompt).images[0]

# Unfuse to restore flexibility
pipe.unfuse_lora()

Fused pipelines are ~5–10% faster. Trade-off: cannot adjust alpha at runtime.

Practical example for ModelForgeLab playground

from fastapi import FastAPI
from diffusers import StableDiffusionXLPipeline
import torch
import uuid

app = FastAPI()

# Load once on startup
pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
)
pipe.to("cuda")

# Preload common adapters
adapters_dir = "./data/adapters/"
for adapter_name in ["sdxl-watercolor-lora", "sdxl-product-photo-lora"]:
    pipe.load_lora_weights(f"{adapters_dir}/{adapter_name}", weight_name=adapter_name)

@app.post("/v1/generate")
async def generate(request):
    """Generate image with optional LoRA."""
    prompt = request["prompt"]
    adapter_name = request.get("adapter", None)

    if adapter_name:
        pipe.set_adapter(adapter_name)
    else:
        pipe.unload_lora_weights()

    image = pipe(prompt, guidance_scale=7.0).images[0]

    # Return as base64 or save
    return {"image_id": str(uuid.uuid4()), "status": "complete"}

SDXL vs FLUX

FLUX support is similar but FLUX uses dual text encoders:

# FLUX with LoRA (requires diffusers nightly)
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.float16
)
pipe.to("cuda")

# Load FLUX LoRA
pipe.load_lora_weights("path/to/flux-linework-lora")

image = pipe(
    prompt="a clean linework sketch",
    guidance_scale=3.5,
    num_inference_steps=50
).images[0]

Syntax is identical; weight interpretation may differ due to FLUX's higher baseline guidance.

Performance

Fusing pays off for static workflows. Hot-swapping adapters is cheap; the overhead is loading from disk, not computation.

Diffusers gives you full control. Use it for experimentation, A/B testing, and production pipelines where you need fine control over adapter behavior.