2025-06-19 · 3 min read

Prompt Sets for Image Regression Testing

Image regression testing is only useful when the input set is stable. A random prompt every time tells you almost nothing about whether a new adapter version improved or regressed. A fixed prompt set gives you a consistent baseline for visual comparison, which is the only way to make version changes intelligible.

The prompt set should cover the adapter's expected use cases. If the adapter is for watercolor portraits, include portrait prompts, close-ups, different lighting conditions, and a few edge cases such as outdoor scenes or multi-subject compositions. If the adapter is for product imagery, include plain backgrounds, angled shots, and detail-heavy prompts. The goal is not to test everything. The goal is to catch regressions in the area the adapter claims to handle.

Seeds matter just as much as prompts. If the seed changes, the visual comparison becomes much noisier. A fixed seed lets you compare outputs across versions and see whether composition, color, or subject fidelity shifted. That makes version-to-version review much more practical.

[
  {"prompt":"a watercolor portrait of a woman with soft window light","seed":1234},
  {"prompt":"a watercolor portrait of a man in a blue coat","seed":1234},
  {"prompt":"a watercolor still life on a wooden table","seed":1234}
]

Negative prompts are useful when the adapter is sensitive to artifacts. They should also be fixed if they are used at all. A changing negative prompt makes it harder to know whether a visual difference came from the adapter or from the control inputs. The same applies to guidance scale, aspect ratio, and sampler choice.

Regression tests do not need to be huge. A few carefully chosen prompts can reveal a lot if they are representative. The best prompt set usually mixes easy cases and hard cases. Easy cases tell you whether the adapter still does its core job. Hard cases show whether it still behaves reasonably when the prompt gets less ideal.

The prompt set should also reflect the expected usage distribution. If most users will ask for portraits, do not test only landscapes. If the adapter is meant to handle stylized product shots, include prompts that stress reflection, symmetry, and background control. A good regression set should feel like the future workload, not a random benchmark.

Versioning matters here because prompt sets are part of the release contract. If the prompt set changes, comparisons across versions become less trustworthy. Store the prompt list with the adapter release so the same evaluation can be repeated later without guessing which inputs were used.

The review process should be visual and repeatable. Put old and new outputs next to each other. Look for changes in color balance, subject shape, texture quality, and prompt adherence. A small drift can be acceptable if the release note explains it. A sudden collapse in composition usually means the new version needs another pass.

It also helps to annotate each prompt with a purpose. Some prompts are there to test style fidelity, some to test composition, and some to catch artifact patterns such as extra fingers, washed-out backgrounds, or oversaturation. A prompt set that is labeled well is much easier to maintain than a flat list with no context.

When the adapter changes, use the same prompts, the same seeds, the same sampler, and the same resolution. Otherwise the comparison becomes noisy enough that the results are hard to trust. A regression set is only useful when the variables stay under control.

ModelForgeLab can make this workflow feel natural by storing prompt sets alongside adapter releases. That way a release page can show a stable reference grid and users can judge changes without reconstructing the test harness themselves.