2025-12-08 · 4 min read

Hot-Swap Strategies for Adapter Serving

Hot-swapping allows a production server to load new adapters without restarting. It requires careful memory management and request routing.

The problem

You have N GPU slots but M adapters to serve (M > N). Not all adapters fit in VRAM simultaneously. Requests arrive for adapters that may not be loaded.

Solutions range from simple (unload/reload) to sophisticated (predictive caching).

Three strategies

Cold swap: Unload current adapter, load new one. Latency: 1–3 seconds per swap. Adapters loaded from disk.

Warm swap: Keep "warm" adapters in CPU RAM; quickly copy to GPU. Latency: 200–500ms. Requires 2–3x adapter storage (VRAM + CPU RAM).

Hot swap: All active adapters in VRAM; multiplex per-request. Latency: <1ms. Requires precise capacity planning.

Cold swap implementation

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM

class ColdSwapServer:
    def __init__(self, base_model_id):
        self.base = AutoModelForCausalLM.from_pretrained(base_model_id)
        self.current_adapter = None
        self.adapters_dir = "./adapters/"

    def load_adapter(self, adapter_name):
        """Load adapter from disk; unload previous."""
        if self.current_adapter:
            # Unload old
            self.model.unload()
            del self.model

        # Load new
        self.model = PeftModel.from_pretrained(
            self.base, 
            f"{self.adapters_dir}/{adapter_name}"
        )
        self.current_adapter = adapter_name

    def infer(self, adapter_name, prompt):
        """Infer with specified adapter."""
        if self.current_adapter != adapter_name:
            self.load_adapter(adapter_name)

        # Generate
        inputs = tokenizer(prompt, return_tensors="pt")
        output = self.model.generate(**inputs)
        return tokenizer.decode(output[0])

Simple but slow. Reasonable only for infrequent adapter changes.

Warm swap: LRU cache

import torch
from collections import OrderedDict

class WarmSwapServer:
    def __init__(self, base_model_id, vram_slots=2, cpu_cache_size=8):
        self.base = AutoModelForCausalLM.from_pretrained(base_model_id)
        self.vram_slots = vram_slots
        self.vram_cache = OrderedDict()  # Currently in VRAM
        self.cpu_cache = OrderedDict()   # Warm in CPU RAM
        self.cpu_cache_size = cpu_cache_size

    def _move_to_vram(self, adapter_name):
        """Move adapter from CPU to GPU VRAM."""
        if adapter_name in self.vram_cache:
            # Already loaded
            self.vram_cache.move_to_end(adapter_name)  # Mark recent
            return

        if len(self.vram_cache) >= self.vram_slots:
            # Evict oldest from VRAM to CPU
            old_name, old_adapter = self.vram_cache.popitem(last=False)
            self.cpu_cache[old_name] = old_adapter.to("cpu")

        # Load from CPU or disk
        if adapter_name in self.cpu_cache:
            adapter = self.cpu_cache.pop(adapter_name).to("cuda")
        else:
            adapter = PeftModel.from_pretrained(self.base, adapter_name)

        self.vram_cache[adapter_name] = adapter

    def infer(self, adapter_name, prompt):
        """Infer with warm-swap."""
        self._move_to_vram(adapter_name)
        model = self.vram_cache[adapter_name]
        inputs = tokenizer(prompt, return_tensors="pt")
        output = model.generate(**inputs)
        return tokenizer.decode(output[0])

LRU eviction ensures frequently-used adapters stay in VRAM.

Hot swap: fixed mux

import torch
from fastapi import FastAPI
from contextlib import asynccontextmanager

class HotSwapServer:
    def __init__(self, adapters_to_load):
        """Load fixed set of adapters; no swapping."""
        self.base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
        self.adapters = {}
        for adapter_name in adapters_to_load:
            peft_model = PeftModel.from_pretrained(self.base, adapter_name)
            self.adapters[adapter_name] = peft_model

    def infer(self, adapter_name, prompt):
        """Infer; adapter is always loaded."""
        if adapter_name not in self.adapters:
            raise ValueError(f"Adapter {adapter_name} not loaded. Load it at server startup.")

        model = self.adapters[adapter_name]
        inputs = tokenizer(prompt, return_tensors="pt")
        output = model.generate(**inputs)
        return tokenizer.decode(output[0])

app = FastAPI()

@asynccontextmanager
async def lifespan(app):
    # Startup: load adapters
    app.state.server = HotSwapServer([
        "qwen25-support-tone",
        "mistral-json-extract",
        "phi3-sql-helper"
    ])
    yield
    # Shutdown

app = FastAPI(lifespan=lifespan)

@app.post("/infer")
async def infer(request):
    adapter = request["adapter"]
    prompt = request["prompt"]
    result = app.state.server.infer(adapter, prompt)
    return {"response": result}

No swapping overhead. Trade-off: adapters are static (set at startup).

Memory planning

For vLLM, the memory math:

Total VRAM = base_model + (num_hot_slots * adapter_size) + overhead

Example: 7B base (14 GB) + 4 LoRA (100 MB each) ≈ 14.4 GB on 24 GB card.

Unused capacity can hold 8–16 "warm" adapters in CPU RAM:

CPU RAM = num_warm_slots * adapter_size

4 warm adapters × 100 MB = 400 MB. Negligible.

Predictive loading

For popular adapters, preemptively load the next likely one:

class PredictiveServer(WarmSwapServer):
    def __init__(self, *args, popularity_log="adapter_hits.json", **kwargs):
        super().__init__(*args, **kwargs)
        self.popularity = load_json(popularity_log)  # {adapter_name: count}

    def infer(self, adapter_name, prompt):
        result = super().infer(adapter_name, prompt)

        # Preload next likely adapter
        top_adapters = sorted(self.popularity.items(), key=lambda x: x[1], reverse=True)
        next_adapter = top_adapters[0][0]  # Most popular
        if next_adapter not in self.vram_cache and next_adapter not in self.cpu_cache:
            self._preload_to_cpu(next_adapter)

        return result

Predictive loading reduces perceived latency when adapter usage is skewed.

Metrics

Track in production:

• Cache hit rate: % of requests served without loading delay
• p95 swap latency: 95th percentile load time
• Memory utilization: % VRAM in use vs capacity
• Adapter popularity: Histogram of requests per adapter

metrics = {
    "cache_hits": 450,
    "cache_misses": 50,
    "p95_latency_ms": 250,
    "vram_utilization": 0.85,
    "adapters": {"support_tone": 350, "json_extract": 100}
}

ModelForgeLab strategy

The registry could adopt a three-tier model:

1. Hot (4): Most popular adapters (support_tone, json_extract, sql_helper, product_photo) always loaded
2. Warm (12): Less common adapters in CPU RAM, loaded on-demand
3. Cold: Remaining adapters on disk or object storage

Requests hit warm/cold timeout of 200–1000ms, but hot adapters are <1ms.

Hot-swap is a scaling problem. Start with simple cold-swap, add warm-swap when adapter count grows, switch to hot-swap at production scale.