2026-02-12 · 4 min read

Continuous Training Pipelines for Adapters

Production adapters degrade. User distributions shift, edge cases accumulate, base models get updated. Continuous retraining keeps adapters fresh without manual intervention.

The pipeline

New data lands → Batch accumulation → Sample review (optional) → Training → Evaluation → Gated release

Each stage has triggers, thresholds, and automation rules.

Data ingestion

New examples flow from production:

from datetime import datetime, timedelta

class DataAccumulator:
    def __init__(self, output_file="data/accumulate.jsonl"):
        self.output_file = output_file

    def log_interaction(self, prompt, expected_output):
        """Log a production interaction."""
        with open(self.output_file, "a") as f:
            record = {
                "prompt": prompt,
                "output": expected_output,
                "timestamp": datetime.utcnow().isoformat(),
            }
            f.write(json.dumps(record) + "\n")

    def get_batch(self, days=7):
        """Fetch examples from the last N days."""
        cutoff = datetime.utcnow() - timedelta(days=days)
        batch = []
        for line in open(self.output_file):
            record = json.loads(line)
            ts = datetime.fromisoformat(record["timestamp"])
            if ts > cutoff:
                batch.append(record)
        return batch

In production, whenever a user interaction completes successfully, log it. Accumulate for N days (e.g., weekly retraining = 7 days).

Triggers for retraining

Time-based: Every Monday 2 AM, retrain.

# In cron or scheduler
0 2 * * MON python retrain_adapter.py --adapter qwen25-7b-support-tone --batch-days 7

Event-based: After 1000 new examples accumulate.

def should_retrain():
    recent_batch = accumulator.get_batch(days=30)
    return len(recent_batch) > 1000

if should_retrain():
    trigger_training_job()

Drift detection: If validation loss degrades by >5%, retrain immediately.

current_loss = evaluate_on_heldout()
if current_loss > previous_loss * 1.05:
    log("Drift detected; triggering retraining")
    trigger_training_job()

Training stage

def retrain_adapter(adapter_name, new_data, base_model_id="Qwen/Qwen2.5-7B-Instruct"):
    """Retrain an adapter with new data."""
    import subprocess

    # Deterministic: freeze seed, base model version, training script
    cmd = [
        "python", "train_lora.py",
        "--base-model", base_model_id,
        "--adapter-name", adapter_name,
        "--train-file", new_data,
        "--output-dir", f"runs/{adapter_name}-retrain-{datetime.now().isoformat()}",
        "--seed", "42",
        "--epochs", "3",
        "--rank", "8",
    ]

    result = subprocess.run(cmd, capture_output=True)
    return result.returncode == 0

Store run metadata:

{
  "adapter": "qwen25-7b-support-tone",
  "version": "1.2.1",
  "training_date": "2026-02-12T02:00:00Z",
  "data_size": 756,
  "training_seed": 42,
  "base_model_version": "Qwen/Qwen2.5-7B-Instruct-abc123",
  "training_script_hash": "sha256:...",
  "output_path": "runs/qwen25-7b-support-tone-retrain-2026-02-12T02:00:00Z"
}

Everything is pinned: seed, base model revision, script version.

Evaluation gate

New adapter passes evaluation before release:

def evaluate_and_gate(new_adapter_path, heldout_eval_set):
    """Compare new vs previous adapter."""
    from transformers import AutoModelForCausalLM
    from peft import PeftModel

    base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

    # Previous (currently in production)
    previous = PeftModel.from_pretrained(base, "current-production-adapter")

    # New (candidate)
    new = PeftModel.from_pretrained(base, new_adapter_path)

    # Test on held-out examples
    previous_correct = 0
    new_correct = 0

    for example in heldout_eval_set:
        prev_output = previous.generate(example["prompt"], max_length=100)[0]
        new_output = new.generate(example["prompt"], max_length=100)[0]

        if compare_outputs(prev_output, example["expected"]) == "correct":
            previous_correct += 1
        if compare_outputs(new_output, example["expected"]) == "correct":
            new_correct += 1

    previous_accuracy = previous_correct / len(heldout_eval_set)
    new_accuracy = new_correct / len(heldout_eval_set)

    # Gate: new must not regress > 2%
    if new_accuracy < previous_accuracy * 0.98:
        return False, f"Regression: {previous_accuracy:.1%} → {new_accuracy:.1%}"

    return True, f"Pass: {new_accuracy:.1%} (was {previous_accuracy:.1%})"

If evaluation fails, send alert and keep the old adapter. Do not auto-release.

Release policy

Automatic: If new > previous by >1%, release immediately.

Previous accuracy: 88.5%
New accuracy: 89.7%
Delta: +1.2%
→ Auto-release v1.2.1

Gated: If new ≈ previous (within ±1%), release with a flag beta=true.

Previous: 88.5%
New: 88.8%
Delta: +0.3%
→ Release as beta; monitor 24h before prod

Held: If new < previous, do not release. Send alert to team.

Previous: 88.5%
New: 87.2%
Delta: -1.3%
→ Held; investigate regression

Rollback

Keep the previous 3 versions available:

v1.2.1 (current, in production)
v1.2.0 (previous, available for rollback)
v1.1.9 (older, kept for reference)
v1.1.8 (deleted; too old)

If v1.2.1 causes issues in production, rollback to v1.2.0:

mfl switch qwen25-7b-support-tone v1.2.0

Monitoring

Track metrics over time:

metrics_log = {
    "adapter": "qwen25-7b-support-tone",
    "versions": [
        {"version": "1.2.0", "accuracy": 0.885, "release_date": "2026-01-29"},
        {"version": "1.2.1", "accuracy": 0.897, "release_date": "2026-02-12"},
    ],
    "production_version": "1.2.1",
    "last_retraining": "2026-02-12T02:00:00Z",
}

Dashboard shows: - Accuracy trend (improving? degrading?) - Retraining cadence (every week? every month?) - Failure rate (how often does eval gate fail?) - Rollback frequency (stability measure)

Cost budget

Each retraining costs GPU time:

1 GPU-hour × $0.50 (spot) = $0.50 per run
Weekly retraining = 52 runs/year = $26/year per adapter
10 adapters = $260/year

Set a soft cap: if budget exceeded, switch to monthly retraining.

Example for support-tone adapter

# adapters/qwen25-7b-support-tone/retrain-config.yaml
adapter: qwen25-7b-support-tone
trigger: weekly  # Every Monday 2 AM
batch_days: 7
base_model: Qwen/Qwen2.5-7B-Instruct
base_model_pin: abc123def  # Locked version
rank: 8
alpha: 16
seed: 42
eval_heldout_size: 100
eval_regression_threshold: 0.02  # 2% regression gates release
eval_improvement_threshold: 0.01  # 1% improvement auto-releases
rollback_versions_kept: 3

Continuous training is the difference between an adapter that works for 3 months and one that works for 3 years. The initial training is the easy part; staying good is hard.