Setting up canary rollouts for a new model version

Canary rollouts let you validate a new model against real production traffic before committing fully. This guide walks through the complete setup in ManyLayers Gateway — from defining the alias to promoting or rolling back.

Prerequisites

• ManyLayers Gateway running and receiving production traffic
• An existing alias pointing to your current production model
• Access to Platform Settings → Aliases and Analytics → Cost
• Baseline eval results for your current model (P95 latency, cost per 1,000 requests, guardrail trigger rate)

Step 1: Record your baseline metrics

Before changing anything, establish the baseline you’ll compare against during the canary.

1. Navigate to Analytics → Cost and note the average cost per 1,000 requests for the alias you’re changing.
2. Navigate to Analytics → Latency and record the P50 and P95 latency for the same alias over the past 7 days.
3. Navigate to Analytics → Guardrails and note the guardrail trigger rate for the alias.
4. Run your eval suite against the current model and record pass rates for your key task types.

Save these numbers somewhere accessible. They are your rollback thresholds.

Step 2: Add the new model target to your alias

Navigate to Platform Settings → Aliases → [Your Alias] → Edit.

Add the new model as a second route with a small weight. Keep the existing model at the majority weight:

aliases:
  chat-production:
    routes:
      - target: openai/gpt-4o
        weight: 95
      - target: openai/gpt-4o-2026-07
        weight: 5
        fallback_to: openai/gpt-4o

Setting fallback_to on the canary target means any 5xx errors or timeouts on the new model automatically fall back to the established model. This caps downside risk during the initial canary window.

Save the configuration. Changes propagate in under a second — no deployment required.

Step 3: Set up monitoring split by route target

The audit log tags every request with the resolved route target. Your analytics dashboard can slice on this field.

1. Navigate to Analytics → Requests and group by route_target.
2. Create two panels: one showing the established model bucket, one showing the canary bucket.
3. Track P95 latency, error rate, and cost per request for each bucket independently.

If your observability stack ingests ManyLayers logs directly, add a route_target dimension to your existing latency and error dashboards.

Step 4: Monitor the 5% canary window (Day 1–2)

Watch the canary bucket for 24–48 hours at 5% weight before increasing. Look for:

• Hard errors (5xx): any error rate above 0.5% on the canary bucket is a rollback signal.
• Catastrophic latency regression: P95 more than 2x the baseline warrants investigation.
• Guardrail trigger rate change: a significant increase suggests the new model is producing different output characteristics that interact differently with your guardrail policies.

If any of these thresholds are breached, set the canary weight to 0 and the established model to 100:

routes:
  - target: openai/gpt-4o
    weight: 100

This is your rollback — a config change, not a deployment.

Step 5: Ramp the canary incrementally

If the 5% window is clean, increase the weight in stages:

Day	Canary weight	Watch for
1–2	5%	Hard errors, catastrophic latency
3–5	20%	Quality regressions on narrow prompt types
6–9	50%	Cost and latency differences at statistical significance
10+	100% (if clean)	Final eval suite comparison

Don’t compress this schedule unless your traffic volume gives you statistical significance faster. At fewer than 500 requests per hour, 5% is 25 requests — not enough data to catch subtle regressions quickly.

Step 6: Connect the eval pipeline (recommended)

ManyLayers Gateway can shadow-evaluate canary requests against your eval suite asynchronously — no latency impact for users.

1. Navigate to Evals → Pipelines → [Your Pipeline] → Routing Integration.
2. Enable shadow_eval: true and link the pipeline to your alias.
3. Set an alert threshold: if the eval pass rate on the canary bucket drops below your baseline pass rate by more than 5 percentage points, trigger a webhook.

Configure the webhook in Platform Settings → Alerts → [Alert] → Destinations. Point it at your Slack channel or PagerDuty integration.

Step 7: Promote or roll back

To promote: Set the new model to weight 100 and remove the old target from the route list. Save the configuration.

To roll back: Set the old model to weight 100 and set the new model weight to 0 (or remove it from the route list). Save. The change takes effect immediately.

In both cases, no application code changes. The alias abstraction means your applications never reference a specific model version directly.

Checklist

• Baseline metrics recorded before any change
• New model added at 5% weight with fallback configured
• Monitoring dashboards split by route_target
• Rollback thresholds defined and documented
• Eval pipeline connected for continuous quality scoring
• Alert webhook configured
• Promotion decision based on data, not schedule
• Audit log reviewed after promotion for any anomalies

Common mistakes to avoid

Skipping the baseline. You cannot evaluate whether the canary is regressing if you don’t know what the baseline looks like.

Compressing the ramp schedule. Statistical significance accumulates with time and volume. Patience is the control.

Forgetting to remove the old target after promotion. An alias with a 0% weight target still shows up in your configuration and can cause confusion. Clean up after promotion.

Using canary routing on low-volume aliases. If an alias handles fewer than 200 requests per day, a 5% canary is 10 requests — too few for meaningful signal. Run offline evals against your request archive instead.