Migrating from a direct OpenAI integration to a gateway

If your application calls the OpenAI API directly, you have a single provider dependency, no centralized cost visibility, no guardrails, and no fallback if OpenAI has an outage. Moving to ManyLayers Gateway adds all of those capabilities — and for most applications, the migration requires changing one environment variable.

This guide covers the complete migration path: from a minimal drop-in replacement to a fully configured gateway deployment with routing, guardrails, and team-level cost control.

Prerequisites

• ManyLayers Gateway deployed (SaaS, hybrid, or self-hosted)
• Your OpenAI API key available for migration to the gateway’s credential store
• Access to your application’s deployment configuration
• Platform Admin access to Platform Settings

Step 1: Add your OpenAI API key to the gateway

The gateway holds your provider credentials centrally. Your applications will authenticate to the gateway, and the gateway authenticates to OpenAI on their behalf.

1. Navigate to Platform Settings → Providers → OpenAI → Add Credential.
2. Enter your OpenAI API key. The gateway encrypts it at rest — it is not accessible to users or visible in the UI after saving.
3. Give the credential a descriptive name: openai-production or openai-main.
4. Save. The gateway now has the credentials it needs to proxy OpenAI requests.

If you have multiple OpenAI API keys (for example, different keys for different cost centers), add each one and label them accordingly.

Step 2: Create a team and generate a gateway API key

Your application will authenticate to the gateway using a gateway-issued key, not the OpenAI key directly. This is how the gateway can apply per-team policies.

1. Navigate to Platform Settings → Teams → New Team. Create a team for the application you’re migrating (e.g., customer-support-app).
2. Navigate to Platform Settings → Teams → [Team] → Keys → New Key.
3. Name the key descriptively (e.g., support-app-prod). Set an expiry date if appropriate.
4. Copy the key. This is the value you’ll put in your application’s OPENAI_API_KEY environment variable.

Step 3: Create a gateway alias pointing to OpenAI

An alias is a named routing target that your application references. Using an alias rather than a direct model string means you can change routing behavior at the gateway level without touching application code.

Navigate to Platform Settings → Aliases → New Alias:

aliases:
  openai-gpt4o:
    routes:
      - target: openai/gpt-4o
        weight: 100

Note the alias name. You can use it as the model string in your application’s API calls, or you can configure the gateway to transparently map your existing model strings to the alias.

Step 4: Update your application’s base URL and API key

ManyLayers Gateway is OpenAI API-compatible. For most applications, the migration is a two-line environment variable change:

# Before
OPENAI_API_KEY=sk-your-openai-key
OPENAI_BASE_URL=https://api.openai.com/v1

# After
OPENAI_API_KEY=ml-your-gateway-key
OPENAI_BASE_URL=https://gateway.manylayers.io/v1

The OpenAI SDK picks up OPENAI_BASE_URL automatically. No code changes required. The gateway accepts requests in the same format as the OpenAI API and returns responses in the same format.

If your application hardcodes the base URL:

# Before
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# After
client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ.get("OPENAI_BASE_URL", "https://gateway.manylayers.io/v1")
)

This single change routes all OpenAI calls through the gateway.

Step 5: Verify traffic is flowing through the gateway

Deploy your updated application to a staging environment and send a few requests. Then:

1. Navigate to Analytics → Requests in the ManyLayers dashboard.
2. Confirm requests appear with the correct team attribution.
3. Check that the model, token counts, and latency look as expected.
4. Verify the audit log shows the requests in Audit → Logs.

If requests aren’t appearing, check:

• The OPENAI_BASE_URL is set correctly and includes the /v1 path suffix.
• The gateway API key is valid and belongs to the expected team.
• Network connectivity from your application to the gateway endpoint.

Step 6: Add a fallback provider (recommended)

Once traffic is flowing through the gateway, add fallback routing. This is one of the primary benefits of running a gateway, and it takes 30 seconds to configure:

aliases:
  openai-gpt4o:
    routes:
      - target: openai/gpt-4o
        weight: 100
        fallback_to: anthropic/claude-sonnet-4-5
    fallback_timeout_ms: 5000

Add Anthropic as a credential in Platform Settings → Providers → Anthropic → Add Credential first. With this configuration, any OpenAI 5xx error or timeout automatically retries against Claude Sonnet — your application sees a successful response, and you see the fallback in the analytics log.

Step 7: Enable guardrails

With all traffic flowing through the gateway, you can enable guardrails without touching application code.

Navigate to Platform Settings → Teams → [Team] → Guardrails:

• Enable PII firewall with your preferred redaction policy.
• Enable prompt injection detection if your application accepts user-supplied content.
• Set content moderation thresholds appropriate for your use case.

These guardrails apply to every request the team makes through the gateway — regardless of which application made the request.

Step 8: Set a budget cap

Navigate to Platform Settings → Teams → [Team] → Budget and configure a monthly budget cap based on your OpenAI spend history. Enable a 75% alert threshold so you’re notified before the cap is reached.

Phased migration for larger applications

If your organization has many applications calling OpenAI directly, migrate in phases:

Phase 1 — Centralize credentials. Move all OpenAI API keys into the gateway. Issue gateway keys to each application. Minimal change per application.

Phase 2 — Establish teams and attribution. Create one team per application or department. Reissue keys under the correct teams. Now you have cost attribution.

Phase 3 — Enable fallback routing. Add fallback providers to all production aliases.

Phase 4 — Enable guardrails. Start with audit-only mode (guardrails log but don’t block), review the logs, then enable blocking mode for the policies that are well-calibrated.

Phase 5 — Optimize routing. Use analytics data to identify high-volume, cost-sensitive workloads that can route to cheaper models. Introduce canary rollouts for model upgrades.

Checklist

• OpenAI API key added to the gateway credential store
• Team and gateway API key created for the migrating application
• Alias configured pointing to openai/gpt-4o (or your current model)
• OPENAI_BASE_URL and OPENAI_API_KEY updated in application config
• Traffic verified in Analytics and Audit logs
• Fallback provider credential added
• Fallback route configured on the alias
• Guardrails reviewed and enabled
• Budget cap and alert threshold configured

Common mistakes to avoid

Not including /v1 in the base URL. The gateway endpoint must include the /v1 path prefix for the OpenAI SDK to route requests correctly.

Using the OpenAI key as the gateway key. After migration, your application should use the gateway-issued key, not the OpenAI key directly. The OpenAI key lives in the gateway’s credential store only.

Skipping the staging verification step. Confirm traffic appears in the gateway analytics before deploying to production. Silent misconfiguration (where requests bypass the gateway entirely) is the most common migration failure.

Migrating all applications at once. Phase the migration. Start with a low-stakes internal application, verify the integration, then expand.