Build vs. buy for LLM gateways: an honest decision framework

Almost every engineering team that starts using LLMs in production builds a proxy. It usually starts as a thin wrapper around the OpenAI client — a few dozen lines that adds an API key from an environment variable, maybe a retry loop, maybe some basic logging. It ships in a week and works fine for six months.

Then the team starts hitting the cases the wrapper did not anticipate. A provider has an outage. A team member leaks a key. Finance asks what the $40,000 cloud bill is for. A compliance officer asks whether the prompts are being logged and whether they contain PII. The wrapper, which solved the original problem efficiently, now has a backlog of requirements it was never designed to handle.

This is the build-vs-buy decision for LLM gateways. This post is an honest assessment of where homegrown solutions struggle, what a realistic total cost of ownership looks like, and what to evaluate when looking at off-the-shelf options.

What a homegrown proxy typically gets right

Be fair to the build path. A custom proxy is not inherently wrong. It solves the immediate problem — centralizing provider credentials, adding a request/response logging hook, providing a stable internal endpoint that teams call instead of provider endpoints directly.

For a small team using a single provider with one or two models, this is often sufficient. The proxy is simple enough to understand completely, has no external dependencies to update, and can be modified freely as requirements change. Teams with strong operational capacity and unusual requirements (a deeply custom routing algorithm, a proprietary data format, integration with internal systems that no standard gateway handles) often find that a bespoke proxy pays for itself.

The problems emerge with scale, not with the initial implementation.

The seven things homegrown proxies reliably miss

1. Fallback routing with state awareness

A retry loop is not a fallback. Retrying the same model at the same provider will fail if the failure mode is an outage or rate limit. Fallback requires knowledge of alternative providers, the ability to re-format the request for a different provider’s API schema, and the ability to detect whether the fallback is currently healthy before attempting it.

Building this correctly requires maintaining a live health model of multiple providers, handling provider-specific error codes (rate limits, content policy rejections, and timeouts each require different responses), and managing partial failures gracefully. Most custom proxies implement a version of this that works for the happy path and breaks in novel failure modes.

2. Per-team budget enforcement with atomic accounting

Log-based spend tracking — write the token counts to a database and query them later — does not enforce budgets in real time. A team that sends 50 concurrent requests will pass all 50 budget checks before any of them complete and post their costs. Reliable enforcement requires atomic reservation of estimated cost before the request is forwarded, with reconciliation after completion.

This is a concurrency problem, and solving it correctly with a high-throughput proxy requires careful design: either a Redis-backed atomic counter or a short-lived reservation system. Teams that try to solve it with a simple database read/write under concurrent load eventually hit a budget violation that slips through.

3. PII detection at latency budgets that work for chat

Scanning prompts for PII before forwarding them is not hard to stub. Building a scanner that reliably catches SSNs, names embedded in prose, medical identifiers, and international phone number formats — at sub-5ms latency — while handling 1,000 requests per second is a different problem. Most custom implementations either accept high false-negative rates (by using only simple regex patterns) or accept high latency (by calling a remote scanning service on the critical path).

A well-optimized scanner runs regex passes on a compiled pattern library in microseconds, and escalates to a lightweight local model only when the cheaper pass is ambiguous. This is not complex to build, but it is time-consuming to get right, and the parameter tuning (confidence thresholds, pattern coverage by locale) is ongoing maintenance.

4. Audit logs that satisfy compliance reviewers

Compliance-grade audit logs require: every request logged with no gaps, immutable storage, field-level evidence of guardrail evaluations, and export to SIEM or compliance tooling. Most homegrown logs are designed for debugging, not compliance. They are mutable (because they live in a database that engineers can modify), incomplete (because someone disabled logging to debug a performance issue and forgot to re-enable it), and poorly structured for compliance queries.

Retrofitting a debugging log into a compliance-grade audit log is expensive. The schema changes are significant, the immutability requirement often requires infrastructure changes, and the export integrations need to be built from scratch.

5. Semantic caching with invalidation

Request-level caching based on exact prompt hash is the version most homegrown proxies implement. It helps with repeated identical requests (CI test runs that hit the same prompts, for example) but misses the majority of caching opportunity. Two prompts that ask the same question in slightly different words are cache misses in a hash-based cache but could share a cached response in a semantic cache.

Building a semantic cache requires embedding incoming prompts, storing embeddings alongside cached responses, and performing nearest-neighbor search at query time with a configurable similarity threshold. Getting the threshold right — high enough to avoid false matches, low enough to actually get cache hits — requires evaluation against real traffic. The infrastructure (an embedding model, a vector index, a cache store with TTL) adds operational overhead.

6. Model aliasing with routing logic

Applications hardcode model identifiers (e.g., gpt-4o). When a better, cheaper model becomes available, every application has to be updated. A gateway that supports model aliases decouples the model identifier used in application code from the model that actually serves the request. The platform team updates the alias target; no application code changes.

This sounds simple to implement and is — for the basic case. The complexity is in routing logic that considers cost, latency, performance characteristics, and current provider health when resolving an alias at runtime. And in the change management that ensures alias updates are tested before being pushed to production traffic.

7. Multi-tenant access control with SSO integration

A shared API key is not access control. As teams multiply, you need per-team keys, per-team rate limits, per-team budget policies, and audit trail linkage to individual users. This requires an identity layer: users authenticate via SSO, their identity propagates to the gateway, and all logging and budget accounting is keyed to their identity.

Building this integration with an enterprise IdP (Okta, Azure AD, Google Workspace) and keeping it current as provider libraries update is ongoing work, not a one-time implementation.

Total cost of ownership

The initial proxy takes a senior engineer a week to build. The ongoing cost is what teams underestimate.

Cost center	Typical annual estimate
Initial build	1–2 engineer-weeks
Provider library maintenance	2–4 engineer-weeks/year
Feature backlog (fallbacks, budgets, PII, caching)	10–20 engineer-weeks/year
Incident response and reliability	4–8 engineer-weeks/year
Compliance and security reviews	2–4 engineer-weeks/year
Total ongoing	18–36 engineer-weeks/year

At a fully-loaded senior engineer cost of $250–400/hour, the ongoing maintenance of a custom proxy runs $225,000–$720,000 per year in engineering time — before accounting for opportunity cost.

This is not an argument that buy always wins. An off-the-shelf gateway has license costs and its own limitations. But teams that choose build often do so without accounting for the ongoing maintenance burden, and discover it when the proxy engineer leaves or the compliance requirement arrives.

Evaluation checklist for off-the-shelf gateways

When evaluating a gateway product, the questions that matter most are often not on the feature comparison table:

Routing and reliability

• Does fallback routing handle provider-specific error codes correctly, or does it retry on every error including content policy rejections?
• Can fallback chains be configured per-route, not just globally?
• Is provider health tracked dynamically, or does failover require manual configuration?

Cost and budget management

• Is budget enforcement atomic under concurrent load, or is there a race condition?
• Can budgets be expressed in USD (not just token counts)?
• Are alert thresholds configurable per-budget with distinct notification channels?
• Can request metadata (project, feature, environment) be used for cost attribution?

Security and compliance

• Does the PII scanner run locally (no data egress for the scanning itself), or does it call a remote API?
• Are audit logs immutable? Can the platform team delete log entries?
• Does the audit log capture guardrail decisions (not just request metadata)?
• Can content-class logs have a shorter retention period than metadata logs?

Integration and operations

• Does the gateway expose its own OpenTelemetry-compatible metrics?
• What is the deployment model? Can it run in an air-gapped environment?
• How are provider credentials stored and rotated?
• Is the access control model compatible with your IdP?

Vendor risk

• Is there a way to export all configuration and audit data if you need to migrate?
• What is the upgrade path? How often do breaking changes occur?
• Can the gateway run on your own infrastructure, or is it SaaS-only?

The last question is increasingly important for regulated industries. A SaaS gateway that processes all your prompts means a third party sees all your AI traffic. For many teams this is acceptable; for others it is a disqualifier regardless of how strong the vendor’s security posture is.

When to build, when to buy

Build when:

• Your requirements are genuinely unusual (a proprietary routing algorithm, a custom data format, deep integration with internal systems that no standard gateway supports).
• You have a small, stable team using one or two providers with no compliance requirements.
• You have the operational capacity to own the proxy long-term, including the maintenance and feature backlog described above.

Buy when:

• Compliance requirements (audit logs, PII handling, budget enforcement) are non-negotiable.
• You have multiple teams with different budget allocations and access policies.
• You want the ability to add providers, models, and routes without code changes.
• Your engineering team’s time is better spent on the product than on proxy infrastructure.

The pivot point is usually a compliance requirement or a finance conversation. Teams that are doing well with a simple proxy often hit one of these and realize that retrofitting the proxy to meet the requirement is more work than switching to a product built for it.

ManyLayers Gateway covers the full set of requirements described here — fallbacks, budgets, PII guardrails, audit logs, semantic caching, model aliasing, SSO integration — and runs on your infrastructure in self-hosted or hybrid mode. If you are in the middle of a build-vs-buy evaluation or finding that your existing proxy has accumulated a backlog of missing features, it is worth a look.