Choosing between hosted APIs and self-hosted open-weight models

The headline question for AI infrastructure in 2026 is not “which model should we use?” — it is “where should that model run?” Hosted APIs and self-hosted open-weight models are not competing philosophies. They are tools with different cost structures, latency characteristics, compliance profiles, and operational complexity. The teams that get this decision right deploy a mix, routing workloads to whichever option best fits each one.

This post is a practical framework for making that call, not a philosophical argument for either side.

The real cost comparison

The sticker price comparison between hosted APIs and self-hosted models is deceptive in both directions.

Hosted APIs look expensive when you compare per-token costs against the theoretical throughput of a GPU you already own. But that comparison ignores the cost of the engineers to operate it, the overhead of capacity planning, the cost of idle capacity during off-peak hours, and the time to market you lose while standing up infrastructure.

Self-hosted models look expensive when you’re calculating GPU hours until you account for the workloads where you need them most: high-volume, steady-state inference where dedicated capacity pays off quickly. A team running 200 million tokens per day on a model that costs $2/M tokens at a hosted API is spending $400/day — or $146,000/year. A well-utilized GPU cluster serving the same throughput often costs significantly less at scale.

The break-even point varies by model size, GPU type, and utilization rate. Run the calculation for your actual workload distribution before deciding. ManyLayers Analytics exports per-model token volume, which gives you the raw numbers to plug into a capacity model.

Latency characteristics

Hosted APIs are faster for cold-start situations: you make a request and the model is already running. For interactive applications, a hosted API that responds in 800ms consistently will outperform a self-hosted model that averages 600ms but spikes to 4,000ms when the serving node is handling a large batch.

Self-hosted models win on predictability when you control the hardware. You set the queue depth, the batch size, and the concurrency limits. There are no noisy-neighbor effects from other tenants. For real-time applications — voice assistants, interactive coding tools, streaming chat — a dedicated self-hosted model with predictable tail latency may justify the operational overhead.

Network topology matters too. If your application servers are colocated with a self-hosted model, the round-trip cost to a cloud API endpoint may be more significant than the inference time itself.

Compliance and data residency

This is where the decision often becomes non-negotiable.

Hosted APIs involve sending your prompts and completions to a third-party provider’s infrastructure. Most enterprise providers offer contractual commitments about data retention, processing location, and training use. Some workloads are compatible with these commitments. Others are not.

Categories that frequently require self-hosted models:

• Prompts containing patient health information (HIPAA covered entities with strict interpretations)
• Legal privileged communications
• National security-classified information (any classification level)
• Data subject to sector-specific residency requirements (financial services in certain jurisdictions, government data in others)
• Prompts containing trade secrets where the risk of inadvertent training exposure is unacceptable

ManyLayers supports air-gapped self-hosted deployments where no traffic leaves your network perimeter. The model library includes both GPU-optimized and CPU-fallback configurations for environments without internet access.

Capability gaps: where they still exist and where they’re closing

The capability gap between frontier hosted models and the best open-weight alternatives has narrowed dramatically over the past 18 months. For many common enterprise tasks — document summarization, code generation, structured data extraction, customer support — an open-weight model fine-tuned on your domain data matches or exceeds the performance of the best hosted model.

The gap remains significant for:

• Tasks requiring very recent world knowledge (open-weight models have fixed training cutoffs)
• Complex multi-step reasoning at long context lengths
• Multimodal tasks involving image understanding at production quality
• Tasks where zero-shot performance matters and fine-tuning isn’t feasible

Evaluate capability against your actual task distribution. The correct capability comparison is not “GPT-4o versus Llama-3.1-70B on a general benchmark.” It is “which model performs best on the specific prompts our application sends?”

Operational complexity

Self-hosted models require infrastructure. Someone on your team needs to own:

• Model deployment and version management
• Hardware capacity planning and scaling
• Serving framework configuration (vLLM, TGI, or comparable)
• Monitoring, alerting, and incident response
• Fine-tuning pipelines if you’re customizing models

ManyLayers simplifies this through the model deployment system, which provides Kubernetes and dstack-based deployments with built-in health checks, autoscaling, and integration into the gateway routing layer. Fine-tuning is available via the LoRA fine-tuning pipeline without requiring ML engineering expertise. But “simplified” is not “free” — you still need engineers who understand what they’re operating.

If your organization doesn’t have the operational bandwidth for model infrastructure, the hosted API path is more rational regardless of the cost comparison.

The hybrid routing pattern

The most practical architecture for most organizations is not “hosted APIs only” or “self-hosted only.” It is a hybrid:

• Default traffic routes to a hosted API for flexibility and zero operational overhead
• Sensitive workloads (tagged by the calling application via request metadata) route to a self-hosted model
• High-volume, cost-sensitive workloads route to a self-hosted model after validation
• Experimental and low-volume workloads route to hosted APIs for easy iteration

ManyLayers Gateway’s conditional routing makes this pattern straightforward to implement:

aliases:
  production-chat:
    conditional:
      - condition: "metadata.data_classification == 'restricted'"
        target: self-hosted/llama-3-1-70b
      - condition: "metadata.volume_tier == 'high'"
        target: self-hosted/mistral-7b-instruct
      - default:
          target: openai/gpt-4o

The routing policy lives in the gateway, not in application code. Switching a workload from hosted to self-hosted is a config change that takes effect in under a second, with no application deployments required.

Practical guidance

• Run hosted APIs for new workloads until you understand your actual token volumes and latency requirements. Don’t optimize prematurely.
• Identify your compliance-constrained workloads first. These go self-hosted regardless of cost.
• Set a volume threshold that triggers a cost analysis. A reasonable starting point is 50 million tokens per month per model — below that, the operational overhead of self-hosting rarely pays off.
• Fine-tune for your domain before concluding a smaller model can’t do the job. The capability gap between a fine-tuned 7B model and a general-purpose 70B model is often smaller than expected for narrow tasks.

Takeaway

The hosted-versus-self-hosted decision is a portfolio question, not a platform choice. Most organizations should answer it per workload, using compliance requirements as a hard filter and cost-volume analysis as the economic guide. The right infrastructure is one where routing is flexible enough to move workloads between hosted and self-hosted without rewriting applications — which is exactly the role the gateway layer plays.