Token budgets and chargeback: making AI spend accountable

AI spend follows a predictable trajectory in most organizations. The first few months feel cheap — a few experiments, some prototypes, a handful of teams with API keys. Then usage grows faster than anyone anticipated, the monthly invoice arrives, and nobody can explain where the money went. Finance asks for a breakdown by team. Engineering shrugs. The API key is shared.

This is the token budget problem. It is not primarily a technical problem; it is a governance problem that manifests as a technical gap. The solution requires both: organizational decisions about who owns AI spend, and infrastructure that can enforce those decisions in real time.

Showback versus chargeback

Before building anything, your organization needs to make a policy decision: do you want showback or chargeback?

Showback means you attribute costs to teams without actually transferring funds. Each team receives a report showing what their AI usage cost. It creates visibility and mild social pressure but no hard constraint. Teams can exceed their “budget” without consequence beyond a conversation.

Chargeback means costs are actually allocated — either as real fund transfers between cost centers or as hard limits that stop the request when the budget is exhausted. It creates accountability, but it also creates friction: teams complain when their features stop working at the end of the month because they burned through their allocation.

The right choice depends on your organization’s maturity and culture. Most teams start with showback to get baseline visibility, discover which teams are the heavy users, and then move to chargeback once they have enough data to set fair initial allocations. Jumping straight to hard limits before anyone understands their typical spend is a recipe for developer frustration and helpdesk tickets.

What “budget” actually means at the gateway layer

A token budget needs to be enforced at the point where every request passes through. If teams can call model providers directly using their own keys, no gateway-layer budget will help. The prerequisite is that all AI traffic flows through a central proxy — every team, every application, every model call.

Given that, a budget is a policy object associated with an identity (team, project, API key, or user) that tracks accumulated spend against a threshold. When a request arrives, the gateway:

1. Identifies the requester from the API key or auth token.
2. Looks up the active budget for that identity.
3. Checks whether the request would exceed the remaining allocation.
4. Either allows the request and charges the tokens against the budget, or rejects it with a 429 and a policy-violation code.

The charge must happen atomically with the request — or at least the reservation must. If you check the budget before forwarding but only deduct after the response arrives, concurrent requests can all pass the check while collectively exceeding the limit. Use an atomic compare-and-subtract on a counter, or reserve an estimated token count before forwarding and reconcile against actual usage after the response.

Defining budget dimensions

Budgets can be expressed in several units. Token counts are the most direct but hardest to plan around — teams think in dollars, not tokens. Cost in USD is more intuitive but requires conversion at runtime based on the model and provider. Requests per period is simpler but ignores the difference between a ten-token classification call and a 100,000-token document summarization.

A practical approach is to store budgets in USD-cents and convert at enforcement time:

budgets:
  - identity: team_data_engineering
    period: monthly
    limit_usd: 1200.00
    alert_thresholds: [0.5, 0.8, 0.95]
    on_exceeded: reject

  - identity: team_product
    period: monthly
    limit_usd: 400.00
    alert_thresholds: [0.8]
    on_exceeded: alert_only

  - identity: key_experimental_sandbox
    period: daily
    limit_usd: 25.00
    on_exceeded: reject

The on_exceeded field distinguishes showback (alert_only) from chargeback (reject). You can run both modes simultaneously: enforce hard limits for well-understood production workloads, and use alert-only for experimental teams who need flexibility while they calibrate.

Alert thresholds and the noise problem

A single alert at 100% budget consumption is useless — by the time it fires, the team is already blocked. Useful alerting fires at multiple thresholds with increasing urgency:

Threshold	Action	Audience
50%	Informational notification	Team lead
80%	Warning notification	Team lead + platform team
95%	Urgent alert, request projected exhaustion date	Team lead, engineering manager, finance
100%	Hard stop (if chargeback) or incident escalation	All of the above

The projected exhaustion date requires a simple calculation: current spend rate (USD/day over the past 7 days) extrapolated against remaining balance. “You will hit your limit in 3 days at current burn rate” is actionable. “You are at 80% of budget” is not.

Calibrate alert channels to urgency. A Slack message to a team channel works for informational notifications. A page or direct message to a manager works for 95% threshold. Sending everything to the same Slack channel guarantees that all alerts will be ignored.

Cost attribution via request metadata

Budgets at the team level answer the “who” question but not the “what.” Teams running multiple products, projects, or workflows need attribution within their budget — not just a monthly total, but a breakdown by application or use case.

This requires metadata on the request, not just on the identity. The OpenAI API supports a metadata field on chat completion requests (and most providers support similar mechanisms):

{
  "model": "gpt-4o",
  "messages": [...],
  "metadata": {
    "project": "support-bot",
    "env": "production",
    "feature": "ticket-classification"
  }
}

A gateway that captures and indexes this metadata can produce attribution reports that are meaningful to teams:

Data Engineering — June 2026: $1,047.22

  By project:
    pipeline-summarizer         $612.40   58.5%
    anomaly-description         $298.17   28.5%
    ad-hoc-analysis             $136.65   13.0%

  By model:
    gpt-4o                      $891.55   85.1%
    claude-3-5-sonnet           $155.67   14.9%

This level of granularity transforms cost reviews from guesswork into engineering conversations. “Why did pipeline-summarizer spike in the second week of June?” is a question the team can actually answer.

Rolling versus calendar budgets

Monthly budgets reset on the first of the month. This creates a perverse incentive: teams that are close to their limit on the 28th will slow down usage, creating artificial gaps in coverage, while teams that renewed on the first will burn freely. Heavy end-of-month usage followed by a reset-day rush is a common pattern.

Rolling budgets track the past N days rather than calendar months. A 30-day rolling budget allows the same total spend but distributes it more evenly, because the budget is always measuring the most recent 30 days of activity. Rolling budgets are slightly harder to explain to finance teams, but they produce more predictable behavior.

A practical compromise: use calendar-month budgets for chargeback (because finance works in calendar months), but add a rolling 7-day spend rate cap to prevent any team from burning their entire monthly budget in the first week.

Handling multi-model workloads

Teams rarely use a single model. A workflow might call GPT-4o for reasoning, an embedding model for retrieval, and a smaller model for classification. Each call has a different cost-per-token. Budget accounting must aggregate across models, not track each separately.

Most teams are surprised to find that embedding calls, which feel “cheap,” represent a significant fraction of their spend at scale. A RAG pipeline that embeds 10,000 chunks per document ingestion at $0.00002/1K tokens will spend $0.20 per document — multiply by 50,000 documents and the embedding cost alone is $10,000. This is often invisible because embedding costs are hidden inside ingestion pipelines that nobody looks at closely.

Gateway-level cost attribution that captures every model call, including embeddings, is the only way to get an accurate picture.

Capacity planning with budget data

Once you have several months of budget data, you can use it for capacity planning. The questions that previously required a data engineer now answer themselves:

• Which teams are growing their AI spend fastest, and is that growth correlated with business value?
• Which model families are being used for which tasks, and are teams using the cheapest model that meets their quality bar?
• What is the organization’s total AI spend trajectory, and when does it become large enough to warrant a negotiated enterprise contract with providers?

Budget data at the gateway layer is the foundation for all of these questions. Without it, you are flying blind on an increasingly significant line item.

ManyLayers Gateway implements per-team and per-key budgets with configurable periods, alert thresholds, and enforcement modes. Spend is attributed in real time against every request, including metadata dimensions, and is queryable via the management API or exported to your BI tooling. If your current setup relies on checking the monthly invoice after the fact, it is worth looking at what enforcement at the gateway layer would change.

The organizational side

Technical budget enforcement solves the measurement and constraint problem. It does not solve the allocation problem — deciding how much each team should get.

Initial allocations are almost always wrong. Teams that have never had a budget set one based on gut feel, either under-estimating (they get blocked and frustrated) or over-estimating (the budget is meaningless because they never approach it). Plan for a calibration quarter where teams run in showback mode, collect real spend data, and then set allocations for the following quarter based on actual usage patterns plus a growth buffer.

Budget ownership should sit with the team or business unit, not the central platform team. The platform team sets the infrastructure — the budgets, the enforcement mechanism, the reporting. The business unit decides how much to allocate and how to distribute it internally. This separation keeps the platform team out of internal prioritization debates they should not be in.

Regular budget reviews — monthly for most organizations, quarterly for stable teams — close the loop between cost data and organizational decisions. The goal is not to minimize AI spend; it is to make sure AI spend is intentional and understood.