Zero-trust access for AI: virtual keys, RBAC, and key rotation without downtime

When your team starts using an LLM API, the typical first step is to grab the provider’s API key, paste it into an environment variable, and ship it. Six months later you have that key in fifteen services, three developer machines, one CI pipeline, and a Slack message from eight months ago that never got deleted. Nobody knows what everything is calling with it, and rotating it means coordinating a flag day across every consumer simultaneously.

This is not a hypothetical scenario — it describes most teams’ actual situation. The fix is to apply zero-trust principles to model access: every caller gets its own credential with exactly the permissions it needs, those credentials are audited, and rotating or revoking any of them takes seconds and affects only that credential.

The problem with shared API keys

A single shared provider API key has three compounding problems:

No attribution. When you get a bill with 40 million output tokens in one day, you cannot tell whether that came from your production search feature, a developer running an experiment, a runaway automated job, or a compromised consumer. You cannot act on information you don’t have.

No least privilege. The key can call any model the account has access to, with no rate limit per consumer, no model restriction, and no spend cap. A developer testing a new workflow has the same permissions as your production traffic.

Catastrophic rotation cost. If the key is compromised, or if you simply want to rotate it on a quarterly schedule, you must update every consumer simultaneously. In practice, this means key rotation rarely happens at all — which means compromised keys often remain active far longer than they should.

Virtual keys as the abstraction layer

The solution is to place a gateway between your consumers and the provider, and issue virtual keys that map to real provider credentials. The virtual key is what consumers hold and rotate. The real provider key lives only in the gateway, visible to no consumer.

A virtual key configuration looks roughly like this:

virtual_key:
  name: "search-feature-prod"
  key_id: "vk_prod_search_a7f3"
  maps_to: openai_prod_account
  scopes:
    allowed_models:
      - gpt-4o
      - text-embedding-3-large
    allowed_endpoints:
      - /v1/chat/completions
      - /v1/embeddings
  limits:
    rpm: 200
    tpm: 500000
    daily_spend_usd: 80.00
  expiry: "2026-12-31"
  owner: team:search

With this model:

• The search feature can call GPT-4o and the embedding model, nothing else
• It is rate-limited to 200 requests per minute regardless of what other consumers are doing
• It cannot spend more than $80/day; if it hits that cap, requests return a budget error, not an unexpected provider bill
• It expires on a specific date, forcing explicit renewal — passive key hygiene

The real OpenAI API key never leaves the gateway. If the vk_prod_search_a7f3 key is compromised, you revoke it in the gateway. Nothing else is affected. You issue a new virtual key, update the one consumer, and the rotation is complete in under five minutes.

Team RBAC

Virtual keys solve the per-service credential problem. Team RBAC solves the human-access problem. The two are related but distinct.

An RBAC model for AI access typically looks like this:

Roles:

• admin — can create and revoke virtual keys, view all audit logs, modify rate limits and budget caps, manage RBAC assignments
• developer — can create virtual keys scoped to allowed models for their team, view their team’s audit logs, cannot modify limits above their team’s quota
• analyst — read-only access to dashboards, cost reports, and audit logs; cannot create keys or make model calls directly
• service-account — non-human identity for automated workloads; cannot create keys, interacts only via its own virtual key

Team hierarchy: Teams correspond to product areas or organizational units. Each team has a model allowlist and a budget allocation. Members of a team can only create virtual keys within their team’s allowlist and cannot exceed the team’s aggregate budget.

This means the security perimeter is not just “can this key call the API” — it’s “can this human or service create credentials, and within what constraints.” An engineer on the search team can create a virtual key for a new experiment, but only for models their team is authorized to use, and only up to the budget their team has been allocated.

Model allowlists at the team level

Model allowlists deserve emphasis. Not every team should have access to every model. A customer-facing chatbot team probably doesn’t need access to experimental preview models that are not production-stable. An internal analytics team probably doesn’t need access to fine-tuned model checkpoints that are licensed for a specific product vertical.

Beyond access control, model allowlists are a compliance tool. If your organization has approved specific model versions for use with customer data — based on a DPA with the provider, or internal security review — allowlists enforce that approved list automatically. Any request to an unapproved model is rejected at the gateway before reaching the provider.

Audit trails

An audit trail for model access is different from an application log. It must capture:

• Who called what: the virtual key identifier, the resolved team and owner, the model called
• What was sent: prompt token count (and optionally the prompt hash, for content auditing without storing sensitive prompts in plain text)
• What was returned: completion token count, finish reason
• When: timestamp with millisecond precision
• From where: source IP, user-agent, request ID that ties back to your application tracing

The audit trail should be append-only and write to a separate, access-controlled store. The same team that has write access to the gateway configuration should not have delete access to the audit log. This is the basic integrity guarantee that makes audits meaningful.

Practical audit log schema:

{
  "ts": "2026-03-05T14:22:07.341Z",
  "request_id": "req_9k2mp",
  "virtual_key_id": "vk_prod_search_a7f3",
  "team": "search",
  "owner_label": "search-feature-prod",
  "model": "openai/gpt-4o-2024-08-06",
  "prompt_tokens": 1842,
  "completion_tokens": 207,
  "finish_reason": "stop",
  "latency_ms": 812,
  "cost_usd": 0.00743,
  "source_ip": "10.0.1.44",
  "user_agent": "my-app/2.1.0"
}

When a security incident requires you to answer “what did this compromised key access, and when?” you want to be able to run a query, not reconstruct events from scattered application logs.

Key rotation without downtime

Rotating API credentials in a running system is a coordination problem. The naive approach — revoke old key, issue new key, update every consumer simultaneously — requires either a maintenance window or a very fast coordinated deploy. Neither is acceptable for production services.

The zero-downtime rotation pattern uses key versioning:

1. Issue a new virtual key alongside the existing one. Both are valid.
2. Update consumers gradually — one service at a time, one environment at a time, using standard deploy workflows.
3. Monitor the old key’s traffic in the audit log. When traffic to the old key drops to zero, it has been fully migrated.
4. Revoke the old key. Zero traffic means zero impact.

This pattern works because the gateway is the single control plane for both keys. The provider’s real API key doesn’t change at all. The only thing rotating is the virtual key that your consumers hold, and because both are valid simultaneously during the transition window, there is no moment where a consumer is left without a working credential.

For automated key rotation — quarterly rotations on a schedule, or rotation triggered by a security event — this same pattern can be run programmatically. The gateway API issues the new key, an automation script updates consumer secrets in your secrets manager (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager), and after a configurable overlap window, the old key is revoked.

Expiry enforcement as a policy mechanism

Key expiry is underused as a security control. A key with no expiry date is implicitly trusted forever. A key with an expiry date must be explicitly renewed, which is an opportunity to ask: does this credential still need to exist? Has the service it was created for been decommissioned? Has the team that owns it changed?

A practical expiry policy:

• Production service keys: 90-day expiry with automated renewal via CI pipeline
• Developer experiment keys: 14-day expiry, manual renewal required
• CI/CD pipeline keys: 30-day expiry, renewed as part of the pipeline itself
• All keys: expiry notifications 14 days before, 7 days before, and 1 day before

Expiry notifications should go to the key’s owner, not to a shared channel. Shared channel alerts get ignored. Owner alerts create accountability.

Detecting anomalous access

Audit trails enable detection, but detection requires alerting on patterns, not just logging them. Signals worth monitoring:

• Requests from new source IPs for a given virtual key — a key being used from an unexpected network may indicate compromise
• Request volume spikes beyond the key’s historical pattern — a 10x spike in one hour warrants investigation even if the rate limit hasn’t been hit
• Model calls outside normal hours for a given key — a production service key making requests at 3am when the service typically has near-zero traffic
• Repeated authentication failures — a virtual key being tried with the wrong credentials or from an unexpected context

These patterns don’t prove compromise, but they create signal worth reviewing. Combine them with your broader security monitoring.

ManyLayers access control

ManyLayers Gateway implements virtual keys, per-key model allowlists, per-key rate limits and budget caps, team RBAC with admin/developer/analyst roles, append-only audit logs with export to S3 or your SIEM, and automated key expiry with configurable notifications. Key rotation follows the dual-validity pattern — new key can be issued before the old one is revoked, with zero downtime. If you’re integrating with SSO and SCIM for identity management, virtual key ownership syncs with your directory’s team structure so offboarding a user automatically orphans their keys for review.

Zero-trust model access is not a complexity you have to build from scratch. It is a set of well-understood patterns applied to a new class of credential. The cost of not applying them is an unaudited, unrotated, over-privileged key that nobody wants to touch because they don’t know what depends on it.