LLM observability: traces, token accounting, and cost attribution

Traditional application observability — error rates, latency percentiles, saturation — is necessary but insufficient for LLM applications. A request that returns HTTP 200 in 800ms can still be a complete failure: the model hallucinated, the response was truncated, the retrieved context was irrelevant, or the user paid for 8,000 tokens to get an answer that could have come from a 200-token prompt. None of those failures show up in your existing dashboards.

This post covers the observability primitives that actually matter for LLM workloads: per-request traces, token accounting, cost attribution by feature and user, latency decomposition, and the alerting thresholds worth setting.

The shape of an LLM request

Before instrumenting anything, understand what a single LLM request actually consists of. What your application calls “one inference” may involve:

1. Embedding the user query (one model call)
2. Vector search over your index (one DB query)
3. Reranking retrieved chunks (optionally another model call)
4. Constructing the final prompt (CPU, no model call)
5. Calling the LLM for generation (one or more model calls if streaming)
6. Post-processing the response (regex, parsing, validation)
7. Storing the result in cache or conversation history (one DB write)

Each of these steps has its own latency, its own failure modes, and in the case of model calls, its own token cost. A trace that shows only step 5 — the generation call — is hiding most of the story. Your observability must span the full request lifecycle.

Per-request traces

A useful LLM trace has at minimum these spans:

[request] /api/chat POST
  [retrieval] vector-search: 12ms, 8 chunks retrieved
  [rerank] cross-encoder: 45ms, 8→3 chunks kept
  [prompt-build] 2ms, 2847 tokens constructed
  [llm-call] openai/gpt-4o: 623ms, 2847 prompt + 341 completion tokens
    [stream-first-token] 201ms
    [stream-complete] 623ms
  [cache-write] 3ms
[request total] 686ms

The fields that matter on each LLM span:

• Provider and model — not just “the LLM” but the specific model string (gpt-4o-2024-08-06, claude-3-5-sonnet-20241022). Models change behavior across versions; you need this for attribution and debugging.
• Prompt token count — input tokens, including system prompt, retrieved context, conversation history
• Completion token count — output tokens
• Time to first token (TTFT) — the latency the user actually perceives during streaming
• Time to last token / total generation time
• Cache hit status — was this a semantic cache hit, exact cache hit, or cache miss
• Finish reason — stop, length, content_filter, tool_calls; a length finish reason at scale means you’re truncating model output

Traces should be linked to a session or conversation ID so you can reconstruct multi-turn interactions. Debugging “the third message in this conversation made the model go off-rails” requires that link.

Token accounting

Token counts are the throughput metric of LLM infrastructure. Track them at every level of granularity:

Per-request: prompt tokens + completion tokens, broken down by segment if possible (system prompt tokens, context tokens, user message tokens). Prompt token breakdown reveals optimization opportunities — a system prompt that’s 3,000 tokens on every request is costing you on every single call.

Per-endpoint or feature: aggregate token consumption by the endpoint or product feature that triggered the request. This tells you which features are your most expensive compute consumers, independent of request count.

Per-user: in multi-tenant applications, which users are driving token consumption? A small number of heavy users often account for a disproportionate share of cost. Identifying them lets you enforce fair-use policies, tier appropriately, or reach out proactively.

Per-model: when you’re routing to multiple providers, track token consumption separately per model. Your cost per token differs dramatically between models — GPT-4o input tokens versus Llama-3.3-70B hosted on your own infrastructure are not fungible.

Prompt token growth over time

One of the most insidious LLM cost patterns is prompt token creep. A system prompt that starts at 800 tokens gets revised over several months — more edge cases handled, more examples added, more instructions layered on — and grows to 3,500 tokens without anyone noticing. At 10,000 requests per day, that’s 27 million extra prompt tokens per day, costing real money. Tracking the p50 and p95 of prompt token count per endpoint over time catches this drift before it becomes a budget problem.

Cost attribution

Token counts are the quantity; cost attribution requires pricing. For each model call, compute:

request_cost = (prompt_tokens / 1_000_000 * input_price_per_million)
             + (completion_tokens / 1_000_000 * output_price_per_million)

Store this on every trace span. Then roll it up:

• Cost per request — the atomic unit; useful for p50/p95 cost distribution analysis
• Cost per feature per day — your primary budget accountability metric
• Cost per active user per day — useful for unit economics and tiering decisions
• Cost per session — for conversational applications, total cost of one conversation
• Cost per successful outcome — if you track task completion, cost/outcome is your efficiency metric

The last one is underused. Optimizing for cost/request in isolation can lead to cheaper but worse experiences. Optimizing for cost/successful-outcome aligns the metric with actual value.

Handling cached responses

Semantic and exact caching complicates cost accounting in a useful way. A cache hit has near-zero marginal model cost, but it consumed resources to build the index. Track cache hits distinctly:

• Hit rate by endpoint (what fraction of requests are served from cache)
• Cost avoided by cache hits (the model call cost that would have been incurred)
• Cache staleness — how old was the cached response when it was served

Cache hit rate is a cost-reduction metric; cache staleness is a quality metric. You need both.

Latency decomposition

LLM request latency has a structure that conventional p50/p95 analysis obscures. The two dimensions that matter are:

Time to first token (TTFT): perceived latency for streaming UIs. Users feel this immediately. TTFT is dominated by queue time at the provider plus prompt processing time. Long system prompts increase TTFT. Context-heavy RAG prompts increase TTFT.

Generation throughput (tokens/second): the speed at which tokens arrive after the first token. This determines how long a user waits for a long response. It is largely determined by the model size and the serving infrastructure; you have limited control over it when using hosted APIs, but it is critical when running self-hosted models.

Track both. Report TTFT and total generation time separately in your dashboards. A request might have excellent TTFT but poor throughput (or vice versa), and aggregating them into a single latency number hides the true user experience.

Latency by provider and model

When routing across multiple providers, latency is not uniform. Keep per-model-per-provider latency histograms. Provider A may have lower p50 latency but much higher p99 due to tail-latency spikes; provider B may be consistently slower but more predictable. Your routing decisions — and your SLA commitments — should be informed by the actual latency distributions you’re observing, not by published benchmarks.

A table your dashboard should be able to produce on demand:

Provider/Model	p50 TTFT	p95 TTFT	p50 total	p95 total	Error rate
openai/gpt-4o	210ms	890ms	1.2s	4.1s	0.3%
anthropic/claude-3-5-sonnet	180ms	760ms	1.4s	5.2s	0.1%
self-hosted/llama-3.3-70b	340ms	1100ms	2.1s	6.8s	0.05%

What to alert on

Not every metric deserves an alert. The signals worth waking someone up for:

Error rate spike above baseline. Define a per-endpoint error rate baseline using a 7-day rolling window. Alert when the rate exceeds 2x baseline for more than 5 minutes. Absolute thresholds miss baseline variation; relative thresholds are more robust.

TTFT p95 exceeding SLA. If your application commits to a user experience with time-to-first-token under 1 second on the 95th percentile, alert when that threshold is breached. p95, not p50 — p50 can look fine while 5% of users are having a terrible experience.

Unexpected length finish reason rate. If more than 5% of your requests are finishing because the model hit the max token limit rather than completing naturally, your generation budget is misconfigured or your prompts are too long. This is a silent quality problem.

Cost per request deviation. Alert when cost-per-request for an endpoint increases by more than 30% compared to a 24-hour rolling baseline. This catches prompt token creep, model routing changes, or new traffic patterns that are unexpectedly expensive.

Cache hit rate collapse. If a cache warming strategy is part of your cost model, a sudden drop in hit rate means you’re spending money you planned not to spend.

Provider error diversity. Alert if you start seeing error types you haven’t seen before from a provider — new error codes, new finish_reason values, or unusual response patterns. These often precede provider incidents.

Connecting traces to application context

Model-level observability is necessary but not sufficient. You need to connect traces to application-level context:

• Which feature or workflow triggered this request
• The user’s session ID and tenant ID
• Whether the response was rated by the user (thumbs up/down)
• Whether a downstream task succeeded or failed

Without this connection, you can tell that costs are rising but not why. With it, you can trace a cost spike to a specific feature shipped last Tuesday, or correlate a latency regression with a model version rollout.

The mechanism is trace context propagation: pass a trace ID from your application layer through the model gateway and into your storage. Every instrumentation decision downstream should preserve this ID so you can join across systems.

ManyLayers observability

ManyLayers Gateway emits per-request traces with token counts, cost attribution, latency decomposition, finish reasons, and cache hit status — all accessible via the dashboard and exportable to your existing observability stack (Datadog, Grafana, OpenTelemetry-compatible backends). Budgets are enforced at the gateway layer; when a per-user or per-feature budget threshold is approached, you get an alert before you exceed it. Model-level latency histograms are tracked per provider and per model string, so routing decisions can be informed by the actual distributions in your traffic rather than generic benchmarks.

Observability is not a nice-to-have for LLM applications — it is how you make principled decisions about model selection, prompt optimization, caching strategy, and budget allocation. Build the instrumentation before you need to debug an incident.