A practical webhook and alerting setup for budget overruns

A budget cap that silently blocks requests until someone notices the support tickets is worse than no cap at all. The point of per-team budgets is proactive visibility — your team lead should know they’re at 75% of their monthly budget before they’re at 100%, and your platform admin should be notified before hard caps start returning 429s to users.

This guide walks through configuring alert webhooks in ManyLayers Gateway, from the initial webhook setup to routing alerts to Slack, PagerDuty, and a custom handler for escalations.

Prerequisites

• ManyLayers Gateway with per-team budgets configured
• Platform Admin access to Platform Settings → Alerts
• A Slack webhook URL, PagerDuty routing key, or custom endpoint URL
• Budget policies set for at least one team (see the per-team budgets guide)

Step 1: Understand the alert event types

ManyLayers Gateway emits structured webhook payloads for three budget-related events:

Event type	Trigger
budget.threshold_reached	Team crosses the configured alert threshold (e.g. 75%)
budget.hard_cap_reached	Team hits the hard cap — requests are now being blocked
budget.daily_spike	A team’s daily spend exceeds 3x its 7-day average (anomaly detection)

Configure alert destinations per event type. A budget.threshold_reached event is informational and belongs in Slack. A budget.hard_cap_reached event is operational and belongs in PagerDuty or an on-call system. budget.daily_spike events warrant both.

Step 2: Configure a Slack destination

1. In Slack, create an incoming webhook for the channel where budget alerts should appear (e.g., #platform-alerts). Copy the webhook URL.

2. Navigate to Platform Settings → Alerts → Destinations → New Destination.

3. Configure:

   {
     "name": "slack-platform-alerts",
     "type": "webhook",
     "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
     "method": "POST",
     "headers": {
       "Content-Type": "application/json"
     },
     "template": "slack"
   }

4. The template: "slack" option formats the payload as a Slack block message. ManyLayers renders the team name, current spend, budget cap, and percentage consumed as a readable card.

5. Save and send a test event using Destinations → [Destination] → Send Test. Verify the message appears in your Slack channel.

Step 3: Configure a PagerDuty destination for hard cap events

Hard cap events mean users are being blocked. These warrant a more urgent notification path.

1. In PagerDuty, create a new service or use an existing one. Copy the Events API v2 routing key.

2. Navigate to Platform Settings → Alerts → Destinations → New Destination.

3. Configure:

   {
     "name": "pagerduty-budget-critical",
     "type": "pagerduty",
     "routing_key": "YOUR_PAGERDUTY_ROUTING_KEY",
     "severity": "error"
   }

4. Save and test.

Step 4: Create alert rules and bind them to destinations

Navigate to Platform Settings → Alerts → Rules → New Rule.

Create a rule for threshold alerts:

{
  "name": "team-budget-75-percent",
  "event_type": "budget.threshold_reached",
  "destinations": ["slack-platform-alerts"],
  "filters": {
    "threshold_gte": 0.75
  },
  "include_fields": ["team_name", "current_spend_usd", "budget_usd", "pct_consumed", "top_models_by_cost"]
}

Create a separate rule for hard cap events:

{
  "name": "team-budget-hard-cap",
  "event_type": "budget.hard_cap_reached",
  "destinations": ["slack-platform-alerts", "pagerduty-budget-critical"],
  "include_fields": ["team_name", "current_spend_usd", "budget_usd", "requests_blocked_count", "top_models_by_cost"]
}

Create a rule for anomalous daily spend:

{
  "name": "team-budget-daily-spike",
  "event_type": "budget.daily_spike",
  "destinations": ["slack-platform-alerts"],
  "include_fields": ["team_name", "today_spend_usd", "7d_avg_daily_spend_usd", "spike_multiplier"]
}

Step 5: Add a 90% alert for advance warning

A single alert at 75% gives teams one warning. Add a second alert at 90% so teams have a second opportunity to request a budget increase or reduce usage before hitting the cap.

{
  "name": "team-budget-90-percent",
  "event_type": "budget.threshold_reached",
  "destinations": ["slack-platform-alerts"],
  "filters": {
    "threshold_gte": 0.90
  },
  "include_fields": ["team_name", "current_spend_usd", "budget_usd", "pct_consumed", "projected_month_end_usd"]
}

The projected_month_end_usd field projects end-of-month spend based on the current daily spend rate. This gives team leads actionable context: “at your current pace, you’ll exhaust your budget on day 22.”

Step 6: Configure team-level alert recipients

By default, budget alerts go to platform admins. To also notify team leads directly:

1. Navigate to Platform Settings → Teams → [Team] → Notifications.
2. Add the team lead’s email address as an alert recipient for budget.threshold_reached and budget.hard_cap_reached.
3. Optionally, add a team-specific Slack channel here. Team members see their own budget alerts in their channel; platform admins see all teams in the central channel.

This two-tier notification approach means team leads can self-manage within their budget without requiring platform admin intervention for every alert.

Step 7: Build a custom handler for escalations (optional)

For organizations that want richer escalation logic — for example, automatically requesting a budget extension when a production team hits 90%, or paging a specific on-call engineer based on which team triggered the alert — a custom webhook handler gives you full control.

Configure a custom destination pointing at your handler:

{
  "name": "custom-escalation-handler",
  "type": "webhook",
  "url": "https://internal-tools.yourcompany.com/manylayers-budget-alert",
  "method": "POST",
  "headers": {
    "Authorization": "Bearer YOUR_INTERNAL_TOKEN"
  }
}

The raw webhook payload ManyLayers sends:

{
  "event_type": "budget.threshold_reached",
  "timestamp": "2026-06-03T14:22:00Z",
  "team_id": "team_abc123",
  "team_name": "Engineering",
  "current_spend_usd": 1125.50,
  "budget_usd": 1500.00,
  "pct_consumed": 0.75,
  "projected_month_end_usd": 1820.00,
  "top_models_by_cost": [
    {"model": "openai/gpt-4o", "spend_usd": 890.00},
    {"model": "anthropic/claude-sonnet-4-5", "spend_usd": 235.50}
  ]
}

Your handler can enrich this payload, look up the team’s on-call contact in your directory, post a threaded message in the team’s Slack channel, or create a ticket in your internal system.

Checklist

• Slack destination configured and test message verified
• PagerDuty destination configured for hard cap events
• Alert rules created for 75%, 90%, and hard cap events
• budget.daily_spike rule configured for anomaly detection
• Team-level notification recipients added
• Alert payload verified to include projected_month_end_usd on 90% alerts
• Custom handler configured if escalation logic is needed
• End-to-end test run: temporarily lower a test team’s budget to trigger an alert and verify the full notification chain

Common mistakes to avoid

Sending all alerts to a single destination. Mixing threshold alerts (informational) with hard cap alerts (operational) in the same PagerDuty service causes alert fatigue. Route by severity.

Not testing the alert chain end-to-end. Webhook configurations that look correct often have subtle URL or authentication errors. Use the test event feature and verify delivery before relying on the alerts in production.

Forgetting to alert team leads. Platform admin alerts catch problems too late for team leads to self-correct. Give teams visibility into their own budget consumption.