Alerts and Notifications

Running AI agents in production means you cannot watch dashboards all day. Agents make hundreds or thousands of LLM calls per hour. Costs accumulate. Policies trigger. Approval requests queue up. You need to know when something important happens without sitting in front of a screen waiting for it.

Govyn's alert system monitors your agents continuously and notifies you through the channels you already use. When an agent exceeds its budget, when an anomaly is detected, when a policy blocks a request, or when an action requires human approval, Govyn sends a notification immediately. You define the rules. Govyn enforces them and tells you about it.

Alerts are available on all paid plans. Configure them in the dashboard under Settings, or manage them programmatically through the API.

Alert types

Govyn supports four categories of alerts, each designed to surface a specific class of operational event. You can enable any combination of these and route each to different notification channels.

Cost threshold alerts

Cost threshold alerts fire when an agent's spending reaches a configurable percentage of its daily or monthly budget. The most common configuration is an 80% warning and a 100% critical alert. When the threshold is crossed, Govyn sends a notification with the agent name, the threshold percentage, the current spend, and the budget limit. This gives you time to intervene before an agent exhausts its budget — or to confirm that the spend is expected and adjust the limit.

Cost tracking is centralized across all providers and models. Whether an agent is calling OpenAI, Anthropic, or a self-hosted model through Govyn, the spend is aggregated into a single budget. The alert evaluates against the total, not individual providers.

Anomaly detection alerts

Anomaly detection alerts fire when an agent's behavior deviates significantly from its established baseline. Govyn tracks request patterns — call frequency, token usage, model distribution, error rates — and builds a statistical baseline over time. When an agent suddenly starts making ten times its normal request volume, or switches from using a mini model to a premium model exclusively, the anomaly detector flags it.

Anomaly alerts are particularly valuable for catching compromised agents, broken retry logic, and unexpected workflow changes before they result in significant cost overruns. They complement budget alerts by detecting problems that have not yet hit a dollar threshold but are clearly abnormal.

Policy violation alerts

Policy violation alerts fire when Govyn blocks a request because it violates a configured policy. This includes model restriction violations (agent tries to use a model not on the allowlist), rate limit violations (agent exceeds its per-minute or per-hour call limit), content pattern violations (request contains a blocked pattern), and schedule violations (agent makes a call outside its permitted hours).

Each violation alert includes the policy name, the agent that triggered it, the specific rule that was violated, and a summary of the blocked request. This gives you the context to decide whether the policy needs adjustment or whether the agent is misbehaving.

Approval pending notifications

Approval pending notifications fire when an agent's request is held in the approval queue waiting for human review. If you have configured approval requirements for certain actions — sending emails, posting to external services, exceeding token thresholds — Govyn queues the request and notifies the designated approvers immediately.

These notifications are time-sensitive. The agent is waiting for a response. Fast delivery to the right person matters. Route approval notifications to channels with the lowest latency: Slack for team visibility, PagerDuty for on-call rotation, or email for individual reviewers.

Notification channels

Govyn delivers alerts through four channels. Each alert rule can be routed to any channel, and you can create multiple rules for the same event type routed to different destinations. A cost threshold alert can go to both Slack (for team awareness) and email (for the budget owner) simultaneously.

In-dashboard notifications

The Govyn dashboard includes a built-in notification center. Alerts appear in real time without page refresh. Each notification shows the alert type, agent name, message, and timestamp. Notifications are tracked as read or unread, and you can filter by type and date range. Dashboard notifications are always enabled — they require no configuration.

Email

Email alerts deliver directly to the inbox of the team member or distribution list you specify. Use email for critical alerts that require immediate attention — budget exceeded, approval waiting, anomaly detected. Email alerts include the full alert context: type, agent, message, current values, and a direct link to the relevant dashboard page.

Webhooks

Webhooks are the most flexible channel. Govyn sends an HTTP POST request with a JSON payload to any public URL you specify. Use webhooks to integrate with any system that accepts HTTP callbacks: incident management tools, custom Slack bots, internal monitoring dashboards, ticketing systems, or your own alert aggregation service.

Webhook URLs are validated for security. Govyn blocks requests to private IP addresses, localhost, and cloud metadata endpoints to prevent SSRF attacks. Only public, routable URLs are accepted. Each webhook delivery is logged in the notification history with the HTTP response status, so you can verify delivery and debug integration issues.

Slack (via webhook)

Slack integration uses Slack's incoming webhook feature. Create an incoming webhook in your Slack workspace, paste the URL into Govyn, and alerts appear as formatted messages in your chosen Slack channel. The message includes the alert type, agent name, a human-readable description, and the timestamp. You can route different alert types to different channels — cost alerts to #billing, policy violations to #security, approvals to #approvals.

Setting up alert rules

Alert rules are configured in the Govyn dashboard under Settings. Each rule defines what triggers an alert and where the notification goes.

1. Navigate to Settings — open the Govyn dashboard and go to the Settings page. Select the Alert Rules section.
2. Click Add Rule — this opens the alert rule configuration form.
3. Name the rule — give it a descriptive name like "Production agent budget warning" or "Security policy violations to Slack".
4. Select the condition — choose the alert type and configure the trigger. For cost thresholds, set the percentage (e.g., 80%). For anomaly detection, the system uses automatic baseline deviation. For policy violations, select specific policies or all policies.
5. Choose the channel — select the notification channel and provide the destination. For webhooks and Slack, enter the URL. For email, enter the recipient address.
6. Test the rule — click "Test" to send a sample alert to the configured destination. Verify that the notification arrives and is formatted correctly.
7. Save — the rule is active immediately. Govyn begins evaluating the condition on every relevant event.

You can create as many rules as you need. Rules can overlap — the same event can trigger multiple rules routed to different channels.

Webhook payload format

Every webhook delivery sends a JSON object with a consistent structure. The type field identifies the alert category. The agent field identifies which agent triggered the alert. The message field contains a human-readable description. Additional fields vary by alert type.

Here is an example payload for a cost threshold alert:

{
  "type": "cost_threshold",
  "agent": "support-bot",
  "message": "Agent exceeded 80% of daily budget",
  "threshold": 80,
  "current": 85.2,
  "budget": 50.00,
  "period": "daily",
  "timestamp": "2025-01-15T14:30:00Z"
}

An anomaly detection alert includes the deviation metric and baseline values:

{
  "type": "anomaly_detected",
  "agent": "research-agent",
  "message": "Request volume 8.3x above baseline",
  "metric": "request_volume",
  "baseline": 120,
  "current": 996,
  "deviation_factor": 8.3,
  "timestamp": "2025-01-15T15:45:00Z"
}

A policy violation alert identifies the specific policy and rule that was triggered:

{
  "type": "policy_violation",
  "agent": "data-processor",
  "message": "Request blocked by model restriction policy",
  "policy": "production-model-allowlist",
  "rule": "model_restrict",
  "blocked_model": "gpt-4o",
  "allowed_models": ["gpt-4o-mini", "gpt-3.5-turbo"],
  "timestamp": "2025-01-15T16:00:00Z"
}

An approval pending notification includes the approval token and the request summary:

{
  "type": "approval_pending",
  "agent": "email-agent",
  "message": "Action requires human approval",
  "approval_token": "gva_abc123...",
  "action_summary": "Send email to customer@example.com",
  "dashboard_url": "https://app.govynai.com/approvals/gva_abc123",
  "timestamp": "2025-01-15T16:30:00Z"
}

SSRF protection

Server-side request forgery (SSRF) is a class of vulnerability where an attacker tricks a server into making HTTP requests to internal resources. Because Govyn sends HTTP requests to user-specified webhook URLs, SSRF protection is critical.

Govyn validates every webhook URL before saving it and before every delivery. The following address ranges are blocked:

• RFC 1918 private addresses — 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
• Localhost — 127.0.0.0/8, ::1
• Link-local addresses — 169.254.0.0/16, fe80::/10
• Cloud metadata endpoints — 169.254.169.254 (AWS, GCP, Azure instance metadata)
• IPv6 mapped IPv4 — addresses like ::ffff:127.0.0.1 that could bypass IPv4 checks

Only public, routable URLs are accepted. DNS resolution is performed at validation time to catch hostnames that resolve to private addresses. This prevents attackers from using DNS rebinding to bypass URL-level checks. The validation runs on both rule creation and every webhook delivery to guard against DNS changes after rule creation.

Notification history

Every alert Govyn sends is recorded in the notification history. This provides a complete audit trail of what was detected, when it was detected, and how the notification was delivered.

The notification history is available in the Settings section of the dashboard. Each entry includes:

• Alert type — cost threshold, anomaly, policy violation, or approval pending
• Channel — dashboard, email, webhook, or Slack
• Destination — the webhook URL, email address, or Slack channel
• Agent — which agent triggered the alert
• Message — the human-readable alert description
• Delivery status — success, failure, or pending
• Timestamp — when the alert was generated and when it was delivered

The history is filterable by alert type, channel, delivery status, and date range. Use it to verify that your alert rules are working, to debug delivery failures, and to review the alert history during incident investigations. Notification history retention follows your plan's log retention limits.

Alert fatigue prevention

Alert fatigue is what happens when a monitoring system sends so many notifications that the team starts ignoring them. A single budget spike can trigger dozens of identical alerts. A flapping policy can generate hundreds of violations per hour. Without controls, the alert channel becomes noise and real issues get missed.

Govyn provides three mechanisms to prevent alert fatigue:

Cooldown periods

Each alert rule can have a cooldown period — a minimum time between consecutive notifications for the same condition. If you set a 15-minute cooldown on a cost threshold rule, Govyn sends the first alert when the threshold is crossed and suppresses duplicate alerts for the next 15 minutes. The cooldown resets when the condition clears (the agent drops below the threshold) or when the cooldown period expires.

Alert grouping

When multiple agents trigger the same alert type within a short window, Govyn can group them into a single notification. Instead of receiving ten separate "budget exceeded" alerts for ten agents, you receive one notification listing all affected agents. This is particularly useful during system-wide events that affect many agents simultaneously.

Severity-based routing

Route alerts by severity to match the urgency of the response required. Critical alerts (budget fully exhausted, approval waiting for more than 30 minutes) go to PagerDuty or the on-call engineer's phone. Warning alerts (approaching budget threshold, anomaly detected) go to Slack for team awareness. Informational alerts (policy violation blocked a request, agent activity summary) go to email or the dashboard only. This ensures that high-urgency channels are reserved for high-urgency events.

Integration examples

Govyn's webhook system integrates with any service that accepts HTTP POST requests. Here are setup instructions for common integrations.

Slack incoming webhook

1. In your Slack workspace, go to Apps and search for "Incoming Webhooks".
2. Click Add to Slack and select the channel where alerts should appear.
3. Copy the webhook URL (it starts with https://hooks.slack.com/services/).
4. In the Govyn dashboard, create an alert rule and paste the Slack webhook URL as the destination.
5. Test the rule. A formatted message should appear in your Slack channel.

PagerDuty Events API

1. In PagerDuty, create a new service or select an existing one.
2. Add an integration of type "Events API v2" and copy the integration URL.
3. In the Govyn dashboard, create an alert rule for critical conditions (budget exceeded, approval pending).
4. Use the PagerDuty Events API URL as the webhook destination.
5. PagerDuty will create an incident for each alert and route it through your on-call schedule.

Custom webhook handler in Node.js

You can build a custom handler to process Govyn alerts and take automated action. Here is a minimal Express server that receives alerts and logs them:

import express from 'express';

const app = express();
app.use(express.json());

app.post('/govyn-alerts', (req, res) => {
  const alert = req.body;

  switch (alert.type) {
    case 'cost_threshold':
      // Notify finance team or auto-scale budget
      handleCostAlert(alert);
      break;
    case 'anomaly_detected':
      // Investigate or pause the agent
      handleAnomalyAlert(alert);
      break;
    case 'policy_violation':
      // Log for security review
      handleViolationAlert(alert);
      break;
    case 'approval_pending':
      // Route to the right reviewer
      handleApprovalAlert(alert);
      break;
  }

  res.status(200).json({ received: true });
});

app.listen(3000);

Your webhook handler should return a 2xx status code to confirm receipt. If Govyn receives a non-2xx response or a timeout, the delivery is marked as failed in the notification history.

Frequently asked questions