Alerts

AgentMark’s alert system monitors critical metrics and notifies you when thresholds are exceeded. Stay informed about your application’s performance, costs, and potential issues.

​

Overview

Alerts help you:

• Monitor important metrics like cost, latency, and error rates
• Set thresholds for acceptable values
• Define time windows for measurement
• Receive notifications via Slack or webhooks
• Track alert history to analyze patterns

​

Available Metrics

​

Creating Alerts

To create a new alert:

1. Navigate to your application’s dashboard
2. Click on the Alerts tab
3. Click Create Alert
4. Configure the alert:
   • Name - Descriptive name for the alert
   • Metric - Choose what to monitor
   • Threshold - Value that triggers the alert
   • Time Window - Period over which the metric is measured
   • Evaluation Name (for eval alerts) - Specific evaluation to monitor
   • Aggregation Type (for eval alerts) - Average or individual scoring
   • Threshold Direction (for eval alerts) - Alert above or below threshold

​

Notification Options

​

Slack Integration

Receive alerts directly in Slack:

1. Enable Send to Slack when creating an alert
2. Click Connect Slack if not already connected
3. Select the channel for notifications

​

Webhooks

For custom integrations:

1. Enable Use Webhook when creating an alert
2. Configure your webhook endpoint in the Developers section
3. Receive alerts as HTTP POST requests with alert details

For webhook payload format and implementation details, see the Webhook documentation.

​

Alert Status

Alerts have two states: Triggered - The monitored metric exceeds the defined threshold. Resolved - The metric returns below the threshold.

​

Alert History

View alert history to analyze patterns:

1. Navigate to the Alerts tab
2. Click on an alert to see:
   • When the alert was triggered
   • The actual value that triggered it
   • When the alert was resolved
   • Frequency and duration of alerts

​

Use Cases

Cost Management - Get notified when daily spending exceeds budget. Performance Monitoring - Alert when latency degrades beyond acceptable levels. Quality Assurance - Track when evaluation scores drop below quality thresholds. Error Detection - Catch error rate spikes before they impact users.

​

Best Practices

Set Realistic Thresholds - Base thresholds on your application’s normal behavior, not arbitrary values. Choose Appropriate Windows - Use time windows that match your usage patterns to avoid alert fatigue. Configure Multiple Channels - Set up both Slack and webhooks for critical alerts. Review Regularly - Adjust thresholds as your usage patterns evolve. Start Conservative - Begin with higher thresholds and tighten them as you understand your baseline.

​

Next Steps