Paid feature. Alerts are available on Growth, Team, and Enterprise plans in the AgentMark Dashboard. They are not available on the Hobby (free) tier.
Webhook endpoints are configured by developers. See Webhook documentation for setup instructions.
Overview
Alerts help you:- Monitor important metrics like cost, latency, and error rates
- Set thresholds for acceptable values
- Define time windows for measurement
- Receive notifications via Slack or webhooks
- Track alert history to analyze patterns
Available metrics
| Metric | Description |
|---|---|
| Cost | Total cost of LLM calls within the time window |
| Latency | Response times for AI requests |
| Error rate | Percentage of failed requests |
| Evaluation scores | Quality of AI responses from scoring pipelines |
Creating alerts
To create a new alert:- Navigate to your app’s Alerts tab in the Dashboard
- Click Create Alert
- Configure the alert:
- Name: descriptive name for the alert
- Metric: what to monitor. Thresholds are validated per metric: error rate 0-100 (percent), latency a positive integer in milliseconds, cost greater than 0 (dollars), evaluation score between 0 and 1
- Threshold: value that triggers the alert
- Time window: period over which the metric is measured, between 5 and 100 minutes
- Evaluation name (for score alerts): specific score to monitor
- Aggregation type (for score alerts): average or individual scores
- Threshold direction (for score alerts): alert above or below threshold
Notification options
Slack integration
Receive alerts directly in Slack:- Connect Slack and pick the notification channel in Settings → Integrations. The channel binding is per environment, so each environment can post to its own channel
- Enable the Slack Notifications toggle when creating an alert
Webhooks
For custom integrations:- Configure your webhook endpoint in Settings → Integrations (per environment)
- Enable the Custom Webhook toggle when creating an alert
- Receive alerts as HTTP POST requests with alert details
Alerts API
You can also create and manage alerts programmatically through/v1/alerts (CRUD, trigger history, and Slack channel listing). See the API reference.
Alert status
Alerts have two states:- Triggered: the monitored metric has crossed the threshold.
- Resolved: the metric has returned to the acceptable side of the threshold.
Alert history
View alert history to analyze patterns:- Navigate to the Alerts tab
- Click an alert to see when it triggered, the value that triggered it, when it resolved, and how frequently it fires
Use cases
- Cost management: get notified when daily spending exceeds budget
- Performance monitoring: alert when latency degrades beyond acceptable levels
- Quality assurance: track when evaluation scores drop below quality thresholds
- Error detection: catch error rate spikes before they impact users
Best practices
- Set realistic thresholds based on your app’s normal behavior, not arbitrary values.
- Choose appropriate windows to match your usage patterns and avoid alert fatigue.
- Configure multiple channels. Use Slack plus a webhook for critical alerts.
- Review regularly and adjust thresholds as your usage patterns evolve.
- Start conservative. Begin with higher thresholds and tighten them once you understand your baseline.
Next steps
Traces and logs
Monitor prompt execution
Sessions
Track user interactions
Dashboards
View overall performance
Evaluations
Set up evaluation alerts
Have questions?
Reach out any time:
- Email us at hello@agentmark.co for support
- Schedule an Enterprise Demo to learn about our business solutions