Overview

AgentMark webhooks allow you to integrate your prompts and datasets directly into your application. When you run a prompt or dataset in the AgentMark platform, it sends the request to your webhook endpoint, processes it, and returns the results.

​

Key Features

• Unified Integration: Single endpoint for all generation types (text, object, image, speech)
• Real-time Processing: Support for both streaming and non-streaming responses
• Security: Built-in signature verification for secure communication
• Flexibility: Handle different event types (prompt runs, dataset runs, alerts)
• Telemetry: Built-in support for tracing and monitoring
• Custom Evaluators: Register custom evaluation functions for dataset runs
• Tool Support: Register and use custom tools in your prompts

​

Generation Types

AgentMark webhooks support four types of generation:

1. Text Generation: Generate text responses with support for tool calls
2. Object Generation: Generate structured JSON responses
3. Image Generation: Generate images from text descriptions
4. Speech Generation: Convert text to speech

​

Event Types

The webhook handles three main types of events:

1. Prompt Run Events: When you run a single prompt
   • Supports all generation types
   • Can return streaming or non-streaming responses
   • Includes metadata like usage and tool calls
2. Dataset Run Events: When you run a dataset
   • Processes multiple items in parallel
   • Supports text and object generation
   • Includes telemetry for monitoring
3. Alert Events: When alerts are triggered or resolved
   • Monitor cost, latency, and error rate thresholds
   • Receive real-time notifications
   • Track alert status changes

​

Security

All webhook requests are secured with:

• Signature verification using HMAC-SHA256
• Required headers for authentication
• Proper error handling and validation

​

Advanced Features

​

Custom Evaluators

Use the EvalRegistry to register custom evaluation functions that automatically score dataset run outputs. This allows you to:

• Define custom metrics (e.g., correctness, hallucination detection, relevance)
• Automatically evaluate model outputs during dataset runs
• Track evaluation scores alongside generation metrics

​

Tool Registry

Register custom tools using VercelAIToolRegistry to extend your prompts with external capabilities:

• API integrations (weather, flights, databases)
• Custom business logic
• Data transformations
• Third-party service calls

​

Model Registry

Register AI models to use in your prompts. Start with text generation models like GPT-4o, GPT-4 Turbo, and GPT-3.5 Turbo. You can also register image and speech models as needed.

​

Getting Started

1. Set up your webhook endpoint
2. Use Webhook Helpers for easy integration
3. Handle prompt run events
4. Process dataset runs
5. Handle alert events