Agentmark allows you to test your inference directly in your application by setting up a webhook endpoint. This endpoint receives test requests from Agentmark and returns the inference results back to the platform. All inference results must be returned in Agentmark format.

​
Setting Up Your Webhook

​
1. Get Your Webhook Credentials

  1. Navigate to the Settings page in your Agentmark dashboard
  2. Under “Webhook Url”, you’ll find:
    • A webhook URL input field
    • A webhook secret input field. This must match the secret in your test endpoint.

Keep your webhook secret secure! This is used to verify that requests are coming from Agentmark.

​
2. Create Your Webhook Endpoint

Here’s an example of setting up a webhook endpoint using Next.js:

import { NextRequest, NextResponse } from "next/server";
import { AgentMarkSDK, trace } from "@agentmark/sdk";
import { verifySignature } from "@agentmark/shared-utils";
import { createAgentMarkClient, VercelAIModelRegistry } from "@agentmark/vercel-ai-v4-adapter";
import { openai } from "@ai-sdk/openai";
import { getFrontMatter } from "@agentmark/templatedx";
import { generateObject, generateText } from "ai";

// Import your configuration
import {
  AGENTMARK_API_KEY,
  AGENTMARK_APP_ID,
  AGENTMARK_WEBHOOK_SECRET,
} from "@config";

export const dynamic = "force-dynamic";

// Initialize AgentMark sdk
const sdk = new AgentMarkSDK({
  apiKey: AGENTMARK_API_KEY,
  appId: AGENTMARK_APP_ID,
});

sdk.initTracing({ disableBatch: true });

// Set up model registry
const modelRegistry = new VercelAIModelRegistry();
modelRegistry.registerModels(
  [
    "gpt-4o",
    "gpt-4o-mini",
    "gpt-4-turbo",
    "gpt-4",
    "o1-mini",
    "o1-preview",
    "gpt-3.5-turbo",
  ],
  (name: string) => openai(name)
);

const agentmark = createAgentMarkClient({
  modelRegistry,
});

​
3. Webhook Handler and Security

The webhook handler processes different types of events and includes security verification:

export async function POST(request: NextRequest) {
  const payload = await request.json();
  const headers = request.headers;
  const xAgentmarkSign = headers.get("x-agentmark-signature-256");

  try {
    // Verify signature
    if (!xAgentmarkSign) {
      throw new Error("Missing x-agentmark-signature-256 header");
    }

    if (!(await verifySignature(
      AGENTMARK_WEBHOOK_SECRET,
      xAgentmarkSign,
      JSON.stringify(payload)
    ))) {
      throw new Error("Invalid signature");
    }

    const event = payload.event;
    // ... handle different event types
  } catch (error) {
    console.log(error);
    return NextResponse.json(
      { message: "Internal server error" },
      { status: 500 }
    );
  }
}

​
Running Prompts

When you click the “Run” button in your Agentmark platform, it sends a prompt-run event to your webhook endpoint. This allows you to test your prompts directly in your application. For prompt runs, the handler supports both object and text configurations.

The response for prompt runs must always conform to the required JSON format structure to ensure proper processing by the Agentmark platform.

​
Response Format

The webhook response must be a JSON object with the following structure:

{
  "type": "text" | "object",
  "result": "string (for text) or object (for object)",
  "usage": {
    "promptTokens": 0,
    "completionTokens": 0,
    "totalTokens": 0
  },
  "finishReason": "string",
  "toolCalls": [
    {
      "toolName": "string",
      "args": {},
      "type": "string",
      "toolCallId": "string"
    }
  ],
  "toolResults": [
    {
      "toolCallId": "string",
      "result": "string or object",
      "type": "string",
      "toolName": "string",
      "args": {}
    }
  ]
}

Required fields:

  • type: Either “text” or “object”
  • result: String for text responses, object for structured responses

Optional fields:

  • usage: Token usage information
  • finishReason: Completion reason
  • toolCalls: Array of tool calls made during generation
  • toolResults: Array of tool execution results

​
Streaming Response Format

For streaming responses, set the Transfer-Encoding: chunked header and stream JSON chunks separated by newlines:

Stream Format:

{"result": "partial_content", "type": "text"}
{"result": "more_content", "type": "text"}
{"usage": {...}, "toolCalls": [...], "finishReason": "stop"}

Required Headers for Streaming:

  • Content-Type: application/json
  • Transfer-Encoding: chunked

Stream Structure:

  1. Content Chunks: Each chunk contains partial results with result and type fields
  2. Final Metadata: Last chunk contains usage, toolCalls, toolResults, and finishReason

​
Code

Agentmark webhooks support both regular JSON responses and streaming responses for prompt runs.

​
Non-Streaming Response

if (event.type === "prompt-run") {
  const data = event.data;
  const frontmatter = getFrontMatter(data.prompt) as any;
  const props = frontmatter.test_settings?.props || {};

  // Handle object configuration
  if (frontmatter.object_config) {
    const prompt = await agentmark.loadObjectPrompt(data.prompt);
    const vercelInput = await prompt.format({ props });
    const output = await generateObject(vercelInput);

    // Return response conforming to required JSON format
    return NextResponse.json({
      type: "object",
      result: output.object,
      usage: output.usage,
      finishReason: output.finishReason,
    });
  }

  // Handle text configuration
  if (frontmatter.text_config) {
    const prompt = await agentmark.loadTextPrompt(data.prompt);
    const vercelInput = await prompt.format({ props });
    const output = await generateText(vercelInput);

    // Return response conforming to required JSON format
    return NextResponse.json({
      type: "text",
      result: output.text,
      usage: output.usage,
      finishReason: output.finishReason,
      toolCalls: output.toolCalls,
      toolResults: output.toolResults,
    });
  }
}

​
Streaming Response

For real-time streaming of results, you can return a streaming response:

if (event.type === "prompt-run") {
  const data = event.data;
  const frontmatter = getFrontMatter(data.prompt) as any;
  const props = frontmatter.test_settings?.props || {};

  // Handle object configuration with streaming
  if (frontmatter.object_config) {
    const prompt = await agentmark.loadObjectPrompt(data.prompt);
    const vercelInput = await prompt.format({ props });
    const { usage, partialObjectStream } = streamObject(vercelInput);
    
    const stream = new ReadableStream({
      async start(controller) {
        const encoder = new TextEncoder();
        
        // Stream partial object chunks
        for await (const chunk of partialObjectStream) {
          const chunkData = encoder.encode(
            JSON.stringify({
              result: chunk,
              type: "object",
            }) + "\n"
          );
          controller.enqueue(chunkData);
        }
        
        // Send final metadata
        const usageData = await usage;
        const metadata = { usage: usageData };
        const metadataChunk = JSON.stringify(metadata);
        const metadataChunkData = encoder.encode(metadataChunk);
        controller.enqueue(metadataChunkData);
        controller.close();
      },
    });

    return new NextResponse(stream, {
      headers: {
        "Content-Type": "application/json",
        "Transfer-Encoding": "chunked",
      },
    });
  }

  // Handle text configuration with streaming
  if (frontmatter.text_config) {
    const prompt = await agentmark.loadTextPrompt(data.prompt);
    const vercelInput = await prompt.format({ props });
    const { textStream, toolCalls, toolResults, finishReason, usage } = 
      streamText(vercelInput);

    const stream = new ReadableStream({
      async start(controller) {
        const encoder = new TextEncoder();
        
        // Stream text chunks
        for await (const chunk of textStream) {
          const chunkData = encoder.encode(
            JSON.stringify({
              result: chunk,
              type: "text",
            }) + "\n"
          );
          controller.enqueue(chunkData);
        }
        
        // Send final metadata
        const toolCallsData = await toolCalls;
        const toolResultsData = await toolResults;
        const finishReasonData = await finishReason;
        const usageData = await usage;
        
        const metadata = {
          usage: usageData,
          toolCalls: toolCallsData,
          toolResults: toolResultsData,
          finishReason: finishReasonData,
        };
        
        const metadataChunk = JSON.stringify(metadata);
        const metadataChunkData = encoder.encode(metadataChunk);
        controller.enqueue(metadataChunkData);
        controller.close();
      },
    });

    return new NextResponse(stream, {
      headers: {
        "Content-Type": "application/json",
        "Transfer-Encoding": "chunked",
      },
    });
  }
}

​
Event Format

{
  "event": {
    "type": "prompt-run",
    "data": {
      "prompt": {
        ...A prompt in Agentmark format
      }
    }
  }
}

​
Running Datasets

For dataset runs, the handler processes each item in the dataset:

​
Code

if (event.type === "dataset-run") {
  const data = event.data;
  const promptAst = await loader.load(data.promptPath);
  const frontmatter = getFrontMatter(promptAst as any) as any;

  await Promise.all(
    data.dataset.map(async (item, index) => {
      return trace(
        { name: `ds-run-${data.runName}-${index}` },
        async () => {
          const telemetry = {
            isEnabled: true,
            metadata: {
              dataset_run_id: data.runId,        // required
              dataset_path: data.datasetPath,    // required
              dataset_run_name: data.runName,    // required
              dataset_item_name: index,          // required
              promptName: frontmatter.name,      // required
              props: item.input,                 // required
            },
          };

          // Handle text configuration
          if (frontmatter.text_config) {
            const prompt = await agentmark.loadTextPrompt(promptAst as any);
            const vercelInput = await prompt.format({
              props: item.input,
              telemetry,
            });
            await generateText(vercelInput);
          }

          // Handle object configuration
          if (frontmatter.object_config) {
            const prompt = await agentmark.loadObjectPrompt(promptAst as any);
            const vercelInput = await prompt.format({
              props: item.input,
              telemetry,
            });
            await generateObject(vercelInput);
          }
        }
      );
    })
  );

  return NextResponse.json({ message: "ok" });
}

​
Event Format

{
  "event": {
    "type": "dataset-run",
    "data": {
      "promptPath": "path/to/prompt",
      "dataset": [...],
      "datasetPath": "path/to/dataset",
      "runId": "run-id",
      "runName": "run-name"
    }
  }
}

​
Handling Alerts

When alerts are triggered or resolved in your Agentmark platform, it sends notifications to your webhook endpoint. This allows you to integrate with your monitoring systems and take appropriate actions.

Notification is only triggered for alerts for which the webhook is configured.

​
Code

if (event.type === "alert") {
  const alertData = event.data;
  console.log('Alert received:', alertData);
  
  // Process alert based on status and type
  if (alertData.alert.status === "triggered") {
    // Handle triggered alert
  } else if (alertData.alert.status === "resolved") {
    // Handle resolved alert
  }
  
  return NextResponse.json({ message: "Alert processed" });
}

​
Event Format

{
  "event": {
    "type": "alert",
    "data": {
      "alert": {
        "id": string,
        "currentValue": number,
        "threshold": number,
        "status": "triggered | resolved",
        "timeWindow": string,
        "type": "cost | latency | error_rate"
      },
      "message": string,
      "timestamp": string
    }
  }
}

​
Testing Your Webhook

  1. Set up your webhook endpoint
  2. Add your webhook URL and secret in Agentmark settings
  3. Create a prompt in Agentmark
  4. Click “Run” in the prompt to send a request to your webhook
  5. View the results in Agentmark’s dashboard

Make sure your webhook endpoint is publicly accessible and can handle POST requests.

​
Troubleshooting

  • Verify your webhook URL is correct and accessible
  • Check that your webhook secret matches the one in Agentmark
  • Ensure proper error handling in your endpoint
  • Make sure your returning a JSON response that matches whats provided by Agentmark