Agentmark allows you to test your inference directly in your application by setting up a webhook endpoint. This endpoint receives test requests from Agentmark and returns the inference results back to the platform. All inference results must be returned in Agentmark format.

​
Setting Up Your Webhook

​
1. Get Your Webhook Credentials

  1. Navigate to the Settings page in your Agentmark dashboard
  2. Under “Webhook Url”, you’ll find:
    • A webhook URL input field
    • A webhook secret input field. This must match the secret in your test endpoint.

Keep your webhook secret secure! This is used to verify that requests are coming from Agentmark.

​
2. Create Your Webhook Endpoint

Here’s an example of setting up a webhook endpoint using Next.js:

import { NextRequest, NextResponse } from "next/server";
import { AgentMarkSDK, trace } from "@agentmark/sdk";
import { verifySignature } from "@agentmark/shared-utils";
import { createAgentMarkClient, VercelAIModelRegistry } from "@agentmark/vercel-ai-v4-adapter";
import { createOpenAI } from "@ai-sdk/openai";
import { getFrontMatter } from "@agentmark/templatedx";
import { generateObject, generateText } from "ai";

// Import your configuration
import {
  AGENTMARK_API_KEY,
  AGENTMARK_APP_ID,
  AGENTMARK_WEBHOOK_SECRET,
} from "@config";

export const dynamic = "force-dynamic";

// Initialize AgentMark sdk
const sdk = new AgentMarkSDK({
  apiKey: AGENTMARK_API_KEY,
  appId: AGENTMARK_APP_ID,
});

sdk.initTracing({ disableBatch: true });

// Set up model registry
const modelRegistry = new VercelAIModelRegistry();
modelRegistry.registerModels(
  [
    "gpt-4o",
    "gpt-4o-mini",
    "gpt-4-turbo",
    "gpt-4",
    "o1-mini",
    "o1-preview",
    "gpt-3.5-turbo",
  ],
  (name: string, options) => {
    const provider = createOpenAI(options);
    return provider(name);
  }
);

const agentmark = createAgentMarkClient({
  modelRegistry,
});

​
3. Webhook Handler and Security

The webhook handler processes different types of events and includes security verification:

export async function POST(request: NextRequest) {
  const payload = await request.json();
  const headers = request.headers;
  const xAgentmarkSign = headers.get("x-agentmark-signature-256");

  try {
    // Verify signature
    if (!xAgentmarkSign) {
      throw new Error("Missing x-agentmark-signature-256 header");
    }

    if (!(await verifySignature(
      AGENTMARK_WEBHOOK_SECRET,
      xAgentmarkSign,
      JSON.stringify(payload)
    ))) {
      throw new Error("Invalid signature");
    }

    const event = payload.event;
    // ... handle different event types
  } catch (error) {
    console.log(error);
    return NextResponse.json(
      { message: "Internal server error" },
      { status: 500 }
    );
  }
}

​
Running Prompts

When you click the “Run” button in your Agentmark platform, it sends a prompt-run event to your webhook endpoint. This allows you to test your prompts directly in your application. For prompt runs, the handler supports both object and text configurations:

​
Code

if (event.type === "prompt-run") {
  const data = event.data;
  const frontmatter = getFrontMatter(data.prompt) as any;
  const props = frontmatter.test_settings?.props || {};

  // Handle object configuration
  if (frontmatter.object_config) {
    const prompt = await agentmark.loadObjectPrompt(data.prompt);
    const vercelInput = await prompt.format({ props });
    const output = await generateObject(vercelInput);

    return NextResponse.json({
      type: "object",
      result: output.object,
      usage: output.usage,
      finishReason: output.finishReason,
    });
  }

  // Handle text configuration
  if (frontmatter.text_config) {
    const prompt = await agentmark.loadTextPrompt(data.prompt);
    const vercelInput = await prompt.format({ props });
    const output = await generateText(vercelInput);

    return NextResponse.json({
      type: "text",
      result: output.text,
      usage: output.usage,
      finishReason: output.finishReason,
      toolCalls: output.toolCalls,
      toolResults: output.toolResults,
    });
  }
}

​
Event Format

{
  "event": {
    "type": "prompt-run",
    "data": {
      "prompt": {
        ...A prompt in Agentmark format
      }
    }
  }
}

​
Running Datasets

For dataset runs, the handler processes each item in the dataset:

​
Code

if (event.type === "dataset-run") {
  const data = event.data;
  const promptAst = await loader.load(data.promptPath);
  const frontmatter = getFrontMatter(promptAst as any) as any;

  await Promise.all(
    data.dataset.map(async (item, index) => {
      return trace(
        { name: `ds-run-${data.runName}-${index}` },
        async () => {
          const telemetry = {
            isEnabled: true,
            metadata: {
              dataset_run_id: data.runId,        // required
              dataset_path: data.datasetPath,    // required
              dataset_run_name: data.runName,    // required
              dataset_item_name: index,          // required
              promptName: frontmatter.name,      // required
              props: item.input,                 // required
            },
          };

          // Handle text configuration
          if (frontmatter.text_config) {
            const prompt = await agentmark.loadTextPrompt(promptAst as any);
            const vercelInput = await prompt.format({
              props: item.input,
              telemetry,
            });
            await generateText(vercelInput);
          }

          // Handle object configuration
          if (frontmatter.object_config) {
            const prompt = await agentmark.loadObjectPrompt(promptAst as any);
            const vercelInput = await prompt.format({
              props: item.input,
              telemetry,
            });
            await generateObject(vercelInput);
          }
        }
      );
    })
  );

  return NextResponse.json({ message: "ok" });
}

​
Event Format

{
  "event": {
    "type": "dataset-run",
    "data": {
      "promptPath": "path/to/prompt",
      "dataset": [...],
      "datasetPath": "path/to/dataset",
      "runId": "run-id",
      "runName": "run-name"
    }
  }
}

​
Handling Alerts

When alerts are triggered or resolved in your Agentmark platform, it sends notifications to your webhook endpoint. This allows you to integrate with your monitoring systems and take appropriate actions.

Notification is only triggered for alerts for which the webhook is configured.

​
Code

if (event.type === "alert") {
  const alertData = event.data;
  console.log('Alert received:', alertData);
  
  // Process alert based on status and type
  if (alertData.alert.status === "triggered") {
    // Handle triggered alert
  } else if (alertData.alert.status === "resolved") {
    // Handle resolved alert
  }
  
  return NextResponse.json({ message: "Alert processed" });
}

​
Event Format

{
  "event": {
    "type": "alert",
    "data": {
      "alert": {
        "id": string,
        "currentValue": number,
        "threshold": number,
        "status": "triggered | resolved",
        "timeWindow": string,
        "type": "cost | latency | error_rate"
      },
      "message": string,
      "timestamp": string
    }
  }
}

​
Testing Your Webhook

  1. Set up your webhook endpoint
  2. Add your webhook URL and secret in Agentmark settings
  3. Create a prompt in Agentmark
  4. Click “Run” in the prompt to send a request to your webhook
  5. View the results in Agentmark’s dashboard

Make sure your webhook endpoint is publicly accessible and can handle POST requests.

​
Troubleshooting

  • Verify your webhook URL is correct and accessible
  • Check that your webhook secret matches the one in Agentmark
  • Ensure proper error handling in your endpoint
  • Make sure your returning a JSON response that matches whats provided by Agentmark