Skip to main content
Instrument with the SDK to capture traces automatically. Explore them in your terminal (local) or in the Dashboard with filtering, search, graph view, dashboards, and alerts. Built on OpenTelemetry, AgentMark automatically collects telemetry data from your prompts so you can monitor and debug them.

Core concepts

Traces

A trace represents a complete request or workflow in your application. Each trace is identified by a unique trace ID and contains one or more spans. Traces carry top-level attributes such as metadata, tags, user ID, and session ID.

Spans

Spans are individual operations within a trace, forming a tree structure. AgentMark classifies every ingested span by type:
  • GENERATION: a model call, including model, tokens, cost, and response. Detected from gen_ai.* attributes or AI SDK span names such as ai.generateText and ai.streamText.
  • SPAN: any other operation: tool calls, retrieval steps, and custom span() / observe() wrappers.
  • EVENT: point-in-time records. The spans API accepts this type alongside the other two.

Span kinds

Every span has a semantic kind that categorizes the operation: function (default), llm, tool, agent, retrieval, embedding, or guardrail. Span kinds determine how spans can be filtered and how analytics are grouped on dashboards. See SpanKind values for the full reference, and set them using observe() or ctx.span().

Sessions

Sessions group related traces together by session ID. Track multi-turn conversations, agent workflows, and batch processing runs. Each session aggregates cost, tokens, and latency across its traces. Learn more about Sessions →

Scores

Numeric evaluations attached to spans or traces. Set scores programmatically via the SDK using sdk.score(), or manually through annotations in the Dashboard.

Metadata and tags

Metadata: custom key-value pairs attached to traces for context (environment, feature flags, customer tier). Automatically discovered as filter fields. Tags: string labels for categorization (environment, team, feature, release). Metadata → · Tags →

What gets tracked

Model calls: full prompt execution lifecycle: token usage, costs, response times, model information, completion status. Tool calls: tool name, parameters, execution duration, success/failure status, return values. Streaming metrics: time to first token, tokens per second, total streaming duration. Sessions: group related traces by user interaction, multi-step workflow, or batch run. Alerts: monitor cost thresholds, latency spikes, error rates, and evaluation scores.

Quick start

With tracing initialized (sdk.initTracing({ registerGlobally: true }), see Tracing setup), render your prompt and enable telemetry on the model call:
import { client } from './agentmark.client';
import { openai } from '@ai-sdk/openai';
import { generateText } from 'ai';

const prompt = await client.loadTextPrompt('greeting.prompt.mdx');
const { messages, text_config } = await prompt.format({
  props: { name: 'Alice' },
});

const result = await generateText({
  model: openai(text_config.model_name.replace(/^openai\//, '')),
  messages,
  experimental_telemetry: {
    isEnabled: true,
    functionId: 'greeting-handler',
    // Session/user keys in metadata must be snake_case to group traces (see Sessions)
    metadata: {
      user_id: 'user-123',
      session_id: 'session-abc',
      session_name: 'Customer Support Chat'
    }
  }
});
The render is neutral: format() returns { messages, text_config } and your code calls the model, so telemetry is enabled per call via the AI SDK’s experimental_telemetry option. For full tracing setup including AgentMarkSDK, child spans, observe(), and span kinds, see Tracing setup.

How data flows

Your application sends telemetry via the AgentMark SDK, which exports OpenTelemetry spans to the AgentMark gateway. The gateway processes and stores the data, powering the traces, metrics, and analytics views.
Spans are exported to the AgentMark Cloud gateway and stored in ClickHouse. View traces, dashboards, alerts, and analytics in the Dashboard.

Programmatic access

You can query traces, spans, sessions, scores, metrics, datasets, experiments, prompts, and runs programmatically through the REST API, or from an IDE agent via the agentmark-mcp MCP server (which exposes one MCP tool per gateway operation). Use either to build custom integrations, pull data into external tools, or automate monitoring workflows. Most endpoints are available on both the local dev server and the AgentMark Cloud gateway. The local server returns 501 not_available_locally for features that require ClickHouse aggregations (/v1/metrics and score analytics). Use the capabilities endpoint to check which features a server supports.
# Query traces from the local dev server
curl -fsS "http://localhost:9418/v1/traces?limit=10"

# Get a specific trace with all its spans
curl -fsS "http://localhost:9418/v1/traces/<traceId>"

# Same calls against Cloud — set AGENTMARK_API_KEY + AGENTMARK_APP_ID
curl -fsS "https://api.agentmark.co/v1/traces?limit=10" \
  -H "Authorization: Bearer $AGENTMARK_API_KEY" \
  -H "X-Agentmark-App-Id: $AGENTMARK_APP_ID"

# Check which features are available on a given server
curl -fsS "http://localhost:9418/v1/capabilities"
The same operations are available as MCP tools (list_traces, get_trace, get_capabilities, …) when you run the agentmark-mcp server alongside your IDE.

Next steps

Tracing setup

Instrument your app with the SDK

Traces and logs

View execution timelines in the Dashboard

Sessions

Group related traces together

Alerts

Get notified of critical issues

Dashboards

Analyze usage, performance, and scores

API reference

Query traces, scores, and metrics via REST API

Have questions?

Reach out any time: