Traces and Logs

AgentMark uses OpenTelemetry to provide distributed tracing for your prompt executions. This gives you complete visibility into how your prompts perform in production.

Developers set up tracing in your application. See Development documentation for setup instructions.

Traces panel showing prompt execution timeline with spans, token usage, and response times

Understanding Traces

A trace represents the complete execution of a prompt, including all its steps, tool calls, and metadata. Each trace contains: Execution Timeline - See exactly when each step occurred and how long it took. Token Usage - Track input tokens, output tokens, and total tokens consumed. Costs - Monitor spending on a per-request basis. Tool Calls - View all tool executions, their parameters, and results. Custom Metadata - Add context like user IDs, session IDs, and custom attributes. Error Information - Detailed error messages and stack traces when issues occur.

Collected Spans

AgentMark records the following OpenTelemetry spans:

Span Type	Description	Attributes
`ai.inference`	Full length of the inference call	`operation.name`, `ai.operationId`, `ai.prompt`, `ai.response.text`, `ai.response.toolCalls`, `ai.response.finishReason`
`ai.toolCall`	Individual tool executions	`operation.name`, `ai.operationId`, `ai.toolCall.name`, `ai.toolCall.args`, `ai.toolCall.result`
`ai.stream`	Streaming response data	`ai.response.msToFirstChunk`, `ai.response.msToFinish`, `ai.response.avgCompletionTokensPerSecond`

Span kinds

Each span carries a semantic kind that categorizes the type of operation it represents. Span kinds affect how spans appear in the graph view, how they can be filtered, and how analytics are grouped on the dashboard.

Kind	Description
function	Generic computation step (default)
llm	A call to a language model
tool	An external tool or API call
agent	An orchestration loop that decides what to do next
retrieval	A vector database query or document search
embedding	A call to an embedding model
guardrail	A content safety or validation check

Span kinds are set in code using the observe() function or the kind option on ctx.span(). See Traces and Logs (Development) for implementation details.

LLM Span Attributes

Each LLM span contains attributes that vary slightly depending on the adapter you use. The table below shows common attributes across integrations:

AI SDK (Vercel)
Claude Agent SDK

Attribute	Description
`ai.model.id`	Model identifier
`ai.model.provider`	Model provider name
`ai.usage.promptTokens`	Number of prompt tokens
`ai.usage.completionTokens`	Number of completion tokens
`ai.settings.maxRetries`	Maximum retry attempts
`ai.telemetry.functionId`	Function identifier
`ai.telemetry.metadata.*`	Custom metadata
`ai.response.text`	Response text
`ai.response.toolCalls`	Tool calls array
`ai.response.finishReason`	Finish reason

Attribute	Description
`gen_ai.request.model`	Requested model name
`gen_ai.system`	AI system identifier (e.g., `anthropic`)
`gen_ai.usage.input_tokens`	Number of input tokens
`gen_ai.usage.output_tokens`	Number of output tokens
`gen_ai.response.output`	Agent response output
`gen_ai.response.finish_reasons`	Completion finish reasons
`gen_ai.tool.name`	Tool name
`gen_ai.operation.name`	Operation type (`chat`, `execute_tool`, `invoke_agent`)

All adapters also support custom metadata via agentmark.metadata.* attributes.

Grouping Traces

Organize related traces together using custom grouping. This is useful for understanding complex workflows that span multiple prompt executions.

You can create hierarchical trace groups to represent:

Multi-step agent workflows
Nested component execution
Parallel processing pipelines

Graph View

For complex AI agent workflows, AgentMark provides an interactive graph visualization that shows the relationships between different components, execution flow, and dependencies. Learn more about Graph View →

Viewing Traces

Access traces in the AgentMark dashboard under the “Traces” tab. Each trace shows:

Complete prompt execution timeline
Tool calls and their durations
Token usage and costs
Custom metadata and attributes
Error information (if any)
Graph visualization (when graph metadata is present)
Manual annotations for quality assessment

Filtering and search

AgentMark provides powerful filtering across all trace dimensions — model, status, latency, cost, tokens, metadata, scores, and more. Filters can be combined, saved as views, and shared via URL. Learn more about filtering and search

Integration

AgentMark works with any application that uses OpenTelemetry. For detailed setup instructions, see the Development Observability documentation.

MCP Trace Server

For debugging traces directly from your IDE, AgentMark provides an MCP server that exposes list_traces and get_trace tools. This lets you query and inspect traces without leaving your development environment.

Traces and spans API

You can query traces and spans programmatically using the REST API or the CLI. The local dev server and cloud gateway expose the same endpoints, so you can develop against local data and switch to cloud without changing your integration.

CLI
REST API (cloud)
REST API (local)

# List traces from the local dev server
agentmark api traces list --limit 20

# Get a specific trace with its spans
agentmark api traces get <traceId>

# Query spans across all traces
agentmark api spans list --limit 50

# Target the cloud gateway instead
agentmark api traces list --remote --limit 20

# List traces with filters
curl "https://api.agentmark.co/v1/traces?status=2&limit=20" \
  -H "Authorization: Bearer am_live_abc123" \
  -H "X-Agentmark-App-Id: app_abc123"

# Get a single trace with all its spans
curl "https://api.agentmark.co/v1/traces/abc123-trace-id" \
  -H "Authorization: Bearer am_live_abc123" \
  -H "X-Agentmark-App-Id: app_abc123"

# Query spans across all traces
curl "https://api.agentmark.co/v1/spans?type=GENERATION&model=gpt-4o&limit=50" \
  -H "Authorization: Bearer am_live_abc123" \
  -H "X-Agentmark-App-Id: app_abc123"

# Same endpoints, no auth required locally
curl "http://localhost:9418/v1/traces?status=2&limit=20"

curl "http://localhost:9418/v1/traces/abc123-trace-id"

curl "http://localhost:9418/v1/spans?type=GENERATION&limit=50"

See the API reference for all available endpoints, filters, and response schemas. You can also export traces in bulk for fine-tuning and analysis, or create scores for spans and traces programmatically.

Best practices

Use Meaningful IDs - Choose descriptive function IDs for easy filtering and debugging. Add Context - Include relevant metadata like user IDs, session IDs, and business context. Monitor Regularly - Check traces frequently to catch issues early. Set Up Alerts - Configure alerts for cost, latency, or error thresholds. Analyze Patterns - Use the dashboard’s filtering to identify trends and patterns.

Next Steps

Sessions

Group related traces together

Alerts

Get notified of critical issues

Annotations

Manually label and score traces

Development Setup

Integrate observability in your app

Have Questions?

We’re here to help! Choose the best way to reach us:

Email us at hello@agentmark.co for support
Schedule an Enterprise Demo to learn about our business solutions

Introduction

Getting Started

Build

Evaluate

Observe

Configure

Deploy

Integrations

Understanding Traces

Collected Spans

Span kinds

LLM Span Attributes

Grouping Traces

Graph View

Viewing Traces

Filtering and search

Integration

MCP Trace Server

Traces and spans API

Best practices

Next Steps

Sessions

Alerts

Annotations

Development Setup

Have Questions?

Introduction

Getting Started

Build

Evaluate

Observe

Configure

Deploy

Integrations

​Understanding Traces

​Collected Spans

​Span kinds

​LLM Span Attributes

​Grouping Traces

​Graph View

​Viewing Traces

​Filtering and search

​Integration

​MCP Trace Server

​Traces and spans API

​Best practices

​Next Steps

Sessions

Alerts

Annotations

Development Setup

​Have Questions?

Understanding Traces

Collected Spans

Span kinds

LLM Span Attributes

Grouping Traces

Graph View

Viewing Traces

Filtering and search

Integration

MCP Trace Server

Traces and spans API

Best practices

Next Steps

Have Questions?