Core concepts
Traces
A trace represents a complete request or workflow in your application. Each trace is identified by a unique trace ID and contains one or more spans. Traces carry top-level attributes such as metadata, tags, user ID, and session ID.Spans
Spans are individual operations within a trace, forming a tree structure. AgentMark classifies every ingested span by type:GENERATION: a model call, including model, tokens, cost, and response. Detected fromgen_ai.*attributes or AI SDK span names such asai.generateTextandai.streamText.SPAN: any other operation: tool calls, retrieval steps, and customspan()/observe()wrappers.EVENT: point-in-time records. The spans API accepts this type alongside the other two.
Span kinds
Every span has a semantic kind that categorizes the operation:function (default), llm, tool, agent, retrieval, embedding, or guardrail. Span kinds determine how spans can be filtered and how analytics are grouped on dashboards.
See SpanKind values for the full reference, and set them using observe() or ctx.span().
Sessions
Sessions group related traces together by session ID. Track multi-turn conversations, agent workflows, and batch processing runs. Each session aggregates cost, tokens, and latency across its traces. Learn more about Sessions →Scores
Numeric evaluations attached to spans or traces. Set scores programmatically via the SDK usingsdk.score(), or manually through annotations in the Dashboard.
Metadata and tags
Metadata: custom key-value pairs attached to traces for context (environment, feature flags, customer tier). Automatically discovered as filter fields. Tags: string labels for categorization (environment, team, feature, release). Metadata → · Tags →What gets tracked
Model calls: full prompt execution lifecycle: token usage, costs, response times, model information, completion status. Tool calls: tool name, parameters, execution duration, success/failure status, return values. Streaming metrics: time to first token, tokens per second, total streaming duration. Sessions: group related traces by user interaction, multi-step workflow, or batch run. Alerts: monitor cost thresholds, latency spikes, error rates, and evaluation scores.Quick start
With tracing initialized (sdk.initTracing({ registerGlobally: true }), see Tracing setup), render your prompt and enable telemetry on the model call:
format() returns { messages, text_config } and your code calls the model, so telemetry is enabled per call via the AI SDK’s experimental_telemetry option. For full tracing setup including AgentMarkSDK, child spans, observe(), and span kinds, see Tracing setup.
How data flows
Your application sends telemetry via the AgentMark SDK, which exports OpenTelemetry spans to the AgentMark gateway. The gateway processes and stores the data, powering the traces, metrics, and analytics views.- Cloud
- Local
Spans are exported to the AgentMark Cloud gateway and stored in ClickHouse. View traces, dashboards, alerts, and analytics in the Dashboard.
Programmatic access
You can query traces, spans, sessions, scores, metrics, datasets, experiments, prompts, and runs programmatically through the REST API, or from an IDE agent via theagentmark-mcp MCP server (which exposes one MCP tool per gateway operation). Use either to build custom integrations, pull data into external tools, or automate monitoring workflows.
Most endpoints are available on both the local dev server and the AgentMark Cloud gateway. The local server returns 501 not_available_locally for features that require ClickHouse aggregations (/v1/metrics and score analytics). Use the capabilities endpoint to check which features a server supports.
list_traces, get_trace, get_capabilities, …) when you run the agentmark-mcp server alongside your IDE.
Next steps
Tracing setup
Instrument your app with the SDK
Traces and logs
View execution timelines in the Dashboard
Sessions
Group related traces together
Alerts
Get notified of critical issues
Dashboards
Analyze usage, performance, and scores
API reference
Query traces, scores, and metrics via REST API
Have questions?
Reach out any time:
- Email us at hello@agentmark.co for support
- Schedule an Enterprise Demo to learn about our business solutions