Skip to main content
AgentMark’s observability is built on OpenTelemetry, providing a structured data model for monitoring and debugging LLM applications. This page defines the core concepts you will encounter throughout the platform.

​
Traces

A trace represents a complete request or workflow in your application. Each trace is identified by a unique trace ID and contains one or more spans. Traces carry top-level attributes such as metadata, tags, user ID, and session ID. Use traces to see the full picture of a single prompt execution, agent workflow, or multi-step pipeline.

​
Spans

Spans are individual operations within a trace. They form a tree structure — parent spans can contain child spans, representing nested operations. AgentMark records three span types:
  • ai.inference — the full lifecycle of an LLM call, including model, tokens, cost, and response
  • ai.toolCall — a single tool execution, including name, arguments, and result
  • ai.stream — streaming response metrics such as time to first token and tokens per second
For detailed span attributes, see Traces and Logs.

​
Sessions

Sessions group related traces together by session ID. They are useful for tracking multi-turn conversations, agent workflows, and batch processing runs. Each session aggregates cost, tokens, and latency across all of its traces. For details on working with sessions, see Sessions.

​
Scores

Scores are numeric evaluations attached to spans or traces. You can set scores programmatically via the SDK using sdk.score(), or manually through annotations in the dashboard. Scores are used for quality tracking, filtering, and alerting. For more on annotations, see Annotations.

​
Metadata

Metadata consists of custom key-value pairs attached to traces via the SDK. Use metadata to add context such as environment, feature flags, customer tier, or A/B test variants. Metadata keys are automatically discovered and available as filter fields. For details on setting and using metadata, see Metadata.

​
Tags

Tags are string labels attached to traces for categorization. Use tags to organize traces by environment, team, feature, experiment, or release. For naming conventions and best practices, see Tags.

​
How data flows

Your application sends telemetry via the AgentMark SDK, which exports OpenTelemetry spans to the AgentMark gateway. The gateway processes and stores the data in ClickHouse, where it powers the dashboard’s traces, metrics, and analytics views.

​
Next steps

Quickstart

See your first trace in under 5 minutes

Traces and Logs

Explore trace details and span attributes

Sessions

Group related traces together

Metrics

Track usage, costs, and performance

​
Have Questions?

We’re here to help! Choose the best way to reach us: