Concepts

AgentMark’s observability is built on OpenTelemetry, providing a structured data model for monitoring and debugging LLM applications. This page defines the core concepts you will encounter throughout the platform.

​

Traces

A trace represents a complete request or workflow in your application. Each trace is identified by a unique trace ID and contains one or more spans. Traces carry top-level attributes such as metadata, tags, user ID, and session ID. Use traces to see the full picture of a single prompt execution, agent workflow, or multi-step pipeline.

​

Spans

Spans are individual operations within a trace. They form a tree structure — parent spans can contain child spans, representing nested operations. AgentMark records three span types:

• ai.inference — the full lifecycle of an LLM call, including model, tokens, cost, and response
• ai.toolCall — a single tool execution, including name, arguments, and result
• ai.stream — streaming response metrics such as time to first token and tokens per second

For detailed span attributes, see Traces and Logs.

​

Span kinds

Every span has a semantic kind that categorizes what type of operation it represents. Span kinds determine how spans appear in the graph view, how they can be filtered, and how analytics are grouped on the dashboard. You can set span kinds in your code using the observe() function or the kind option on ctx.span(). See Traces and Logs for usage examples.

​

Sessions

Sessions group related traces together by session ID. They are useful for tracking multi-turn conversations, agent workflows, and batch processing runs. Each session aggregates cost, tokens, and latency across all of its traces. For details on working with sessions, see Sessions.

​

Scores

Scores are numeric evaluations attached to spans or traces. You can set scores programmatically via the SDK using sdk.score(), or manually through annotations in the dashboard. Scores are used for quality tracking, filtering, and alerting. For more on annotations, see Annotations.

​

Metadata

Metadata consists of custom key-value pairs attached to traces via the SDK. Use metadata to add context such as environment, feature flags, customer tier, or A/B test variants. Metadata keys are automatically discovered and available as filter fields. For details on setting and using metadata, see Metadata.

​

Tags

Tags are string labels attached to traces for categorization. Use tags to organize traces by environment, team, feature, experiment, or release. For naming conventions and best practices, see Tags.

​

How data flows

Your application sends telemetry via the AgentMark SDK, which exports OpenTelemetry spans to the AgentMark gateway. The gateway processes and stores the data in ClickHouse, where it powers the dashboard’s traces, metrics, and analytics views.

​

Next steps