> ## Documentation Index > Fetch the complete documentation index at: https://docs.agentmark.co/llms.txt > Use this file to discover all available pages before exploring further. # Observe > Monitor, debug, and optimize your LLM applications with tracing, dashboards, and alerts Instrument with the SDK to capture traces automatically. Explore them in your terminal (local) or in the Dashboard with filtering, search, graph view, dashboards, and alerts. Built on [OpenTelemetry](https://opentelemetry.io/), AgentMark automatically collects telemetry data from your prompts so you can monitor and debug them. ## Core concepts ### Traces A trace represents a complete request or workflow in your application. A unique trace ID identifies each trace, which contains one or more spans. Traces carry top-level attributes such as metadata, tags, user ID, and session ID. ### Spans Spans are individual operations within a trace, forming a tree structure. AgentMark classifies every ingested span by type: * **`GENERATION`**: a model call, including model, tokens, cost, and response. Detected from `gen_ai.*` attributes or AI SDK span names such as `ai.generateText` and `ai.streamText`. * **`SPAN`**: any other operation: tool calls, retrieval steps, and custom `span()` / `observe()` wrappers. * **`EVENT`**: point-in-time records. The spans API accepts this type alongside the other two. ### Span kinds Every span has a semantic kind that categorizes the operation: `function` (default), `llm`, `tool`, `agent`, `retrieval`, `embedding`, or `guardrail`. Span kinds determine how you [filter](/observe/filtering-and-search) spans and how [dashboards](/observe/dashboards) group analytics. See [SpanKind values](/observe/span-reference#spankind-values) for the full reference, and set them using [`observe()`](/observe/tracing-setup#wrapping-functions-with-observe) or `ctx.span()`. ### Sessions Sessions group related traces together by session ID. Track multi-turn conversations, agent workflows, and batch processing runs. Each session aggregates cost, tokens, and latency across its traces. [Learn more about Sessions →](/observe/sessions) ### Scores Numeric evaluations attached to spans or traces. Set scores programmatically via the SDK using `sdk.score()`, or manually through [annotations](/evaluate/annotations) in the Dashboard. ### Metadata and tags **Metadata**: custom key-value pairs attached to traces for context (environment, feature flags, customer tier). Automatically discovered as filter fields. **Tags**: string labels for categorization (environment, team, feature, release). [Metadata →](/observe/metadata) · [Tags →](/observe/tags) ## What gets tracked **Model calls**: full prompt execution lifecycle: token usage, costs, response times, model information, completion status. **Tool calls**: tool name, parameters, execution duration, success/failure status, return values. **Streaming metrics**: time to first token, tokens per second, total streaming duration. **Sessions**: group related traces by user interaction, multi-step workflow, or batch run. **Alerts**: monitor cost thresholds, latency spikes, error rates, and evaluation scores. ## Quick start With tracing initialized (`sdk.initTracing({ registerGlobally: true })`, see [Tracing setup](/observe/tracing-setup)), render your prompt and enable telemetry on the model call: ```typescript theme={null} import { client } from './agentmark.client'; import { openai } from '@ai-sdk/openai'; import { generateText } from 'ai'; const prompt = await client.loadTextPrompt('greeting.prompt.mdx'); const { messages, text_config } = await prompt.format({ props: { name: 'Alice' }, }); const result = await generateText({ model: openai(text_config.model_name.replace(/^openai\//, '')), messages, experimental_telemetry: { isEnabled: true, functionId: 'greeting-handler', // Session/user keys in metadata must be snake_case to group traces (see Sessions) metadata: { user_id: 'user-123', session_id: 'session-abc', session_name: 'Customer Support Chat' } } }); ``` The render is neutral: `format()` returns `{ messages, text_config }` and your code calls the model, so you enable telemetry per call via the AI SDK's `experimental_telemetry` option. For full tracing setup including `AgentMarkSDK`, child spans, `observe()`, and span kinds, see [Tracing setup](/observe/tracing-setup). ## How data flows Your application sends telemetry via the AgentMark SDK, which exports OpenTelemetry spans to the AgentMark gateway. The gateway processes and stores the data, powering the traces, metrics, and analytics views. The SDK exports spans to the AgentMark Cloud gateway, which stores them for querying. View traces, dashboards, alerts, and analytics in the Dashboard. When you run `agentmark dev`, the SDK sends traces to `http://localhost:9418` automatically. View them in the local dev server UI at `http://localhost:3000`. The local dev server exposes most of the REST API so the same endpoints work against local data. The `/v1/capabilities` endpoint reports which features a given server supports; `metrics` and `score_analytics` are Cloud-only and return `501 not_available_locally`. ## Programmatic access You can query traces, spans, sessions, scores, metrics, datasets, experiments, prompts, and runs programmatically through the [REST API](/api-reference/overview), or from an IDE agent via the [`agentmark-mcp`](/coding-agents/gateway-mcp) MCP server (which exposes one MCP tool per gateway operation). Use either to build custom integrations, pull data into external tools, or automate monitoring workflows. Most endpoints are available on both the local dev server and the AgentMark Cloud gateway. The local server returns `501 not_available_locally` for features that require server-side aggregations (`/v1/metrics` and score analytics). Use the [`capabilities`](/api-reference/overview) endpoint to check which features a server supports. ```bash theme={null} # Query traces from the local dev server curl -fsS "http://localhost:9418/v1/traces?limit=10" # Get a specific trace with all its spans curl -fsS "http://localhost:9418/v1/traces/" # Same calls against Cloud — set AGENTMARK_API_KEY + AGENTMARK_APP_ID curl -fsS "https://api.agentmark.co/v1/traces?limit=10" \ -H "Authorization: Bearer $AGENTMARK_API_KEY" \ -H "X-Agentmark-App-Id: $AGENTMARK_APP_ID" # Check which features are available on a given server curl -fsS "http://localhost:9418/v1/capabilities" ``` The same operations are available as MCP tools (`list_traces`, `get_trace`, `get_capabilities`, …) when you run the `agentmark-mcp` server alongside your IDE. ## Next steps Instrument your app with the SDK View execution timelines in the Dashboard Group related traces together Get notified of critical issues Analyze usage, performance, and scores Query traces, scores, and metrics via REST API

Have questions?

Reach out any time: