> ## Documentation Index > Fetch the complete documentation index at: https://docs.agentmark.co/llms.txt > Use this file to discover all available pages before exploring further. # Tracing setup > Instrument your application with the AgentMark SDK to capture traces, spans, and metrics AgentMark uses [OpenTelemetry](https://opentelemetry.io/) to collect distributed traces and metrics for your prompt executions. This page covers setup and the tracing patterns built on top of it. Already instrumenting with a framework? Apps traced by [OpenInference](/integrations/tracing/openinference) (LangChain, LlamaIndex, CrewAI, …) or [OpenLLMetry](/integrations/tracing/openllmetry) (Traceloop, OpenLIT) can send traces to AgentMark without the SDK. Point their OTLP exporter at AgentMark. See [Tracing integrations](/integrations/tracing/opentelemetry). ## Install the SDK ```bash theme={null} npm install @agentmark-ai/sdk ``` ```bash theme={null} pip install agentmark-sdk ``` ## Initialize tracing **TypeScript: pass `registerGlobally: true`, or your model-call spans silently vanish.** The AI SDK emits the model-call ("generation") span through the *global* OpenTelemetry tracer. AgentMark's tracer stays isolated by default so it never clobbers an existing OTel setup in your app. Without `registerGlobally: true`, your custom `span()` / `observe()` wrappers still arrive, so it looks like "tracing works", but every model span goes to a no-op tracer and never shows up. The result is a trace tree with no model data, no tokens, and no cost, which is easy to miss because the rest of the trace looks healthy. ```typescript theme={null} const tracer = sdk.initTracing({ registerGlobally: true }); ``` Omit `registerGlobally` only when your app already registers its own global OTel provider. **Python: `init_tracing()` always registers the provider globally**, with no separate flag. This is the equivalent of TypeScript's `registerGlobally: true`, handled for you. This is the single most common reason traces look like they're working but carry no model data. If model spans are missing, this is the first thing to check. For the other silent failures that drop spans (telemetry not enabled, env vars not loaded, or a script exiting before the batch flushes), see [Troubleshooting traces](/observe/troubleshooting). ```typescript theme={null} import { AgentMarkSDK } from "@agentmark-ai/sdk"; import { createAgentMark } from "@agentmark-ai/prompt-core"; import { openai } from "@ai-sdk/openai"; import { generateText } from "ai"; const sdk = new AgentMarkSDK({ apiKey: process.env.AGENTMARK_API_KEY, appId: process.env.AGENTMARK_APP_ID, baseUrl: process.env.AGENTMARK_BASE_URL // defaults to https://api.agentmark.co }); // Initialize tracing. // registerGlobally: true is REQUIRED for the model-call (generation) span: // the AI SDK emits it through the global tracer. Omit it only when your app // already registers its own global OTel provider. const tracer = sdk.initTracing({ registerGlobally: true }); // The neutral client only loads + renders prompts; your code calls the model. const client = createAgentMark({ loader: sdk.getApiLoader() }); // Load and render the prompt, then call your model SDK. const prompt = await client.loadTextPrompt("greeting.prompt.mdx"); const { messages, text_config } = await prompt.format({ props: { name: 'Alice' }, }); // Telemetry is per call: without experimental_telemetry the AI SDK // emits no spans for this call. const result = await generateText({ model: openai(text_config.model_name.replace(/^openai\//, "")), messages, experimental_telemetry: { isEnabled: true, functionId: "greeting-function", metadata: { user_id: "123", environment: "production" } }, }); // Shutdown tracer (only for short-running scripts) await tracer.shutdown(); ``` ```python theme={null} import os from agentmark_sdk import AgentMarkSDK from agentmark_client import client sdk = AgentMarkSDK( api_key=os.environ["AGENTMARK_API_KEY"], app_id=os.environ["AGENTMARK_APP_ID"], ) # Initialize tracing (always registers the global tracer provider) tracer = sdk.init_tracing() # Load and render the prompt, then call your model SDK. prompt = await client.load_text_prompt("greeting.prompt.mdx") formatted = await prompt.format(props={"name": "Alice"}) # The render is a pydantic model: read `formatted.messages` and # `formatted.text_config.model_name`. The generation span comes from # your model SDK's own OpenTelemetry instrumentation. result = await call_your_model(formatted) # Shutdown tracer (only for short-running scripts) tracer.shutdown() ``` For local development with `agentmark dev`, the SDK sends traces to `http://localhost:9418` automatically. Pass `disableBatch: true` for short-running scripts: ```typescript theme={null} const tracer = sdk.initTracing({ disableBatch: true }); ``` For local development with `agentmark dev`, the SDK sends traces to `http://localhost:9418` automatically. Pass `disable_batch=True` for short-running scripts: ```python theme={null} tracer = sdk.init_tracing(disable_batch=True) ``` Once spans are flowing, AgentMark classifies each one by type (`GENERATION` for model calls, `SPAN` for everything else) and reads attributes like model, tokens, and response off it. For the full list of span types, attributes, how traces link to a prompt version, and how AgentMark derives trace-level input/output, see the [Span and attribute reference](/observe/span-reference). ## Attributing traces to an environment By default, AgentMark pins an API key to one [environment](/concepts/environments), and every trace it sends lands there. A key scoped instead to environment **kinds** (see [API key environment scope](/api-reference/authentication#api-key-environment-scope)) carries no pin, so it lets the SDK pick the target environment per request. Pass `environment` or `prNumber` to `initTracing()`: ```typescript theme={null} // Send these traces to the env named "staging". const tracer = sdk.initTracing({ registerGlobally: true, environment: "staging" }); // Or send them to the preview env for PR #482. const tracer = sdk.initTracing({ registerGlobally: true, prNumber: 482 }); ``` * `environment` sets the `X-Agentmark-Environment` header. The gateway authorizes the named environment against the key's allowed kinds; if the key can't write to that environment's kind, the gateway drops the traces rather than writing them to the wrong environment. * `prNumber` sets the `X-Agentmark-Pr-Number` header. The gateway maps it to that pull request's [preview environment](/deploy/environments-and-promotions). When you set both, `prNumber` wins. Omitting both falls back to the key's pinned environment, so a pinned key never needs these options. ### Zero-config selection in CI For CI runs where adding code isn't convenient, `initTracing()` reads two environment variable fallbacks: * `AGENTMARK_ENVIRONMENT`, the fallback for `environment`. * `AGENTMARK_PR_NUMBER`, the fallback for `prNumber`. The precedence is: an explicit option, then the matching `AGENTMARK_*` variable, then the Vercel auto-derivation below. The [eval Action and component](/deploy/ci-cd) set these variables for you, so eval traces from a pull request land in that PR's preview environment with no configuration. ### Automatic selection on Vercel When your app runs on Vercel, `initTracing()` derives the environment from the system variables Vercel already exposes, so a Vercel deploy needs no selector configuration. A Preview deployment for an open pull request attributes traces to that PR's preview environment, a branch Preview deployment with no open PR uses the branch ref as the environment name, and a Development deployment uses the `dev` environment. A Production deployment sends to the key's pinned environment. The derived selector is the lowest-precedence source, so an explicit option or `AGENTMARK_*` variable still overrides it. See [Vercel integration](/integrations/vercel-connect) for the full mapping. ## Grouping operations into a span Use `span()` (TypeScript) or `span_context()` (Python) to wrap a block of work as a single parent span. Nested SDK calls automatically attach as children. ```typescript theme={null} import { span } from "@agentmark-ai/sdk"; const { result, traceId } = await span( { name: 'user-request-handler' }, async (ctx) => { const prompt = await client.loadTextPrompt('handler.prompt.mdx'); const { messages, text_config } = await prompt.format({ props: { query: 'What is AgentMark?' }, }); return await generateText({ model: openai(text_config.model_name.replace(/^openai\//, "")), messages, experimental_telemetry: { isEnabled: true }, }); } ); console.log('Trace ID:', traceId); const output = await result; ``` ```python theme={null} from agentmark_sdk import span_context, SpanOptions from agentmark_client import client async with span_context(SpanOptions(name="user-request-handler")) as ctx: prompt = await client.load_text_prompt("handler.prompt.mdx") formatted = await prompt.format(props={"query": "What is AgentMark?"}) result = await call_your_model(formatted) print(f"Trace ID: {ctx.trace_id}") ``` In TypeScript, `span()` returns `{ result, traceId }` where `result` is `Promise`, not `T`. You need to `await` it to get the resolved value. In Python, `span_context()` is an async context manager that exposes `ctx.trace_id` immediately. `SpanOptions` accept far more than `name`: a `sessionId` to group traces into a [session](/observe/sessions), a `userId`, `metadata`, and the dataset/experiment fields used by experiment runs. For every field and its Python snake\_case equivalent, see [SpanOptions](/observe/span-reference#spanoptions) in the reference. ### Creating child spans Use `ctx.span()` inside a callback to create child spans under the current parent: ```typescript theme={null} import { span } from "@agentmark-ai/sdk"; const { result, traceId } = await span( { name: 'multi-step-workflow' }, async (ctx) => { await ctx.span({ name: 'validate-input' }, async (spanCtx) => { spanCtx.setAttribute('input.length', 42); }); const output = await ctx.span({ name: 'process-request' }, async (spanCtx) => { const prompt = await client.loadTextPrompt('process.prompt.mdx'); const { messages, text_config } = await prompt.format({ props: { query: 'process this' }, }); return await generateText({ model: openai(text_config.model_name.replace(/^openai\//, "")), messages, experimental_telemetry: { isEnabled: true }, }); }); await ctx.span({ name: 'format-response' }, async (spanCtx) => { spanCtx.addEvent('formatting-complete'); }); return output; } ); ``` `ctx.span()` accepts `{ name: string; metadata?: Record }`. Use `observe()` (below) if you need to set a `SpanKind` on a span. ```python theme={null} from agentmark_sdk import span_context, SpanOptions from agentmark_client import client async with span_context(SpanOptions(name="multi-step-workflow")) as ctx: async with ctx.span("validate-input") as span_ctx: span_ctx.set_attribute("input.length", 42) async with ctx.span("process-request") as span_ctx: prompt = await client.load_text_prompt("process.prompt.mdx") formatted = await prompt.format(props={"query": "process this"}) output = await call_your_model(formatted) async with ctx.span("format-response") as span_ctx: span_ctx.add_event("formatting-complete") ``` ## Wrapping functions with `observe()` `observe()` wraps an async function with automatic input/output capture AND lets you set a `SpanKind`. Unlike `span()` / `ctx.span()` which create inline spans, `observe()` produces a reusable function so every call is automatically traced. ```typescript theme={null} import { observe, SpanKind } from "@agentmark-ai/sdk"; const searchWeb = observe( async (query: string) => { const response = await fetch(`https://api.search.com?q=${query}`); return response.json(); }, { name: "search-web", kind: SpanKind.TOOL } ); // Every call is now automatically traced const results = await searchWeb("AgentMark tracing"); ``` ```python theme={null} from agentmark_sdk import observe, SpanKind @observe(name="search-web", kind=SpanKind.TOOL) async def search_web(query: str) -> dict: async with httpx.AsyncClient() as http: response = await http.get(f"https://api.search.com?q={query}") return response.json() # Every call is now automatically traced results = await search_web("AgentMark tracing") ``` Observed functions automatically attach to the active trace context, so they nest correctly inside `span()` / `span_context()` without extra wiring. You can also redact arguments or return values before they're recorded with the `processInputs` / `processOutputs` options. For every `observe()` option and the full list of [`SpanKind` values](/observe/span-reference#spankind-values), see the [Span and attribute reference](/observe/span-reference). ### Using `SpanKind` in a pipeline To set `SpanKind` on individual steps of a pipeline, wrap each step with `observe()` and call the wrapped functions inside `span()`: ```typescript theme={null} import { span, observe, SpanKind } from "@agentmark-ai/sdk"; const searchKB = observe( async (query: string) => vectorDb.query({ query, topK: 5 }), { name: 'search-knowledge-base', kind: SpanKind.RETRIEVAL } ); const guardrail = observe( async (question: string) => moderationService.check(question), { name: 'check-content-policy', kind: SpanKind.GUARDRAIL } ); const generateAnswer = observe( async (question: string, context: unknown) => { const prompt = await client.loadTextPrompt('answer.prompt.mdx'); const { messages, text_config } = await prompt.format({ props: { question, context }, }); return generateText({ model: openai(text_config.model_name.replace(/^openai\//, "")), messages, experimental_telemetry: { isEnabled: true }, }); }, { name: 'generate-answer', kind: SpanKind.LLM } ); const { result } = await span( { name: 'rag-pipeline' }, async () => { const docs = await searchKB(userQuestion); await guardrail(userQuestion); return generateAnswer(userQuestion, docs); } ); ``` ```python theme={null} from agentmark_sdk import span_context, SpanOptions, observe, SpanKind @observe(name="search-knowledge-base", kind=SpanKind.RETRIEVAL) async def search_kb(query: str): return await vector_db.query(query=query, top_k=5) @observe(name="check-content-policy", kind=SpanKind.GUARDRAIL) async def guardrail(question: str): await moderation_service.check(question) @observe(name="generate-answer", kind=SpanKind.LLM) async def generate_answer(question: str, context) -> str: prompt = await client.load_text_prompt("answer.prompt.mdx") formatted = await prompt.format( props={"question": question, "context": context}, ) return await call_your_model(formatted) async with span_context(SpanOptions(name="rag-pipeline")): docs = await search_kb(user_question) await guardrail(user_question) answer = await generate_answer(user_question, docs) ``` ## Scoring traces Use `sdk.score()` to attach quality scores to traces or spans: ```typescript theme={null} const { result, traceId } = await span( { name: 'scored-workflow' }, async (ctx) => { const output = await myAsyncFunction(); return output; } ); await sdk.score({ resourceId: traceId, name: 'correctness', score: 0.95, label: 'correct', reason: 'Output matches expected result' }); ``` ```python theme={null} async with span_context(SpanOptions(name="scored-workflow")) as ctx: output = await my_async_function() await sdk.score( resource_id=ctx.trace_id, name="correctness", score=0.95, label="correct", reason="Output matches expected result", ) ``` ## Best practices * **Use meaningful function IDs**: `"customer-support-greeting"`, not `"func1"` * **Add relevant metadata**: `user_id`, environment, query parameters * **Always enable telemetry in production**: monitor performance and set up alerts * **Shutdown tracer for short scripts**: call `tracer.shutdown()` before the process exits ## Next steps Span types, attributes, SpanOptions, and SpanKind values Group related traces together Add custom context to traces Categorize traces with labels Redact sensitive data from traces Fix traces that aren't showing up

Have questions?

Reach out any time: