Tracing setup - AgentMark Docs

AgentMark uses OpenTelemetry to collect distributed traces and metrics for your prompt executions. This page covers setup and the tracing patterns built on top of it.

Already instrumenting with a framework? Apps traced by OpenInference (LangChain, LlamaIndex, CrewAI, …) or OpenLLMetry (Traceloop, OpenLIT) can send traces to AgentMark without the SDK. Point their OTLP exporter at AgentMark. See Tracing integrations.

Install the SDK

TypeScript
Python

npm install @agentmark-ai/sdk

pip install agentmark-sdk

Initialize tracing

TypeScript: pass registerGlobally: true, or your model-call spans silently vanish.The AI SDK emits the model-call (“generation”) span through the global OpenTelemetry tracer. AgentMark’s tracer stays isolated by default so it never clobbers an existing OTel setup in your app. Without registerGlobally: true, your custom span() / observe() wrappers still arrive, so it looks like “tracing works”, but every model span goes to a no-op tracer and never shows up. The result is a trace tree with no model data, no tokens, and no cost, which is easy to miss because the rest of the trace looks healthy.

const tracer = sdk.initTracing({ registerGlobally: true });

Omit registerGlobally only when your app already registers its own global OTel provider.Python: init_tracing() always registers the provider globally, with no separate flag. This is the equivalent of TypeScript’s registerGlobally: true, handled for you.

This is the single most common reason traces look like they’re working but carry no model data. If model spans are missing, this is the first thing to check. For the other silent failures that drop spans (telemetry not enabled, env vars not loaded, or a script exiting before the batch flushes), see Troubleshooting traces.

TypeScript
Python

import { AgentMarkSDK } from "@agentmark-ai/sdk";
import { createAgentMark } from "@agentmark-ai/prompt-core";
import { openai } from "@ai-sdk/openai";
import { generateText } from "ai";

const sdk = new AgentMarkSDK({
  apiKey: process.env.AGENTMARK_API_KEY,
  appId: process.env.AGENTMARK_APP_ID,
  baseUrl: process.env.AGENTMARK_BASE_URL  // defaults to https://api.agentmark.co
});

// Initialize tracing.
// registerGlobally: true is REQUIRED for the model-call (generation) span:
// the AI SDK emits it through the global tracer. Omit it only when your app
// already registers its own global OTel provider.
const tracer = sdk.initTracing({ registerGlobally: true });

// The neutral client only loads + renders prompts; your code calls the model.
const client = createAgentMark({ loader: sdk.getApiLoader() });

// Load and render the prompt, then call your model SDK.
const prompt = await client.loadTextPrompt("greeting.prompt.mdx");
const { messages, text_config } = await prompt.format({
  props: { name: 'Alice' },
});

// Telemetry is per call: without experimental_telemetry the AI SDK
// emits no spans for this call.
const result = await generateText({
  model: openai(text_config.model_name.replace(/^openai\//, "")),
  messages,
  experimental_telemetry: {
    isEnabled: true,
    functionId: "greeting-function",
    metadata: {
      user_id: "123",
      environment: "production"
    }
  },
});

// Shutdown tracer (only for short-running scripts)
await tracer.shutdown();

import os
from agentmark_sdk import AgentMarkSDK
from agentmark_client import client

sdk = AgentMarkSDK(
    api_key=os.environ["AGENTMARK_API_KEY"],
    app_id=os.environ["AGENTMARK_APP_ID"],
)

# Initialize tracing (always registers the global tracer provider)
tracer = sdk.init_tracing()

# Load and render the prompt, then call your model SDK.
prompt = await client.load_text_prompt("greeting.prompt.mdx")
formatted = await prompt.format(props={"name": "Alice"})

# The render is a pydantic model: read `formatted.messages` and
# `formatted.text_config.model_name`. The generation span comes from
# your model SDK's own OpenTelemetry instrumentation.
result = await call_your_model(formatted)

# Shutdown tracer (only for short-running scripts)
tracer.shutdown()

TypeScript
Python

For local development with agentmark dev, the SDK sends traces to http://localhost:9418 automatically. Pass disableBatch: true for short-running scripts:

const tracer = sdk.initTracing({ disableBatch: true });

For local development with agentmark dev, the SDK sends traces to http://localhost:9418 automatically. Pass disable_batch=True for short-running scripts:

tracer = sdk.init_tracing(disable_batch=True)

Once spans are flowing, AgentMark classifies each one by type (GENERATION for model calls, SPAN for everything else) and reads attributes like model, tokens, and response off it. For the full list of span types, attributes, how traces link to a prompt version, and how AgentMark derives trace-level input/output, see the Span and attribute reference.

Attributing traces to an environment

By default, AgentMark pins an API key to one environment, and every trace it sends lands there. A key scoped instead to environment kinds (see API key environment scope) carries no pin, so it lets the SDK pick the target environment per request. Pass environment or prNumber to initTracing():

// Send these traces to the env named "staging".
const tracer = sdk.initTracing({ registerGlobally: true, environment: "staging" });

// Or send them to the preview env for PR #482.
const tracer = sdk.initTracing({ registerGlobally: true, prNumber: 482 });

environment sets the X-Agentmark-Environment header. The gateway authorizes the named environment against the key’s allowed kinds; if the key can’t write to that environment’s kind, the gateway drops the traces rather than writing them to the wrong environment.
prNumber sets the X-Agentmark-Pr-Number header. The gateway maps it to that pull request’s preview environment. When you set both, prNumber wins.

Omitting both falls back to the key’s pinned environment, so a pinned key never needs these options.

Zero-config selection in CI

For CI runs where adding code isn’t convenient, initTracing() reads two environment variable fallbacks:

AGENTMARK_ENVIRONMENT, the fallback for environment.
AGENTMARK_PR_NUMBER, the fallback for prNumber.

The precedence is: an explicit option, then the matching AGENTMARK_* variable, then the Vercel auto-derivation below. The eval Action and component set these variables for you, so eval traces from a pull request land in that PR’s preview environment with no configuration.

Automatic selection on Vercel

When your app runs on Vercel, initTracing() derives the environment from the system variables Vercel already exposes, so a Vercel deploy needs no selector configuration. A Preview deployment for an open pull request attributes traces to that PR’s preview environment, a branch Preview deployment with no open PR uses the branch ref as the environment name, and a Development deployment uses the dev environment. A Production deployment sends to the key’s pinned environment. The derived selector is the lowest-precedence source, so an explicit option or AGENTMARK_* variable still overrides it. See Vercel integration for the full mapping.

Grouping operations into a span

Use span() (TypeScript) or span_context() (Python) to wrap a block of work as a single parent span. Nested SDK calls automatically attach as children.

TypeScript
Python

import { span } from "@agentmark-ai/sdk";

const { result, traceId } = await span(
  { name: 'user-request-handler' },
  async (ctx) => {
    const prompt = await client.loadTextPrompt('handler.prompt.mdx');
    const { messages, text_config } = await prompt.format({
      props: { query: 'What is AgentMark?' },
    });

    return await generateText({
      model: openai(text_config.model_name.replace(/^openai\//, "")),
      messages,
      experimental_telemetry: { isEnabled: true },
    });
  }
);

console.log('Trace ID:', traceId);
const output = await result;

from agentmark_sdk import span_context, SpanOptions
from agentmark_client import client

async with span_context(SpanOptions(name="user-request-handler")) as ctx:
    prompt = await client.load_text_prompt("handler.prompt.mdx")
    formatted = await prompt.format(props={"query": "What is AgentMark?"})

    result = await call_your_model(formatted)

print(f"Trace ID: {ctx.trace_id}")

In TypeScript, span() returns { result, traceId } where result is Promise<T>, not T. You need to await it to get the resolved value. In Python, span_context() is an async context manager that exposes ctx.trace_id immediately.

SpanOptions accept far more than name: a sessionId to group traces into a session, a userId, metadata, and the dataset/experiment fields used by experiment runs. For every field and its Python snake_case equivalent, see SpanOptions in the reference.

Creating child spans

Use ctx.span() inside a callback to create child spans under the current parent:

TypeScript
Python

import { span } from "@agentmark-ai/sdk";

const { result, traceId } = await span(
  { name: 'multi-step-workflow' },
  async (ctx) => {
    await ctx.span({ name: 'validate-input' }, async (spanCtx) => {
      spanCtx.setAttribute('input.length', 42);
    });

    const output = await ctx.span({ name: 'process-request' }, async (spanCtx) => {
      const prompt = await client.loadTextPrompt('process.prompt.mdx');
      const { messages, text_config } = await prompt.format({
        props: { query: 'process this' },
      });
      return await generateText({
        model: openai(text_config.model_name.replace(/^openai\//, "")),
        messages,
        experimental_telemetry: { isEnabled: true },
      });
    });

    await ctx.span({ name: 'format-response' }, async (spanCtx) => {
      spanCtx.addEvent('formatting-complete');
    });

    return output;
  }
);

ctx.span() accepts { name: string; metadata?: Record<string, string> }. Use observe() (below) if you need to set a SpanKind on a span.

from agentmark_sdk import span_context, SpanOptions
from agentmark_client import client

async with span_context(SpanOptions(name="multi-step-workflow")) as ctx:
    async with ctx.span("validate-input") as span_ctx:
        span_ctx.set_attribute("input.length", 42)

    async with ctx.span("process-request") as span_ctx:
        prompt = await client.load_text_prompt("process.prompt.mdx")
        formatted = await prompt.format(props={"query": "process this"})
        output = await call_your_model(formatted)

    async with ctx.span("format-response") as span_ctx:
        span_ctx.add_event("formatting-complete")

Wrapping functions with `observe()`

observe() wraps an async function with automatic input/output capture AND lets you set a SpanKind. Unlike span() / ctx.span() which create inline spans, observe() produces a reusable function so every call is automatically traced.

TypeScript
Python

import { observe, SpanKind } from "@agentmark-ai/sdk";

const searchWeb = observe(
  async (query: string) => {
    const response = await fetch(`https://api.search.com?q=${query}`);
    return response.json();
  },
  { name: "search-web", kind: SpanKind.TOOL }
);

// Every call is now automatically traced
const results = await searchWeb("AgentMark tracing");

from agentmark_sdk import observe, SpanKind

@observe(name="search-web", kind=SpanKind.TOOL)
async def search_web(query: str) -> dict:
    async with httpx.AsyncClient() as http:
        response = await http.get(f"https://api.search.com?q={query}")
        return response.json()

# Every call is now automatically traced
results = await search_web("AgentMark tracing")

Observed functions automatically attach to the active trace context, so they nest correctly inside span() / span_context() without extra wiring. You can also redact arguments or return values before they’re recorded with the processInputs / processOutputs options. For every observe() option and the full list of SpanKind values, see the Span and attribute reference.

Using `SpanKind` in a pipeline

To set SpanKind on individual steps of a pipeline, wrap each step with observe() and call the wrapped functions inside span():

TypeScript
Python

import { span, observe, SpanKind } from "@agentmark-ai/sdk";

const searchKB = observe(
  async (query: string) => vectorDb.query({ query, topK: 5 }),
  { name: 'search-knowledge-base', kind: SpanKind.RETRIEVAL }
);

const guardrail = observe(
  async (question: string) => moderationService.check(question),
  { name: 'check-content-policy', kind: SpanKind.GUARDRAIL }
);

const generateAnswer = observe(
  async (question: string, context: unknown) => {
    const prompt = await client.loadTextPrompt('answer.prompt.mdx');
    const { messages, text_config } = await prompt.format({
      props: { question, context },
    });
    return generateText({
      model: openai(text_config.model_name.replace(/^openai\//, "")),
      messages,
      experimental_telemetry: { isEnabled: true },
    });
  },
  { name: 'generate-answer', kind: SpanKind.LLM }
);

const { result } = await span(
  { name: 'rag-pipeline' },
  async () => {
    const docs = await searchKB(userQuestion);
    await guardrail(userQuestion);
    return generateAnswer(userQuestion, docs);
  }
);

from agentmark_sdk import span_context, SpanOptions, observe, SpanKind

@observe(name="search-knowledge-base", kind=SpanKind.RETRIEVAL)
async def search_kb(query: str):
    return await vector_db.query(query=query, top_k=5)

@observe(name="check-content-policy", kind=SpanKind.GUARDRAIL)
async def guardrail(question: str):
    await moderation_service.check(question)

@observe(name="generate-answer", kind=SpanKind.LLM)
async def generate_answer(question: str, context) -> str:
    prompt = await client.load_text_prompt("answer.prompt.mdx")
    formatted = await prompt.format(
        props={"question": question, "context": context},
    )
    return await call_your_model(formatted)

async with span_context(SpanOptions(name="rag-pipeline")):
    docs = await search_kb(user_question)
    await guardrail(user_question)
    answer = await generate_answer(user_question, docs)

Scoring traces

Use sdk.score() to attach quality scores to traces or spans:

TypeScript
Python

const { result, traceId } = await span(
  { name: 'scored-workflow' },
  async (ctx) => {
    const output = await myAsyncFunction();
    return output;
  }
);

await sdk.score({
  resourceId: traceId,
  name: 'correctness',
  score: 0.95,
  label: 'correct',
  reason: 'Output matches expected result'
});

async with span_context(SpanOptions(name="scored-workflow")) as ctx:
    output = await my_async_function()

await sdk.score(
    resource_id=ctx.trace_id,
    name="correctness",
    score=0.95,
    label="correct",
    reason="Output matches expected result",
)

Best practices

Use meaningful function IDs: "customer-support-greeting", not "func1"
Add relevant metadata: user_id, environment, query parameters
Always enable telemetry in production: monitor performance and set up alerts
Shutdown tracer for short scripts: call tracer.shutdown() before the process exits

Next steps

Span reference

Span types, attributes, SpanOptions, and SpanKind values

Sessions

Group related traces together

Metadata

Add custom context to traces

PII masking

Redact sensitive data from traces

Troubleshooting

Fix traces that aren’t showing up

Have questions?

Reach out any time:

Email the team at hello@agentmark.co for support
Schedule an Enterprise Demo to learn about AgentMark’s business solutions

​Install the SDK

​Initialize tracing

​Attributing traces to an environment

​Zero-config selection in CI

​Automatic selection on Vercel

​Grouping operations into a span

​Creating child spans

​Wrapping functions with observe()

​Using SpanKind in a pipeline

​Scoring traces

​Best practices

​Next steps

Span reference

Sessions

Metadata

Tags

PII masking

Troubleshooting

​Have questions?

Install the SDK

Initialize tracing

Attributing traces to an environment

Zero-config selection in CI

Automatic selection on Vercel

Grouping operations into a span

Creating child spans

Wrapping functions with `observe()`

Using `SpanKind` in a pipeline

Scoring traces

Best practices

Next steps

Have questions?