AgentMark builds on top of OpenTelemetry for collecting telemetry data from your prompts. This helps you monitor, debug, and optimize your LLM applications in production.

Traces

​
What We Track

AgentMark automatically collects:

  • Inference Spans: Full lifecycle of prompt execution

    • Token usage
    • Response times
    • Model information
    • Completion status
    • Cost
    • Response status

  • Tool Calls: When your prompts use tools

    • Tool name and parameters
    • Execution duration
    • Success/failure status
    • Return values

  • Streaming Metrics: For streaming responses

    • Time to first token
    • Tokens per second
    • Total streaming duration

  • Sessions: Group related traces together

    • Organize traces by user interaction
    • Track multi-step workflows
    • Monitor batch processing jobs
    • Analyze performance across related operations

  • Alerts: Monitor critical metrics and get notified

    • Cost thresholds
    • Latency monitoring
    • Error rate tracking
    • Notifications via Slack or webhooks
    • Alert history for pattern analysis

​
Basic Usage

Enable telemetry in your AgentMark client:

const myPrompt = agentmark.loadTextPrompt('my-prompt.prompt.mdx');
const props = { num: 1 };
const result = myPrompt.format({
  props,
  telemetry: {
    isEnabled: true,
    metadata: {
      userId: '1234567890',
      sessionId: 'session-123',
      sessionName: 'Example Session'
    }
  }
});

​
Learn More

For detailed information about spans, metrics, and custom configuration, see:

Have Questions?

We’re here to help! Choose the best way to reach us: