Overview

Puzzlet builds on top of OpenTelemetry for collecting telemetry data from your prompts. This helps you monitor, debug, and optimize your LLM applications in production.

​

What We Track

Puzzlet automatically collects:

• Inference Spans: Full lifecycle of prompt execution

  • Token usage
  • Response times
  • Model information
  • Completion status
  • Cost
  • Response status

• Tool Calls: When your prompts use tools

  • Tool name and parameters
  • Execution duration
  • Success/failure status
  • Return values

• Streaming Metrics: For streaming responses

  • Time to first token
  • Tokens per second
  • Total streaming duration

• Sessions: Group related traces together

  • Organize traces by user interaction
  • Track multi-step workflows
  • Monitor batch processing jobs
  • Analyze performance across related operations

• Alerts: Monitor critical metrics and get notified

  • Cost thresholds
  • Latency monitoring
  • Error rate tracking
  • Notifications via Slack or webhooks
  • Alert history for pattern analysis

​

Basic Usage

Enable telemetry in your Puzzlet client:

const myPrompt = puzzletClient.fetchPrompt('my-prompt.prompt.mdx');
const props = { num: 1 };
const result = myPrompt.run(props, {
  telemetry: {
    enabled: true,
    metadata: {
      userId: '1234567890',
      sessionId: 'session-123',
      sessionName: 'Example Session'
    }
  }
});

​

Learn More

For detailed information about spans, metrics, and custom configuration, see:

• Spans & Traces
• Alerts
• Sessions
• Metrics