Developers set up tracing in your application. See Development documentation for setup instructions.
Understanding Traces
A trace represents the complete execution of a prompt, including all its steps, tool calls, and metadata. Each trace contains: Execution Timeline - See exactly when each step occurred and how long it took. Token Usage - Track input tokens, output tokens, and total tokens consumed. Costs - Monitor spending on a per-request basis. Tool Calls - View all tool executions, their parameters, and results. Custom Metadata - Add context like user IDs, session IDs, and custom attributes. Error Information - Detailed error messages and stack traces when issues occur.Collected Spans
AgentMark records the following OpenTelemetry spans:| Span Type | Description | Attributes |
|---|---|---|
ai.inference | Full length of the inference call | operation.name, ai.operationId, ai.prompt, ai.response.text, ai.response.toolCalls, ai.response.finishReason |
ai.toolCall | Individual tool executions | operation.name, ai.operationId, ai.toolCall.name, ai.toolCall.args, ai.toolCall.result |
ai.stream | Streaming response data | ai.response.msToFirstChunk, ai.response.msToFinish, ai.response.avgCompletionTokensPerSecond |
LLM Span Attributes
Each LLM span contains:| Attribute | Description |
|---|---|
ai.model.id | Model identifier |
ai.model.provider | Model provider name |
ai.usage.promptTokens | Number of prompt tokens |
ai.usage.completionTokens | Number of completion tokens |
ai.settings.maxRetries | Maximum retry attempts |
ai.telemetry.functionId | Function identifier |
ai.telemetry.metadata.* | Custom metadata |
Grouping Traces
Organize related traces together using custom grouping. This is useful for understanding complex workflows that span multiple prompt executions.
- Multi-step agent workflows
- Nested component execution
- Parallel processing pipelines
Graph View
For complex AI agent workflows, AgentMark provides an interactive graph visualization that shows the relationships between different components, execution flow, and dependencies. Learn more about Graph View →Viewing Traces
Access traces in the AgentMark dashboard under the “Traces” tab. Each trace shows:- Complete prompt execution timeline
- Tool calls and their durations
- Token usage and costs
- Custom metadata and attributes
- Error information (if any)
- Graph visualization (when graph metadata is present)
- Manual annotations for quality assessment
Filtering and Search
Find specific traces using:- Function ID - Filter by specific prompt or function
- Session ID - View all traces in a session
- User ID - See all activity for a specific user
- Time Range - Narrow results to specific periods
- Status - Filter by success, error, or specific finish reasons
- Model - View traces for specific LLM models
Integration
AgentMark works with any application that uses OpenTelemetry. For detailed setup instructions, see the Development Observability documentation.Best Practices
Use Meaningful IDs - Choose descriptive function IDs for easy filtering and debugging. Add Context - Include relevant metadata like user IDs, session IDs, and business context. Monitor Regularly - Check traces frequently to catch issues early. Set Up Alerts - Configure alerts for cost, latency, or error thresholds. Analyze Patterns - Use the dashboard’s filtering to identify trends and patterns.Next Steps
Sessions
Group related traces together
Alerts
Get notified of critical issues
Annotations
Manually label and score traces
Development Setup
Integrate observability in your app
Have Questions?
We’re here to help! Choose the best way to reach us:
- Join our Discord community for quick answers and discussions
- Email us at [email protected] for support
- Schedule an Enterprise Demo to learn about our business solutions