Human annotation

Cloud feature. Annotations are available in the AgentMark Dashboard.

Human annotation adds manual scores, labels, and feedback to your traces. Use it to evaluate subjective quality, flag edge cases, curate training datasets, and calibrate your automated evals. AgentMark supports two annotation workflows:

Inline annotation — score a single trace directly from the trace drawer
Annotation queues — batch traces into structured review queues with assignment, progress tracking, and multi-reviewer support

Animated walkthrough of the annotation queue review flow: queue list, detail view, and review panel

When to use human annotation

Use case	Example	Workflow
Quality audits	Review a sample of production traces for correctness and tone	Create a queue, add traces, assign to domain experts
Edge case triage	Flag and investigate unexpected model behavior	Inline annotation from the trace drawer
Dataset curation	Build high-quality test datasets from real production data	Review in queue, save passing traces to a dataset
Calibrate automated evals	Align your LLM-as-judge scorers with human judgment	Score the same traces manually that your evals score, compare results
Multi-reviewer consensus	Get independent assessments from multiple team members	Set reviewers required > 1 on a queue

Score types

Score configs define what reviewers score on. They are declared in your agentmark.json file under the scores field and persisted to the platform database on agentmark deploy. When creating a queue, you select which score configs to include.

Score configs must be deployed before you can create a queue. Run agentmark deploy after adding scores to your agentmark.json. Once deployed, score configs are always available in the dashboard — no worker dependency required. See Project configuration for the scores schema and Evaluations for adding automated eval functions.

Boolean (pass/fail)
Numeric (scale)
Categorical (labels)

A binary judgment. The reviewer clicks Pass or Fail.

Name: factual_accuracy
Type: boolean

Saved as score 1 (pass) or 0 (fail). Best for clear-cut criteria: “Is the response factually correct?”

A number within a configurable range. The reviewer enters a value.

Name: helpfulness
Type: numeric
Min: 1, Max: 5

Best for graded assessments: “How helpful is this response on a 1-5 scale?”

A dropdown of predefined options. The reviewer picks one.

Name: tone
Type: categorical
Categories:
  - { label: "professional", value: 1 }
  - { label: "casual", value: 0.5 }
  - { label: "inappropriate", value: 0 }

Each category is a {label, value} pair. The label is shown in the reviewer dropdown, and the value is the numeric score recorded when selected.Best for classification: “What is the tone of this response?”

Every score type includes an optional reason field where the reviewer can explain their judgment.

Inline annotation

Add a score to any trace directly from the trace drawer — no queue required.

Open a trace

Navigate to Traces and click on any trace to open the detail drawer.

Select a span

Choose the span you want to annotate from the trace tree.

Go to the evaluations tab

Click the Evaluations tab in the drawer.

Add annotation

Click Add annotation, fill in the name, label, score, and reason, then click Save.

Inline annotations appear alongside automated eval scores, distinguished by an “annotation” badge.

Annotation queues

For batch review, use annotation queues. Queues let you organize items, assign reviewers, track progress, and require multiple independent reviews.

Review queues list showing active queues with progress bars and pending badge in sidebar

Create a queue

Navigate to Review Queues in the sidebar and click Create Queue.

Create review queue dialog with name, instructions, reviewers required, and score config fields

Field	Required	Description
Name	Yes	Descriptive name for the review batch
Description	No	Context for what this queue covers
Instructions for annotators	No	Guidance shown during review (e.g., “Mark PASS if factually correct and professional”)
Reviewers required	Yes	Independent reviews needed per item (default: 1)
Score configs	Yes	Which scoring dimensions to show during review
Default dataset	No	Pre-selects a dataset for the “Save to dataset” action

Add items

Bulk from traces
Individual spans
From experiments
Via API

Select traces

Go to the Traces page and select traces using the checkboxes.

Add to queue

Click Add to Queue in the bulk actions bar, choose a queue, and confirm.

Add items programmatically using the REST API:

curl -X POST /api/annotation-queues/QUEUE_ID/items?appId=YOUR_APP_ID \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"resource_id": "trace-abc-123"},
      {"resource_id": "span-def-456", "resource_type": "span"},
      {"resource_id": "session-789", "resource_type": "session"}
    ]
  }'

Queue detail

Click any queue to see its items, progress, and reviewer assignments.

Filter items using the tabs:

Tab	Shows
All	Every item in the queue
Pending	Items waiting for review
Completed	Reviewed or skipped items
Assigned to me	Items assigned to you

Click the assign icon on any row to assign it to a team member. Use the three-dot menu to archive a queue when review is complete.

Review workflow

Click Start Review to begin. The review view splits into two panels. Left panel — trace content:

Metadata bar with trace name, latency, cost, tokens, and model
Root span input/output formatted as JSON
Expandable spans tree — click any span to see its I/O
For session items, a conversation timeline showing all turns

Right panel — annotation:

Annotator instructions (collapsible, from queue config)
Score controls for each configured dimension
Prior annotations on this resource (read-only)
Save to dataset section with auto-extracted I/O

Action	Shortcut	What it does
Complete + Next	`Enter`	Save scores, mark complete, advance
Skip	—	Mark as skipped, advance
Back	—	Return to queue detail

Multi-reviewer

When reviewers required is set above 1, each reviewer annotates independently:

The review header shows a progress badge (e.g., “0/2 reviewed”) tracking how many reviewers have completed their assessment
Each reviewer sees their own fresh annotation form — they don’t see other reviewers’ scores while annotating
An item is only marked complete when the required number of independent reviews is reached
The /next endpoint automatically skips items the current reviewer has already reviewed, so each reviewer only sees items they haven’t scored yet

Use multi-reviewer for high-stakes evaluations like safety reviews or fine-tuning dataset curation where a single reviewer’s judgment isn’t sufficient.

Resource types

Queues support three item types:

Type	When to use	What the reviewer sees
Trace	Review a complete request	Full trace with expandable per-span I/O
Span	Review a single LLM call or tool invocation	Individual span content
Session	Review a multi-turn conversation	Conversation timeline across traces

Programmatic queue management

All queue operations are available via the REST API. Use these to integrate annotation into your CI pipelines or automated workflows.

Create a queue

curl -X POST /api/annotation-queues?appId=YOUR_APP_ID \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Weekly quality review",
    "score_config_names": ["accuracy", "tone"],
    "reviewers_required": 2,
    "instructions": "Review for factual accuracy and professional tone."
  }'

const response = await fetch(`/api/annotation-queues?appId=${appId}`, {
  method: 'POST',
  headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({
    name: 'Weekly quality review',
    score_config_names: ['accuracy', 'tone'],
    reviewers_required: 2,
    instructions: 'Review for factual accuracy and professional tone.',
  }),
});
const { queue } = await response.json();

Add items to a queue

curl -X POST /api/annotation-queues/QUEUE_ID/items?appId=YOUR_APP_ID \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"resource_id": "trace-abc-123"},
      {"resource_id": "span-def-456", "resource_type": "span"}
    ]
  }'

await fetch(`/api/annotation-queues/${queueId}/items?appId=${appId}`, {
  method: 'POST',
  headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({
    items: traceIds.map(id => ({ resource_id: id })),
  }),
});

Get next item for review

curl /api/annotation-queues/QUEUE_ID/next?appId=YOUR_APP_ID \
  -H "Authorization: Bearer YOUR_API_KEY"

# Returns 200 with item, or 204 if queue is fully reviewed

const res = await fetch(`/api/annotation-queues/${queueId}/next?appId=${appId}`, {
  headers: { 'Authorization': `Bearer ${apiKey}` },
});

if (res.status === 204) {
  console.log('All items reviewed');
} else {
  const { item } = await res.json();
  console.log(`Next item: ${item.resourceId} (${item.resourceType})`);
}

Update item status or assignment

# Assign an item to a reviewer
curl -X PATCH /api/annotation-queues/QUEUE_ID/items/ITEM_ID?appId=YOUR_APP_ID \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"assigned_to": "USER_UUID"}'

# Mark an item as completed
curl -X PATCH /api/annotation-queues/QUEUE_ID/items/ITEM_ID?appId=YOUR_APP_ID \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"status": "completed"}'

End-to-end example: dataset curation

A common workflow is using annotation queues to curate high-quality datasets from production traces.

Create a queue

Create a queue with a boolean score config (e.g., dataset_quality) and set the default dataset to your target dataset.

Add production traces

Go to Traces, filter to interesting traces (errors, low automated scores, specific prompts), select them, and add to the queue.

Review and score

Click Start Review. For each trace, read the I/O, mark Pass or Fail, and optionally edit the input/output before saving to the dataset.

Save to dataset

Expand the Save to dataset section, verify the auto-extracted fields, and click Save. The default dataset is pre-selected.

Use in experiments

Run experiments against the curated dataset to validate prompt changes against human-verified examples.