Human annotation

Human annotation adds manual scores, labels, and feedback to your traces. Use it to evaluate subjective quality, flag edge cases, curate training datasets, and calibrate your automated evals. AgentMark supports two annotation workflows:

• Inline annotation: score a single trace directly from the trace drawer
• Annotation queues: batch traces into structured review queues with assignment, progress tracking, and multi-reviewer support

The animation shows a reviewer moving through a queue: list view → queue detail with assigned items → side-by-side review panel (trace on the left, score controls on the right) → next item.

​

When to use human annotation

​

Score types

Score configs define what reviewers score on. They’re declared as JSON in your agentmark.json under the top-level scores field and synced to AgentMark Cloud through the deployment pipeline. When creating a queue, you select which score configs to include. The same three score types power both automated evals and human annotation, so Declare score configs documents the JSON schema once. How each type appears to a reviewer:

• Boolean (pass/fail): the reviewer clicks Pass or Fail, saved as 1 or 0. Best for clear-cut criteria.
• Numeric (scale): the reviewer enters a value within the config’s min/max range. Best for graded assessments.
• Categorical (labels): the reviewer picks one option from a dropdown; each option is a {label, value} pair, and value is the recorded score. Best for classification.

Every score type includes an optional reason field where the reviewer can explain their judgment.

​

Inline annotation

Add a score to any trace directly from the trace drawer, with no queue required. The Evaluations tab lists every score attached to the selected span: both automated eval results and human annotations. Each row shows the score name, label/value, and reason; annotations carry an annotation badge to distinguish them from automated scores. Inline annotations appear alongside automated eval scores, distinguished by an “annotation” badge.

​

Annotation queues

For batch review, use annotation queues. Queues let you organize items, assign reviewers, track progress, and require multiple independent reviews. The Review Queues page lists every queue in the app with its name, status, progress, and creation time. Filter tabs narrow by status (All, Active, Completed, Archived), and an Assigned to me toggle restricts the list to queues with items assigned to you. A sidebar badge surfaces the total number of pending items across all active queues.

​

Create a queue

Navigate to Review Queues in the sidebar and click Create Queue. The Create Review Queue dialog takes a name, optional description and annotator instructions, the number of independent reviews required per item, and the set of score configs to show to reviewers. You can select a default dataset to pre-fill the Add to Dataset action during review.

​

Add items

​

Queue detail

Click any queue to see its items, progress, and reviewer assignments. The queue detail view lists every item with status (pending, completed, skipped), resource type (trace, span, or session), and assignee. Filter tabs across the top narrow by status, and a multi-reviewer badge shows how many independent reviews remain per item. Filter items using the tabs: Click the assign icon on any row to assign it to a team member. Use the three-dot menu to archive a queue when review is complete.

​

Review workflow

Click Start Review to begin. The review view splits into two panels. Left panel (trace content):

• Header showing the trace name
• Root span input and output, falling back to trace-level data
• A Spans (N) accordion, collapsed by default; expand it and click any span to see its I/O
• For session items, a conversation timeline showing all turns

Right panel (annotation):

• Annotator instructions (collapsible, from queue config)
• Existing annotations: a read-only list of annotation scores already saved on this resource
• Score controls for each configured dimension
• Add to Dataset section with auto-extracted input and an editable expected output

The review footer shows the same hints: Enter — complete · p/f — pass/fail · 1-9 — category. The review view ignores the letter and digit shortcuts while you’re typing in a text field.

​

Multi-reviewer

When you set reviewers required above 1, each reviewer annotates independently:

• The review header shows a progress badge (for example, “0/2 reviewed”) tracking how many reviewers have completed their assessment
• Each reviewer fills out their own fresh annotation form. Scores saved by earlier reviewers appear in the read-only Existing annotations list, so reviews are independent but not blind
• The queue marks an item complete only once it collects the required number of independent reviews
• The /next endpoint automatically skips items the current reviewer has already reviewed, so each reviewer only sees items they haven’t scored yet

​

Resource types

Queues support three item types:

​

Programmatic queue management

Annotation queues are fully exposed on the public REST API at /v1/annotation-queues (Cloud only; the local dev server returns 404). CI pipelines can create queues, enqueue traces, and, via the /reviews endpoint, submit annotations through the same path a human reviewer clicks in the Dashboard. The review-submission endpoint is what takes this past plain queue CRUD: posting { "status": "completed" } records the authenticated user as a reviewer, and once the review count meets the queue’s reviewers_required threshold, the item auto-advances to completed. That lets an LLM-as-judge pipeline submit annotations that count toward the same threshold as human reviewers. API keys need the annotation_queue.read, annotation_queue.write, annotation_queue.delete, and annotation_queue.review permissions (split so you can grant CI pipelines review without queue-CRUD access). See the API reference for full endpoint schemas.

​

End-to-end example: dataset curation

A common workflow is using annotation queues to curate high-quality datasets from production traces.

​

Human annotation vs automated evals

Use both together. They serve different purposes. Automated evals catch regressions at scale. Human annotations handle the cases machines can’t judge, and they give you the ground truth to calibrate your automated scorers against.

​

Related