> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentmark.co/llms.txt
> Use this file to discover all available pages before exploring further.
# Datasets
> Create test datasets for your prompts
Datasets are JSONL files containing test cases to validate prompt behavior. Each line has an `input` (required) and an optional `expected_output`. You keep them alongside your prompts and run them directly from the CLI.
## Quick start
**1. Create a dataset file** (`agentmark/datasets/sentiment.jsonl`):
```jsonl theme={null}
{"input": {"text": "I love this!"}, "expected_output": "positive"}
{"input": {"text": "Terrible product"}, "expected_output": "negative"}
{"input": {"text": ""}}
```
**2. Link to your prompt** (frontmatter):
```mdx theme={null}
---
name: sentiment-classifier
test_settings:
dataset: ./datasets/sentiment.jsonl
---
Classify the sentiment
{props.text}
```
**3. Run experiments**:
```bash theme={null}
agentmark run-experiment agentmark/sentiment.prompt.mdx
```
## Dataset structure
Each line must be valid JSON:
* **`input`** (required) - Props passed to your prompt
* **`expected_output`** (optional) - Expected result for evaluation
**With expected output** (enables evaluations):
```jsonl theme={null}
{"input": {"text": "Great!", "category": "electronics"}, "expected_output": "positive"}
```
**Without expected output** (output-only mode):
```jsonl theme={null}
{"input": {"text": "Great!", "category": "electronics"}}
```
## What to test
**Common cases**:
```jsonl theme={null}
{"input": {"query": "What is AI?"}, "expected_output": "explanation"}
{"input": {"query": "Explain ML"}, "expected_output": "explanation"}
```
**Edge cases**:
```jsonl theme={null}
{"input": {"text": ""}, "expected_output": "error"}
{"input": {"text": "a"}, "expected_output": "too_short"}
{"input": {"text": "Lorem ipsum... [5000 chars]"}, "expected_output": "truncated"}
```
**Failure modes**:
```jsonl theme={null}
{"input": {"email": "invalid-email"}, "expected_output": "error: invalid email"}
{"input": {"amount": -100}, "expected_output": "error: amount must be positive"}
```
**Real-world data** - Use production data with identifying details removed when possible.
**LLM-assisted generation** - Use LLMs to generate test cases, but have humans verify outputs before using them.
## Expected-output types
**Strings** (classification):
```jsonl theme={null}
{"input": {"text": "sunny day"}, "expected_output": "positive"}
```
**Objects** (structured data):
```jsonl theme={null}
{"input": {"text": "John, john@example.com"}, "expected_output": {"name": "John", "email": "john@example.com"}}
```
**Flexible** (patterns, not exact matches):
```jsonl theme={null}
{"input": {"topic": "AI"}, "expected_output": "explanation containing: artificial intelligence"}
```
Your evaluation function validates flexible expectations.
## Dataset size
**Start small** (10-20 cases):
* 5-7 common scenarios
* 3-5 edge cases
* 2-3 failure modes
**Scale based on needs**:
* **Initial development**: 50-100 cases (recommended by [Confident AI](https://www.confident-ai.com/blog/evaluating-llm-systems-metrics-benchmarks-and-best-practices))
* **Statistical significance**: \~250 cases (for 95% confidence, 5% margin of error)
* **Production systems**: 100-300 cases minimum
* **High-stakes applications**: 300+ cases
Quality > quantity. Start with 50-100 high-quality cases, then grow based on statistical power analysis and real-world findings.
## Best practices
* One test case per line (valid JSONL)
* Use descriptive inputs that clearly show what the test case validates
* Version control datasets alongside prompts
* Avoid duplicates - each case should validate something unique
* Always anonymize data (never leak sensitive information)
## Advanced: held-out test sets
Create separate datasets to avoid over-fitting:
```text theme={null}
datasets/
├── development.jsonl # Use during iteration (60-70%)
├── validation.jsonl # Check progress periodically (15-20%)
└── held-out.jsonl # Final test before production (15-20%)
```
**Critical rules**:
* Never iterate on held-out data
* Don't peek at held-out results during development
* If you look at held-out results and make changes, create a new held-out set
**Example workflow**:
```text theme={null}
Week 1-2: Iterate on development set
├─ Test prompt v1 → 75% pass rate
└─ Test prompt v2 → 82% pass rate
Week 3: Check validation set
└─ Test prompt v2 → 79% pass rate (close to dev, good sign!)
Before deploy: Test held-out set
└─ Test prompt v3 → 81% pass rate → Deploy if meets requirements
```
## Advanced: statistical significance
**Sample size requirements** ([source](https://www.confident-ai.com/blog/evaluating-llm-systems-metrics-benchmarks-and-best-practices)):
* **Quick iteration**: 10-20 cases (directional feedback only)
* **Initial development**: 50-100 cases (industry standard)
* **Statistical rigor**: \~250 cases (95% confidence, 5% margin of error)
* **Production deployment**: 100-300 cases minimum
* **High-stakes systems**: 300+ cases
**Why size matters**: with 10 cases, one failure = 10% change. With 100 cases, one failure = 1% change. Research shows datasets with N ≤ 300 often overestimate performance.
**Confidence intervals** - Report uncertainty:
```text theme={null}
Pass rate: 85% (85 passed out of 100 tests)
Standard error: √(0.85 × 0.15 / 100) = 0.036
95% confidence interval: 85% ± 7% → [78%, 92%]
```
Report: "Pass rate: 85% \[CI: 78%-92%]"
Avoid: "Pass rate: 85%"
**Comparing prompts** - Use paired comparisons on same dataset:
```typescript theme={null}
// For each test case, record if new prompt performed better
const improvements = testCases.map(tc => {
const oldPassed = evaluateOld(tc);
const newPassed = evaluateNew(tc);
return newPassed && !oldPassed ? 1 : (oldPassed && !newPassed ? -1 : 0);
});
const netImprovement = improvements.reduce((a, b) => a + b, 0);
// netImprovement > 10 with 100 cases suggests real improvement
```
**Power analysis** - Determine how many samples you need before creating your dataset. It answers how many test cases reliably detect a meaningful improvement. The smaller the improvement you want to catch, the more cases you need:
| Minimum detectable difference | Required sample size (per group) |
| ----------------------------- | -------------------------------- |
| 10% (for example, 80% → 90%) | \~100 samples |
| 5% (for example, 80% → 85%) | \~400 samples |
| 2% (for example, 80% → 82%) | \~2,500 samples |
| 1% (for example, 80% → 81%) | \~10,000 samples |
If you only have 50 test cases, you can only reliably detect large improvements (>15%); smaller improvements look like noise. Plan your dataset size around the smallest improvement that matters to your application before you start collecting cases.
## Next steps
Write evaluation functions
Test your datasets
Learn testing concepts
Have questions?
Reach out any time: