Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.agentmark.co/llms.txt

Use this file to discover all available pages before exploring further.

The AgentMark CLI (@agentmark-ai/cli) provides tools for developing, testing, and building your AI prompts.

Installation

# Use directly with npx (no global install needed)
npx agentmark <command>

# Or install globally
npm install -g @agentmark-ai/cli
agentmark <command>

Environment variables

The CLI automatically loads environment variables from a .env file in the current working directory. This happens before any command execution, so you can store API keys and configuration there.
# .env
OPENAI_API_KEY=sk-...
AGENTMARK_API_KEY=...
AGENTMARK_APP_ID=...
See Environment variables for the full list.

Update notifications

The CLI checks for updates asynchronously when you run commands. If a newer version is available, you’ll see a notification after your command completes. This check is non-blocking and won’t slow down your workflow. To disable update checks, set the environment variable:
export AGENTMARK_NO_UPDATE_NOTIFIER=1

Commands

agentmark dev

Start the local development environment with API server, webhook server, and local dev server UI. With --remote, the CLI also connects to AgentMark Cloud via WebSocket Connect, enabling you to run prompts and experiments from the Dashboard.
npx agentmark dev [options]
Options:
OptionDescriptionDefault
--api-port <number>API server port9418
--webhook-port <number>Webhook server port9417
--app-port <number>Local dev server UI port3000
-r, --remoteConnect to AgentMark Cloud via WebSocketfalse
--no-forwardDisable trace forwarding (only relevant with --remote)forwarding is on by default with --remote
--no-uiSkip the local dev server UI (API + webhook only). For CI, headless, or test contexts where the Next.js UI isn’t needed.UI starts by default
Project detection: The dev server automatically detects your project type:
  • TypeScript projects: Looks for agentmark.client.ts in the project root
  • Python projects: Looks for pyproject.toml, agentmark_client.py, or .agentmark/dev_server.py
Dev server entry points (TypeScript): The CLI looks for dev server files in this order:
  1. dev-server.ts - Custom override (project root)
  2. dev-entry.ts - Default location (project root)
  3. .agentmark/dev-entry.ts - Legacy location
Python virtual environment: For Python projects, the CLI automatically detects and uses virtual environments in .venv/ or venv/ directories. Remote connection: When using --remote, the CLI establishes a WebSocket connection to the AgentMark Cloud gateway. This enables running prompts and experiments directly from the Dashboard. The CLI handles authentication and app linking automatically:
  1. If not logged in, opens a browser for OAuth login.
  2. If the project is not linked, prompts you to select an app.
  3. Opens a WebSocket connection with automatic heartbeat and reconnection.
See Connect for details on how WebSocket Connect works. Example:
# Local-only development
npx agentmark dev

# Connect to AgentMark Cloud (recommended)
npx agentmark dev --remote

# Custom ports with Cloud connection
npx agentmark dev --api-port 9500 --webhook-port 9501 --remote

agentmark login

Authenticate with AgentMark Cloud via browser OAuth. The CLI opens your default browser to complete the login flow, then stores credentials locally for subsequent commands.
npx agentmark login
Stored credentials are used automatically by agentmark dev --remote and agentmark link. You can override stored credentials with the --api-key flag or AGENTMARK_API_KEY environment variable on any command.

agentmark logout

Clear stored CLI authentication credentials and revoke any dev API keys created during agentmark link.
npx agentmark logout [options]
Options:
OptionDescriptionDefault
--base-url <url>AgentMark Cloud URLhttps://app.agentmark.co

Link your local project to an app in AgentMark Cloud. The CLI prompts you to select an app from your account (or use --app-id to skip the prompt), then stores the app ID and a dev API key in your local project configuration.
npx agentmark link [options]
Options:
OptionDescriptionDefault
--app-id <uuid>App ID to link (skips interactive selection)
--base-url <url>AgentMark Cloud URLhttps://app.agentmark.co
After linking, commands like agentmark dev --remote automatically use the linked app without requiring --app-id or AGENTMARK_APP_ID.
agentmark dev --remote runs agentmark link automatically if your project is not yet linked. You only need to run agentmark link manually if you want to change which app your project is linked to.

agentmark run-prompt

Run a single prompt file with test props.
npx agentmark run-prompt <filepath> [options]
Arguments:
ArgumentDescription
filepathPath to the .prompt.mdx file
Options:
OptionDescriptionDefault
--server <url>Webhook server URLhttp://localhost:9417
--props <json>Props as JSON string-
--props-file <path>Path to JSON or YAML file containing props-
Example:
# Run with inline props
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --props '{"name": "Alice"}'

# Run with props from file
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --props-file ./test-props.yaml

# Run against a remote server
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --server https://my-webhook.example.com

agentmark run-experiment

Run an experiment against its dataset, with evaluations by default.
npx agentmark run-experiment <filepath> [options]
Arguments:
ArgumentDescription
filepathPath to the .prompt.mdx file with test configuration
Options:
OptionDescriptionDefault
--server <url>Webhook server URLhttp://localhost:9417
--skip-evalSkip running evals even if they existfalse
--format <format>Output format: table, csv, json, jsonl, junittable
--threshold <percent>Fail if pass rate is below threshold (0-100)-
--sample <percent>Sample N% of dataset rows randomly (1-100)-
--rows <spec>Select specific rows by index/range (e.g. 0,3-5,9)-
--split <spec>Train/test split (e.g. train:80, test:80)-
--seed <number>Seed for reproducible sampling/splitting-
--truncate <chars>Truncate table cell content to N chars (0 = no limit)1000
Example:
# Run experiment with table output
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx

# Run experiment with JSON output, skip evals
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format json --skip-eval

# Run with CI threshold (fails if <80% pass rate)
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --threshold 80

# Emit JUnit XML for CI gating (GitHub Actions, GitLab, Jenkins, etc.)
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format junit > results.xml

agentmark generate-types

Generate TypeScript type definitions from your prompt schemas.
npx agentmark generate-types [options]
Options:
OptionDescriptionDefault
-l, --language <language>Target languagetypescript
--local <port>Local server port to fetch prompts from-
--root-dir <path>Root directory containing agentmark files-
Output: The command outputs TypeScript definitions to stdout. Redirect to a file:
npx agentmark generate-types --root-dir ./agentmark > agentmark.types.ts
Generated types include:
  • Input types based on input_schema
  • Output types based on the model’s schema
  • A mapping of prompt paths to their respective types
  • Tool argument types
Example:
# Generate from local files
npx agentmark generate-types --root-dir ./agentmark > agentmark.types.ts

# Generate from local dev server
npx agentmark generate-types --local 9418 > agentmark.types.ts
See Type safety for usage examples.

agentmark generate-schema

Generate a JSON Schema file for .prompt.mdx frontmatter. This enables IDE validation (squiggles) for fields like model_name in your prompt files.
npx agentmark generate-schema [options]
Options:
OptionDescriptionDefault
-o, --out <directory>Output directory.agentmark
Example:
npx agentmark generate-schema
npx agentmark generate-schema --out ./schemas

agentmark build

Build prompts into pre-compiled JSON files for static loading with FileLoader.
npx agentmark build [options]
Options:
OptionDescriptionDefault
-o, --out <directory>Output directorydist/agentmark
Requirements:
  • An agentmark.json config file must exist in the current directory
  • Prompts are read from the directory specified by agentmarkPath in the config
Output Structure:
dist/agentmark/
  manifest.json           # Build manifest with all prompts
  greeting.prompt.json    # Compiled prompt (mirrors source structure)
  nested/
    helper.prompt.json
Example:
# Build with default output directory
npx agentmark build

# Build to custom directory
npx agentmark build --out ./build/prompts
See Loaders for using built prompts with FileLoader.

agentmark pull-models

Interactive command to pull and configure models from a provider.
npx agentmark pull-models
This command opens an interactive prompt to:
  1. Select a model provider
  2. Choose models to enable
  3. Update your local configuration

agentmark api

Access the AgentMark gateway API from the command line. Subcommands are auto-generated from the gateway’s OpenAPI spec, so the full list is only available after the local dev server is reachable (or with --remote). npx agentmark api --help run cold (before agentmark dev is up) shows only the top-level usage.
npx agentmark api [options]
# Then run --help for a specific resource after the server is up:
npx agentmark api traces --help
Options:
OptionDescriptionDefault
--remoteTarget AgentMark Cloud gateway instead of the local dev serverlocal (localhost:9418)
--refreshForce re-fetch of the OpenAPI spec (cached for 24 hours)false
By default, agentmark api targets your local dev server. Use --remote to target the AgentMark Cloud gateway (requires agentmark login and agentmark link). Available resources:
ResourceActionsDescription
traceslist, getList and retrieve traces
sessionslist, getList and retrieve sessions
spanslistList spans across traces
scoreslistList scores for spans and traces
metricsgetRetrieve aggregated metrics
datasetslist, getList and retrieve datasets
experimentslist, getList and retrieve experiment results
promptslist, getList and retrieve prompt execution logs
runslist, getList and retrieve individual runs within experiments
capabilitiesgetCheck which features the server supports
healthgetCheck gateway health
npx agentmark api __schema prints the full resource/action tree from the live OpenAPI spec. This command only works once the local dev server is running, or with --remote + agentmark login + agentmark link.
Self-documenting help: Every resource and action has built-in help. Use --help to see available actions and their parameters (after the server is reachable):
# List all actions for a resource
npx agentmark api traces --help

# See parameters for a specific action
npx agentmark api traces list --help
Example:
# List recent traces from the local dev server
npx agentmark api traces list --limit 10

# List traces from the AgentMark Cloud gateway
npx agentmark api traces list --remote --limit 10

# Get a specific trace by ID
npx agentmark api traces get <traceId>

# List sessions from AgentMark Cloud
npx agentmark api sessions list --remote --limit 5

# List scores for review
npx agentmark api scores list --remote

# List datasets
npx agentmark api datasets list

# List experiment results
npx agentmark api experiments list --limit 5

# List prompt execution logs
npx agentmark api prompts list --limit 10

# Check which features are available
npx agentmark api capabilities get
The OpenAPI spec is cached locally for 24 hours. If endpoints were recently added, pass --refresh to pick up changes: npx agentmark api --refresh traces list.

Configuration files

agentmark.json

Project configuration file in your project root. See Project config for the full schema.
{
  "agentmarkPath": ".",
  "version": "2.0.0",
  "mdxVersion": "1.0"
}
FieldDescription
agentmarkPathBase path for agentmark files (contains the agentmark/ directory) — use "." for the canonical layout, not "/"
versionConfiguration version
mdxVersionMDX syntax version

.agentmark/dev-config.json

Auto-generated local development configuration (gitignored):
{
  "createdAt": "2026-04-15T10:30:00.000Z",
  "appPort": 3000,
  "forwarding": {
    "appId": "app_xxxxx",
    "appName": "my-app",
    "orgName": "my-org",
    "tenantId": "tenant_xxxxx",
    "apiKey": "am_dev_xxxxx",
    "apiKeyId": "key_xxxxx",
    "expiresAt": "2026-05-15T10:30:00.000Z",
    "baseUrl": "https://app.agentmark.co"
  }
}
This file stores:
  • appPort — local dev server UI port (updated when dev server starts).
  • forwarding — linked app metadata (app ID, dev API key, token expiry) used by agentmark dev --remote. Populated by agentmark link and cleared by agentmark logout.
The configuration expires after 30 days and is regenerated on the next agentmark link.

Have Questions?

We’re here to help! Choose the best way to reach us: