CLI reference

The AgentMark CLI (@agentmark-ai/cli) provides tools for developing, testing, and building your AI prompts.

Installation

# Use directly with npx (no global install needed)
npx agentmark <command>

# Or install globally
npm install -g @agentmark-ai/cli
agentmark <command>

Environment variables

The CLI automatically loads environment variables from a .env file in the current working directory. This happens before any command execution, so you can store API keys and configuration there.

# .env
OPENAI_API_KEY=sk-...
AGENTMARK_API_KEY=...
AGENTMARK_APP_ID=...

See Environment variables for the full list.

Update notifications

The CLI checks for updates asynchronously when you run commands. If a newer version is available, you’ll see a notification after your command completes. This check is non-blocking and won’t slow down your workflow. To disable update checks, set the environment variable:

export AGENTMARK_NO_UPDATE_NOTIFIER=1

Commands

agentmark dev

Start the local development environment with API server, webhook server, and local dev server UI. With --remote, the CLI also connects to AgentMark Cloud via WebSocket Connect, enabling you to run prompts and experiments from the Dashboard.

npx agentmark dev [options]

Options:

Option	Description	Default
`--api-port <number>`	API server port	`9418`
`--webhook-port <number>`	Webhook server port	`9417`
`--app-port <number>`	Local dev server UI port	`3000`
`-r, --remote`	Connect to AgentMark Cloud via WebSocket	`false`
`--no-forward`	Disable trace forwarding (only relevant with `--remote`)	forwarding is on by default with `--remote`
`--no-ui`	Skip the local dev server UI (API + webhook only). For CI, headless, or test contexts where the Next.js UI isn’t needed.	UI starts by default

Project detection: The dev server automatically detects your project type:

TypeScript projects: Looks for agentmark.client.ts in the project root
Python projects: Looks for pyproject.toml, agentmark_client.py, or .agentmark/dev_server.py

Dev server entry points (TypeScript): The CLI looks for dev server files in this order:

dev-server.ts - Custom override (project root)
dev-entry.ts - Default location (project root)
.agentmark/dev-entry.ts - Legacy location

Python virtual environment: For Python projects, the CLI automatically detects and uses virtual environments in .venv/ or venv/ directories. Remote connection: When using --remote, the CLI establishes a WebSocket connection to the AgentMark Cloud gateway. This enables running prompts and experiments directly from the Dashboard. The CLI handles authentication and app linking automatically:

If not logged in, opens a browser for OAuth login.
If the project is not linked, prompts you to select an app.
Opens a WebSocket connection with automatic heartbeat and reconnection.

See Connect for details on how WebSocket Connect works. Example:

# Local-only development
npx agentmark dev

# Connect to AgentMark Cloud (recommended)
npx agentmark dev --remote

# Custom ports with Cloud connection
npx agentmark dev --api-port 9500 --webhook-port 9501 --remote

Authenticate with AgentMark Cloud via browser OAuth. The CLI opens your default browser to complete the login flow, then stores credentials locally for subsequent commands.

npx agentmark login

Stored credentials are used automatically by agentmark dev --remote and agentmark link. You can override stored credentials with the --api-key flag or AGENTMARK_API_KEY environment variable on any command.

agentmark logout

Clear stored CLI authentication credentials and revoke any dev API keys created during agentmark link.

npx agentmark logout [options]

Options:

Option	Description	Default
`--base-url <url>`	AgentMark Cloud URL	`https://app.agentmark.co`

agentmark link

Link your local project to an app in AgentMark Cloud. The CLI prompts you to select an app from your account (or use --app-id to skip the prompt), then stores the app ID and a dev API key in your local project configuration.

npx agentmark link [options]

Options:

Option	Description	Default
`--app-id <uuid>`	App ID to link (skips interactive selection)	—
`--base-url <url>`	AgentMark Cloud URL	`https://app.agentmark.co`

After linking, commands like agentmark dev --remote automatically use the linked app without requiring --app-id or AGENTMARK_APP_ID.

agentmark dev --remote runs agentmark link automatically if your project is not yet linked. You only need to run agentmark link manually if you want to change which app your project is linked to.

agentmark run-prompt

Run a single prompt file with test props.

npx agentmark run-prompt <filepath> [options]

Arguments:

Argument	Description
`filepath`	Path to the `.prompt.mdx` file

Options:

Option	Description	Default
`--server <url>`	Webhook server URL	`http://localhost:9417`
`--props <json>`	Props as JSON string	-
`--props-file <path>`	Path to JSON or YAML file containing props	-

Example:

# Run with inline props
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --props '{"name": "Alice"}'

# Run with props from file
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --props-file ./test-props.yaml

# Run against a remote server
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --server https://my-webhook.example.com

agentmark run-experiment

Run an experiment against its dataset, with evaluations by default.

npx agentmark run-experiment <filepath> [options]

Arguments:

Argument	Description
`filepath`	Path to the `.prompt.mdx` file with test configuration

Options:

Option	Description	Default
`--server <url>`	Webhook server URL	`http://localhost:9417`
`--skip-eval`	Skip running evals even if they exist	`false`
`--format <format>`	Output format: `table`, `csv`, `json`, `jsonl`, `junit`	`table`
`--threshold <percent>`	Fail if pass rate is below threshold (0-100)	-
`--sample <percent>`	Sample N% of dataset rows randomly (1-100)	-
`--rows <spec>`	Select specific rows by index/range (e.g. `0,3-5,9`)	-
`--split <spec>`	Train/test split (e.g. `train:80`, `test:80`)	-
`--seed <number>`	Seed for reproducible sampling/splitting	-
`--truncate <chars>`	Truncate table cell content to N chars (0 = no limit)	`1000`

Example:

# Run experiment with table output
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx

# Run experiment with JSON output, skip evals
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format json --skip-eval

# Run with CI threshold (fails if <80% pass rate)
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --threshold 80

# Emit JUnit XML for CI gating (GitHub Actions, GitLab, Jenkins, etc.)
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format junit > results.xml

agentmark generate-types

Generate TypeScript type definitions from your prompt schemas.

npx agentmark generate-types [options]

Options:

Option	Description	Default
`-l, --language <language>`	Target language	`typescript`
`--local <port>`	Local server port to fetch prompts from	-
`--root-dir <path>`	Root directory containing agentmark files	-

Output: The command outputs TypeScript definitions to stdout. Redirect to a file:

npx agentmark generate-types --root-dir ./agentmark > agentmark.types.ts

Generated types include:

Input types based on input_schema
Output types based on the model’s schema
A mapping of prompt paths to their respective types
Tool argument types

Example:

# Generate from local files
npx agentmark generate-types --root-dir ./agentmark > agentmark.types.ts

# Generate from local dev server
npx agentmark generate-types --local 9418 > agentmark.types.ts

See Type safety for usage examples.

agentmark generate-schema

Generate a JSON Schema file for .prompt.mdx frontmatter. This enables IDE validation (squiggles) for fields like model_name in your prompt files.

npx agentmark generate-schema [options]

Options:

Option	Description	Default
`-o, --out <directory>`	Output directory	`.agentmark`

Example:

npx agentmark generate-schema
npx agentmark generate-schema --out ./schemas

agentmark build

Build prompts into pre-compiled JSON files for static loading with FileLoader.

npx agentmark build [options]

Options:

Option	Description	Default
`-o, --out <directory>`	Output directory	`dist/agentmark`

Requirements:

An agentmark.json config file must exist in the current directory
Prompts are read from the directory specified by agentmarkPath in the config

Output Structure:

dist/agentmark/
  manifest.json           # Build manifest with all prompts
  greeting.prompt.json    # Compiled prompt (mirrors source structure)
  nested/
    helper.prompt.json

Example:

# Build with default output directory
npx agentmark build

# Build to custom directory
npx agentmark build --out ./build/prompts

See Loaders for using built prompts with FileLoader.

agentmark pull-models

Interactive command to pull and configure models from a provider.

npx agentmark pull-models

This command opens an interactive prompt to:

Select a model provider
Choose models to enable
Update your local configuration

agentmark api

Access the AgentMark gateway API from the command line. Subcommands are auto-generated from the gateway’s OpenAPI spec, so the full list is only available after the local dev server is reachable (or with --remote). npx agentmark api --help run cold (before agentmark dev is up) shows only the top-level usage.

npx agentmark api [options]
# Then run --help for a specific resource after the server is up:
npx agentmark api traces --help

Options:

Option	Description	Default
`--remote`	Target AgentMark Cloud gateway instead of the local dev server	local (`localhost:9418`)
`--refresh`	Force re-fetch of the OpenAPI spec (cached for 24 hours)	`false`

By default, agentmark api targets your local dev server. Use --remote to target the AgentMark Cloud gateway (requires agentmark login and agentmark link). Available resources:

Resource	Actions	Description
`traces`	`list`, `get`	List and retrieve traces
`sessions`	`list`, `get`	List and retrieve sessions
`spans`	`list`	List spans across traces
`scores`	`list`	List scores for spans and traces
`metrics`	`get`	Retrieve aggregated metrics
`datasets`	`list`, `get`	List and retrieve datasets
`experiments`	`list`, `get`	List and retrieve experiment results
`prompts`	`list`, `get`	List and retrieve prompt execution logs
`runs`	`list`, `get`	List and retrieve individual runs within experiments
`capabilities`	`get`	Check which features the server supports
`health`	`get`	Check gateway health

npx agentmark api __schema prints the full resource/action tree from the live OpenAPI spec. This command only works once the local dev server is running, or with --remote + agentmark login + agentmark link.

Self-documenting help: Every resource and action has built-in help. Use --help to see available actions and their parameters (after the server is reachable):

# List all actions for a resource
npx agentmark api traces --help

# See parameters for a specific action
npx agentmark api traces list --help

Example:

# List recent traces from the local dev server
npx agentmark api traces list --limit 10

# List traces from the AgentMark Cloud gateway
npx agentmark api traces list --remote --limit 10

# Get a specific trace by ID
npx agentmark api traces get <traceId>

# List sessions from AgentMark Cloud
npx agentmark api sessions list --remote --limit 5

# List scores for review
npx agentmark api scores list --remote

# List datasets
npx agentmark api datasets list

# List experiment results
npx agentmark api experiments list --limit 5

# List prompt execution logs
npx agentmark api prompts list --limit 10

# Check which features are available
npx agentmark api capabilities get

The OpenAPI spec is cached locally for 24 hours. If endpoints were recently added, pass --refresh to pick up changes: npx agentmark api --refresh traces list.

Configuration files

`agentmark.json`

Project configuration file in your project root. See Project config for the full schema.

{
  "agentmarkPath": ".",
  "version": "2.0.0",
  "mdxVersion": "1.0"
}

Field	Description
`agentmarkPath`	Base path for agentmark files (contains the `agentmark/` directory) — use `"."` for the canonical layout, not `"/"`
`version`	Configuration version
`mdxVersion`	MDX syntax version

`.agentmark/dev-config.json`

Auto-generated local development configuration (gitignored):

{
  "createdAt": "2026-04-15T10:30:00.000Z",
  "appPort": 3000,
  "forwarding": {
    "appId": "app_xxxxx",
    "appName": "my-app",
    "orgName": "my-org",
    "tenantId": "tenant_xxxxx",
    "apiKey": "am_dev_xxxxx",
    "apiKeyId": "key_xxxxx",
    "expiresAt": "2026-05-15T10:30:00.000Z",
    "baseUrl": "https://app.agentmark.co"
  }
}

This file stores:

appPort — local dev server UI port (updated when dev server starts).
forwarding — linked app metadata (app ID, dev API key, token expiry) used by agentmark dev --remote. Populated by agentmark link and cleared by agentmark logout.

The configuration expires after 30 days and is regenerated on the next agentmark link.

Have Questions?

We’re here to help! Choose the best way to reach us:

Email us at hello@agentmark.co for support
Schedule an Enterprise Demo to learn about our business solutions

TypeScript SDK

Python SDK

CLI

Developer Tools

Help

Installation

Environment variables

Update notifications

Commands

agentmark dev

agentmark logout

agentmark link

agentmark run-prompt

agentmark run-experiment

agentmark generate-types

agentmark generate-schema

agentmark build

agentmark pull-models

agentmark api

Configuration files

`agentmark.json`

`.agentmark/dev-config.json`

Have Questions?

TypeScript SDK

Python SDK

CLI

Developer Tools

Help

Documentation Index

​Installation

​Environment variables

​Update notifications

​Commands

​agentmark dev

​agentmark login

​agentmark logout

​agentmark link

​agentmark run-prompt

​agentmark run-experiment

​agentmark generate-types

​agentmark generate-schema

​agentmark build

​agentmark pull-models

​agentmark api

​Configuration files

​agentmark.json

​.agentmark/dev-config.json

​Have Questions?

Installation

Environment variables

Update notifications

Commands

agentmark dev

agentmark login

agentmark logout

agentmark link

agentmark run-prompt

agentmark run-experiment

agentmark generate-types

agentmark generate-schema

agentmark build

agentmark pull-models

agentmark api

Configuration files

`agentmark.json`

`.agentmark/dev-config.json`

Have Questions?