CLI reference - AgentMark Docs

The AgentMark CLI (@agentmark-ai/cli) provides tools for developing, testing, and building your AI prompts.

Installation

# Use directly with npx (no global install needed)
npx agentmark <command>

# Or install globally
npm install -g @agentmark-ai/cli
agentmark <command>

Environment variables

The CLI automatically loads environment variables from a .env file in the current working directory. This happens before any command execution, so you can store API keys and configuration there.

# .env
OPENAI_API_KEY=sk-...
AGENTMARK_API_KEY=...
AGENTMARK_APP_ID=...

See Environment variables for the full list.

Update notifications

The CLI checks for updates asynchronously when you run commands. If a newer version is available, you’ll see a notification after your command completes. This check is non-blocking and won’t slow down your workflow. To disable update checks, set the environment variable:

export AGENTMARK_NO_UPDATE_NOTIFIER=1

Commands

agentmark dev

Start the local development environment — API server, webhook server, and the local dev UI app. When the project is linked (agentmark link), traces from local runs automatically forward to AgentMark Cloud.

npx agentmark dev [options]

Options:

Option	Description	Default
`--api-port <number>`	API server port	`9418`
`--webhook-port <number>`	Webhook server port	`9417`
`--app-port <number>`	Local dev UI port	`3000`
`--no-forward`	Disable trace forwarding to AgentMark Cloud (forwarding is on by default when the project is linked)	forward
`--no-ui`	Skip the UI app (API + webhook only) — for CI / headless / test use	UI on

The --remote flag and WebSocket Connect feature were removed in @agentmark-ai/cli 0.13.0. For programmatic Cloud access (what --remote used to enable), run the agentmark-mcp MCP server or call the gateway REST API directly.

Project detection:

TypeScript projects: agentmark.client.ts in the project root
Python projects: pyproject.toml, agentmark_client.py, or .agentmark/dev_server.py

Dev server entry points (TypeScript): The CLI looks for dev server files in this order:

dev-server.ts — custom override (project root)
dev-entry.ts — default location (project root)
.agentmark/dev-entry.ts — legacy location

Python virtual environment: For Python projects, the CLI auto-detects .venv/ or venv/ directories. Example:

# Default — API + webhook + UI on 9418/9417/3000
npx agentmark dev

# Custom ports
npx agentmark dev --api-port 9500 --webhook-port 9501

# CI / headless — no UI app
npx agentmark dev --no-ui

# Linked project, but don't forward traces to Cloud
npx agentmark dev --no-forward

Authenticate with AgentMark Cloud via browser OAuth. The CLI opens your default browser to complete the login flow, then stores credentials locally for subsequent commands.

npx agentmark login [options]

Options:

Option	Description	Default
`--base-url <url>`	AgentMark Cloud URL	`$AGENTMARK_PLATFORM_URL` or `https://app.agentmark.co`
`--print-url`	Print the auth URL instead of opening a browser (for SSH’d shells, CI runners, or IDE-embedded agents)	open browser
`--json`	Emit a single line of JSON on completion instead of human text — useful for wrapper scripts that need to capture `user_id` / `email` programmatically	human text
`--timeout <seconds>`	How long to wait for the browser handoff before failing	`120` (2 minutes)

Stored credentials are used automatically by agentmark link and by the agentmark-mcp MCP server. You can override stored credentials by setting the AGENTMARK_API_KEY environment variable, which takes precedence over the cached session bearer.

agentmark logout

Clear stored CLI authentication credentials and revoke any dev API keys created during agentmark link.

npx agentmark logout [options]

Options:

Option	Description	Default
`--base-url <url>`	AgentMark Cloud URL	`$AGENTMARK_PLATFORM_URL` or `https://app.agentmark.co`
`--json`	Emit a single line of JSON on completion instead of human text. Shape: `{"logged_out": true, "was_logged_in": <bool>, "revoked_dev_key": <bool>}`	human text

agentmark link

Link your local project to an app in AgentMark Cloud. The CLI prompts you to select an app from your account (or use --app-id to skip the prompt), then stores the app ID and a dev API key in your local project configuration.

npx agentmark link [options]

Options:

Option	Description	Default
`--app-id <uuid>`	App ID to link (skips interactive selection)	—
`--base-url <url>`	AgentMark Cloud URL	`$AGENTMARK_PLATFORM_URL` or `https://app.agentmark.co`
`--json`	Emit a single line of JSON on completion (e.g. for CI to capture the linked appId). Shape: `{"linked": true, "appId": "...", "appName": "...", "tenantId": "...", "orgName": "...", "baseUrl": "..."}`	human text

After linking, agentmark dev automatically forwards traces from local prompt runs to the linked app — no flag needed. The linked appId is read from .agentmark/dev-config.json (per-developer, gitignored). The forwarder authenticates with the session bearer from ~/.agentmark/auth.json (auto-refreshed); AGENTMARK_API_KEY overrides if set.

agentmark run-prompt

Run a single prompt file with test props.

npx agentmark run-prompt <filepath> [options]

Arguments:

Argument	Description
`filepath`	Path to the `.prompt.mdx` file

Options:

Option	Description	Default
`--server <url>`	Webhook server URL	`http://localhost:9417`
`--props <json>`	Props as JSON string	-
`--props-file <path>`	Path to JSON or YAML file containing props	-

Example:

# Run with inline props
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --props '{"name": "Alice"}'

# Run with props from file
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --props-file ./test-props.yaml

# Run against a remote server
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --server https://my-webhook.example.com

agentmark run-experiment

Run an experiment against its dataset, with evaluations by default.

npx agentmark run-experiment <filepath> [options]

Arguments:

Argument	Description
`filepath`	Path to the `.prompt.mdx` file with test configuration

Options:

Option	Description	Default
`--server <url>`	Webhook server URL	`http://localhost:9417`
`--skip-eval`	Skip running evals even if they exist	`false`
`--format <format>`	Output format: `table`, `csv`, `json`, `jsonl`, `junit`	`table`
`--threshold <percent>`	Fail if pass rate is below threshold (0-100)	-
`--sample <percent>`	Sample N% of dataset rows randomly (1-100)	-
`--rows <spec>`	Select specific rows by index/range (e.g. `0,3-5,9`)	-
`--split <spec>`	Train/test split (e.g. `train:80`, `test:80`)	-
`--seed <number>`	Seed for reproducible sampling/splitting	-
`--truncate <chars>`	Truncate table cell content to N chars (0 = no limit)	`1000`
`--concurrency <number>`	Dataset rows to run in parallel	`20`
`--baseline-commit <ref>`	Git ref (or tree hash) of a prior run to compare against; enables the regression gate via `test_settings.regression_tolerance`	-

Example:

# Run experiment with table output
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx

# Run experiment with JSON output, skip evals
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format json --skip-eval

# Run with CI threshold (fails if <80% pass rate)
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --threshold 80

# Emit JUnit XML for CI gating (GitHub Actions, GitLab, Jenkins, etc.)
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format junit > results.xml

# Gate against a baseline run — fails rows whose scorer regresses beyond
# test_settings.regression_tolerance relative to the baseline commit
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --baseline-commit main

agentmark generate-types

Generate TypeScript type definitions from your prompt schemas.

npx agentmark generate-types [options]

Options:

Option	Description	Default
`-l, --language <language>`	Target language	`typescript`
`--local <port>`	Local server port to fetch prompts from	-
`--root-dir <path>`	Root directory containing agentmark files	-

Output: The command outputs TypeScript definitions to stdout. Redirect to a file:

npx agentmark generate-types --root-dir ./agentmark > agentmark.types.ts

Generated types include:

Input types based on input_schema
Output types based on the model’s schema
A mapping of prompt paths to their respective types
Tool argument types

Example:

# Generate from local files
npx agentmark generate-types --root-dir ./agentmark > agentmark.types.ts

# Generate from local dev server
npx agentmark generate-types --local 9418 > agentmark.types.ts

See Type safety for usage examples.

agentmark generate-schema

Generate a JSON Schema file for .prompt.mdx frontmatter. This enables IDE validation (squiggles) for fields like model_name in your prompt files.

npx agentmark generate-schema [options]

Options:

Option	Description	Default
`-o, --out <directory>`	Output directory	`.agentmark`

Example:

npx agentmark generate-schema
npx agentmark generate-schema --out ./schemas

agentmark build

Build prompts into pre-compiled JSON files for static loading with FileLoader.

npx agentmark build [options]

Options:

Option	Description	Default
`-o, --out <directory>`	Output directory	`dist/agentmark`

Requirements:

An agentmark.json config file must exist in the current directory
Prompts are read from the directory specified by agentmarkPath in the config

Output Structure:

dist/agentmark/
  manifest.json           # Build manifest with all prompts
  greeting.prompt.json    # Compiled prompt (mirrors source structure)
  nested/
    helper.prompt.json

Example:

# Build with default output directory
npx agentmark build

# Build to custom directory
npx agentmark build --out ./build/prompts

See Loaders for using built prompts with FileLoader.

agentmark pull-models

Pull and configure models from a provider. Runs interactively by default; pass --provider + --models to skip the prompts (e.g. for CI).

npx agentmark pull-models [options]

Options:

Option	Description	Default
`--provider <name>`	Provider key (skips the interactive picker)	prompt
`--models <csv>`	Comma-separated model IDs to add (skips the interactive multi-select)	prompt

With both --provider and --models set, the command runs fully non-interactively and is safe for CI. This command opens an interactive prompt (when no flags are passed) to:

Select a model provider
Choose models to enable
Update your local configuration

Programmatic gateway access (for agents and scripts)

The agentmark api CLI command was retired in favor of two protocol-level surfaces that stay in lock-step with the gateway’s OpenAPI spec:

IDE agents (Claude Code, Cursor, VS Code, Zed): run the agentmark-mcp MCP server. It fetches the gateway’s OpenAPI spec at startup and exposes one MCP tool per operation (e.g. create_app, list_traces, start_app_git_connect). The agent calls those tools directly; no CLI invocation needed.
CI / shell scripts: call the gateway REST API with curl and an AGENTMARK_API_KEY. The MCP tools are generated from the same OpenAPI spec, so request shapes are identical — only the transport differs.

Both targets honor the same auth chain: AGENTMARK_API_KEY env var first, then the session bearer from ~/.agentmark/auth.json (written by agentmark login).

Configuration files

`agentmark.json`

Project configuration file in your project root. See Project config for the full schema.

{
  "agentmarkPath": ".",
  "version": "2.0.0",
  "mdxVersion": "1.0"
}

Field	Description
`agentmarkPath`	Base path for agentmark files (contains the `agentmark/` directory) — use `"."` for the canonical layout, not `"/"`
`version`	Configuration version
`mdxVersion`	MDX syntax version

`.agentmark/dev-config.json`

Auto-generated local development configuration (gitignored):

{
  "createdAt": "2026-04-15T10:30:00.000Z",
  "appPort": 3000,
  "forwarding": {
    "appId": "app_xxxxx",
    "appName": "my-app",
    "orgName": "my-org",
    "tenantId": "tenant_xxxxx",
    "apiKey": "am_dev_xxxxx",
    "apiKeyId": "key_xxxxx",
    "expiresAt": "2026-05-15T10:30:00.000Z",
    "baseUrl": "https://app.agentmark.co"
  }
}

This file stores:

appPort — local dev server UI port (updated when dev server starts).
forwarding — linked app metadata (app ID, dev API key, token expiry) used by agentmark dev when forwarding traces to AgentMark Cloud. Populated by agentmark link and cleared by agentmark logout.

The configuration expires after 30 days and is regenerated on the next agentmark link.

Have Questions?

We’re here to help! Choose the best way to reach us:

Email us at hello@agentmark.co for support
Schedule an Enterprise Demo to learn about our business solutions

Documentation Index

​Installation

​Environment variables

​Update notifications

​Commands

​agentmark dev

​agentmark login

​agentmark logout

​agentmark link

​agentmark run-prompt

​agentmark run-experiment

​agentmark generate-types

​agentmark generate-schema

​agentmark build

​agentmark pull-models

​Programmatic gateway access (for agents and scripts)

​Configuration files

​agentmark.json

​.agentmark/dev-config.json

​Have Questions?

Installation

Environment variables

Update notifications

Commands

agentmark dev

agentmark login

agentmark logout

agentmark link

agentmark run-prompt

agentmark run-experiment

agentmark generate-types

agentmark generate-schema

agentmark build

agentmark pull-models

Programmatic gateway access (for agents and scripts)

Configuration files

`agentmark.json`

`.agentmark/dev-config.json`

Have Questions?