CLI reference - AgentMark Docs

The AgentMark CLI (@agentmark-ai/cli) provides tools for developing, testing, and building your AI prompts.

Commands at a glance

Jump to any command’s full reference:

agentmark init: set up AgentMark in a new or existing project
agentmark doctor: check that you configured your project correctly
agentmark dev: start the local dev environment (API, webhook, UI)
agentmark login: authenticate with AgentMark Cloud
agentmark logout: clear stored CLI credentials
agentmark link: link your local project to a Cloud app
agentmark run-prompt: run a single prompt with test props
agentmark run-experiment: run an experiment against its dataset
agentmark generate-types: generate type definitions from prompt schemas
agentmark generate-schema: generate a JSON Schema for prompt frontmatter
agentmark build: compile prompts to JSON for static loading
agentmark pull-models: pull and configure models from a provider

Installation

# Install globally so the bare `agentmark` command is on your PATH
npm install -g @agentmark-ai/cli
agentmark <command>

# Or run without installing, via npx
npx @agentmark-ai/cli <command>

In a project scaffolded by agentmark init, @agentmark-ai/cli is also pinned as a local dev dependency, so npm run dev and CI resolve that pinned version from node_modules/.bin regardless of what’s installed globally.

Environment variables

The CLI automatically loads environment variables from a .env file in the current working directory. This happens before any command execution, so you can store API keys and configuration there.

# .env
OPENAI_API_KEY=sk-...
AGENTMARK_API_KEY=...
AGENTMARK_APP_ID=...

See Environment variables for the full list.

Update notifications

The CLI checks for updates asynchronously when you run commands. If a newer version is available, you’ll see a notification after your command completes. This check is non-blocking. To disable update checks, set the environment variable:

export AGENTMARK_NO_UPDATE_NOTIFIER=1

Commands

`agentmark init`

Set up AgentMark in a new or existing project. Writes agentmark.json, creates the agentmark/ prompts directory, pins @agentmark-ai/cli as a local dev dependency (plus npm scripts), wires IDE MCP configs for the clients you pick, and installs the agent skill. Run it once per project: from inside an existing repo (the common case) or with a folder name to scaffold a fresh one.

agentmark init [folder] [options]

Options:

Option	Description	Default
`[folder]` / `--path <folder>`	Target directory	`.` inside an existing project, else `my-agentmark-app`
`--client <ids>`	IDE clients to wire MCP configs for, comma-separated: `claude-code`, `codex`, `cursor`, `vscode`, `zed` (or `all`)	prompt (pre-selects detected editors)
`-y, --yes`	Non-interactive: accept the default for every prompt (folder default, all IDE clients, keep an existing `agentmark.json`). For CI and coding agents	interactive
`--overwrite`	Replace an existing `agentmark.json` with the default config	keep existing
`--api-url <url>`	Override the AgentMark gateway URL for the cloud MCP entry (internal staging / self-host)	`https://api.agentmark.co`

The local pin is non-destructive: an existing dev / build script is never overwritten. AgentMark’s scripts land under namespaced keys (agentmark:dev, …) instead. npm create agentmark is a thin wrapper that runs this exact flow.

`agentmark doctor`

Check that you set up your AgentMark project correctly. The default pass is a static health check (no network calls, no servers spawned) that inspects your config, prompts, client, and dependencies and reports each finding with a fix. The optional --smoke tier adds a live end-to-end run. Run it right after scaffolding, or when a command fails with a config or dependency error.

agentmark doctor [options]

Options:

Option	Description	Default
`--json`	Emit the report as JSON instead of human-readable text	human text
`--strict`	Exit non-zero on warnings too (useful in CI)	off
`--smoke`	Also run a live tier: execute a prompt against a running `agentmark dev` and verify the emitted trace’s shape (model, tokens, input, output)	static only
`--boot`	With `--smoke`, start `agentmark dev` automatically and tear it down afterward (one command, for CI / agents)	assumes dev is running
`--prompt <path>`	Prompt to run for `--smoke` (defaults to the first prompt found)	first prompt
`--props <json>`	Props as a JSON string for the `--smoke` prompt. Requires `--prompt` (for example `'{"name":"Alice"}'`)	none
`--webhook-port <number>`	Webhook port `--smoke` targets (and `--boot` starts dev on)	`9417`
`--api-port <number>`	API-server port `--smoke` reads traces from (and `--boot` starts dev on)	`9418`

What it checks:

Project: agentmark.json is present and valid (including field/schema shape: required keys present, no unknown-key typos); agentmarkPath resolves (catches the "/" mistake); the client file (agentmark.client.ts / agentmark_client.py), the dev-server entry, and the managed-deploy handler (handler.ts / handler.py) exist; AGENTMARK_API_KEY / AGENTMARK_APP_ID are present; .env is gitignored; Node.js is 20.6+.
Config & prompts: every .prompt.mdx parses and declares a model_name; you declare prompt models in builtInModels (doctor enforces a non-empty list as an allowlist); the catalog recognizes the models.
Dependencies: doctor checks that you installed the runtime packages (resolvable from the project), not just listed them in package.json. The client and dev-entry import @agentmark-ai/prompt-core, so doctor fails the check when package.json declares prompt-core but node_modules lacks it, and points you to npm install; a missing install is the most common reason agentmark dev or doctor --smoke --boot exits 1. @agentmark-ai/sdk carries tracing and the cloud-execution runner; doctor warns (rather than fails) when node_modules lacks it or package.json omits it. agentmark init pins @agentmark-ai/cli locally so CI and teammates run the same version; a missing pin is an advisory warning. For Python projects, doctor checks that agentmark-prompt-core and agentmark-sdk import, and skips with a reminder when it can’t verify them. There is no SDK-specific adapter to require: you bring your own SDK through @agentmark-ai/prompt-core’s neutral render or an executor.
Live run (--smoke): runs one prompt end-to-end through agentmark dev and confirms real output + token usage came back, then fetches the emitted trace and validates its shape, exercising your SDK, provider keys, and tracing indirectly, with no provider-specific knowledge.

Exit code: 0 when nothing failed (warnings don’t fail the run), 1 on any failure, or on any non-advisory warning with --strict. Advisory warnings (like a missing local @agentmark-ai/cli pin) are informational and never fail the run, so adding agentmark init to an existing project won’t turn a green --strict CI gate red. So it drops straight into CI. Example:

# Human-readable report
agentmark doctor

# Machine-readable, for scripts
agentmark doctor --json

# CI: treat warnings as failures too
agentmark doctor --strict

# Live end-to-end check: boot dev, run a prompt, verify the trace, tear down
agentmark doctor --smoke --boot

# Smoke a specific prompt that requires variables
agentmark doctor --smoke --prompt agentmark/greeting.prompt.mdx --props '{"name":"Alice"}'

For the failures it surfaces, see Troubleshooting.

`agentmark dev`

Start the local development environment: the API server, the webhook server, and the local dev UI app. Once you link the project (agentmark link), traces from local runs automatically forward to AgentMark Cloud.

agentmark dev [options]

Options:

Option	Description	Default
`--api-port <number>`	API server port	`9418`
`--webhook-port <number>`	Webhook server port	`9417`
`--app-port <number>`	Local dev UI port	`3000`
`--no-forward`	Disable trace forwarding to AgentMark Cloud (forwarding is on by default once you link the project)	forward
`--no-ui`	Skip the UI app (API + webhook only) for CI / headless / test use	UI on
`--no-watch`	Don’t restart on file changes; exit on a dev-entry crash so the error surfaces (for CI / headless / boot use)	watch on

For programmatic Cloud access, run the agentmark-mcp MCP server or call the gateway REST API directly.

Project detection:

TypeScript projects: agentmark.client.ts in the project root
Python projects: pyproject.toml, agentmark_client.py, or .agentmark/dev_server.py

Dev server entry points (TypeScript): The CLI looks for dev server files in this order:

dev-server.ts (custom override, project root)
dev-entry.ts (default location, project root)

Python virtual environment: For Python projects, the CLI auto-detects .venv/ or venv/ directories. Example:

# Default — API + webhook + UI on 9418/9417/3000
agentmark dev

# Custom ports
agentmark dev --api-port 9500 --webhook-port 9501

# CI / headless — no UI app
agentmark dev --no-ui

# Linked project, but don't forward traces to Cloud
agentmark dev --no-forward

Authenticate with AgentMark Cloud via browser OAuth. The CLI opens your default browser to complete the login flow, then stores credentials locally for subsequent commands.

agentmark login [options]

Options:

Option	Description	Default
`--base-url <url>`	AgentMark Cloud URL	`$AGENTMARK_PLATFORM_URL` or `https://app.agentmark.co`
`--print-url`	Print the auth URL instead of opening a browser (for SSH’d shells, CI runners, or IDE-embedded agents)	open browser
`--json`	Emit a single line of JSON on completion instead of human text; useful for wrapper scripts that need to capture `user_id` / `email` programmatically	human text
`--timeout <seconds>`	How long to wait for the browser handoff before failing	`120` (2 minutes)

agentmark link, trace forwarding in agentmark dev, and the agentmark-mcp MCP server all use the stored credentials automatically. For the MCP server, the AGENTMARK_API_KEY environment variable takes precedence over the cached session bearer; agentmark link itself always requires a login session.

`agentmark logout`

Clear stored CLI authentication credentials.

agentmark logout [options]

Options:

Option	Description	Default
`--base-url <url>`	AgentMark Cloud URL	`$AGENTMARK_PLATFORM_URL` or `https://app.agentmark.co`
`--json`	Emit a single line of JSON on completion instead of human text. Shape: `{"logged_out": true, "was_logged_in": <bool>, "revoked_dev_key": <bool>}`	human text

`agentmark link`

Link your local project to an app in AgentMark Cloud. The CLI prompts you to select an app from your account (or use --app-id to skip the prompt), then stores the app ID and app/org metadata in .agentmark/dev-config.json. The CLI creates no API key: trace forwarding authenticates with your login session.

agentmark link [options]

Options:

Option	Description	Default
`--app-id <uuid>`	App ID to link (skips interactive selection)	None
`--base-url <url>`	AgentMark Cloud URL	`$AGENTMARK_PLATFORM_URL` or `https://app.agentmark.co`
`--json`	Emit a single line of JSON on completion (for example, for CI to capture the linked appId). Shape: `{"linked": true, "appId": "...", "appName": "...", "tenantId": "...", "orgName": "...", "baseUrl": "..."}`	human text

After linking, agentmark dev automatically forwards traces from local prompt runs to the linked app, with no flag needed. agentmark dev reads the linked appId from .agentmark/dev-config.json (per-developer, gitignored). The forwarder authenticates with the session bearer from ~/.agentmark/auth.json (auto-refreshed).

`agentmark run-prompt`

Run a single prompt file with test props.

agentmark run-prompt <filepath> [options]

Arguments:

Argument	Description
`filepath`	Path to the `.prompt.mdx` file

Options:

Option	Description	Default
`--server <url>`	Webhook server URL	`http://localhost:9417`
`--props <json>`	Props as JSON string	-
`--props-file <path>`	Path to JSON or YAML file containing props	-

Example:

# Run with inline props
agentmark run-prompt ./agentmark/greeting.prompt.mdx --props '{"name": "Alice"}'

# Run with props from file
agentmark run-prompt ./agentmark/greeting.prompt.mdx --props-file ./test-props.yaml

# Run against a remote server
agentmark run-prompt ./agentmark/greeting.prompt.mdx --server https://my-webhook.example.com

`agentmark run-experiment`

Run an experiment against its dataset, with evaluations by default.

agentmark run-experiment <filepath> [options]

Arguments:

Argument	Description
`filepath`	Path to the `.prompt.mdx` file with test configuration

Options:

Option	Description	Default
`--server <url>`	Webhook server URL	`http://localhost:9417`
`--skip-eval`	Skip running evals even if they exist	`false`
`--format <format>`	Output format: `table`, `csv`, `json`, `jsonl`, `junit`	`table`
`--threshold <percent>`	Fail if pass rate is below threshold (0-100)	-
`--sample <percent>`	Sample N% of dataset rows randomly (1-100)	-
`--rows <spec>`	Select specific rows by index/range (for example `0,3-5,9`)	-
`--split <spec>`	Train/test split (for example `train:80`, `test:80`)	-
`--seed <number>`	Seed for reproducible sampling/splitting	-
`--truncate <chars>`	Truncate table cell content to N chars (0 = no limit)	`1000`
`--concurrency <number>`	Dataset rows to run in parallel	`20`
`--baseline-commit <ref>`	Git ref (or tree hash) of a prior run to compare against; enables the regression gate via `test_settings.regression_tolerance`	-

Example:

# Run experiment with table output
agentmark run-experiment ./agentmark/qa-bot.prompt.mdx

# Run experiment with JSON output, skip evals
agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format json --skip-eval

# Run with CI threshold (fails if <80% pass rate)
agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --threshold 80

# Emit JUnit XML for CI gating (GitHub Actions, GitLab, Jenkins, etc.)
agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format junit > results.xml

# Gate against a baseline run — fails rows whose scorer regresses beyond
# test_settings.regression_tolerance relative to the baseline commit
agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --baseline-commit main

`agentmark generate-types`

Generate TypeScript type definitions from your prompt schemas.

agentmark generate-types [options]

Options:

Option	Description	Default
`-l, --language <language>`	Target language (`typescript` or `python`)	`typescript`
`--local <port>`	Local server port to fetch prompts from	-
`--root-dir <path>`	Root directory containing agentmark files	-

Output: The command outputs TypeScript definitions to stdout. Redirect to a file:

agentmark generate-types --root-dir ./agentmark > agentmark.types.ts

Generated types include:

Input types based on input_schema
Output types based on the model’s schema
A mapping of prompt paths to their respective types
Tool argument types

Example:

# Generate from local files
agentmark generate-types --root-dir ./agentmark > agentmark.types.ts

# Generate from local dev server
agentmark generate-types --local 9418 > agentmark.types.ts

See Type safety for usage examples.

`agentmark generate-schema`

Generate a JSON Schema file for .prompt.mdx frontmatter. This enables IDE validation (squiggles) for fields like model_name in your prompt files.

agentmark generate-schema [options]

Options:

Option	Description	Default
`-o, --out <directory>`	Output directory	`.agentmark`

Example:

agentmark generate-schema
agentmark generate-schema --out ./schemas

`agentmark build`

Build prompts into pre-compiled JSON files for static loading with FileLoader.

agentmark build [options]

Options:

Option	Description	Default
`-o, --out <directory>`	Output directory	`dist/agentmark`

Requirements:

An agentmark.json config file must exist in the current directory
agentmark build reads prompts from the directory that agentmarkPath names in the config

Output structure:

dist/agentmark/
  manifest.json           # Build manifest with all prompts
  greeting.prompt.json    # Compiled prompt (mirrors source structure)
  nested/
    helper.prompt.json

Example:

# Build with default output directory
agentmark build

# Build to custom directory
agentmark build --out ./build/prompts

See Loaders for using built prompts with FileLoader.

`agentmark pull-models`

Pull and configure models from a provider. Runs interactively by default; pass --provider + --models to skip the prompts (for example, for CI or agents).

agentmark pull-models [options]

Options:

Option	Description	Default
`--provider <name>`	Provider key (skips the interactive picker)	prompt
`--models <csv>`	Comma-separated model IDs to add (skips the interactive multi-select)	prompt
`--list`	Print available providers (or models when combined with `--provider`) as JSON and exit; doesn’t modify `agentmark.json`	off

With both --provider and --models set, the command runs fully non-interactively and is safe for CI. This command opens an interactive prompt (when you pass no flags) to:

Select a model provider
Choose models to enable
Update your local configuration

Agent / CI discovery workflow Use --list to discover valid providers and model IDs before modifying config (no agentmark.json required):

# List all available providers
agentmark pull-models --list

# List models for a specific provider
agentmark pull-models --list --provider anthropic

# Add models non-interactively
agentmark pull-models --provider anthropic --models anthropic/claude-opus-4-8

--list output is stable JSON, safe to pipe into jq or capture in scripts.

Programmatic gateway access (for agents and scripts)

Programmatic gateway access is available through two protocol-level surfaces that stay in lock-step with the gateway’s OpenAPI spec:

IDE agents (Claude Code, Cursor, VS Code, Zed): run the agentmark-mcp MCP server. It fetches the gateway’s OpenAPI spec at startup and exposes one MCP tool per operation (for example create_app, list_traces, start_app_git_connect). The agent calls those tools directly; no CLI invocation needed.
CI / shell scripts: call the gateway REST API with curl and an AGENTMARK_API_KEY. AgentMark generates the MCP tools from the same OpenAPI spec, so request shapes are identical; only the transport differs.

Both targets honor the same auth chain: AGENTMARK_API_KEY env var first, then the session bearer from ~/.agentmark/auth.json (written by agentmark login).

Configuration files

`agentmark.json`

Project configuration file in your project root. See Project config for the full schema.

{
  "agentmarkPath": ".",
  "version": "2.0.0",
  "mdxVersion": "1.0"
}

Field	Description
`agentmarkPath`	Base path for agentmark files (contains the `agentmark/` directory); use `"."` for the canonical layout, not `"/"`
`version`	Configuration version
`mdxVersion`	MDX syntax version

`.agentmark/dev-config.json`

Auto-generated local development configuration (gitignored):

{
  "createdAt": "2026-04-15T10:30:00.000Z",
  "appPort": 3000,
  "forwarding": {
    "appId": "1c5f9e2a-7b43-4a1e-9c0d-3f8a2b6e4d71",
    "appName": "my-app",
    "orgName": "my-org",
    "tenantId": "9d2b4c6e-1f3a-4e5d-8a7b-0c9e2f4a6b8d",
    "baseUrl": "https://api.agentmark.co"
  }
}

This file stores:

appPort: local dev server UI port (updated when dev server starts).
forwarding: linked app metadata (app ID, app and org names, gateway base URL) used by agentmark dev when forwarding traces to AgentMark Cloud. Populated by agentmark link and cleared by agentmark logout. No credentials live here: the forwarder authenticates with your login session.

The configuration expires 30 days after creation: the next CLI command that loads it regenerates a fresh file, and running agentmark link again restores the app binding.

Have questions?

Reach out any time:

Email the team at hello@agentmark.co for support
Schedule an Enterprise Demo to learn about AgentMark’s business solutions

​Commands at a glance

​Installation

​Environment variables

​Update notifications

​Commands

​agentmark init

​agentmark doctor

​agentmark dev

​agentmark login

​agentmark logout

​agentmark link

​agentmark run-prompt

​agentmark run-experiment

​agentmark generate-types

​agentmark generate-schema

​agentmark build

​agentmark pull-models

​Programmatic gateway access (for agents and scripts)

​Configuration files

​agentmark.json

​.agentmark/dev-config.json

​Have questions?

Commands at a glance

Installation

Environment variables

Update notifications

Commands

`agentmark init`

`agentmark doctor`

`agentmark dev`

`agentmark login`

`agentmark logout`

`agentmark link`

`agentmark run-prompt`

`agentmark run-experiment`

`agentmark generate-types`

`agentmark generate-schema`

`agentmark build`

`agentmark pull-models`

Programmatic gateway access (for agents and scripts)

Configuration files

`agentmark.json`

`.agentmark/dev-config.json`

Have questions?