Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.agentmark.co/llms.txt

Use this file to discover all available pages before exploring further.

The AgentMark CLI (@agentmark-ai/cli) provides tools for developing, testing, and building your AI prompts.

Installation

# Use directly with npx (no global install needed)
npx agentmark <command>

# Or install globally
npm install -g @agentmark-ai/cli
agentmark <command>

Environment variables

The CLI automatically loads environment variables from a .env file in the current working directory. This happens before any command execution, so you can store API keys and configuration there.
# .env
OPENAI_API_KEY=sk-...
AGENTMARK_API_KEY=...
AGENTMARK_APP_ID=...
See Environment variables for the full list.

Update notifications

The CLI checks for updates asynchronously when you run commands. If a newer version is available, you’ll see a notification after your command completes. This check is non-blocking and won’t slow down your workflow. To disable update checks, set the environment variable:
export AGENTMARK_NO_UPDATE_NOTIFIER=1

Commands

agentmark dev

Start the local development environment — API server, webhook server, and the local dev UI app. When the project is linked (agentmark link), traces from local runs automatically forward to AgentMark Cloud.
npx agentmark dev [options]
Options:
OptionDescriptionDefault
--api-port <number>API server port9418
--webhook-port <number>Webhook server port9417
--app-port <number>Local dev UI port3000
--no-forwardDisable trace forwarding to AgentMark Cloud (forwarding is on by default when the project is linked)forward
--no-uiSkip the UI app (API + webhook only) — for CI / headless / test useUI on
The --remote flag and WebSocket Connect feature were removed in @agentmark-ai/cli 0.13.0. For programmatic Cloud access (what --remote used to enable), run the agentmark-mcp MCP server or call the gateway REST API directly.
Project detection:
  • TypeScript projects: agentmark.client.ts in the project root
  • Python projects: pyproject.toml, agentmark_client.py, or .agentmark/dev_server.py
Dev server entry points (TypeScript): The CLI looks for dev server files in this order:
  1. dev-server.ts — custom override (project root)
  2. dev-entry.ts — default location (project root)
  3. .agentmark/dev-entry.ts — legacy location
Python virtual environment: For Python projects, the CLI auto-detects .venv/ or venv/ directories. Example:
# Default — API + webhook + UI on 9418/9417/3000
npx agentmark dev

# Custom ports
npx agentmark dev --api-port 9500 --webhook-port 9501

# CI / headless — no UI app
npx agentmark dev --no-ui

# Linked project, but don't forward traces to Cloud
npx agentmark dev --no-forward

agentmark login

Authenticate with AgentMark Cloud via browser OAuth. The CLI opens your default browser to complete the login flow, then stores credentials locally for subsequent commands.
npx agentmark login [options]
Options:
OptionDescriptionDefault
--base-url <url>AgentMark Cloud URL$AGENTMARK_PLATFORM_URL or https://app.agentmark.co
--print-urlPrint the auth URL instead of opening a browser (for SSH’d shells, CI runners, or IDE-embedded agents)open browser
--jsonEmit a single line of JSON on completion instead of human text — useful for wrapper scripts that need to capture user_id / email programmaticallyhuman text
--timeout <seconds>How long to wait for the browser handoff before failing120 (2 minutes)
Stored credentials are used automatically by agentmark link and by the agentmark-mcp MCP server. You can override stored credentials by setting the AGENTMARK_API_KEY environment variable, which takes precedence over the cached session bearer.

agentmark logout

Clear stored CLI authentication credentials and revoke any dev API keys created during agentmark link.
npx agentmark logout [options]
Options:
OptionDescriptionDefault
--base-url <url>AgentMark Cloud URL$AGENTMARK_PLATFORM_URL or https://app.agentmark.co
--jsonEmit a single line of JSON on completion instead of human text. Shape: {"logged_out": true, "was_logged_in": <bool>, "revoked_dev_key": <bool>}human text

Link your local project to an app in AgentMark Cloud. The CLI prompts you to select an app from your account (or use --app-id to skip the prompt), then stores the app ID and a dev API key in your local project configuration.
npx agentmark link [options]
Options:
OptionDescriptionDefault
--app-id <uuid>App ID to link (skips interactive selection)
--base-url <url>AgentMark Cloud URL$AGENTMARK_PLATFORM_URL or https://app.agentmark.co
--jsonEmit a single line of JSON on completion (e.g. for CI to capture the linked appId). Shape: {"linked": true, "appId": "...", "appName": "...", "tenantId": "...", "orgName": "...", "baseUrl": "..."}human text
After linking, agentmark dev automatically forwards traces from local prompt runs to the linked app — no flag needed. The linked appId is read from .agentmark/dev-config.json (per-developer, gitignored). The forwarder authenticates with the session bearer from ~/.agentmark/auth.json (auto-refreshed); AGENTMARK_API_KEY overrides if set.

agentmark run-prompt

Run a single prompt file with test props.
npx agentmark run-prompt <filepath> [options]
Arguments:
ArgumentDescription
filepathPath to the .prompt.mdx file
Options:
OptionDescriptionDefault
--server <url>Webhook server URLhttp://localhost:9417
--props <json>Props as JSON string-
--props-file <path>Path to JSON or YAML file containing props-
Example:
# Run with inline props
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --props '{"name": "Alice"}'

# Run with props from file
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --props-file ./test-props.yaml

# Run against a remote server
npx agentmark run-prompt ./agentmark/greeting.prompt.mdx --server https://my-webhook.example.com

agentmark run-experiment

Run an experiment against its dataset, with evaluations by default.
npx agentmark run-experiment <filepath> [options]
Arguments:
ArgumentDescription
filepathPath to the .prompt.mdx file with test configuration
Options:
OptionDescriptionDefault
--server <url>Webhook server URLhttp://localhost:9417
--skip-evalSkip running evals even if they existfalse
--format <format>Output format: table, csv, json, jsonl, junittable
--threshold <percent>Fail if pass rate is below threshold (0-100)-
--sample <percent>Sample N% of dataset rows randomly (1-100)-
--rows <spec>Select specific rows by index/range (e.g. 0,3-5,9)-
--split <spec>Train/test split (e.g. train:80, test:80)-
--seed <number>Seed for reproducible sampling/splitting-
--truncate <chars>Truncate table cell content to N chars (0 = no limit)1000
--concurrency <number>Dataset rows to run in parallel20
--baseline-commit <ref>Git ref (or tree hash) of a prior run to compare against; enables the regression gate via test_settings.regression_tolerance-
Example:
# Run experiment with table output
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx

# Run experiment with JSON output, skip evals
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format json --skip-eval

# Run with CI threshold (fails if <80% pass rate)
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --threshold 80

# Emit JUnit XML for CI gating (GitHub Actions, GitLab, Jenkins, etc.)
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format junit > results.xml

# Gate against a baseline run — fails rows whose scorer regresses beyond
# test_settings.regression_tolerance relative to the baseline commit
npx agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --baseline-commit main

agentmark generate-types

Generate TypeScript type definitions from your prompt schemas.
npx agentmark generate-types [options]
Options:
OptionDescriptionDefault
-l, --language <language>Target languagetypescript
--local <port>Local server port to fetch prompts from-
--root-dir <path>Root directory containing agentmark files-
Output: The command outputs TypeScript definitions to stdout. Redirect to a file:
npx agentmark generate-types --root-dir ./agentmark > agentmark.types.ts
Generated types include:
  • Input types based on input_schema
  • Output types based on the model’s schema
  • A mapping of prompt paths to their respective types
  • Tool argument types
Example:
# Generate from local files
npx agentmark generate-types --root-dir ./agentmark > agentmark.types.ts

# Generate from local dev server
npx agentmark generate-types --local 9418 > agentmark.types.ts
See Type safety for usage examples.

agentmark generate-schema

Generate a JSON Schema file for .prompt.mdx frontmatter. This enables IDE validation (squiggles) for fields like model_name in your prompt files.
npx agentmark generate-schema [options]
Options:
OptionDescriptionDefault
-o, --out <directory>Output directory.agentmark
Example:
npx agentmark generate-schema
npx agentmark generate-schema --out ./schemas

agentmark build

Build prompts into pre-compiled JSON files for static loading with FileLoader.
npx agentmark build [options]
Options:
OptionDescriptionDefault
-o, --out <directory>Output directorydist/agentmark
Requirements:
  • An agentmark.json config file must exist in the current directory
  • Prompts are read from the directory specified by agentmarkPath in the config
Output Structure:
dist/agentmark/
  manifest.json           # Build manifest with all prompts
  greeting.prompt.json    # Compiled prompt (mirrors source structure)
  nested/
    helper.prompt.json
Example:
# Build with default output directory
npx agentmark build

# Build to custom directory
npx agentmark build --out ./build/prompts
See Loaders for using built prompts with FileLoader.

agentmark pull-models

Pull and configure models from a provider. Runs interactively by default; pass --provider + --models to skip the prompts (e.g. for CI).
npx agentmark pull-models [options]
Options:
OptionDescriptionDefault
--provider <name>Provider key (skips the interactive picker)prompt
--models <csv>Comma-separated model IDs to add (skips the interactive multi-select)prompt
With both --provider and --models set, the command runs fully non-interactively and is safe for CI. This command opens an interactive prompt (when no flags are passed) to:
  1. Select a model provider
  2. Choose models to enable
  3. Update your local configuration

Programmatic gateway access (for agents and scripts)

The agentmark api CLI command was retired in favor of two protocol-level surfaces that stay in lock-step with the gateway’s OpenAPI spec:
  • IDE agents (Claude Code, Cursor, VS Code, Zed): run the agentmark-mcp MCP server. It fetches the gateway’s OpenAPI spec at startup and exposes one MCP tool per operation (e.g. create_app, list_traces, start_app_git_connect). The agent calls those tools directly; no CLI invocation needed.
  • CI / shell scripts: call the gateway REST API with curl and an AGENTMARK_API_KEY. The MCP tools are generated from the same OpenAPI spec, so request shapes are identical — only the transport differs.
Both targets honor the same auth chain: AGENTMARK_API_KEY env var first, then the session bearer from ~/.agentmark/auth.json (written by agentmark login).

Configuration files

agentmark.json

Project configuration file in your project root. See Project config for the full schema.
{
  "agentmarkPath": ".",
  "version": "2.0.0",
  "mdxVersion": "1.0"
}
FieldDescription
agentmarkPathBase path for agentmark files (contains the agentmark/ directory) — use "." for the canonical layout, not "/"
versionConfiguration version
mdxVersionMDX syntax version

.agentmark/dev-config.json

Auto-generated local development configuration (gitignored):
{
  "createdAt": "2026-04-15T10:30:00.000Z",
  "appPort": 3000,
  "forwarding": {
    "appId": "app_xxxxx",
    "appName": "my-app",
    "orgName": "my-org",
    "tenantId": "tenant_xxxxx",
    "apiKey": "am_dev_xxxxx",
    "apiKeyId": "key_xxxxx",
    "expiresAt": "2026-05-15T10:30:00.000Z",
    "baseUrl": "https://app.agentmark.co"
  }
}
This file stores:
  • appPort — local dev server UI port (updated when dev server starts).
  • forwarding — linked app metadata (app ID, dev API key, token expiry) used by agentmark dev when forwarding traces to AgentMark Cloud. Populated by agentmark link and cleared by agentmark logout.
The configuration expires after 30 days and is regenerated on the next agentmark link.

Have Questions?

We’re here to help! Choose the best way to reach us: