Skip to main content

​
CLI Reference

The AgentMark CLI (@agentmark-ai/cli) provides tools for developing, testing, building, and deploying your AI prompts.

​
Installation

npm install -g @agentmark-ai/cli
# or use with npx
npx @agentmark-ai/cli <command>

​
Environment Variables

The CLI automatically loads environment variables from a .env file in the current working directory. This happens before any command execution, so you can store API keys and configuration there. 
# .env
OPENAI_API_KEY=sk-...
AGENTMARK_API_KEY=...
AGENTMARK_APP_ID=...

​
Update Notifications

The CLI checks for updates asynchronously when you run commands. If a newer version is available, you’ll see a notification after your command completes. This check is non-blocking and won’t slow down your workflow. To disable update checks, set the environment variable: 
export AGENTMARK_DISABLE_UPDATE_CHECK=1

​
Commands

​
agentmark dev

Start the local development environment with API server, webhook server, and UI app. 
agentmark dev [options]
Options:
OptionDescriptionDefault
--api-port <number>API server port9418
--webhook-port <number>Webhook server port9417
--app-port <number>AgentMark UI app port3000
-t, --tunnelExpose webhook server publicly via tunnelfalse
Project Detection: The dev server automatically detects your project type:
  • TypeScript projects: Looks for agentmark.client.ts in the project root
  • Python projects: Looks for pyproject.toml, agentmark_client.py, or .agentmark/dev_server.py
Dev Server Entry Points (TypeScript): The CLI looks for dev server files in this order:
  1. dev-server.ts - Custom override (project root)
  2. dev-entry.ts - Default location (project root)
  3. .agentmark/dev-entry.ts - Legacy location
Python Virtual Environment: For Python projects, the CLI automatically detects and uses virtual environments in .venv/ or venv/ directories. Tunnel Feature: When using --tunnel, the CLI creates a public URL via Cloudflare Tunnel (trycloudflare.com):
  • Automatically downloads and manages the cloudflared binary
  • Generates a webhook secret stored in .agentmark/dev-config.json
  • Configuration persists for 30 days
Example: 
# Start with default ports
agentmark dev

# Start with custom ports and public tunnel
agentmark dev --api-port 9500 --webhook-port 9501 --tunnel

​
agentmark run-prompt

Run a single prompt file with test props. 
agentmark run-prompt <filepath> [options]
Arguments:
ArgumentDescription
filepathPath to the .prompt.mdx file
Options:
OptionDescriptionDefault
--server <url>Webhook server URLhttp://localhost:9417
--props <json>Props as JSON string-
--props-file <path>Path to JSON or YAML file containing props-
Example: 
# Run with inline props
agentmark run-prompt ./agentmark/greeting.prompt.mdx --props '{"name": "Alice"}'

# Run with props from file
agentmark run-prompt ./agentmark/greeting.prompt.mdx --props-file ./test-props.yaml

# Run against a remote server
agentmark run-prompt ./agentmark/greeting.prompt.mdx --server https://my-webhook.example.com

​
agentmark run-experiment

Run an experiment against its dataset, with evaluations by default. 
agentmark run-experiment <filepath> [options]
Arguments:
ArgumentDescription
filepathPath to the .prompt.mdx file with test configuration
Options:
OptionDescriptionDefault
--server <url>Webhook server URLhttp://localhost:9417
--skip-evalSkip running evals even if they existfalse
--format <format>Output format: table, csv, json, jsonltable
--threshold <percent>Fail if pass rate is below threshold (0-100)-
Example: 
# Run experiment with table output
agentmark run-experiment ./agentmark/qa-bot.prompt.mdx

# Run experiment with JSON output, skip evals
agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format json --skip-eval

# Run with CI threshold (fails if <80% pass rate)
agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --threshold 80

​
agentmark generate-types

Generate TypeScript type definitions from your prompt schemas. 
agentmark generate-types [options]
Options:
OptionDescriptionDefault
-l, --language <language>Target languagetypescript
--local <port>Local server port to fetch prompts from-
--root-dir <path>Root directory containing agentmark files-
Output: The command outputs TypeScript definitions to stdout. Redirect to a file: 
agentmark generate-types --root-dir ./agentmark > agentmark.types.ts
Generated Types Include:
  • Input types based on input_schema
  • Output types based on the model’s schema
  • A mapping of prompt paths to their respective types
  • Tool argument types
Example: 
# Generate from local files
agentmark generate-types --root-dir ./prompts > agentmark.types.ts

# Generate from local dev server
agentmark generate-types --local 9418 > agentmark.types.ts
See Type Safety for usage examples.

​
agentmark build

Build prompts into pre-compiled JSON files for static loading with FileLoader. 
agentmark build [options]
Options:
OptionDescriptionDefault
-o, --out <directory>Output directorydist/agentmark
Requirements:
  • An agentmark.json config file must exist in the current directory
  • Prompts are read from the directory specified by agentmarkPath in the config
Output Structure: 
dist/agentmark/
  manifest.json           # Build manifest with all prompts
  greeting.prompt.json    # Compiled prompt (mirrors source structure)
  nested/
    helper.prompt.json
Example: 
# Build with default output directory
agentmark build

# Build to custom directory
agentmark build --out ./build/prompts
See Loaders for using built prompts with FileLoader.

​
agentmark pull-models

Interactive command to pull and configure models from a provider. 
agentmark pull-models
This command opens an interactive prompt to:
  1. Select a model provider
  2. Choose models to enable
  3. Update your local configuration

​
Configuration Files

​
agentmark.json

Project configuration file in your project root: 
{
  "agentmarkPath": ".",
  "version": "1.0",
  "mdxVersion": "1.0"
}
FieldDescription
agentmarkPathBase path for agentmark files (contains the agentmark/ directory)
versionConfiguration version
mdxVersionMDX syntax version

​
.agentmark/dev-config.json

Auto-generated local development configuration (gitignored): 
{
  "webhookSecret": "...",
  "tunnelSubdomain": "agentmark-abc123",
  "createdAt": "2024-01-15T10:30:00.000Z",
  "appPort": 3000
}
This file stores:
  • Webhook secrets for tunnel authentication
  • Tunnel subdomain preferences
  • Current app port (updated when dev server starts)
The configuration expires after 30 days and is automatically regenerated.

​
Have Questions?

We’re here to help! Choose the best way to reach us: