Skip to main content
The AgentMark CLI automatically detects and runs Python projects with the appropriate dev server configuration.

​
Starting the dev server

npx agentmark dev
The CLI detects Python projects and spawns the Python webhook server alongside the API server and UI.

​
Project detection

The CLI identifies Python projects by checking for:
  1. pyproject.toml - Python project manifest
  2. agentmark_client.py - AgentMark client configuration
  3. .agentmark/dev_server.py - Auto-generated entry point
If any of these files exist, the CLI runs in Python mode.

​
Virtual environment detection

The CLI automatically detects and uses virtual environments: 
Priority order:
1. .venv/bin/python (or .venv\Scripts\python.exe on Windows)
2. venv/bin/python (or venv\Scripts\python.exe on Windows)
3. System python
When a virtual environment is found, the CLI prints: 
Using virtual environment: .venv/

​
Entry point resolution

The dev server entry point is resolved in this order:
LocationDescription
dev_server.pyCustom dev server (project root)
.agentmark/dev_server.pyAuto-generated server

​
Custom dev server

Create a dev_server.py in your project root to customize the webhook server:
dev_server.py
"""Custom webhook server for AgentMark development."""

import argparse
import os
import sys
from pathlib import Path

# Mark this process as development BEFORE the client loads, so the client's
# loader switch picks the local dev server over the cloud.
os.environ.setdefault("NODE_ENV", "development")

sys.path.insert(0, str(Path(__file__).parent))

from agentmark_pydantic_ai_v0 import create_webhook_server
from agentmark_client import client

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--webhook-port", type=int, default=9417)
    parser.add_argument("--api-server-port", type=int, default=9418)
    args = parser.parse_args()

    create_webhook_server(client, args.webhook_port, args.api_server_port)

​
Auto-generated server

When you run npm create agentmark@latest, an entry point is created at .agentmark/dev_server.py:
.agentmark/dev_server.py
"""Auto-generated webhook server for AgentMark development."""

import argparse
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent.parent))

from agentmark_pydantic_ai_v0 import create_webhook_server
from agentmark_client import client

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--webhook-port", type=int, default=9417)
    parser.add_argument("--api-server-port", type=int, default=9418)
    args = parser.parse_args()

    create_webhook_server(client, args.webhook_port, args.api_server_port)

​
Environment variables

The dev server sets the following environment variables:
VariableValueDescription
PYTHONDONTWRITEBYTECODE1Prevents __pycache__ creation
PYTHONUNBUFFERED1Ensures real-time output
AGENTMARK_BASE_URLhttp://localhost:{api_port}API server URL for telemetry

​
Server architecture

When you run npx agentmark dev, three servers start: 
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   API Server    │────▶│ Webhook Server  │────▶│    UI Server    │
│   (port 9418)   │     │   (port 9417)   │     │   (port 3000)   │
│                 │     │                 │     │                 │
│  Telemetry API  │     │ Python Process  │     │    Next.js      │
│  Trace Storage  │     │ Prompt Executor │     │  Local dev UI   │
└─────────────────┘     └─────────────────┘     └─────────────────┘
The Python webhook server:
  • Receives prompt execution requests from the CLI
  • Uses your agentmark_client.py configuration
  • Executes prompts via the configured adapter (Pydantic AI or Claude Agent SDK)
  • Returns streaming or non-streaming responses

​
Port configuration

Override default ports with CLI options: 
npx agentmark dev --webhook-port 8080 --api-port 8081 --app-port 8082
OptionDefaultDescription
--webhook-port9417Webhook server port
--api-port9418API server port
--app-port3000UI server port

​
Webhook handler

The webhook server implements two event types:

​
prompt-run

Executes a single prompt: 
{
  "type": "prompt-run",
  "data": {
    "ast": { ... },
    "options": {
      "shouldStream": true
    },
    "customProps": { ... }
  }
}

​
dataset-run

Executes a prompt across a dataset: 
{
  "type": "dataset-run",
  "data": {
    "ast": { ... },
    "experimentId": "exp-123",
    "datasetPath": "./datasets/test.jsonl"
  }
}

​
Running prompts

With the dev server running, execute prompts from another terminal: 
npx agentmark run-prompt ./agentmark/<your-prompt>.prompt.mdx
Or run experiments: 
npx agentmark run-experiment ./agentmark/<your-prompt>.prompt.mdx
Need a working starter? See Example prompts: four copy-paste recipes (object, text+tools, image, speech) you can drop into your agentmark/ directory.

​
Troubleshooting

​
Virtual environment not found

If you see “python not found” errors: 
# Create a virtual environment
python -m venv .venv

# Activate it
source .venv/bin/activate  # macOS/Linux
.venv\Scripts\activate     # Windows

# Install dependencies directly — `pip install -e ".[dev]"` has known
# "resolution too deep" issues in scaffolded projects, so install the
# packages you need explicitly.
pip install agentmark-sdk agentmark-pydantic-ai-v0 agentmark-prompt-core python-dotenv

​
Module not found

Confirm dependencies are installed in the correct virtual environment: 
pip install agentmark-pydantic-ai-v0 agentmark-prompt-core python-dotenv

​
Port already in use

If ports are busy, specify alternative ports: 
npx agentmark dev --webhook-port 9500 --api-port 9501

​
agentmark_client.py not found

The CLI requires agentmark_client.py in your project root: 
# Create a new project
npm create agentmark@latest

# Or manually create agentmark_client.py

​
Adapter-specific considerations

​
Pydantic AI

The Pydantic AI dev server uses aiohttp for async HTTP handling: 
from agentmark_pydantic_ai_v0 import create_webhook_server

create_webhook_server(client, webhook_port=9417, api_server_port=9418)

​
Claude Agent SDK

The Claude Agent SDK dev server handles agentic execution: 
from agentmark_claude_agent_sdk_v0 import create_webhook_server

create_webhook_server(client, webhook_port=9417, api_server_port=9418)

​
Next steps

Python overview

Python SDK and project setup

Pydantic AI

Type-safe LLM interactions

Claude Agent SDK

Agentic task execution

Running prompts

Execute prompts from CLI

​
Have Questions?

We’re here to help! Choose the best way to reach us: