Python dev server - AgentMark Docs

The AgentMark CLI automatically detects and runs Python projects with the appropriate dev server configuration.

Starting the dev server

agentmark dev

The CLI detects Python projects and spawns the Python webhook server alongside the API server and UI.

Project detection

The CLI identifies Python projects by checking, in order:

AgentMark setup files: agentmark_client.py, .agentmark/dev_server.py, or a root dev_server.py. These are decisive: once you set up your project, an unrelated manifest file no longer overrides TypeScript/Python detection. (agentmark.client.ts is the equally decisive TypeScript marker.)
Python manifests: pyproject.toml, requirements.txt, or setup.py. These matter before any AgentMark files exist, so a fresh pip project gets Python guidance from doctor and dev on first contact.

If none match, the CLI assumes TypeScript.

Virtual environment detection

The CLI automatically detects and uses virtual environments:

Priority order:
.venv/bin/python (or .venv\Scripts\python.exe on Windows)
venv/bin/python (or venv\Scripts\python.exe on Windows)
System python

When the CLI finds a virtual environment, it prints:

Using virtual environment: .venv/

Entry point resolution

The CLI resolves the dev server entry point in this order:

Location	Description
`dev_server.py`	Custom dev server (project root)
`.agentmark/dev_server.py`	Standard entry point (created by you or your AI tool)

The entry point serves a webhook runner so the CLI and the local dev UI can execute your prompts (once deployed, the same handler serves the Dashboard). You build the runner once from an executor (the function that calls your LLM SDK), and the runner’s dispatch method handles every job it receives (prompt-run, dataset-run, and get-evals, which lists the evals registered on your client).

Neutral client and runner

The entry point builds a webhook runner from three pieces and serves it. Keep agentmark_client.py minimal (loader, client, and your evals) and put the executor, tracing init, and runner in .agentmark/dev_server.py:

Client (agentmark_client.py): create_agentmark(loader=ApiLoader.local(base_url="http://localhost:9418")), plus your evals.
Executor + runner (.agentmark/dev_server.py): write a create_executor for your SDK, then runner = create_webhook_runner(client, executor). The full executor contract and copy-paste handlers (object, streaming, every SDK) live on Connect your SDK.
Serve: serve_webhook_runner(runner) (see Entry-point file).

For the complete agentmark_client.py + .agentmark/dev_server.py files, see Client setup.

Local tracing differs from production. The runner wires span hooks, but spans only export once you initialize tracing, and in local dev you point the exporter at the dev API server (unauthenticated, no cloud keys) so agentmark doctor --smoke and the local trace UI see your runs:

import os
from agentmark_sdk import AgentMarkSDK

AgentMarkSDK(
    api_key="local-dev",
    app_id="local-dev",
    base_url=os.environ.get("AGENTMARK_DEV_SERVER", "http://localhost:9418"),
).init_tracing(disable_batch=True)

See Tracing setup for the production configuration.

Entry-point file

The CLI boots .agentmark/dev_server.py (or dev_server.py at your project root) and passes --webhook-port / --api-server-port. It strips cloud credentials from the spawned environment, so a client that picks its loader by API-key presence (the client setup pattern) lands on the local loader; the example above pins ApiLoader.local directly, which behaves the same in local dev. The entry point must keep serving HTTP: serve_webhook_runner(runner) is the line that does that. It serves runner.dispatch, the same handler a managed deployment exposes as handler = runner.dispatch. Without it the process builds the runner, exits, and agentmark dev reports Webhook server stopped. serve_webhook_runner is the Python counterpart of the TypeScript createWebhookServer (@agentmark-ai/cli/runner-server).

npm create agentmark@latest scaffolds agentmark.json, the agentmark/ prompt directory, MCP configs for your editor, and the AgentMark agent skill (plus git init in new directories). It doesn’t create the client or dev-server entry: you (or the agent skill) write agentmark_client.py and .agentmark/dev_server.py. See Client setup for the full dev-server and handler.py files.

Environment variables

The CLI sets the following environment variables for the spawned dev_server.py:

Variable	Value	Description
`AGENTMARK_DEV_SERVER`	`http://localhost:{api_port}`	Local API server URL; the example entry point reads it to point the tracing exporter
`PYTHONPATH`	project root (prepended)	Lets `.agentmark/dev_server.py` import project-root modules like `agentmark_client`
`PYTHONDONTWRITEBYTECODE`	`1`	Prevents `__pycache__` creation
`PYTHONUNBUFFERED`	`1`	Ensures real-time output
`PYTHONPYCACHEPREFIX`	per-run temp directory	Keeps stale bytecode from masking source edits between restarts

The CLI also removes AGENTMARK_API_KEY, AGENTMARK_APP_ID, and AGENTMARK_BASE_URL from the spawned process, so a client that picks its loader by API-key presence stays in local mode instead of loading deployed prompts from the Cloud.

Server architecture

When you run agentmark dev, three servers start:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   API Server    │────▶│ Webhook Server  │────▶│    UI Server    │
│   (port 9418)   │     │   (port 9417)   │     │   (port 3000)   │
│                 │     │                 │     │                 │
│  Telemetry API  │     │ Python Process  │     │    Next.js      │
│  Trace Storage  │     │ Prompt Executor │     │  Local dev UI   │
└─────────────────┘     └─────────────────┘     └─────────────────┘

The Python webhook server:

Receives prompt execution requests from the CLI
Uses your agentmark_client.py configuration
Executes prompts through your executor, which calls your LLM SDK
Returns streaming or non-streaming responses

Port configuration

Override default ports with CLI options:

agentmark dev --webhook-port 8080 --api-port 8081 --app-port 8082

Option	Default	Description
`--webhook-port`	9417	Webhook server port
`--api-port`	9418	API server port
`--app-port`	3000	UI server port

Wire contract

You normally never implement this; serve_webhook_runner(runner) is the whole server. The contract below is for custom entry points (your own framework, extra routes) and for debugging with curl. Every job arrives as POST / with a JSON {type, data} body. Three event types exist:

`prompt-run`

Executes a single prompt:

{
  "type": "prompt-run",
  "data": {
    "ast": { ... },
    "options": {
      "shouldStream": true
    },
    "customProps": { ... }
  }
}

`dataset-run`

Executes a prompt across a dataset:

{
  "type": "dataset-run",
  "data": {
    "ast": { ... },
    "experimentId": "exp-123",
    "datasetPath": "./datasets/test.jsonl"
  }
}

get-evals

Control-plane job: lists the eval names registered on your client, which populates the Dashboard’s New Experiment dialog. Carries no AST; the response is a flat JSON body {"type": "evals", "result": [...names], "traceId": null}.

Responses

The CLI (run-prompt, run-experiment) and the Dashboard switch parsing on one response header:

Streaming (runner.dispatch returned a result with a stream): respond with AgentMark-Streaming: true and Content-Type: application/x-ndjson, then write the stream’s NDJSON lines as they arrive (one JSON event per line: text / object / dataset / error chunks). After the stream drains, append a final {"type": "done", "traceId": "..."} line when the result carries a traceId; that’s where the CLI reads the trace link from.
Non-streaming: respond 200 with the dispatch result as a plain JSON body.
Errors: respond with a non-2xx status and a {"message": "..."} JSON body (400 for malformed/unknown jobs, 500 for executor failures).

This is the same contract the TypeScript dev server (createWebhookServer) and the managed-deployment server implement; serve_webhook_runner keeps the Python side pinned to it.

Running prompts

With the dev server running, execute prompts from another terminal:

agentmark run-prompt ./agentmark/<your-prompt>.prompt.mdx

Or run experiments:

agentmark run-experiment ./agentmark/<your-prompt>.prompt.mdx

Need a working starter? See Example prompts: four copy-paste recipes (object, text+tools, image, speech) you can drop into your agentmark/ directory.

Troubleshooting

`agentmark doctor` reports missing dependencies

agentmark doctor checks .venv/, then venv/, then the system pip, in that priority order, so it correctly finds packages installed inside a virtual environment. If you still see a missing-package warning after installing into your venv, confirm the venv pip resolves the packages:

.venv/bin/pip show agentmark-prompt-core agentmark-sdk

Virtual environment not found

If you see “python not found” errors:

# Create a virtual environment
python -m venv .venv

# Activate it
source .venv/bin/activate  # macOS/Linux
.venv\Scripts\activate     # Windows

# Install dependencies (plus your own LLM SDK, e.g. openai)
pip install agentmark-prompt-core agentmark-sdk python-dotenv

Module not found

Confirm you installed dependencies in the correct virtual environment:

pip install agentmark-prompt-core agentmark-sdk python-dotenv

Port already in use

If ports are busy, specify alternative ports:

agentmark dev --webhook-port 9500 --api-port 9501

`agentmark_client.py` not found

The CLI requires agentmark_client.py in your project root:

# Create a new project
npm create agentmark@latest

# Or manually create agentmark_client.py

Agent frameworks

An agent-framework executor works the same in the dev server. Your handler runs an agent loop instead of a single completion, but it’s still a create_executor wired into the runner above. See Connect your SDK for the agent-loop executor, including streaming with tool-call events.

Next steps

Client setup

Python client, dev server, and deploy handler files

Connect your SDK

Wire any LLM SDK with an executor

Reference executors

Copy-paste executors for OpenAI, Anthropic, agent frameworks

Running prompts

Execute prompts from CLI

Have questions?

Reach out any time:

Email the team at hello@agentmark.co for support
Schedule an Enterprise Demo to learn about AgentMark’s business solutions

​Starting the dev server

​Project detection

​Virtual environment detection

​Entry point resolution

​Neutral client and runner

​Entry-point file

​Environment variables

​Server architecture

​Port configuration

​Wire contract

​prompt-run

​dataset-run

​get-evals

​Responses

​Running prompts

​Troubleshooting

​agentmark doctor reports missing dependencies

​Virtual environment not found

​Module not found

​Port already in use

​agentmark_client.py not found

​Agent frameworks

​Next steps

Client setup

Connect your SDK

Reference executors

Running prompts

​Have questions?

Starting the dev server

Project detection

Virtual environment detection

Entry point resolution

Neutral client and runner

Entry-point file

Environment variables

Server architecture

Port configuration

Wire contract

`prompt-run`

`dataset-run`

get-evals

Responses

Running prompts

Troubleshooting

`agentmark doctor` reports missing dependencies

Virtual environment not found

Module not found

Port already in use

`agentmark_client.py` not found

Agent frameworks

Next steps

Have questions?