Project config

AgentMark projects are configured through two main files: agentmark.json for project-level settings, and agentmark.client.ts (or agentmark_client.py) for runtime configuration like models, tools, and loaders.

agentmark.json

The agentmark.json file lives at your project root and configures your AgentMark application. It is read by both the CLI and AgentMark Cloud.

Basic example

agentmark.json

{
  "$schema": "https://raw.githubusercontent.com/agentmark-ai/agentmark/refs/heads/main/packages/cli/agentmark.schema.json",
  "version": "2.0.0",
  "mdxVersion": "1.0",
  "agentmarkPath": ".",
  "builtInModels": ["openai/gpt-4o"]
}

Configuration properties

$schema (optional)

Points to the JSON Schema for editor autocompletion and validation.

"$schema": "https://raw.githubusercontent.com/agentmark-ai/agentmark/refs/heads/main/packages/cli/agentmark.schema.json"

agentmarkPath (required)

The base directory (relative to your project root) where AgentMark looks for the agentmark/ folder containing prompts, components, and datasets. Projects scaffolded with npm create agentmark@latest use "." — the agentmark/ directory at the project root.

"agentmarkPath": "."

In a monorepo, set this to the relative path of the package containing your AgentMark files (e.g., "packages/ai").

Use "." (or a relative path like "packages/ai"), not "/". A leading slash resolves to the filesystem root and breaks agentmark build.

version (required)

The AgentMark configuration version. Use "2.0.0" for new projects. AgentMark Cloud uses this to choose the storage folder for deployed prompts — versions >= "2.0.0" use the agentmark/ folder, earlier versions use the legacy puzzlet/ folder.

mdxVersion (optional)

The prompt format version. Accepts "1.0" (current) or "0.0" (legacy). Use "1.0" for new projects.

builtInModels (optional)

An array of model IDs allowed in prompts. When set and non-empty, prompt-core rejects any prompt whose model_name is not in the list. IDs use the provider/model format (e.g., openai/gpt-4o) so the adapter’s model registry can auto-resolve the provider when you call .registerProviders({ openai, anthropic }). Pricing and settings for these models come from the bundled AgentMark model registry.

"builtInModels": ["openai/gpt-4o", "openai/gpt-4o-mini", "anthropic/claude-sonnet-4-20250514"]

Use the pull-models CLI command to interactively add models from supported providers — it emits the correct provider/model format automatically:

npx agentmark pull-models

See Model schemas for details.

evals (deprecated)

Use scores instead. The evals field listed evaluation function names but did not include schema definitions. It is still supported for backward compatibility.

"evals": ["correctness", "hallucination", "relevance"]

scores (optional)

Define score schemas for evaluation and human annotation. Each entry declares a score name and its type (boolean, numeric, or categorical). These schemas are synced to AgentMark Cloud through the deployment pipeline and used by both the annotation UI and experiment runner.

"scores": {
  "accuracy": {
    "type": "boolean",
    "description": "Was the response factually correct?"
  },
  "helpfulness": {
    "type": "numeric",
    "min": 1,
    "max": 5,
    "description": "Rate helpfulness on a 1-5 scale"
  },
  "tone": {
    "type": "categorical",
    "description": "Response tone",
    "categories": [
      { "label": "professional", "value": 1 },
      { "label": "casual", "value": 0.5 },
      { "label": "inappropriate", "value": 0 }
    ]
  }
}

To add automated eval functions for these scores, define them in your client config using the evals option. See Evaluations for details.

modelSchemas (optional)

Define custom model configurations with settings, pricing, and UI controls. Use this for models not covered by builtInModels, or to customize settings for existing models.

"modelSchemas": {
  "my-custom-model": {
    "label": "My Custom Model",
    "cost": {
      "inputCost": 0.01,
      "outputCost": 0.03,
      "unitScale": 1000000
    },
    "settings": {
      "temperature": {
        "label": "Temperature",
        "order": 1,
        "default": 0.7,
        "minimum": 0,
        "maximum": 2,
        "multipleOf": 0.1,
        "type": "slider"
      }
    }
  }
}

See Adding Models for the full schema reference.

mcpServers (optional)

Configure Model Context Protocol (MCP) servers that your prompts can reference as tools. Servers listed here are registered with the adapter at runtime (AI SDK, Mastra, Claude Agent SDK) and become available to prompts that reference them as mcp://<server-name>/<tool> in the tools: frontmatter.

URL / SSE
Stdio

For remote MCP servers accessible via HTTP:

"mcpServers": {
  "docs": {
    "url": "https://example.com/mcp",
    "headers": {
      "Authorization": "Bearer your-token"
    }
  }
}

For local MCP servers that run as a subprocess:

"mcpServers": {
  "local-tools": {
    "command": "node",
    "args": ["./mcp-server.js"],
    "cwd": "/path/to/server",
    "env": {
      "API_KEY": "secret"
    }
  }
}

See MCP Integration for usage in prompts.

handler (optional)

Path to your handler file for managed code deployment. AgentMark Cloud bundles and deploys this file so prompts can be executed from the Dashboard. The file extension determines the runtime (.py → Python, anything else → Node.js).

agentmark.json

"handler": "handler.ts"

If omitted, AgentMark Cloud checks for handler.py at the repository root first (Python runtime), then falls back to handler.ts (Node.js runtime). If neither is found, managed code deployment is skipped. Projects scaffolded with npm create agentmark@latest --cloud write this field automatically (handler.ts for TypeScript, handler.py for Python).

Full example

An illustrative config showing every top-level field (not all are written by the scaffolder — see each field’s section above for when it applies):

agentmark.json

{
  "$schema": "https://raw.githubusercontent.com/agentmark-ai/agentmark/refs/heads/main/packages/cli/agentmark.schema.json",
  "version": "2.0.0",
  "mdxVersion": "1.0",
  "agentmarkPath": ".",
  "builtInModels": ["openai/gpt-4o", "openai/gpt-4o-mini", "anthropic/claude-sonnet-4-20250514"],
  "scores": {
    "correctness": {
      "type": "boolean",
      "description": "Was the response correct?"
    },
    "hallucination": {
      "type": "boolean",
      "description": "Did the response contain hallucinated content?"
    },
    "helpfulness": {
      "type": "numeric",
      "min": 1,
      "max": 5,
      "description": "Rate helpfulness on a 1-5 scale"
    }
  },
  "handler": "handler.ts",
  "mcpServers": {
    "docs": {
      "url": "https://example.com/mcp"
    }
  },
  "modelSchemas": {
    "my-fine-tuned-model": {
      "label": "My Fine-tuned Model",
      "cost": {
        "inputCost": 0.005,
        "outputCost": 0.015,
        "unitScale": 1000000
      },
      "settings": {
        "temperature": {
          "label": "Temperature",
          "order": 1,
          "default": 0.7,
          "minimum": 0,
          "maximum": 2,
          "multipleOf": 0.1,
          "type": "slider"
        }
      }
    }
  }
}

Client configuration

The client configuration file (agentmark.client.ts or agentmark_client.py) defines your runtime setup: which models to use, what tools are available, how to load prompts, and which evaluations to run. This file is auto-generated by npm create agentmark@latest and can be customized for your project.

Cloud mode
Self-hosted mode

In Cloud mode, prompts are loaded from the AgentMark API in production and from your local dev server during development:

agentmark.client.ts

import { ApiLoader } from "@agentmark-ai/loader-api";

const loader = process.env.NODE_ENV === 'development'
  ? ApiLoader.local({
      baseUrl: process.env.AGENTMARK_BASE_URL || 'http://localhost:9418'
    })
  : ApiLoader.cloud({
      apiKey: process.env.AGENTMARK_API_KEY!,
      appId: process.env.AGENTMARK_APP_ID!,
    });

In self-hosted mode, prompts are loaded from pre-built files in production:

agentmark.client.ts

import { ApiLoader } from "@agentmark-ai/loader-api";
import { FileLoader } from "@agentmark-ai/loader-file";

const loader = process.env.NODE_ENV === 'development'
  ? ApiLoader.local({
      baseUrl: process.env.AGENTMARK_BASE_URL || 'http://localhost:9418'
    })
  : new FileLoader('./dist/agentmark');

Self-hosted mode requires running npx agentmark build --out dist/agentmark to pre-compile your prompts before deployment.

Environment variables

Variable	Required	Description
`AGENTMARK_API_KEY`	Cloud mode	API key from AgentMark Dashboard settings
`AGENTMARK_APP_ID`	Cloud mode	App ID from AgentMark Dashboard settings
`AGENTMARK_BASE_URL`	No	Override the local dev server URL in scaffolded clients (default: `http://localhost:9418`)
`OPENAI_API_KEY`	Depends on adapter	OpenAI API key for AI SDK, Mastra, or Pydantic AI adapters
`ANTHROPIC_API_KEY`	Depends on adapter	Anthropic API key for Claude Agent SDK adapter

See Environment variables for the complete list.

Have Questions?

We’re here to help! Choose the best way to reach us:

Email us at hello@agentmark.co for support
Schedule an Enterprise Demo to learn about our business solutions

Introduction

Getting Started

Build

Evaluate

Observe

Configure

Deploy

Integrations

agentmark.json

Basic example

Configuration properties

$schema (optional)

agentmarkPath (required)

version (required)

mdxVersion (optional)

builtInModels (optional)

evals (deprecated)

scores (optional)

modelSchemas (optional)

mcpServers (optional)

handler (optional)

Full example

Client configuration

Environment variables

Have Questions?

Introduction

Getting Started

Build

Evaluate

Observe

Configure

Deploy

Integrations

Documentation Index

​agentmark.json

​Basic example

​Configuration properties

​$schema (optional)

​agentmarkPath (required)

​version (required)

​mdxVersion (optional)

​builtInModels (optional)

​evals (deprecated)

​scores (optional)

​modelSchemas (optional)

​mcpServers (optional)

​handler (optional)

​Full example

​Client configuration

​Environment variables

​Have Questions?

agentmark.json

Basic example

Configuration properties

$schema (optional)

agentmarkPath (required)

version (required)

mdxVersion (optional)

builtInModels (optional)

evals (deprecated)

scores (optional)

modelSchemas (optional)

mcpServers (optional)

handler (optional)

Full example

Client configuration

Environment variables

Have Questions?