Running prompts - AgentMark Docs

Cloud
Local

Run from the Dashboard

Open any prompt in the Dashboard editor, fill in your input variables, and click Run. Results stream back in real time.

When the run finishes, the output pane streams the response and the footer shows the tokens used, the cost, and the model that ran.AgentMark traces every run automatically. Open Traces to see the execution timeline, token usage, cost, and model for each run.

Run from the Playground

The Playground lets you run the same prompt across multiple models and parameter configurations side-by-side. Compare outputs, tweak prompt text per variant, and apply the winning configuration back to your editor.

CLI usage

Run prompts from the command line for quick testing during development.

agentmark run-prompt agentmark/greeting.prompt.mdx

Requires the development server running (agentmark dev).

Passing props

Inline JSON:

agentmark run-prompt agentmark/greeting.prompt.mdx \
  --props '{"name": "Alice"}'

From file:

agentmark run-prompt agentmark/greeting.prompt.mdx \
  --props-file ./test-data.json

Output examples

Text generation:

=== Text Prompt Results ===
Once upon a time...

────────────────────────────────────────────────────────────
🪙 250 in, 100 out, 350 total
────────────────────────────────────────────────────────────

📊 View trace: http://localhost:3000/traces?traceId=<id>

Object generation:

=== Object Prompt Results ===
{
  "name": "John Smith",
  "email": "john@example.com"
}

────────────────────────────────────────────────────────────
🪙 180 in, 45 out, 225 total
────────────────────────────────────────────────────────────

📊 View trace: http://localhost:3000/traces?traceId=<id>

Image and speech generation, saved to .agentmark-outputs/ in your project (paths print absolute):

=== Image Prompt Results ===
Saved 2 image(s) to:
- /path/to/your-project/.agentmark-outputs/image-1-1698765432.png
- /path/to/your-project/.agentmark-outputs/image-2-1698765432.png

SDK usage

AgentMark’s client renders a prompt to its neutral shape (for a text prompt, { messages, text_config }), which you pass to whatever LLM SDK you already use. The pattern is always:

Load the prompt with client.loadTextPrompt() (or loadObjectPrompt, loadImagePrompt, loadSpeechPrompt)
Format with props (and optionally telemetry) to get the neutral render
Call your SDK, mapping text_config.model_name to a provider model

Model resolution happens at your call site, not on the client. See Client setup for how the client is set up.

Address a prompt by its path relative to your agentmark/ prompts root: <name>.prompt.mdx, with no agentmark/ prefix. The prefix belongs to the CLI (agentmark run-prompt agentmark/<name>.prompt.mdx takes a file path from the project root); the SDK key resolves against the prompts root, so adding the prefix doubles it (agentmark/agentmark/...) and 404s. The local file loader also accepts the bare slug (<name>), but the cloud loader is strict and matches the full .prompt.mdx path, so write that form from the start (code that works locally on the short form 404s against Cloud).

Text generation

TypeScript
Python

import { client } from './agentmark.client';
import { generateText } from 'ai';
import { openai } from '@ai-sdk/openai';

const prompt = await client.loadTextPrompt('greeting.prompt.mdx');
const { messages, text_config } = await prompt.format({
  props: { name: 'Alice' }
});

const result = await generateText({
  model: openai(text_config.model_name.replace(/^openai\//, '')),
  messages,
});
console.log(result.text);

from agentmark_client import client
from openai import OpenAI

openai = OpenAI()

prompt = await client.load_text_prompt("greeting.prompt.mdx")
formatted = await prompt.format(props={"name": "Alice"})

result = openai.chat.completions.create(
    model=formatted.text_config.model_name.removeprefix("openai/"),
    messages=formatted.messages,
)
print(result.choices[0].message.content)

Streaming

Pass the neutral render to your SDK’s streaming call. With the Vercel AI SDK, that’s streamText() and streamObject():

TypeScript
Python

import { client } from './agentmark.client';
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';

const prompt = await client.loadTextPrompt('story.prompt.mdx');
const { messages, text_config } = await prompt.format({
  props: { topic: 'space exploration' }
});

const result = streamText({
  model: openai(text_config.model_name.replace(/^openai\//, '')),
  messages,
});

for await (const chunk of result.textStream) {
  process.stdout.write(chunk);
}

For structured output:

import { streamObject, jsonSchema } from 'ai';
import { openai } from '@ai-sdk/openai';

const prompt = await client.loadObjectPrompt('extract-data.prompt.mdx');
const { messages, object_config } = await prompt.format({
  props: { input: 'Contact John Smith at john@example.com' }
});

const result = streamObject({
  model: openai(object_config.model_name.replace(/^openai\//, '')),
  messages,
  schema: jsonSchema(object_config.schema),
});

for await (const partial of result.partialObjectStream) {
  console.log(partial);
}

from agentmark_client import client
from openai import OpenAI

openai = OpenAI()

prompt = await client.load_text_prompt("story.prompt.mdx")
formatted = await prompt.format(props={"topic": "space exploration"})

stream = openai.chat.completions.create(
    model=formatted.text_config.model_name.removeprefix("openai/"),
    messages=formatted.messages,
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Object generation

Object prompts return object_config (including the output schema) alongside messages. Pass the schema to your SDK’s structured-output call:

TypeScript
Python

import { client } from './agentmark.client';
import { generateObject, jsonSchema } from 'ai';
import { openai } from '@ai-sdk/openai';

const prompt = await client.loadObjectPrompt('extract-data.prompt.mdx');
const { messages, object_config } = await prompt.format({
  props: { input: 'Contact John Smith at john@example.com' }
});

const result = await generateObject({
  model: openai(object_config.model_name.replace(/^openai\//, '')),
  messages,
  schema: jsonSchema(object_config.schema),
});
console.log(result.object);
// { name: "John Smith", email: "john@example.com" }

import json
from agentmark_client import client
from openai import OpenAI

openai = OpenAI()

prompt = await client.load_object_prompt("extract-data.prompt.mdx")
formatted = await prompt.format(
    props={"input": "Contact John Smith at john@example.com"}
)

result = openai.chat.completions.create(
    model=formatted.object_config.model_name.removeprefix("openai/"),
    messages=formatted.messages,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "response",
            "schema": formatted.object_config.schema_,
            "strict": True,
        },
    },
)
print(json.loads(result.choices[0].message.content or "{}"))

The Python attribute is schema_ (trailing underscore), not schema. Plain schema collides with Pydantic’s deprecated BaseModel.schema() method, so reading .schema hands you that method ('method' object is not subscriptable the moment you index it), not your JSON schema. The frontmatter key stays schema; only the Python attribute carries the underscore.

Image generation

Image prompts have no messages: the render returns image_config with the rendered image description in image_config.prompt. Map image_config.model_name to your provider and pass the description as the prompt. With the Vercel AI SDK that’s experimental_generateImage (experimental_ prefix is from the Vercel AI SDK):

import { client } from './agentmark.client';
import { experimental_generateImage as generateImage } from 'ai';
import { openai } from '@ai-sdk/openai';
import fs from 'fs';

const prompt = await client.loadImagePrompt('logo.prompt.mdx');
const { image_config } = await prompt.format({
  props: { company: 'Acme Corp', style: 'modern' }
});

const result = await generateImage({
  model: openai.image(image_config.model_name.replace(/^openai\//, '')),
  prompt: image_config.prompt,
});
result.images.forEach((image, i) => {
  fs.writeFileSync(`logo-${i}.png`, image.uint8Array);
});

The image generation example targets the Vercel AI SDK. No Python equivalent yet.

Speech generation

Speech prompts have no messages either: the render returns speech_config with the rendered text to speak in speech_config.text (any <System> content lands in speech_config.instructions). Map speech_config.model_name to your provider and pass the text. With the Vercel AI SDK that’s experimental_generateSpeech:

import { client } from './agentmark.client';
import { experimental_generateSpeech as generateSpeech } from 'ai';
import { openai } from '@ai-sdk/openai';
import fs from 'fs';

const prompt = await client.loadSpeechPrompt('narration.prompt.mdx');
const { speech_config } = await prompt.format({
  props: { script: 'Welcome to our podcast' }
});

const result = await generateSpeech({
  model: openai.speech(speech_config.model_name.replace(/^openai\//, '')),
  text: speech_config.text,
});
fs.writeFileSync('narration.mp3', result.audio.uint8Array);

The speech generation example targets the Vercel AI SDK. No Python equivalent yet.

Using other SDKs

The examples above call the Vercel AI SDK, but the prompt.format() → pass-to-SDK pattern is identical for any SDK. Only the generation call and the model mapping differ. For text and object prompts the neutral render gives you messages plus the matching config (text_config or object_config); image and speech prompts return the rendered content inside their config (image_config.prompt, speech_config.text). Either way, you map model_name to a provider model and call your SDK:

Vercel AI SDK: generateText(), generateObject(), streamText(), streamObject()
OpenAI (raw SDK): chat.completions.create()
Anthropic (raw SDK): messages.create() (split the system message out of messages)
Agent frameworks (Pydantic AI, Mastra, Claude Agent SDK): run your agent loop over messages, return its final output
Custom: any provider or hand-rolled HTTP client

To have AgentMark Cloud run these prompts instead (the Dashboard Run button and managed experiments), see Connect your SDK.

Tracing prompt runs

Enable telemetry to trace every prompt execution. Traces capture input/output, token usage, cost, latency, and custom metadata. Initialize tracing once with the AgentMark SDK, pass telemetry to format(), and your model call is captured. registerGlobally: true is required so AgentMark sees the model-call span your SDK emits.

import { AgentMarkSDK } from '@agentmark-ai/sdk';
import { client } from './agentmark.client';
import { generateText } from 'ai';
import { openai } from '@ai-sdk/openai';

const sdk = new AgentMarkSDK({
  apiKey: process.env.AGENTMARK_API_KEY!,
  appId: process.env.AGENTMARK_APP_ID!,
});
sdk.initTracing({ registerGlobally: true });

const prompt = await client.loadTextPrompt('greeting.prompt.mdx');
const { messages, text_config } = await prompt.format({
  props: { name: 'Alice' },
  telemetry: {
    isEnabled: true,
    functionId: 'greeting-handler',
    // user_id must be snake_case in metadata to populate the trace's user field
    metadata: {
      user_id: 'user-123',
      environment: 'production'
    }
  }
});

const result = await generateText({
  model: openai(text_config.model_name.replace(/^openai\//, '')),
  messages,
});

View traces locally at http://localhost:3000 or in the Dashboard under Traces. See Tracing Setup for the full API.

Caching

Load the same prompt twice within 60 seconds and the second call returns instantly from cache, with no network request. The API loader caches each loaded prompt for 60 seconds by default (its time-to-live, or TTL). This applies to both ApiLoader.cloud() and ApiLoader.local(); FileLoader, which reads pre-built prompts from disk, does not cache.Caching is automatic, with no configuration needed, though the TTL is configurable via the loader’s cache.ttl option. After the TTL expires, the next request re-fetches from the server.

Troubleshooting

Server connection error: ensure agentmark dev is running. Check ports 9417 and 9418 are available.File not found: verify the file path and .prompt.mdx extension.Invalid JSON in props: use valid JSON with double quotes.

Next steps

Running experiments

Test prompts against datasets

Generation types

Text, objects, images, and audio

Version control

Track changes and rollback to previous versions

Run from AgentMark Cloud

Let the Dashboard and experiments run your prompts

Have questions?

Reach out any time:

Email the team at hello@agentmark.co for support
Schedule an Enterprise Demo to learn about AgentMark’s business solutions

​Run from the Dashboard

​Run from the Playground

​CLI usage

​Passing props

​Output examples

​SDK usage

​Text generation

​Streaming

​Object generation

​Image generation

​Speech generation

​Using other SDKs

​Tracing prompt runs

​Caching

​Troubleshooting

​Next steps

Running experiments

Generation types

Version control

Run from AgentMark Cloud

​Have questions?

Run from the Dashboard

Run from the Playground

CLI usage

Passing props

Output examples

SDK usage

Text generation

Streaming

Object generation

Image generation

Speech generation

Using other SDKs

Tracing prompt runs

Caching

Troubleshooting

Next steps

Have questions?