Datasets

​

Creating Datasets

In Puzzlet, you can create datasets directly through the user interface (UI), or locally as a JSON object.

​

Dataset Items

Each item in a dataset must have the following components:

• Unique Name: A distinct identifier for the dataset item. This is recommended by Puzzlet to differentiate between items within the dataset.
• Input: This represents the input props that the prompt expects.
• Expected Output: The expected output from the prompt when the given input is provided. This serves as the ground truth for evaluating the prompt’s performance.

Here’s an example of how dataset might look:

{
  "items": [
    {
      "name": "item_1",
      "input": "What is the capital of France?",
      "expected_output": "Paris"
    },
    {
      "name": "item_2",
      "input": "Translate 'Hello' to French.",
      "expected_output": "Bonjour"
    }
  ]
}

​

Supported Formats

Currently, Puzzlet only supports datasets in JSON format.

​

Run Datasets

​

Bulk Testing

Once you have created a dataset, you can run it in bulk against your prompts. This allows you to evaluate how well your prompt performs across all the items in the dataset. Puzzlet will execute each item in the dataset and compare the actual output of the prompt with the expected output.

​

Viewing Results

After running a dataset, you can view the results of each run. Puzzlet provides detailed insights, including:

• Inputs and Outputs: For each item in the dataset, you can see the input provided to the prompt and the output generated by the prompt.

• Traces: You can also view the traces associated with each run item. Traces provide a detailed breakdown of the prompt’s execution, helping you understand how the output was generated.

​

Webhook Integration

Puzzlet gives you full control over the inference process by allowing you to set up a webhook. The webhook is responsible for calling the inference when a dataset is run. This setup ensures that you can integrate Puzzlet with your existing infrastructure seamlessly.

To learn more about setting up a webhook, check out the documentation.