Create Experiment

Create an A/B experiment comparing two configurations

POST /v1/experiments

cURL

curl -X POST https://api.playgent.com/v1/experiments \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "System Prompt v2 Test",
    "agent_id": "agent_abc123",
    "variants": [
      {
        "name": "control",
        "config": { "system_prompt": "You are a helpful agent..." }
      },
      {
        "name": "treatment",
        "config": { "system_prompt": "You are a customer support agent for Acme Corp..." }
      }
    ],
    "test_case_ids": ["tc_1", "tc_2", "tc_3"],
    "metrics": ["relevance", "faithfulness", "latency_ms"],
    "config": {
      "runs_per_variant": 10,
      "randomize": true
    }
  }'

Response

{
  "experiment_id": "exp_yza567",
  "status": "completed",
  "results": {
    "control": {
      "runs": 10,
      "pass_rate": 0.70,
      "avg_scores": { "relevance": 0.85, "faithfulness": 0.82 },
      "avg_latency_ms": 2340
    },
    "treatment": {
      "runs": 10,
      "pass_rate": 0.90,
      "avg_scores": { "relevance": 0.91, "faithfulness": 0.94 },
      "avg_latency_ms": 2180
    }
  },
  "statistical_analysis": {
    "winner": "treatment",
    "confidence": 0.95,
    "p_value": 0.023,
    "effect_size": {
      "pass_rate": "+28.6%",
      "faithfulness": "+14.6%"
    }
  },
  "recommendation": "Treatment variant shows statistically significant improvement. Recommend deploying."
}

Create an A/B experiment to compare different agent configurations with statistical analysis and recommendations.

name · string · required

Experiment name

agent_id · string · required

Base agent for the experiment

variants · array · required

Experiment variants (minimum 2)

Show variant object

name · string · required

Variant name (e.g., “control”, “treatment”)

config · object · required

Variant configuration

Show properties

system_prompt · string

Override system prompt

model · string

Override model

temperature · number

Override temperature

test_case_ids · array · required

Test cases to run in the experiment

metrics · array · required

Metrics to measure (e.g., “relevance”, “faithfulness”, “latency_ms”)

config · object

Experiment configuration

Show properties

runs_per_variant · integer

Runs per variant for statistical power (default: 10)

randomize · boolean

Randomize run order (default: true)

experiment_id · string · required

Experiment identifier

status · string · required

Status: pending, running, completed

results · object

Results per variant (when completed)

Show variant result

runs · integer

Number of runs

pass_rate · number

Pass rate

avg_scores · object

Average scores per metric

avg_latency_ms · integer

Average latency

statistical_analysis · object

Statistical analysis (when completed)

Show properties

winner · string

Winning variant

confidence · number

Confidence level (0-1)

p_value · number

P-value

effect_size · object

Effect sizes per metric

recommendation · string

Human-readable recommendation

cURL

curl -X POST https://api.playgent.com/v1/experiments \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "System Prompt v2 Test",
    "agent_id": "agent_abc123",
    "variants": [
      {
        "name": "control",
        "config": { "system_prompt": "You are a helpful agent..." }
      },
      {
        "name": "treatment",
        "config": { "system_prompt": "You are a customer support agent for Acme Corp..." }
      }
    ],
    "test_case_ids": ["tc_1", "tc_2", "tc_3"],
    "metrics": ["relevance", "faithfulness", "latency_ms"],
    "config": {
      "runs_per_variant": 10,
      "randomize": true
    }
  }'

Response

{
  "experiment_id": "exp_yza567",
  "status": "completed",
  "results": {
    "control": {
      "runs": 10,
      "pass_rate": 0.70,
      "avg_scores": { "relevance": 0.85, "faithfulness": 0.82 },
      "avg_latency_ms": 2340
    },
    "treatment": {
      "runs": 10,
      "pass_rate": 0.90,
      "avg_scores": { "relevance": 0.91, "faithfulness": 0.94 },
      "avg_latency_ms": 2180
    }
  },
  "statistical_analysis": {
    "winner": "treatment",
    "confidence": 0.95,
    "p_value": 0.023,
    "effect_size": {
      "pass_rate": "+28.6%",
      "faithfulness": "+14.6%"
    }
  },
  "recommendation": "Treatment variant shows statistically significant improvement. Recommend deploying."
}