A/B Decision Testing Kit

Run controlled experiments on agent decision strategies with statistical significance.

Configuration

Test type

Auto-promote winner

A/B Decision Testing Dashboard

Active Tests

0/2

Traffic Split

50/50

P-value

Variant AConservative Strategy

Samples: 0Accuracy: 0%

Variant BAggressive Strategy

Samples: 0Accuracy: 0%

Integration Code

import { createABTest } from 'agent-tools-kit/experimentation'

const test = createABTest({
  type: 'bayesian',
  metric: 'accuracy',
  minSamples: 300,
  significanceLevel: 0.05,
})

await test.start({
  variantA: { policy: conservativePolicy },
  variantB: { policy: aggressivePolicy },
  trafficSplit: '50/50',
})

// Monitor real-time results
test.on('update', (results) => {
  console.log('Variant A:', results.a.accuracy)
  console.log('Variant B:', results.b.accuracy)
  console.log('P-value:', results.pValue)
})

const winner = await test.waitForSignificance()
console.log('Winner:', winner.variant, winner.accuracy)