Domain Context
This skill implements a proven product management framework. The approach combines best practices from industry leaders and is designed for practical application in day-to-day PM work.
Input Requirements
- Context about your product, feature, or problem
- Relevant data, research, or constraints (recommended but optional)
- Clear articulation of what you're trying to achieve
A/B Test Designer
When to Use
- Testing a new feature or design variation
- Validating a hypothesis before full rollout
- Optimizing conversion rates or key metrics
- Choosing between multiple design approaches
- Need to make a data-driven decision on a change
What This Skill Does
Helps you design rigorous A/B tests with clear hypotheses, success metrics, sample size calculations, and analysis plans.
Instructions
Help me design an A/B test for [feature/change]. Include:
-
Hypothesis
- Current situation and metrics
- Proposed change
- Expected impact and why
-
Test Design
- Primary success metric
- Secondary metrics
- Sample size needed
- Test duration
- User segments to include/exclude
-
Variants
- Control (A): current experience
- Variant (B): new experience
- Any additional variants (C, D, etc.)
-
Risks and Controls
- Potential negative impacts
- Guardrail metrics
- When to stop the test early
-
Analysis Plan
- Statistical significance threshold
- How to handle edge cases
- Decision criteria
Feature context: [Add context about the change you want to test]
Best Practices
- Start with a clear, falsifiable hypothesis
- Choose one primary metric to avoid multiple comparison issues
- Calculate sample size upfront based on expected effect size
- Run tests for full weekly cycles to account for day-of-week effects
- Set a minimum test duration (usually 1-2 weeks)
- Define success criteria before running the test
- Monitor guardrail metrics (revenue, errors, performance)
Example
Input: Testing new onboarding flow vs current 3-step process Output: Hypothesis (new 1-step flow will increase co...