Experimentation Maturity & Data Trust Assessment

Welcome! This survey explores your experimentation practices and confidence in results. It takes approximately 11 minutes to complete. Your participation is entirely voluntary, and you may stop at any time. There are no right or wrong answers—we are interested in your honest experience. All responses are confidential, anonymized, and reported only in aggregate to improve experimentation practices.

How often are experiments (e.g., A/B tests, feature experiments) part of your work?

• Regularly (monthly or more)
• Occasionally (quarterly)
• Rarely (yearly or less)
• Never

Which platforms or approaches do you currently use for experimentation? Select all that apply.

• In-house experimentation framework
• Feature flag platform (e.g., LaunchDarkly, Flagsmith)
• Third-party A/B tool (e.g., Optimizely, VWO, AB Tasty)
• SQL / notebooks only (no dedicated tool)
• Dashboarding tool (e.g., internal BI)
• None of the above
• Other (please specify)

In the last 3 months, approximately how many experiments did you help design, run, or analyze?

• 0
• 1–2
• 3–5
• 6–10
• 11–20
• More than 20

How easy or difficult is it to set up a standard A/B test using your current tools and processes?

How easy or difficult is it to analyze a completed experiment and interpret its results?

Which of the following quality controls are currently enforced in your experimentation workflow? Select all that apply.

• Pre-launch checklist
• Blocking deployment on missing instrumentation
• Automated SRM (sample ratio mismatch) alerting
• Sequential testing / alpha spending
• Max exposure or blast-radius limits
• Quality gates for key metrics
• Post-experiment QA template
• None of the above
• Other (please specify)

What is the primary decision rule your team uses to determine whether an experiment's results are conclusive?

• Fixed p-value threshold (e.g., 0.05)
• Bayesian decision rule
• Business threshold / minimum detectable effect
• Case-by-case judgement
• No standard rule / not sure
• Other (please specify)

How much do you trust your organization's experiment results to inform product decisions?

How confident are you in acting on an experiment's outcome to make a product or business decision?

What, if anything, most undermines your trust in experiment results today? Please share specifics.

Rank the following blockers to reliable experimentation from biggest (top) to smallest (bottom).

1. Data quality / instrumentation issues
2. Metric definitions ambiguity
3. Sample contamination / overlap
4. Insufficient traffic / power
5. Engineering constraints / time
6. Organizational pressure to ship

Rank the following phases of a typical experiment by how much effort they require (most effort at top).

1. Planning and design
2. Instrumentation and data validation
3. Implementation and rollout setup
4. Running and monitoring
5. Analysis and interpretation
6. Decision and rollout
7. Documentation and communication

What one change would most improve your experimentation workflow?

The next two questions are for those who do not currently run experiments. If you do run experiments, please skip ahead.

What are the main reasons you do not currently run experiments? Select all that apply.

• Not enough traffic to test
• Missing instrumentation / metrics
• Tooling is hard to use
• Unclear process or approvals
• Lack of statistical support
• Feature timelines too tight
• We prioritize other methods (e.g., user research)
• Other (please specify)

What resources, tools, or support would help you start running experiments confidently?

Based on your responses, we'd like to explore your experimentation experience in a bit more depth. Please share your thoughts openly—an AI moderator may ask a follow-up question or two.

What is your primary role?

• Product manager
• Engineer
• Data scientist / analyst
• Designer / UX
• Growth / marketing
• Other (please specify)

Which team are you primarily part of?

• Core product
• Platform / infrastructure
• Growth / monetization
• Data / analytics
• Other / cross-functional

How many years have you been involved in running or analyzing experiments?

• Less than 1 year
• 1–2 years
• 3–5 years
• 6–9 years
• 10+ years

Where are you primarily located?

• Americas
• EMEA
• APAC
• Prefer not to say

Approximately how many employees are in your company?

• 1–49
• 50–249
• 250–999
• 1,000–4,999
• 5,000+
• Prefer not to say

All set—thank you for sharing your perspective! Your responses will help us identify ways to improve experimentation practices across the organization.