A/B Experimentation Trust & Data Quality Assessment

Welcome to the Experimentation Trust & Quality Survey. We're gathering candid feedback on how A/B test results are used and trusted across the organization. Your responses are confidential and will be reported only in aggregate — there are no right or wrong answers. Participation is voluntary, and you may exit at any time. The survey takes approximately 12 minutes. Results will be used internally to improve our experimentation practices and communication.

Which functional areas best describe your role? (Select up to three.)

• Product Management
• Engineering
• Data Science / Analytics
• Design / UX
• Marketing / Growth
• Operations / Support
• Leadership / Strategy
• Other

In the last 6 months, how often have you reviewed or acted on A/B test results?

• Weekly or more
• 1 to 3 times per month
• A few times total
• Not in the last 6 months
• Never

The following questions are for those who have not actively used A/B test results recently. If you regularly work with test results, you may skip ahead.

Based on your general impression, how reliable are our A/B test results overall?

What limits your use of A/B test results today? (Select all that apply.)

• Hard to access results
• Unsure how to interpret results
• Don't trust the data quality
• Not relevant to my work
• No tests run in my area
• Lack of time
• Other

How useful would a short guide explaining key experimentation concepts (e.g., statistical power, minimum detectable effect, confidence intervals) be for your work?

The following questions are for those who have actively worked with A/B test results in the past 3–6 months.

Approximately how many distinct A/B tests did you work on or review results from in the last 3 months?

• 1–2
• 3–5
• 6–10
• 11–20
• More than 20

Where are the A/B tests you work with primarily run? (Select all that apply.)

• Web
• iOS app
• Android app
• Backend systems
• Marketing channels (email / ads)
• Other

How much do you trust the validity of our A/B test conclusions over the past 3 months?

How often do A/B test results meaningfully change your team's decisions?

• Almost always
• Often
• Sometimes
• Rarely
• Almost never

In the past 3 months, have you observed flaky or inconsistent A/B test outcomes on key metrics?

• No
• Yes, occasionally
• Yes, frequently
• Unsure

If you observed flaky or inconsistent outcomes, please share one or two examples and what you think caused them.

How often do each of the following contribute to flaky or unreliable A/B test results in your area?

• Insufficient sample size or test duration
• Instrumentation or logging bugs
• Peeking at results before reaching significance
• Interactions between concurrent experiments
• Unstable or delayed data pipelines
• Poorly defined or overly sensitive metrics
• External events or seasonality
• Other

How clearly do shipped experiment reports communicate uncertainty (e.g., confidence intervals, statistical significance)?

Before launch, how often are minimum detectable effect (MDE) and statistical power planned explicitly for experiments?

• Always
• Often
• Sometimes
• Rarely
• Never
• Unsure

When deciding to ship based on a test result, what minimum effect size on the primary metric is typically meaningful for your team?

• It depends on context
• Any positive change
• At least 0.5 percentage points
• At least 1 percentage point
• At least 2 percentage points
• At least 5 percentage points

Rank the following improvements by how much they would increase your trust in A/B test results. (Drag to reorder; most impactful first.)

1. Better instrumentation and QA
2. Guardrails against peeking at results early
3. Faster and more stable data pipelines
4. Pre-registration of hypotheses and metrics
5. Automated power / MDE checks before launch
6. Clearer result summaries and decision guidance

Based on your responses in this survey, please share any additional thoughts or concerns about the trustworthiness or reliability of our A/B testing program.

Finally, a few questions about your background for analysis purposes.

How long have you been at the company?

• Less than 6 months
• 6 to 12 months
• 1 to 2 years
• 3 to 5 years
• More than 5 years

How many years of total professional experience do you have?

• 0 to 2
• 3 to 5
• 6 to 10
• 11 to 15
• More than 15

What is your seniority level?

• Individual contributor
• People manager
• Director+
• Prefer not to say

Where are you primarily located?

• Americas
• Europe
• Middle East & Africa
• Asia-Pacific
• Multiple regions
• Prefer not to say

Which product area(s) do you mostly support? (Select up to three.)

• Consumer-facing experience
• B2B / Enterprise
• Infrastructure / Platform
• Monetization / Payments
• Marketing / Growth
• Internal tools
• Other
• Prefer not to say

Thank you for your time. Your feedback will directly inform improvements to our experimentation practices, tooling, and communication. Results will be shared in aggregate with the broader team.