Thank you for participating. Responses are confidential and analyzed in aggregate. Please be candid.
Which areas best describe your role? Select up to three.
- Product Management
- Engineering
- Data Science / Analytics
- Design / UX
- Marketing / Growth
- Operations / Support
- Leadership / Strategy
- Other
In the last 6 months, how often have you consumed or acted on A/B test results?
- Weekly or more
- 1 to 3 times per month
- A few times total
- Not in the last 6 months
- Never
Based on what you see and hear, how reliable are our A/B results overall?
What limits your use of A/B test results today? Select all that apply.
- Hard to access results
- Unsure how to interpret results
- Don’t trust data quality
- Not relevant to my work
- No tests run in my area
- Lack of time
- Other
Would a brief primer on power, minimum detectable effect (MDE), and uncertainty be helpful to you?
About how many distinct A/B tests did you work on or consume results from in the last 3 months?
Where are the tests you touch primarily run? Select all that apply.
- Web
- iOS app
- Android app
- Backend systems
- Marketing channels (email/ads)
- Other
How much do you trust the validity of our A/B test conclusions lately?
Recently, have you observed flaky or inconsistent A/B outcomes on key metrics?
- No
- Yes, occasionally
- Yes, frequently
- Unsure
Please share one or two recent examples of flakiness and what you think caused them.
Max 600 chars
How often do the following contribute to flaky results in your area?
When deciding to ship based on a test, what effect size on the primary metric is typically meaningful for you?
How often do A/B results meaningfully change your team’s decisions?
How clear is the communication of uncertainty (confidence intervals, p-values, power) in shipped reports?
Before launch, how often are MDE and power planned explicitly?
- Always
- Often
- Sometimes
- Rarely
- Never
- Unsure
Rank the top improvements that would most increase your trust.
Attention check: To confirm you’re paying attention, please select “Often” here.
- Never
- Rarely
- Sometimes
- Often
- Always
How long have you been at the company?
Total years of professional experience
Where are you primarily located?
- Americas
- Europe
- Middle East & Africa
- Asia-Pacific
- Multiple regions
- Prefer not to say
What is your seniority level?
Which product area(s) do you mostly support? Select up to three.
- Consumer-facing experience
- B2B / Enterprise
- Infrastructure / Platform
- Monetization / Payments
- Marketing / Growth
- Internal tools
- Other
- Prefer not to say
Anything else we should know about trust or flakiness in our experiments?
Max 600 chars
AI Interview: 2 Follow-up Questions on A/B Testing Trust and Flakiness
Thanks for your time—your feedback will help us improve our experimentation quality and communication.