Evaluation Fairness & Representation Perceptions Survey for Developers

Welcome! This survey explores your experiences and perspectives on fairness, bias, and representativeness in software and model evaluations. Your participation is completely voluntary and you may stop at any time. All responses are anonymous and will be reported only in aggregate. There are no right or wrong answers — we are interested in your honest opinions. Estimated time: 8–10 minutes.

Which best describes your primary development focus?

• Frontend
• Backend
• ML/AI
• Data engineering/MLOps
• Mobile
• DevOps/SRE
• Security
• Full-stack
• QA/Test automation
• Other (please specify)

Have you been involved in evaluating software, systems, or models in the last 12 months?

• Yes, in the last 6 months
• Yes, 6–12 months ago
• Yes, over a year ago
• No

In a typical month, approximately how much of your time is spent on evaluation activities?

• 0–10%
• 11–25%
• 26–50%
• 51–75%
• 76–100%
• Prefer not to say

Which evaluation method did you rely on most in the last 6 months?

• Unit tests/assertions
• Offline benchmarks
• Human ratings/annotation
• A/B or canary releases
• Synthetic data tests
• Red-teaming/adversarial testing
• Bias/fairness audits
• Other (please specify)
• None/Not applicable

In your recent evaluations, did you consider sensitive attributes (e.g., gender, ethnicity, income)?

• Yes
• No
• Not applicable

Which safeguard was most important when handling sensitive attributes in your evaluations?

• IRB/ethics review
• Legal/privacy review
• Data minimization
• Aggregation/anonymization
• Differential privacy or noise
• Limited access/approvals
• Stakeholder consent
• Bias detection/remediation
• Other (please specify)
• Not applicable

How important is fairness in your evaluation decisions?

To what extent do you agree: Our evaluation criteria are applied consistently across different user groups.

To what extent do you agree: I have adequate tools and methods to detect bias in evaluation outcomes.

To what extent do you agree: Stakeholders from diverse backgrounds are involved in designing our evaluations.

To what extent do you agree: Fairness considerations sometimes conflict with other priorities such as speed or cost.

In one or two sentences, how do you define a "fair" evaluation?

How concerned are you that unrepresentative samples may have affected your evaluation results in the last 12 months?

Which sampling strategy did you use most often in the last 12 months?

• Random sampling
• Stratified sampling
• User segment quotas
• Synthetic augmentation
• Convenience/availability sampling
• Production traffic replay
• Telemetry-driven sampling
• Other (please specify)
• None/Not applicable

Rank the following segments by priority for coverage in your evaluations (top = highest priority).

1. New users
2. Power users
3. Underrepresented regions/locales
4. Low-resource devices
5. Harm-sensitive contexts
6. Long-tail queries

Approximately what minimum sample size do you typically need to trust a feature-level evaluation decision?

• Under 100
• 100–499
• 500–999
• 1,000–4,999
• 5,000–9,999
• 10,000+
• I don't have a specific threshold
• Prefer not to say

How confident are you that your evaluations fairly represent real-world use?

If you faced any trade-offs between accuracy, speed, and fairness in recent evaluations, please briefly describe them.

Based on your responses in this survey, what would most improve fairness and representativeness in your evaluations?

We'd like to explore your thoughts on fairness and representativeness in evaluations a bit further. Please share your perspective and our AI moderator will ask a couple of follow-up questions.

How many years of professional development experience do you have?

• Less than 1
• 1–3
• 4–6
• 7–10
• 11–15
• 16+
• Prefer not to say

What is your current seniority level?

• Student/Intern
• Junior/Associate
• Mid-level
• Senior
• Staff/Principal
• Manager/Lead
• Other
• Prefer not to say

Which region do you primarily work in?

• Africa
• Asia-Pacific
• Europe
• Latin America
• Middle East
• North America
• Oceania
• Prefer not to say

Approximately how many employees are in your organization?

• 1
• 2–10
• 11–50
• 51–200
• 201–1,000
• 1,001–10,000
• 10,001+
• Prefer not to say

How many people are on the team you primarily work with?

• 1
• 2–5
• 6–10
• 11–20
• 21–50
• 51+
• Prefer not to say

What is your primary industry or domain?

• Consumer software
• Enterprise/B2B
• Finance/Fintech
• Healthcare
• Education
• E-commerce
• Gaming
• Government/Public sector
• Research/Academia
• Other
• Prefer not to say

Thank you for completing the survey! Your responses are anonymous and will be used in aggregate to improve evaluation practices. We appreciate your time.