Evaluation Fairness & Representation Perceptions Survey for Developers

Welcome! This survey explores your experiences and perspectives on fairness, bias, and representativeness in software and model evaluations. Your participation is completely voluntary and you may stop at any time. All responses are anonymous and will be reported only in aggregate. There are no right or wrong answers — we are interested in your honest opinions. Estimated time: 8–10 minutes.

Which best describes your primary development focus?

• Frontend
• Backend
• ML/AI
• Data engineering/MLOps
• Mobile
• DevOps/SRE
• Security
• Full-stack
• QA/Test automation
• Other (please specify)

Which evaluation method did you rely on most in the last 6 months?

• Unit tests/assertions
• Offline benchmarks
• Human ratings/annotation
• A/B or canary releases
• Synthetic data tests
• Red-teaming/adversarial testing
• Bias/fairness audits
• Other (please specify)
• None/Not applicable

How important is fairness in your evaluation decisions?

How concerned are you that unrepresentative samples may have affected your evaluation results in the last 12 months?

If you faced any trade-offs between accuracy, speed, and fairness in recent evaluations, please briefly describe them.

How many years of professional development experience do you have?

• Less than 1
• 1–3
• 4–6
• 7–10
• 11–15
• 16+
• Prefer not to say

Thank you for completing the survey! Your responses are anonymous and will be used in aggregate to improve evaluation practices. We appreciate your time.

Have you been involved in evaluating software, systems, or models in the last 12 months?

• Yes, in the last 6 months
• Yes, 6–12 months ago
• Yes, over a year ago
• No

In your recent evaluations, did you consider sensitive attributes (e.g., gender, ethnicity, income)?

• Yes
• No
• Not applicable

To what extent do you agree: Our evaluation criteria are applied consistently across different user groups.

Which sampling strategy did you use most often in the last 12 months?

• Random sampling
• Stratified sampling
• User segment quotas
• Synthetic augmentation
• Convenience/availability sampling
• Production traffic replay
• Telemetry-driven sampling
• Other (please specify)
• None/Not applicable

Based on your responses in this survey, what would most improve fairness and representativeness in your evaluations?

What is your current seniority level?

• Student/Intern
• Junior/Associate
• Mid-level
• Senior
• Staff/Principal
• Manager/Lead
• Other
• Prefer not to say

In a typical month, approximately how much of your time is spent on evaluation activities?

• 0–10%
• 11–25%
• 26–50%
• 51–75%
• 76–100%
• Prefer not to say

Which safeguard was most important when handling sensitive attributes in your evaluations?

• IRB/ethics review
• Legal/privacy review
• Data minimization
• Aggregation/anonymization
• Differential privacy or noise
• Limited access/approvals
• Stakeholder consent
• Bias detection/remediation
• Other (please specify)
• Not applicable

To what extent do you agree: I have adequate tools and methods to detect bias in evaluation outcomes.

Rank the following segments by priority for coverage in your evaluations (top = highest priority).

1. New users
2. Power users
3. Underrepresented regions/locales
4. Low-resource devices
5. Harm-sensitive contexts
6. Long-tail queries

We'd like to explore your thoughts on fairness and representativeness in evaluations a bit further. Please share your perspective and our AI moderator will ask a couple of follow-up questions.

Which region do you primarily work in?

• Africa
• Asia-Pacific
• Europe
• Latin America
• Middle East
• North America
• Oceania
• Prefer not to say

To what extent do you agree: Stakeholders from diverse backgrounds are involved in designing our evaluations.

Approximately what minimum sample size do you typically need to trust a feature-level evaluation decision?

• Under 100
• 100–499
• 500–999
• 1,000–4,999
• 5,000–9,999
• 10,000+
• I don't have a specific threshold
• Prefer not to say

Approximately how many employees are in your organization?

• 1
• 2–10
• 11–50
• 51–200
• 201–1,000
• 1,001–10,000
• 10,001+
• Prefer not to say

To what extent do you agree: Fairness considerations sometimes conflict with other priorities such as speed or cost.

How confident are you that your evaluations fairly represent real-world use?

How many people are on the team you primarily work with?

• 1
• 2–5
• 6–10
• 11–20
• 21–50
• 51+
• Prefer not to say

In one or two sentences, how do you define a "fair" evaluation?

What is your primary industry or domain?

• Consumer software
• Enterprise/B2B
• Finance/Fintech
• Healthcare
• Education
• E-commerce
• Gaming
• Government/Public sector
• Research/Academia
• Other
• Prefer not to say