Data Labeling QA, Bias & Instruction Clarity Audit

Welcome! This survey takes about 11 minutes and asks about your data labeling work over the last 30 days. Your participation is entirely voluntary, and you may stop at any time. There are no right or wrong answers — we are interested in your honest experience. All responses are confidential and will be reported only in aggregate to improve labeling operations. By continuing, you agree to participate.

In the past 30 days, which of the following tasks have you performed? Select all that apply.

• Labeling / annotation
• Reviewing / QA
• Both labeling and reviewing
• Other (please specify)

How long have you worked on this labeling program?

• Less than 1 month
• 1–3 months
• 4–6 months
• 7–12 months
• 1–2 years
• More than 2 years

Overall, how clear were the task instructions you received in the last 30 days?

In the last 30 days, how often did task instructions change mid-project?

If you encountered any unclear or conflicting instructions in the last 30 days, please briefly describe one example. If none, you may skip this question.

Which of the following bias topics are covered in your current labeling guidelines? Select all that apply.

• Demographic bias (e.g., gender, race, age)
• Domain or jargon bias
• Geographic / vernacular variation
• Label leakage or proxy signals
• Harmful stereotypes and toxicity
• Context / translation bias
• None of the above
• Other (please specify)

In the last 30 days, how often did you encounter inputs or labels that appeared biased?

When bias is suspected, how clear is the process for escalating the issue?

If you encountered a potentially biased input or label recently, please briefly describe the example and how you handled it. If none, you may skip this question.

How clear are the acceptance criteria used for reviewing labeled work?

Which review approach is used most often on your current program?

• Blind double review with adjudication
• Spot checks (fixed percentage)
• Heuristic-triggered review (rules-based)
• Peer review within team
• Self-review before submit
• Not sure
• Other (please specify)

How useful was the review feedback you received in the last 30 days for improving your labeling accuracy?

How timely was the review feedback you received in the last 30 days?

Approximately what percentage of your labeled items were returned for rework in the last 30 days?

• 0%
• 1–5%
• 6–10%
• 11–20%
• 21–30%
• 31–50%
• More than 50%
• Not sure

From the list below, rank the top causes of rework you observed in the last 30 days, from most common to least common.

1. Unclear or changing guidelines
2. Reviewer–labeler disagreement
3. Edge cases not covered
4. Tooling or platform issues
5. Time pressure or quotas
6. Insufficient training or context

Which of the following activities takes the largest share of your typical work week on this program?

• Labeling / annotation
• Review / QA
• Guideline reading / updating
• Meetings / syncs
• Training / onboarding
• Escalations or questions
• Other (please specify)

Which of the following tooling issues most slowed your quality or speed in the last 30 days? Select all that apply.

• Slow loading or lag
• Limited shortcuts or templates
• Poor diff / compare views
• Unclear error messages
• Hard to flag bias or edge cases
• Limited audit trail / metadata
• None of the above
• Other (please specify)

If you could make one change to improve clarity, fairness, or quality assurance in your labeling work, what would it be?

Based on your responses, we'd like to explore a few of your experiences in more depth. An AI moderator will ask you 1–2 follow-up questions about your labeling operations.

What is your primary working region?

• North America
• Latin America
• Europe
• Middle East
• Africa
• South Asia
• East Asia
• Southeast Asia
• Oceania
• Prefer not to say

What is your primary working language?

• English
• Spanish
• Portuguese
• French
• German
• Chinese
• Japanese
• Korean
• Hindi
• Arabic
• Other (please specify)
• Prefer not to say

How much total experience do you have in data labeling or annotation?

• Less than 6 months
• 6–12 months
• 1–2 years
• 3–5 years
• 6+ years

What is your employment type on this program?

• Full-time
• Part-time
• Contract / Freelance
• Prefer not to say

Thank you for completing this survey. Your feedback will directly inform improvements to instruction clarity, bias mitigation, and quality assurance processes.