Reproducibility in AI-Moderated Research: A Researcher Assessment

Welcome to this study on reproducibility in AI-moderated research. This survey is part of a research effort to understand how researchers document, replicate, and evaluate AI-moderated studies. Your participation is entirely voluntary, and you may stop at any time. There are no right or wrong answers — we are interested in your honest professional opinions and experiences. Your responses will be kept confidential, analyzed in aggregate, and used for academic and methodological research purposes only. Estimated completion time: 12–15 minutes.

Have you personally designed, conducted, or evaluated research that uses AI moderation (e.g., AI-driven interview probes, AI-moderated qualitative discussions, or AI-assisted survey logic)?

• Yes, I have direct experience with AI-moderated research
• No, but I am familiar with AI-moderated research methods
• No, I am not familiar with AI-moderated research

Approximately how many AI-moderated research projects have you been involved with in the past 24 months?

• 1–2 projects
• 3–5 projects
• 6–10 projects
• More than 10 projects

Which AI-moderated research platforms or tools have you used? (Select all that apply)

• QuestionPunk
• Remesh
• Discuss.io (AI features)
• Forsta / Confirmit (AI features)
• Qualtrics (AI-assisted logic)
• Custom-built AI moderation tools
• Other (please specify)

<p>For the purposes of this study, <strong>'AI-moderated research'</strong> refers to any research methodology where an AI system actively conducts or adapts participant interactions in real time. This includes:</p><p>AI-driven qualitative interviews (where AI asks follow-up probes)</p><p>AI-moderated focus groups or discussions</p><p>AI-assisted survey logic that adapts questions based on prior responses</p><p>This does <strong>NOT</strong> include:</p><p>Using AI solely for data analysis after collection</p><p>Using AI to write survey questions without real-time moderation</p><p>Simple chatbot-based data collection with fully scripted flows</p>

When you have conducted or contributed to an AI-moderated study, how thoroughly did you typically document the AI system prompts and instructions used?

How thoroughly did you typically document the AI model version and configuration parameters (e.g., temperature, model name, token limits)?

How thoroughly did you typically document the logic flow and branching rules governing the AI moderation?

How thoroughly did you typically document the sampling criteria and participant recruitment procedures?

What are the primary barriers you face when documenting AI-moderated study protocols? (Select all that apply)

• Platform does not expose configuration details
• AI model versions change without notice
• No established documentation standards exist
• Time constraints during project execution
• Proprietary restrictions from the platform vendor
• Lack of institutional requirements for AI documentation
• Uncertainty about which elements are necessary to document
• Other (please specify)

The following section presents elements that could be disclosed to support transparency and reproducibility in AI-moderated research. For each element, please indicate whether the AI research platform you have used most recently makes this information accessible to you as a researcher.

On your most recently used AI research platform, is the specific AI model name and version (e.g., GPT-4o, Claude 3.5) accessible to you?

• Yes, fully accessible
• Partially accessible (some details available)
• Not accessible
• I don't know / Never checked

Are the system prompts and instructions given to the AI accessible to you?

• Yes, fully accessible
• Partially accessible (some details available)
• Not accessible
• I don't know / Never checked

Are the AI configuration parameters (e.g., temperature, max tokens, top-p) accessible to you?

• Yes, fully accessible
• Partially accessible (some details available)
• Not accessible
• I don't know / Never checked

Is the logic flow and branching structure of the AI moderation accessible to you?

• Yes, fully accessible
• Partially accessible (some details available)
• Not accessible
• I don't know / Never checked

Are the complete raw transcripts of AI-participant interactions accessible to you?

• Yes, fully accessible
• Partially accessible (some details available)
• Not accessible
• I don't know / Never checked

How important is each of the following for enabling reproducibility of AI-moderated research? — Disclosure of AI model name and version

— Disclosure of system prompts and instructions

— Disclosure of AI configuration parameters

— Disclosure of logic flow and branching rules

— Access to complete raw interaction transcripts

Compared to traditional (human-moderated) qualitative research, how would you rate the overall data quality of AI-moderated research?

Compared to traditional human-moderated research, how would you rate AI-moderated research on: depth of participant responses

Compared to traditional human-moderated research, how would you rate AI-moderated research on: consistency of moderation across participants

Compared to traditional human-moderated research, how would you rate AI-moderated research on: participant comfort and candor

Compared to traditional human-moderated research, how would you rate AI-moderated research on: ability to capture unexpected insights

Compared to traditional human-moderated research, how would you rate AI-moderated research on: reproducibility of findings across studies

Have you ever attempted to replicate an AI-moderated study (either your own or another researcher's)?

• Yes, I have replicated my own AI-moderated study
• Yes, I have replicated another researcher's AI-moderated study
• Yes, both my own and others' studies
• No, I have not attempted a replication

How confident are you that an AI-moderated study you have conducted could be successfully replicated by another researcher using the same platform and configuration?

Please rank the following challenges to reproducibility in AI-moderated research from most significant (1) to least significant.

1. AI model updates change behavior over time
2. Platform-specific 'black box' configurations
3. Non-deterministic AI outputs (randomness in responses)
4. Lack of standardized documentation protocols
5. Insufficient access to system prompts and parameters
6. Variations in participant behavior across samples

To what extent do you agree or disagree with the following statement: AI research platforms that share their prompts, models, and logic flows openly produce more reproducible research than platforms that do not disclose these elements.

Please describe a specific experience where you encountered a reproducibility challenge in AI-moderated research. What happened, and how did you attempt to address it?

Based on your experiences, what single change to AI-moderated research platforms or practices would most improve the reproducibility of studies conducted on these platforms?

Based on your responses throughout this survey, please share any additional thoughts or observations about reproducibility in AI-moderated research.

Which of the following best describes your primary professional role?

• Academic researcher
• Market research professional
• UX / Design researcher
• Data scientist / AI specialist
• Research operations / insights manager
• Consultant / Independent researcher
• Student (graduate or postgraduate)
• Other (please specify)

How many years of experience do you have conducting research (any methodology)?

• Less than 2 years
• 2–5 years
• 6–10 years
• 11–20 years
• More than 20 years

In which sector do you primarily conduct research?

• Academia / Higher education
• Technology / Software
• Healthcare / Pharmaceuticals
• Financial services
• Consumer packaged goods
• Media / Entertainment
• Government / Public sector
• Non-profit / NGO
• Consulting / Agency
• Other (please specify)