Reproducibility in AI-Moderated Research: A Researcher Assessment

This survey explores researcher experiences with reproducibility, transparency, and data quality in AI-moderated research. It serves as a replication study template to understand current practices, identify barriers, and assess the role of platform transparency in enabling reproducible AI-moderated studies.

Use this template Browse more templates

Sample questions

A preview of what’s in the template. Every question is editable before you launch.

37 questions · ~4 min

Q01

Long Text

Welcome to this study on reproducibility in AI-moderated research. This survey is part of a research effort to understand how researchers document, replicate, and evaluate AI-moderated studies. Your participation is entirely voluntary, and you may stop at any time. There are no right or wrong answers — we are interested in your honest professional opinions and experiences. Your responses will be kept confidential, analyzed in aggregate, and used for academic and methodological research purposes only. Estimated completion time: 12–15 minutes.

Q02

Multiple Choice

Have you personally designed, conducted, or evaluated research that uses AI moderation (e.g., AI-driven interview probes, AI-moderated qualitative discussions, or AI-assisted survey logic)?

Q03

Multiple Choice

Approximately how many AI-moderated research projects have you been involved with in the past 24 months?

Q04

Long Text

For the purposes of this study, 'AI-moderated research' refers to any research methodology where an AI system actively conducts or adapts participant interactions in real time. This includes:AI-driven qualitative interviews (where AI asks follow-up probes)AI-moderated focus groups or discussionsAI-assisted survey logic that adapts questions based on prior responsesThis does NOT include:Using AI solely for data analysis after collectionUsing AI to write survey questions without real-time moderationSimple chatbot-based data collection with fully scripted flows

Q05

Long Text

When you have conducted or contributed to an AI-moderated study, how thoroughly did you typically document the AI system prompts and instructions used?

Q06

Long Text

The following section presents elements that could be disclosed to support transparency and reproducibility in AI-moderated research. For each element, please indicate whether the AI research platform you have used most recently makes this information accessible to you as a researcher.

Q07

Long Text

Compared to traditional (human-moderated) qualitative research, how would you rate the overall data quality of AI-moderated research?

Q08

Multiple Choice

Have you ever attempted to replicate an AI-moderated study (either your own or another researcher's)?

Q09

Long Text

Based on your experiences, what single change to AI-moderated research platforms or practices would most improve the reproducibility of studies conducted on these platforms?

Q10

Long Text

Which of the following best describes your primary professional role?

Q11

Multiple Choice

Which AI-moderated research platforms or tools have you used? (Select all that apply)

Q12

Long Text

How thoroughly did you typically document the AI model version and configuration parameters (e.g., temperature, model name, token limits)?

Q13

Multiple Choice

On your most recently used AI research platform, is the specific AI model name and version (e.g., GPT-4o, Claude 3.5) accessible to you?

Q14

Long Text

Compared to traditional human-moderated research, how would you rate AI-moderated research on: depth of participant responses

Q15

Long Text

How confident are you that an AI-moderated study you have conducted could be successfully replicated by another researcher using the same platform and configuration?

Q16

Long Text

Based on your responses throughout this survey, please share any additional thoughts or observations about reproducibility in AI-moderated research.

Q17

Long Text

How many years of experience do you have conducting research (any methodology)?

Q18

Long Text

How thoroughly did you typically document the logic flow and branching rules governing the AI moderation?

Q19

Multiple Choice

Are the system prompts and instructions given to the AI accessible to you?

Q20

Long Text

Compared to traditional human-moderated research, how would you rate AI-moderated research on: consistency of moderation across participants

Q21

Long Text

Please rank the following challenges to reproducibility in AI-moderated research from most significant (1) to least significant.

Q22

Long Text

In which sector do you primarily conduct research?

Q23

Long Text

How thoroughly did you typically document the sampling criteria and participant recruitment procedures?

Q24

Multiple Choice

Are the AI configuration parameters (e.g., temperature, max tokens, top-p) accessible to you?

Q25

Long Text

Compared to traditional human-moderated research, how would you rate AI-moderated research on: participant comfort and candor

Q26

Long Text

To what extent do you agree or disagree with the following statement: AI research platforms that share their prompts, models, and logic flows openly produce more reproducible research than platforms that do not disclose these elements.

Q27

Multiple Choice

What are the primary barriers you face when documenting AI-moderated study protocols? (Select all that apply)

Q28

Multiple Choice

Is the logic flow and branching structure of the AI moderation accessible to you?

Q29

Long Text

Compared to traditional human-moderated research, how would you rate AI-moderated research on: ability to capture unexpected insights

Q30

AI Interview

Please describe a specific experience where you encountered a reproducibility challenge in AI-moderated research. What happened, and how did you attempt to address it?

Q31

Multiple Choice

Are the complete raw transcripts of AI-participant interactions accessible to you?

Q32

Long Text

Compared to traditional human-moderated research, how would you rate AI-moderated research on: reproducibility of findings across studies

Q33

Long Text

How important is each of the following for enabling reproducibility of AI-moderated research? — Disclosure of AI model name and version

Q34

Long Text

— Disclosure of system prompts and instructions

Q35

Long Text

— Disclosure of AI configuration parameters

Q36

Long Text

— Disclosure of logic flow and branching rules

Q37

Long Text

— Access to complete raw interaction transcripts

What’s included

AI follow-ups
Adaptive probes on open-ended answers that pull out detail a static form would miss.
Attention checks
Built-in safeguards against rushed answers and low-quality respondents.
AI-drafted copy
Wording, ordering, and branching written by the AI — tuned to your research goal.
Auto report
Themes, quotes, and a plain-English summary write themselves once responses come in.

Ready to launch?

Open this template in the editor. Every part is yours to change before the first respondent sees it.

Use this template Start free

Sample questions

Have you personally designed, conducted, or evaluated research that uses AI moderation (e.g., AI-driven interview probes, AI-moderated qualitative discussions, or AI-assisted survey logic)?

Approximately how many AI-moderated research projects have you been involved with in the past 24 months?

When you have conducted or contributed to an AI-moderated study, how thoroughly did you typically document the AI system prompts and instructions used?

The following section presents elements that could be disclosed to support transparency and reproducibility in AI-moderated research. For each element, please indicate whether the AI research platform you have used most recently makes this information accessible to you as a researcher.

Compared to traditional (human-moderated) qualitative research, how would you rate the overall data quality of AI-moderated research?

Have you ever attempted to replicate an AI-moderated study (either your own or another researcher's)?

Based on your experiences, what single change to AI-moderated research platforms or practices would most improve the reproducibility of studies conducted on these platforms?

Which of the following best describes your primary professional role?

Which AI-moderated research platforms or tools have you used? (Select all that apply)

How thoroughly did you typically document the AI model version and configuration parameters (e.g., temperature, model name, token limits)?

On your most recently used AI research platform, is the specific AI model name and version (e.g., GPT-4o, Claude 3.5) accessible to you?

Compared to traditional human-moderated research, how would you rate AI-moderated research on: depth of participant responses

How confident are you that an AI-moderated study you have conducted could be successfully replicated by another researcher using the same platform and configuration?

Based on your responses throughout this survey, please share any additional thoughts or observations about reproducibility in AI-moderated research.

How many years of experience do you have conducting research (any methodology)?

How thoroughly did you typically document the logic flow and branching rules governing the AI moderation?

Are the system prompts and instructions given to the AI accessible to you?

Compared to traditional human-moderated research, how would you rate AI-moderated research on: consistency of moderation across participants

Please rank the following challenges to reproducibility in AI-moderated research from most significant (1) to least significant.

In which sector do you primarily conduct research?

How thoroughly did you typically document the sampling criteria and participant recruitment procedures?

Are the AI configuration parameters (e.g., temperature, max tokens, top-p) accessible to you?

Compared to traditional human-moderated research, how would you rate AI-moderated research on: participant comfort and candor

To what extent do you agree or disagree with the following statement: AI research platforms that share their prompts, models, and logic flows openly produce more reproducible research than platforms that do not disclose these elements.

What are the primary barriers you face when documenting AI-moderated study protocols? (Select all that apply)

Is the logic flow and branching structure of the AI moderation accessible to you?

Compared to traditional human-moderated research, how would you rate AI-moderated research on: ability to capture unexpected insights

Please describe a specific experience where you encountered a reproducibility challenge in AI-moderated research. What happened, and how did you attempt to address it?

Are the complete raw transcripts of AI-participant interactions accessible to you?

Compared to traditional human-moderated research, how would you rate AI-moderated research on: reproducibility of findings across studies

How important is each of the following for enabling reproducibility of AI-moderated research? — Disclosure of AI model name and version

— Disclosure of system prompts and instructions

— Disclosure of AI configuration parameters

— Disclosure of logic flow and branching rules

— Access to complete raw interaction transcripts

What’s included

AI follow-ups

Attention checks

AI-drafted copy

Auto report

Ready to launch?