Pre-Analysis Plan Peer Review Assessment

Welcome, and thank you for participating in this pre-analysis plan peer review. This survey asks for your expert feedback on the clarity, completeness, and pre-specification of a research plan. There are no right or wrong answers—we value your honest professional judgment. Your responses are confidential and will be reported only in aggregate to the research team. Participation is voluntary, and you may stop at any time. The survey takes approximately 15–20 minutes to complete. Please base your responses only on the materials provided today.

Which materials did you review for this assessment? Select all that apply.

• Research question brief
• Full PAP/preregistration draft
• Hypotheses section
• Outcome definitions
• Analysis plan
• Power analysis
• Data dictionary / variable list
• Inclusion/exclusion rules
• Mock tables/figures
• Other

What best describes the planned study design?

• Randomized controlled trial
• Quasi-experimental (e.g., DiD, IV, RD)
• Observational cross-sectional
• Longitudinal / panel study
• Lab / online experiment
• Qualitative or mixed methods
• Other

In your own words, summarize the research question in one or two sentences.

Each hypothesis is clearly labeled (e.g., H1, H2) and easy to identify.

Each hypothesis specifies the expected direction of the effect.

Primary and secondary/exploratory hypotheses are clearly distinguished.

How well do the planned statistical tests align with the stated hypotheses?

Please note any hypotheses that seem ambiguous, double-barreled, or underspecified, and briefly explain why.

The primary outcome is operationally defined (i.e., exact variable, measurement method, and timing are specified).

Measurement instruments or scales are described in sufficient detail to be replicated.

Secondary outcomes and any index/composite construction rules are clearly specified.

Which types of outcomes are planned? Select all that apply.

• Behavioral / administrative
• Survey scale or index
• Physiological / biomarker
• Derived composite/index
• Binary event
• Time-to-event
• Other

The primary estimator or model is clearly specified (e.g., OLS, logistic regression, ANCOVA).

Key assumptions of the chosen model are stated and justified.

Robustness checks or sensitivity analyses are pre-specified.

Is the estimand (e.g., ATE, ITT, CACE) explicitly defined?

• Yes, clearly defined
• Partially defined
• Not defined
• Not applicable

Is a power analysis or sample size justification included?

• Yes, with calculations and inputs
• Yes, but minimal detail
• No
• Not applicable

Given the planned tests, how adequate is the statistical power justification?

If applicable, note the minimum detectable effect (MDE), key inputs, or any concerns about the power justification.

Rank the following potential threats to interpretability for this study, from most to least concerning.

1. Measurement error / instrument validity
2. Confounding / selection bias
3. Noncompliance / attrition
4. Selective reporting / researcher degrees of freedom
5. Model misspecification
6. Multiplicity / p-hacking

Are there ethical considerations that may influence analysis choices? Select all that apply.

• None noted
• Privacy or data security risk
• Potential harm to participants
• Equity/fairness bias concerns
• Data governance/consent constraints
• Other

Which additions would most improve reproducibility before data collection? Select up to three.

• Mock registry entry (final wording)
• Code template or skeleton analysis script
• Data schema / variable naming plan
• Versioned package/dependency list
• Defined file/folder structure with README
• Plan for where materials will be shared
• Other

Overall, how clear is the pre-analysis plan as currently written?

Based on your review, is this plan ready to proceed to data collection?

• Yes, proceed as planned
• Mostly ready; minor edits recommended
• Hold; needs substantive revisions
• Unsure

What are your most actionable suggestions to improve the clarity, pre-specification, or reproducibility of this plan?

Based on your responses, we'd like to explore your key recommendations in more depth. Please share your thoughts with our AI moderator.

Approximately how long did you spend reviewing the materials today?

• Less than 15 minutes
• 15–30 minutes
• 31–60 minutes
• 1–2 hours
• More than 2 hours

What is your primary role?

• Academic researcher
• Graduate student
• Data scientist / analyst
• Policy researcher / evaluator
• Practitioner / NGO
• Other

How many years of experience do you have with preregistrations or pre-analysis plans?

• Less than 1 year
• 1–3 years
• 4–6 years
• 7–10 years
• More than 10 years

What is your primary field or domain?

• Economics
• Political science
• Public health
• Education
• Psychology
• Sociology
• Computer science / data science
• Other

Have you previously authored a preregistration or pre-analysis plan?

• Yes
• No

How familiar are you with this study's topic area?

• Novice
• Intermediate
• Advanced
• Expert

Thank you for your thoughtful review. Your feedback will be used to improve the clarity and pre-specification of this plan before data collection begins.