Pre-Analysis Plan Peer Review Assessment

Welcome, and thank you for participating in this pre-analysis plan peer review. This survey asks for your expert feedback on the clarity, completeness, and pre-specification of a research plan. There are no right or wrong answers—we value your honest professional judgment. Your responses are confidential and will be reported only in aggregate to the research team. Participation is voluntary, and you may stop at any time. The survey takes approximately 15–20 minutes to complete. Please base your responses only on the materials provided today.

Which materials did you review for this assessment? Select all that apply.

• Research question brief
• Full PAP/preregistration draft
• Hypotheses section
• Outcome definitions
• Analysis plan
• Power analysis
• Data dictionary / variable list
• Inclusion/exclusion rules
• Mock tables/figures
• Other

Each hypothesis is clearly labeled (e.g., H1, H2) and easy to identify.

The primary outcome is operationally defined (i.e., exact variable, measurement method, and timing are specified).

The primary estimator or model is clearly specified (e.g., OLS, logistic regression, ANCOVA).

Is a power analysis or sample size justification included?

• Yes, with calculations and inputs
• Yes, but minimal detail
• No
• Not applicable

Rank the following potential threats to interpretability for this study, from most to least concerning.

1. Measurement error / instrument validity
2. Confounding / selection bias
3. Noncompliance / attrition
4. Selective reporting / researcher degrees of freedom
5. Model misspecification
6. Multiplicity / p-hacking

Overall, how clear is the pre-analysis plan as currently written?

Approximately how long did you spend reviewing the materials today?

• Less than 15 minutes
• 15–30 minutes
• 31–60 minutes
• 1–2 hours
• More than 2 hours

What is your primary role?

• Academic researcher
• Graduate student
• Data scientist / analyst
• Policy researcher / evaluator
• Practitioner / NGO
• Other

Thank you for your thoughtful review. Your feedback will be used to improve the clarity and pre-specification of this plan before data collection begins.

What best describes the planned study design?

• Randomized controlled trial
• Quasi-experimental (e.g., DiD, IV, RD)
• Observational cross-sectional
• Longitudinal / panel study
• Lab / online experiment
• Qualitative or mixed methods
• Other

Each hypothesis specifies the expected direction of the effect.

Measurement instruments or scales are described in sufficient detail to be replicated.

Key assumptions of the chosen model are stated and justified.

Given the planned tests, how adequate is the statistical power justification?

Are there ethical considerations that may influence analysis choices? Select all that apply.

• None noted
• Privacy or data security risk
• Potential harm to participants
• Equity/fairness bias concerns
• Data governance/consent constraints
• Other

Based on your review, is this plan ready to proceed to data collection?

• Yes, proceed as planned
• Mostly ready; minor edits recommended
• Hold; needs substantive revisions
• Unsure

How many years of experience do you have with preregistrations or pre-analysis plans?

• Less than 1 year
• 1–3 years
• 4–6 years
• 7–10 years
• More than 10 years

In your own words, summarize the research question in one or two sentences.

Primary and secondary/exploratory hypotheses are clearly distinguished.

Secondary outcomes and any index/composite construction rules are clearly specified.

Robustness checks or sensitivity analyses are pre-specified.

If applicable, note the minimum detectable effect (MDE), key inputs, or any concerns about the power justification.

Which additions would most improve reproducibility before data collection? Select up to three.

• Mock registry entry (final wording)
• Code template or skeleton analysis script
• Data schema / variable naming plan
• Versioned package/dependency list
• Defined file/folder structure with README
• Plan for where materials will be shared
• Other

What are your most actionable suggestions to improve the clarity, pre-specification, or reproducibility of this plan?

What is your primary field or domain?

• Economics
• Political science
• Public health
• Education
• Psychology
• Sociology
• Computer science / data science
• Other

How well do the planned statistical tests align with the stated hypotheses?

Which types of outcomes are planned? Select all that apply.

• Behavioral / administrative
• Survey scale or index
• Physiological / biomarker
• Derived composite/index
• Binary event
• Time-to-event
• Other

Is the estimand (e.g., ATE, ITT, CACE) explicitly defined?

• Yes, clearly defined
• Partially defined
• Not defined
• Not applicable

Based on your responses, we'd like to explore your key recommendations in more depth. Please share your thoughts with our AI moderator.

Have you previously authored a preregistration or pre-analysis plan?

• Yes
• No

Please note any hypotheses that seem ambiguous, double-barreled, or underspecified, and briefly explain why.

How familiar are you with this study's topic area?

• Novice
• Intermediate
• Advanced
• Expert