5 Statistical Flaws AI-Assisted Peer Review Can Flag

AI-assisted peer review is most useful as a consistent statistical second pass. It can compare claims across the abstract, methods, results, tables, and supplement; surface patterns that deserve scrutiny; and point a reviewer to the relevant passage. It cannot determine validity from a p-value alone or replace statistical and domain expertise.

The five flaws below are high-value checks because each can change the interpretation of a paper. For each one, the goal is not to generate an accusation. It is to identify the claim, locate the supporting analysis, and decide whether the evidence justifies the wording.

1. Multiplicity without a clear error-control plan

Every formal hypothesis test creates an opportunity for a false positive. If six independent null hypotheses are tested at an unadjusted 5% level, the expected number of false positives is 0.30, while the probability of at least one false positive is about 26.5%: 1 − 0.95⁶.

The practical problem is broader than a list of outcomes. Multiplicity can arise from:

• multiple primary or secondary endpoints;
• several treatment doses or comparator groups;
• repeated time points;
• subgroup analyses;
• alternative model specifications;
• interim analyses.

An AI-assisted review can inventory these analysis branches and search for a prespecified strategy such as Holm adjustment, a hierarchical testing sequence, or false-discovery-rate control. It should not assume that every p-value in a paper belongs to one family or that Bonferroni is always the right answer.

Reviewer check: Identify which results support confirmatory claims, how those tests were grouped, and whether the stated procedure controls the relevant error rate. The FDA guidance on multiple endpoints explains why endpoint grouping and ordering matter in confirmatory trials.

For the full statistical explanation, use the dedicated multiple comparisons guide.

2. Testing change within groups instead of the difference between groups

A common trial-reporting error has this form:

The treatment group improved significantly from baseline, while the control group did not; therefore, the treatment worked.

That conclusion does not follow. A significant change in one group and a non-significant change in another is not itself evidence that the groups differ. The relevant analysis estimates the between-group contrast—often a group-by-time interaction, a difference in change, or another prespecified treatment-effect estimand—with its uncertainty.

The issue can hide behind correct-looking individual tests. An automated pass can align the claim with the model specification and ask whether the coefficient that directly represents the treatment contrast is reported.

Reviewer check: Find the estimand first. Then locate the estimate, confidence interval, and model term that correspond to it. Separate t-tests at each time point or separate within-arm changes may not answer the trial’s primary question, and repeated or clustered observations require their dependence to be handled.

This check belongs inside a broader clinical trial critical-appraisal workflow, not as a standalone pass/fail rule.

3. Outcome switching and selective emphasis

A statistically polished paper can still mislead if the outcome being celebrated was not the outcome the study set out to test. Warning patterns include:

• a registered primary outcome missing from the paper;
• a secondary outcome presented as primary after results were known;
• a changed time point, scale, or analysis metric;
• an abstract that highlights a favorable exploratory result while the primary result was null;
• several reasonable analyses with only the favorable version reported.

The strongest check compares the publication with the dated trial registration, protocol, and statistical analysis plan. ClinicalTrials.gov defines a primary outcome as the prespecified measure of greatest importance and exposes a record history that can help readers inspect changes. The ClinicalTrials.gov study-record guide shows where those fields and revisions appear.

AI can accelerate this comparison when all documents are available, but timing matters. A difference between documents is not automatically misconduct: amendments can be legitimate. Record what changed, when it changed, whether a reason was given, and whether the paper labels the analysis accurately.

Reviewer check: Build a small outcome map with the protocol definition, publication definition, time point, analysis population, and reported result. Treat unexplained discrepancies as a risk to interpretation, not proof of intent.

4. Treating imprecision or non-significance as proof of no effect

p ≥ 0.05 does not establish equivalence, absence of effect, or absence of clinically important harm. A small or noisy study may produce a wide confidence interval compatible with meaningful benefit, no material effect, and meaningful harm at the same time.

An AI-assisted review can flag phrases such as “no difference,” “equivalent,” or “safe” and compare them with the interval estimate. The important questions are:

1. What range of effects is compatible with the data and model?
2. Does that range cross a clinically important threshold?
3. Was the trial designed as a superiority, non-inferiority, or equivalence study?
4. Were the margin and analysis population prespecified for an equivalence-type claim?

Power calculations are useful design information, but post hoc power does not repair an imprecise estimate. Read the effect size and confidence interval directly. The American Statistical Association’s p-value statement emphasizes that scientific conclusions should not rest only on whether a p-value crosses a threshold.

Reviewer check: Rewrite the claim in interval language. “The study found no statistically significant difference” is descriptive; “the treatments are equivalent” requires a design and evidence capable of supporting equivalence.

5. A composite endpoint driven by its weakest component

Composite endpoints combine several events into one outcome—for example, death, hospitalization, and a less severe clinical intervention. They can increase event counts and answer a meaningful patient-level question, but the combined result can obscure what actually changed.

Interpretation becomes fragile when components:

• differ greatly in clinical importance;
• differ greatly in frequency;
• are affected in opposite directions;
• use subjective and objective events together;
• are reported only as a combined estimate.

A positive composite may be driven by the most frequent, least important component while the clinically critical component is unchanged. Conversely, splitting every component into separate significance tests can create a new multiplicity problem.

Reviewer check: Confirm that the composite was prespecified, inspect the definition and time-to-first-event rules, review each component’s event count and estimate, and calibrate the conclusion to the component pattern. The FDA’s multiple-endpoints guidance discusses both composite endpoints and the uncertainty they can create about which component drove the overall effect.

A verification workflow for AI-assisted statistical review

Use automation to make review more systematic, not more certain than the evidence permits:

1. Extract the claim. Record the population, comparison, outcome, time point, and direction of effect.
2. Locate its evidence. Link the claim to a table, figure, model coefficient, or supplementary analysis.
3. Classify the analysis. Mark it as prespecified primary, secondary, sensitivity, subgroup, or post hoc.
4. Surface the concern. Describe the exact mismatch—do not jump from “not reported” to “not done.”
5. Verify manually. Check the source passage, protocol, registry history, and statistical assumptions.
6. Calibrate the wording. State how the issue changes certainty or interpretation and what clarification would resolve it.

Before critiquing, create an accurate structured summary of the research paper. That neutral record reduces the risk of reviewing a claim the authors did not actually make.

Where AI-assisted review stops

Text analysis can detect internal inconsistencies and missing explanations. It usually cannot determine whether model assumptions hold, reproduce an analysis without data and code, select a clinically meaningful margin, or decide whether an unexpected finding is biologically plausible.

Confidentiality also matters. Do not upload an unpublished manuscript to an external tool unless the journal, editor, data owner, and applicable policy permit it. The final judgment and wording remain the reviewer’s responsibility.

Related appraisal guides

• Apply the complete research paper peer-review checklist to methods, reporting, conclusions, and reviewer comments.
• Read intervention evidence with the eight-step clinical trial appraisal guide.
• Compare the same claim across studies using a multi-paper evidence matrix.
• See how these checks fit inside a systematic literature review.