Background. Scholarly peer review is “the lynchpin about which the whole business of science is pivoted” (Ziman 1968). Most researchers believe peer review is effective (Ware 2008), but empirical research consistently shows that reviewers cannot reliable distinguish methodologically sound from fundamentally flawed studies (Cole 1981; Peters & Ceci 1982; Lock 1991; Rothwell and Martyn 2000; Price 2014; Ralph 2016). Consequently, we created comprehensive evidence standards and tools to improve peer review in software engineering and related fields. Objective. The objective of this study is to investigate the impact of evidence standards on scholarly peer review. Method. A randomized controlled experiment was conducted at an A-ranked software engineering conference. The program committee was randomly divided into two groups: one using a typical conference review process; the other using a standardized process based on the ACM SIGSOFT Empirical Standards for Software Engineering Research (https://acmsigsoft.github.io/EmpiricalStandards/) Results. Evidence standards significantly improve inter-reviewer reliability without harming authors’ or reviewers’ attitudes toward the review process. Reviewers using evidence standards gave more praise and focused more on research methods than style. Discussion. Asking reviewers to write free-text comments about a paper and score it on a 6-point scale from strong reject to strong accept produces data statistically indistinguishable from random noise. This means that decisions are determined entirely by reviewer selection, not the merits of the research. Conventional review processes are therefore scientifically and morally indefensible. While they are not a silver bullet, evidence standards significantly improves reliability, and the data collected in this study facilitates further refinement of the standards and tooling toward still greater reliability.