Remaining code and data to be added soon. Stay tuned!
Abstract: Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.
Contact person: Rohan Nayak Ilia Kuznetsov
Don't hesitate to send us an e-mail if you have further questions.
- analysis.ipynb has the code for all the results used in the paper (results may be different as NeurIPS2024 and NeurIPS2025 data are absent)
- openReviewCrawler.ipynb has the code used for crawling data from OpenReview
Please use the following citation (To be added after arxiv upload):
@misc{kuznetsov2026peerreviewreallydecline,
title={Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time},
author={Ilia Kuznetsov and Rohan Nayak and Alla Rozovskaya and Iryna Gurevych},
year={2026},
eprint={2601.15172},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2601.15172},
}
This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.