SurveyReview

A Reviewer-Aligned Benchmark for Survey Evaluators

SurveyReview is a reviewer-aligned benchmark for evaluating survey papers. It converts real peer-review reports into multidimensional evaluation labels and rationales, allowing models to be tested against how human reviewers judge survey quality.

Version

v1.0/: the original SurveyReview release for paper reproduction.
v1.1/: the updated release using cleaned article texts.

The benchmark focuses on four survey-review dimensions: Readability, Criticalness, Comprehensiveness, and Structure. It provides standardized train/test splits, article metadata, prompt files, and an API-based evaluation pipeline.

Quick Start

Create an environment and install dependencies:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Configure the API client:

cp .env.example .env

Then edit .env:

API_KEY=your-api-key-here
BASE_URL=https://api.openai.com/v1
MODEL_NAME=gpt-5.2
JUDGE_MODEL=gpt-5.2
EVALUATE_REASONS=True

Run the default test-set evaluation:

python src/api_base_evaluate.py

Run on the train split:

EVAL_SPLIT=train python src/api_base_evaluate.py

Outputs are written to result/<timestamp>/:

File	Description
`results.csv`	MSE, MAE, accuracy, and sample counts for each dimension.
`predictions_<dimension>.jsonl`	Per-sample prediction records.
`run_config.json`	Runtime configuration and split statistics.
`rqs_<dimension>.json`	Rationale quality results when `EVALUATE_REASONS=True`.

Leaderboard(v1.0)

Lower MSE/MAE is better. Higher HAS/RQS is better.

Rank	Model	HAS	Read.		Crit.		Comp.		Stru.		Average		RQS
Rank	Model	HAS	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	RQS
1	SurveyReviewer	0.74	1.43	0.72	1.52	0.82	1.26	0.56	1.29	0.65	1.38	0.69	0.36
2	GPT-5.2	0.68	2.13	1.07	1.97	0.97	2.04	1.08	2.98	1.47	2.28	1.15	0.42
3	Claude-Opus-4.5	0.68	2.91	1.29	1.88	0.88	2.66	1.23	3.65	1.58	2.77	1.25	0.48
4	Qwen3-32B	0.61	3.05	1.45	3.24	1.51	3.22	1.54	3.35	1.53	3.21	1.51	0.36
5	GLM-4.7	0.60	3.43	1.50	2.58	1.21	3.66	1.57	4.83	1.95	3.62	1.56	0.37
6	gemini-3-pro	0.58	3.84	1.52	2.25	1.00	3.91	1.49	5.76	2.11	3.94	1.53	0.29
7	DeepSeek-v3.2	0.58	4.78	1.88	2.49	1.15	4.59	1.82	4.02	1.76	3.97	1.65	0.37

Notes

v1.0/ contains the original SurveyReview release and v1.0-paper data for paper reproduction.
v1.1/ contains the updated release using cleaned article texts.
Article files are split into JSON shards to stay within GitHub file-size limits.
Both releases keep the SurveyReview evaluation style: MSE/MAE plus optional RQS. If you only want to verify the pipeline, set EVALUATE_REASONS=False to skip the judge-model stage.

Citation

🌟 If you find our work helpful, please leave us a star and cite our paper.

@inproceedings{zhang2026surveyreview,
  author = {Zhang, Yuheng and Wang, Yuanchun and Zhang, Fanjin and Zhao, Ruyu and Li, Juanzi and Tang, Jie and Zhang, Jing},
  title = {{SurveyReview}: A Reviewer-Aligned Benchmark for Survey Evaluators},
  booktitle = {Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2},
  year = {2026},
  pages = {10302--10313},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  location = {Jeju Island, Republic of Korea},
  series = {KDD '26},
  isbn = {979-8-4007-2259-2},
  doi = {10.1145/3770855.3817505},
  url = {https://doi.org/10.1145/3770855.3817505}
}