Gao Tianxi

Anti-Contamination Evaluation: Verifiable Online Benchmark Generation with LLM

Tianxi Gao, Yufan Cai, Jun Niu, Zhe Hou, Jin Song Dong, NeurIPS 2025(under review)  

Recent large language models (LLMs) perform well on math benchmarks, but current evaluation datasets are static, often  contaminated, and fail to distinguish real reasoning from memorization. The paper introduces an automated framework that  uses formal methods to generate clean, randomized, and verifiable problems for robust LLM evaluation. The system includes:  (1) translating natural language into formal code, (2) randomizing variables within valid ranges, (3) verifying problem  correctness using formal tools, and (4) comparing LLM outputs with verified answers on newly generated test sets.

National E-Commerce “Innovation, Creation, Entrepreneurship” Challenge Competition

Second prize in Sichuan province

Scholarship

Leader student scholarship (top 30%)