OpenZeppelin says EVMbench’s Dataset Contains Training Data Leaks
OpenZeppelin found data contamination and methodological flaws in OpenAI's EVMbench blockchain security benchmark.Summary
Blockchain security firm OpenZeppelin audited OpenAI's EVMbench, an AI benchmark for smart contract security developed with Paradigm, and identified significant issues. The primary concern is training data contamination, as the highest-scoring AI agents were likely exposed to the benchmark's vulnerability reports during pretraining, given the dataset's reliance on audits up to mid-2025. Additionally, OpenZeppelin noted methodological flaws, including the classification of at least four non-exploitable issues as high-severity vulnerabilities. While acknowledging AI's future role in blockchain security, OpenZeppelin stressed the necessity of rigorous, contamination-free data and benchmarks to properly evaluate these tools.
(Source：Cointelegraph)