AI Benchmark Democratization and Carpentry

📝 Original Info

• Title: AI Benchmark Democratization and Carpentry
• ArXiv ID: 2512.11588
• Date: 2025-12-12
• Authors: Gregor von Laszewski, Wesley Brewer, Jeyan Thiyagalingam, Juri Papay, Armstrong Foundjem, Piotr Luszczek, Murali Emani, Shirley V. Moore, Vijay Janapa Reddi, Matthew D. Sinclair, Sebastian Lobentanzer, Sujata Goswami, Benjamin Hawks, Marco Colombo, Nhan Tran, Christine R. Kirkpatrick, Abdulkareem Alsudais, Gregg Barrett, Tianhao Li, Kirsten Morehouse, Shivaram Venkataraman, Rutwik Jain, Kartik Mathur, Victor Lu, Tejinder Singh, Khojasteh Z. Mirza, Kongtao Chen, Sasidhar Kunapuli, Gavin Farrell, Renato Umeton, Geoffrey C. Fox

📝 Abstract

Benchmarks are one cornerstone of modern machine learning practice, providing standardized evaluations that enable reproducibility, comparison, and scientific progress. However, AI benchmarks are becoming increasingly complex, requiring special care, including AI focused dynamic workflows. This is evident by the rapid evolution of AI models in architecture, scale, and capability; the evolution of datasets; and deployment contexts continuously change, creating a moving target for evaluation. Large language models in particular are known for their memorization of static benchmarks, which causes a drastic difference between benchmark results and real-world performance. Beyond the accepted static benchmarks we know from the traditional computing community, we need to develop and evolve contin...

📄 Full Content

...(본문 내용이 길어 생략되었습니다. 사이트에서 전문을 확인해 주세요.)