MLCommons Announces Formation of MLPerf Client Working Group
Creating Benchmarks for AI Acceleration on Consumer Client Systems
January 24, 2024
San Francisco, CA
Executive Summary
With the growing impact of artificial intelligence (AI) and its transformative capabilities, MLCommons, the leading organization for developing AI benchmarks, has announced the formation of a new MLPerf Client working group. This group’s mission is to create machine learning benchmarks specifically designed for client systems like desktops, laptops, and workstations running Microsoft Windows and other operating systems.
The MLPerf suite of benchmarks has set the standard for AI benchmarks in data centers, and MLCommons is now applying its collaborative, community-focused approach and expertise in machine learning (ML) to establish a consumer client systems benchmark suite.
Background
AI has become an integral part of the computing experience, seamlessly integrated into various aspects of our daily lives. Client systems are increasingly equipped with AI-specific hardware acceleration capabilities, and software vendors are incorporating AI-driven features to enhance productivity and unleash creativity for millions of end users. As these capabilities proliferate, many ML models will be executed locally on client systems, necessitating reliable and standardized methods to measure the performance and efficiency of AI acceleration solutions.
MLPerf Client Benchmarks
The MLPerf Client benchmarks will take a scenario-driven approach, focusing on real-world end-user use cases and incorporating feedback from the community. The initial benchmark will center around a large language model (LLM), specifically the Llama 2 LLM.
The MLCommons community has already encountered and overcome many of the challenges LLMs present in client systems, such as balancing performance with output quality, addressing licensing issues related to datasets and models, and ensuring safety through the integration of Llama 2-based workloads in the MLCommons training and inference benchmark suites. This experience will be instrumental in jump-starting this new client work.
Initial Participants
The MLPerf Client working group has attracted a diverse group of initial participants, including representatives from AMD, Arm, ASUSTeK, Dell Technologies, Intel, Lenovo, Microsoft, NVIDIA, and Qualcomm Technologies, Inc., among others.
Quotes from Co-Chairs
Ramesh Jaladi, Senior Director of Engineering, IP Performance Group, Intel
“Good measurements are the cornerstone of advancing AI acceleration. They enable us to set targets, track progress, and deliver enhanced end-user experiences with each product generation. The industry benefits tremendously when benchmarks align with customer needs, and that’s precisely the role we expect the MLPerf Client suite to play in consumer computing.”
Yannis Minadakis, Partner GM, Software Development, Microsoft
“Microsoft recognizes the need for tailored benchmarking tools that cater to the AI acceleration capabilities of Windows client systems. We are excited to collaborate with the MLCommons community to tackle this challenge.”
Jani Joki, Director of Performance Benchmarking, NVIDIA
“The MLPerf benchmarks have been instrumental in driving substantial advancements in machine learning performance and efficiency in data center solutions. We look forward to contributing to the creation of benchmarks that will serve a similar role in client systems.”
Vinesh Sukumar, Senior Director, AI/ML Product Management, Qualcomm
“Qualcomm is dedicated to advancing the client ecosystem. We eagerly anticipate the innovative benchmarks that the MLPerf Working Group will establish for machine learning. Benchmarks remain crucial in the development and fine-tuning of silicon, and MLCommons’ focus on end-user use cases will be key to on-device AI testing.”
Invitation to Participate
MLCommons extends an open invitation to all interested parties to join this collaborative effort. For more information on the MLPerf Client working group, including details on how to join and contribute to the benchmarks, please visit the working group page or contact the chairs via email at [email protected].
About MLCommons
MLCommons is the world leader in building benchmarks for AI. As an open engineering consortium, its mission is to make machine learning better for everyone through benchmarks and data. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly gained recognition as a set of industry metrics to measure machine learning performance and promote transparency in machine learning techniques. In collaboration with its 125+ members, including global technology providers, academics, and researchers, MLCommons focuses on collaborative engineering work to build tools for the entire machine learning industry through benchmarks and metrics, public datasets, and best practices. For additional information regarding MLCommons and details on becoming a Member or Affiliate of the organization, please visit the MLCommons website and contact [email protected].