Search Results for author:
Found 18 papers, 12 papers with code
AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
This work represents a crucial step toward establishing global standards for AI risk and reliability evaluation while acknowledging the need for continued development in areas such as multiturn interactions, multimodal understanding, coverage of additional languages, and emerging hazard categories.
Introducing v0.5 of the AI Safety Benchmark from MLCommons
We created a new taxonomy of 13 hazard categories, of which 7 have tests in the v0. 5 benchmark.
Croissant: A Metadata Format for ML-Ready Datasets
Data is a critical resource for machine learning (ML), yet working with data remains a key friction point.
DMLR: Data-centric Machine Learning Research -- Past, Present and Future
Drawing from discussions at the inaugural DMLR workshop at ICML 2023 and meetings prior, in this report we outline the relevance of community engagement and infrastructure development for the creation of next-generation public datasets that will advance machine learning science.
Benchmarking Neural Network Training Algorithms
In order to address these challenges, we introduce a new, competitive, time-to-result benchmark using multiple workloads running on fixed hardware, the AlgoPerf: Training Algorithms benchmark.
Understanding metric-related pitfalls in image analysis validation
Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice.
DataPerf: Benchmarks for Data-Centric AI Development
Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems.
Metrics reloaded: Recommendations for image analysis validation
The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint - a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), data set and algorithm output.
Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks
We introduce Dynatask: an open source system for setting up custom NLP tasks that aims to greatly lower the technical knowledge and effort required for hosting and evaluating state-of-the-art NLP models, as well as for conducting model in the loop data collection with crowdworkers.
MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems
Scientific communities are increasingly adopting machine learning and deep learning models in their applications to accelerate scientific insights.
MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation
Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience.
Common Limitations of Image Processing Metrics: A Picture Story
While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation.
GaNDLF: A Generally Nuanced Deep Learning Framework for Scalable End-to-End Clinical Workflows in Medical Imaging
Deep Learning (DL) has the potential to optimize machine learning in both the scientific and clinical communities.
Data Engineering for Everyone
This article shows that open-source data sets are the rocket fuel for research and innovation at even some of the largest AI organizations.
MLPerf Mobile Inference Benchmark
This paper presents the first industry-standard open-source machine learning (ML) benchmark to allow perfor mance and accuracy evaluation of mobile devices with different AI chips and software stacks.
MLPerf Training Benchmark
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML.
Scale MLPerf-0.6 models on Google TPU-v3 Pods
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0. 6 training benchmark demonstrates the scalability of a suite of industry relevant ML models.
