Search Results for author:
Found 17 papers, 3 papers with code
Deep Learning Scaling is Predictable, Empirically
As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art.
A Proposed Hierarchy of Deep Learning Tasks
As the pace of deep learning innovation accelerates, it becomes increasingly important to organize the space of problems by relative difficultly.
Empirically Characterizing Overparameterization Impact on Convergence
A long-held conventional wisdom states that larger models train more slowly when using gradient descent.
A Static Analysis-based Cross-Architecture Performance Prediction Using Machine Learning
Porting code from CPU to GPU is costly and time-consuming; Unless much time is invested in development and optimization, it is not obvious, a priori, how much speed-up is achievable or how much room is left for improvement.
Beyond Human-Level Accuracy: Computational Challenges in Deep Learning
However, recent prior work shows that as dataset sizes grow, DL model accuracy and model size grow predictably.
Model Architecture Controls Gradient Descent Dynamics: A Combinatorial Path-Based Formula
However, our theoretical understanding of how model architecture affects performance or accuracy is limited.
Sustainable AI: Environmental Implications, Challenges and Opportunities
This paper explores the environmental impact of the super-linear growth trends for AI from a holistic perspective, spanning Data, Algorithms, and System Hardware.
Time and the Value of Data
In addition, we argue that increasing the stock of data by including older datasets may, in fact, damage the model's accuracy.
Time Dependency, Data Flow, and Competitive Advantage
Relating the text topics to various business areas of interest, we argue that competing in a business area in which data value decays rapidly alters strategies to acquire competitive advantage.
DataPerf: Benchmarks for Data-Centric AI Development
Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems.
Understanding Scaling Laws for Recommendation Models
We show that parameter scaling is out of steam for the model architecture under study, and until a higher-performing model architecture emerges, data scaling is the path forward.
MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation
Based on our characterization of various embedding representations, we propose a hybrid embedding representation that achieves higher quality embeddings at the cost of increased memory and compute requirements.
Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
We propose three optimization techniques to mitigate sources of inefficiencies, namely (1) Dynamic gating, (2) Expert Buffering, and (3) Expert load balancing.
Sieve: Multimodal Dataset Pruning Using Image Captioning Models
We propose a pruning signal, Sieve, that employs synthetic captions generated by image-captioning models pretrained on small, diverse, and well-aligned image-text pairs to evaluate the alignment of noisy image-text pairs.
MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems
Training and deploying large machine learning (ML) models is time-consuming and requires significant distributed computing infrastructures.
Data Acquisition: A New Frontier in Data-centric AI
As Machine Learning (ML) systems continue to grow, the demand for relevant and comprehensive datasets becomes imperative.
Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Data
Armed with this knowledge, we devise novel pruning metrics that operate in embedding space to identify and remove low-quality entries in the Stack dataset.
