Search Results for author:
Found 18 papers, 3 papers with code
Text Quality-Based Pruning for Efficient Training of Language Models
In recent times training Language Models (LMs) have relied on computationally heavy training over massive datasets which makes this training process extremely laborious.
Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Data
Armed with this knowledge, we devise novel pruning metrics that operate in embedding space to identify and remove low-quality entries in the Stack dataset.
Data Acquisition: A New Frontier in Data-centric AI
As Machine Learning (ML) systems continue to grow, the demand for relevant and comprehensive datasets becomes imperative.
MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems
Training and deploying large-scale machine learning models is time-consuming, requires significant distributed computing infrastructures, and incurs high operational costs.
Sieve: Multimodal Dataset Pruning Using Image Captioning Models
We propose a pruning signal, Sieve, that employs synthetic captions generated by image-captioning models pretrained on small, diverse, and well-aligned image-text pairs to evaluate the alignment of noisy image-text pairs.
Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
We propose three optimization techniques to mitigate sources of inefficiencies, namely (1) Dynamic gating, (2) Expert Buffering, and (3) Expert load balancing.
MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation
Based on our characterization of various embedding representations, we propose a hybrid embedding representation that achieves higher quality embeddings at the cost of increased memory and compute requirements.
Understanding Scaling Laws for Recommendation Models
We show that parameter scaling is out of steam for the model architecture under study, and until a higher-performing model architecture emerges, data scaling is the path forward.
DataPerf: Benchmarks for Data-Centric AI Development
Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems.
Time Dependency, Data Flow, and Competitive Advantage
Relating the text topics to various business areas of interest, we argue that competing in a business area in which data value decays rapidly alters strategies to acquire competitive advantage.
Time and the Value of Data
In addition, we argue that increasing the stock of data by including older datasets may, in fact, damage the model's accuracy.
Sustainable AI: Environmental Implications, Challenges and Opportunities
This paper explores the environmental impact of the super-linear growth trends for AI from a holistic perspective, spanning Data, Algorithms, and System Hardware.
Model Architecture Controls Gradient Descent Dynamics: A Combinatorial Path-Based Formula
However, our theoretical understanding of how model architecture affects performance or accuracy is limited.
Beyond Human-Level Accuracy: Computational Challenges in Deep Learning
However, recent prior work shows that as dataset sizes grow, DL model accuracy and model size grow predictably.
A Static Analysis-based Cross-Architecture Performance Prediction Using Machine Learning
Porting code from CPU to GPU is costly and time-consuming; Unless much time is invested in development and optimization, it is not obvious, a priori, how much speed-up is achievable or how much room is left for improvement.
Empirically Characterizing Overparameterization Impact on Convergence
A long-held conventional wisdom states that larger models train more slowly when using gradient descent.
A Proposed Hierarchy of Deep Learning Tasks
As the pace of deep learning innovation accelerates, it becomes increasingly important to organize the space of problems by relative difficultly.
Deep Learning Scaling is Predictable, Empirically
As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art.
