Search Results for author:
Found 20 papers, 4 papers with code
Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM).
Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems
We find empirically that across multiple language tasks, surprisingly, Voting Inference Systems' performance first increases but then decreases as a function of the number of LLM calls.
Data Acquisition: A New Frontier in Data-centric AI
As Machine Learning (ML) systems continue to grow, the demand for relevant and comprehensive datasets becomes imperative.
How is ChatGPT's behavior changing over time?
We find that the performance and behavior of both GPT-3. 5 and GPT-4 can vary greatly over time.
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
There is a rapidly growing number of large language models (LLMs) that users can query for a fee.
SEAL : Interactive Tool for Systematic Error Analysis and Labeling
With the advent of Transformers, large language models (LLMs) have saturated well-known NLP benchmarks and leaderboards with high aggregate performance.
HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictions
HAPI is the first large-scale dataset of ML API usages and is a unique resource for studying ML-as-a-service (MLaaS).
Estimating and Explaining Model Performance When Both Covariates and Labels Shift
We further propose SEES, an algorithmic framework to characterize the distribution shift under SJS and to estimate a model's performance on new data without any labels.
DataPerf: Benchmarks for Data-Centric AI Development
Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems.
Solon: Communication-efficient Byzantine-resilient Distributed Training via Redundant Gradients
There has been a growing need to provide Byzantine-resilience in distributed model training.
How Did the Model Change? Efficiently Assessing Machine Learning API Shifts
ML prediction APIs from providers like Amazon and Google have made it simple to use ML in applications.
Did the Model Change? Efficiently Assessing Machine Learning API Shifts
This motivated us to formulate the API shift assessment problem at a more fine-grained level as estimating how the API model's confusion matrix changes over time when the data distribution is constant.
Efficient Online ML API Selection for Multi-Label Classification Tasks
In this work, we propose FrugalMCT, a principled framework that adaptively selects the APIs to use for different data in an online fashion while respecting user's budget.
FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply
Prediction APIs offered for a fee are a fast-growing industry and an important part of machine learning as a service.
The Effect of Network Width on the Performance of Large-batch Training
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training.
Model-based Pricing for Machine Learning in a Data Marketplace
Finally, we conduct extensive experiments, which validate that the MBP framework can provide high revenue to the seller, high affordability to the buyer, and also operate on low runtime cost.
DRACO: Byzantine-resilient Distributed Training via Redundant Gradients
Distributed model training is vulnerable to byzantine system failures and adversarial compute nodes, i. e., nodes that use malicious updates to corrupt the global model stored at a parameter server (PS).
The Manifold Assumption and Defenses Against Adversarial Perturbations
Interestingly, we find that a recent objective by Madry et al. encourages training a model that satisfies well our formal version of the goodness property, but has a weak control of points that are wrong but with low confidence.
Reinforcing Adversarial Robustness using Model Confidence Induced by Adversarial Training
In this paper we study leveraging confidence information induced by adversarial training to reinforce adversarial robustness of a given adversarially trained model.
Tuple-oriented Compression for Large-scale Mini-batch Stochastic Gradient Descent
We fill this crucial research gap by proposing a new lossless compression scheme we call tuple-oriented compression (TOC) that is inspired by an unlikely source, the string/text compression scheme Lempel-Ziv-Welch, but tailored to MGD in a way that preserves tuple boundaries within mini-batches.
