Search Results for author:
Found 133 papers, 85 papers with code
Mechanistic Design and Scaling of Hybrid Architectures
The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale model training and evaluation.
Simple linear attention language models balance the recall-throughput tradeoff
In this work, we explore whether we can improve language model efficiency (e. g. by reducing memory consumption) without compromising on recall.
Prospector Heads: Generalized Feature Attribution for Large Models & Data
Feature attribution, the ability to localize regions of the input data that are relevant for classification, is an important capability for machine learning models in scientific and biomedical domains.
Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT
Retrieval pipelines-an integral component of many machine learning systems-perform poorly in domains where documents are long (e. g., 10K tokens or more) and where identifying the relevant document requires synthesizing information across the entire text.
Hydragen: High-Throughput LLM Inference with Shared Prefixes
Decoding in this large-batch setting can be bottlenecked by the attention operation, which reads large key-value (KV) caches from memory and computes inefficient matrix-vector products for every sequence in the batch.
The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry
Experiments show Hedgehog recovers over 99% of standard Transformer quality in train-from-scratch and finetuned-conversion settings, outperforming prior linear attentions up to 6 perplexity points on WikiText-103 with causal GPTs, and up to 8. 7 GLUE score points on finetuned bidirectional BERTs.
Zoology: Measuring and Improving Recall in Efficient Language Models
To close the gap between synthetics and real language, we develop a new formalization of the task called multi-query associative recall (MQAR) that better reflects actual language.
FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores
FlashFFTConv uses a matrix decomposition that computes the FFT using matrix multiply units and enables kernel fusion for long sequences, reducing I/O.
Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture
We ask: are there performant architectures that can scale sub-quadratically along sequence length and model dimension?
LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform?
H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Based on these insights, we propose Heavy Hitter Oracle (H$_2$O), a KV cache eviction policy that dynamically retains a balance of recent and H$_2$ tokens.
Towards trustworthy seizure onset detection using workflow notes
We find that our multilabel model significantly improves overall seizure onset detection performance (+5. 9 AUROC points) while greatly improving performance among subgroups (up to +8. 3 AUROC points), and decreases false positives on non-epileptiform abnormalities by 8 FPR points.
Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes
Code synthesis is cheap, but far less accurate than directly processing each document with the LLM.
Effectively Modeling Time Series with Simple Discrete State Spaces
For expressivity, we propose a new SSM parameterization based on the companion matrix -- a canonical representation for discrete-time processes -- which enables SpaceTime's SSM layers to learn desirable autoregressive processes.
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144.
Collage Diffusion
We seek to give users precise control over diffusion-based image generation by modeling complex scenes as sequences of layers, which define the desired spatial arrangement and visual attributes of objects in the scene.
Hyena Hierarchy: Towards Larger Convolutional Language Models
Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale.
Ranked #37 on
Language Modelling
on WikiText-103
Simple Hardware-Efficient Long Convolutions for Sequence Modeling
We find that a key requirement to achieving high performance is keeping the convolution kernels smooth.
Hungry Hungry Hippos: Towards Language Modeling with State Space Models
First, we use synthetic language modeling tasks to understand the gap between SSMs and attention.
Ranked #2 on
Language Modelling
on The Pile
(Test perplexity metric)
Transform Once: Efficient Operator Learning in Frequency Domain
Instead, this work introduces a blueprint for frequency domain learning through a single transform: transform once (T1).
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces
On ImageNet-1k, S4ND exceeds the performance of a Vision Transformer baseline by $1. 5\%$ when training with a $1$D sequence of patches, and matches ConvNeXt when modeling images in $2$D.
Ask Me Anything: A simple strategy for prompting language models
Prompting is a brittle process wherein small modifications to the prompt can cause large variations in the model predictions, and therefore significant effort is dedicated towards designing a painstakingly "perfect prompt" for a task.
Ranked #1 on
Question Answering
on Story Cloze
HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictions
HAPI is the first large-scale dataset of ML API usages and is a unique resource for studying ML-as-a-service (MLaaS).
LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoning
Finally-inspired by the Open Science movement-we make a call for the legal and computer science communities to join our efforts by contributing new tasks.
Contrastive Adapters for Foundation Model Group Robustness
We also find that efficient ways to improve model inference (e. g., via adapters, lightweight networks with FM embeddings as inputs) do not consistently improve and can sometimes hurt group robustness compared to zero-shot (e. g., increasing the accuracy gap by 50. 1 pp on CelebA).
How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections
Linear time-invariant state space models (SSM) are a classical model from engineering and statistics, that have recently been shown to be very promising in machine learning through the Structured State Space sequence model (S4).
Ranked #7 on
Long-range modeling
on LRA
On the Parameterization and Initialization of Diagonal State Space Models
On the other hand, a recent variant of S4 called DSS showed that restricting the state matrix to be fully diagonal can still preserve the performance of the original model when using a specific initialization based on approximating S4's matrix.
Ranked #9 on
Long-range modeling
on LRA
Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data
At their core, these frameworks learn the dynamics of brain activity by modeling sequences of activity akin to how sequences of text are modeled in NLP.
The Importance of Background Information for Out of Distribution Generalization
Domain generalization in medical image classification is an important problem for trustworthy machine learning to be deployed in healthcare.
Comparing interpretation methods in mental state decoding analyses with deep learning models
Deep learning (DL) models find increasing application in mental state decoding, where researchers seek to understand the mapping between mental states (e. g., perceiving fear or joy) and brain activity by identifying those brain regions (and networks) whose activity allows to accurately identify (i. e., decode) these states.
Can Foundation Models Help Us Achieve Perfect Secrecy?
However, privacy and quality appear to be in tension in existing systems for personal tasks.
Can Foundation Models Wrangle Your Data?
Foundation Models (FMs) are models trained on large corpora of data that, at very large scale, can generalize to new tasks without any task-specific finetuning.
Ranked #10 on
Entity Resolution
on Amazon-Google
TABi: Type-Aware Bi-Encoders for Open-Domain Entity Retrieval
Entity retrieval--retrieving information about entity mentions in a query--is a key step in open-domain tasks, such as question answering or fact checking.
Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learning
We first prove that adding a weighted class-conditional InfoNCE loss to SupCon controls the degree of spread.
Monarch: Expressive Structured Matrices for Efficient and Accurate Training
To address these issues, we propose a class of matrices (Monarch) that is hardware-efficient (they are parameterized as products of two block-diagonal matrices for better hardware utilization) and expressive (they can represent many commonly used transforms).
Domino: Discovering Systematic Errors with Cross-Modal Embeddings
In this work, we address these challenges by first designing a principled evaluation framework that enables a quantitative comparison of SDMs across 1, 235 slice discovery settings in three input domains (natural images, medical images, and time-series data).
Shoring Up the Foundations: Fusing Model Embeddings and Weak Supervision
Despite the black-box nature of foundation models, we prove results characterizing how our approach improves performance and show that lift scales with the smoothness of label distributions in embedding space.
SKM-TEA: A Dataset for Accelerated MRI Reconstruction with Dense Image Labels for Quantitative Clinical Evaluation
While recent machine learning methods for MRI reconstruction and analysis have shown promise for reducing this burden, these techniques are primarily validated with imperfect image quality metrics, which are discordant with clinically-relevant measures that ultimately hamper clinical deployment and clinician trust.
Reasoning over Public and Private Data in Retrieval-Based Systems
We first define the PUBLIC-PRIVATE AUTOREGRESSIVE INFORMATION RETRIEVAL (PAIR) privacy framework for the novel retrieval setting over multiple privacy scopes.
Ranked #1 on
Multi-hop Question Answering
on ConcurrentQA
Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations
As ERM models can be good spurious attribute predictors, CNC works by (1) using a trained ERM model's outputs to identify samples with the same class but dissimilar spurious features, and (2) training a robust model with contrastive learning to learn similar representations for same-class samples.
It's Raw! Audio Generation with State-Space Models
SaShiMi yields state-of-the-art performance for unconditional waveform generation in the autoregressive setting.
BARACK: Partially Supervised Group Robustness With Guarantees
Theoretically, we provide generalization bounds for our approach in terms of the worst-group performance, which scale with respect to both the total number of training points and the number of training points with group labels.
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
To address this, our main insight is to optimize over a continuous superset of sparse matrices with a fixed structure known as products of butterfly matrices.
Personalized Benchmarking with the Ludwig Benchmarking Toolkit
LBT provides a configurable interface for controlling training and customizing evaluation, a standardized training framework for eliminating confounding variables, and support for multi-objective evaluation.
VORTEX: Physics-Driven Data Augmentations Using Consistency Training for Robust Accelerated MRI Reconstruction
Deep neural networks have enabled improved image quality and fast inference times for various inverse problems, including accelerated magnetic resonance imaging (MRI) reconstruction.
Efficiently Modeling Long Sequences with Structured State Spaces
A central goal of sequence modeling is designing a single principled model that can address sequence data across a range of modalities and tasks, particularly on long-range dependencies.
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
Recent advances in efficient Transformers have exploited either the sparsity or low-rank properties of attention matrices to reduce the computational and memory bottlenecks of modeling long sequences.
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
Recurrent neural networks (RNNs), temporal convolutions, and neural differential equations (NDEs) are popular families of deep learning models for time-series data, each with unique strengths and tradeoffs in modeling power and computational efficiency.
Ranked #2 on
Sequential Image Classification
on Sequential MNIST
Cross-Domain Data Integration for Named Entity Disambiguation in Biomedical Text
Named entity disambiguation (NED), which involves mapping textual mentions to structured entities, is particularly challenging in the medical domain due to the presence of rare entities.
The Details Matter: Preventing Class Collapse in Supervised Contrastive Learning
Supervised contrastive learning optimizes a loss that pushes together embeddings of points from the same class while pulling apart embeddings of points from different classes.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Challenges for cognitive decoding using deep learning methods
In cognitive decoding, researchers aim to characterize a brain region's representations by identifying the cognitive states (e. g., accepting/rejecting a gamble) that can be identified from the region's activity.
Declarative Machine Learning Systems
In this article we will describe how ML systems are currently structured, highlight important factors for their success and adoption, what are the issues current ML systems are facing and how the systems we developed addressed them.
Mandoline: Model Evaluation under Distribution Shift
If an unlabeled sample from the target distribution is available, along with a labeled sample from a possibly different source distribution, standard approaches such as importance weighting can be applied to estimate performance on the target.
HoroPCA: Hyperbolic Dimensionality Reduction via Horospherical Projections
Given directions, PCA relies on: (1) a parameterization of subspaces spanned by these directions, (2) a method of projection onto subspaces that preserves information in these directions, and (3) an objective to optimize, namely the variance explained by projections.
Ember: No-Code Context Enrichment via Similarity-Based Keyless Joins
Context enrichment, or rebuilding fragmented context, using keyless joins is an implicit or explicit step in machine learning (ML) pipelines over structured data sources.
Scatterbrain: Unifying Sparse and Low-rank Attention
Recent advances in efficient Transformers have exploited either the sparsity or low-rank properties of attention matrices to reduce the computational and memory bottlenecks of modeling long sequences.
Rethinking Neural Operations for Diverse Tasks
An important goal of AutoML is to automate-away the design of neural networks on new tasks in under-explored domains.
Comparing the Value of Labeled and Unlabeled Data in Method-of-Moments Latent Variable Estimation
We apply our decomposition framework to three scenarios -- well-specified, misspecified, and corrected models -- to 1) choose between labeled and unlabeled data and 2) learn from their combination.
Robustness Gym: Unifying the NLP Evaluation Landscape
Despite impressive performance on standard benchmarks, deep neural networks are often brittle when deployed in real-world systems.
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
Modern neural network architectures use structured linear transformations, such as low-rank matrices, sparse matrices, permutations, and the Fourier transform, to improve inference speed and reduce memory usage compared to general linear maps.
No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems
As the subclass labels are frequently unavailable, models trained using only the coarser-grained class labels often exhibit highly variable performance across different subclasses.
Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transfers
Intuitively, the transfer effect from one task to another task depends on dataset shifts such as sample sizes and covariance matrices.
From Trees to Continuous Embeddings and Back: Hyperbolic Hierarchical Clustering
Recently, Dasgupta reframed HC as a discrete optimization problem by introducing a global cost function measuring the quality of a given tree.
Model Patching: Closing the Subgroup Performance Gap with Data Augmentation
Particularly concerning are models with inconsistent performance on specific subgroups of a class, e. g., exhibiting disparities in skin cancer classification in the presence or absence of a spurious bandage.
Train and You'll Miss It: Interactive Model Iteration with Weak Supervision and Pre-Trained Embeddings
Our goal is to enable machine learning systems to be trained interactively.
Contextual Embeddings: When Are They Worth It?
We study the settings for which deep contextual embeddings (e. g., BERT) give large improvements in performance relative to classic pretrained embeddings (e. g., GloVe), and an even simpler baseline---random word embeddings---focusing on the impact of the training set size and the linguistic properties of the task.
Machine Learning on Graphs: A Model and Comprehensive Taxonomy
The second, graph regularized neural networks, leverages graphs to augment neural network losses with a regularization objective for semi-supervised learning.
On the Generalization Effects of Linear Transformations in Data Augmentation
We validate our proposed scheme on image and text datasets.
Understanding and Improving Information Transfer in Multi-Task Learning
We investigate multi-task learning approaches that use a shared feature representation for all tasks.
Low-Dimensional Hyperbolic Knowledge Graph Embeddings
However, existing hyperbolic embedding methods do not account for the rich logical patterns in KGs.
Ranked #5 on
Link Prediction
on YAGO3-10
Ivy: Instrumental Variable Synthesis for Causal Inference
To relax these assumptions, we propose Ivy, a new method to combine IV candidates that can handle correlated and invalid IV candidates in a robust manner.
Assessing Robustness to Noise: Low-Cost Head CT Triage
We then show that the trained model is robust to reduced tube current and fewer projections, with the AUROC dropping only 0. 65% for images acquired with a 16x reduction in tube current and 0. 22% for images acquired with 8x fewer projections.
Understanding the Downstream Instability of Word Embeddings
To theoretically explain this tradeoff, we introduce a new measure of embedding instability---the eigenspace instability measure---which we prove bounds the disagreement in downstream predictions introduced by the change in word embeddings.
Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods
In this work, we show that, for a class of latent variable models highly applicable to weak supervision, we can find a closed-form solution to model parameters, obviating the need for iterative solutions like stochastic gradient descent (SGD).
Hyperbolic Graph Convolutional Neural Networks
Here we propose Hyperbolic Graph Convolutional Neural Network (HGCN), the first inductive hyperbolic GCN that leverages both the expressiveness of GCNs and hyperbolic geometry to learn inductive node representations for hierarchical and scale-free graphs.
Ranked #1 on
Link Prediction
on PPI
(Accuracy metric)
Multi-Resolution Weak Supervision for Sequential Data
Multi-resolution sources exacerbate this challenge due to complex correlations and sample complexity that scales in the length of the sequence.
PipeMare: Asynchronous Pipeline Parallel DNN Training
Pipeline parallelism (PP) when training neural networks enables larger models to be partitioned spatially, leading to both lower network communication and overall higher hardware utilization.
Rekall: Specifying Video Events using Compositions of Spatiotemporal Labels
Many real-world video analysis applications require the ability to identify domain-specific events in video, such as interviews and commercials in TV news broadcasts, or action sequences in film.
Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging
Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing.
Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices
In real-world machine learning applications, data subsets correspond to especially critical outcomes: vulnerable cyclist detections are safety-critical in an autonomous driving task, and "question" sentences might be important to a dialogue agent's language understanding for product purposes.
Overton: A Data System for Monitoring and Improving Machine-Learned Products
We describe a system called Overton, whose main design goal is to support engineers in building, monitoring, and improving production machine learning systems.
On the Downstream Performance of Compressed Word Embeddings
Finally, we show that by using the eigenspace overlap score as a selection criterion between embeddings drawn from a representative set we compressed, we can efficiently identify the better performing embedding with up to $2\times$ lower selection error rates than the next best measure of compression quality, and avoid the cost of training a model for each task of interest.
Learning Mixed-Curvature Representations in Product Spaces
The quality of the representations achieved by embeddings is determined by how well the geometry of the embedding space matches the structure of the data.
Low-Memory Neural Network Training: A Technical Report
In this paper we study a fundamental question: How much memory is actually needed to train a neural network?
Medical device surveillance with electronic health records
Using hip replacements as a test case, our methods accurately extracted implant details and reports of complications and pain from electronic health records with up to 96. 3% precision, 98. 5% recall, and 97. 4% F1, improved classification performance by 12. 7- 53. 0% over rule-based methods, and detected over 6 times as many complication events compared to using structured data alone.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Cross-Modal Data Programming Enables Rapid Medical Machine Learning
Labeling training datasets has become a key barrier to building medical machine learning models.
Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations
Fast linear transforms are ubiquitous in machine learning, including the discrete Fourier transform, discrete cosine transform, and other structured transformations such as convolutions.
Learning Dependency Structures for Weak Supervision Models
Labeling training data is a key bottleneck in the modern machine learning pipeline.
Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale
Labeling training data is one of the most costly bottlenecks in developing machine learning-based applications.
Low-Precision Random Fourier Features for Memory-Constrained Kernel Approximation
We investigate how to train kernel approximation methods that generalize well under a memory budget.
Training Complex Models with Multi-Task Weak Supervision
Snorkel MeTaL: A framework for training models with multi-task weak supervision
Ranked #1 on
Semantic Textual Similarity
on SentEval
Learning Compressed Transforms with Low Displacement Rank
The low displacement rank (LDR) framework for structured matrices represents a matrix through two displacement operators and a low-rank residual.
Hypertree Decompositions Revisited for PGMs
We revisit the classical problem of exact inference on probabilistic graphical models (PGMs).
Training Classifiers with Natural Language Explanations
Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification).
Representation Tradeoffs for Hyperbolic Embeddings
Given a tree, we give a combinatorial construction that embeds the tree in hyperbolic space with arbitrarily low distortion without using optimization.
Hypertree Decompositions Revisited for PGMs
We revisit the classical problem of exact inference on probabilistic graphical models (PGMs).
A Kernel Theory of Modern Data Augmentation
Data augmentation, a technique in which a training set is expanded with class-preserving transformations, is ubiquitous in modern machine learning pipelines.
High-Accuracy Low-Precision Training
Low-precision computation is often used to lower the time and energy cost of machine learning, and recently hardware accelerators have been developed to support it.
Snorkel: Rapid Training Data Creation with Weak Supervision
In a user study, subject matter experts build models 2. 8x faster and increase predictive performance an average 45. 5% versus seven hours of hand labeling.
Gaussian Quadrature for Kernel Features
We show that deterministic feature maps can be constructed, for any $\gamma > 0$, to achieve error $\epsilon$ with $O(e^{e^\gamma} + \epsilon^{-1/\gamma})$ samples as $\epsilon$ goes to 0.
Inferring Generative Model Structure with Static Analysis
Obtaining enough labeled data to robustly train complex discriminative models is a major bottleneck in the machine learning pipeline.
Learning to Compose Domain-Specific Transformations for Data Augmentation
Data augmentation is a ubiquitous technique for increasing the size of labeled training sets by leveraging task-specific data transformations that preserve class labels.
Accelerated Stochastic Power Iteration
We propose a simple variant of the power iteration with an added momentum term, that achieves both the optimal sample and iteration complexity.
ShortFuse: Biomedical Time Series Representations in the Presence of Structured Information
In healthcare applications, temporal variables that encode movement, health status and longitudinal patient evolution are often accompanied by rich structured information such as demographics, diagnostics and medical exam data.
SwellShark: A Generative Model for Biomedical Named Entity Recognition without Labeled Data
We present SwellShark, a framework for building biomedical named entity recognition (NER) systems quickly and without hand-labeled data.
Ranked #2 on
Weakly-Supervised Named Entity Recognition
on BC5CDR
Fonduer: Knowledge Base Construction from Richly Formatted Data
We focus on knowledge base construction (KBC) from richly formatted data.
Databases
Learning the Structure of Generative Models without Labeled Data
Curating labeled training data has become the primary bottleneck in machine learning.
Socratic Learning: Augmenting Generative Models to Incorporate Latent Subsets in Training Data
Prior work has explored learning accuracies for these sources even without ground truth labels, but they assume that a single accuracy parameter is sufficient to model the behavior of these sources over the entire training set.
Sub-sampled Newton Methods with Non-uniform Sampling
As second-order methods prove to be effective in finding the minimizer to a high-precision, in this work, we propose randomized Newton-type algorithms that exploit \textit{non-uniform} sub-sampling of $\{\nabla^2 f_i(w)\}_{i=1}^{n}$, as well as inexact updates, as means to reduce the computational complexity.
Parallel SGD: When does averaging help?
Consider a number of workers running SGD independently on the same pool of data and averaging the models every once in a while -- a common but not well understood practice.
Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs
Given a specification of a convolutional neural network, our goal is to minimize the time to train this model on a cluster of commodity CPUs and GPUs.
Scan Order in Gibbs Sampling: Models in Which it Matters and Bounds on How Much
Gibbs sampling is a Markov Chain Monte Carlo sampling technique that iteratively samples variables from their conditional distributions.
Asynchrony begets Momentum, with an Application to Deep Learning
Since asynchronous methods have better hardware efficiency, this result may shed light on when asynchronous execution is more efficient for deep learning systems.
Data Programming: Creating Large Training Sets, Quickly
Additionally, in initial user studies we observed that data programming may be an easier way for non-experts to create machine learning models when training data is limited or unavailable.
Ensuring Rapid Mixing and Low Bias for Asynchronous Gibbs Sampling
Gibbs sampling is a Markov chain Monte Carlo technique commonly used for estimating marginal distributions.
Asynchronous stochastic convex optimization: the noise is in the noise and SGD don't care
We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures.
Taming the Wild: A Unified Analysis of Hogwild-Style Algorithms
Stochastic gradient descent (SGD) is a ubiquitous algorithm for a variety of machine learning problems.
Rapidly Mixing Gibbs Sampling for a Class of Factor Graphs Using Hierarchy Width
Gibbs sampling on factor graphs is a widely used inference technique, which often produces good empirical results.
Asynchronous stochastic convex optimization
We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures.
Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries
The complexity of the visual world creates significant challenges for comprehensive visual understanding.
Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
with relaxed assumptions on the sparsity of the problem; (2) we analyze asynchronous SGD algorithms for non-convex matrix problems including matrix completion; and (3) we design and analyze an asynchronous SGD algorithm, called Buckwild!, that uses lower-precision arithmetic.
Caffe con Troll: Shallow Ideas to Speed Up Deep Learning
We present Caffe con Troll (CcT), a fully compatible end-to-end version of the popular framework Caffe with rebuilt internals.
Weighted SGD for $\ell_p$ Regression with Randomized Preconditioning
We aim to bridge the gap between these two methods in solving constrained overdetermined linear regression problems---e. g., $\ell_2$ and $\ell_1$ regression problems.
Incremental Knowledge Base Construction Using DeepDive
Populating a database with unstructured information is a long-standing problem in industry and research that encompasses problems of extraction, cleaning, and integration.
Parallel Feature Selection Inspired by Group Testing
Superior performance of our method is demonstrated on a challenging relation extraction task from a very large data set that have both redundant features and sample size in the order of millions.
Global Convergence of Stochastic Gradient Descent for Some Non-convex Matrix Problems
Stochastic gradient descent (SGD) on a low-rank factorization is commonly employed to speed up matrix problems including matrix completion, subspace tracking, and SDP relaxation.
Feature Engineering for Knowledge Base Construction
Our approach to KBC is based on joint probabilistic inference and learning, but we do not see inference as either a panacea or a magic bullet: inference is a tool that allows us to be systematic in how we construct, debug, and improve the quality of such systems.
A machine-compiled macroevolutionary history of Phanerozoic life
Many aspects of macroevolutionary theory and our understanding of biotic responses to global environmental change derive from literature-based compilations of palaeontological data.
DimmWitted: A Study of Main-Memory Statistical Analytics
We perform the first study of the tradeoff space of access methods and replication to support statistical analytics using first-order methods executed in the main memory of a Non-Uniform Memory Access (NUMA) machine.
