Search Results for author:
Found 21 papers, 5 papers with code
MagicPIG: LSH Sampling for Efficient LLM Generation
Large language models (LLMs) with long context windows have gained significant attention.
Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need
Third, it provides a proper active learning framework yielding low-cost solutions to annotate datasets, arguably bringing the gap between theory and practice of active learning that is based on simple-to-answer-by-non-experts queries of semantic relationships between inputs.
The Effects of Regularization and Data Augmentation are Class Dependent
The optimal amount of DA or weight decay found from cross-validation leads to disastrous model performances on some classes e. g. on Imagenet with a resnet50, the "barn spider" classification test accuracy falls from $68\%$ to $46\%$ only by introducing random crop DA during training.
Match Prediction Using Learned History Embeddings
Contemporary ranking systems that are based on win/loss history, such as Elo or TrueSkill represent each player using a scalar estimate of ability (plus variance, in the latter case).
Pseudo-Euclidean Attract-Repel Embeddings for Undirected Graphs
Dot product embeddings take a graph and construct vectors for nodes such that dot products between two vectors give the strength of the edge.
Linear unit-tests for invariance discovery
There is an increasing interest in algorithms to learn invariant correlations across training environments.
Scaling Laws for the Principled Design, Initialization, and Preconditioning of ReLU Networks
Abstract In this work, we describe a set of rules for the design and initialization of well-conditioned neural networks, guided by the goal of naturally balancing the diagonal blocks of the Hessian at the start of training.
Mean Replacement Pruning
Finally, we show that the units selected by the best performing scoring functions are somewhat consistent over the course of training, implying the dead parts of the network appear during the stages of training.
AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such methods have gained widespread use in large-scale optimization for their ability to converge robustly, without the need to fine-tune the stepsize schedule.
Geometrical Insights for Implicit Generative Modeling
Learning algorithms for implicit generative models can optimize a variety of criteria that measure how the data distribution differs from the implicit model distribution, including the Wasserstein distance, the Energy distance, and the Maximum Mean Discrepancy criterion.
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.
Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
We look at the eigenvalues of the Hessian of a loss function before and after training.
Is Object Localization for Free? - Weakly-Supervised Learning With Convolutional Neural Networks
Successful visual object recognition methods typically rely on training datasets containing lots of richly annotated images.
A Lower Bound for the Optimization of Finite Sums
This paper presents a lower bound for optimizing a finite sum of $n$ functions, where each function is $L$-smooth and the sum is $\mu$-strongly convex.
ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
Based on the machine's output, the teacher can revise the definition of the task or make it more precise.
Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks
We show that despite differences in image statistics and tasks in the two datasets, the transferred representation leads to significantly improved results for object and action classification, outperforming the current state of the art on Pascal VOC 2007 and 2012 datasets.
An efficient distributed learning algorithm based on effective local functional approximations
In this paper we give a novel approach to the distributed training of linear classifiers (involving smooth losses and L2 regularization) that is designed to reduce the total communication costs.
Para-active learning
We leverage the same observation to build a generic strategy for parallelizing learning algorithms.
Natural Language Processing (almost) from Scratch
We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling.
From Machine Learning to Machine Reasoning
This observation suggests a conceptual continuity between algebraically rich inference systems, such as logical or probabilistic inference, and simple manipulations, such as the mere concatenation of trainable learning systems.
