Search Results for author:
Found 35 papers, 13 papers with code
Causal Layering via Conditional Entropy
Under appropriate assumptions and conditioning, we can separate the sources or sinks from the remainder of the nodes by comparing their conditional entropy to the unconditional entropy of their noise.
Editing Arbitrary Propositions in LLMs without Subject Labels
On datasets of binary propositions derived from the CounterFact dataset, we show that our method -- without access to subject labels -- performs close to state-of-the-art L\&E methods which has access subject labels.
BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents
The massive successes of large language models (LLMs) encourage the emerging exploration of LLM-augmented Autonomous Agents (LAAs).
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
This demonstrates that using policy gradient optimization to improve language agents, for which we believe our work is one of the first, seems promising and can be applied to optimize other models in the agent architecture to enhance agent performances over time.
REX: Rapid Exploration and eXploitation for AI Agents
In this paper, we propose an enhanced approach for Rapid Exploration and eXploitation for AI Agents called REX.
On the Unlikelihood of D-Separation
We then provide an analytic average case analysis of the PC Algorithm for causal discovery, as well as a variant of the SGS Algorithm we call UniformSGS.
Salesforce CausalAI Library: A Fast and Scalable Framework for Causal Analysis of Time Series and Tabular Data
Finally, we provide a user interface (UI) that allows users to perform causal analysis on data without coding.
Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
We first show that this chaotic behavior exists even along the training optimization trajectory of a single model, and propose a simple model averaging protocol that both significantly boosts domain generalization and diminishes the impact of stochasticity by improving the rank correlation between the in-domain validation accuracy and out-domain test accuracy, which is crucial for reliable early stopping.
Ranked #5 on
Domain Generalization
on TerraIncognita
Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE
Wasserstein autoencoder (WAE) shows that matching two distributions is equivalent to minimizing a simple autoencoder (AE) loss under the constraint that the latent space of this AE matches a pre-specified prior distribution.
Learning Rich Nearest Neighbor Representations from Self-supervised Ensembles
Pretraining convolutional neural networks via self-supervision, and applying them in transfer learning, is an incredibly fast-growing field that is rapidly and iteratively improving performance across practically all image domains.
Merlion: A Machine Learning Library for Time Series
We introduce Merlion, an open-source machine learning library for time series.
Momentum Contrastive Autoencoder
Quantitatively, we show that our algorithm achieves a new state-of-the-art FID of 54. 36 on CIFAR-10, and performs competitively with existing models on CelebA in terms of FID score.
Neural Bayes: A Generic Parameterization Method for Unsupervised Learning
Disjoint Manifold Separation: Neural Bayes allows us to formulate an objective which can optimally label samples from disjoint manifolds present in the support of a continuous distribution.
Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization
The early phase of training a deep neural network has a dramatic effect on the local curvature of the loss function.
The Break-Even Point on Optimization Trajectories of Deep Neural Networks
We argue for the existence of the "break-even" point on this trajectory, beyond which the curvature of the loss surface and noise in the gradient are implicitly regularized by SGD.
Neural Bayes: A Generic Parameterization Method for Unsupervised Representation Learning
Disjoint Manifold Labeling: Neural Bayes allows us to formulate an objective which can optimally label samples from disjoint manifolds present in the support of a continuous distribution.
Predicting with High Correlation Features
In this paper, we consider distribution shift as a shift in the distribution of input features during test time that exhibit low correlation with targets in the training set.
Entropy Penalty: Towards Generalization Beyond the IID Assumption
This allows deep networks trained with Entropy Penalty to generalize well even under distribution shift of spurious features.
How to Initialize your Network? Robust Initialization for WeightNorm & ResNets
Finally, we show that using our initialization in conjunction with learning rate warmup is able to reduce the gap between the performance of weight normalized and batch normalized networks.
A Walk with SGD: How SGD Explores Regions of Deep Network Loss?
The non-convex nature of the loss landscape of deep neural networks (DNN) lends them the intuition that over the course of training, stochastic optimization algorithms explore different regions of the loss surface by entering and escaping many local minima due to the noise induced by mini-batches.
The Benefits of Over-parameterization at Initialization in Deep ReLU Networks
These results are derived using the PAC analysis framework, and hold true for finitely sized datasets such that the width of the ReLU network only needs to be larger than a certain finite lower bound.
h-detach: Modifying the LSTM Gradient Towards Better Optimization
This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately over a large number of steps.
On the Spectral Bias of Neural Networks
Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with $100\%$ accuracy.
A Walk with SGD
Based on this and other metrics, we deduce that for most of the training update steps, SGD moves in valley like regions of the loss surface by jumping from one valley wall to another at a height above the valley floor.
Variational Bi-LSTMs
Bidirectional LSTMs (Bi-LSTMs) on the other hand model sequences along both forward and backward directions and are generally known to perform better at such tasks because they capture a richer representation of the data.
Three Factors Influencing Minima in SGD
In particular we find that the ratio of learning rate to batch size is a key determinant of SGD dynamics and of the width of the final minima, and that higher values of the ratio lead to wider minima and often better generalization.
Fraternal Dropout
We show that our regularization term is upper bounded by the expectation-linear dropout objective which has been shown to address the gap due to the difference between the train and inference phases of dropout.
Ranked #28 on
Language Modelling
on Penn Treebank (Word Level)
Residual Connections Encourage Iterative Inference
In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features.
A Closer Look at Memorization in Deep Networks
We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness.
On Optimality Conditions for Auto-Encoder Signal Recovery
Auto-Encoders are unsupervised models that aim to learn patterns from observed data by minimizing a reconstruction cost.
Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks.
Why Regularized Auto-Encoders learn Sparse Representation?
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- \textit{Internal Covariate Shift}-- the current solution has certain drawbacks.
Dimensionality Reduction with Subspace Structure Preservation
Modeling data as being sampled from a union of independent subspaces has been widely applied to a number of real world applications.
Is Joint Training Better for Deep Auto-Encoders?
But due to the greedy scheme of the layerwise training technique, the parameters of lower layers are fixed when training higher layers.
An Analysis of Random Projections in Cancelable Biometrics
With increasing concerns about security, the need for highly secure physical biometrics-based authentication systems utilizing \emph{cancelable biometric} technologies is on the rise.
