Search Results for author:
Found 26 papers, 8 papers with code
Centered Self-Attention Layers
The self-attention mechanism in transformers and the message-passing mechanism in graph neural networks are repeatedly applied within deep learning architectures.
Reverse Engineering Self-Supervised Learning
Self-supervised learning (SSL) is a powerful tool in machine learning, but understanding the learned representations and their underlying mechanisms remains a challenge.
The Probabilistic Stability of Stochastic Gradient Descent
Characterizing and understanding the stability of Stochastic Gradient Descent (SGD) remains an open problem in deep learning.
Norm-based Generalization Bounds for Compositionally Sparse Neural Networks
In this paper, we investigate the Rademacher complexity of deep sparse neural networks, where each neuron receives a small number of inputs.
Exploring the Approximation Capabilities of Multiplicative Neural Networks for Smooth Functions
Multiplication layers are a key component in various influential neural network modules, including self-attention and hypernetwork layers.
Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifiers
We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes.
Characterizing the Implicit Bias of Regularized SGD in Rank Minimization
We study the bias of Stochastic Gradient Descent (SGD) to learn low-rank weight matrices when training deep neural networks.
On the Implicit Bias Towards Minimal Depth of Deep Neural Networks
Finally, we empirically show that the effective depth of a trained neural network monotonically increases when increasing the number of random labels in data.
On the Role of Neural Collapse in Transfer Learning
We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes.
Meta Internal Learning
Internal learning for single-image generation is a framework, where a generator is trained to produce novel images based on a single image.
Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models
We discover that we can indeed use the same architecture and pretrained weights of a neural net model to understand both images and point-clouds.
Weakly Supervised Recovery of Semantic Attributes
We consider the problem of the extraction of semantic attributes, supervised only with classification labels.
Evaluation Metrics for Conditional Image Generation
We present two new metrics for evaluating generative models in the class-conditional image generation setting.
On Infinite-Width Hypernetworks
{\em Hypernetworks} are architectures that produce the weights of a task-specific {\em primary network}.
A Critical View of the Structural Causal Model
In the multivariate case, where one can ensure that the complexities of the cause and effect are balanced, we propose a new adversarial training method that mimics the disentangled structure of the causal model.
On the Modularity of Hypernetworks
Besides, we show that for a structured target function, the overall number of trainable parameters in a hypernetwork is smaller by orders of magnitude than the number of trainable parameters of a standard neural network and an embedding method.
On Random Kernels of Residual Architectures
We derive finite width and depth corrections for the Neural Tangent Kernel (NTK) of ResNets and DenseNets.
A Formal Approach to Explainability
We regard explanations as a blending of the input sample and the model's output and offer a few definitions that capture various desired properties of the function that generates these explanations.
Emerging Disentanglement in Auto-Encoder Based Unsupervised Image Content Transfer
Thus, in the above example, we can create, for every person without glasses a version with the glasses observed in any face image.
Unsupervised Learning of the Set of Local Maxima
Two functions are learned: (i) a set indicator c, which is a binary classifier, and (ii) a comparator function h that given two nearby samples, predicts which sample has the higher value of the unknown function v. Loss terms are used to ensure that all training samples x are a local maxima of v, according to h and satisfy c(x)=1.
Variable Complexity in the Univariate and Multivariate Structural Causal Model
Comparing the reconstruction errors of the two autoencoders, one for each variable, is shown to perform well on the accepted benchmarks of the field.
Domain Intersection and Domain Difference
We present a method for recovering the shared content between two visual domains as well as the content that is unique to each domain.
Risk Bounds for Unsupervised Cross-Domain Mapping with IPMs
The recent empirical success of unsupervised cross-domain mapping algorithms, between two domains that share common characteristics, is not well-supported by theoretical justifications.
Estimating the Success of Unsupervised Image to Image Translation
While in supervised learning, the validation error is an unbiased estimator of the generalization (test) error and complexity-based generalization bounds are abundant, no such bounds exist for learning a mapping in an unsupervised way.
The Role of Minimal Complexity Functions in Unsupervised Learning of Semantic Mappings
We discuss the feasibility of the following learning problem: given unmatched samples from two domains and nothing else, learn a mapping between the two, which preserves semantics.
A Theory of Output-Side Unsupervised Domain Adaptation
We consider the complementary problem in which the unlabeled samples are given post mapping, i. e., we are given the outputs of the mapping of unknown samples from the shifted domain.
