Search Results for author:
Found 64 papers, 16 papers with code
Incremental Few-Shot Meta-Learning via Indirect Discriminant Alignment
This process enables incrementally improving the model by processing multiple learning episodes, each representing a different learning task, even with few training examples.
Long-context Protein Language Model
Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design.
Compositional Structures in Neural Embedding and Interaction Decompositions
Specifically, we introduce a characterization of compositional structures in terms of "interaction decompositions," and we establish necessary and sufficient conditions for the presence of such structures within the representations of a model.
B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory
Recent hybrid architectures have combined eidetic and fading memory, but with limitations that do not allow the designer or the learning process to seamlessly modulate the two, nor to extend the eidetic memory span.
Diffusion Soup: Model Merging for Text-to-Image Diffusion Models
We present Diffusion Soup, a compartmentalization method for Text-to-Image Generation that averages the weights of diffusion models trained on sharded data.
Principles of Designing Robust Remote Face Anti-Spoofing Systems
Protecting digital identities of human face from various attack vectors is paramount, and face anti-spoofing plays a crucial role in this endeavor.
NeRF-Insert: 3D Local Editing with Multimodal Control Signals
We propose NeRF-Insert, a NeRF editing framework that allows users to make high-quality local edits with a flexible level of control.
CPR: Retrieval Augmented Generation for Copyright Protection
To reduce risks of leaking private information contained in the retrieved set, we introduce Copy-Protected generation with Retrieval (CPR), a new method for RAG with strong copyright protection guarantees in a mixed-private setting for diffusion models. CPR allows to condition the output of diffusion models on a set of retrieved images, while also guaranteeing that unique identifiable information about those example is not exposed in the generated outputs.
Multi-Modal Hallucination Control by Visual Information Grounding
In particular, we show that as more tokens are generated, the reliance on the visual prompt decreases, and this behavior strongly correlates with the emergence of hallucinations.
Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding
Quantifying the degree of similarity between images is a key copyright issue for image-based machine learning.
Meaning Representations from Trajectories in Autoregressive Models
We propose to extract meaning representations from autoregressive language models by considering the distribution of all possible trajectories extending an input text.
Critical Learning Periods Emerge Even in Deep Linear Networks
Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations.
Training Data Protection with Compositional Diffusion Models
We introduce Compartmentalized Diffusion Models (CDM), a method to train different diffusion models (or prompts) on distinct data sources and arbitrarily compose them at inference time.
Towards Visual Foundational Models of Physical Scenes
We describe a first step towards learning general-purpose visual representations of physical scenes using only image prediction as a training criterion.
Prompt Algebra for Task Composition
We investigate whether prompts learned independently for different tasks can be later combined through prompt algebra to obtain a model that supports composition of tasks.
SAFE: Machine Unlearning With Shard Graphs
We present Synergy Aware Forgetting Ensemble (SAFE), a method to adapt large models on a diverse collection of data while minimizing the expected cost to remove the influence of training samples from the trained model.
Leveraging sparse and shared feature activations for disentangled representation learning
Recovering the latent factors of variation of high dimensional data has so far focused on simple synthetic settings.
AI Model Disgorgement: Methods and Choices
One potential fix for training corpus data defects is model disgorgement -- the elimination of not just the improperly used data, but also the effects of improperly used data on any component of an ML model.
Train/Test-Time Adaptation with Retrieval
Second, we apply ${\rm T^3AR}$ for test-time adaptation and show that exploiting a pool of external images at test-time leads to more robust representations over existing methods on DomainNet-126 and VISDA-C, especially when few adaptation data are available (up to 8%).
Spacetime-Efficient Low-Depth Quantum State Preparation with Applications
When our protocol is compiled into CNOT and arbitrary single-qubit gates, it prepares an $N$-dimensional state in depth $O(\log(N))$ and spacetime allocation (a metric that accounts for the fact that oftentimes some ancilla qubits need not be active for the entire circuit) $O(N)$, which are both optimal.
A Meta-Learning Approach to Predicting Performance and Data Requirements
The PPL improves the performance estimation on average by 37% across 16 classification and 33% across 10 detection datasets, compared to the power law.
Linear Spaces of Meanings: Compositional Structures in Vision-Language Models
These vectors can be seen as "ideal words" for generating concepts directly within the embedding space of the model.
À-la-carte Prompt Tuning (APT): Combining Distinct Data Via Composable Prompting
During inference, models can be assembled based on arbitrary selections of data sources, which we call "\`a-la-carte learning".
A-La-Carte Prompt Tuning (APT): Combining Distinct Data via Composable Prompting
During inference, models can be assembled based on arbitrary selections of data sources, which we call a-la-carte learning.
Integral Continual Learning Along the Tangent Vector Field of Tasks
We propose a lightweight continual learning method which incorporates information from specialized datasets incrementally, by integrating it along the vector field of "generalist" models.
Critical Learning Periods for Multisensory Integration in Deep Networks
We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training.
On the Learnability of Physical Concepts: Can a Neural Network Understand What's Real?
We revisit the classic signal-to-symbol barrier in light of the remarkable ability of deep neural networks to generate realistic synthetic data.
On Leave-One-Out Conditional Mutual Information For Generalization
We derive information theoretic generalization bounds for supervised learning algorithms based on a new measure of leave-one-out conditional mutual information (loo-CMI).
Gacs-Korner Common Information Variational Autoencoder
We propose a notion of common information that allows one to quantify and separate the information that is shared between two random variables from the information that is unique to each.
Task Adaptive Parameter Sharing for Multi-Task Learning
TAPS solves a joint optimization problem which determines which layers to share with the base model and the value of the task-specific weights.
Towards Differential Relational Privacy and its use in Question Answering
While bounding general memorization can have detrimental effects on the performance of a trained model, bounding RM does not prevent effective learning.
Mixed Differential Privacy in Computer Vision
AdaMix incorporates few-shot training, or cross-modal zero-shot learning, on public data prior to private fine-tuning, to improve the trade-off.
On Plasticity, Invariance, and Mutually Frozen Weights in Sequential Task Learning
In particular, we show that, when using weight decay, weights in successive layers of a deep network may become "mutually frozen".
DIVA: Dataset Derivative of a Learning Task
A learning task is a function from a training set to the validation error, which can be represented by a trained deep neural network (DNN).
STRIC: Stacked Residuals of Interpretable Components for Time Series Anomaly Detection
After modeling the signals, we use an anomaly detection system based on the classic CUMSUM algorithm and a variational approximation of the $f$-divergence to detect both isolated point anomalies and change-points in statistics of the signals.
Redundant Information Neural Estimation
We introduce the Redundant Information Neural Estimator (RINE), a method that allows efficient estimation for the component of information about a target variable that is common to a set of sources, previously referred to as the “redundant information.” We show that existing definitions of the redundant information can be recast in terms of an optimization over a family of deterministic or stochastic functions.
A linearized framework and a new benchmark for model selection for fine-tuning
Since all model selection algorithms in the literature have been tested on different use-cases and never compared directly, we introduce a new comprehensive benchmark for model selection comprising of: i) A model zoo of single and multi-domain models, and ii) Many target tasks.
Supervised Momentum Contrastive Learning for Few-Shot Classification
In this work we investigate the complementary roles of these two sources of information by combining instance-discriminative contrastive learning and supervised learning in a single framework called Supervised Momentum Contrastive learning (SUPMOCO).
Estimating informativeness of samples with Smooth Unique Information
We define a notion of information that an individual sample provides to the training of a neural network, and we specialize it to measure both how much a sample informs the final weights and how much it informs the function computed by the weights.
Structured Prediction as Translation between Augmented Natural Languages
We propose a new framework, Translation between Augmented Natural Languages (TANL), to solve many structured prediction language tasks including joint entity and relation extraction, nested named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, and dialogue state tracking.
Ranked #3 on
Relation Classification
on TACRED
Multimodal Attention for Layout Synthesis in Diverse Domains
We address the problem of scene layout generation for diverse domains such as images, mobile applications, documents and 3D objects.
Mixed-Privacy Forgetting in Deep Networks
We show that the influence of a subset of the training samples can be removed -- or "forgotten" -- from the weights of a network trained on large-scale image classification tasks, and we provide strong computable bounds on the amount of remaining information after forgetting.
LQF: Linear Quadratic Fine-Tuning
Classifiers that are linear in their parameters, and trained by optimizing a convex loss function, have predictable behavior with respect to changes in the training data, initial conditions, and optimization.
Usable Information and Evolution of Optimal Representations During Training
We introduce a notion of usable information contained in the representation learned by a deep network, and use it to study how optimal representations for the task emerge during training.
Predicting Training Time Without Training
We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function.
Adversarial Training Reduces Information and Improves Transferability
Recent results show that features of adversarially trained networks for classification, in addition to being robust, enable desirable properties such as invertibility.
LayoutTransformer: Layout Generation and Completion with Self-attention
Generating a new layout or extending an existing layout requires understanding the relationships between these primitives.
Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations
We describe a procedure for removing dependency on a cohort of training data from a trained deep network that improves upon and generalizes previous methods to different readout functions and can be extended to ensure forgetting in the activations of the network.
Incremental Meta-Learning via Indirect Discriminant Alignment
Majority of the modern meta-learning methods for few-shot classification tasks operate in two phases: a meta-training phase where the meta-learner learns a generic representation by solving multiple few-shot tasks sampled from a large dataset and a testing phase, where the meta-learner leverages its learnt internal representation for a specific few-shot task involving classes which were not seen during the meta-training phase.
TextTubes for Detecting Curved Text in the Wild
We present a detector for curved text in natural images.
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
We explore the problem of selectively forgetting a particular subset of the data used for training a deep neural network.
Where is the Information in a Deep Network?
We relate this to the Information in the Weights, and use this result to show that models of low (information) complexity not only generalize better, but are bound to learn invariant representations of future inputs.
Toward Understanding Catastrophic Forgetting in Continual Learning
As an application, we apply our procedure to study two properties of a task sequence: (1) total complexity and (2) sequential heterogeneity.
Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
Deep neural networks (DNNs), however, challenge this view: We show that removing regularization after an initial transient period has little effect on generalization, even if the final loss landscape is the same as if there had been no regularization.
Where is the Information in a Deep Neural Network?
We establish a novel relation between the information in the weights and the effective information in the activations, and use this result to show that models with low (information) complexity not only generalize better, but are bound to learn invariant representations of future inputs.
Critical Learning Periods in Deep Networks
Deficits that do not affect low-level statistics, such as vertical flipping of the images, have no lasting effect on performance and can be overcome with further training.
The Information Complexity of Learning Tasks, their Structure and their Distance
Our framework is the first to measure complexity in a way that accounts for the effect of the optimization scheme, which is critical in Deep Learning.
Task2Vec: Task Embedding for Meta-Learning
We demonstrate that this embedding is capable of predicting task similarities that match our intuition about semantic and taxonomic relations between different visual tasks (e. g., tasks based on classifying different types of plants are similar) We also demonstrate the practical value of this framework for the meta-task of selecting a pre-trained feature extractor for a new task.
Dynamics and Reachability of Learning Tasks
We compute the transition probability between two learning tasks, and show that it decomposes into two factors.
Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies
Intelligent behaviour in the real-world requires the ability to acquire new knowledge from an ongoing sequence of experiences while preserving and reusing past knowledge.
Critical Learning Periods in Deep Neural Networks
Deficits that do not affect low-level statistics, such as vertical flipping of the images, have no lasting effect on performance and can be overcome with further training.
A Separation Principle for Control in the Age of Deep Learning
Again this can be finitely-parametrized using a deep neural network, and already some applications are beginning to emerge.
Emergence of Invariance and Disentanglement in Deep Representations
Using established principles from Statistics and Information Theory, we show that invariance to nuisance factors in a deep neural network is equivalent to information minimality of the learned representation, and that stacking layers and injecting noise during training naturally bias the network towards learning invariant representations.
Information Dropout: Learning Optimal Representations Through Noisy Computation
The cross-entropy loss commonly used in deep learning is closely related to the defining properties of optimal representations, but does not enforce some of the key properties.
