Search Results for author:
Found 195 papers, 56 papers with code
Incremental Few-Shot Meta-Learning via Indirect Discriminant Alignment
This process enables incrementally improving the model by processing multiple learning episodes, each representing a different learning task, even with few training examples.
PICASO: Permutation-Invariant Context Composition with State Space Models
A key challenge arises when attempting to leverage information present across multiple contexts, since there is no straightforward way to condition generation on multiple independent states in existing SSMs.
Descriminative-Generative Custom Tokens for Vision-Language Models
We operate on CLIP text features and propose to use a combination of a textual inversion loss and a classification loss to ensure that text features of the learned token are aligned with image features of the concept in the CLIP embedding space.
Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models
We describe a method to expand the memory span of the hybrid state by "reserving" a fraction of the Attention context for tokens retrieved from arbitrarily distant in the past, thus expanding the eidetic memory span of the overall state.
Efficient Scaling of Diffusion Transformers for Text-to-Image Generation
We empirically study the scaling properties of various Diffusion Transformers (DiTs) for text-to-image generation by performing extensive and rigorous ablations, including training scaled DiTs ranging from 0. 3B upto 8B parameters on datasets up to 600M images.
The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation
Speculative decoding aims to speed up autoregressive generation of a language model by verifying in parallel the tokens generated by a smaller draft model. In this work, we explore the effectiveness of learning-free, negligible-cost draft strategies, namely $N$-grams obtained from the model weights and the context.
Conjuring Semantic Similarity
Therefore, we characterize the semantic similarity between two textual expressions simply as the distance between image distributions they induce, or 'conjure.'
DocKD: Knowledge Distillation from LLMs for Open-World Document Understanding Models
Visual document understanding (VDU) is a challenging task that involves understanding documents across various modalities (text and image) and layouts (forms, tables, etc.).
RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions
Our method, RSA, takes as input a text caption describing objects present in an image and outputs the parameters of a linear transformation which can be applied globally to a relative depth map to yield metric-scaled depth predictions.
NAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionality
The benchmark is constructed by generating negative texts with incorrect action descriptions for a given video and the model is expected to pair a positive text with its corresponding video.
Compositional Structures in Neural Embedding and Interaction Decompositions
Specifically, we introduce a characterization of compositional structures in terms of "interaction decompositions," and we establish necessary and sufficient conditions for the presence of such structures within the representations of a model.
B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory
Recent hybrid architectures have combined eidetic and fading memory, but with limitations that do not allow the designer or the learning process to seamlessly modulate the two, nor to extend the eidetic memory span.
Diffusion Soup: Model Merging for Text-to-Image Diffusion Models
We present Diffusion Soup, a compartmentalization method for Text-to-Image Generation that averages the weights of diffusion models trained on sharded data.
Cycles of Thought: Measuring LLM Confidence through Stable Explanations
We instead propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
Meanings and Feelings of Large Language Models: Observability of Latent States in Generative AI
But if there are 'system prompts' not visible to the user, then the set of indistinguishable trajectories becomes non-trivial, and there can be multiple state trajectories that yield the same verbalized output.
THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models
By evaluating a large selection of recent LVLMs using public datasets, we show that an improvement in existing metrics do not lead to a reduction in Type I hallucinations, and that established benchmarks for measuring Type I hallucinations are incomplete.
Diffeomorphic Template Registration for Atmospheric Turbulence Mitigation
Rather than initializing a latent irradiance ("template") by heuristics to estimate deformation, we select one of the images as a reference, and model the deformation in this image by the aggregation of the optical flow from it to other images, exploiting a prior imposed by Central Limit Theorem.
NeRF-Insert: 3D Local Editing with Multimodal Control Signals
We propose NeRF-Insert, a NeRF editing framework that allows users to make high-quality local edits with a flexible level of control.
Grounded Compositional and Diverse Text-to-3D with Pretrained Multi-View Diffusion Model
In this paper, we propose an effective two-stage approach named Grounded-Dreamer to generate 3D assets that can accurately follow complex, compositional text prompts while achieving high fidelity by using a pre-trained multi-view diffusion model.
Fewer Truncations Improve Language Modeling
In large language model training, input documents are typically concatenated together and then split into sequences of equal length to avoid padding tokens.
Mixed-Query Transformer: A Unified Image Segmentation Architecture
Existing unified image segmentation models either employ a unified architecture across multiple tasks but use separate weights tailored to each dataset, or apply a single set of weights to multiple datasets but are limited to a single task.
WorDepth: Variational Language Prior for Monocular Depth Estimation
To test this, we focus on monocular depth estimation, the problem of predicting a dense depth map from a single image, but with an additional text caption describing the scene.
Heat Death of Generative Models in Closed-Loop Learning
The aim of this paper is to provide insights into this process (that we refer to as "generative closed-loop learning") by studying the learning dynamics of generative models that are fed back their own produced content in addition to their original training dataset.
CPR: Retrieval Augmented Generation for Copyright Protection
To reduce risks of leaking private information contained in the retrieved set, we introduce Copy-Protected generation with Retrieval (CPR), a new method for RAG with strong copyright protection guarantees in a mixed-private setting for diffusion models. CPR allows to condition the output of diffusion models on a set of retrieved images, while also guaranteeing that unique identifiable information about those example is not exposed in the generated outputs.
Multi-Modal Hallucination Control by Visual Information Grounding
In particular, we show that as more tokens are generated, the reliance on the visual prompt decreases, and this behavior strongly correlates with the emergence of hallucinations.
Fast Sparse View Guided NeRF Update for Object Reconfigurations
Neural Radiance Field (NeRF), as an implicit 3D scene representation, lacks inherent ability to accommodate changes made to the initial static scene.
Enhancing Vision-Language Pre-training with Rich Supervisions
We propose Strongly Supervised pre-training with ScreenShots (S4) - a novel pre-training paradigm for Vision-Language Models using data from large-scale web screenshot rendering.
Non-autoregressive Sequence-to-Sequence Vision-Language Models
Sequence-to-sequence vision-language models are showing promise, but their applicability is limited by their inference latency due to their autoregressive way of generating predictions.
A Quantitative Evaluation of Score Distillation Sampling Based Text-to-3D
However, the SDS method is also the source of several artifacts, such as the Janus problem, the misalignment between the text prompt and the generated 3D model, and 3D model inaccuracies.
Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding
Quantifying the degree of similarity between images is a key copyright issue for image-based machine learning.
Meaning Representations from Trajectories in Autoregressive Models
We propose to extract meaning representations from autoregressive language models by considering the distribution of all possible trajectories extending an input text.
AugUndo: Scaling Up Augmentations for Monocular Depth Completion and Estimation
The sparse depth modality in depth completion have seen even less use as intensity transformations alter the scale of the 3D scene, and geometric transformations may decimate the sparse points during resampling.
Sub-token ViT Embedding via Stochastic Resonance Transformers
Vision Transformer (ViT) architectures represent images as collections of high-dimensional vectorized tokens, each corresponding to a rectangular non-overlapping patch.
Critical Learning Periods Emerge Even in Deep Linear Networks
Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations.
Training Data Protection with Compositional Diffusion Models
We introduce Compartmentalized Diffusion Models (CDM), a method to train different diffusion models (or prompts) on distinct data sources and arbitrarily compose them at inference time.
Tangent Model Composition for Ensembling and Continual Fine-tuning
Component models are composed at inference time via scalar combination, reducing the cost of ensembling to that of a single model.
Tangent Transformers for Composition, Privacy and Removal
We introduce Tangent Attention Fine-Tuning (TAFT), a method for fine-tuning linearized transformers obtained by computing a First-order Taylor Expansion around a pre-trained initialization.
Towards Visual Foundational Models of Physical Scenes
We describe a first step towards learning general-purpose visual representations of physical scenes using only image prediction as a training criterion.
Prompt Algebra for Task Composition
We investigate whether prompts learned independently for different tasks can be later combined through prompt algebra to obtain a model that supports composition of tasks.
Taming AI Bots: Controllability of Neural States in Large Language Models
We then characterize the subset of meanings that can be reached by the state of the LLMs for some input prompt, and show that a well-trained bot can reach any meaning albeit with small probability.
Learning for Transductive Threshold Calibration in Open-World Recognition
In deep metric learning for visual recognition, the calibration of distance thresholds is crucial for achieving desired model performance in the true positive rates (TPR) or true negative rates (TNR).
Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts
We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks which may interfere with each other, resulting in a single model which we named Musketeer.
SAFE: Machine Unlearning With Shard Graphs
We present Synergy Aware Forgetting Ensemble (SAFE), a method to adapt large models on a diverse collection of data while minimizing the expected cost to remove the influence of training samples from the trained model.
Leveraging sparse and shared feature activations for disentangled representation learning
Recovering the latent factors of variation of high dimensional data has so far focused on simple synthetic settings.
AI Model Disgorgement: Methods and Choices
One potential fix for training corpus data defects is model disgorgement -- the elimination of not just the improperly used data, but also the effects of improperly used data on any component of an ML model.
Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots
We introduce a method to segment the visual field into independently moving regions, trained with no ground truth or supervision.
Train/Test-Time Adaptation with Retrieval
Second, we apply ${\rm T^3AR}$ for test-time adaptation and show that exploiting a pool of external images at test-time leads to more robust representations over existing methods on DomainNet-126 and VISDA-C, especially when few adaptation data are available (up to 8%).
Feature Tracks are not Zero-Mean Gaussian
In state estimation algorithms that use feature tracks as input, it is customary to assume that the errors in feature track positions are zero-mean Gaussian.
A Meta-Learning Approach to Predicting Performance and Data Requirements
The PPL improves the performance estimation on average by 37% across 16 classification and 33% across 10 detection datasets, compared to the power law.
Linear Spaces of Meanings: Compositional Structures in Vision-Language Models
These vectors can be seen as "ideal words" for generating concepts directly within the embedding space of the model.
À-la-carte Prompt Tuning (APT): Combining Distinct Data Via Composable Prompting
During inference, models can be assembled based on arbitrary selections of data sources, which we call "\`a-la-carte learning".
Guided Recommendation for Model Fine-Tuning
With thousands of historical training jobs, a recommendation system can be learned to predict the model selection score given the features of the dataset and the model as input.
A-La-Carte Prompt Tuning (APT): Combining Distinct Data via Composable Prompting
During inference, models can be assembled based on arbitrary selections of data sources, which we call a-la-carte learning.
Depth Estimation From Camera Image and mmWave Radar Point Cloud
To fuse radar depth with an image, we propose a gated fusion scheme that accounts for the confidence scores of the correspondence so that we selectively combine radar and camera embeddings to yield a dense depth map.
Integral Continual Learning Along the Tangent Vector Field of Tasks
We propose a lightweight continual learning method which incorporates information from specialized datasets incrementally, by integrating it along the vector field of "generalist" models.
Stain-invariant self supervised learning for histopathology image analysis
Our method achieves the state-of-the-art performance on several publicly available breast cancer datasets ranging from tumor classification (CAMELYON17) and subtyping (BRACS) to HER2 status classification and treatment response prediction.
Critical Learning Periods for Multisensory Integration in Deep Networks
We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training.
Semi-supervised Vision Transformers at Scale
We study semi-supervised learning (SSL) for vision transformers (ViT), an under-explored topic despite the wide adoption of the ViT architectures to different tasks.
Masked Vision and Language Modeling for Multi-modal Representation Learning
Instead of developing masked language modeling (MLM) and masked image modeling (MIM) independently, we propose to build joint masked vision and language modeling, where the masked signal of one modality is reconstructed with the help from another modality.
On the Learnability of Physical Concepts: Can a Neural Network Understand What's Real?
We revisit the classic signal-to-symbol barrier in light of the remarkable ability of deep neural networks to generate realistic synthetic data.
On Leave-One-Out Conditional Mutual Information For Generalization
We derive information theoretic generalization bounds for supervised learning algorithms based on a new measure of leave-one-out conditional mutual information (loo-CMI).
Not Just Streaks: Towards Ground Truth for Single Image Deraining
We propose a large-scale dataset of real-world rainy and clean image pairs and a method to remove degradations, induced by rain streaks and rain accumulation, from the image.
Gacs-Korner Common Information Variational Autoencoder
We propose a notion of common information that allows one to quantify and separate the information that is shared between two random variables from the information that is unique to each.
ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training
Based on the observation, we present a method, called Ensemble Logit Difference Inhibition (ELODI), to train a classification system that achieves paragon performance in both error rate and NFR, at the inference cost of a single model.
Graph Spectral Embedding using the Geodesic Betweeness Centrality
We introduce the Graph Sylvester Embedding (GSE), an unsupervised graph representation of local similarity, connectivity, and global structure.
X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks
In this paper, we study the challenging instance-wise vision-language tasks, where the free-form language is required to align with the objects instead of the whole image.
Class-Incremental Learning with Strong Pre-trained Models
We hypothesize that a strong base model can provide a good representation for novel classes and incremental learning can be done with small adaptations.
MeMOT: Multi-Object Tracking with Memory
We propose an online tracking algorithm that performs the object detection and data association under a common framework, capable of linking objects after a long time span.
Towards Differential Relational Privacy and its use in Question Answering
While bounding general memorization can have detrimental effects on the performance of a trained model, bounding RM does not prevent effective learning.
Task Adaptive Parameter Sharing for Multi-Task Learning
TAPS solves a joint optimization problem which determines which layers to share with the base model and the value of the task-specific weights.
Omni-DETR: Omni-Supervised Object Detection with Transformers
This is enabled by a unified architecture, Omni-DETR, based on the recent progress on student-teacher framework and end-to-end transformer based object detection.
Ranked #14 on
Semi-Supervised Object Detection
on COCO 2% labeled data
On the Viability of Monocular Depth Pre-training for Semantic Segmentation
We choose monocular depth prediction as the geometric task, and semantic segmentation as the downstream semantic task, and design a collection of empirical tests by exploring different forms of supervision, training pipelines, and data sources for both depth pre-training and semantic fine-tuning.
Mixed Differential Privacy in Computer Vision
AdaMix incorporates few-shot training, or cross-modal zero-shot learning, on public data prior to private fine-tuning, to improve the trade-off.
Contrastive Neighborhood Alignment
We present Contrastive Neighborhood Alignment (CNA), a manifold learning approach to maintain the topology of learned features whereby data points that are mapped to nearby representations by the source (teacher) model are also mapped to neighbors by the target (student) model.
Stereoscopic Universal Perturbations across Different Architectures and Datasets
We study the effect of adversarial perturbations of images on deep stereo matching networks for the disparity estimation task.
DIVA: Dataset Derivative of a Learning Task
A learning task is a function from a training set to the validation error, which can be represented by a trained deep neural network (DNN).
Representation Consolidation from Multiple Expert Teachers
Indeed, we observe experimentally that standard distillation of task-specific teachers, or using these teacher representations directly, **reduces** downstream transferability compared to a task-agnostic generalist model.
STRIC: Stacked Residuals of Interpretable Components for Time Series Anomaly Detection
After modeling the signals, we use an anomaly detection system based on the classic CUMSUM algorithm and a variational approximation of the $f$-divergence to detect both isolated point anomalies and change-points in statistics of the signals.
Small Lesion Segmentation in Brain MRIs with Subpixel Embedding
We propose a neural network architecture in the form of a standard encoder-decoder where predictions are guided by a spatial expansion embedding network.
Unsupervised Depth Completion with Calibrated Backprojection Layers
At inference time, the calibration of the camera, which can be different than the one used for training, is fed as an input to the network along with the sparse point cloud and a single image.
Ranked #2 on
Depth Completion
on VOID
ARCH++: Animation-Ready Clothed Human Reconstruction Revisited
We present ARCH++, an image-based method to reconstruct 3D avatars with arbitrary clothing styles.
Ranked #1 on
3D Object Reconstruction From A Single Image
on RenderPeople
(using extra training data)
3D Object Reconstruction From A Single Image
Image-to-Image Translation
SABER: Data-Driven Motion Planner for Autonomously Navigating Heterogeneous Robots
We present an end-to-end online motion planning framework that uses a data-driven approach to navigate a heterogeneous robot team towards a global goal while avoiding obstacles in uncertain environments.
Uniform Sampling over Episode Difficulty
Episodic training is a core ingredient of few-shot learning to train models on tasks with limited labelled data.
Representation Consolidation for Training Expert Students
Traditionally, distillation has been used to train a student model to emulate the input/output functionality of a teacher.
Long Short-Term Transformer for Online Action Detection
We present Long Short-term TRansformer (LSTR), a temporal modeling algorithm for online action detection, which employs a long- and short-term memory mechanism to model prolonged sequence data.
Ranked #3 on
Online Action Detection
on TVSeries
Learning Hierarchical Graph Neural Networks for Image Clustering
Our hierarchical GNN uses a novel approach to merge connected components predicted at each level of the hierarchy to form a new graph at the next level.
Scene Uncertainty and the Wellington Posterior of Deterministic Image Classifiers
We propose a method to estimate the uncertainty of the outcome of an image classifier on a given input datum.
Dynamically Grown Generative Adversarial Networks
Recent work introduced progressive network growing as a promising way to ease the training for large GANs, but the model design and architecture-growing strategy still remain under-explored and needs manual design for different image data.
Harnessing Unrecognizable Faces for Improving Face Recognition
The common implementation of face recognition systems as a cascade of a detection stage and a recognition or verification stage can cause problems beyond failures of the detector.
Learning Topology from Synthetic Data for Unsupervised Depth Completion
We present a method for inferring dense depth maps from images and sparse depth measurements by leveraging synthetic data to learn the association of sparse point clouds with dense natural shapes, and using the image as evidence to validate the predicted depth map.
Ranked #3 on
Depth Completion
on VOID
An Adaptive Framework for Learning Unsupervised Depth Completion
We present a method to infer a dense depth map from a color image and associated sparse depth measurements.
Compatibility-aware Heterogeneous Visual Search
Existing systems use the same embedding model to compute representations (embeddings) for the query and gallery images.
Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates
Behavior of deep neural networks can be inconsistent between different versions.
Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queries
Computer vision applications such as visual relationship detection and human object interaction can be formulated as a composite (structured) set detection problem in which both the parts (subject, object, and predicate) and the sum (triplet as a whole) are to be detected in a hierarchical fashion.
Learning Semantic-Aware Dynamics for Video Prediction
The appearance of the scene is warped from past frames using the predicted motion in co-visible regions; dis-occluded regions are synthesized with content-aware inpainting utilizing the predicted scene layout.
Redundant Information Neural Estimation
We introduce the Redundant Information Neural Estimator (RINE), a method that allows efficient estimation for the component of information about a target variable that is common to a set of sources, previously referred to as the “redundant information.” We show that existing definitions of the redundant information can be recast in terms of an optimization over a family of deterministic or stochastic functions.
A linearized framework and a new benchmark for model selection for fine-tuning
Since all model selection algorithms in the literature have been tested on different use-cases and never compared directly, we introduce a new comprehensive benchmark for model selection comprising of: i) A model zoo of single and multi-domain models, and ii) Many target tasks.
Supervised Momentum Contrastive Learning for Few-Shot Classification
In this work we investigate the complementary roles of these two sources of information by combining instance-discriminative contrastive learning and supervised learning in a single framework called Supervised Momentum Contrastive learning (SUPMOCO).
Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
We present a plug-in replacement for batch normalization (BN) called exponential moving average normalization (EMAN), which improves the performance of existing student-teacher based self- and semi-supervised learning techniques.
Self-Supervised Learning
Semi-Supervised Image Classification
Estimating informativeness of samples with Smooth Unique Information
We define a notion of information that an individual sample provides to the training of a neural network, and we specialize it to measure both how much a sample informs the final weights and how much it informs the function computed by the weights.
Structured Prediction as Translation between Augmented Natural Languages
We propose a new framework, Translation between Augmented Natural Languages (TANL), to solve many structured prediction language tasks including joint entity and relation extraction, nested named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, and dialogue state tracking.
Ranked #3 on
Relation Classification
on TACRED
Mixed-Privacy Forgetting in Deep Networks
We show that the influence of a subset of the training samples can be removed -- or "forgotten" -- from the weights of a network trained on large-scale image classification tasks, and we provide strong computable bounds on the amount of remaining information after forgetting.
LQF: Linear Quadratic Fine-Tuning
Classifiers that are linear in their parameters, and trained by optimizing a convex loss function, have predictable behavior with respect to changes in the training data, initial conditions, and optimization.
Positive-Congruent Training: Towards Regression-Free Model Updates
Reducing inconsistencies in the behavior of different versions of an AI system can be as important in practice as reducing its overall error.
Spectral Embedding of Graph Networks
We introduce an unsupervised graph embedding that trades off local node similarity and connectivity, and global structure.
Stereopagnosia: Fooling Stereo Networks with Adversarial Perturbations
We study the effect of adversarial perturbations of images on the estimates of disparity by deep learning models trained for stereo.
Predicting Training Time Without Training
We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function.
DyStaB: Unsupervised Object Segmentation via Dynamic-Static Bootstrapping
Then, it uses the segments to learn object models that can be used for detection in a static image.
Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance
In this paper, we propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties for safer navigation through cluttered environments.
Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction
We propose Geo-PIFu, a method to recover a 3D mesh from a monocular color image of a clothed person.
Targeted Adversarial Perturbations for Monocular Depth Prediction
We study the effect of adversarial perturbations on the task of monocular depth prediction.
Stochastic batch size for adaptive regularization in deep network optimization
We propose a first-order stochastic optimization algorithm incorporating adaptive regularization applicable to machine learning problems in deep learning framework.
Learning to Manipulate Individual Objects in an Image
We describe a method to train a generative model with latent factors that are (approximately) independent and localized.
FDA: Fourier Domain Adaptation for Semantic Segmentation
We describe a simple method for unsupervised domain adaptation, whereby the discrepancy between the source and target distributions is reduced by swapping the low-frequency spectrum of one with the other.
Ranked #4 on
Domain Adaptation
on Panoptic SYNTHIA-to-Mapillary
Phase Consistent Ecological Domain Adaptation
We introduce two criteria to regularize the optimization involved in learning a classifier in a domain where no annotated data are available, leveraging annotated data in a different domain, a problem known as unsupervised domain adaptation.
Towards Backward-Compatible Representation Learning
Backward compatibility is critical to quickly deploy new embedding models that leverage ever-growing large-scale training datasets and improvements in deep learning architectures and training methods.
Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations
We describe a procedure for removing dependency on a cohort of training data from a trained deep network that improves upon and generalizes previous methods to different readout functions and can be extended to ensure forgetting in the activations of the network.
Rethinking the Hyperparameters for Fine-tuning
Our findings challenge common practices of fine-tuning and encourages deep learning practitioners to rethink the hyperparameters for fine-tuning.
Multi-Task Incremental Learning for Object Detection
For the difficult cases, where the domain gaps and especially category differences are large, we explore three different exemplar sampling methods and show the proposed adaptive sampling method is effective to select diverse and informative samples from entire datasets, to further prevent forgetting.
Incremental Meta-Learning via Indirect Discriminant Alignment
Majority of the modern meta-learning methods for few-shot classification tasks operate in two phases: a meta-training phase where the meta-learner learns a generic representation by solving multiple few-shot tasks sampled from a large dataset and a testing phase, where the meta-learner leverages its learnt internal representation for a specific few-shot task involving classes which were not seen during the meta-training phase.
SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning
To learn this representation, we train a squeeze network to drive using annotations for the side task as input.
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
We explore the problem of selectively forgetting a particular subset of the data used for training a deep neural network.
Meta-Q-Learning
This paper introduces Meta-Q-Learning (MQL), a new off-policy algorithm for meta-Reinforcement Learning (meta-RL).
Where is the Information in a Deep Network?
We relate this to the Information in the Weights, and use this result to show that models of low (information) complexity not only generalize better, but are bound to learn invariant representations of future inputs.
A Baseline for Few-Shot Image Classification
When fine-tuned transductively, this outperforms the current state-of-the-art on standard datasets such as Mini-ImageNet, Tiered-ImageNet, CIFAR-FS and FC-100 with the same hyper-parameters.
Toward Understanding Catastrophic Forgetting in Continual Learning
As an application, we apply our procedure to study two properties of a task sequence: (1) total complexity and (2) sequential heterogeneity.
Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
Deep neural networks (DNNs), however, challenge this view: We show that removing regularization after an initial transient period has little effect on generalization, even if the final loss landscape is the same as if there had been no regularization.
Where is the Information in a Deep Neural Network?
We establish a novel relation between the information in the weights and the effective information in the activations, and use this result to show that models with low (information) complexity not only generalize better, but are bound to learn invariant representations of future inputs.
Unsupervised Domain Adaptation via Regularized Conditional Alignment
We propose a method for unsupervised domain adaptation that trains a shared embedding to align the joint distributions of inputs (domain) and outputs (classes), making any classifier agnostic to the domain.
Unsupervised Depth Completion from Visual Inertial Odometry
Our method first constructs a piecewise planar scaffolding of the scene, and then uses it to infer dense depth using the image along with the sparse points.
Ranked #4 on
Depth Completion
on VOID
Few-Shot Learning with Embedded Class Models and Shot-Free Meta Training
We propose a method for learning embeddings for few-shot learning that is suitable for use with any number of ways and any number of shots (shot-free).
Critical Learning Periods in Deep Networks
Deficits that do not affect low-level statistics, such as vertical flipping of the images, have no lasting effect on performance and can be overcome with further training.
Meta-Learning with Differentiable Convex Optimization
We propose to use these predictors as base learners to learn representations for few-shot learning and show they offer better tradeoffs between feature size and performance across a range of few-shot recognition benchmarks.
Ranked #12 on
Few-Shot Image Classification
on FC100 5-way (1-shot)
The Information Complexity of Learning Tasks, their Structure and their Distance
Our framework is the first to measure complexity in a way that accounts for the effect of the optimization scheme, which is critical in Deep Learning.
Bilateral Cyclic Constraint and Adaptive Regularization for Unsupervised Monocular Depth Prediction
Supervised learning methods to infer (hypothesize) depth of a scene from a single image require costly per-pixel ground-truth.
Zero Shot Learning with the Isoperimetric Loss
We introduce the isoperimetric loss as a regularization criterion for learning the map from a visual representation to a semantic embedding, to be used to transfer knowledge to unknown classes in a zero-shot learning setting.
Task2Vec: Task Embedding for Meta-Learning
We demonstrate that this embedding is capable of predicting task similarities that match our intuition about semantic and taxonomic relations between different visual tasks (e. g., tasks based on classifying different types of plants are similar) We also demonstrate the practical value of this framework for the meta-task of selecting a pre-trained feature extractor for a new task.
Dense Depth Posterior (DDP) from Single Image and Sparse Range
We present a deep learning system to infer the posterior distribution of a dense depth map associated with an image, by exploiting sparse range measurements, for instance from a lidar.
Ranked #5 on
Depth Completion
on VOID
Mono3D++: Monocular 3D Vehicle Detection with Two-Scale 3D Hypotheses and Task Priors
We present a method to infer 3D pose and shape of vehicles from a single image.
Unsupervised Moving Object Detection via Contextual Information Separation
We propose an adversarial contextual model for detecting moving objects in images.
GeoNet: Deep Geodesic Networks for Point Cloud Analysis
Surface-based geodesic topology provides strong cues for object semantic analysis and geometric modeling.
Dynamics and Reachability of Learning Tasks
We compute the transition probability between two learning tasks, and show that it decomposes into two factors.
Geo-Supervised Visual Depth Prediction
We propose using global orientation from inertial measurements, and the bias it induces on the shape of objects populating the scene, to inform visual 3D reconstruction.
Conditional Prior Networks for Optical Flow
On the other hand, fully supervised methods learn the regularity in the annotated data, without explicit regularization and with the risk of overfitting.
Visual-Inertial Object Detection and Mapping
We present a method to populate an unknown environment with models of previously seen objects, placed in a Euclidean reference frame that is inferred causally and on-line using monocular video along with inertial sensors.
Empirical Study of the Topology and Geometry of Deep Networks
We specifically study the topology of classification regions created by deep networks, as well as their associated decision boundary.
Input and Weight Space Smoothing for Semi-supervised Learning
We propose regularizing the empirical loss for semi-supervised learning by acting on both the input (data) space, and the weight (parameter) space.
SaaS: Speed as a Supervisor for Semi-supervised Learning
We introduce the SaaS Algorithm for semi-supervised learning, which uses learning speed during stochastic gradient descent in a deep neural network to measure the quality of an iterative estimate of the posterior probability of unknown labels.
OATM: Occlusion Aware Template Matching by Consensus Set Maximization
We present a novel approach to template matching that is efficient, can handle partial occlusions, and comes with provable performance guarantees.
Mathematics of Deep Learning
Recently there has been a dramatic increase in the performance of recognition systems due to the introduction of deep architectures for representation learning and classification.
DeepRadiologyNet: Radiologist Level Pathology Detection in CT Head Images
Thus, DeepRadiologyNet enables significant reduction in the workload of human radiologists by automatically filtering studies and reporting on the high-confidence ones at an operating point well below the literal error rate for US Board Certified radiologists, estimated at 0. 82%.
Critical Learning Periods in Deep Neural Networks
Deficits that do not affect low-level statistics, such as vertical flipping of the images, have no lasting effect on performance and can be overcome with further training.
Block-Cyclic Stochastic Coordinate Descent for Deep Neural Networks
We present a stochastic first-order optimization algorithm, named BCSC, that adds a cyclic constraint to stochastic block-coordinate descent.
A Separation Principle for Control in the Age of Deep Learning
Again this can be finitely-parametrized using a deep neural network, and already some applications are beginning to emerge.
Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks
So SGD does perform variational inference, but for a different loss than the one used to compute the gradients.
Parle: parallelizing stochastic gradient descent
We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4x faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additional hyper-parameters.
S2F: Slow-To-Fast Interpolator Flow
We introduce a method to compute optical flow at multiple scales of motion, without resorting to multi- resolution or combinatorial methods.
Zero Shot Learning via Multi-Scale Manifold Regularization
We address zero-shot learning using a new manifold alignment framework based on a localized multi-scale transform on graphs.
Visual-Inertial-Semantic Scene Representation for 3D Object Detection
We describe a system to detect objects in three-dimensional space using video and inertial sensors (accelerometer and gyrometer), ubiquitous in modern mobile platforms from phones to drones.
Emergence of Invariance and Disentanglement in Deep Representations
Using established principles from Statistics and Information Theory, we show that invariance to nuisance factors in a deep neural network is equivalent to information minimality of the learned representation, and that stacking layers and injecting noise during training naturally bias the network towards learning invariant representations.
Reinforced Temporal Attention and Split-Rate Transfer for Depth-Based Person Re-Identification
We address the problem of person re-identification from commodity depth sensors.
Classification regions of deep neural networks
The goal of this paper is to analyze the geometric properties of deep neural network classifiers in the input space.
Robustness of classifiers to universal perturbations: a geometric perspective
Deep networks have recently been shown to be vulnerable to universal perturbations: there exist very small image-agnostic perturbations that cause most natural images to be misclassified by such classifiers.
Adaptive Regularization of Some Inverse Problems in Image Analysis
We present an adaptive regularization scheme for optimizing composite energy functionals arising in image analysis problems.
Deep Relaxation: partial differential equations for optimizing deep neural networks
In this paper we establish a connection between non-convex optimization methods for training deep neural networks and nonlinear partial differential equations (PDEs).
Multi-Label Segmentation via Residual-Driven Adaptive Regularization
We present a variational multi-label segmentation algorithm based on a robust Huber loss for both the data and the regularizer, minimized within a convex optimization framework.
Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape.
Information Dropout: Learning Optimal Representations Through Noisy Computation
The cross-entropy loss commonly used in deep learning is closely related to the defining properties of optimal representations, but does not enforce some of the key properties.
ShapeFit and ShapeKick for Robust, Scalable Structure from Motion
We introduce a new method for location recovery from pair-wise directions that leverages an efficient convex program that comes with exact recovery guarantees, even in the presence of adversarial outliers.
Visual-Inertial-Semantic Scene Representation for 3-D Object Detection
We describe a system to detect objects in three-dimensional space using video and inertial sensors (accelerometer and gyrometer), ubiquitous in modern mobile platforms from phones to drones.
A Theory of Local Matching: SIFT and Beyond
Can it suggest new algorithms with reduced computational complexity or new descriptors with better accuracy for matching?
Self-Occlusions and Disocclusions in Causal Video Object Segmentation
We propose a method to detect disocclusion in video sequences of three-dimensional scenes and to partition the disoccluded regions into objects, defined by coherent deformation corresponding to surfaces in the scene.
A Simple Hierarchical Pooling Data Structure for Loop Closure
We propose a data structure obtained by hierarchically averaging bag-of-word descriptors during a sequence of views that achieves average speedups in large-scale loop closure applications ranging from 4 to 20 times on benchmark datasets.
On the energy landscape of deep networks
Specifically, we show that a regularization term akin to a magnetic field can be modulated with a single scalar parameter to transition the loss function from a complex, non-convex landscape with exponentially many local minima, to a phase with a polynomial number of minima, all the way down to a trivial landscape with a unique minimum.
Multi-View Feature Engineering and Learning
We frame the problem of local representation of imaging data as the computation of minimal sufficient statistics that are invariant to nuisance variability induced by viewpoint and illumination.
Efficient Minimal-Surface Regularization of Perspective Depth Maps in Variational Stereo
We propose a method for dense three-dimensional surface reconstruction that leverages the strengths of shape-based approaches, by imposing regularization that respects the geometry of the surface, and the strength of depth-map-based stereo, by avoiding costly computation of surface topology.
Texture Representations for Image and Video Synthesis
In texture synthesis and classification, algorithms require a small texture to be provided as an input, which is assumed to be representative of a larger region to be re-synthesized or categorized.
Causal Video Object Segmentation From Persistence of Occlusions
Occlusion relations inform the partition of the image domain into ``objects'' but are difficult to determine from a single image or short-baseline video.
An Empirical Evaluation of Current Convolutional Architectures' Ability to Manage Nuisance Location and Scale Variability
We conduct an empirical study to test the ability of Convolutional Neural Networks (CNNs) to reduce the effects of nuisance transformations of the input data, such as location, scale and aspect ratio.
Boosting Convolutional Features for Robust Object Proposals
Modern detection algorithms like Regions with CNNs (Girshick et al., 2014) rely on Selective Search (Uijlings et al., 2013) to propose regions which with high probability represent objects, where in turn CNNs are deployed for classification.
Domain-Size Pooling in Local Descriptors: DSP-SIFT
We introduce a simple modification of local image descriptors, such as SIFT, based on pooling gradient orientations across different domain sizes, in addition to spatial locations.
Visual Scene Representations: Contrast, Scaling and Occlusion
We study the structure of representations, defined as approximations of minimal sufficient statistics that are maximal invariants to nuisance factors, for visual data subject to scaling and occlusion of line-of-sight.
Visual Representations: Defining Properties and Deep Approximations
Visual representations are defined in terms of minimal sufficient statistics of visual data, for a class of tasks, that are also invariant to nuisance variability.
Cavlectometry: Towards Holistic Reconstruction of Large Mirror Objects
We introduce a method based on the deflectometry principle for the reconstruction of specular objects exhibiting significant size and geometric complexity.
Active Frame, Location, and Detector Selection for Automated and Manual Video Annotation
We describe an information-driven active selection approach to determine which detectors to deploy at which location in which frame of a video to minimize semantic class label uncertainty at every pixel, with the smallest computational cost that ensures a given uncertainty bound.
Asymmetric Sparse Kernel Approximations for Large-scale Visual Search
We introduce an asymmetric sparse approximate embedding optimized for fast kernel comparison operations arising in large-scale visual search.
Second-Order Shape Optimization for Geometric Inverse Problems in Vision
We develop a method for optimization in shape spaces, i. e., sets of surfaces modulo re-parametrization.
On the Design and Analysis of Multiple View Descriptors
We propose an extension of popular descriptors based on gradient orientation histograms (HOG, computed in a single image) to multiple views.
Nonlinearly Constrained MRFs: Exploring the Intrinsic Dimensions of Higher-Order Cliques
This paper introduces an efficient approach to integrating non-local statistics into the higher-order Markov Random Fields (MRFs) framework.
CLAM: Coupled Localization and Mapping with Efficient Outlier Handling
We describe a method to efficiently generate a model (map) of small-scale objects from video.
Controlled Recognition Bounds for Visual Learning and Exploration
We describe the tradeoff between the performance in a visual recognition problem and the control authority that the agent can exercise on the sensing process.
Multiple Instance Filtering
We propose a robust filtering approach based on semi-supervised and multiple instance learning (MIL).
Steps Towards a Theory of Visual Information: Active Perception, Signal-to-Symbol Conversion and the Interplay Between Sensing and Control
The concept of Actionable Information is described, that relates to a notion of information championed by J. Gibson, and a notion of "complete information" that relates to the minimal sufficient statistics of a complete representation.
Occlusion Detection and Motion Estimation with Convex Optimization
We tackle the problem of simultaneously detecting occlusions and estimating optical flow.
