Search Results for author:
Found 207 papers, 116 papers with code
GDumb: A Simple Approach that Questions Our Progress in Continual Learning
We discuss a general formulation for the Continual Learning (CL) problem for classification---a learning task where a stream provides samples to a learner and the goal of the learner, depending on the samples it receives, is to continually upgrade its knowledge about the old classes and learn new ones.
No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation.
NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning
Furthermore, the cascading learning of NeRF-VPT introduces adaptability to scenarios with sparse inputs, resulting in a significant enhancement of accuracy for sparse-view novel view synthesis.
Placing Objects in Context via Inpainting for Out-of-distribution Segmentation
In our experiments, we present different anomaly segmentation datasets based on POC-generated data and show that POC can improve the performance of recent state-of-the-art anomaly fine-tuning methods in several standardized benchmarks.
Prompting a Pretrained Transformer Can Be a Universal Approximator
Despite the widespread adoption of prompting, prompt tuning and prefix-tuning of transformer models, our theoretical understanding of these fine-tuning methods remains limited.
Self-consistent Validation for Machine Learning Electronic Structure
Machine learning has emerged as a significant approach to efficiently tackle electronic structure problems.
Revealing Decurve Flows for Generalized Graph Propagation
\textbf{Secondly}, we propose the {\em Continuous Unified Ricci Curvature} (\textbf{CURC}), an extension of celebrated {\em Ollivier-Ricci Curvature} for directed and weighted graphs.
RanDumb: A Simple Approach that Questions the Efficacy of Continual Representation Learning
Our investigation is both surprising and alarming as it questions our understanding of how to effectively design and train models that require efficient continual representation learning, and necessitates a principled reinvestigation of the widely explored problem formulation itself.
Secret Collusion Among Generative AI Agents
In this paper, we comprehensively formalise the problem of secret collusion in systems of generative AI agents by drawing on relevant concepts from both the AI and security literature.
From Categories to Classifier: Name-Only Continual Learning by Exploring the Web
Continual Learning (CL) often relies on the availability of extensive annotated datasets, an assumption that is unrealistically time-consuming and costly in practice.
When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations
Context-based fine-tuning methods, including prompting, in-context learning, soft prompting (also known as prompt tuning), and prefix-tuning, have gained popularity due to their ability to often match the performance of full fine-tuning with a fraction of the parameters.
Segment, Select, Correct: A Framework for Weakly-Supervised Referring Segmentation
Referring Image Segmentation (RIS) - the problem of identifying objects in images through natural language sentences - is a challenging task currently mostly solved through supervised learning.
Fine-tuning can cripple your foundation model; preserving features may be the solution
This is an undesirable effect of fine-tuning as a substantial amount of resources was used to learn these pre-trained concepts in the first place.
Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer
To bridge these gaps, in this paper, we propose Tem-Adapter, which enables the learning of temporal dynamics and complex semantics by a visual Temporal Aligner and a textual Semantic Aligner.
Ranked #1 on
Video Question Answering
on SUTD-TrafficQA
Faithful Knowledge Distillation
To address these questions, we introduce a faithful imitation framework to discuss the relative calibration of confidences and provide empirical and certified methods to evaluate the relative calibration of a student w. r. t.
Language Model Tokenizers Introduce Unfairness Between Languages
Recent language models have shown impressive multilingual performance, even when not explicitly trained for it.
Rapid Adaptation in Online Continual Learning: Are We Evaluating It Right?
We revisit the common practice of evaluating adaptation of Online Continual Learning (OCL) algorithms through the metric of online accuracy, which measures the accuracy of the model on the immediate next few samples.
Certifying Ensembles: A General Certification Theory with S-Lipschitzness
Improving and guaranteeing the robustness of deep learning models has been a topic of intense research.
Fairness in AI and Its Long-Term Implications on Society
Successful deployment of artificial intelligence (AI) in various settings has led to numerous positive outcomes for individuals and society.
Don't FREAK Out: A Frequency-Inspired Approach to Detecting Backdoor Poisoned Samples in DNNs
In this paper we investigate the frequency sensitivity of Deep Neural Networks (DNNs) when presented with clean samples versus poisoned samples.
Computationally Budgeted Continual Learning: What Does Matter?
Our conclusions are consistent in a different number of stream time steps, e. g., 20 to 200, and under several computational budgets.
Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentation
Referring image segmentation segments an image from a language expression.
MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices
The distinct data modality offered by high-resolution RGB images and low-resolution depth maps captured on a mobile device, when combined with precise 3D geometry annotations, presents a unique opportunity for future research on high-fidelity 3D reconstruction.
MOSE: A New Dataset for Video Object Segmentation in Complex Scenes
However, since the target objects in these existing datasets are usually relatively salient, dominant, and isolated, VOS under complex scenes has rarely been studied.
Real-Time Evaluation in Online Continual Learning: A New Hope
We show that a simple baseline outperforms state-of-the-art CL methods under this evaluation, questioning the applicability of existing methods in realistic settings.
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning
We introduce Patch Aligned Contrastive Learning (PACL), a modified compatibility function for CLIP's contrastive loss, intending to train an alignment between the patch tokens of the vision encoder and the CLS token of the text encoder.
Traditional Classification Neural Networks are Good Generators: They are Competitive with DDPMs and GANs
Proving that classifiers have learned the data distribution and are ready for image generation has far-reaching implications, for classifiers are much easier to train than generative models like DDPMs and GANs.
Label Alignment Regularization for Distribution Shift
Recent work has highlighted the label alignment property (LAP) in supervised learning, where the vector of all labels in the dataset is mostly in the span of the top few singular vectors of the data matrix.
Structure-Preserving 3D Garment Modeling with Neural Sewing Machines
In this paper, we propose a novel Neural Sewing Machine (NSM), a learning-based framework for structure-preserving 3D garment modeling, which is capable of learning representations for garments with diverse shapes and topologies and is successfully applied to 3D garment reconstruction and controllable manipulation.
Bipartite Graph Reasoning GANs for Person Pose and Facial Image Synthesis
To further capture the change in pose of each part more precisely, we propose a novel part-aware bipartite graph reasoning (PBGR) block to decompose the task of reasoning the global structure transformation with a bipartite graph into learning different local transformations for different semantic body/face parts.
Holistically-Attracted Wireframe Parsing: From Supervised to Self-Supervised Learning
This article presents Holistically-Attracted Wireframe Parsing (HAWP), a method for geometric analysis of 2D images containing wireframes formed by line segments and junctions.
Raising the Bar on the Evaluation of Out-of-Distribution Detection
In this paper, we define 2 categories of OoD data using the subtly different concepts of perceptual/visual and semantic similarity to in-distribution (iD) data.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
+2
Learn what matters: cross-domain imitation learning with task-relevant embeddings
We study how an autonomous agent learns to perform a task from demonstrations in a different domain, such as a different environment or different agent.
Dynamic Graph Message Passing Networks for Visual Recognition
A fully-connected graph, such as the self-attention operation in Transformers, is beneficial for such modelling, however, its computational overhead is prohibitive.
Memory-Driven Text-to-Image Generation
We introduce a memory-driven semi-parametric approach to text-to-image generation, which is based on both parametric and non-parametric techniques.
Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation
In this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of semantic annotations in the 360-degree imagery.
Ranked #1 on
Semantic Segmentation
on SynPASS
An Impartial Take to the CNN vs Transformer Robustness Contest
Following the surge of popularity of Transformers in Computer Vision, several studies have attempted to determine whether they could be more robust to distribution shifts and provide better uncertainty estimates than Convolutional Neural Networks (CNNs).
Illusory Attacks: Detectability Matters in Adversarial Attacks on Sequential Decision-Makers
Autonomous agents deployed in the real world need to be robust against adversarial attacks on sensory inputs.
Vision Transformers: From Semantic Segmentation to Dense Prediction
In this work, for the first time we explore the global context learning potentials of ViTs for dense visual prediction (e. g., semantic segmentation).
Sample-dependent Adaptive Temperature Scaling for Improved Calibration
The most common post-hoc approach to compensate for this is to perform temperature scaling, which adjusts the confidences of the predictions on any input by scaling the logits by a fixed value.
SiamMask: A Framework for Fast Online Object Tracking and Segmentation
In this paper we introduce SiamMask, a framework to perform both visual object tracking and video object segmentation, in real-time, with the same simple method.
RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness
We show that the effectiveness of the well celebrated Mixup [Zhang et al., 2018] can be further improved if instead of using it as the sole learning objective, it is utilized as an additional regularizer to the standard cross-entropy loss.
How Robust is Unsupervised Representation Learning to Distribution Shift?
As such, there is a lack of insight on the robustness of the representations learned from unsupervised methods, such as self-supervised learning (SSL) and auto-encoder based algorithms (AE), to distribution shift.
Catastrophic overfitting can be induced with discriminative non-robust features
Through extensive experiments we analyze this novel phenomenon and discover that the presence of these easy features induces a learning shortcut that leads to CO. Our findings provide new insights into the mechanisms of CO and improve our understanding of the dynamics of AT.
Zero-Shot Logit Adjustment
As a consequence of our derivation, the aforementioned two properties are incorporated into the classifier training as seen-unseen priors via logit adjustment.
Ranked #1 on
Generalized Zero-Shot Learning
on AwA2
(Accuracy Unseen metric)
Deconstructed Generation-Based Zero-Shot Model
Recent research on Generalized Zero-Shot Learning (GZSL) has focused primarily on generation-based methods.
BNV-Fusion: Dense 3D Reconstruction using Bi-level Neural Volume Fusion
Dense 3D reconstruction from a stream of depth images is the key to many mixed reality and robotic applications.
Local and Global GANs with Semantic-Aware Upsampling for Image Generation
To learn more discriminative class-specific feature representations for the local generation, we also propose a novel classification module.
Make Some Noise: Reliable and Efficient Single-Step Adversarial Training
Recently, Wong et al. showed that adversarial training with single-step FGSM leads to a characteristic failure mode named Catastrophic Overfitting (CO), in which a model becomes suddenly vulnerable to multi-step attacks.
On the Robustness of Quality Measures for GANs
Finally, we show that the FID can be robustified by simply replacing the standard Inception with a robust Inception.
Adversarial Masking for Self-Supervised Learning
We propose ADIOS, a masked image model (MIM) framework for self-supervised learning, which simultaneously learns a masking function and an image encoder using an adversarial objective.
Learning to Hash Naturally Sorts
In this paper, we tackle this problem by introducing Naturally-Sorted Hashing (NSH).
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
Referring image segmentation is a fundamental vision-language task that aims to segment out an object referred to by a natural language expression from an image.
Generalized Referring Expression Segmentation
Image Segmentation
+2
Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AI-NIDS
We then argue that attacker-defender fixed points are themselves general-sum games with complex phase transitions, and introduce a temporally extended multi-agent reinforcement learning framework in which the resultant dynamics can be studied.
Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge
To promote the development of occlusion understanding, we collect a large-scale dataset called OVIS for video instance segmentation in the occluded scenario.
Deep Deterministic Uncertainty for Semantic Segmentation
We extend Deep Deterministic Uncertainty (DDU), a method for uncertainty estimation using feature space densities, to semantic segmentation.
Detecting and Quantifying Malicious Activity with Simulation-based Inference
We propose the use of probabilistic programming techniques to tackle the malicious user identification problem in a recommendation algorithm.
RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
Thus, we propose RobustART, the first comprehensive Robustness investigation benchmark on ImageNet regarding ARchitecture design (49 human-designed off-the-shelf architectures and 1200+ networks from neural architecture search) and Training techniques (10+ techniques, e. g., data augmentation) towards diverse noises (adversarial, natural, and system noises).
ANCER: Anisotropic Certification via Sample-wise Volume Maximization
Randomized smoothing has recently emerged as an effective tool that enables certification of deep neural network classifiers at scale.
Do Different Tracking Tasks Require Different Appearance Models?
We show how most tracking tasks can be solved within this framework, and that the same appearance model can be successfully used to obtain results that are competitive against specialised methods for most of the tasks considered.
Ranked #2 on
Video Object Segmentation
on DAVIS 2017
(mIoU metric)
Multi-Object Tracking
Multi-Object Tracking and Segmentation
+10
DeformRS: Certifying Input Deformations with Randomized Smoothing
Deep neural networks are vulnerable to input deformations in the form of vector fields of pixel displacements and to other parameterized geometric deformations e. g. translations, rotations, etc.
Learning Multimodal VAEs through Mutual Supervision
Here we introduce a novel alternative, the MEME, that avoids such explicit combinations by repurposing semi-supervised VAEs to combine information between modalities implicitly through mutual supervision.
KL Guided Domain Adaptation
A common approach in the domain adaptation literature is to learn a representation of the input that has the same (marginal) distribution over the source and the target domain.
You Never Cluster Alone
On one hand, with the corresponding assignment variables being the weight, a weighted aggregation along the data points implements the set representation of a cluster.
Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning
Previous cycle-consistency correspondence learning methods usually leverage image patches for training.
Gradient Matching for Domain Generalization
We perform experiments on both the Wilds benchmark, which captures distribution shift in the real world, as well as datasets in DomainBed benchmark that focuses more on synthetic-to-real transfer.
Solving Inefficiency of Self-supervised Representation Learning
In this paper, we reveal two contradictory phenomena in contrastive learning that we call under-clustering and over-clustering problems, which are major obstacles to learning efficiency.
Ranked #1 on
Self-Supervised Person Re-Identification
on SYSU-30k
Improved Branch and Bound for Neural Network Verification via Lagrangian Decomposition
Finally, we design a BaB framework, named Branch and Dual Network Bound (BaDNB), based on our novel bounding and branching algorithms.
Cloth Interactive Transformer for Virtual Try-On
In the second stage, we put forth a CIT reasoning block for establishing global mutual interactive dependencies among person representation, the warped clothing item, and the corresponding warped cloth mask.
Combating Adversaries with Anti-Adversaries
Deep neural networks are vulnerable to small input perturbations known as adversarial attacks.
Deep Deterministic Uncertainty: A Simple Baseline
Reliable uncertainty from deterministic single-forward pass models is sought after because conventional methods of uncertainty quantification are computationally expensive.
Shape-Tailored Deep Neural Networks
We stack multiple PDE layers to generalize a deep CNN to arbitrary regions, and apply it to segmentation.
Occluded Video Instance Segmentation: A Benchmark
On the OVIS dataset, the highest AP achieved by state-of-the-art algorithms is only 16. 3, which reveals that we are still at a nascent stage for understanding objects, instances, and videos in a real-world scenario.
Ranked #39 on
Video Instance Segmentation
on OVIS validation
Scaling the Convex Barrier with Sparse Dual Algorithms
Tight and efficient neural network bounding is crucial to the scaling of neural network verification systems.
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
In this paper, we aim to provide an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task.
Ranked #2 on
Semantic Segmentation
on FoodSeg103
(using extra training data)
On Batch Normalisation for Approximate Bayesian Inference
We study batch normalisation in the context of variational inference methods in Bayesian neural networks, such as mean-field or MC Dropout.
Multi-shot Temporal Event Localization: a Benchmark
Current developments in temporal event or action localization usually target actions captured by a single camera.
Ranked #2 on
Temporal Action Localization
on MUSES
GeoNet++: Iterative Geometric Neural Network with Edge-Aware Refinement for Joint Depth and Surface Normal Estimation
Note that GeoNet++ is generic and can be used in other depth/normal prediction frameworks to improve the quality of 3D reconstruction and pixel-wise accuracy of depth and surface normals.
Data-Dependent Randomized Smoothing
In this work, we revisit Gaussian randomized smoothing and show that the variance of the Gaussian distribution can be optimized at each input so as to maximize the certification radius for the construction of the smooth classifier.
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
Most recently developed approaches to cooperative multi-agent reinforcement learning in the \emph{centralized training with decentralized execution} setting involve estimating a centralized, joint value function.
Lightweight Generative Adversarial Networks for Text-Guided Image Manipulation
To achieve this, a new word-level discriminator is proposed, which provides the generator with fine-grained training feedback at word-level, to facilitate training a lightweight generator that has a small number of parameters, but can still correctly focus on specific visual attributes of an image, and then edit them without affecting other contents that are not described in the text.
Continual Learning in Low-rank Orthogonal Subspaces
In continual learning (CL), a learner is faced with a sequence of tasks, arriving one after the other, and the goal is to remember all the tasks once the continual learning experience is finished.
AutoSimulate: (Quickly) Learning Synthetic Data Generation
Simulation is increasingly being used for generating large labelled datasets in many machine learning problems.
Bipartite Graph Reasoning GANs for Person Image Generation
We present a novel Bipartite Graph Reasoning GAN (BiGraphGAN) for the challenging person image generation task.
Ranked #1 on
Pose Transfer
on Market-1501
(PCKh metric)
XingGAN for Person Image Generation
We propose a novel Generative Adversarial Network (XingGAN or CrossingGAN) for person image generation tasks, i. e., translating the pose of a given person to a desired one.
Ranked #1 on
Pose Transfer
on Market-1501
(IS metric)
WordCraft: An Environment for Benchmarking Commonsense Agents
This is partly due to the lack of lightweight simulation environments that sufficiently reflect the semantics of the real world and provide knowledge sources grounded with respect to observations in an RL environment.
How benign is benign overfitting?
We investigate two causes for adversarial vulnerability in deep neural networks: bad data and (poorly) trained models.
Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models
Multimodal learning for generative models often refers to the learning of abstract concepts from the commonality of information in multiple modalities, such as vision and language.
STEER: Simple Temporal Regularization For Neural ODEs
Training Neural Ordinary Differential Equations (ODEs) is often computationally expensive.
Capturing Label Characteristics in VAEs
We present a principled approach to incorporating labels in VAEs that captures the rich characteristic information associated with those labels.
Progressive Skeletonization: Trimming more fat from a network at initialization
Recent studies have shown that skeletonization (pruning parameters) of networks \textit{at initialization} provides all the practical benefits of sparsity both at inference and training time, while only marginally degrading their performance.
A Revised Generative Evaluation of Visual Dialogue
Evaluating Visual Dialogue, the task of answering a sequence of questions relating to a visual input, remains an open research challenge.
Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
As a result, we find across various workloads of data set, network model, and optimization algorithm that there exists a general scaling trend between batch size and number of training steps to convergence for the effect of data parallelism, and further, difficulty of training under sparsity.
Cross-modal Deep Face Normals with Deactivable Skip Connections
Core to our approach is a novel module that we call deactivable skip connections, which allows integrating both the auto-encoded and image-to-normal branches within the same architecture that can be trained end-to-end.
FACMAC: Factored Multi-Agent Centralised Policy Gradients
We propose FACtored Multi-Agent Centralised policy gradients (FACMAC), a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces.
Holistically-Attracted Wireframe Parsing
For computing line segment proposals, a novel exact dual representation is proposed which exploits a parsimonious geometric reparameterization for line segments and forms a holistic 4-dimensional attraction field map for an input image.
Ranked #4 on
Line Segment Detection
on York Urban Dataset
Lagrangian Decomposition for Neural Network Verification
Both the algorithms offer three advantages: (i) they yield bounds that are provably at least as tight as previous dual algorithms relying on Lagrangian relaxations; (ii) they are based on operations analogous to forward/backward pass of neural networks layers and are therefore easily parallelizable, amenable to GPU implementation and able to take advantage of the convolutional structure of problems; and (iii) they allow for anytime stopping while still providing valid bounds.
Calibrating Deep Neural Networks using Focal Loss
To facilitate the use of focal loss in practice, we also provide a principled approach to automatically select the hyperparameter involved in the loss function.
Image-to-Image Translation with Text Guidance
The goal of this paper is to embed controllable factors, i. e., natural language descriptions, into image-to-image translation with generative adversarial networks, which allows text descriptions to determine the visual attributes of synthetic images.
Multi-Channel Attention Selection GANs for Guided Image-to-Image Translation
In the first stage, the input image and the conditional semantic guidance are fed into a cycled semantic-guided generation network to produce initial coarse results.
Generative Adversarial Network
Image-to-Image Translation
+1
Unifying Training and Inference for Panoptic Segmentation
This panoptic submodule gives rise to a novel propagation mechanism for panoptic logits and enables the network to output a coherent panoptic segmentation map for both "stuff" and "thing" classes, without any post-processing.
Rethinking Class Relations: Absolute-relative Supervised and Unsupervised Few-shot Learning
The majority of existing few-shot learning methods describe image relations with binary labels.
Few-shot Action Recognition with Permutation-invariant Attention
Such encoded blocks are aggregated by permutation-invariant pooling to make our approach robust to varying action lengths and long-range temporal dependencies whose patterns are unlikely to repeat even in clips of the same class.
Ranked #6 on
Few Shot Action Recognition
on Kinetics-100
Improving Few-shot Learning by Spatially-aware Matching and CrossTransformer
In this paper, we study the impact of scale and location mismatch in the few-shot learning scenario, and propose a novel Spatially-aware Matching (SM) scheme to effectively perform matching across multiple scales and locations, and learn image relations by giving the highest weights to the best matching pairs.
Local Class-Specific and Global Image-Level Generative Adversarial Networks for Semantic-Guided Scene Generation
To tackle this issue, in this work we consider learning the scene generation in a local context, and correspondingly design a local class-specific generative network with semantic maps as a guidance, which separately constructs and learns sub-generators concentrating on the generation of different classes, and is able to provide more scene details.
Learning Regional Attraction for Line Segment Detection
Given a line segment map, the proposed regional attraction first establishes the relationship between line segments and regions in the image lattice.
Lessons from reinforcement learning for biological representations of space
Neuroscientists postulate 3D representations in the brain in a variety of different coordinate frames (e. g. 'head-centred', 'hand-centred' and 'world-based').
ManiGAN: Text-Guided Image Manipulation
The goal of our paper is to semantically edit parts of an image matching a given text that describes desired attributes (e. g., texture, colour, and background), while preserving other contents that are irrelevant to the text.
Transflow Learning: Repurposing Flow Models Without Retraining
It is well known that deep generative models have a rich latent space, and that it is possible to smoothly manipulate their outputs by traversing this latent space.
Siam R-CNN: Visual Tracking by Re-Detection
We present Siam R-CNN, a Siamese re-detection architecture which unleashes the full power of two-stage object detection approaches for visual object tracking.
Ranked #5 on
Object Tracking
on COESOT
AttentionGAN: Unpaired Image-to-Image Translation using Attention-Guided Generative Adversarial Networks
State-of-the-art methods in image-to-image translation are capable of learning a mapping from a source domain to a target domain with unpaired image data.
Ranked #1 on
Facial Expression Translation
on CelebA
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models
In this work, we characterise successful learning of such models as the fulfillment of four criteria: i) implicit latent decomposition into shared and private subspaces, ii) coherent joint generation over all modalities, iii) coherent cross-generation across individual modalities, and iv) improved model learning for individual modalities through multi-modal integration.
Anchor Diffusion for Unsupervised Video Object Segmentation
Unsupervised video object segmentation has often been tackled by methods based on recurrent neural networks and optical flow.
Ranked #18 on
Unsupervised Video Object Segmentation
on DAVIS 2016 val
Amortized Rejection Sampling in Universal Probabilistic Programming
Naive approaches to amortized inference in probabilistic programs with unbounded loops can produce estimators with infinite variance.
Mirror Descent View for Neural Network Quantization
Quantizing large Neural Networks (NN) while maintaining the performance is highly desirable for resource-limited devices due to reduced memory and time complexity.
Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation
We propose an interactive GAN-based sketch-to-image translation method that helps novice users create images of simple objects.
Controllable Text-to-Image Generation
In this paper, we propose a novel controllable text-to-image generative adversarial network (ControlGAN), which can effectively synthesise high-quality images and also control parts of the image generation according to natural language descriptions.
Ranked #7 on
Text-to-Image Generation
on Multi-Modal-CelebA-HQ
Branch and Bound for Piecewise Linear Neural Network Verification
We use the data sets to conduct a thorough experimental comparison of existing and new algorithms and to provide an inclusive analysis of the factors impacting the hardness of verification problems.
Dual Graph Convolutional Network for Semantic Segmentation
Exploiting long-range contextual information is key for pixel-wise prediction tasks such as semantic segmentation.
Ranked #32 on
Semantic Segmentation
on Cityscapes test
Dynamic Graph Message Passing Networks
We propose a dynamic graph message passing network, that significantly reduces the computational complexity compared to related works modelling a fully-connected graph.
Real-Time Highly Accurate Dense Depth on a Power Budget using an FPGA-CPU Hybrid SoC
Obtaining highly accurate depth from stereo images in real time has many applications across computer vision and robotics, but in some contexts, upper bounds on power consumption constrain the feasible hardware to embedded platforms such as FPGAs.
A Signal Propagation Perspective for Pruning Neural Networks at Initialization
Alternatively, a recent approach shows that pruning can be done at initialization prior to training, based on a saliency criterion called connection sensitivity.
Stable Rank Normalization for Improved Generalization in Neural Networks and GANs
Exciting new work on the generalization bounds for neural networks (NN) given by Neyshabur et al. , Bartlett et al. closely depend on two parameter-depenedent quantities: the Lipschitz constant upper-bound and the stable rank (a softer version of the rank operator).
Ranked #122 on
Image Generation
on CIFAR-10
Hijacking Malaria Simulators with Probabilistic Programming
Epidemiology simulations have become a fundamental tool in the fight against the epidemics of various infectious diseases like AIDS and malaria.
Straight to Shapes++: Real-time Instance Segmentation Made More Accurate
The STS model can run at 35 FPS on a high-end desktop, but its accuracy is significantly worse than that of offline state-of-the-art methods.
Alpha MAML: Adaptive Model-Agnostic Meta-Learning
Model-agnostic meta-learning (MAML) is a meta-learning technique to train a model on a multitude of learning tasks in a way that primes the model for few-shot learning of new tasks.
GA-Net: Guided Aggregation Net for End-to-end Stereo Matching
In the stereo matching task, matching cost aggregation is crucial in both traditional methods and deep neural network models in order to accurately estimate disparities.
Ranked #4 on
Stereo Depth Estimation
on Spring
Deep Virtual Networks for Memory Efficient Inference of Multiple Tasks
To this end, we propose a novel network architecture producing multiple networks of different configurations, termed deep virtual networks (DVNs), for different tasks.
Learning to Adapt for Stereo
Real world applications of stereo depth estimation require models that are robust to dynamic variations in the environment.
On Tiny Episodic Memories in Continual Learning
But for a successful knowledge transfer, the learner needs to remember how to perform previous tasks.
Ranked #7 on
Class Incremental Learning
on cifar100
Domain Partitioning Network
Standard adversarial training involves two agents, namely a generator and a discriminator, playing a mini-max game.
The StarCraft Multi-Agent Challenge
In this paper, we propose the StarCraft Multi-Agent Challenge (SMAC) as a benchmark problem to fill this gap.
Ranked #6 on
SMAC
on SMAC 6h_vs_8z
3D Hand Shape and Pose from Images in the Wild
We present in this work the first end-to-end deep learning based method that predicts both 3D hand shape and pose from RGB images in the wild.
Ranked #10 on
3D Hand Pose Estimation
on FreiHAND
(PA-MPVPE metric)
Adversarial Metric Attack and Defense for Person Re-identification
However, our work observes the extreme vulnerability of existing distance metrics to adversarial examples, generated by simply adding human-imperceptible perturbations to person images.
Hypergraph Convolution and Hypergraph Attention
To efficiently learn deep embeddings on the high-order graph-structured data, we introduce two end-to-end trainable operators to the family of graph neural networks, i. e., hypergraph convolution and hypergraph attention.
Learn to Interpret Atari Agents
Deep reinforcement learning (DeepRL) agents surpass human-level performance in many tasks.
Visual Dialogue without Vision or Dialogue
We characterise some of the quirks and shortcomings in the exploration of Visual Dialogue - a sequential question-answering task where the questions and corresponding answers are related through given visual stimuli.
Fast Online Object Tracking and Segmentation: A Unifying Approach
In this paper we illustrate how to perform both visual object tracking and semi-supervised video object segmentation, in real-time, with a single simple approach.
Ranked #3 on
Visual Object Tracking
on YouTube-VOS 2018
Proximal Mean-field for Neural Network Quantization
Compressing large Neural Networks (NN) by quantizing the parameters, while maintaining the performance is highly desirable due to reduced memory and time complexity.
Meta Learning Deep Visual Words for Fast Video Object Segmentation
We address this problem by considering the task of video object segmentation.
Deeper Interpretability of Deep Networks
However, understanding the information represented and processed in CNNs remains in most cases challenging.
R$^3$SGM: Real-time Raster-Respecting Semi-Global Matching for Power-Constrained Systems
Stereo depth estimation is used for many computer vision applications.
Real-Time RGB-D Camera Pose Estimation in Novel Scenes using a Relocalisation Cascade
The adapted forests achieved relocalisation performance that was on par with that of offline forests, and our approach was able to estimate the camera pose in close to real time.
Multi-Agent Common Knowledge Reinforcement Learning
In this paper, we show that common knowledge between agents allows for complex decentralised coordination.
Multi-agent Reinforcement Learning
reinforcement-learning
+3
SNIP: Single-shot Network Pruning based on Connection Sensitivity
To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task.
Weakly- and Semi-Supervised Panoptic Segmentation
We present a weakly supervised model that jointly performs both semantic- and instance-segmentation -- a particularly relevant problem given the substantial cost of obtaining pixel-perfect annotation for these tasks.
Ranked #33 on
Panoptic Segmentation
on Cityscapes val
With Friends Like These, Who Needs Adversaries?
Via a novel experimental analysis, we illustrate some facts about deep convolutional networks for image classification that shed new light on their behaviour and how it connects to the problem of adversaries.
Value Propagation Networks
We present Value Propagation (VProp), a set of parameter-efficient differentiable planning modules built on Value Iteration which can successfully be trained using reinforcement learning to solve unseen tasks, has the capability to generalize to larger map sizes, and can learn to navigate in dynamic environments.
Efficient Relaxations for Dense CRFs with Sparse Higher Order Potentials
The presented algorithms can be applied to any labelling problem using a dense CRF with sparse higher-order potentials.
Meta-learning with differentiable closed-form solvers
The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data.
Robustness via Deep Low-Rank Representations
To achieve low dimensionality of learned representations, we propose an easy-to-use, end-to-end trainable, low-rank regularizer (LR) that can be applied to any intermediate layer representation of a DNN.
Learn To Pay Attention
We propose an end-to-end-trainable attention module for convolutional neural network (CNN) architectures built for image classification.
WebSeg: Learning Semantic Segmentation from Web Searches
In this paper, we improve semantic segmentation by automatically learning from Flickr images associated with a particular keyword, without relying on any explicit user annotations, thus substantially alleviating the dependence on accurate annotations when compared to previous weakly supervised methods.
Three Birds One Stone: A General Architecture for Salient Object Segmentation, Edge Detection and Skeleton Extraction
Although these tasks are inherently very different, we show that our unified approach performs very well on all of them and works far better than current single-purpose state-of-the-art methods.
Real-Time Dense Stereo Matching With ELAS on FPGA Accelerated Embedded Devices
For many applications in low-power real-time robotics, stereo cameras are the sensors of choice for depth perception as they are typically cheaper and more versatile than their active counterparts.
Devon: Deformable Volume Network for Learning Optical Flow
State-of-the-art neural network models estimate large displacement optical flow in multi-resolution and use warping to propagate the estimation between two resolutions.
FlipDial: A Generative Model for Two-Way Visual Dialogue
We are the first to extend this paradigm to full two-way visual dialogue, where our model is capable of generating both questions and answers in sequence based on a visual input, for which we propose a set of novel evaluation measures and metrics.
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
We observe that, in addition to forgetting, a known issue while preserving knowledge, IL also suffers from a problem we call intransigence, inability of a model to update its knowledge.
Collaborative Large-Scale Dense 3D Reconstruction with Online Inter-Agent Pose Optimisation
Reconstructing dense, volumetric models of real-world 3D scenes is important for many tasks, but capturing large scenes can take significant time, and the risk of transient changes to the scene goes up as the capture time increases.
Piecewise Linear Neural Networks verification: A comparative study
Motivated by the need of accelerating progress in this very important area, we investigate the trade-offs of a number of different approaches based on Mixed Integer Programming, Satisfiability Modulo Theory, as well as a novel method based on the Branch-and-Bound framework.
On the Robustness of Semantic Segmentation Models to Adversarial Attacks
Deep Neural Networks (DNNs) have demonstrated exceptional performance on most recognition tasks such as image classification and segmentation.
Learning to Compare: Relation Network for Few-Shot Learning
Once trained, a RN is able to classify images of new classes by computing relation scores between query images and the few examples of each new class without further updating the network.
A Unified View of Piecewise Linear Neural Network Verification
The success of Deep Learning and its potential use in many safety-critical applications has motivated research on formal verification of Neural Network (NN) models.
Holistic, Instance-Level Human Parsing
We address this problem by segmenting the parts of objects at an instance-level, such that each pixel in the image is assigned a part label, as well as the identity of the object it belongs to.
Ranked #2 on
Multi-Human Parsing
on PASCAL-Part
InfiniTAM v3: A Framework for Large-Scale 3D Reconstruction with Loop Closure
Representing the reconstruction volumetrically as a TSDF leads to most of the simplicity and efficiency that can be achieved with GPU implementations of these systems.
Spatio-temporal Human Action Localisation and Instance Segmentation in Temporally Untrimmed Videos
Current state-of-the-art human action recognition is focused on the classification of temporally trimmed videos in which only one action occurs per frame.
Discovering Class-Specific Pixels for Weakly-Supervised Semantic Segmentation
We propose an approach to discover class-specific pixels for the weakly-supervised semantic segmentation task.
Weakly supervised Semantic Segmentation
Weakly-Supervised Semantic Segmentation
Learning Disentangled Representations with Semi-Supervised Deep Generative Models
We propose to learn such representations using model architectures that generalise from standard VAEs, employing a general graphical model structure in the encoder and decoder.
End-to-end representation learning for Correlation Filter based tracking
The Correlation Filter is an algorithm that trains a linear template to discriminate between images and their translations.
Ranked #3 on
Visual Object Tracking
on OTB-50
DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents
DESIRE effectively predicts future locations of objects in multiple scenes by 1) accounting for the multi-modal nature of the future prediction (i. e., given the same context, future may vary), 2) foreseeing the potential future outcomes and make a strategic prediction based on that, and 3) reasoning not only from the past motion history, but also from the scene context as well as the interactions among the agents.
Ranked #1 on
Trajectory Prediction
on PAID
Multi-Agent Diverse Generative Adversarial Networks
Second, to enforce that different generators capture diverse high probability modes, the discriminator of MAD-GAN is designed such that along with finding the real and fake samples, it is also required to identify the generator that generated the given fake sample.
Pixelwise Instance Segmentation with a Dynamically Instantiated Network
This subnetwork uses the initial category-level segmentation, along with cues from the output of an object detector, within an end-to-end CRF to predict instances.
Ranked #10 on
Panoptic Segmentation
on Cityscapes test
Incremental Tube Construction for Human Action Detection
In this work, we introduce a real-time and online joint-labelling and association algorithm for action detection that can incrementally construct space-time action tubes on the most challenging action videos in which different action categories occur concurrently.
Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems.
On-the-Fly Adaptation of Regression Forests for Online Camera Relocalisation
Camera relocalisation is an important problem in computer vision, with applications in simultaneous localisation and mapping, virtual/augmented reality and navigation.
ROAM: a Rich Object Appearance Model with Application to Rotoscoping
Rotoscoping, the detailed delineation of scene elements through a video shot, is a painstaking task of tremendous importance in professional post-production pipelines.
Learning to superoptimize programs - Workshop Version
Superoptimization requires the estimation of the best program for a given computational task.
Playing Doom with SLAM-Augmented Deep Reinforcement Learning
A number of recent approaches to policy learning in 2D game domains have been successful going directly from raw input images to actions.
Efficient Linear Programming for Dense CRFs
To this end, we develop a proximal minimization framework, where the dual of each proximal problem is optimized via block coordinate descent.
Straight to Shapes: Real-time Detection of Encoded Shapes
To achieve this, we use a denoising convolutional auto-encoder to establish an embedding space, and place the decoder after a fast end-to-end network trained to regress directly to the encoded shape vectors.
Ranked #5 on
Semantic Contour Prediction
on Sbd val
Inducing Interpretable Representations with Variational Autoencoders
We develop a framework for incorporating structured graphical models in the \emph{encoders} of variational autoencoders (VAEs) that allows us to induce interpretable representations through approximate variational inference.
Learning to superoptimize programs
This approach involves repeated sampling of modifications to the program from a proposal distribution, which are accepted or rejected based on whether they preserve correctness, and the improvement they achieve.
Random Forests versus Neural Networks - What's Best for Camera Localization?
This work addresses the task of camera localization in a known 3D scene given a single input RGB image.
Fully-Trainable Deep Matching
Deep Matching (DM) is a popular high-quality method for quasi-dense image matching.
Bottom-up Instance Segmentation using Deep Higher-Order CRFs
Traditional Scene Understanding problems such as Object Detection and Semantic Segmentation have made breakthroughs in recent years due to the adoption of deep learning.
Efficient Continuous Relaxations for Dense CRF
In contrast to the continuous relaxation-based energy minimisation algorithms used for sparse CRFs, the mean-field algorithm fails to provide strong theoretical guarantees on the quality of its solutions.
Deep Learning for Detecting Multiple Space-Time Action Tubes in Videos
In stage 2, the appearance network detections are boosted by combining them with the motion detection scores, in proportion to their respective spatial overlap.
Fully-Convolutional Siamese Networks for Object Tracking
The problem of arbitrary object tracking has traditionally been tackled by learning a model of the object's appearance exclusively online, using as sole training data the video itself.
Ranked #3 on
Visual Object Tracking
on OTB-50
Learning feed-forward one-shot learners
In this paper, we propose a method to learn the parameters of a deep model in one shot.
Adaptive Neural Compilation
We show that it is possible to compile programs written in a low-level language to a differentiable representation.
Recurrent Instance Segmentation
Instance segmentation is the problem of detecting and delineating each distinct object of interest appearing in an image.
Joint Training of Generic CNN-CRF Models with Stochastic Optimization
We propose a new CNN-CRF end-to-end learning framework, which is based on joint stochastic optimization with respect to both Convolutional Neural Network (CNN) and Conditional Random Field (CRF) parameters.
Sequential Optimization for Efficient High-Quality Object Proposal Generation
We propose a novel object proposal algorithm, BING++, which inherits the virtue of good computational efficiency of BING but significantly improves its proposal localization quality.
SemanticPaint: A Framework for the Interactive Segmentation of 3D Scenes
We present an open-source, real-time implementation of SemanticPaint, a system for geometric reconstruction, object-class segmentation and learning of 3D scenes.
An embarrassingly simple approach to zero-shot learning
Zero-shot learning consists in learning how to recognise new concepts by just having a description of them.
Ranked #8 on
Zero-Shot Action Recognition
on Olympics
Exploiting Uncertainty in Regression Forests for Accurate Camera Relocalization
Recent advances in camera relocalization use predictions from a regression forest to guide the camera pose optimization procedure.
Target Identity-Aware Network Flow for Online Multiple Target Tracking
In this paper we show that multiple object tracking (MOT) can be formulated in a framework, where the detection and data-association are performed simultaneously.
Conditional Random Fields as Recurrent Neural Networks
Pixel-level labelling tasks, such as semantic segmentation, play a central role in image understanding.
Ranked #36 on
Semantic Segmentation
on PASCAL VOC 2012 test
Large-scale Binary Quadratic Optimization Using Semidefinite Relaxation and Applications
Two standard relaxation methods are widely used for solving general BQPs--spectral methods and semidefinite programming (SDP), each with their own advantages and disadvantages.
A Framework for the Volumetric Integration of Depth Images
Along with the framework we also provide a set of components for scalable reconstruction: two implementations of camera trackers, based on RGB data and on depth data, two representations of the 3D volumetric data, a dense volume and one based on hashes of subblocks, and an optional module for swapping subblocks in and out of the typically limited GPU memory.
