Search Results for author:
Found 101 papers, 44 papers with code
Temporally Grounding Instructional Diagrams in Unconstrained Videos
In this paper, we tackle this issue by simultaneously grounding a sequence of step diagrams.
Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation
We evaluate our segmentation approach and unsupervised learning pipeline on the Breakfast, 50-Salads, YouTube Instructions and Desktop Assembly datasets, yielding state-of-the-art results for the unsupervised video action segmentation task.
The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?
This study uses linear probing to shed light on the hidden knowledge at the output layers of LVLMs.
An Empirical Study Into What Matters for Calibrating Vision-Language Models
Vision-Language Models (VLMs) have emerged as the dominant approach for zero-shot recognition, adept at handling diverse scenarios and significant distribution changes.
Towards Optimal Feature-Shaping Methods for Out-of-Distribution Detection
Feature shaping refers to a family of methods that exhibit state-of-the-art performance for out-of-distribution (OOD) detection.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
Differentiable Neural Surface Refinement for Modeling Transparent Objects
Neural implicit surface reconstruction leveraging volume rendering has led to significant advances in multi-view reconstruction.
Small Steps and Level Sets: Fitting Neural Surface Models with Point Guidance
We circumvent this problem by introducing guiding points and use them to steer the optimization towards the true shape via small incremental changes for which the loss formulation has a good descent direction.
Learning to Select Views for Efficient Multi-View Understanding
Inspired by this we study selective view pipelining for efficient multi-view understanding which breaks computation of multiple views into steps and only computes the most helpful views/steps in a parallel manner for the best efficiency.
Zero-shot Retrieval: Augmenting Pre-trained Models with Search Engines
Large pre-trained models can dramatically reduce the amount of task-specific data required to solve a problem, but they often fail to capture domain-specific nuances out of the box.
3D-GPT: Procedural 3D Modeling with Large Language Models
Our work highlights the potential of LLMs in 3D modeling, offering a basic framework for future advancements in scene generation and animation.
Exploring Predicate Visual Context in Detecting Human-Object Interactions
Recently, the DETR framework has emerged as the dominant approach for human--object interaction (HOI) research.
Ranked #2 on
Human-Object Interaction Detection
on HICO-DET
Scaling Data Generation in Vision-and-Language Navigation
Recent research in language-guided visual navigation has demonstrated a significant demand for the diversity of traversable environments and the quantity of supervision for training generalizable agents.
Learning Navigational Visual Representations with Semantic Map Supervision
Being able to perceive the semantics and the spatial structure of the environment is essential for visual navigation of a household robot.
PMaF: Deep Declarative Layers for Principal Matrix Features
We explore two differentiable deep declarative layers, namely least squares on sphere (LESS) and implicit eigen decomposition (IED), for learning the principal matrix features (PMaF).
Towards Understanding Gradient Approximation in Equality Constrained Deep Declarative Networks
We explore conditions for when the gradient of a deep declarative node can be approximated by ignoring constraint terms and still result in a descent direction for the global loss function.
Candidate Set Re-ranking for Composed Image Retrieval with Dual Multi-modal Encoder
An alternative approach is to allow interactions between the query and every possible candidate, i. e., reference-text-candidate triplets, and pick the best from the entire set.
Ranked #3 on
Image Retrieval
on CIRR
GoferBot: A Visual Guided Human-Robot Collaborative Assembly System
It is composed of a visual servoing module that reaches and grasps assembly parts in an unstructured multi-instance and dynamic environment, an action recognition module that performs human action prediction for implicit communication, and a visual handover module that uses the perceptual understanding of human behaviour to produce an intuitive and efficient collaborative assembly experience.
Adaptive Cross Batch Normalization for Metric Learning
In particular, it is necessary to circumvent the representational drift between the accumulated embeddings and the feature embeddings at the current training iteration as the learnable parameters are being updated.
Bi-directional Training for Composed Image Retrieval via Text Prompt Learning
Composed image retrieval searches for a target image based on a multi-modal user query comprised of a reference image and modification text describing the desired changes.
Ranked #8 on
Image Retrieval
on Fashion IQ
Aligning Step-by-Step Instructional Diagrams to Video Demonstrations
In this paper, we consider a novel setting where such an alignment is between (i) instruction steps that are depicted as assembly diagrams (commonly seen in Ikea assembly manuals) and (ii) video segments from in-the-wild videos; these videos comprising an enactment of the assembly actions in the real world.
Deep Declarative Dynamic Time Warping for End-to-End Learning of Alignment Paths
An interesting byproduct of this formulation is that DecDTW outputs the optimal warping path between two time series as opposed to a soft approximation, recoverable from Soft-DTW.
3DInAction: Understanding Human Actions in 3D Point Clouds
We propose a novel method for 3D point cloud action recognition.
Learning to Select Camera Views: Efficient Multiview Understanding at Few Glances
Multiview camera setups have proven useful in many computer vision applications for reducing ambiguities, mitigating occlusions, and increasing field-of-view coverage.
Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation
This work aims to assess how well a model performs under distribution shifts without using labels.
Octree Guided Unoriented Surface Reconstruction
We propose a two-step approach, OG-INR, where we (1) construct a discrete octree and label what is inside and outside (2) optimize for a continuous and high-fidelity shape using an INR that is initially guided by the octree's labelling.
Semi-Supervised Semantic Segmentation under Label Noise via Diverse Learning Groups
Each learning group consists of a teacher network, a student network and a novel filter module.
Understanding and Improving the Role of Projection Head in Self-Supervised Learning
Most current contrastive learning approaches append a parametrized projection head to the end of some backbone network to optimize the InfoNCE objective and then discard the learned projection head after training.
NeRFEditor: Differentiable Style Decomposition for Full 3D Scene Editing
Specifically, we use a NeRF model to generate numerous image-angle pairs to train an adjustor, which can adjust the StyleGAN latent code to generate high-fidelity stylized images for any given angle.
High-Fidelity Guided Image Synthesis with Latent Diffusion Models
The user scribbles control the color composition while the text prompt provides control over the overall image semantics.
Learning to Structure an Image with Few Colors and Beyond
To this end, we propose a color quantization network, ColorCNN, which learns to structure an image in limited color spaces by minimizing the classification loss.
Multi-View Correlation Consistency for Semi-Supervised Semantic Segmentation
In this paper, we take the best of both worlds and propose multi-view correlation consistency (MVCC) learning: it considers rich pairwise relationships in self-correlation matrices and matches them across views to provide robust supervision.
On the Strong Correlation Between Model Invariance and Generalization
Generalization and invariance are two essential properties of any machine learning model.
Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation
To bridge the discrete-to-continuous gap, we propose a predictor to generate a set of candidate waypoints during navigation, so that agents designed with high-level actions can be transferred to and trained in continuous environments.
Exploiting Problem Structure in Deep Declarative Networks: Two Case Studies
Deep declarative networks and other recent related works have shown how to differentiate the solution map of a (continuous) parametrized optimization problem, opening up the possibility of embedding mathematical optimization problems into end-to-end learnable models.
DiGS: Divergence Guided Shape Implicit Neural Representation for Unoriented Point Clouds
In this paper, we propose a divergence guided shape representation learning approach that does not require normal vectors as input.
Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer
Recent developments in transformer models for visual data have led to significant improvements in recognition and detection tasks.
Ranked #10 on
Human-Object Interaction Detection
on V-COCO
A Regularized Wasserstein Framework for Graph Kernels
This framework provides a novel optimal transport distance metric, namely Regularized Wasserstein (RW) discrepancy, which can preserve both features and structure of graphs via Wasserstein distances on features and their local variations, local barycenters and global connectivity.
Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models
We demonstrate that with a relatively simple architecture, CIRPLANT outperforms existing methods on open-domain images, while matching state-of-the-art accuracy on the existing narrow datasets, such as fashion.
Ranked #12 on
Image Retrieval
on CIRR
DiGS : Divergence guided shape implicit neural representation for unoriented point clouds
In this paper, we propose a divergence guided shape representation learning approach that does not require normal vectors as input.
VLN BERT: A Recurrent Vision-and-Language BERT for Navigation
In this paper we propose a recurrent BERT model that is time-aware for use in VLN.
What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?
In this work, we train semantic classification and rotation prediction in a multi-task way.
Semantics for Robotic Mapping, Perception and Interaction: A Survey
In robotics and related research fields, the study of understanding is often referred to as semantics, which dictates what does the world "mean" to a robot, and is strongly tied to the question of how to represent that meaning.
Fine-grained Classification via Categorical Memory Networks
The memory module stores the prototypical feature representation for each category as a moving average.
Ranked #51 on
Fine-Grained Image Classification
on CUB-200-2011
Spatially Conditioned Graphs for Detecting Human-Object Interactions
We address the problem of detecting human-object interactions in images using graphical neural networks.
Ranked #18 on
Human-Object Interaction Detection
on V-COCO
(using extra training data)
Probabilistic Tracklet Scoring and Inpainting for Multiple Object Tracking
Despite the recent advances in multiple object tracking (MOT), achieved by joint detection and tracking, dealing with long occlusions remains a challenge.
A Recurrent Vision-and-Language BERT for Navigation
In this paper we propose a recurrent BERT model that is time-aware for use in VLN.
Ranked #7 on
Visual Navigation
on R2R
Rethinking conditional GAN training: An approach using geometrically structured latent manifolds
Conditional GANs (cGAN), in their rudimentary form, suffer from critical drawbacks such as the lack of diversity in generated outputs and distortion between the latent and output manifolds.
Language and Visual Entity Relationship Graph for Agent Navigation
From both the textual and visual perspectives, we find that the relationships among the scene, its objects, and directional clues are essential for the agent to interpret complex instructions and correctly perceive the environment.
DORi: Discovering Object Relationship for Moment Localization of a Natural-Language Query in Video
This paper studies the task of temporal moment localization in a long untrimmed video using natural language query.
Conditional Generative Modeling via Learning the Latent Space
Although deep learning has achieved appealing results on several machine learning tasks, most of the models are deterministic at inference, limiting their application to single-modal settings.
Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization
We instead propose the first fully end-to-end trainable network for solving the blind PnP problem efficiently and globally, that is, without the need for pose priors.
Multiview Detection with Feature Perspective Transformation
First, how should we aggregate cues from the multiple views?
Ranked #5 on
Multiview Detection
on MultiviewX
The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose
The availability of a large labeled dataset is a key requirement for applying deep learning methods to solve various computer vision tasks.
Bidirectionally Self-Normalizing Neural Networks
The problem of vanishing and exploding gradients has been a long-standing obstacle that hinders the effective training of neural networks.
A Multi-modal Approach to Fine-grained Opinion Mining on Video Reviews
Despite the recent advances in opinion mining for written reviews, few works have tackled the problem on other sources of reviews.
Inferring Temporal Compositions of Actions Using Probabilistic Automata
This paper presents a framework to recognize temporal compositions of atomic actions in videos.
ArTIST: Autoregressive Trajectory Inpainting and Scoring for Tracking
One of the core components in online multiple object tracking (MOT) frameworks is associating new detections with existing tracklets, typically done via a scoring function.
Sub-Instruction Aware Vision-and-Language Navigation
Vision-and-language navigation requires an agent to navigate through a real 3D environment following natural language instructions.
DeepFit: 3D Surface Fitting via Neural Network Weighted Least Squares
We propose a surface fitting method for unstructured 3D point clouds.
Ranked #6 on
Surface Normals Estimation
on PCPNet
Learning to Structure an Image with Few Colors
Color and structure are the two pillars that construct an image.
Joint Unsupervised Learning of Optical Flow and Egomotion with Bi-Level Optimization
We address the problem of joint optical flow and camera motion estimation in rigid scenes by incorporating geometric constraints into an unsupervised deep learning framework.
Contextually Plausible and Diverse 3D Human Motion Prediction
We tackle the task of diverse 3D human motion prediction, that is, forecasting multiple plausible future 3D poses given a sequence of observed 3D poses.
Spectral-GANs for High-Resolution 3D Point-cloud Generation
Point-clouds are a popular choice for vision and graphics tasks due to their accurate shape description and direct acquisition from range-scanners.
Representation Learning on Unit Ball with 3D Roto-Translational Equivariance
In this work, we propose a novel `\emph{volumetric convolution}' operation that can effectively model and convolve arbitrary functions in $\mathbb{B}^3$.
Deep Declarative Networks: A New Hope
We show how these declarative processing nodes can be implemented in the popular PyTorch deep learning software library allowing declarative and imperative nodes to co-exist within the same network.
Blended Convolution and Synthesis for Efficient Discrimination of 3D Shapes
Existing networks directly learn feature representations on 3D point clouds for shape analysis.
Proposal-free Temporal Moment Localization of a Natural-Language Query in Video using Guided Attention
Given an untrimmed video and a sentence as the query, the goal is to determine the starting, and the ending, of the relevant visual moment in the video, that corresponds to the query sentence.
Learning Variations in Human Motion via Mix-and-Match Perturbation
In this paper, we introduce an approach to stochastically combine the root of variations with previous pose information, which forces the model to take the noise into account.
A Signal Propagation Perspective for Pruning Neural Networks at Initialization
Alternatively, a recent approach shows that pruning can be done at initialization prior to training, based on a saliency criterion called connection sensitivity.
Learning to Find Common Objects Across Few Image Collections
We model the selection as an energy minimization problem with unary and pairwise potential functions.
The Alignment of the Spheres: Globally-Optimal Spherical Mixture Alignment for Camera Pose Estimation
Determining the position and orientation of a calibrated camera from a single image with respect to a 3D model is an essential task for many applications.
Partially-Supervised Image Captioning
To address this problem, we teach image captioning models new visual concepts from labeled images and object detection datasets.
Non-Linear Temporal Subspace Representations for Activity Recognition
As these features are often non-linear, we propose a novel pooling method, kernelized rank pooling, that represents a given sequence compactly as the pre-image of the parameters of a hyperplane in a reproducing kernel Hilbert space, projections of data onto which captures their temporal order.
Video Representation Learning Using Discriminative Pooling
In an attempt to tackle this problem, we propose discriminative pooling, based on the notion that among the deep features generated on all short clips, there is at least one that characterizes the action.
Neural Algebra of Classifiers
In this paper, we build on the compositionality principle and develop an "algebra" to compose classifiers for complex visual concepts.
Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments
This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering.
Ranked #10 on
Visual Navigation
on R2R
Human Action Forecasting by Learning Task Grammars
For effective human-robot interaction, it is important that a robotic assistant can forecast the next action a human will consider in a given task.
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning.
Ranked #29 on
Visual Question Answering (VQA)
on VQA v2 test-std
Human Pose Forecasting via Deep Markov Models
Human pose forecasting is an important problem in computer vision with applications to human-robot interaction, visual surveillance, and autonomous driving.
Incorporating Network Built-in Priors in Weakly-supervised Semantic Segmentation
We then show how to obtain multi-class masks by the fusion of foreground/background ones with information extracted from a weakly-supervised localization network.
Discriminatively Learned Hierarchical Rank Pooling Networks
First, we present "discriminative rank pooling" in which the shared weights of our video representation and the parameters of the action classifiers are estimated jointly for a given training dataset of labelled vector sequences using a bilevel optimization formulation of the learning problem.
Second-order Temporal Pooling for Action Recognition
We also propose higher-order extensions of this scheme by computing correlations after embedding the CNN features in a reproducing kernel Hilbert space.
DeepPermNet: Visual Permutation Learning
Unrolling these iterations in a Sinkhorn network layer, we propose DeepPermNet, an end-to-end CNN model for this task.
Generalized Rank Pooling for Activity Recognition
Most popular deep models for action recognition split video sequences into short sub-sequences consisting of a few frames; frame-based features are then pooled for recognizing the activity.
Action Representation Using Classifier Decision Boundaries
Applying multiple instance learning in an SVM setup, we use the parameters of this separating hyperplane as a descriptor for the video.
Higher-order Pooling of CNN Features via Kernel Linearization for Action Recognition
The HOK descriptors are then generated from the higher-order co-occurrences of these feature maps, and are then used as input to a video-level classifier.
Guided Open Vocabulary Image Captioning with Constrained Beam Search
Existing image captioning models do not generalize well to out-of-domain images containing novel scenes or objects.
Unsupervised Human Action Detection by Action Matching
On the MPII Cooking dataset we detect action segments with a precision of 21. 6% and recall of 11. 7% over 946 long video pairs and over 5000 ground truth action segments.
Self-Supervised Video Representation Learning With Odd-One-Out Networks
On action classification, our method obtains 60. 3\% on the UCF101 dataset using only UCF101 data for training which is approximately 10% better than current state-of-the-art self-supervised learning methods.
Ranked #47 on
Self-Supervised Action Recognition
on UCF101
Built-in Foreground/Background Prior for Weakly-Supervised Semantic Segmentation
Hence, weak supervision using only image tags could have a significant impact in semantic segmentation.
SPICE: Semantic Propositional Image Caption Evaluation
There is considerable interest in the task of automatically generating image captions.
On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization
Some recent works in machine learning and computer vision involve the solution of a bi-level optimization problem.
Dynamic Image Networks for Action Recognition
We introduce the concept of dynamic image, a novel compact representation of videos useful for video analysis especially when convolutional neural networks (CNNs) are used.
Ranked #62 on
Action Recognition
on HMDB-51
Discriminative Hierarchical Rank Pooling for Activity Recognition
We present hierarchical rank pooling, a video sequence encoding method for activity recognition.
Hierarchical Higher-Order Regression Forest Fields: An Application to 3D Indoor Scene Labelling
Specifically, we relax the labelling problem to a regression, and generalize the higher-order associative P n Potts model to a new family of arbitrary higher-order models based on regression forests.
Multi-Target Tracking with Time-Varying Clutter Rate and Detection Profile: Application to Time-lapse Cell Microscopy Sequences
To deal with this, we propose a bootstrap filter composed of an estimator and a tracker.
Deep CNN Ensemble with Data Augmentation for Object Detection
We report on the methods used in our recent DeepEnsembleCoco submission to the PASCAL VOC 2012 challenge, which achieves state-of-the-art performance on the object detection task.
An Exemplar-based CRF for Multi-instance Object Segmentation
We address the problem of joint detection and segmentation of multiple object instances in an image, a key step towards scene understanding.
Region-based Segmentation and Object Detection
Object detection and multi-class image segmentation are two closely related tasks that can be greatly improved when solved jointly by feeding information from one task to the other.
Cascaded Classification Models: Combining Models for Holistic Scene Understanding
We demonstrate the effectiveness of our method on a large set of natural images by combining the subtasks of scene categorization, object detection, multiclass image segmentation, and 3d scene reconstruction.
Learning Bounded Treewidth Bayesian Networks
In this work we present a novel method for learning Bayesian networks of bounded treewidth that employs global structure modifications and that is polynomial in the size of the graph and the treewidth bound.
