Search Results for author:
Found 104 papers, 70 papers with code
Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning
Neural discrete reasoning (NDR) has shown remarkable progress in combining deep models with discrete reasoning.
Tuning Multi-mode Token-level Prompt Alignment across Modalities
To address the limitation, we propose a multi-mode token-level tuning framework that leverages the optimal transportation to learn and align a set of prompt tokens across modalities.
Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation
Domain Adaptation (DA) is always challenged by the spurious correlation between domain-invariant features (e. g., class identity) and domain-specific features (e. g., environment) that does not generalize to the target domain.
Towards Debiasing Frame Length Bias in Text-Video Retrieval via Causal Intervention
Many studies focus on improving pretraining or developing new backbones in text-video retrieval.
Empowering Dynamics-aware Text-to-Video Diffusion with Large Language Models
In this work, we investigate strengthening the awareness of video dynamics for DMs, for high-quality T2V generation.
Invariant Training 2D-3D Joint Hard Samples for Few-Shot Point Cloud Recognition
We tackle the data scarcity challenge in few-shot point cloud recognition of 3D objects by using a joint prediction from a conventional 3D model and a well-trained 2D model.
Empowering Vision-Language Models to Follow Interleaved Vision-Language Instructions
To address this issue, we propose a generic and lightweight controllable knowledge re-injection module, which utilizes the sophisticated reasoning ability of LLMs to control the VPG to conditionally extract instruction-specific visual information and re-inject it into the LLM.
Random Boxes Are Open-world Object Detectors
First, as the randomization is independent of the distribution of the limited known objects, the random proposals become the instrumental variable that prevents the training from being confounded by the known objects.
DisCo: Disentangled Control for Referring Human Dance Generation in Real World
Generative AI has made significant strides in computer vision, particularly in image/video synthesis conditioned on text descriptions.
Fast Diffusion Model
To mitigate this, we propose a Fast Diffusion Model (FDM) which improves the diffusion process of DMs from a stochastic optimization perspective to speed up both training and sampling.
An Overview of Challenges in Egocentric Text-Video Retrieval
Text-video retrieval contains various challenges, including biases coming from diverse sources.
Decoupled Kullback-Leibler Divergence Loss
In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and observe that it is equivalent to the Doupled Kullback-Leibler (DKL) Divergence loss that consists of 1) a weighted Mean Square Error (wMSE) loss and 2) a Cross-Entropy loss incorporating soft labels.
Equivariant Similarity for Vision-Language Foundation Models
Unlike the existing image-text similarity objective which only categorizes matched pairs as similar and unmatched pairs as dissimilar, equivariance also requires similarity to vary faithfully according to the semantic changes.
Unbiased Multiple Instance Learning for Weakly Supervised Video Anomaly Detection
At each MIL training iteration, we use the current detector to divide the samples into two groups with different context biases: the most confident abnormal/normal snippets and the rest ambiguous ones.
Semantic Scene Completion with Cleaner Self
SSC is a well-known ill-posed problem as the prediction model has to "imagine" what is behind the visible surface, which is usually represented by Truncated Signed Distance Function (TSDF).
Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection
Without bells and whistles, our RePro achieves a new state-of-the-art performance on two VidVRD benchmarks of not only the base training object and predicate categories, but also the unseen ones.
Debiased Fine-Tuning for Vision-language Models by Prompt Regularization
We present a new paradigm for fine-tuning large-scale visionlanguage pre-trained models on downstream task, dubbed Prompt Regularization (ProReg).
Adaptively Clustering Neighbor Elements for Image Captioning
We design a novel global-local Transformer named \textbf{Ada-ClustFormer} (\textbf{ACF}) to generate captions.
Learning Trajectory-Word Alignments for Video-Language Tasks
To amend this, we propose a novel TW-BERT to learn Trajectory-Word alignment by a newly designed trajectory-to-word (T2W) attention for solving video-language tasks.
Bootstrap Your Own Prior: Towards Distribution-Agnostic Novel Class Discovery
Novel Class Discovery (NCD) aims to discover unknown classes without any annotation, by exploiting the transferable knowledge already learned from a base set of known classes.
Mitigating and Evaluating Static Bias of Action Representations in the Background and the Foreground
In video action recognition, shortcut static features can interfere with the learning of motion features, resulting in poor out-of-distribution (OOD) generalization.
Attention-based Class Activation Diffusion for Weakly-Supervised Semantic Segmentation
An intuitive solution is ``coupling'' the CAM with the long-range attention matrix of visual transformers (ViT) We find that the direct ``coupling'', e. g., pixel-wise multiplication of attention and activation, achieves a more global coverage (on the foreground), but unfortunately goes with a great increase of false positives, i. e., background pixels are mistakenly included.
Weakly supervised Semantic Segmentation
Weakly-Supervised Semantic Segmentation
Respecting Transfer Gap in Knowledge Distillation
The network response serves as additional supervision to formulate the machine domain, which uses the data collected from the human domain as a transfer set.
Learning to Collocate Visual-Linguistic Neural Modules for Image Captioning
This is because the language is only partially observable, for which we need to dynamically collocate the modules during the process of image captioning.
Class Is Invariant to Context and Vice Versa: On Learning Invariance for Out-Of-Distribution Generalization
If the context in every class is evenly distributed, OOD would be trivial because the context can be easily removed due to an underlying principle: class is invariant to context.
NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation
To this end, we propose a novel NoIsy label CorrEction and Sample Training strategy for SGG: NICEST.
Identifying Hard Noise in Long-Tailed Sample Distribution
Such imbalanced training data makes a classifier less discriminative for the tail classes, whose previously "easy" noises are now turned into "hard" ones -- they are almost as outliers as the clean tail samples.
Equivariance and Invariance Inductive Bias for Learning from Insufficient Data
We are interested in learning robust models from insufficient data, without the need for any externally pre-trained checkpoints.
Invariant Feature Learning for Generalized Long-Tailed Classification
In fact, even if the class is balanced, samples within each class may still be long-tailed due to the varying attributes.
Ranked #1 on
Long-tail Learning
on ImageNet-GLT
On Non-Random Missing Labels in Semi-Supervised Learning
Our method is three-fold: 1) We propose Class-Aware Propensity (CAP) that exploits the unlabeled data to train an improved classifier using the biased labeled data.
Exploiting Semantic Role Contextualized Video Features for Multi-Instance Text-Video Retrieval EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022
In this report, we present our approach for EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022.
Ranked #7 on
Multi-Instance Retrieval
on EPIC-KITCHENS-100
RoME: Role-aware Mixture-of-Expert Transformer for Text-to-Video Retrieval
Most methods consider only one joint embedding space between global visual and textual features without considering the local structures of each modality.
Ranked #12 on
Video Retrieval
on YouCook2
Prompt-aligned Gradient for Prompt Tuning
Thanks to the large pre-trained vision-language models (VLMs) like CLIP, we can craft a zero-shot classifier by "prompt", e. g., the confidence score of an image being "[CLASS]" can be obtained by using the VLM provided similarity measure between the image and the prompt sentence "a photo of a [CLASS]".
Certified Robustness Against Natural Language Attacks by Causal Intervention
Deep learning models have achieved great success in many fields, yet they are vulnerable to adversarial examples.
Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation
Specifically, due to the sum-over-class pooling nature of BCE, each pixel in CAM may be responsive to multiple classes co-occurring in the same receptive field.
Weakly supervised Semantic Segmentation
Weakly-Supervised Semantic Segmentation
Deconfounded Visual Grounding
We focus on the confounding bias between language and location in the visual grounding pipeline, where we find that the bias is the major visual reasoning bottleneck.
Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification
We address the overlooked unbiasedness in existing long-tailed classification methods: we find that their overall improvement is mostly attributed to the biased preference of tail over head, as the test distribution is assumed to be balanced; however, when the test is as imbalanced as the long-tailed training data -- let the test respect Zipf's law of nature -- the tail bias is no longer beneficial overall because it hurts the head majorities.
How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?
The fine-tuning of pre-trained language models has a great success in many NLP fields.
Introspective Distillation for Robust Question Answering
Question answering (QA) models are well-known to exploit data bias, e. g., the language prior in visual QA and the position bias in reading comprehension.
Self-Supervised Learning Disentangled Group Representation as Feature
A good visual representation is an inference map from observations (images) to features (vectors) that faithfully reflects the hidden modularized generative factors (semantics).
Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering
Specifically, CSST is composed of two parts: Counterfactual Samples Synthesizing (CSS) and Counterfactual Samples Training (CST).
Auto-Parsing Network for Image Captioning and Visual Question Answering
We propose an Auto-Parsing Network (APN) to discover and exploit the input data's hidden tree structures for improving the effectiveness of the Transformer-based vision-language systems.
Causal Attention for Unbiased Visual Recognition
Attention module does not always help deep models learn causal features that are robust in any confounding context, e. g., a foreground object feature is invariant to different backgrounds.
Are Missing Links Predictable? An Inferential Benchmark for Knowledge Graph Completion
We present InferWiki, a Knowledge Graph Completion (KGC) dataset that improves upon existing benchmarks in inferential ability, assumptions, and patterns.
Transporting Causal Mechanisms for Unsupervised Domain Adaptation
However, the theoretical solution provided by transportability is far from practical for UDA, because it requires the stratification and representation of the unobserved confounder that is the cause of the domain gap.
Adversarial Visual Robustness by Causal Intervention
As these visual confounders are imperceptible in general, we propose to use the instrumental variable that achieves causal intervention without the need for confounder observation.
Empowering Language Understanding with Counterfactual Reasoning
Present language understanding methods have demonstrated extraordinary ability of recognizing patterns in texts via machine learning.
VL-NMS: Breaking Proposal Bottlenecks in Two-Stage Visual-Language Matching
In this paper, we argue that these methods overlook an obvious \emph{mismatch} between the roles of proposals in the two stages: they generate proposals solely based on the detection confidence (i. e., query-agnostic), hoping that the proposals contain all instances mentioned in the text query (i. e., query-aware).
TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph
Multi-hop Question Answering (QA) is a challenging task because it requires precise reasoning with entity relations at every step towards the answer.
The Blessings of Unlabeled Background in Untrimmed Videos
The key challenge is how to distinguish the action of interest segments from the background, which is unlabelled even on the video-level.
Weakly-supervised Temporal Action Localization
Weakly Supervised Temporal Action Localization
Causal Attention for Vision-Language Tasks
Specifically, CATT is implemented as a combination of 1) In-Sample Attention (IS-ATT) and 2) Cross-Sample Attention (CS-ATT), where the latter forcibly brings other samples into every IS-ATT, mimicking the causal intervention.
Distilling Causal Effect of Data in Class-Incremental Learning
We propose a causal framework to explain the catastrophic forgetting in Class-Incremental Learning (CIL) and then derive a novel distillation method that is orthogonal to the existing anti-forgetting techniques, such as data replay and feature/label distillation.
Counterfactual Zero-Shot and Open-Set Visual Recognition
We show that the key reason is that the generation is not Counterfactual Faithful, and thus we propose a faithful one, whose generation is from the sample-specific counterfactual question: What would the sample look like, if we set its class attribute to a certain class, while keeping its sample attribute unchanged?
Hierarchical Scene Graph Encoder-Decoder for Image Paragraph Captioning
We propose irredundant attention in SSG-RNN to improve the possibility of abstracting topics from rarely described sub-graphs and inheriting attention in WSG-RNN to generate more grounded sentences with the abstracted topics, both of which give rise to more distinctive paragraphs.
Interventional Few-Shot Learning
Specifically, we develop three effective IFSL algorithmic implementations based on the backdoor adjustment, which is essentially a causal intervention towards the SCM of many-shot learning: the upper-bound of FSL in a causal view.
Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect
On one hand, it has a harmful causal effect that misleads the tail prediction biased towards the head.
Ranked #34 on
Long-tail Learning
on CIFAR-10-LT (ρ=10)
Causal Intervention for Weakly-Supervised Semantic Segmentation
We present a causal inference framework to improve Weakly-Supervised Semantic Segmentation (WSSS).
Ranked #26 on
Weakly-Supervised Semantic Segmentation
on COCO 2014 val
Clicks can be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue
However, we argue that there is a significant gap between clicks and user satisfaction -- it is common that a user is "cheated" to click an item by the attractive title/cover of the item.
Ref-NMS: Breaking Proposal Bottlenecks in Two-Stage Referring Expression Grounding
The prevailing framework for solving referring expression grounding is based on a two-stage process: 1) detecting proposals with an object detector and 2) grounding the referent to one of the proposals.
Feature Pyramid Transformer
Yet, the non-local spatial interactions are not across scales, and thus they fail to capture the non-local contexts of objects (or parts) residing in different scales.
KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base
To this end, we introduce KQA Pro, a dataset for Complex KBQA including ~120K diverse natural language questions.
Counterfactual VQA: A Cause-Effect Look at Language Bias
VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language.
Iterative Context-Aware Graph Inference for Visual Dialog
Visual dialog is a challenging task that requires the comprehension of the semantic dependencies among implicit visual and textual contexts.
Ranked #12 on
Visual Dialog
on VisDial v0.9 val
Learning to Segment the Tail
Real-world visual recognition requires handling the extreme sample imbalance in large-scale long-tailed data.
More Grounded Image Captioning by Distilling Image-Text Matching Model
To improve the grounding accuracy while retaining the captioning quality, it is expensive to collect the word-region alignment as strong supervision.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
To reduce the language biases, several recent works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on VQA-CP.
Ranked #1 on
Visual Question Answering (VQA)
on VQA-CP
(using extra training data)
Deconfounded Image Captioning: A Causal Retrospect
The dataset bias in vision-language tasks is becoming one of the main problems that hinder the progress of our community.
Cross-GCN: Enhancing Graph Convolutional Network with $k$-Order Feature Interactions
Graph Convolutional Network (GCN) is an emerging technique that performs learning and reasoning on graph data.
Visual Commonsense R-CNN
We present a novel unsupervised feature representation learning method, Visual Commonsense Region-based Convolutional Neural Network (VC R-CNN), to serve as an improved visual region encoder for high-level tasks such as captioning and VQA.
Ranked #23 on
Image Captioning
on COCO Captions
Unbiased Scene Graph Generation from Biased Training
Today's scene graph generation (SGG) task is still far from practical, mainly due to the severe training bias, e. g., collapsing diverse "human walk on / sit on / lay on beach" into "human on beach".
Ranked #1 on
Scene Graph Generation
on Visual Genome
General Partial Label Learning via Dual Bipartite Graph Autoencoder
Compared to the traditional Partial Label Learning (PLL) problem, GPLL relaxes the supervision assumption from instance-level -- a label set partially labels an instance -- to group-level: 1) a label set partially labels a group of instances, where the within-group instance-label link annotations are missing, and 2) cross-group links are allowed -- instances in a group may be partially linked to the label set from another group.
Ranked #1 on
Partial Label Learning
on MPII Movie Description
Two Causal Principles for Improving Visual Dialog
This paper unravels the design tricks adopted by us, the champion team MReaL-BDAI, for Visual Dialog Challenge 2019: two causal principles for improving Visual Dialog (VisDial).
Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions
Specifically, our framework exploits the reciprocal relation between the referent and context, i. e., either of them influences estimation of the posterior distribution of the other, and thereby the search space of context can be greatly reduced.
Joint Visual Grounding with Language Scene Graphs
In this paper, we alleviate the missing-annotation problem and enable the joint reasoning by leveraging the language scene graph which covers both labeled referent and unlabeled contexts (other objects, attributes, and relationships).
Context-Aware Visual Policy Network for Fine-Grained Image Captioning
With the maturity of visual detection techniques, we are more ambitious in describing visual content with open-vocabulary, fine-grained and free-form language, i. e., the task of image captioning.
Learning to Compose and Reason with Language Tree Structures for Visual Grounding
Grounding natural language in images, such as localizing "the black dog on the left of the tree", is one of the core problems in artificial intelligence, as it needs to comprehend the fine-grained and compositional language space.
Learning to Collocate Neural Modules for Image Captioning
To this end, we make the following technical contributions for CNM training: 1) compact module design --- one for function words and three for visual content words (eg, noun, adjective, and verb), 2) soft module fusion and multi-step module execution, robustifying the visual reasoning in partial observation, 3) a linguistic loss for module controller being faithful to part-of-speech collocations (eg, adjective is before noun).
Making History Matter: History-Advantage Sequence Training for Visual Dialog
We study the multi-round response generation in visual dialog, where a response is generated according to a visually grounded conversational history.
Ranked #10 on
Visual Dialog
on VisDial v0.9 val
Learning to Assemble Neural Module Tree Networks for Visual Grounding
In particular, we develop a novel modular network called Neural Module Tree network (NMTree) that regularizes the visual grounding along the dependency parsing tree of the sentence, where each node is a neural module that calculates visual attention according to its linguistic feature, and the grounding score is accumulated in a bottom-up direction where as needed.
Auto-Encoding Scene Graphs for Image Captioning
We propose Scene Graph Auto-Encoder (SGAE) that incorporates the language inductive bias into the encoder-decoder image captioning framework for more human-like captions.
Recursive Visual Attention in Visual Dialog
Visual dialog is a challenging vision-language task, which requires the agent to answer multi-round questions about an image.
Ranked #13 on
Visual Dialog
on VisDial v0.9 val
Counterfactual Critic Multi-Agent Training for Scene Graph Generation
CMAT is a multi-agent policy gradient method that frames objects as cooperative agents, and then directly maximizes a graph-level metric as the reward.
Learning to Compose Dynamic Tree Structures for Visual Contexts
We propose to compose dynamic tree structures that place the objects in an image into a visual context, helping visual reasoning tasks such as scene graph generation and visual Q&A.
Ranked #5 on
Panoptic Scene Graph Generation
on PSG Dataset
Explainable and Explicit Visual Reasoning over Scene Graphs
We aim to dismantle the prevalent black-box neural architectures used in complex visual reasoning tasks, into the proposed eXplainable and eXplicit Neural Modules (XNMs), which advance beyond existing neural module networks towards using scene graphs --- objects as nodes and the pairwise relationships as edges --- for explainable and explicit reasoning with structured knowledge.
Ranked #10 on
Visual Question Answering (VQA)
on CLEVR
DeepChannel: Salience Estimation by Contrastive Learning for Extractive Document Summarization
We propose DeepChannel, a robust, data-efficient, and interpretable neural model for extractive document summarization.
Learning to Embed Sentences Using Attentive Recursive Trees
Sentence embedding is an effective feature representation for most deep learning-based NLP tasks.
Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networks
Deep neural networks suffer from over-fitting and catastrophic forgetting when trained with small data.
Stochastic Dynamics for Video Infilling
In this paper, we introduce a stochastic dynamics video infilling (SDVI) framework to generate frames between long intervals in a video.
Context-Aware Visual Policy Network for Sequence-Level Image Captioning
To fill the gap, we propose a Context-Aware Visual Policy network (CAVP) for sequence-level image captioning.
Shuffle-Then-Assemble: Learning Object-Agnostic Visual Relationship Features
By "agnostic", we mean that the feature is less likely biased to the classes of paired objects.
Discrete Factorization Machines for Fast Feature-based Recommendation
In this paper, we develop a generic feature-based recommendation model, called Discrete Factorization Machine (DFM), for fast and accurate recommendation.
Learning to Guide Decoding for Image Captioning
Recently, much advance has been made in image captioning, and an encoder-decoder framework has achieved outstanding performance for this task.
Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder
Existing video hash functions are built on three isolated stages: frame pooling, relaxed learning, and binarization, which have not adequately explored the temporal order of video frames in a joint binary optimization model, resulting in severe information loss.
Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks
We propose a novel framework called Semantics-Preserving Adversarial Embedding Network (SP-AEN) for zero-shot visual recognition (ZSL), where test images and their classes are both unseen during training.
Grounding Referring Expressions in Images by Variational Context
This is a general yet challenging vision-language task since it does not only require the localization of objects, but also the multimodal comprehension of context --- visual attributes (e. g., "largest", "baby") and relationships (e. g., "behind") that help to distinguish the referent from other objects, especially those of the same category.
Neural Collaborative Filtering
When it comes to model the key factor in collaborative filtering -- the interaction between user and item features, they still resorted to matrix factorization and applied an inner product on the latent features of users and items.
Fast Matrix Factorization for Online Recommendation with Implicit Feedback
To address this, we specifically design a new learning algorithm based on the element-wise Alternating Least Squares (eALS) technique, for efficiently optimizing a MF model with variably-weighted missing data.
Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks
Factorization Machines (FMs) are a supervised learning approach that enhances the linear regression model by incorporating the second-order feature interactions.
PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
We aim to tackle a novel vision task called Weakly Supervised Visual Relation Detection (WSVRD) to detect "subject-predicate-object" relations in an image with object relation groundtruths available only at the image level.
Attributed Social Network Embedding
For social networks, besides the network structure, there also exists rich information about social actors, such as user profiles of friendship networks and textual content of citation networks.
Social and Information Networks
Visual Translation Embedding Network for Visual Relation Detection
To the best of our knowledge, VTransE is the first end-to-end relation detection network.
SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
Existing visual attention models are generally spatial, i. e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of a CNN encoding an input image.
Online Collaborative Learning for Open-Vocabulary Visual Classifiers
Leveraging on the structure of the proposed collaborative learning formulation, we develop an efficient online algorithm that can jointly learn the label embeddings and visual classifiers.
Learning Image and User Features for Recommendation in Social Networks
It is often a great challenge for traditional recommender systems to learn representative features of both users and images in large social networks, in particular, social curation networks, which are characterized as the extremely sparse links between users and images, and the extremely diverse visual contents of images.
