Search Results for author:
Found 143 papers, 93 papers with code
Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning
Neural discrete reasoning (NDR) has shown remarkable progress in combining deep models with discrete reasoning.
Pushing Rendering Boundaries: Hard Gaussian Splatting
To address this problem, we propose Hard Gaussian Splatting, dubbed HGS, which considers multi-view significant positional gradients and rendering errors to grow hard Gaussians that fill the gaps of classical Gaussian Splatting on 3D scenes, thus achieving superior NVS results.
HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editing
HumanEdit bridges this gap by employing human annotators to construct data pairs and administrators to provide feedback.
LoRA of Change: Learning to Generate LoRA for the Editing Instruction from A Single Before-After Image Pair
Furthermore, generalizable models for image editing with visual instructions typically require quad data, i. e., a before-after image pair, along with query and target images.
Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing
For causal generation, it introduces unidirectional feature computation, which ensures that the cache of conditional frames can be precomputed in previous autoregression steps and reused in every subsequent step, eliminating redundant computations.
CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction
In this paper, we propose a new de\textbf{C}oupled du\textbf{A}l-interactive linea\textbf{R} att\textbf{E}ntion (CARE) mechanism, revealing that features' decoupling and interaction can fully unleash the power of linear attention.
AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea
Instruction-based image editing aims to modify specific image elements with natural language instructions.
Robust Fine-tuning of Zero-shot Models via Variance Reduction
When fine-tuning zero-shot models like CLIP, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD).
Unified Generative and Discriminative Training for Multi-modal Large Language Models
Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval, yet struggles with complex scenarios requiring fine-grained semantic differentiation.
Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting
Vision-language models, such as CLIP, have shown impressive generalization capacities when using appropriate text descriptions.
Few-shot NeRF by Adaptive Rendering Loss Regularization
In this work, we reveal that there exists an inconsistency between the frequency regularization of PE and rendering loss.
Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing
Recent developments of vision large language models (LLMs) have seen remarkable progress, yet still encounter challenges towards multimodal generalists, such as coarse-grained instance-level understanding, lack of unified support for both images and videos, and insufficient coverage across various vision tasks.
Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration
In this paper, we propose UniKE, a novel multimodal editing method that establishes a unified perspective and paradigm for intrinsic knowledge editing and external knowledge resorting.
Instruction Tuning-free Visual Token Complement for Multimodal LLMs
However, current research is inherently constrained by challenges such as the need for high-quality instruction pairs and the loss of visual information in image-to-text training objectives.
Selective Vision-Language Subspace Projection for Few-shot CLIP
Vision-language models such as CLIP are capable of mapping the different modality data into a unified feature space, enabling zero/few-shot inference by measuring the similarity of given images and texts.
Visual Prompt Selection for In-Context Learning Segmentation
As a fundamental and extensively studied task in computer vision, image segmentation aims to locate and identify different semantic concepts at the pixel level.
ViD-GPT: Introducing GPT-style Autoregressive Generation in Video Diffusion Models
Inspired from the huge success of large language models (LLMs) and following GPT (generative pre-trained transformer), we bring causal (i. e., unidirectional) generation into VDMs, and use past frames as prompt to generate future frames.
EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal Prompts
Recent advancements in image generation have enabled the creation of high-quality images from text conditions.
MVGamba: Unify 3D Content Generation as State Space Sequence Modeling
Recent 3D large reconstruction models (LRMs) can generate high-quality 3D content in sub-seconds by integrating multi-view diffusion models with scalable multi-view reconstructors.
Towards Semantic Equivalence of Tokenization in Multimodal LLM
The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features.
Ranked #72 on
Visual Question Answering
on MM-Vet
A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training
Training diffusion models is always a computation-intensive task.
Non-confusing Generation of Customized Concepts in Diffusion Models
We tackle the common challenge of inter-concept visual confusion in compositional concept generation using text-guided diffusion models (TGDMs).
Auto-Encoding Morph-Tokens for Multimodal LLM
For multimodal LLMs, the synergy of visual comprehension (textual output) and generation (visual output) presents an ongoing challenge.
Dual-Modal Prompting for Sketch-Based Image Retrieval
In this study, we aim to tackle two major challenges of this task simultaneously: i) zero-shot, dealing with unseen categories, and ii) fine-grained, referring to intra-category instance-level retrieval.
Diffusion Time-step Curriculum for One Image to 3D Generation
Score distillation sampling~(SDS) has been widely adopted to overcome the absence of unseen views in reconstructing 3D objects from a \textbf{single} image.
Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstruction
We tackle the challenge of efficiently reconstructing a 3D asset from a single image at millisecond speed.
View-Consistent 3D Editing with Gaussian Splatting
However, this approach faces a critical issue of multi-view inconsistency, where the guidance images exhibit significant discrepancies across views, leading to mode collapse and visual artifacts of 3DGS.
Controllable Relation Disentanglement for Few-Shot Class-Incremental Learning
In this paper, we propose to tackle Few-Shot Class-Incremental Learning (FSCIL) from a new perspective, i. e., relation disentanglement, which means enhancing FSCIL via disentangling spurious relation between categories.
Distributionally Generative Augmentation for Fair Facial Attribute Classification
This work proposes a novel, generation-based two-stage framework to train a fair FAC model on biased data without additional annotation.
Discriminative Probing and Tuning for Text-to-Image Generation
We present a discriminative adapter built on T2I models to probe their discriminative abilities on two representative tasks and leverage discriminative fine-tuning to improve their text-image alignment.
Doubly Abductive Counterfactual Inference for Text-based Image Editing
Through the lens of the formulation, we find that the crux of TBIE is that existing techniques hardly achieve a good trade-off between editability and fidelity, mainly due to the overfitting of the single-image fine-tuning.
Few-shot Learner Parameterization by Diffusion Time-steps
To this end, we find an inductive bias that the time-steps of a Diffusion Model (DM) can isolate the nuanced class attributes, i. e., as the forward diffusion adds noise to an image at each time-step, nuanced attributes are usually lost at an earlier time-step than the spurious attributes that are visually prominent.
Classes Are Not Equal: An Empirical Study on Image Recognition Fairness
Second, with the proposed concept of Model Prediction Bias, we investigate the origins of problematic representation during optimization.
Exploring Diffusion Time-steps for Unsupervised Representation Learning
Representation learning is all about discovering the hidden modular attributes that generate the data faithfully.
Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior
To solve this issue, we first deeply analyze the SDS and find that its distillation sampling process indeed corresponds to the trajectory sampling of a stochastic differential equation (SDE): SDS samples along an SDE trajectory to yield a less noisy sample which then serves as a guidance to optimize a 3D model.
Dual-Perspective Knowledge Enrichment for Semi-Supervised 3D Object Detection
Our DPKE enriches the knowledge of limited training data, particularly unlabeled data, from two perspectives: data-perspective and feature-perspective.
MGNet: Learning Correspondences via Multiple Graphs
To obtain information integrating implicit and explicit local graphs, we construct local graphs from implicit and explicit aspects and combine them effectively, which is used to build a global graph.
Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models
As Archimedes famously said, ``Give me a lever long enough and a fulcrum on which to place it, and I shall move the world'', in this study, we propose to use a tiny Language Model (LM), \eg, a Transformer with 67M parameters, to lever much larger Vision-Language Models (LVLMs) with 9B parameters.
ChartLlama: A Multimodal LLM for Chart Understanding and Generation
Next, we introduce ChartLlama, a multi-modal large language model that we've trained using our created dataset.
Invariant Feature Regularization for Fair Face Recognition
Unfortunately, face datasets inevitably capture the imbalanced demographic attributes that are ubiquitous in real-world observations, and the model learns biased feature that generalizes poorly in the minority group.
Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models
In this study, we systematically examine the biases in foundation models and demonstrate the efficacy of our proposed Generalized Logit Adjustment (GLA) method.
Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation
Domain Adaptation (DA) is always challenged by the spurious correlation between domain-invariant features (e. g., class identity) and domain-specific features (e. g., environment) that does not generalize to the target domain.
Towards Debiasing Frame Length Bias in Text-Video Retrieval via Causal Intervention
Many studies focus on improving pretraining or developing new backbones in text-video retrieval.
Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs
In this work, we investigate strengthening the awareness of video dynamics for DMs, for high-quality T2V generation.
Invariant Training 2D-3D Joint Hard Samples for Few-Shot Point Cloud Recognition
We tackle the data scarcity challenge in few-shot point cloud recognition of 3D objects by using a joint prediction from a conventional 3D model and a well-trained 2D model.
Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions
This shortcoming results in MLLMs' underperformance in comprehending demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task.
Random Boxes Are Open-world Object Detectors
First, as the randomization is independent of the distribution of the limited known objects, the random proposals become the instrumental variable that prevents the training from being confounded by the known objects.
DisCo: Disentangled Control for Realistic Human Dance Generation
In this paper, we depart from the traditional paradigm of human motion transfer and emphasize two additional critical attributes for the synthesis of human dance content in social media contexts: (i) Generalizability: the model should be able to generalize beyond generic human viewpoints as well as unseen human subjects, backgrounds, and poses; (ii) Compositionality: it should allow for the seamless composition of seen/unseen subjects, backgrounds, and poses from different sources.
Fast Diffusion Model
In this paper, we propose a Fast Diffusion Model (FDM) to significantly speed up DMs from a stochastic optimization perspective for both faster training and sampling.
An Overview of Challenges in Egocentric Text-Video Retrieval
Text-video retrieval contains various challenges, including biases coming from diverse sources.
Decoupled Kullback-Leibler Divergence Loss
In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of 1) a weighted Mean Square Error (wMSE) loss and 2) a Cross-Entropy loss incorporating soft labels.
Equivariant Similarity for Vision-Language Foundation Models
Unlike the existing image-text similarity objective which only categorizes matched pairs as similar and unmatched pairs as dissimilar, equivariance also requires similarity to vary faithfully according to the semantic changes.
Ranked #7 on
Visual Reasoning
on Winoground
Unbiased Multiple Instance Learning for Weakly Supervised Video Anomaly Detection
At each MIL training iteration, we use the current detector to divide the samples into two groups with different context biases: the most confident abnormal/normal snippets and the rest ambiguous ones.
Semantic Scene Completion with Cleaner Self
SSC is a well-known ill-posed problem as the prediction model has to "imagine" what is behind the visible surface, which is usually represented by Truncated Signed Distance Function (TSDF).
Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection
Without bells and whistles, our RePro achieves a new state-of-the-art performance on two VidVRD benchmarks of not only the base training object and predicate categories, but also the unseen ones.
Debiased Fine-Tuning for Vision-language Models by Prompt Regularization
We present a new paradigm for fine-tuning large-scale visionlanguage pre-trained models on downstream task, dubbed Prompt Regularization (ProReg).
Adaptively Clustering Neighbor Elements for Image-Text Generation
In this gradual clustering process, a parsing tree is generated which embeds the hierarchical knowledge of the input sequence.
Learning Trajectory-Word Alignments for Video-Language Tasks
To amend this, we propose a novel TW-BERT to learn Trajectory-Word alignment by a newly designed trajectory-to-word (T2W) attention for solving video-language tasks.
Bootstrap Your Own Prior: Towards Distribution-Agnostic Novel Class Discovery
Novel Class Discovery (NCD) aims to discover unknown classes without any annotation, by exploiting the transferable knowledge already learned from a base set of known classes.
Mitigating and Evaluating Static Bias of Action Representations in the Background and the Foreground
The video background is clearly a source of static bias, but the video foreground, such as the clothing of the actor, can also provide static bias.
Attention-based Class Activation Diffusion for Weakly-Supervised Semantic Segmentation
An intuitive solution is ``coupling'' the CAM with the long-range attention matrix of visual transformers (ViT) We find that the direct ``coupling'', e. g., pixel-wise multiplication of attention and activation, achieves a more global coverage (on the foreground), but unfortunately goes with a great increase of false positives, i. e., background pixels are mistakenly included.
Weakly supervised Semantic Segmentation
Weakly-Supervised Semantic Segmentation
Respecting Transfer Gap in Knowledge Distillation
The network response serves as additional supervision to formulate the machine domain, which uses the data collected from the human domain as a transfer set.
Learning to Collocate Visual-Linguistic Neural Modules for Image Captioning
This is because the language is only partially observable, for which we need to dynamically collocate the modules during the process of image captioning.
Class Is Invariant to Context and Vice Versa: On Learning Invariance for Out-Of-Distribution Generalization
If the context in every class is evenly distributed, OOD would be trivial because the context can be easily removed due to an underlying principle: class is invariant to context.
NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation
To this end, we propose a novel NoIsy label CorrEction and Sample Training strategy for SGG: NICEST.
Identifying Hard Noise in Long-Tailed Sample Distribution
Such imbalanced training data makes a classifier less discriminative for the tail classes, whose previously "easy" noises are now turned into "hard" ones -- they are almost as outliers as the clean tail samples.
Equivariance and Invariance Inductive Bias for Learning from Insufficient Data
We are interested in learning robust models from insufficient data, without the need for any externally pre-trained checkpoints.
Invariant Feature Learning for Generalized Long-Tailed Classification
In fact, even if the class is balanced, samples within each class may still be long-tailed due to the varying attributes.
Ranked #1 on
Long-tail Learning
on ImageNet-GLT
On Non-Random Missing Labels in Semi-Supervised Learning
Our method is three-fold: 1) We propose Class-Aware Propensity (CAP) that exploits the unlabeled data to train an improved classifier using the biased labeled data.
Exploiting Semantic Role Contextualized Video Features for Multi-Instance Text-Video Retrieval EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022
In this report, we present our approach for EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022.
Ranked #10 on
Multi-Instance Retrieval
on EPIC-KITCHENS-100
RoME: Role-aware Mixture-of-Expert Transformer for Text-to-Video Retrieval
Most methods consider only one joint embedding space between global visual and textual features without considering the local structures of each modality.
Ranked #12 on
Video Retrieval
on YouCook2
Prompt-aligned Gradient for Prompt Tuning
Thanks to the large pre-trained vision-language models (VLMs) like CLIP, we can craft a zero-shot classifier by "prompt", e. g., the confidence score of an image being "[CLASS]" can be obtained by using the VLM provided similarity measure between the image and the prompt sentence "a photo of a [CLASS]".
Certified Robustness Against Natural Language Attacks by Causal Intervention
Deep learning models have achieved great success in many fields, yet they are vulnerable to adversarial examples.
Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation
Specifically, due to the sum-over-class pooling nature of BCE, each pixel in CAM may be responsive to multiple classes co-occurring in the same receptive field.
Weakly supervised Semantic Segmentation
Weakly-Supervised Semantic Segmentation
Deconfounded Visual Grounding
We focus on the confounding bias between language and location in the visual grounding pipeline, where we find that the bias is the major visual reasoning bottleneck.
Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification
We address the overlooked unbiasedness in existing long-tailed classification methods: we find that their overall improvement is mostly attributed to the biased preference of tail over head, as the test distribution is assumed to be balanced; however, when the test is as imbalanced as the long-tailed training data -- let the test respect Zipf's law of nature -- the tail bias is no longer beneficial overall because it hurts the head majorities.
How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?
The fine-tuning of pre-trained language models has a great success in many NLP fields.
Introspective Distillation for Robust Question Answering
Question answering (QA) models are well-known to exploit data bias, e. g., the language prior in visual QA and the position bias in reading comprehension.
Self-Supervised Learning Disentangled Group Representation as Feature
A good visual representation is an inference map from observations (images) to features (vectors) that faithfully reflects the hidden modularized generative factors (semantics).
Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering
Specifically, CSST is composed of two parts: Counterfactual Samples Synthesizing (CSS) and Counterfactual Samples Training (CST).
Auto-Parsing Network for Image Captioning and Visual Question Answering
We propose an Auto-Parsing Network (APN) to discover and exploit the input data's hidden tree structures for improving the effectiveness of the Transformer-based vision-language systems.
Causal Attention for Unbiased Visual Recognition
Attention module does not always help deep models learn causal features that are robust in any confounding context, e. g., a foreground object feature is invariant to different backgrounds.
Are Missing Links Predictable? An Inferential Benchmark for Knowledge Graph Completion
We present InferWiki, a Knowledge Graph Completion (KGC) dataset that improves upon existing benchmarks in inferential ability, assumptions, and patterns.
Transporting Causal Mechanisms for Unsupervised Domain Adaptation
However, the theoretical solution provided by transportability is far from practical for UDA, because it requires the stratification and representation of the unobserved confounder that is the cause of the domain gap.
Adversarial Visual Robustness by Causal Intervention
As these visual confounders are imperceptible in general, we propose to use the instrumental variable that achieves causal intervention without the need for confounder observation.
Empowering Language Understanding with Counterfactual Reasoning
Present language understanding methods have demonstrated extraordinary ability of recognizing patterns in texts via machine learning.
VL-NMS: Breaking Proposal Bottlenecks in Two-Stage Visual-Language Matching
In this paper, we argue that these methods overlook an obvious \emph{mismatch} between the roles of proposals in the two stages: they generate proposals solely based on the detection confidence (i. e., query-agnostic), hoping that the proposals contain all instances mentioned in the text query (i. e., query-aware).
TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph
Multi-hop Question Answering (QA) is a challenging task because it requires precise reasoning with entity relations at every step towards the answer.
The Blessings of Unlabeled Background in Untrimmed Videos
The key challenge is how to distinguish the action of interest segments from the background, which is unlabelled even on the video-level.
Weakly-supervised Temporal Action Localization
Weakly Supervised Temporal Action Localization
Causal Attention for Vision-Language Tasks
Specifically, CATT is implemented as a combination of 1) In-Sample Attention (IS-ATT) and 2) Cross-Sample Attention (CS-ATT), where the latter forcibly brings other samples into every IS-ATT, mimicking the causal intervention.
Distilling Causal Effect of Data in Class-Incremental Learning
We propose a causal framework to explain the catastrophic forgetting in Class-Incremental Learning (CIL) and then derive a novel distillation method that is orthogonal to the existing anti-forgetting techniques, such as data replay and feature/label distillation.
Counterfactual Zero-Shot and Open-Set Visual Recognition
We show that the key reason is that the generation is not Counterfactual Faithful, and thus we propose a faithful one, whose generation is from the sample-specific counterfactual question: What would the sample look like, if we set its class attribute to a certain class, while keeping its sample attribute unchanged?
Hierarchical Scene Graph Encoder-Decoder for Image Paragraph Captioning
We propose irredundant attention in SSG-RNN to improve the possibility of abstracting topics from rarely described sub-graphs and inheriting attention in WSG-RNN to generate more grounded sentences with the abstracted topics, both of which give rise to more distinctive paragraphs.
Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect
On one hand, it has a harmful causal effect that misleads the tail prediction biased towards the head.
Ranked #36 on
Long-tail Learning
on CIFAR-10-LT (ρ=10)
Interventional Few-Shot Learning
Specifically, we develop three effective IFSL algorithmic implementations based on the backdoor adjustment, which is essentially a causal intervention towards the SCM of many-shot learning: the upper-bound of FSL in a causal view.
Causal Intervention for Weakly-Supervised Semantic Segmentation
We present a causal inference framework to improve Weakly-Supervised Semantic Segmentation (WSSS).
Ranked #37 on
Weakly-Supervised Semantic Segmentation
on COCO 2014 val
Clicks can be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue
However, we argue that there is a significant gap between clicks and user satisfaction -- it is common that a user is "cheated" to click an item by the attractive title/cover of the item.
Ref-NMS: Breaking Proposal Bottlenecks in Two-Stage Referring Expression Grounding
The prevailing framework for solving referring expression grounding is based on a two-stage process: 1) detecting proposals with an object detector and 2) grounding the referent to one of the proposals.
Feature Pyramid Transformer
Yet, the non-local spatial interactions are not across scales, and thus they fail to capture the non-local contexts of objects (or parts) residing in different scales.
Counterfactual VQA: A Cause-Effect Look at Language Bias
VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language.
Iterative Context-Aware Graph Inference for Visual Dialog
Visual dialog is a challenging task that requires the comprehension of the semantic dependencies among implicit visual and textual contexts.
Ranked #12 on
Visual Dialog
on VisDial v0.9 val
Learning to Segment the Tail
Real-world visual recognition requires handling the extreme sample imbalance in large-scale long-tailed data.
More Grounded Image Captioning by Distilling Image-Text Matching Model
To improve the grounding accuracy while retaining the captioning quality, it is expensive to collect the word-region alignment as strong supervision.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
To reduce the language biases, several recent works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on VQA-CP.
Ranked #1 on
Visual Question Answering (VQA)
on VQA-CP
(using extra training data)
Deconfounded Image Captioning: A Causal Retrospect
Dataset bias in vision-language tasks is becoming one of the main problems which hinders the progress of our community.
Cross-GCN: Enhancing Graph Convolutional Network with $k$-Order Feature Interactions
Graph Convolutional Network (GCN) is an emerging technique that performs learning and reasoning on graph data.
Unbiased Scene Graph Generation from Biased Training
Today's scene graph generation (SGG) task is still far from practical, mainly due to the severe training bias, e. g., collapsing diverse "human walk on / sit on / lay on beach" into "human on beach".
Ranked #3 on
Scene Graph Generation
on Visual Genome
Visual Commonsense R-CNN
We present a novel unsupervised feature representation learning method, Visual Commonsense Region-based Convolutional Neural Network (VC R-CNN), to serve as an improved visual region encoder for high-level tasks such as captioning and VQA.
Ranked #23 on
Image Captioning
on COCO Captions
General Partial Label Learning via Dual Bipartite Graph Autoencoder
Compared to the traditional Partial Label Learning (PLL) problem, GPLL relaxes the supervision assumption from instance-level -- a label set partially labels an instance -- to group-level: 1) a label set partially labels a group of instances, where the within-group instance-label link annotations are missing, and 2) cross-group links are allowed -- instances in a group may be partially linked to the label set from another group.
Ranked #1 on
Partial Label Learning
on MPII Movie Description
Two Causal Principles for Improving Visual Dialog
This paper unravels the design tricks adopted by us, the champion team MReaL-BDAI, for Visual Dialog Challenge 2019: two causal principles for improving Visual Dialog (VisDial).
Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions
Specifically, our framework exploits the reciprocal relation between the referent and context, i. e., either of them influences estimation of the posterior distribution of the other, and thereby the search space of context can be greatly reduced.
Joint Visual Grounding with Language Scene Graphs
In this paper, we alleviate the missing-annotation problem and enable the joint reasoning by leveraging the language scene graph which covers both labeled referent and unlabeled contexts (other objects, attributes, and relationships).
Context-Aware Visual Policy Network for Fine-Grained Image Captioning
With the maturity of visual detection techniques, we are more ambitious in describing visual content with open-vocabulary, fine-grained and free-form language, i. e., the task of image captioning.
Learning to Compose and Reason with Language Tree Structures for Visual Grounding
Grounding natural language in images, such as localizing "the black dog on the left of the tree", is one of the core problems in artificial intelligence, as it needs to comprehend the fine-grained and compositional language space.
Learning to Collocate Neural Modules for Image Captioning
To this end, we make the following technical contributions for CNM training: 1) compact module design --- one for function words and three for visual content words (eg, noun, adjective, and verb), 2) soft module fusion and multi-step module execution, robustifying the visual reasoning in partial observation, 3) a linguistic loss for module controller being faithful to part-of-speech collocations (eg, adjective is before noun).
Making History Matter: History-Advantage Sequence Training for Visual Dialog
We study the multi-round response generation in visual dialog, where a response is generated according to a visually grounded conversational history.
Ranked #10 on
Visual Dialog
on VisDial v0.9 val
Learning to Assemble Neural Module Tree Networks for Visual Grounding
In particular, we develop a novel modular network called Neural Module Tree network (NMTree) that regularizes the visual grounding along the dependency parsing tree of the sentence, where each node is a neural module that calculates visual attention according to its linguistic feature, and the grounding score is accumulated in a bottom-up direction where as needed.
Auto-Encoding Scene Graphs for Image Captioning
We propose Scene Graph Auto-Encoder (SGAE) that incorporates the language inductive bias into the encoder-decoder image captioning framework for more human-like captions.
Counterfactual Critic Multi-Agent Training for Scene Graph Generation
CMAT is a multi-agent policy gradient method that frames objects as cooperative agents, and then directly maximizes a graph-level metric as the reward.
Recursive Visual Attention in Visual Dialog
Visual dialog is a challenging vision-language task, which requires the agent to answer multi-round questions about an image.
Ranked #13 on
Visual Dialog
on VisDial v0.9 val
Learning to Compose Dynamic Tree Structures for Visual Contexts
We propose to compose dynamic tree structures that place the objects in an image into a visual context, helping visual reasoning tasks such as scene graph generation and visual Q&A.
Ranked #7 on
Panoptic Scene Graph Generation
on PSG Dataset
Explainable and Explicit Visual Reasoning over Scene Graphs
We aim to dismantle the prevalent black-box neural architectures used in complex visual reasoning tasks, into the proposed eXplainable and eXplicit Neural Modules (XNMs), which advance beyond existing neural module networks towards using scene graphs --- objects as nodes and the pairwise relationships as edges --- for explainable and explicit reasoning with structured knowledge.
Ranked #11 on
Visual Question Answering (VQA)
on CLEVR
Learning to Embed Sentences Using Attentive Recursive Trees
Sentence embedding is an effective feature representation for most deep learning-based NLP tasks.
DeepChannel: Salience Estimation by Contrastive Learning for Extractive Document Summarization
We propose DeepChannel, a robust, data-efficient, and interpretable neural model for extractive document summarization.
Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networks
Deep neural networks suffer from over-fitting and catastrophic forgetting when trained with small data.
Stochastic Dynamics for Video Infilling
In this paper, we introduce a stochastic dynamics video infilling (SDVI) framework to generate frames between long intervals in a video.
Context-Aware Visual Policy Network for Sequence-Level Image Captioning
To fill the gap, we propose a Context-Aware Visual Policy network (CAVP) for sequence-level image captioning.
Shuffle-Then-Assemble: Learning Object-Agnostic Visual Relationship Features
By "agnostic", we mean that the feature is less likely biased to the classes of paired objects.
Discrete Factorization Machines for Fast Feature-based Recommendation
In this paper, we develop a generic feature-based recommendation model, called Discrete Factorization Machine (DFM), for fast and accurate recommendation.
Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder
Existing video hash functions are built on three isolated stages: frame pooling, relaxed learning, and binarization, which have not adequately explored the temporal order of video frames in a joint binary optimization model, resulting in severe information loss.
Grounding Referring Expressions in Images by Variational Context
This is a general yet challenging vision-language task since it does not only require the localization of objects, but also the multimodal comprehension of context --- visual attributes (e. g., "largest", "baby") and relationships (e. g., "behind") that help to distinguish the referent from other objects, especially those of the same category.
Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks
We propose a novel framework called Semantics-Preserving Adversarial Embedding Network (SP-AEN) for zero-shot visual recognition (ZSL), where test images and their classes are both unseen during training.
Neural Collaborative Filtering
When it comes to model the key factor in collaborative filtering -- the interaction between user and item features, they still resorted to matrix factorization and applied an inner product on the latent features of users and items.
Fast Matrix Factorization for Online Recommendation with Implicit Feedback
To address this, we specifically design a new learning algorithm based on the element-wise Alternating Least Squares (eALS) technique, for efficiently optimizing a MF model with variably-weighted missing data.
Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks
Factorization Machines (FMs) are a supervised learning approach that enhances the linear regression model by incorporating the second-order feature interactions.
PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
We aim to tackle a novel vision task called Weakly Supervised Visual Relation Detection (WSVRD) to detect "subject-predicate-object" relations in an image with object relation groundtruths available only at the image level.
Attributed Social Network Embedding
For social networks, besides the network structure, there also exists rich information about social actors, such as user profiles of friendship networks and textual content of citation networks.
Social and Information Networks
Visual Translation Embedding Network for Visual Relation Detection
To the best of our knowledge, VTransE is the first end-to-end relation detection network.
SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
Existing visual attention models are generally spatial, i. e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of a CNN encoding an input image.
Online Collaborative Learning for Open-Vocabulary Visual Classifiers
Leveraging on the structure of the proposed collaborative learning formulation, we develop an efficient online algorithm that can jointly learn the label embeddings and visual classifiers.
Learning Image and User Features for Recommendation in Social Networks
It is often a great challenge for traditional recommender systems to learn representative features of both users and images in large social networks, in particular, social curation networks, which are characterized as the extremely sparse links between users and images, and the extremely diverse visual contents of images.
