Search Results for author:
Found 27 papers, 17 papers with code
RAP: Retrieval-Augmented Planner for Adaptive Procedure Planning in Instructional Videos
(3) Annotation cost: Annotating instructional videos with step-level labels (i. e., timestamp) or sequence-level labels (i. e., action category) is demanding and labor-intensive, limiting its generalizability to large-scale datasets. In this work, we propose a new and practical setting, called adaptive procedure planning in instructional videos, where the procedure length is not fixed or pre-determined.
SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos
We study the problem of procedure planning in instructional videos, which aims to make a goal-oriented sequence of action steps given partial visual state observations.
Language Models are Causal Knowledge Extractors for Zero-shot Video Question Answering
Causal Video Question Answering (CVidQA) queries not only association or temporal relations but also causal relations in a video.
DiGeo: Discriminative Geometry-Aware Learning for Generalized Few-Shot Object Detection
Generalized few-shot object detection aims to achieve precise detection on both base classes with abundant annotations and novel classes with limited training data.
Debiased Fine-Tuning for Vision-language Models by Prompt Regularization
We present a new paradigm for fine-tuning large-scale visionlanguage pre-trained models on downstream task, dubbed Prompt Regularization (ProReg).
In Defense of Structural Symbolic Representation for Video Event-Relation Prediction
Understanding event relationships in videos requires a model to understand the underlying structures of events (i. e. the event type, the associated argument roles, and corresponding entities) and factual knowledge for reasoning.
Respecting Transfer Gap in Knowledge Distillation
The network response serves as additional supervision to formulate the machine domain, which uses the data collected from the human domain as a transfer set.
Weakly-Supervised Temporal Article Grounding
Specifically, given an article and a relevant video, WSAG aims to localize all ``groundable'' sentences to the video, and these sentences are possibly at different semantic scales.
Explicit Image Caption Editing
Given an image and a reference caption, the image caption editing task aims to correct the misalignment errors and generate a refined caption.
On Non-Random Missing Labels in Semi-Supervised Learning
Our method is three-fold: 1) We propose Class-Aware Propensity (CAP) that exploits the unlabeled data to train an improved classifier using the biased labeled data.
Beyond Grounding: Extracting Fine-Grained Event Hierarchies Across Modalities
To support research on this task, we introduce the Multimodal Hierarchical Events (MultiHiEve) dataset.
Prompt-aligned Gradient for Prompt Tuning
Thanks to the large pre-trained vision-language models (VLMs) like CLIP, we can craft a zero-shot classifier by "prompt", e. g., the confidence score of an image being "[CLASS]" can be obtained by using the VLM provided similarity measure between the image and the prompt sentence "a photo of a [CLASS]".
Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification
We address the overlooked unbiasedness in existing long-tailed classification methods: we find that their overall improvement is mostly attributed to the biased preference of tail over head, as the test distribution is assumed to be balanced; however, when the test is as imbalanced as the long-tailed training data -- let the test respect Zipf's law of nature -- the tail bias is no longer beneficial overall because it hurts the head majorities.
Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphs
To this end, we propose a new classification-then-grounding framework for VidSGG, which can avoid all the three overlooked drawbacks.
Introspective Distillation for Robust Question Answering
Question answering (QA) models are well-known to exploit data bias, e. g., the language prior in visual QA and the position bias in reading comprehension.
Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering
Specifically, CSST is composed of two parts: Counterfactual Samples Synthesizing (CSS) and Counterfactual Samples Training (CST).
COSY: COunterfactual SYntax for Cross-Lingual Understanding
Pre-trained multilingual language models, e. g., multilingual-BERT, are widely used in cross-lingual tasks, yielding the state-of-the-art performance.
Counterfactual Variable Control for Robust and Interpretable Question Answering
We then conduct two novel CVC inference methods (on trained models) to capture the effect of comprehensive reasoning as the final prediction.
Counterfactual VQA: A Cause-Effect Look at Language Bias
VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language.
Domain-Adaptive Few-Shot Learning
Existing few-shot learning (FSL) methods make the implicit assumption that the few target class samples are from the same domain as the source class samples.
Unbiased Scene Graph Generation from Biased Training
Today's scene graph generation (SGG) task is still far from practical, mainly due to the severe training bias, e. g., collapsing diverse "human walk on / sit on / lay on beach" into "human on beach".
Ranked #3 on
Scene Graph Generation
on Visual Genome
Two Causal Principles for Improving Visual Dialog
This paper unravels the design tricks adopted by us, the champion team MReaL-BDAI, for Visual Dialog Challenge 2019: two causal principles for improving Visual Dialog (VisDial).
Mobile Video Action Recognition
In addition to motion vectors, we also provide a temporal fusion method to explicitly induce the temporal context.
Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions
Specifically, our framework exploits the reciprocal relation between the referent and context, i. e., either of them influences estimation of the posterior distribution of the other, and thereby the search space of context can be greatly reduced.
Recursive Visual Attention in Visual Dialog
Visual dialog is a challenging vision-language task, which requires the agent to answer multi-round questions about an image.
Ranked #13 on
Visual Dialog
on VisDial v0.9 val
Grounding Referring Expressions in Images by Variational Context
This is a general yet challenging vision-language task since it does not only require the localization of objects, but also the multimodal comprehension of context --- visual attributes (e. g., "largest", "baby") and relationships (e. g., "behind") that help to distinguish the referent from other objects, especially those of the same category.
Multi-Modal Multi-Scale Deep Learning for Large-Scale Image Annotation
In this paper, we address two main issues in large-scale image annotation: 1) how to learn a rich feature representation suitable for predicting a diverse set of visual concepts ranging from object, scene to abstract concept; 2) how to annotate an image with the optimal number of class labels.
