Search Results for author:
Found 24 papers, 13 papers with code
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampler
In particular, we present a model-agnostic adversarial path sampler (APS) that learns to sample challenging paths that force the navigator to improve based on the navigation performance.
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis
In this work, we improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions.
Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation
Inspired by this, we introduce a novel task, text-guided video completion (TVC), which requests the model to generate a video from partial frames guided by an instruction.
Ranked #3 on
Video Prediction
on BAIR Robot Pushing
CPL: Counterfactual Prompt Learning for Vision and Language Models
Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP.
ULN: Towards Underspecified Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) is a task to guide an embodied agent moving to a target position using language instructions.
An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling
Masked visual modeling (MVM) has been recently proven effective for visual pre-training.
Ranked #1 on
Video Question Answering
on LSMDC-MC
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
Further, unlike previous studies that found pre-training tasks on video inputs (e. g., masked frame modeling) not very effective, we design a new pre-training task, Masked Visual-token Modeling (MVM), for better video modeling.
Language-Driven Image Style Transfer
We propose contrastive language visual artist (CLVA) that learns to extract visual semantics from style instructions and accomplish LDAST by the patch-wise style discriminator.
Semi-Supervised Policy Initialization for Playing Games with Language Hints
Using natural language as a hint can supply an additional reward for playing sparse-reward games.
M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformers
LBVE contains two features: 1) the scenario of the source video is preserved instead of generating a completely different video; 2) the semantic is presented differently in the target video, and all changes are controlled by the given instruction.
L2C: Describing Visual Differences Needs Semantic Understanding of Individuals
Recent advances in language and vision push forward the research of captioning a single image to describing visual differences between image pairs.
DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents
Creating presentation materials requires complex multimodal reasoning skills to summarize key concepts and arrange them in a logical and visually pleasing manner.
H-FND: Hierarchical False-Negative Denoising for Distant Supervision Relation Extraction
Experiments on SemEval-2010 and TACRED were conducted with controlled FN ratios that randomly turn the relations of training and validation instances into negatives to generate FN instances.
SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning
In this paper, we introduce a Self-Supervised Counterfactual Reasoning (SSCR) framework that incorporates counterfactual thinking to overcome data scarcity.
Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation
Outdoor vision-and-language navigation (VLN) is such a task where an agent follows natural language instructions and navigates a real-life urban environment.
Ranked #4 on
Vision and Language Navigation
on Touchdown Dataset
(using extra training data)
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling
In particular, we present a model-agnostic adversarial path sampler (APS) that learns to sample challenging paths that force the navigator to improve based on the navigation performance.
Why Attention? Analyzing and Remedying BiLSTM Deficiency in Modeling Cross-Context for NER
State-of-the-art approaches of NER have used sequence-labeling BiLSTM as a core module.
Why Attention? Analyze BiLSTM Deficiency and Its Remedies in the Case of NER
We test the practical impacts of the deficiency on real-world NER datasets, OntoNotes 5. 0 and WNUT 2017, with clear and consistent improvements over the baseline, up to 8. 7% on some of the multi-token entity mentions.
Ranked #17 on
Named Entity Recognition (NER)
on WNUT 2017
GraphRel: Modeling Text as Relational Graphs for Joint Entity and Relation Extraction
In contrast to previous baselines, we consider the interaction between named entities and relations via a 2nd-phase relation-weighted GCN to better extract relations.
Adversarial Exploration Strategy for Self-Supervised Imitation Learning
Our framework consists of a deep reinforcement learning (DRL) agent and an inverse dynamics model contesting with each other.
Speed Reading: Learning to Read ForBackward via Shuttle
We present LSTM-Shuttle, which applies human speed reading techniques to natural language processing tasks for accurate and efficient comprehension.
Visual Relationship Prediction via Label Clustering and Incorporation of Depth Information
The additional depth prediction path supplements the relationship prediction model in a way that bounding boxes or segmentation masks are unable to deliver.
Adversarial Active Exploration for Inverse Dynamics Model Learning
Our framework consists of a deep reinforcement learning (DRL) agent and an inverse dynamics model contesting with each other.
Dynamic Video Segmentation Network
We explore the use of a decision network to adaptively assign different frame regions to different networks based on a metric called expected confidence score.
