Search Results for author:
Found 32 papers, 18 papers with code
Dynamic Video Segmentation Network
We explore the use of a decision network to adaptively assign different frame regions to different networks based on a metric called expected confidence score.
Adversarial Active Exploration for Inverse Dynamics Model Learning
Our framework consists of a deep reinforcement learning (DRL) agent and an inverse dynamics model contesting with each other.
Visual Relationship Prediction via Label Clustering and Incorporation of Depth Information
The additional depth prediction path supplements the relationship prediction model in a way that bounding boxes or segmentation masks are unable to deliver.
Speed Reading: Learning to Read ForBackward via Shuttle
We present LSTM-Shuttle, which applies human speed reading techniques to natural language processing tasks for accurate and efficient comprehension.
Adversarial Exploration Strategy for Self-Supervised Imitation Learning
Our framework consists of a deep reinforcement learning (DRL) agent and an inverse dynamics model contesting with each other.
GraphRel: Modeling Text as Relational Graphs for Joint Entity and Relation Extraction
In contrast to previous baselines, we consider the interaction between named entities and relations via a 2nd-phase relation-weighted GCN to better extract relations.
Why Attention? Analyze BiLSTM Deficiency and Its Remedies in the Case of NER
We test the practical impacts of the deficiency on real-world NER datasets, OntoNotes 5. 0 and WNUT 2017, with clear and consistent improvements over the baseline, up to 8. 7% on some of the multi-token entity mentions.
Ranked #19 on
Named Entity Recognition (NER)
on WNUT 2017
Why Attention? Analyzing and Remedying BiLSTM Deficiency in Modeling Cross-Context for NER
State-of-the-art approaches of NER have used sequence-labeling BiLSTM as a core module.
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling
In particular, we present a model-agnostic adversarial path sampler (APS) that learns to sample challenging paths that force the navigator to improve based on the navigation performance.
Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation
Outdoor vision-and-language navigation (VLN) is such a task where an agent follows natural language instructions and navigates a real-life urban environment.
Ranked #4 on
Vision and Language Navigation
on Touchdown Dataset
(using extra training data)
SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning
In this paper, we introduce a Self-Supervised Counterfactual Reasoning (SSCR) framework that incorporates counterfactual thinking to overcome data scarcity.
H-FND: Hierarchical False-Negative Denoising for Distant Supervision Relation Extraction
Experiments on SemEval-2010 and TACRED were conducted with controlled FN ratios that randomly turn the relations of training and validation instances into negatives to generate FN instances.
DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents
Creating presentation materials requires complex multimodal reasoning skills to summarize key concepts and arrange them in a logical and visually pleasing manner.
L2C: Describing Visual Differences Needs Semantic Understanding of Individuals
Recent advances in language and vision push forward the research of captioning a single image to describing visual differences between image pairs.
M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformers
LBVE contains two features: 1) the scenario of the source video is preserved instead of generating a completely different video; 2) the semantic is presented differently in the target video, and all changes are controlled by the given instruction.
Semi-Supervised Policy Initialization for Playing Games with Language Hints
Using natural language as a hint can supply an additional reward for playing sparse-reward games.
Language-Driven Image Style Transfer
We propose contrastive language visual artist (CLVA) that learns to extract visual semantics from style instructions and accomplish LDAST by the patch-wise style discriminator.
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
Further, unlike previous studies that found pre-training tasks on video inputs (e. g., masked frame modeling) not very effective, we design a new pre-training task, Masked Visual-token Modeling (MVM), for better video modeling.
Ranked #20 on
Zero-Shot Video Retrieval
on DiDeMo
An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling
Masked visual modeling (MVM) has been recently proven effective for visual pre-training.
Ranked #1 on
Video Question Answering
on LSMDC-MC
ULN: Towards Underspecified Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) is a task to guide an embodied agent moving to a target position using language instructions.
CPL: Counterfactual Prompt Learning for Vision and Language Models
Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP.
Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation
Inspired by this, we introduce a novel task, text-guided video completion (TVC), which requests the model to generate a video from partial frames guided by an instruction.
Ranked #3 on
Video Prediction
on BAIR Robot Pushing
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis
In this work, we improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions.
Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation
We conduct a series of experiments to compare the common edits made by humans and GPT-k, evaluate the performance of GPT-k in prompting T2I, and examine factors that may influence this process.
Discriminative Diffusion Models as Few-shot Vision and Language Learners
Diffusion models, such as Stable Diffusion, have shown incredible performance on text-to-image generation.
Text-guided 3D Human Generation from 2D Collections
To address this \texttt{T3H} task, we propose Compositional Cross-modal Human (CCH).
Ranked #1 on
Text-to-3D-Human Generation
on SHHQ
EDIS: Entity-Driven Image Search over Multimodal Web Content
Making image retrieval methods practical for real-world search applications requires significant progress in dataset scales, entity comprehension, and multimodal information fusion.
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness.
VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View
In this work, we propose VELMA, an embodied LLM agent that uses a verbalization of the trajectory and of visual environment observations as contextual prompt for the next action.
Guiding Instruction-based Image Editing via Multimodal Large Language Models
Extensive experimental results demonstrate that expressive instructions are crucial to instruction-based image editing, and our MGIE can lead to a notable improvement in automatic metrics and human evaluation while maintaining competitive inference efficiency.
Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual encoder and failed to perform well on broader tasks.
Ranked #56 on
Visual Question Answering
on MM-Vet
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampler
In particular, we present a model-agnostic adversarial path sampler (APS) that learns to sample challenging paths that force the navigator to improve based on the navigation performance.
