Search Results for author:
Found 46 papers, 21 papers with code
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampler
In particular, we present a model-agnostic adversarial path sampler (APS) that learns to sample challenging paths that force the navigator to improve based on the navigation performance.
T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation
In this work, we seek to measure more complex human biases exist in the task of text-to-image generations.
Photoswap: Personalized Subject Swapping in Images
In an era where images and visual content dominate our digital landscape, the ability to manipulate and personalize these images has become a necessity.
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness.
R2H: Building Multimodal Navigation Helpers that Respond to Help
Furthermore, we propose a novel task-oriented multimodal response generation model that can see and respond, named SeeRee, as the navigation helper to guide the task performer in embodied tasks.
Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation
We conduct a series of experiments to compare the common edits made by humans and GPT-k, evaluate the performance of GPT-k in prompting T2I, and examine factors that may influence this process.
Discriminative Diffusion Models as Few-shot Vision and Language Learners
Diffusion models, such as Stable Diffusion, have shown incredible performance on text-to-image generation.
LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation
Existing automatic evaluation on text-to-image synthesis can only provide an image-text matching score, without considering the object-level compositionality, which results in poor correlation with human judgments.
Multimodal Procedural Planning via Dual Text-Image Prompting
The key challenges of MPP are to ensure the informativeness, temporal coherence, and accuracy of plans across modalities.
Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment
Pre-trained vision and language models such as CLIP have witnessed remarkable success in connecting images and texts with a primary focus on English texts.
Multimodal Graph Transformer for Multimodal Question Answering
Despite the success of Transformer models in vision and language tasks, they often learn knowledge from enormous data implicitly and cannot utilize structured input data directly.
ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation
Such object navigation tasks usually require large-scale training in visual environments with labeled objects, which generalizes poorly to novel objects in unknown environments.
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis
In this work, we improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions.
Navigation as Attackers Wish? Towards Building Byzantine-Robust Embodied Agents under Federated Learning
However, since the local data is inaccessible to the server under federated learning, attackers may easily poison the training data of the local client to build a backdoor in the agent without notice.
ComCLIP: Training-Free Compositional Image and Text Matching
Contrastive Language-Image Pretraining (CLIP) has demonstrated great zero-shot performance for matching images and text.
CPL: Counterfactual Prompt Learning for Vision and Language Models
Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP.
Visualize Before You Write: Imagination-Guided Open-Ended Text Generation
Recent advances in text-to-image synthesis make it possible to visualize machine imaginations for a given context.
Anticipating the Unseen Discrepancy for Vision and Language Navigation
In this paper, we propose an Unseen Discrepancy Anticipating Vision and Language Navigation (DAVIS) that learns to generalize to unseen environments via encouraging test-time visual consistency.
JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents
Building a conversational embodied agent to execute real-life tasks has been a long-standing yet quite challenging research goal, as it requires effective human-agent communication, multi-modal understanding, long-range sequential decision making, etc.
Understanding Instance-Level Impact of Fairness Constraints
A variety of fairness constraints have been proposed in the literature to mitigate group-level statistical bias.
VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation
We hope the new simulator and benchmark will facilitate future research on language-guided robotic manipulation.
Neuro-Symbolic Procedural Planning with Commonsense Prompting
Procedural planning aims to implement complex high-level goals by decomposition into sequential simpler low-level steps.
Aerial Vision-and-Dialog Navigation
To this end, we introduce Aerial Vision-and-Dialog Navigation (AVDN), to navigate a drone via natural language conversation.
Imagination-Augmented Natural Language Understanding
Human brains integrate linguistic and perceptual information simultaneously to understand natural language, and hold the critical ability to render imaginations.
Parameter-efficient Model Adaptation for Vision Transformers
In this paper, we aim to study parameter-efficient model adaptation strategies for vision transformers on the image classification task.
FedVLN: Privacy-preserving Federated Vision-and-Language Navigation
Data privacy is a central problem for embodied agents that can perceive the environment, communicate with humans, and act in the real world.
Interpretable Research Replication Prediction via Variational Contextual Consistency Sentence Masking
Research Replication Prediction (RRP) is the task of predicting whether a published research result can be replicated or not.
Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning
To systematically measure the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i. e., Charades-CG and ActivityNet-CG.
Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
A long-term goal of AI research is to build intelligent agents that can communicate with humans in natural language, perceive the environment, and perform real-world tasks.
Relational Graph Learning for Grounded Video Description Generation
Such a setting can help explain the decisions of captioning models and prevents the model from hallucinating object words in its description.
Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search
Internet search affects people's cognition of the world, so mitigating biases in search results and learning fair models is imperative for social good.
CUDA-GHR: Controllable Unsupervised Domain Adaptation for Gaze and Head Redirection
However, adopting such generative models to new domains while maintaining their ability to provide fine-grained control over different image attributes, \eg, gaze and head pose directions, has been a challenging problem.
Assessing Multilingual Fairness in Pre-trained Multimodal Representations
To answer these questions, we view language as the fairness recipient and introduce two new fairness notions, multilingual individual fairness and multilingual group fairness, for pre-trained multimodal models.
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation
Automatic evaluations for natural language generation (NLG) conventionally rely on token-level or embedding-level comparisons with text references.
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task.
Language-Driven Image Style Transfer
We propose contrastive language visual artist (CLVA) that learns to extract visual semantics from style instructions and accomplish LDAST by the patch-wise style discriminator.
M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformers
LBVE contains two features: 1) the scenario of the source video is preserved instead of generating a completely different video; 2) the semantic is presented differently in the target video, and all changes are controlled by the given instruction.
Diagnosing Vision-and-Language Navigation: What Really Matters
Results show that indoor navigation agents refer to both object and direction tokens when making decisions.
L2C: Describing Visual Differences Needs Semantic Understanding of Individuals
Recent advances in language and vision push forward the research of captioning a single image to describing visual differences between image pairs.
Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations
A major challenge in visually grounded language generation is to build robust benchmark datasets and models that can generalize well in real-world settings.
Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigation
Vision-and-Language Navigation (VLN) is a natural language grounding task where an agent learns to follow language instructions and navigate to specified destinations in real-world environments.
Ranked #3 on
Vision and Language Navigation
on Touchdown Dataset
SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning
In this paper, we introduce a Self-Supervised Counterfactual Reasoning (SSCR) framework that incorporates counterfactual thinking to overcome data scarcity.
Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation
Outdoor vision-and-language navigation (VLN) is such a task where an agent follows natural language instructions and navigates a real-life urban environment.
Ranked #4 on
Vision and Language Navigation
on Touchdown Dataset
(using extra training data)
Environment-agnostic Multitask Learning for Natural Language Grounded Navigation
Recent research efforts enable study for natural language grounded navigation in photo-realistic environments, e. g., following natural language instructions or dialog.
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling
In particular, we present a model-agnostic adversarial path sampler (APS) that learns to sample challenging paths that force the navigator to improve based on the navigation performance.
Cross-Lingual Vision-Language Navigation
Commanding a robot to navigate with natural language instructions is a long-term goal for grounded language understanding and robotics.
