Search Results for author:
Found 52 papers, 36 papers with code
Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use
In real-world scenarios, it is desirable for embodied agents to have the ability to leverage human language to gain explicit or implicit knowledge for learning tasks.
Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities
Spatial expressions in situated communication can be ambiguous, as their meanings vary depending on the frames of reference (FoR) adopted by speakers and listeners.
RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning
Developing robust and correctable visuomotor policies for robotic manipulation is challenging due to the lack of self-recovery mechanisms from failures and the limitations of simple language instructions in guiding robot actions.
Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models
Vision-and-Language Navigation (VLN) has gained increasing attention over recent years and many approaches have emerged to advance their development.
Multi-Object Hallucination in Vision-Language Models
Large vision language models (LVLMs) often suffer from object hallucination, producing objects not present in the given images.
Towards Bidirectional Human-AI Alignment: A Systematic Review for Clarifications, Framework, and Future Directions
From this, we present a conceptual framework of "Bidirectional Human-AI Alignment" to organize the literature from a human-centered perspective.
3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination
The integration of language and 3D perception is crucial for developing embodied agents and robots that comprehend and interact with the physical world.
LinkGPT: Teaching Large Language Models To Predict Missing Links
To address the efficiency challenges at inference time, we introduce a retrieval-reranking scheme.
DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiences
Recent advancements in foundation models (FMs) have unlocked new prospects in autonomous driving, yet the experimental settings of these studies are preliminary, over-simplified, and fail to capture the complexity of real-world driving scenarios in human environments.
Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations
In this work, we aim to examine how corrective feedback from interactions influences neural language acquisition from the ground up through systematically controlled experiments, assessing whether it contributes to learning efficiency in language models.
GROUNDHOG: Grounding Large Language Models to Holistic Segmentation
Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences of location tokens.
Ranked #2 on
Referring Expression Segmentation
on PhraseCut
Causal Language Modeling
Generalized Referring Expression Segmentation
+3
Inversion-Free Image Editing with Language-Guided Diffusion Models
Despite recent advances in inversion-based editing text-guided image manipulation remains challenging for diffusion models.
Inversion-Free Image Editing with Natural Language
We show that when the initial sample is known, a special variance schedule reduces the denoising step to the same form as the multi-step consistency sampling.
Ranked #1 on
Text-based Image Editing
on PIE-Bench
Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties
Our results, analysis, and \eilev{}-trained models yield numerous insights about the emergence of in-context learning over video and text, creating a foundation for future work to optimize and scale VLMs for open-domain video understanding and reasoning.
GIPCOL: Graph-Injected Soft Prompting for Compositional Zero-Shot Learning
In this work, we propose GIP-COL (Graph-Injected Soft Prompting for COmpositional Learning) to better explore the compositional zero-shot learning (CZSL) ability of VLMs within the prompt-based learning framework.
MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition
Inspired by this observation, in this paper, we propose MetaReVision, a retrieval-enhanced meta-learning model to address the visually grounded compositional concept learning problem.
Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?
Despite tremendous advances in AI, it remains a significant challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks.
Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?
Vision-Language Models (VLMs) are trained on vast amounts of data captured by humans emulating our understanding of the world.
Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models
Such situated evaluation provides a more comprehensive assessment of mental states and potentially mitigates the risk of shortcuts and data leakage.
From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning
We incorporate these interlinked dual processes in fine-tuning and in-context learning with PLMs, applying them to two language understanding tasks that require coherent physical commonsense reasoning.
CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulation
Our empirical studies show that Cyclenet is superior in translation consistency and quality, and can generate high-quality images for out-of-domain distributions with a simple change of the textual prompt.
Think, Act, and Ask: Open-World Interactive Personalized Robot Navigation
To address these limitations, we introduce Zero-shot Interactive Personalized Object Navigation (ZIPON), where robots need to navigate to personalized goal objects while engaging in conversations with users.
LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent
While existing approaches often rely on extensive labeled data or exhibit limitations in handling complex language queries, we propose LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model (LLM)-based 3D visual grounding pipeline.
Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition
Human language acquisition is an efficient, supervised, and continual process.
World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models
The ability to connect language units to their referents in the physical world, referred to as grounding, is crucial to learning and understanding grounded meanings of words.
In-Context Analogical Reasoning with Pre-Trained Language Models
Analogical reasoning is a fundamental capacity of human cognition that allows us to reason abstractly about novel situations by relating them to past experiences.
NLP Reproducibility For All: Understanding Experiences of Beginners
As natural language processing (NLP) has recently seen an unprecedented level of excitement, and more people are eager to enter the field, it is unclear whether current research reproducibility efforts are sufficient for this group of beginners to apply the latest developments.
Towards Collaborative Plan Acquisition through Theory of Mind Modeling in Situated Dialogue
To complete these tasks, agents need to engage in situated communication with their partners and coordinate their partial plans towards a complete plan to achieve a joint task goal.
BAD: BiAs Detection for Large Language Models in the context of candidate screening
Application Tracking Systems (ATS) have allowed talent managers, recruiters, and college admissions committees to process large volumes of potential candidate applications efficiently.
Ranked #1 on
Bias Detection
on ICAT LLM bias
Prompting Large Pre-trained Vision-Language Models For Compositional Concept Learning
This work explores the zero-shot compositional learning ability of large pre-trained vision-language models(VLMs) within the prompt-based learning framework and propose a model (\textit{PromptCompVL}) to solve the compositonal zero-shot learning (CZSL) problem.
DANLI: Deliberative Agent for Following Natural Language Instructions
These reactive agents are insufficient for long-horizon complex tasks.
DOROTHIE: Spoken Dialogue for Handling Unexpected Situations in Interactive Autonomous Driving Agents
To this end, we introduce Dialogue On the ROad To Handle Irregular Events (DOROTHIE), a novel interactive simulation platform that enables the creation of unexpected situations on the fly to support empirical studies on situated communication with autonomous driving agents.
Reproducibility Beyond the Research Community: Experience from NLP Beginners
As NLP research attracts public attention and excitement, it becomes increasingly important for it to be accessible to a broad audience.
Learning to Mediate Disparities Towards Pragmatic Communication
The PRS attempts to learn the speaker-listener disparity and adjust the speech accordingly, by adding a light-weighted disparity adjustment layer into working memory on top of speaker's long-term memory system.
Partition-Based Active Learning for Graph Neural Networks
We study the problem of semi-supervised learning with Graph Neural Networks (GNNs) in an active learning setup.
MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks
An ideal integration of autonomous agents in a human world implies that they are able to collaborate on human terms.
Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers
As large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks, statistical bias in benchmark data and probing studies have recently called into question their true capabilities.
Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding
However, evaluations only based on end task performance shed little light on machines' true ability in language understanding and reasoning.
CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Models
More concretely, our CX-ToM framework generates sequence of explanations in a dialog by mediating the differences between the minds of machine and human user.
Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring
On the ALFRED benchmark for task learning, the published state-of-the-art system only achieves a task success rate of less than 10% in an unseen environment, compared to the human performance of over 90%.
Experience Grounds Language
Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates.
Commonsense Justification for Action Explanation
To enable collaboration and communication between humans and agents, this paper investigates learning to acquire commonsense evidence for action justification.
What Action Causes This? Towards Naive Physical Action-Effect Prediction
Despite recent advances in knowledge representation, automated reasoning, and machine learning, artificial agents still lack the ability to understand basic action-effect relations regarding the physical world, for example, the action of cutting a cucumber most likely leads to the state where the cucumber is broken apart into smaller pieces.
Interactive Learning of Grounded Verb Semantics towards Human-Robot Communication
To enable human-robot communication and collaboration, previous works represent grounded verb semantics as the potential change of state to the physical world caused by these verbs.
