Search Results for author:
Found 41 papers, 27 papers with code
GROUNDHOG: Grounding Large Language Models to Holistic Segmentation
Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences of location tokens.
Ranked #1 on
Generalized Referring Expression Segmentation
on gRefCOCO
(using extra training data)
Causal Language Modeling
Generalized Referring Expression Segmentation
+2
Inversion-Free Image Editing with Natural Language
We show that when the initial sample is known, a special variance schedule reduces the denoising step to the same form as the multi-step consistency sampling.
Ranked #1 on
Text-based Image Editing
on PIE-Bench
Efficient In-Context Learning in Vision-Language Models for Egocentric Videos
Recent advancements in text-only large language models (LLMs) have highlighted the benefit of in-context learning for adapting to new tasks with a few demonstrations.
GIPCOL: Graph-Injected Soft Prompting for Compositional Zero-Shot Learning
In this work, we propose GIP-COL (Graph-Injected Soft Prompting for COmpositional Learning) to better explore the compositional zero-shot learning (CZSL) ability of VLMs within the prompt-based learning framework.
MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition
Inspired by this observation, in this paper, we propose MetaReVision, a retrieval-enhanced meta-learning model to address the visually grounded compositional concept learning problem.
Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?
Despite tremendous advances in AI, it remains a significant challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks.
Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?
Vision-Language Models (VLMs) are trained on vast amounts of data captured by humans emulating our understanding of the world.
Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models
Such situated evaluation provides a more comprehensive assessment of mental states and potentially mitigates the risk of shortcuts and data leakage.
From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning
We incorporate these interlinked dual processes in fine-tuning and in-context learning with PLMs, applying them to two language understanding tasks that require coherent physical commonsense reasoning.
CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulation
Our empirical studies show that Cyclenet is superior in translation consistency and quality, and can generate high-quality images for out-of-domain distributions with a simple change of the textual prompt.
Think, Act, and Ask: Open-World Interactive Personalized Robot Navigation
To address these limitations, we introduce Zero-shot Interactive Personalized Object Navigation (ZIPON), where robots need to navigate to personalized goal objects while engaging in conversations with users.
LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent
While existing approaches often rely on extensive labeled data or exhibit limitations in handling complex language queries, we propose LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model (LLM)-based 3D visual grounding pipeline.
Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition
Human language acquisition is an efficient, supervised, and continual process.
World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models
The ability to connect language units to their referents in the physical world, referred to as grounding, is crucial to learning and understanding grounded meanings of words.
In-Context Analogical Reasoning with Pre-Trained Language Models
Analogical reasoning is a fundamental capacity of human cognition that allows us to reason abstractly about novel situations by relating them to past experiences.
NLP Reproducibility For All: Understanding Experiences of Beginners
As natural language processing (NLP) has recently seen an unprecedented level of excitement, and more people are eager to enter the field, it is unclear whether current research reproducibility efforts are sufficient for this group of beginners to apply the latest developments.
Towards Collaborative Plan Acquisition through Theory of Mind Modeling in Situated Dialogue
To complete these tasks, agents need to engage in situated communication with their partners and coordinate their partial plans towards a complete plan to achieve a joint task goal.
BAD: BiAs Detection for Large Language Models in the context of candidate screening
Application Tracking Systems (ATS) have allowed talent managers, recruiters, and college admissions committees to process large volumes of potential candidate applications efficiently.
Ranked #1 on
Bias Detection
on ICAT LLM bias
Prompting Large Pre-trained Vision-Language Models For Compositional Concept Learning
This work explores the zero-shot compositional learning ability of large pre-trained vision-language models(VLMs) within the prompt-based learning framework and propose a model (\textit{PromptCompVL}) to solve the compositonal zero-shot learning (CZSL) problem.
DANLI: Deliberative Agent for Following Natural Language Instructions
These reactive agents are insufficient for long-horizon complex tasks.
DOROTHIE: Spoken Dialogue for Handling Unexpected Situations in Interactive Autonomous Driving Agents
To this end, we introduce Dialogue On the ROad To Handle Irregular Events (DOROTHIE), a novel interactive simulation platform that enables the creation of unexpected situations on the fly to support empirical studies on situated communication with autonomous driving agents.
Reproducibility Beyond the Research Community: Experience from NLP Beginners
As NLP research attracts public attention and excitement, it becomes increasingly important for it to be accessible to a broad audience.
Learning to Mediate Disparities Towards Pragmatic Communication
The PRS attempts to learn the speaker-listener disparity and adjust the speech accordingly, by adding a light-weighted disparity adjustment layer into working memory on top of speaker's long-term memory system.
Partition-Based Active Learning for Graph Neural Networks
We study the problem of semi-supervised learning with Graph Neural Networks (GNNs) in an active learning setup.
MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks
An ideal integration of autonomous agents in a human world implies that they are able to collaborate on human terms.
Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers
As large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks, statistical bias in benchmark data and probing studies have recently called into question their true capabilities.
Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding
However, evaluations only based on end task performance shed little light on machines' true ability in language understanding and reasoning.
CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Models
More concretely, our CX-ToM framework generates sequence of explanations in a dialog by mediating the differences between the minds of machine and human user.
Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring
On the ALFRED benchmark for task learning, the published state-of-the-art system only achieves a task success rate of less than 10% in an unseen environment, compared to the human performance of over 90%.
Experience Grounds Language
Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates.
Commonsense Justification for Action Explanation
To enable collaboration and communication between humans and agents, this paper investigates learning to acquire commonsense evidence for action justification.
What Action Causes This? Towards Naive Physical Action-Effect Prediction
Despite recent advances in knowledge representation, automated reasoning, and machine learning, artificial agents still lack the ability to understand basic action-effect relations regarding the physical world, for example, the action of cutting a cucumber most likely leads to the state where the cucumber is broken apart into smaller pieces.
Interactive Learning of Grounded Verb Semantics towards Human-Robot Communication
To enable human-robot communication and collaboration, previous works represent grounded verb semantics as the potential change of state to the physical world caused by these verbs.
