Search Results for author:
Found 24 papers, 11 papers with code
NICE: Neural Image Commenting with Empathy
Based on these annotations, we define two different tasks for the NICE dataset.
Position Paper: Agent AI Towards a Holistic Intelligence
Recent advancements in large foundation models have remarkably enhanced our understanding of sensory information in open-world environments.
An Interactive Agent Foundation Model
We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents across a wide range of domains, datasets, and tasks.
Agent AI: Surveying the Horizons of Multimodal Interaction
To accelerate research on agent-based multimodal intelligence, we define "Agent AI" as a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data, and can produce meaningful embodied actions.
Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Empirical results and human evaluations in a zero-shot setup demonstrate that our distillation method results in more precise VL models of reasoning compared to a baseline of passing a generated referring expression to an LLM.
MindAgent: Emergent Gaming Interaction
Large Language Models (LLMs) have the capacity of performing complex scheduling in a multi-agent system and can coordinate these agents into completing sophisticated tasks that require extensive collaboration.
ArK: Augmented Reality with Knowledge Interactive Emergent Ability
In this study, we develop an infinite agent that learns to transfer knowledge memory from general foundation models (e. g. GPT4, DALLE) to novel domains or scenarios for scene understanding and generation in the physical or virtual world.
Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
Large language models (LLMs), such as ChatGPT, are able to generate human-like, fluent responses for many downstream tasks, e. g., task-oriented dialog and question answering.
Training Vision-Language Transformers from Captions
Vision-Language Transformers can be learned without low-level human labels (e. g. class labels, bounding boxes, etc).
KAT: A Knowledge Augmented Transformer for Vision-and-Language
The primary focus of recent work with largescale transformers has been on optimizing the amount of information packed into the model's parameters.
Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations
The encoder of TP-N2F employs TPR `binding' to encode natural-language symbolic structure in vector space and the decoder uses TPR `unbinding' to generate, in symbolic space, a sequential program represented by relational tuples, each consisting of a relation (or operation) and a number of arguments.
Natural- to formal-language generation using Tensor Product Representations
Generating formal-language represented by relational tuples, such as Lisp programs or mathematical expressions, from a natural-language input is an extremely challenging task because it requires to explicitly capture discrete symbolic structural information from the input to generate the output.
REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning
In this study, we present a fine-grained evaluation method REO for automatically measuring the performance of image captioning systems.
TIGEr: Text-to-Image Grounding for Image Caption Evaluation
This paper presents a new metric called TIGEr for the automatic evaluation of image captioning systems.
Object-driven Text-to-Image Synthesis via Adversarial Training
In this paper, we propose Object-driven Attentive Generative Adversarial Newtorks (Obj-GANs) that allow object-centered text-to-image synthesis for complex scenes.
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments.
Ranked #2 on
Vision-Language Navigation
on Room2Room
Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation
We propose a hierarchically structured reinforcement learning approach to address the challenges of planning for generating coherent multi-sentence stories for the visual storytelling task.
Ranked #24 on
Visual Storytelling
on VIST
Turbo Learning for Captionbot and Drawingbot
We study in this paper the problems of both image captioning and text-to-image generation, and present a novel turbo learning approach to jointly training an image-to-text generator (a. k. a.
Generating Diverse and Accurate Visual Captions by Comparative Adversarial Learning
By contrasting with human-written captions and image-mismatched captions, the caption generator effectively exploits the inherent characteristics of human languages, and generates more discriminative captions.
Attentive Tensor Product Learning
This paper proposes a new architecture - Attentive Tensor Product Learning (ATPL) - to represent grammatical structures in deep learning models.
Structured Memory based Deep Model to Detect as well as Characterize Novel Inputs
While deep learning has pushed the boundaries in various machine learning tasks, the current models are still far away from replicating many functions that a normal human brain can do.
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
In this paper, we propose an Attentional Generative Adversarial Network (AttnGAN) that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation.
Ranked #1 on
Text-to-Image Generation
on MS-COCO
A Neural-Symbolic Approach to Design of CAPTCHA
To address this, this paper promotes image/visual captioning based CAPTCHAs, which is robust against machine-learning-based attacks.
Tensor Product Generation Networks for Deep NLP Modeling
We present a new approach to the design of deep networks for natural language processing (NLP), based on the general technique of Tensor Product Representations (TPRs) for encoding and processing symbol structures in distributed neural networks.
