Search Results for author:
Found 322 papers, 190 papers with code
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks.
Ranked #1 on
Common Sense Reasoning
on SWAG
On the Variance of the Adaptive Learning Rate and Beyond
The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam.
MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition
In this paper, we design a benchmark task and provide the associated datasets for recognizing face images and link them to corresponding entity keys in a knowledge base.
Embedding Entities and Relations for Learning and Inference in Knowledge Bases
We consider learning representations of entities and relations in KBs using the neural-embedding approach.
Ranked #10 on
Link Prediction
on UMLS
Unified Language Model Pre-training for Natural Language Understanding and Generation
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks.
Ranked #2 on
Generative Question Answering
on CoQA
(using extra training data)
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Ranked #4 on
Question Generation
on SQuAD1.1
(using extra training data)
Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
MIND: A Large-scale Dataset for News Recommendation
News recommendation is an important technique for personalized news service.
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
We find that scaling LMM consistently enhances model performance and improves language capabilities, and performance of LoRA/QLoRA tuning of LMM are comparable to the performance of full-model fine-tuning.
Ranked #47 on
Visual Question Answering
on MM-Vet
Segment Everything Everywhere All at Once
In SEEM, we propose a novel decoding mechanism that enables diverse prompting for all types of segmentation tasks, aiming at a universal segmentation interface that behaves like large language models (LLMs).
Focal Modulation Networks
For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2. 4, and beats Swin at multi-scale (50. 5 v. s.
Ranked #8 on
Object Detection
on COCO minival
(using extra training data)
Learning deep structured semantic models for web search using clickthrough data
The proposed deep structured semantic models are discriminatively trained by maximizing the conditional likelihood of the clicked documents given a query using the clickthrough data.
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
We present Set-of-Mark (SoM), a new visual prompting method, to unleash the visual grounding abilities of large multimodal models (LMMs), such as GPT-4V.
LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing
LLaVA-Interactive is a research prototype for multimodal human-AI interaction.
Instruction Tuning with GPT-4
Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks, and no human-written instructions are needed.
DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
We present a large, tunable neural conversational response generation model, DialoGPT (dialogue generative pre-trained transformer).
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
We present a large, tunable neural conversational response generation model, DIALOGPT (dialogue generative pre-trained transformer).
Multi-Task Deep Neural Networks for Natural Language Understanding
In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks.
Ranked #2 on
Natural Language Inference
on SciTail
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
This paper explores the use of knowledge distillation to improve a Multi-Task Deep Neural Network (MT-DNN) (Liu et al., 2019) for learning text representations across multiple natural language understanding tasks.
Ranked #1 on
Semantic Textual Similarity
on SentEval
A Hybrid Neural Network Model for Commonsense Reasoning
An HNN consists of two component models, a masked language model and a semantic similarity model, which share a BERT-based contextual encoder but use different model-specific input and output layers.
Ranked #1 on
Natural Language Understanding
on PDP60
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and forget the knowledge of the pre-trained model.
Ranked #1 on
Natural Language Inference
on QNLI
The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models.
Adversarial Training for Large Neural Language Models
In natural language processing (NLP), pre-training large neural language models such as BERT have demonstrated impressive gain in generalization for a variety of tasks, with further improvement from adversarial fine-tuning.
Ranked #6 on
Natural Language Inference
on ANLI test
(using extra training data)
Posterior Differential Regularization with f-divergence for Improving Model Robustness
Theoretically, we provide a connection of two recent methods, Jacobian Regularization and Virtual Adversarial Training, under this framework.
Targeted Adversarial Training for Natural Language Understanding
We present a simple yet effective Targeted Adversarial Training (TAT) algorithm to improve adversarial training for natural language understanding.
Grounded Language-Image Pre-training
The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representation semantic-rich.
Ranked #1 on
2D Object Detection
on RF100
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models
In general, these language-augmented visual models demonstrate strong transferability to a variety of datasets and tasks.
Ranked #1 on
Object Detection
on ELEVATER
GLIPv2: Unifying Localization and Vision-Language Understanding
We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e. g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e. g., VQA, image captioning).
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Test
(using extra training data)
Semantic-SAM: Segment and Recognize Anything at Any Granularity
In this paper, we introduce Semantic-SAM, a universal image segmentation model to enable segment and recognize anything at any desired granularity.
Visual In-Context Prompting
In-context prompting in large language models (LLMs) has become a prevalent approach to improve zero-shot capabilities, but this idea is less explored in the vision domain.
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model.
Ranked #1 on
Natural Language Inference
on MRPC
Natural Language Inference
Natural Language Understanding
+2
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Z-Code++ creates new state of the art on 9 out of 13 text summarization tasks across 5 languages.
GLIGEN: Open-Set Grounded Text-to-Image Generation
Large-scale text-to-image diffusion models have made amazing advances.
Ranked #4 on
Conditional Text-to-Image Synthesis
on COCO-MIG
Generalized Decoding for Pixel, Image, and Language
We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly.
Ranked #4 on
Instance Segmentation
on ADE20K val
(using extra training data)
A Simple Framework for Open-Vocabulary Segmentation and Detection
We present OpenSeeD, a simple Open-vocabulary Segmentation and Detection framework that jointly learns from different segmentation and detection datasets.
Ranked #2 on
Instance Segmentation
on ADE20K val
(using extra training data)
Unified Vision-Language Pre-Training for Image Captioning and VQA
The model is unified in that (1) it can be fine-tuned for either vision-language generation (e. g., image captioning) or understanding (e. g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models.
Ranked #1 on
Image Captioning
on Flickr30k Captions test
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks.
Ranked #1 on
Image Retrieval
on MS COCO
(Recall@10 metric)
Focal Self-attention for Local-Global Interactions in Vision Transformers
With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers on a range of public image classification and object detection benchmarks.
Ranked #17 on
Instance Segmentation
on COCO test-dev
Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization ($\mu$P), many optimal HPs remain stable even as model size changes.
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters.
VinVL: Revisiting Visual Representations in Vision-Language Models
In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model \oscar \cite{li2020oscar}, and utilize an improved approach \short\ to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks.
Ranked #2 on
Image-text matching
on CommercialAdsDataset
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response.
Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years.
Multimodal Foundation Models: From Specialists to General-Purpose Assistants
This paper presents a comprehensive survey of the taxonomy and evolution of multimodal foundation models that demonstrate vision and vision-language capabilities, focusing on the transition from specialist models to general-purpose assistants.
A Diversity-Promoting Objective Function for Neural Conversation Models
Sequence-to-sequence neural network models for generation of conversational responses tend to generate safe, commonplace responses (e. g., "I don't know") regardless of the input.
GODEL: Large-Scale Pre-Training for Goal-Directed Dialog
We introduce GODEL (Grounded Open Dialogue Language Model), a large pre-trained language model for dialog.
ConvLab: Multi-Domain End-to-End Dialog System Platform
We present ConvLab, an open-source multi-domain end-to-end dialog system platform, that enables researchers to quickly set up experiments with reusable components and compare a large set of different approaches, ranging from conventional pipeline systems to end-to-end neural models, in common environments.
End-to-End Task-Completion Neural Dialogue Systems
One of the major drawbacks of modularized task-completion dialogue systems is that each module is trained individually, which presents several challenges.
A User Simulator for Task-Completion Dialogues
Then, one can train reinforcement learning agents in an online fashion as they interact with the simulator.
RegionCLIP: Region-based Language-Image Pretraining
However, we show that directly applying such models to recognize image regions for object detection leads to poor performance due to a domain shift: CLIP was trained to match an image as a whole to a text description, without capturing the fine-grained alignment between image regions and text spans.
Ranked #11 on
Open Vocabulary Object Detection
on MSCOCO
(using extra training data)
LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
LLaVA-Plus is a general-purpose multimodal assistant that expands the capabilities of large multimodal models.
Ranked #1 on
LMM real-life tasks
on Leaderboard
Deep Learning Based Text Classification: A Comprehensive Review
Deep learning based models have surpassed classical machine learning based approaches in various text classification tasks, including sentiment analysis, news categorization, question answering, and natural language inference.
Focal Attention for Long-Range Interactions in Vision Transformers
With focal attention, we propose a new variant of Vision Transformer models, called Focal Transformers, which achieve superior performance over the state-of-the-art (SoTA) Vision Transformers on a range of public image classification and object detection benchmarks.
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models
Sora is a text-to-video generative AI model, released by OpenAI in February 2024.
CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search
Neural rankers based on deep pretrained language models (LMs) have been shown to improve many information retrieval benchmarks.
ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems
We present ConvLab-2, an open-source toolkit that enables researchers to build task-oriented dialogue systems with state-of-the-art models, perform an end-to-end evaluation, and diagnose the weakness of systems.
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding
This paper presents a new Vision Transformer (ViT) architecture Multi-Scale Vision Longformer, which significantly enhances the ViT of \cite{dosovitskiy2020image} for encoding high-resolution images using two techniques.
Ranked #45 on
Instance Segmentation
on COCO minival
Efficient Self-supervised Vision Transformers for Representation Learning
This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning.
Ranked #16 on
Self-Supervised Image Classification
on ImageNet
Representation Learning
Self-Supervised Image Classification
Image Scene Graph Generation (SGG) Benchmark
There is a surge of interest in image scene graph generation (object, attribute and relationship detection) due to the need of building fine-grained image understanding models that go beyond object detection.
Florence: A New Foundation Model for Computer Vision
Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications.
Ranked #1 on
Action Recognition In Videos
on Kinetics-600
Unified Contrastive Learning in Image-Text-Label Space
Particularly, it attains gains up to 9. 2% and 14. 5% in average on zero-shot recognition benchmarks over the language-image contrastive learning and supervised learning methods, respectively.
K-LITE: Learning Transferable Visual Models with External Knowledge
We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts.
Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
We hope that our first pre-trained big VAE language model itself and results can help the NLP community renew the interests of deep generative models in the era of large-scale pre-training, and make these principled methods more practical.
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering.
Understanding the Difficulty of Training Transformers
Transformers have proved effective in many NLP tasks.
Ranked #5 on
Machine Translation
on WMT2014 English-French
Very Deep Transformers for Neural Machine Translation
We explore the application of very deep Transformer models for Neural Machine Translation (NMT).
Ranked #1 on
Machine Translation
on WMT2014 English-French
(using extra training data)
Object-driven Text-to-Image Synthesis via Adversarial Training
In this paper, we propose Object-driven Attentive Generative Adversarial Newtorks (Obj-GANs) that allow object-centered text-to-image synthesis for complex scenes.
TrustLLM: Trustworthiness in Large Language Models
This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions.
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
To address this issue, we have created GVC data that allows for the combination of grounding and chat capabilities.
StoryGAN: A Sequential Conditional GAN for Story Visualization
We therefore propose a new story-to-image-sequence generation model, StoryGAN, based on the sequential conditional GAN framework.
Deep Reinforcement Learning for Dialogue Generation
Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be shortsighted, predicting utterances one at a time while ignoring their influence on future outcomes.
Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems
This proposal introduces a Dialogue Challenge for building end-to-end task-completion dialogue systems, with the goal of encouraging the dialogue research community to collaborate and benchmark on standard datasets and unified experimental environment.
Few-shot Natural Language Generation for Task-Oriented Dialog
It is pre-trained on a large set of annotated NLG corpus to acquire the controllable generation ability, and fine-tuned with only a few domain-specific labels to adapt to new domains.
Ranked #4 on
Data-to-Text Generation
on MULTIWOZ 2.1
Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access
In this paper, we address this limitation by replacing symbolic queries with an induced "soft" posterior distribution over the KB that indicates which entities the user is interested in.
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
To bridge this gap, we present MathVista, a benchmark designed to combine challenges from diverse mathematical and visual tasks.
A Knowledge-Grounded Neural Conversation Model
We generalize the widely-used Seq2Seq approach by conditioning responses on both conversation history and external "facts", allowing the model to be versatile and applicable in an open-domain setting.
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
Variational autoencoders (VAEs) with an auto-regressive decoder have been applied for many natural language processing (NLP) tasks.
Feature Quantization Improves GAN Training
The instability in GAN training has been a long-standing problem despite remarkable research efforts.
Ranked #1 on
Conditional Image Generation
on CIFAR-100
Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning
During dialogue policy learning, the world model is constantly updated with real user experience to approach real user behavior, and in turn, the dialogue agent is optimized using both real experience and simulated experience.
Reinforcement Learning (RL)
Task-Completion Dialogue Policy Learning
From Captions to Visual Concepts and Back
The language model learns from a set of over 400, 000 image descriptions to capture the statistics of word usage.
Ranked #1 on
Image Captioning
on COCO Captions test
XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
While deep and large pre-trained models are the state-of-the-art for various natural language processing tasks, their huge size poses significant challenges for practical uses in resource constrained settings.
Stochastic Answer Networks for Machine Reading Comprehension
We propose a simple yet robust stochastic answer network (SAN) that simulates multi-step reasoning in machine reading comprehension.
Ranked #24 on
Question Answering
on SQuAD1.1 dev
Stochastic Answer Networks for Natural Language Inference
We propose a stochastic answer network (SAN) to explore multi-step inference strategies in Natural Language Inference.
Ranked #32 on
Natural Language Inference
on SNLI
Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
We propose a multi-task learning framework to learn a joint Machine Reading Comprehension (MRC) model that can be applied to a wide range of MRC tasks in different domains.
Stochastic Answer Networks for SQuAD 2.0
This paper presents an extension of the Stochastic Answer Network (SAN), one of the state-of-the-art machine reading comprehension models, to be able to judge whether a question is unanswerable or not.
Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge
We propose a novel open-domain question answering (ODQA) framework for answering single/multi-hop questions across heterogeneous knowledge sources.
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Vision-language (VL) pre-training has recently received considerable attention.
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Dev
Learning Customized Visual Models with Retrieval-Augmented Knowledge
Image-text contrastive learning models such as CLIP have demonstrated strong task transfer ability.
Ranked #1 on
Semi-Supervised Image Classification
on ImageNet - 1% labeled data
(using extra training data)
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models.
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models.
RetGen: A Joint framework for Retrieval and Grounded Text Generation Modeling
We propose a framework that alleviates this data constraint by jointly training a grounded generator and document retriever on the language model signal.
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
In particular, we focus on the task of Commonsense Reasoning, demonstrating that the proposed external attention mechanism can augment existing transformer models and significantly improve the model's reasoning capabilities.
Ranked #1 on
Common Sense Reasoning
on CommonsenseQA
(using extra training data)
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
We first benchmark MM-Navigator on our collected iOS screen dataset.
HittER: Hierarchical Transformers for Knowledge Graph Embeddings
Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from outputs of the bottom block.
Ranked #1 on
Link Prediction
on FB15k-237
(Hit@10 metric)
Stacked Attention Networks for Image Question Answering
Thus, we develop a multiple-layer SAN in which we query an image multiple times to infer the answer progressively.
Ranked #5 on
Visual Question Answering (VQA)
on VQA v1 test-std
ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format
Task-oriented dialogue (TOD) systems function as digital assistants, guiding users through various tasks such as booking flights or finding restaurants.
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs
Therefore, training an effective generalist biomedical model requires high-quality multimodal data, such as parallel image-text pairs.
Ranked #3 on
Medical Visual Question Answering
on SLAKE-English
Learning a Decision Tree Algorithm with Transformers
We then train MetaTree to produce the trees that achieve strong generalization performance.
Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs
In human-written articles, we often leverage the subtleties of text style, such as bold and italics, to guide the attention of readers.
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
By training on a large amount of image-text-action triplets in a self-supervised learning manner, the pre-trained model provides generic representations of visual environments and language instructions.
Ranked #1 on
Visual Navigation
on Help, Anna! (HANNA)
PIQA: Reasoning about Physical Commonsense in Natural Language
Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems.
Ranked #36 on
Question Answering
on PIQA
Natural Language Understanding
Physical Commonsense Reasoning
+1
Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading
Although neural conversation models are effective in learning how to produce fluent responses, their primary challenge lies in knowing what to say to make the conversation contentful and non-vacuous.
Data Augmentation for Spoken Language Understanding via Pretrained Language Models
The training of spoken language understanding (SLU) models often faces the problem of data scarcity.
SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching
We present a new method SOLOIST that uses transfer learning and machine teaching to build task bots at scale.
Ranked #4 on
End-To-End Dialogue Modelling
on MULTIWOZ 2.0
Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learning
We present a bi-encoder framework for named entity recognition (NER), which applies contrastive learning to map candidate text spans and entity types into the same vector representation space.
Ranked #1 on
Named Entity Recognition (NER)
on BC5CDR
End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager
Natural language understanding and dialogue policy learning are both essential in conversational systems that predict the next system actions in response to a current user utterance.
Augmenting Interpretable Models with LLMs during Training
Recent large language models (LLMs) have demonstrated remarkable prediction performance for a growing array of tasks.
Explaining Patterns in Data with Language Models via Interpretable Autoprompting
Large language models (LLMs) have displayed an impressive ability to harness natural language to perform complex tasks.
Explaining black box text modules in natural language with language models
Here, we ask whether we can automatically obtain natural language explanations for black box text modules.
Rethinking Interpretability in the Era of Large Language Models
We highlight two emerging research priorities for LLM interpretation: using LLMs to directly analyze new datasets and to generate interactive explanations.
Semantic Compositional Networks for Visual Captioning
The degree to which each member of the ensemble is used to generate an image caption is tied to the image-dependent probability of the corresponding tag.
Generation-Augmented Retrieval for Open-domain Question Answering
We demonstrate that the generated contexts substantially enrich the semantics of the queries and GAR with sparse representations (BM25) achieves comparable or better performance than state-of-the-art dense retrieval methods such as DPR.
Ranked #9 on
Passage Retrieval
on Natural Questions
Rider: Reader-Guided Passage Reranking for Open-Domain Question Answering
Current open-domain question answering systems often follow a Retriever-Reader architecture, where the retriever first retrieves relevant passages and the reader then reads the retrieved passages to form an answer.
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training.
PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking
We propose the task of outline-conditioned story generation: given an outline as a set of phrases that describe key characters and events to appear in a story, the task is to generate a coherent narrative that is consistent with the provided outline.
Few-Shot Generative Conversational Query Rewriting
Conversational query rewriting aims to reformulate a concise conversational query to a fully specified, context-independent query that can be effectively handled by existing information retrieval systems.
Structuring Latent Spaces for Stylized Response Generation
This structure allows the system to generate stylized relevant responses by sampling in the neighborhood of the conversation model prediction, and continuously control the style level.
Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
We present the Frontier Aware Search with backTracking (FAST) Navigator, a general framework for action decoding, that achieves state-of-the-art results on the Room-to-Room (R2R) Vision-and-Language navigation challenge of Anderson et.
Ranked #3 on
Vision-Language Navigation
on Room2Room
Taming Sparsely Activated Transformer with Stochastic Experts
While most on-going research focuses on improving SAMs models by exploring methods of routing inputs to experts, our analysis reveals that such research might not lead to the solution we expect, i. e., the commonly-used routing methods based on gating mechanisms do not work better than randomly routing inputs to experts.
Implicit Deep Latent Variable Models for Text Generation
Deep latent variable models (LVM) such as variational auto-encoder (VAE) have recently played an important role in text generation.
Few-Shot Named Entity Recognition: A Comprehensive Study
This paper presents a comprehensive study to efficiently build named entity recognition (NER) systems when a small number of in-domain labeled data is available.
Visually-Augmented Language Modeling
With the visually-augmented context, VaLM uses a visual knowledge fusion layer to enable multimodal grounded language modeling by attending to both text context and visual knowledge in images.
Open Domain Question Answering with A Unified Knowledge Interface
The retriever-reader framework is popular for open-domain question answering (ODQA) due to its ability to use explicit knowledge.
Chain-of-Skills: A Configurable Model for Open-domain Question Answering
Our approach outperforms recent self-supervised retrievers in zero-shot evaluations and achieves state-of-the-art fine-tuned retrieval performance on NQ, HotpotQA and OTT-QA.
Ranked #4 on
Question Answering
on HotpotQA
TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics.
KAT: A Knowledge Augmented Transformer for Vision-and-Language
The primary focus of recent work with largescale transformers has been on optimizing the amount of information packed into the model's parameters.
Agent AI: Surveying the Horizons of Multimodal Interaction
To accelerate research on agent-based multimodal intelligence, we define "Agent AI" as a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data, and can produce meaningful embodied actions.
Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving
We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure.
Ranked #1 on
Question Answering
on Mathematics Dataset
EmailSum: Abstractive Email Thread Summarization
Furthermore, we find that widely used automatic evaluation metrics (ROUGE, BERTScore) are weakly correlated with human judgments on this email thread summarization task.
Ranked #1 on
Email Thread Summarization
on EmailSum (short)
Self-Verification Improves Few-Shot Clinical Information Extraction
Extracting patient information from unstructured text is a critical task in health decision-support and clinical research.
Efficient Long Sequence Modeling via State Space Augmented Transformer
Specifically, we augment a SSM into the bottom layer of SPADE, and we employ efficient local attention methods for the other layers.
Deep Reinforcement Learning with a Natural Language Action Space
This paper introduces a novel architecture for reinforcement learning with deep neural networks designed to handle state and action spaces characterized by natural language, as found in text-based games.
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
Furthermore, existing MoE works do not consider computational constraints (e. g., FLOPs, latency) to guide their design.
Towards Amortized Ranking-Critical Training for Collaborative Filtering
In this paper we investigate new methods for training collaborative filtering models based on actor-critic reinforcement learning, to directly optimize the non-differentiable quality metrics of interest.
Ranked #4 on
Recommendation Systems
on Million Song Dataset
RaCT: Toward Amortized Ranking-Critical Training For Collaborative Filtering
We investigate new methods for training collaborative filtering models based on actor-critic reinforcement learning, to more directly maximize ranking-based objective functions.
Training Vision-Language Transformers from Captions
Vision-Language Transformers can be learned without low-level human labels (e. g. class labels, bounding boxes, etc).
REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning
In this study, we present a fine-grained evaluation method REO for automatically measuring the performance of image captioning systems.
TIGEr: Text-to-Image Grounding for Image Caption Evaluation
This paper presents a new metric called TIGEr for the automatic evaluation of image captioning systems.
CLUES: Few-Shot Learning Evaluation in Natural Language Understanding
We demonstrate that while recent models reach human performance when they have access to large amounts of labeled data, there is a huge gap in performance in the few-shot setting for most tasks.
LiST: Lite Prompted Self-training Makes Parameter-Efficient Few-shot Learners
The first is the use of self-training to leverage large amounts of unlabeled data for prompt-based FN in few-shot settings.
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
Analysis shows that the proposed schedule indeed reduces the redundancy and improves generalization performance.
Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning
Given a text description, most existing semantic parsers synthesize a program in one shot.
Hierarchical Reinforcement Learning
reinforcement-learning
+2
Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning
This paper presents a Discriminative Deep Dyna-Q (D3Q) approach to improving the effectiveness and robustness of Deep Dyna-Q (DDQ), a recently proposed framework that extends the Dyna-Q algorithm to integrate planning for task-completion dialogue policy learning.
What Makes A Good Story? Designing Composite Rewards for Visual Storytelling
Previous storytelling approaches mostly focused on optimizing traditional metrics such as BLEU, ROUGE and CIDEr.
Ranked #10 on
Visual Storytelling
on VIST
Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutions
We show that our use of self-sampled correct and partially-correct solutions can benefit learning and help guide the sampling process, leading to more efficient exploration of the solution space.
Ranked #138 on
Arithmetic Reasoning
on GSM8K
Tree Prompting: Efficient Task Adaptation without Fine-Tuning
Prompting language models (LMs) is the main interface for applying them to new tasks.
WebQA: Multihop and Multimodal QA
Scaling Visual Question Answering (VQA) to the open-domain and multi-hop nature of web searches, requires fundamental advances in visual representation learning, knowledge aggregation, and language generation.
A Persona-Based Neural Conversation Model
We present persona-based models for handling the issue of speaker consistency in neural response generation.
A Hybrid Retrieval-Generation Neural Conversation Model
In this paper, we propose a hybrid neural conversation model that combines the merits of both response retrieval and generation methods.
Fault-Aware Neural Code Rankers
Large language models (LLMs) have demonstrated an impressive ability to generate code for various programming tasks.
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization
Responses generated by neural conversational models tend to lack informativeness and diversity.
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5.
Deep Reinforcement Learning with a Combinatorial Action Space for Predicting Popular Reddit Threads
We introduce an online popularity prediction and tracking task as a benchmark task for reinforcement learning with a combinatorial, natural language action space.
End-to-end Learning of LDA by Mirror-Descent Back Propagation over a Deep Architecture
We develop a fully discriminative learning approach for supervised Latent Dirichlet Allocation (LDA) model using Back Propagation (i. e., BP-sLDA), which maximizes the posterior probability of the prediction variable given the input document.
Consistent Dialogue Generation with Self-supervised Feature Learning
Generating responses that are consistent with the dialogue context is one of the central challenges in building engaging conversational agents.
CodeExp: Explanatory Code Document Generation
Our experiments show that (1) our refined training dataset lets models achieve better performance in the explanation generation tasks compared to larger unrefined data (15x larger), and (2) fine-tuned models can generate well-structured long docstrings comparable to human-written ones.
Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language
In this paper, we propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning.
Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations
The encoder of TP-N2F employs TPR `binding' to encode natural-language symbolic structure in vector space and the decoder uses TPR `unbinding' to generate, in symbolic space, a sequential program represented by relational tuples, each consisting of a relation (or operation) and a number of arguments.
A Controllable Model of Grounded Response Generation
Current end-to-end neural conversation models inherently lack the flexibility to impose semantic control in the response generation process, often resulting in uninteresting responses.
Data Augmentation for Abstractive Query-Focused Multi-Document Summarization
The progress in Query-focused Multi-Document Summarization (QMDS) has been limited by the lack of sufficient largescale high-quality training datasets.
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks.
Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization
On several syntactic and semantic probing tasks, we demonstrate the emergent structural information in the role vectors and improved syntactic interpretability in the TPR layer outputs.
Contrastive Multi-document Question Generation
In the SL stage, a single-document question generator is trained.
RMM: A Recursive Mental Model for Dialog Navigation
In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers.
RMM: A Recursive Mental Model for Dialogue Navigation
In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers.
Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering
Given its effectiveness on knowledge-intensive natural language processing tasks, dense retrieval models have become increasingly popular.
Guided Dialog Policy Learning without Adversarial Learning in the Loop
Reinforcement Learning (RL) methods have emerged as a popular choice for training an efficient and effective dialogue policy.
Guided Dialogue Policy Learning without Adversarial Learning in the Loop
Reinforcement learning methods have emerged as a popular choice for training an efficient and effective dialogue policy.
Compositional Processing Emerges in Neural Networks Solving Math Problems
A longstanding question in cognitive science concerns the learning mechanisms underlying compositionality in human cognition.
Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
In this paper, we present Adversarial GLUE (AdvGLUE), a new multi-task benchmark to quantitatively and thoroughly explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks.
Ranked #1 on
Adversarial Robustness
on AdvGLUE
Teaching Language Models to Self-Improve through Interactive Demonstrations
The self-improving ability of large language models (LLMs), enabled by prompting them to analyze and revise their own outputs, has garnered significant interest in recent research.
Language-Based Image Editing with Recurrent Attentive Models
First, we introduce a synthetic dataset, called CoSaL, to evaluate the end-to-end performance of our LBIE system.
Bi-directional Attention with Agreement for Dependency Parsing
We develop a novel bi-directional attention model for dependency parsing, which learns to agree on headword predictions from the forward and backward parsing directions.
Ranked #4 on
Chinese Dependency Parsing
on Chinese Pennbank
Switch-based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning
Training task-completion dialogue agents with reinforcement learning usually requires a large number of real user experiences.
Robust Navigation with Language Pretraining and Stochastic Sampling
Core to the vision-and-language navigation (VLN) challenge is building robust instruction representations and action decoding schemes, which can generalize well to previously unseen instructions and environments.
Differentiable Tree Operations Promote Compositional Generalization
To facilitate the learning of these symbolic sequences, we introduce a differentiable tree interpreter that compiles high-level symbolic tree operations into subsymbolic matrix operations on tensors.
ARCH: Efficient Adversarial Regularized Training with Caching
Adversarial regularization can improve model generalization in many natural language processing tasks.
Execution-based Evaluation for Data Science Code Generation Models
Code generation models can benefit data scientists' productivity by automatically generating code from context and text descriptions.
Is Self-Repair a Silver Bullet for Code Generation?
We hypothesize that this is because self-repair is bottlenecked by the model's ability to provide feedback on its own code; using a stronger model to artificially boost the quality of the feedback, we observe substantially larger performance gains.
Character-level Deep Conflation for Business Data Analytics
Connecting different text attributes associated with the same entity (conflation) is important in business data analytics since it could help merge two different tables in a database to provide a more comprehensive profile of an entity.
HUBERT Untangles BERT to Improve Transfer across NLP Tasks
We introduce HUBERT which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model.
Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Empirical results and human evaluations in a zero-shot setup demonstrate that our distillation method results in more precise VL models of reasoning compared to a baseline of passing a generated referring expression to an LLM.
Attribute Structuring Improves LLM-Based Evaluation of Clinical Text Summaries
Summarizing clinical text is crucial in health decision-support and clinical research.
Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
In this paper, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models.
Ranked #2 on
Participant Intervention Comparison Outcome Extraction
on EBM-NLP
(using extra training data)
Language Models as Inductive Reasoners
To this end, we propose a new paradigm (task) for inductive reasoning, which is to induce natural language rules from natural language facts, and create a dataset termed DEER containing 1. 2k rule-fact pairs for the task, where rules and facts are written in natural language.
OPERA: Harmonizing Task-Oriented Dialogs and Information Seeking Experience
Existing studies in conversational AI mostly treat task-oriented dialog (TOD) and question answering (QA) as separate tasks.
Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning
We propose explanation-consistency finetuning (EC-finetuning), a method that adapts LLMs to generate more consistent natural-language explanations on related examples.
The Neural Painter: Multi-Turn Image Generation
In this work we combine two research threads from Vision/ Graphics and Natural Language Processing to formulate an image generation task conditioned on attributes in a multi-turn setting.
Navigating with Graph Representations for Fast and Scalable Decoding of Neural Language Models
Neural language models (NLMs) have recently gained a renewed interest by achieving state-of-the-art performance across many natural language processing (NLP) tasks.
M-Walk: Learning to Walk over Graphs using Monte Carlo Tree Search
In order to effectively train the agent from sparse rewards, we combine MCTS with the neural policy to generate trajectories yielding more positive rewards.
Ranked #44 on
Link Prediction
on WN18RR
(Hits@3 metric)
Discourse-Aware Neural Rewards for Coherent Text Generation
In this paper, we investigate the use of discourse-aware rewards with reinforcement learning to guide a model to generate long, coherent text.
Link Prediction using Embedded Knowledge Graphs
Since large knowledge bases are typically incomplete, missing facts need to be inferred from observed facts in a task called knowledge base completion.
Subgoal Discovery for Hierarchical Dialogue Policy Learning
Experiments with simulated and real users show that our approach performs competitively against a state-of-the-art method that requires human-defined subgoals.
Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
We introduce intrinsic fear (IF), a learned reward shaping that guards DRL agents against periodic catastrophes.
Dynamic Fusion Networks for Machine Reading Comprehension
This paper presents a novel neural model - Dynamic Fusion Network (DFN), for machine reading comprehension (MRC).
Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning
This paper presents a new method --- adversarial advantage actor-critic (Adversarial A2C), which significantly improves the efficiency of dialogue policy learning in task-completion dialogue systems.
BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems.
BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems.
