Search Results for author:
Found 192 papers, 108 papers with code
XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation
In this paper, we introduce XGLUE, a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora, and evaluate their performance across a diverse set of cross-lingual tasks.
CULG: Commercial Universal Language Generation
Pre-trained language models (PLMs) have dramatically improved performance for many natural language processing (NLP) tasks in domains such as finance and healthcare.
Analytical Reasoning of Text
In this paper, we study the challenge of analytical reasoning of text and collect a new dataset consisting of questions from the Law School Admission Test from 1991 to 2016.
KFCNet: Knowledge Filtering and Contrastive Learning for Generative Commonsense Reasoning
Pre-trained language models have led to substantial gains over a broad range of natural language processing (NLP) tasks, but have been shown to have limitations for natural language generation tasks with high-quality requirements on the output, such as commonsense generation and ad keyword generation.
Step-Video-TI2V Technical Report: A State-of-the-Art Text-Driven Image-to-Video Generation Model
We present Step-Video-TI2V, a state-of-the-art text-driven image-to-video generation model with 30B parameters, capable of generating videos up to 102 frames based on both text and image inputs.
Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model
We present Step-Video-T2V, a state-of-the-art text-to-video pre-trained model with 30B parameters and the ability to generate videos up to 204 frames in length.
DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training
Diffusion Transformers (DiTs) have shown remarkable performance in generating high-quality videos.
Taming Teacher Forcing for Masked Autoregressive Video Generation
We introduce MAGI, a hybrid video generation framework that combines masked modeling for intra-frame generation with causal modeling for next-frame generation.
Alchemy: Amplifying Theorem-Proving Capability through Symbolic Mutation
Experimental results demonstrate the effectiveness of our approach, achieving a 5% absolute performance improvement on Leandojo benchmark.
Automated Proof Generation for Rust Code via Self-Evolution
SAFE also re-purposes the large number of synthesized incorrect proofs to train the self-debugging capability of the fine-tuned models, empowering them to fix incorrect proofs based on the verifier's feedback.
Contextualized Data-Wrangling Code Generation in Computational Notebooks
With CoCoMine, we construct CoCoNote, a dataset containing 58, 221 examples for Contextualized Data-wrangling Code generation in Notebooks.
AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition
With the advancement of generative models, the synthesis of different sensory elements such as music, visuals, and speech has achieved significant realism.
Rho-1: Not All Tokens Are What You Need
Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution.
LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models
In this work, we present a novel interactive application aimed towards understanding the internal mechanisms of large vision-language models.
Exploring the Mystery of Influential Data for Mathematical Reasoning
Additionally, we showcase the use of QaDS in creating efficient fine-tuning mixtures with various selection ratios, and analyze the quality of a wide range of open-source datasets, which can perform as a reference for future works on mathematical reasoning tasks.
PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion
The growing dependence on Large Language Models (LLMs) for finishing user instructions necessitates a comprehensive understanding of their robustness to complex task completion in real-world situations.
Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning
Large language models (LLMs) have shown great potential in complex reasoning tasks, yet their performance is often hampered by the scarcity of high-quality and reasoning-focused training datasets.
Ranked #51 on
Math Word Problem Solving
on MATH
Using Left and Right Brains Together: Towards Vision and Language Planning
Large Language Models (LLMs) and Large Multi-modality Models (LMMs) have demonstrated remarkable decision masking capabilities on a variety of tasks.
StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis
To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model's ability to capture the true semantic representation of visual scenes.
Voila-A: Aligning Vision-Language Models with User's Gaze Attention
In this paper, we introduce gaze information, feasibly collected by AR or VR devices, as a proxy for human attention to guide VLMs and propose a novel approach, Voila-A, for gaze alignment to enhance the interpretability and effectiveness of these models in real-world applications.
Competition-Level Problems are Effective LLM Evaluators
Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about these abilities and the potential data contamination problem recently.
PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion
Recent evaluations of Large Language Models (LLMs) have centered around testing their zero-shot/few-shot capabilities for basic natural language tasks and their ability to translate instructions into tool APIs.
EIPE-text: Evaluation-Guided Iterative Plan Extraction for Long-Form Narrative Text Generation
In this paper, we propose a new framework called Evaluation-guided Iterative Plan Extraction for long-form narrative text generation (EIPE-text), which extracts plans from the corpus of narratives and utilizes the extracted plans to construct a better planner.
Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency
We propose the Multi-Perspective Self-Consistency (MPSC) framework incorporating both inter- and intra-consistency across outputs from multiple perspectives.
ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
Large language models have made significant progress in various language tasks, yet they still struggle with complex mathematics.
Ranked #27 on
Math Word Problem Solving
on MATH
(using extra training data)
LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models
Graphic layout generation, a growing research field, plays a significant role in user engagement and information perception.
ORES: Open-vocabulary Responsible Visual Synthesis
In this work, we formalize a new task, Open-vocabulary Responsible Visual Synthesis (ORES), where the synthesis model is able to avoid forbidden visual concepts while allowing users to input any desired content.
GameEval: Evaluating LLMs on Conversational Games
In this paper, we propose GameEval, a novel approach to evaluating LLMs through goal-driven conversational games, overcoming the limitations of previous methods.
DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory
Our experiments validate the effectiveness of DragNUWA, demonstrating its superior performance in fine-grained control in video generation.
GroundNLQ @ Ego4D Natural Language Queries Challenge 2023
Motivated by this, we leverage a two-stage pre-training strategy to train egocentric feature extractors and the grounding model on video narrations, and further fine-tune the model on annotated data.
Constructing Multilingual Code Search Dataset Using Neural Machine Translation
Code search is a task to find programming codes that semantically match the given natural language queries.
LongCoder: A Long-Range Pre-trained Language Model for Code Completion
In this paper, we introduce a new task for code completion that focuses on handling long code input and propose a sparse Transformer model, called LongCoder, to address this task.
CMMLU: Measuring massive multitask language understanding in Chinese
As the capabilities of large language models (LLMs) continue to advance, evaluating their performance becomes increasingly crucial and challenging.
ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning
With only 4M VLP data, ManagerTower achieves superior performances on various downstream VL tasks, especially 79. 15% accuracy on VQAv2 Test-Std, 86. 56% IR@1 and 95. 64% TR@1 on Flickr30K.
Allies: Prompting Large Language Model with Beam Search
With the advance of large language models (LLMs), the research field of LLM applications becomes more and more popular and the idea of constructing pipelines to accomplish complex tasks by stacking LLM API calls come true.
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
In this paper, we show that strong performance can be achieved by a method we call Iter-RetGen, which synergizes retrieval and generation in an iterative manner.
Query Rewriting for Retrieval-Augmented Large Language Models
Furthermore, to better align the query to the frozen modules, we propose a trainable scheme for our pipeline.
Machine-Created Universal Language for Cross-lingual Transfer
There are two primary approaches to addressing cross-lingual transfer: multilingual pre-training, which implicitly aligns the hidden representations of various languages, and translate-test, which explicitly translates different languages into an intermediate language, such as English.
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Unlike these models, humans typically utilize external tools to cross-check and refine their initial content, like using a search engine for fact-checking, or a code interpreter for debugging.
AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance.
PROM: A Phrase-level Copying Mechanism with Pre-training for Abstractive Summarization
Based on the remarkable achievements of pre-trained language models in abstractive summarization, the copying mechanism has proved helpful by improving the factuality, stability, and overall performance.
Code Execution with Pre-trained Language Models
Code execution is a fundamental aspect of programming language semantics that reflects the exact behavior of the code.
Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models
By utilizing iterative bootstrapping, our approach enables LLMs to autonomously rectify errors, resulting in more precise and comprehensive reasoning chains.
Learning to Plan with Natural Language
To obtain it, we propose the Learning to Plan method, which involves two phases: (1) In the first learning task plan phase, it iteratively updates the task plan with new step-by-step solutions and behavioral instructions, which are obtained by prompting LLMs to derive from training error feedback.
Low-code LLM: Graphical User Interface over Large Language Models
By introducing this framework, we aim to bridge the gap between humans and LLMs, enabling more effective and efficient utilization of LLMs for complex tasks.
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92. 5% accuracy on the English test of the Chinese national college entrance exam.
Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data
Furthermore, we propose a new technique called Self-Distill with Feedback, to further improve the performance of the Baize models with feedback from ChatGPT.
AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
Many natural language processing (NLP) tasks rely on labeled data to train machine learning models with high performance.
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
On the other hand, there are also many existing models and systems (symbolic-based or neural-based) that can do some domain-specific tasks very well.
NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation.
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
To this end, We build a system called \textbf{Visual ChatGPT}, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple AI models with multi-steps.
Learning 3D Photography Videos via Self-supervised Diffusion on Single Images
3D photography renders a static image into a video with appealing 3D visual effects.
Ranked #1 on
Image Outpainting
on MSCOCO
Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval
Specifically, we propose a multilingual PLM called masked sentence model (MSM), which consists of a sentence encoder to generate the sentence representations, and a document encoder applied to a sequence of sentence vectors from a document.
Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models
However, the quality of the prompts depends on the demonstrations given to the models, and creating many of them by hand is costly.
CAPSTONE: Curriculum Sampling for Dense Retrieval with Document Expansion
The dual-encoder has become the de facto architecture for dense retrieval.
MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrievers
Pre-trained Transformers (\eg BERT) have been commonly used in existing dense retrieval methods for parameter initialization, and recent studies are exploring more effective pre-training tasks for further improving the quality of dense vectors.
LEAD: Liberal Feature-based Distillation for Dense Retrieval
Knowledge distillation is often used to transfer knowledge from a strong teacher model to a relatively weak student model.
CodeExp: Explanatory Code Document Generation
Our experiments show that (1) our refined training dataset lets models achieve better performance in the explanation generation tasks compared to larger unrefined data (15x larger), and (2) fine-tuned models can generate well-structured long docstrings comparable to human-written ones.
ReCo: Region-Controlled Text-to-Image Generation
Human evaluation on PaintSkill shows that ReCo is +19. 28% and +17. 21% more accurate in generating images with correct object count and spatial relationship than the T2I model.
Conditional Text-to-Image Synthesis
Layout-to-Image Generation
+1
GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation
We introduce GENIUS: a conditional text generation model using sketches as input, which can fill in the missing contexts for a given sketch (key information consisting of textual spans, phrases, or words, concatenated by mask tokens).
Execution-based Evaluation for Data Science Code Generation Models
Code generation models can benefit data scientists' productivity by automatically generating code from context and text descriptions.
An Efficient COarse-to-fiNE Alignment Framework @ Ego4D Natural Language Queries Challenge 2022
This technical report describes the CONE approach for Ego4D Natural Language Queries (NLQ) Challenge in ECCV 2022.
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Thus, we propose a simple ambiguous negatives sampling method, SimANS, which incorporates a new sampling probability distribution to sample more ambiguous negatives.
Metric-guided Distillation: Distilling Knowledge from the Metric to Ranker and Retriever for Generative Commonsense Reasoning
Commonsense generation aims to generate a realistic sentence describing a daily scene under the given concepts, which is very challenging, since it requires models to have relational reasoning and compositional generalization capabilities.
Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers
This paper presents ReasonFormer, a unified reasoning framework for mirroring the modular and compositional reasoning process of humans in complex decision-making.
Soft-Labeled Contrastive Pre-training for Function-level Code Representation
In this paper, we present \textbf{SCodeR}, a \textbf{S}oft-labeled contrastive pre-training framework with two positive sample construction methods to learn functional-level \textbf{Code} \textbf{R}epresentation.
Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis
Most existing pre-trained language representation models (PLMs) are sub-optimal in sentiment analysis tasks, as they capture the sentiment information from word-level while under-considering sentence-level information.
Mixed-modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQA
However, training an effective dense table-text retriever is difficult due to the challenges of table-text discrepancy and data sparsity problem.
HORIZON: High-Resolution Semantically Controlled Panorama Synthesis
Panorama synthesis endeavors to craft captivating 360-degree visual landscapes, immersing users in the heart of virtual worlds.
AdsCVLR: Commercial Visual-Linguistic Representation Modeling in Sponsored Search
Sponsored search advertisements (ads) appear next to search results when consumers look for products and services on search engines.
Ranked #3 on
Image-text matching
on CommercialAdsDataset
PROD: Progressive Distillation for Dense Retrieval
It is common that a better teacher model results in a bad student via distillation due to the nonnegligible gap between teacher and student.
CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding
This paper tackles an emerging and challenging problem of long video temporal grounding~(VTG) that localizes video moments related to a natural language (NL) query.
Improving Task Generalization via Unified Schema Prompt
Task generalization has been a long standing challenge in Natural Language Processing (NLP).
NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis
In this paper, we present NUWA-Infinity, a generative model for infinite visual synthesis, which is defined as the task of generating arbitrarily-sized high-resolution images or long-duration videos.
Ranked #1 on
Image Outpainting
on LHQC
Joint Generator-Ranker Learning for Natural Language Generation
Generate-then-rank is a widely used mechanism for text generation, where a generator produces multiple text candidates and a ranker chooses the best one among the text candidates.
BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning
Vision-Language (VL) models with the Two-Tower architecture have dominated visual-language representation learning in recent years.
Unsupervised Context Aware Sentence Representation Pretraining for Multi-lingual Dense Retrieval
On the multilingual sentence retrieval task Tatoeba, our model achieves new SOTA results among methods without using bilingual data.
A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation
To further illustrate the commercial value of our approach, we conduct experiments on three generation tasks in real-world advertisements applications.
LogiGAN: Learning Logical Reasoning via Adversarial Pre-training
We present LogiGAN, an unsupervised adversarial pre-training framework for improving logical reasoning abilities of language models.
ProQA: Structural Prompt-based Pre-training for Unified Question Answering
Furthermore, ProQA exhibits strong ability in both continual learning and transfer learning by taking the advantages of the structural prompt.
VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers
Breakthroughs in transformer-based models have revolutionized not only the NLP field, but also vision and multimodal systems.
Automating Code Review Activities by Large-Scale Pre-training
In this research, we focus on utilizing pre-training techniques for the tasks in the code review scenario.
Multi-View Document Representation Learning for Open-Domain Dense Retrieval
Second, to prevent multi-view embeddings from collapsing to the same one, we further propose a global-local loss with annealed temperature to encourage the multiple viewers to better align with different potential queries.
Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure
Our main conclusion is that the contribution of constituent order and word co-occurrence is limited, while the composition is more crucial to the success of cross-linguistic transfer.
ReACC: A Retrieval-Augmented Code Completion Framework
Code completion, which aims to predict the following code token(s) according to the code context, can improve the productivity of software development.
LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval
Experimental results show that LaPraDoR achieves state-of-the-art performance compared with supervised dense retrieval models, and further analysis reveals the effectiveness of our training strategy and objectives.
UniXcoder: Unified Cross-Modal Pre-training for Code Representation
Furthermore, we propose to utilize multi-modal contents to learn representation of code fragment with contrastive learning, and then align representations among programming languages using a cross-modal generation task.
NÜWA-LIP: Language Guided Image Inpainting with Defect-free VQGAN
Language guided image inpainting aims to fill in the defective regions of an image under the guidance of text while keeping non-defective regions unchanged.
CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search
For bimodal contrastive learning, we leverage the documentation and in-line comments of code to build code-text pairs.
Reasoning over Hybrid Chain for Table-and-Text Open Domain QA
CARP utilizes hybrid chain to model the explicit intermediate reasoning process across table and text for question answering.
Ranked #2 on
Question Answering
on OTT-QA
Learning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering
To consider the interdependent knowledge between contextual clips into the network inference, we propose a Siamese Sampling and Reasoning (SiaSamRea) approach, which consists of a siamese sampling mechanism to generate sparse and similar clips (i. e., siamese clips) from the same video, and a novel reasoning strategy for integrating the interdependent knowledge between contextual clips into the network.
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion
To cover language, image, and video at the same time for different scenarios, a 3D transformer encoder-decoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively.
Ranked #1 on
Text-to-Video Generation
on Kinetics
Adversarial Retriever-Ranker for dense text retrieval
To address these challenges, we present Adversarial Retriever-Ranker (AR2), which consists of a dual-encoder retriever plus a cross-encoder ranker.
XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge
The results on MLQA and NER exhibit the superiority of XLM-K in knowledge related tasks.
Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations
We study the problem of coarse-grained response selection in retrieval-based dialogue systems.
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation
Self-supervised vision-and-language pretraining (VLP) aims to learn transferable multi-modal representations from large-scale image-text data and to achieve strong performances on a broad scope of vision-language tasks after finetuning.
Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
While there are many efforts to extend the context window, we introduce an architecture-independent approach for leveraging the syntactic hierarchies of source code for incorporating entire file-level context into a fixed-length window.
KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
Pre-trained language models have led to substantial gains over a broad range of natural language processing (NLP) tasks, but have been shown to have limitations for natural language generation tasks with high-quality requirements on the output, such as commonsense generation and ad keyword generation.
Discovering Representation Sprachbund For Multilingual Pre-Training
Then we cluster all the target languages into multiple groups and name each group as a representation sprachbund.
Hybrid Reasoning Network for Video-based Commonsense Captioning
Firstly, we develop multi-commonsense learning for semantic-level reasoning by jointly training different commonsense types in a unified network, which encourages the interaction between the clues of multiple commonsense descriptions, event-wise captions and videos.
From LSAT: The Progress and Challenges of Complex Reasoning
Complex reasoning aims to draw a correct inference based on complex rules.
Control Image Captioning Spatially and Temporally
Moreover, the controllability and explainability of LoopCAG are validated by analyzing spatial and temporal sensitivity during the generation process.
Ranked #1 on
Image Captioning
on Localized Narratives
Compare to The Knowledge: Graph Neural Fake News Detection with External Knowledge
Specifically, we first construct a \textit{directed heterogeneous document graph} for each news incorporating topics and entities.
Learning to Complete Code with Sketches
To evaluate models, we consider both ROUGE as well as a new metric RegexAcc that measures success of generating completions matching long outputs with as few holes as possible.
GEM: A General Evaluation Benchmark for Multimodal Tasks
Comparing with existing multimodal datasets such as MSCOCO and Flicker30K for image-language tasks, YouCook2 and MSR-VTT for video-language tasks, GEM is not only the largest vision-language dataset covering image-language tasks and video-language tasks at the same time, but also labeled in multiple languages.
FastSeq: Make Sequence Generation Faster
Transformer-based models have made tremendous impacts in natural language generation.
CoSQA: 20,000+ Web Queries for Code Search and Question Answering
Finding codes given natural language query isb eneficial to the productivity of software developers.
EL-Attention: Memory Efficient Lossless Attention for Generation
Transformer model with multi-head attention requires caching intermediate results for efficient inference in generation tasks.
Poolingformer: Long Document Modeling with Pooling Attention
We first evaluate Poolingformer on two long sequence QA tasks: the monolingual NQ and the multilingual TyDi QA.
Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text
Logical reasoning of text requires understanding critical logical information in the text and performing inference over them.
Ranked #7 on
Reading Comprehension
on ReClor
GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
Generating videos from text is a challenging task due to its high computational requirements for training and infinite possible answers for evaluation.
Ranked #17 on
Text-to-Video Generation
on MSR-VTT
(CLIPSIM metric)
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
In this paper, we propose a CLIP4Clip model to transfer the knowledge of the CLIP model to video-language retrieval in an end-to-end manner.
Ranked #1 on
Text to Video Retrieval
on MSR-VTT
ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation
ProphetNet is a pre-training based natural language generation method which shows powerful performance on English text summarization and question generation tasks.
AR-LSAT: Investigating Analytical Reasoning of Text
Analytical reasoning is an essential and challenging task that requires a system to analyze a scenario involving a set of particular circumstances and perform reasoning over it to make conclusions.
Mask Attention Networks: Rethinking and Strengthen Transformer
We therefore introduce a new layer named dynamic mask attention network (DMAN) with a learnable mask matrix which is able to model localness adaptively.
Ranked #11 on
Machine Translation
on WMT2014 English-German
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Benchmark datasets have a significant impact on accelerating research in programming language tasks.
Ranked #1 on
Cloze Test
on CodeXGLUE - CT-maxmin
BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation.
Syntax-Enhanced Pre-trained Model
We study the problem of leveraging the syntactic structure of text to enhance pre-trained models such as BERT and RoBERTa.
Multi-level Alignment Pretraining for Multi-lingual Semantic Parsing
In this paper, we present a multi-level alignment pretraining method in a unified architecture formulti-lingual semantic parsing.
GLGE: A New General Language Generation Evaluation Benchmark
Multi-task benchmarks such as GLUE and SuperGLUE have driven great progress of pretraining and transfer learning in Natural Language Processing (NLP).
Machine Reasoning: Technology, Dilemma and Future
Machine reasoning research aims to build interpretable AI systems that can solve problems or draw conclusions from what they are told (i. e. facts and observations) and already know (i. e. models, common sense and knowledge) under certain constraints.
ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training
This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism.
ProphetNet-Ads: A Looking Ahead Strategy for Generative Retrieval Models in Sponsored Search Engine
We build a dataset from a real-word sponsored search engine and carry out experiments to analyze different generative retrieval models.
Neural Deepfake Detection with Factual Structure of Text
To address this, we propose a graph-based model that utilizes the factual structure of a document for deepfake detection of text.
Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space
In this paper, we propose a novel data augmentation method, referred to as Controllable Rewriting based Question Data Augmentation (CRQDA), for machine reading comprehension (MRC), question generation, and question-answering natural language inference tasks.
No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehension
The Natural Questions (NQ) benchmark set brings new challenges to Machine Reading Comprehension: the answers are not only at different levels of granularity (long and short), but also of richer types (including no-answer, yes/no, single-span and multi-span).
GRACE: Gradient Harmonized and Cascaded Labeling for Aspect-based Sentiment Analysis
Specifically, a cascaded labeling module is developed to enhance the interchange between aspect terms and improve the attention of sentiment tokens when labeling sentiment polarities.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
+4
GraphCodeBERT: Pre-training Code Representations with Data Flow
Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables.
Ranked #3 on
Type prediction
on ManyTypes4TypeScript
Tag and Correct: Question aware Open Information Extraction with Two-stage Decoding
The semi-structured answer has two advantages which are more readable and falsifiable compared to span answer.
Graph Neural News Recommendation with Unsupervised Preference Disentanglement
Furthermore, the learned representations are disentangled with latent preference factors by a neighborhood routing algorithm, which can enhance expressiveness and interpretability.
Evidence-Aware Inferential Text Generation with Vector Quantised Variational AutoEncoder
Generating inferential texts about an event in different perspectives requires reasoning over different contexts that the event occurs.
Ranked #1 on
Common Sense Reasoning
on Event2Mind test
(BLEU metric)
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training.
Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension
Natural Questions is a new challenging machine reading comprehension benchmark with two-grained answers, which are a long answer (typically a paragraph) and a short answer (one or more entities inside the long answer).
A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
Watching instructional videos are often used to learn about procedures.
RikiNet: Reading Wikipedia Pages for Natural Question Answering
The representations are then fed into the predictor to obtain the span of the short answer, the paragraph of the long answer, and the answer type in a cascaded manner.
Leveraging Declarative Knowledge in Text and First-Order Logic for Fine-Grained Propaganda Detection
We study the detection of propagandistic text fragments in news articles.
Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension
Multilingual pre-trained models could leverage the training data from a rich source language (such as English) to improve performance on low resource languages.
LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Network
The graph is used to obtain graph-enhanced contextual representations of words in Transformer-based architecture.
A Heterogeneous Graph with Factual, Temporal and Logical Knowledge for Question Answering Over Dynamic Contexts
We study question answering over a dynamic textual environment.
Pre-training Text Representations as Meta Learning
In this work, we introduce a learning algorithm which directly optimizes model's ability to learn text representations for effective learning of downstream tasks.
Diverse, Controllable, and Keyphrase-Aware: A Corpus and Method for News Multi-Headline Generation
Furthermore, we propose a simple and effective method to mine the keyphrases of interest in the news article and build a first large-scale keyphrase-aware news headline corpus, which contains over 180K aligned triples of $<$news article, headline, keyphrase$>$.
Inferential Text Generation with Multiple Knowledge Sources and Meta-Learning
In this work, we use multiple knowledge sources as fuels for the model.
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks.
XGPT: Cross-modal Generative Pre-Training for Image Captioning
While many BERT-based cross-modal pre-trained models produce excellent results on downstream understanding tasks like image-text retrieval and VQA, they cannot be applied to generation tasks directly.
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation tasks.
Ranked #1 on
Code Documentation Generation
on CodeSearchNet - Go
UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
However, most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks.
Ranked #2 on
Action Segmentation
on COIN
(using extra training data)
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa.
Ranked #1 on
Entity Typing
on Open Entity
ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism.
Ranked #6 on
Question Generation
on SQuAD1.1
(using extra training data)
Asking Clarification Questions in Knowledge-Based Question Answering
We provide representative baselines for these tasks and further introduce a coarse-to-fine model for clarification question generation.
Aggregating Bidirectional Encoder Representations Using MatchLSTM for Sequence Matching
Given a sentence pair, we extract the output representations of it from BERT.
Multi-Task Learning for Conversational Question Answering over a Large-Scale Knowledge Base
We consider the problem of conversational question answering over a large-scale knowledge base.
Interpretable Network Structure for Modeling Contextual Dependency
Then, we show that the lower bound of such a separation rank can reveal the quantitative relation between the network structure (e. g. depth/width) and the modeling ability for the contextual dependency.
Neural Semantic Parsing in Low-Resource Settings with Back-Translation and Meta-Learning
Neural semantic parsing has achieved impressive results in recent years, yet its success relies on the availability of large amounts of supervised data.
Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering
In this work, we propose to automatically extract evidence from heterogeneous knowledge sources, and answer questions based on the extracted evidence.
Ranked #14 on
Common Sense Reasoning
on CommonsenseQA
Reasoning Over Semantic-Level Graph for Fact Checking
We evaluate our system on FEVER, a benchmark dataset for fact checking, and find that rich structural information is helpful and both our graph-based mechanisms improve the accuracy.
Ranked #2 on
Fact Verification
on FEVER
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
On XNLI, 1. 8% averaged accuracy improvement (on 15 languages) is obtained.
Cross-Lingual Natural Language Inference
Cross-Lingual Question Answering
+2
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner.
Ranked #5 on
Image-to-Text Retrieval
on MS COCO
(Recall@10 metric)
Knowledge Aware Semantic Concept Expansion for Image-Text Matching
In this paper, we develop a Scene Concept Graph (SCG) by aggregating image scene graphs and extracting frequently co-occurred concept pairs as scene common-sense knowledge.
Joint Type Inference on Entities and Relations via Graph Convolutional Networks
We develop a new paradigm for the task of joint entity relation extraction.
Ranked #1 on
Relation Extraction
on ACE 2005
(Sentence Encoder metric)
Dense Procedure Captioning in Narrated Instructional Videos
Understanding narrated instructional videos is important for both research and real-world web applications.
A Tensorized Transformer for Language Modeling
In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD).
Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing
In this paper, we present an approach to incorporate retrieved datapoints as supporting evidence for context-dependent semantic parsing, such as generating source code conditioned on the class environment.
Deep Reason: A Strong Baseline for Real-World Visual Reasoning
This paper presents a strong baseline for real-world visual reasoning (GQA), which achieves 60. 93% in GQA 2019 challenge and won the sixth place.
PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph
Therefore, to generate the images with preferred objects and rich interactions, we propose a semi-parametric method, PasteGAN, for generating the image from the scene graph and the image crops, where spatial arrangements of the objects and their pair-wise relationships are defined by the scene graph and the object appearances are determined by the given object crops.
Dialog-to-Action: Conversational Question Answering Over a Large-Scale Knowledge Base
We present an approach to map utterances in conversation to logical forms, which will be executed on a large-scale knowledge base.
Knowledge-Aware Conversational Semantic Parsing Over Web Tables
Results show that our knowledge-aware model outperforms the state-of-the-art approaches.
Knowledge Based Machine Reading Comprehension
Machine reading comprehension (MRC) requires reasoning about both the knowledge involved in a document and knowledge about the world.
Improving Question Answering by Commonsense-Based Pre-Training
Although neural network approaches achieve remarkable success on a variety of NLP tasks, many of them struggle to answer questions that require commonsense knowledge.
Question Generation from SQL Queries Improves Neural Semantic Parsing
We study how to learn a semantic parser of state-of-the-art accuracy with less supervised training data.
Learning to Collaborate for Question Answering and Asking
Secondly, directly applying GAN that regards all the generated questions as negative instances could not improve the accuracy of the QA model.
Table-to-Text: Describing Table Region with Natural Language
The model maps a row from a table to a continuous vector and then generates a natural language sentence by leveraging the semantics of a table.
R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering
To better utilize semantic knowledge in images, we propose a novel framework to learn visual relation facts for VQA.
Semantic Parsing with Syntax- and Table-Aware SQL Generation
We present a generative model to map natural language questions into SQL queries.
Ranked #4 on
Code Generation
on WikiSQL
Assertion-based QA with Question-Aware Open Information Extraction
We present assertion based question answering (ABQA), an open domain question answering task that takes a question and a passage as inputs, and outputs a semi-structured assertion consisting of a subject, a predicate and a list of arguments.
Visual Question Generation as Dual Task of Visual Question Answering
Recently visual question answering (VQA) and visual question generation (VQG) are two trending topics in the computer vision, which have been explored separately.
Question Generation for Question Answering
This paper presents how to generate questions from given passages using neural networks, where large scale QA pairs are automatically crawled and processed from Community-QA website, and used as training data.
Content-Based Table Retrieval for Web Queries
Understanding the connections between unstructured text and semi-structured table is an important yet neglected problem in natural language processing.
Question Answering and Question Generation as Dual Tasks
On one side, the QA model judges whether the generated question of a QG model is relevant to the answer.
Constraint-Based Question Answering with Knowledge Graph
WebQuestions and SimpleQuestions are two benchmark data-sets commonly used in recent knowledge-based question answering (KBQA) work.
