Search Results for author:
Found 317 papers, 84 papers with code
End-to-End Simultaneous Speech Translation with Pretraining and Distillation: Huawei Noah’s System for AutoSimTranS 2022
This paper describes the system submitted to AutoSimTrans 2022 from Huawei Noah’s Ark Lab, which won the first place in the audio input track of the Chinese-English translation task.
Multilingual Speech Translation with Unified Transformer: Huawei Noah’s Ark Lab at IWSLT 2021
We use a unified transformer architecture for our MultiST model, so that the data from different modalities (i. e., speech and text) and different tasks (i. e., Speech Recognition, Machine Translation, and Speech Translation) can be exploited to enhance the model’s ability.
MTRec: Multi-Task Learning over BERT for News Recommendation
With the adoption of large pre-trained models like BERT in news recommendation, the above way to incorporate multi-field information may encounter challenges: the shallow feature encoding to compress the category and entity information is not compatible with the deep BERT encoding.
ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer
Other sparse methods use clustering patterns to select words, but the clustering process is separate from the training process of the target task, which causes a decrease in effectiveness.
Findings of the Third Workshop on Automatic Simultaneous Translation
This paper reports the results of the shared task we hosted on the Third Workshop of Automatic Simultaneous Translation (AutoSimTrans).
Huawei’s Submissions to the WMT20 Biomedical Translation Task
This paper describes Huawei’s submissions to the WMT20 biomedical translation shared task.
Neural Machine Translation with Heterogeneous Topic Knowledge Embeddings
Neural Machine Translation (NMT) has shown a strong ability to utilize local context to disambiguate the meaning of words.
Self-Supervised Quality Estimation for Machine Translation
Quality estimation (QE) of machine translation (MT) aims to evaluate the quality of machine-translated sentences without references and is important in practical applications of MT.
Chinese WPLC: A Chinese Dataset for Evaluating Pretrained Language Models on Word Prediction Given Long-Range Context
Experiment results show that the Chinese pretrained language model PanGu-\alpha is 45 points behind human in terms of top-1 word prediction accuracy, indicating that Chinese WPLC is a challenging dataset.
NoahNMT at WMT 2021: Dual Transfer for Very Low Resource Supervised Machine Translation
This paper describes the NoahNMT system submitted to the WMT 2021 shared task of Very Low Resource Supervised Machine Translation.
MINER: Multi-Interest Matching Network for News Recommendation
Accurately matching user’s interests and candidate news is the key to news recommendation.
Controlled Text Generation Using Dictionary Prior in Variational Autoencoders
While variational autoencoders (VAEs) have been widely applied in text generation tasks, they are troubled by two challenges: insufficient representation capacity and poor controllability.
Huawei AARC’s Submissions to the WMT21 Biomedical Translation Task: Domain Adaption from a Practical Perspective
This paper describes Huawei Artificial Intelligence Application Research Center’s neural machine translation systems and submissions to the WMT21 biomedical translation shared task.
ToolACE-R: Tool Learning with Adaptive Self-Refinement
Tool learning, which allows Large Language Models (LLMs) to leverage external tools for solving complex user tasks, has emerged as a promising avenue for extending model capabilities.
DAST: Difficulty-Aware Self-Training on Large Language Models
Present Large Language Models (LLM) self-training methods always under-sample on challenging queries, leading to inadequate learning on difficult problems which limits LLMs' ability.
KnowLogic: A Benchmark for Commonsense Reasoning via Knowledge-Driven Data Synthesis
This approach, along with our benchmark, provides a valuable tool for assessing and enhancing LLMs' commonsense reasoning capabilities and can be applied to a wide range of knowledge domains.
The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation
Our experiments and analyses demonstrate that neural machine translation performs poorly on commonsense reasoning of the three ambiguity types in terms of both reasoning accuracy (60. 1%) and reasoning consistency (31%).
Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework
Large Language Models (LLMs) are being used more and more extensively for automated evaluation in various scenarios.
Thus Spake Long-Context Large Language Model
Moreover, the research on long-context LLMs has expanded from length extrapolation to a comprehensive focus on architecture, infrastructure, training, and evaluation technologies.
Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solving
Existing approaches to mathematical reasoning with large language models (LLMs) rely on Chain-of-Thought (CoT) for generalizability or Tool-Integrated Reasoning (TIR) for precise computation.
Path-of-Thoughts: Extracting and Following Paths for Robust Relational Reasoning with Large Language Models
Large language models (LLMs) possess vast semantic knowledge but often struggle with complex reasoning tasks, particularly in relational reasoning problems such as kinship or spatial reasoning.
Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding
Based on this Friends-MMC dataset, we further study two fundamental MMC tasks: conversation speaker identification and conversation response prediction, both of which have the multi-party nature with the video or image as visual context.
More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
As large language models (LLMs) process increasing context windows, the memory usage of KV cache has become a critical bottleneck during inference.
LongSafetyBench: Long-Context LLMs Struggle with Safety Issues
With the development of large language models (LLMs), the sequence length of these models continues to increase, drawing significant attention to long-context language models.
ToolFlow: Boosting LLM Tool-Calling Through Natural and Coherent Dialogue Synthesis
Data quality assessments demonstrate improvements in the naturalness and coherence of our synthesized dialogues.
Roadmap towards Superhuman Speech Understanding using Large Language Models
The success of large language models (LLMs) has prompted efforts to integrate speech and audio data, aiming to create general foundation models capable of processing both textual and non-textual inputs.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Subtle Errors Matter: Preference Learning via Error-injected Self-editing
In this work, we propose a novel preference learning framework called eRror-Injected Self-Editing (RISE), which injects predefined subtle errors into partial tokens of correct solutions to construct hard pairs for error mitigation.
Meta-RTL: Reinforcement-Based Meta-Transfer Learning for Low-Resource Commonsense Reasoning
In this framework, we present a reinforcement-based approach to dynamically estimating source task weights that measure the contribution of the corresponding tasks to the target task in the meta-transfer learning.
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models.
GISExplainer: On Explainability of Graph Neural Networks via Game-theoretic Interaction Subgraphs
First, GISExplainer defines a causal attribution mechanism that considers the game-theoretic interaction of multi-granularity coalitions in candidate explanatory subgraph to quantify the causal effect of an edge on the prediction.
DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels
DetectiveQA focuses on evaluating the long-context reasoning ability of LLMs, which not only requires a full understanding of context but also requires extracting important evidences from the context and reasoning according to extracted evidences to answer the given questions.
ToolACE: Winning the Points of LLM Function Calling
Function calling significantly extends the application boundary of large language models, where high-quality and diverse training data is critical for unlocking this capability.
End-to-End Video Question Answering with Frame Scoring Mechanisms and Adaptive Sampling
Video Question Answering (VideoQA) has emerged as a challenging frontier in the field of multimedia processing, requiring intricate interactions between visual and textual modalities.
ReAttention: Training-Free Infinite Context with Finite Attention Scope
The long-context capability of the Large Language Models (LLM) has made significant breakthroughs, but the maximum supported context length remains a critical bottleneck limiting their practical applications.
Chain-of-Probe: Examing the Necessity and Accuracy of CoT Step-by-Step
To address these questions, we propose a method, namely Chain-of-Probe (CoP), to probe changes in the mind during the model's reasoning.
EWEK-QA: Enhanced Web and Efficient Knowledge Graph Retrieval for Citation-based Question Answering Systems
To mitigate these issues, we propose our enhanced web and efficient knowledge graph (KG) retrieval solution (EWEK-QA) to enrich the content of the extracted knowledge fed to the system.
CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search
In this paper, we study how open-source large language models (LLMs) can be effectively deployed for improving query rewriting in conversational search, especially for ambiguous queries.
Evaluating the External and Parametric Knowledge Fusion of Large Language Models
Integrating external knowledge into large language models (LLMs) presents a promising solution to overcome the limitations imposed by their antiquated and static parametric memory.
LFED: A Literary Fiction Evaluation Dataset for Large Language Models
The rapid evolution of large language models (LLMs) has ushered in the need for comprehensive assessments of their performance across various dimensions.
Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment
As the capabilities of large language models (LLMs) have expanded dramatically, aligning these models with human values presents a significant challenge.
Visually Guided Generative Text-Layout Pre-training for Document Intelligence
Prior study shows that pre-training techniques can boost the performance of visual document understanding (VDU), which typically requires models to gain abilities to perceive and reason both document texts and layouts (e. g., locations of texts and table-cells).
HawkEye: Training Video-Text LLMs for Grounding Text in Videos
Video-text Large Language Models (video-text LLMs) have shown remarkable performance in answering questions and holding conversations on simple videos.
Ranked #13 on
Video Question Answering
on MVBench
WIKIGENBENCH: Exploring Full-length Wikipedia Generation under Real-World Scenario
It presents significant challenges to generate comprehensive and accurate Wikipedia articles for newly emerging events under a real-world scenario.
Evaluating Robustness of Generative Search Engine on Adversarial Factual Questions
To this end, we propose evaluating the robustness of generative search engines in the realistic and high-risk setting, where adversaries have only black-box system access and seek to deceive the model into returning incorrect responses.
Contact Complexity in Customer Service
To tackle this, a machine learning model that accurately predicts the complexity of customer issues is highly desirable.
Teacher-Student Learning on Complexity in Intelligent Routing
Customer service is often the most time-consuming aspect for e-commerce websites, with each contact typically taking 10-15 minutes.
Universal Model in Online Customer Service
Our novel approach involves using machine learning techniques to tag customer questions in transcripts and create a repository of questions and corresponding labels.
Learning to Edit: Aligning LLMs with Knowledge Editing
Knowledge editing techniques, aiming to efficiently modify a minor proportion of knowledge in large language models (LLMs) without negatively impacting performance across other inputs, have garnered widespread attention.
Findings of the First Workshop on Simulating Conversational Intelligence in Chat
SCI-CHAT follows previous workshops on open domain dialogue but in contrast the focus of the shared task is simulation of intelligent conversation as judged in a live human evaluation.
Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios
The recent trend of using Large Language Models (LLMs) as tool agents in real-world applications underscores the necessity for comprehensive evaluations of their capabilities, particularly in complex scenarios involving planning, creating, and using tools.
MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models
Large language models (LLMs) are increasingly relied upon for complex multi-turn conversations across diverse real-world applications.
YODA: Teacher-Student Progressive Learning for Language Models
With the teacher's guidance, the student learns to iteratively refine its answer with feedback, and forms a robust and comprehensive understanding of the posed questions.
PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models
Large Language Models (LLMs) have succeeded remarkably in understanding long-form contents.
Preparing Lessons for Progressive Training on Language Models
The rapid progress of Transformers in artificial intelligence has come at the cost of increased resource consumption and greenhouse gas emissions due to growing model sizes.
PanGu-$π$: Enhancing Language Model Architectures via Nonlinearity Compensation
We then demonstrate that the proposed approach is significantly effective for enhancing the model nonlinearity through carefully designed ablations; thus, we present a new efficient model architecture for establishing modern, namely, PanGu-$\pi$.
"Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation
NoMIRACL includes both a non-relevant and a relevant subset.
A Survey of Reasoning with Foundation Models
Reasoning, a crucial ability for complex problem-solving, plays a pivotal role in various real-world settings such as negotiation, medical diagnosis, and criminal investigation.
Unsupervised Extractive Summarization with Learnable Length Control Strategies
Different from the centrality-based ranking methods, our extractive scorer can be trained in an end-to-end manner, with no other requirement of positional assumption.
Data Management For Training Large Language Models: A Survey
This survey aims to provide a comprehensive overview of current research in data management within both the pretraining and supervised fine-tuning stages of LLMs, covering various aspects of data management strategy design.
FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models
To fill this research gap, in this paper, we propose FollowBench, a Multi-level Fine-grained Constraints Following Benchmark for LLMs.
M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models
In this paper, we propose M4LE, a Multi-ability, Multi-range, Multi-task, Multi-domain benchmark for Long-context Evaluation.
TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models
Automated theorem proving (ATP) has become an appealing domain for exploring the reasoning ability of the recent successful generative language models.
Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
The rapid development of large language models (LLMs) has not only provided numerous opportunities but also presented significant challenges.
Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment
Inspired by previous work which identified that feed-forward networks (FFNs) within Transformers are responsible for factual knowledge expressions, we investigate two methods to efficiently improve the factual expression capability {of FFNs} by knowledge enhancement and alignment respectively.
SELF: Self-Evolution with Language Feedback
SELF initiates with a meta-skill learning process that equips the LLMs with capabilities for self-feedback and self-refinement.
FIMO: A Challenge Formal Dataset for Automated Theorem Proving
We present FIMO, an innovative dataset comprising formal mathematical problem statements sourced from the International Mathematical Olympiad (IMO) Shortlisted Problems.
Prompt-Based Length Controlled Generation with Reinforcement Learning
Large language models (LLMs) like ChatGPT and GPT-4 have attracted great attention given their surprising performance on a wide range of NLP tasks.
AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models
Information-seeking conversation, which aims to help users gather information through conversation, has achieved great progress in recent years.
NewsDialogues: Towards Proactive News Grounded Conversation
In this paper, we propose a novel task, Proactive News Grounded Conversation, in which a dialogue system can proactively lead the conversation based on some key topics of the news.
Aligning Large Language Models with Human: A Survey
(2) Training methodologies: a detailed review of the prevailing training methods employed for LLM alignment.
DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering
Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability.
mCLIP: Multilingual CLIP via Cross-lingual Transfer
Furthermore, to enhance the token- and sentence-level multilingual representation of the MTE, we propose to train it with machine translation and contrastive learning jointly before the TriKD to provide a better initialization.
INGB: Informed Nonlinear Granular Ball Oversampling Framework for Noisy Imbalanced Classification
It uses granular balls to simulate the spatial distribution characteristics of datasets, and informed entropy is utilized to further optimize the granular-ball space.
Enhancing Coherence of Extractive Summarization with Multitask Learning
This study proposes a multitask learning architecture for extractive summarization with coherence boosting.
M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models
Comprehensively evaluating the capability of large language models in multiple tasks is of great importance.
Learning Summary-Worthy Visual Representation for Abstractive Summarization in Video
Multimodal abstractive summarization for videos (MAS) requires generating a concise textual summary to describe the highlights of a video according to multimodal resources, in our case, the video content and its transcript.
End-to-end Training and Decoding for Pivot-based Cascaded Translation Model
Utilizing pivot language effectively can significantly improve low-resource machine translation.
Evaluating the Efficacy of Length-Controllable Machine Translation
We can use automatic summarization or machine translation evaluation metrics for length-controllable machine translation, but this is not necessarily suitable and accurate.
Learning Homographic Disambiguation Representation for Neural Machine Translation
Visualization methods like heatmaps, T-SNE and translation examples are also utilized to demonstrate the effects of the proposed method.
PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing
In this work, we develop a system that trained a trillion-parameter language model on a cluster of Ascend 910 AI processors and MindSpore framework, and present the language model with 1. 085T parameters named PanGu-{\Sigma}.
Adapting Pre-trained Language Models for Quantum Natural Language Processing
The emerging classical-quantum transfer learning paradigm has brought a decent performance to quantum computational models in many tasks, such as computer vision, by enabling a combination of quantum models and classical pre-trained neural networks.
WL-Align: Weisfeiler-Lehman Relabeling for Aligning Users across Networks via Regularized Representation Learning
Aligning users across networks using graph representation learning has been found effective where the alignment is accomplished in a low-dimensional embedding space.
MoralDial: A Framework to Train and Evaluate Moral Dialogue Systems via Moral Discussions
In this paper, we propose a framework, MoralDial to train and evaluate moral dialogue systems.
Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding
While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far.
AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation
To alleviate the data scarcity problem in End-to-end speech translation (ST), pre-training on data for speech recognition and machine translation is considered as an important technique.
Retrieval-based Disentangled Representation Learning with Natural Language Supervision
In light of this, we present Vocabulary Disentangled Retrieval (VDR), a retrieval-based framework that harnesses natural language as proxies of the underlying data variation to drive disentangled representation learning.
G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks
Recently, domain-specific PLMs have been proposed to boost the task performance of specific domains (e. g., biomedical and computer science) by continuing to pre-train general PLMs with domain-specific corpora.
KPT: Keyword-guided Pre-training for Grounded Dialog Generation
For the former, the grounding knowledge consists of keywords extracted from the response.
SongRewriter: A Chinese Song Rewriting System with Controllable Content and Rhyme Scheme
Although lyrics generation has achieved significant progress in recent years, it has limited practical applications because the generated lyrics cannot be performed without composing compatible melodies.
Lexicon-injected Semantic Parsing for Task-Oriented Dialog
Then we present a novel lexicon-injected semantic parser, which collects slot labels of tree representation as a lexicon, and injects lexical features to the span representation of parser.
FPT: Improving Prompt Tuning Efficiency via Progressive Training
Inspired by these observations, we propose Fast Prompt Tuning (FPT), which starts by conducting PT using a small-scale partial PLM, and then progressively expands its depth and width until the full-model size.
COPEN: Probing Conceptual Knowledge in Pre-trained Language Models
We believe this is a critical bottleneck for realizing human-like cognition in PLMs.
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling
Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks.
Pre-training Language Models with Deterministic Factual Knowledge
Further experiments on question-answering datasets show that trying to learn a deterministic relationship with the proposed methods can also help other knowledge-intensive tasks.
Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages
MIRACL (Multilingual Information Retrieval Across a Continuum of Languages) is a multilingual dataset we have built for the WSDM 2023 Cup challenge that focuses on ad hoc retrieval across 18 different languages, which collectively encompass over three billion native speakers around the world.
ShortcutLens: A Visual Analytics Approach for Exploring Shortcuts in Natural Language Understanding Dataset
Since shortcuts vary in coverage, productivity, and semantic meaning, it is challenging for NLU experts to systematically understand and avoid them when creating benchmark datasets.
PanGu-Coder: Program Synthesis with Function-Level Language Modeling
We present PanGu-Coder, a pretrained decoder-only language model adopting the PanGu-Alpha architecture for text-to-code generation, i. e. the synthesis of programming language solutions given a natural language problem description.
FreeTransfer-X: Safe and Label-Free Cross-Lingual Transfer from Off-the-Shelf Models
However, labeled cross-lingual corpus is expensive or even inaccessible, especially in the fields where labels are private, such as diagnostic results of symptoms in medicine and user profiles in business.
PERT: A New Solution to Pinyin to Character Conversion Task
Pinyin to Character conversion (P2C) task is the key task of Input Method Engine (IME) in commercial input software for Asian languages, such as Chinese, Japanese, Thai language and so on.
Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding
There is a growing body of work in recent years to develop pre-trained language models (PLMs) for the Arabic language.
Exploring Extreme Parameter Compression for Pre-trained Language Models
A tiny version achieves $96. 7\%$ performance of BERT-base with $ {1}/{48} $ encoder parameters (i. e., less than 2M parameters excluding the embedding layer) and $2. 7 \times$ faster on inference.
UTC: A Unified Transformer with Inter-Task Contrastive Learning for Visual Dialog
In this paper, we propose a contrastive learning-based framework UTC to unify and facilitate both discriminative and generative tasks in visual dialog with a single model.
PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model
We investigate different aspects of responses generated by PanGu-Bot, including response quality, knowledge, and safety.
How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis
We check the words that have three typical associations with the missing words: knowledge-dependent, positionally close, and highly co-occurred.
Compression of Generative Pre-trained Language Models via Quantization
We find that previous quantization methods fail on generative tasks due to the \textit{homogeneous word embeddings} caused by reduced capacity, and \textit{varied distribution of weights}.
Triangular Transfer: Freezing the Pivot for Triangular Machine Translation
Triangular machine translation is a special case of low-resource machine translation where the language pair of interest has limited parallel data, but both languages have abundant parallel data with a pivot language.
Universal Conditional Masked Language Pre-training for Neural Machine Translation
To the best of our knowledge, this is the first work to pre-train a unified model for fine-tuning on both NMT tasks.
Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering
To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR).
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.
Achieving Reliable Human Assessment of Open-Domain Dialogue Systems
Answering the distress call of competitions that have emphasized the urgent need for better evaluation techniques in dialogue, we present the successful development of human evaluation that is highly reliable while still remaining feasible and low cost.
Compilable Neural Code Generation with Compiler Feedback
Automatically generating compilable programs with (or without) natural language descriptions has always been a touchstone problem for computational linguistics and automated software engineering.
HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks
In this paper, we design a novel unified parameter-efficient transfer learning framework that works effectively on both pure language and V&L tasks.
Read before Generate! Faithful Long Form Question Answering with Machine Reading
Long-form question answering (LFQA) aims to generate a paragraph-length answer for a given question.
Ranked #1 on
Question Answering
on KILT: ELI5
Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarks
The research of open-domain dialog systems has been greatly prospered by neural models trained on large-scale corpora, however, such corpora often introduce various safety problems (e. g., offensive languages, biases, and toxic behaviors) that significantly hinder the deployment of dialog systems in practice.
Pan More Gold from the Sand: Refining Open-domain Dialogue Training with Noisy Self-Retrieval Generation
Experiments over publicly available datasets demonstrate that our method can help models generate better responses, even such training data are usually impressed as low-quality data.
SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training
The student network is trained to output representation resembling that of the teacher.
LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework
We propose LMTurk, a novel approach that treats few-shot learners as crowdsourcing workers.
JABER and SABER: Junior and Senior Arabic BERt
Language-specific pre-trained models have proven to be more accurate than multilingual ones in a monolingual evaluation setting, Arabic is no exception.
Towards Improving Embedding Based Models of Social Network Alignment via Pseudo Anchors
Social network alignment aims at aligning person identities across social networks.
CoCA-MDD: A Coupled Cross-Attention based Framework for Streaming Mispronunciation Detection and Diagnosis
We utilize conv-transformer structure to encode input speech in a streaming manner.
UniDS: A Unified Dialogue System for Chit-Chat and Task-oriented Dialogues
With the advances in deep learning, tremendous progress has been made with chit-chat dialogue systems and task-oriented dialogue systems.
bert2BERT: Towards Reusable Pretrained Language Models
However, large language model pre-training costs intensive computational resources and most of the models are trained from scratch without reusing the existing pre-trained models, which is wasteful.
Speech-MLP: a simple MLP architecture for speech processing
Overparameterized transformer-based architectures have shown remarkable performance in recent years, achieving state-of-the-art results in speech processing tasks such as speech recognition, speech synthesis, keyword spotting, and speech enhancement et al.
Multi-Semantic Image Recognition Model and Evaluating Index for explaining the deep learning models
Although deep learning models are powerful among various applications, most deep learning models are still a black box, lacking verifiability and interpretability, which means the decision-making process that human beings cannot understand.
DyLex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling
Incorporating lexical knowledge into deep learning models has been proved to be very effective for sequence labeling tasks.
Improving Unsupervised Question Answering via Summarization-Informed Question Generation
Template-based QG uses linguistically-informed heuristics to transform declarative sentences into interrogatives, whereas supervised QG uses existing Question Answering (QA) datasets to train a system to generate a question given a passage and an answer.
UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation
Specially, we adopt knowledge distillation from a vision-language pretrained model to improve image selection, which avoids any requirement on the existence and quality of image captions.
CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems
As labeling cost for different modules in task-oriented dialog (ToD) systems is high, a major challenge in practice is to learn different tasks with the least amount of labeled data.
KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs
Incorporating factual knowledge into pre-trained language models (PLM) such as BERT is an emerging trend in recent NLP studies.
Ranked #11 on
Common Sense Reasoning
on ReCoRD
Integrating Regular Expressions with Neural Networks via DFA
Human-designed rules are widely used to build industry applications.
NumGPT: Improving Numeracy Ability of Generative Pre-trained Models
However, those models do not consider the numerical properties of numbers and cannot perform robustly on numerical reasoning tasks (e. g., math word problems and measurement estimation).
Generate & Rank: A Multi-task Framework for Math Word Problems
Math word problem (MWP) is a challenging and critical task in natural language processing.
Ranked #3 on
Math Word Problem Solving
on Math23K
Uncertainty-Aware Balancing for Multilingual and Multi-Domain Neural Machine Translation Training
In this work, we propose an approach, MultiUAT, that dynamically adjusts the training data usage based on the model's uncertainty on a small set of trusted clean data for multi-corpus machine translation.
GhostBERT: Generate More Features with Cheap Operations for BERT
Transformer-based pre-trained language models like BERT, though powerful in many tasks, are expensive in both memory and computation, due to their large number of parameters.
TGEA: An Error-Annotated Dataset and Benchmark Tasks for TextGeneration from Pretrained Language Models
Each error is hence manually labeled with comprehensive annotations, including the span of the error, the associated span, minimal correction to the error, the type of the error, and rationale behind the error.
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
Specifically, we carefully design the techniques of one-shot learning and the search space to provide an adaptive and efficient development way of tiny PLMs for various latency constraints.
A Mutual Information Maximization Approach for the Spurious Solution Problem in Weakly Supervised Question Answering
This setting gives rise to the spurious solution problem: there may exist many spurious solutions that coincidentally derive the correct answer, but training on such solutions can hurt model performance (e. g., producing wrong solutions or answers).
Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision
Recently, pre-training multilingual language models has shown great potential in learning multilingual representation, a crucial topic of natural language processing.
RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer
To bridge the modality gap between speech and text, RealTranS gradually downsamples the input speech with interleaved convolution and unidirectional Transformer layers for acoustic modeling, and then maps speech features into text space with a weighted-shrinking operation and a semantic encoder.
Sub-Character Tokenization for Chinese Pretrained Language Models
2) Pronunciation-based SubChar tokenizers can encode Chinese homophones into the same transliteration sequences and produce the same tokenization output, hence being robust to homophone typos.
Multilingual Speech Translation with Unified Transformer: Huawei Noah's Ark Lab at IWSLT 2021
We use a unified transformer architecture for our MultiST model, so that the data from different modalities (i. e., speech and text) and different tasks (i. e., Speech Recognition, Machine Translation, and Speech Translation) can be exploited to enhance the model's ability.
Improved OOD Generalization via Adversarial Training and Pre-training
In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data.
Dynamic Multi-Branch Layers for On-Device Neural Machine Translation
With the rapid development of artificial intelligence (AI), there is a trend in moving AI applications, such as neural machine translation (NMT), from cloud to mobile devices.
HyKnow: End-to-End Task-Oriented Dialog Modeling with Hybrid Knowledge Management
To address this task, we propose a TOD system with hybrid knowledge management, HyKnow.
PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
To enhance the generalization ability of PanGu-$\alpha$, we collect 1. 1TB high-quality Chinese data from a wide range of domains to pretrain the model.
Ranked #1 on
Reading Comprehension (One-Shot)
on DuReader
Cloze (multi-choices) (Few-Shot)
Cloze (multi-choices) (One-Shot)
+19
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
Task-agnostic knowledge distillation, a teacher-student framework, has been proved effective for BERT compression.
From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables
Embedding matrices are key components in neural natural language processing (NLP) models that are responsible to provide numerical representations of input tokens.\footnote{In this paper words and subwords are referred to as \textit{tokens} and the term \textit{embedding} only refers to embeddings of inputs.}
An Approach to Improve Robustness of NLP Systems against ASR Errors
Speech-enabled systems typically first convert audio to text through an automatic speech recognition (ASR) model and then feed the text to downstream natural language processing (NLP) modules.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Dependency Graph-to-String Statistical Machine Translation
We present graph-based translation models which translate source graphs into target strings.
Reweighting Augmented Samples by Minimizing the Maximal Expected Loss
Inspired by adversarial training, we minimize this maximal expected loss (MMEL) and obtain a simple and interpretable closed-form solution: more attention should be paid to augmented samples with large loss values (i. e., harder examples).
LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
The multilingual pre-trained language models (e. g, mBERT, XLM and XLM-R) have shown impressive performance on cross-lingual natural language understanding tasks.
Training Multilingual Pre-trained Language Model with Byte-level Subwords
The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora.
On Position Embeddings in BERT
Various Position Embeddings (PEs) have been proposed in Transformer based architectures~(e. g. BERT) to model word order.
Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning
In this work, we propose a simple and effective method to cover a much larger proportion of the attack search space, called Adversarial and Mixup Data Augmentation (AMDA).
BinaryBERT: Pushing the Limit of BERT Quantization
In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization.
HopRetriever: Retrieve Hops over Wikipedia to Answer Complex Questions
In this paper, we propose a new retrieval target, hop, to collect the hidden reasoning evidence from Wikipedia for complex question answering.
Ranked #6 on
Question Answering
on HotpotQA
Revisiting Robust Neural Machine Translation: A Transformer Case Study
There is a large body of work in the NMT literature on analyzing the behavior of conventional models for the problem of noise but Transformers are relatively understudied in this context.
ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
Knowledge distillation is considered as a training and compression strategy in which two neural networks, namely a teacher and a student, are coupled together during training.
Improving Task-Agnostic BERT Distillation with Layer Mapping Search
Comprehensive experiments on the evaluation benchmarks demonstrate that 1) layer mapping strategy has a significant effect on task-agnostic BERT distillation and different layer mappings can result in quite different performances; 2) the optimal layer mapping strategy from the proposed search process consistently outperforms the other heuristic ones; 3) with the optimal layer mapping, our student model achieves state-of-the-art performance on the GLUE tasks.
PPKE: Knowledge Representation Learning by Path-based Pre-training
Entities may have complex interactions in a knowledge graph (KG), such as multi-step relationships, which can be viewed as graph contextual information of the entities.
KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
Recent studies on pre-trained language models have demonstrated their ability to capture factual knowledge and applications in knowledge-aware downstream tasks.
Document Graph for Neural Machine Translation
Previous works have shown that contextual information can improve the performance of neural machine translation (NMT).
From Unsupervised Machine Translation To Adversarial Text Generation
B-GAN is able to generate a distributed latent space representation which can be paired with an attention based decoder to generate fluent sentences.
Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads
To measure the informativeness of attention heads, we train our Single-Shot Meta-Pruner (SMP) with a meta-learning paradigm aiming to maintain the distribution of text representations after pruning.
BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models
Complex node interactions are common in knowledge graphs (KGs), and these interactions can be considered as contextualized knowledge exists in the topological structure of KGs.
HyperText: Endowing FastText with Hyperbolic Geometry
Natural language data exhibit tree-like hierarchical structures such as the hypernym-hyponym relations in WordNet.
LiDAM: Semi-Supervised Learning with Localized Domain Adaptation and Iterative Matching
Although data is abundant, data labeling is expensive.
Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers
With the growth of computing power neural machine translation (NMT) models also grow accordingly and become better.
SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval
Term-based sparse representations dominate the first-stage text retrieval in industrial applications, due to its advantage in efficiency, interpretability, and exact term matching.
TernaryBERT: Distillation-aware Ultra-low Bit BERT
Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks. However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices.
TensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language Modeling
Transformer has been widely-used in many Natural Language Processing (NLP) tasks and the scaled dot-product attention between tokens is a core module of Transformer.
Learning to Detect Unacceptable Machine Translations for Downstream Tasks
In this work, we put machine translation in a cross-lingual pipeline and introduce downstream tasks to define task-specific acceptability of machine translations.
Accurate Word Alignment Induction from Neural Machine Translation
Shift-Att is an interpretation method that induces alignments from the attention weights of Transformer and does not require parameter update or architecture change.
Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
However, this approach of evaluating a language model is undermined by the uncertainty of the amount of knowledge that is learned by the probe itself.
Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
Masked language model and autoregressive language model are two types of language models.
DynaBERT: Dynamic BERT with Adaptive Width and Depth
The pre-trained language models like BERT, though powerful in many natural language processing tasks, are both computation and memory expensive.
Dictionary-based Data Augmentation for Cross-Domain Neural Machine Translation
Existing data augmentation approaches for neural machine translation (NMT) have predominantly relied on back-translating in-domain (IND) monolingual corpora.
A One-Shot Learning Framework for Assessment of Fibrillar Collagen from Second Harmonic Generation Images of an Infarcted Myocardium
Myocardial infarction (MI) is a scientific term that refers to heart attack.
Context-Aware Design of Cyber-Physical Human Systems (CPHS)
The impacts of context factors driving human system interaction are challenging and are difficult to capture and replicate in existing design models.
Multi-channel Reverse Dictionary Model
A reverse dictionary takes the description of a target word as input and outputs the target word together with other words that match the description.
Deep Learning-Based Feature-Aware Data Modeling for Complex Physics Simulations
Data modeling and reduction for in situ is important.
Learning to Predict Explainable Plots for Neural Story Generation
While the use of neural networks has proven effective in improving story generation, how to learn to generate an explainable high-level plot still remains a major challenge.
Integrating Graph Contextualized Knowledge into Pre-trained Language Models
Complex node interactions are common in knowledge graphs, and these interactions also contain rich knowledge information.
Deep-seismic-prior-based reconstruction of seismic data using convolutional neural networks
Synthetic and field data were tested to assess the performance of the proposed algorithm (DSPRecon algorithm); the advantages of using our method were evaluated by comparing it with the singular spectrum analysis (SSA) method for irregular data reconstruction and de-aliased Cadzow method for regular data reconstruction.
DeepSat V2: Feature Augmented Convolutional Neural Nets for Satellite Image Classification
Satellite image classification is a challenging problem that lies at the crossroads of remote sensing, computer vision, and machine learning.
Ranked #1 on
Satellite Image Classification
on SAT-4
Pretrained Language Models for Document-Level Neural Machine Translation
Previous work on document-level NMT usually focuses on limited contexts because of degraded performance on larger contexts.
A General Framework for Adaptation of Neural Machine Translation to Simultaneous Translation
Despite the success of neural machine translation (NMT), simultaneous neural machine translation (SNMT), the task of translating in real time before a full sentence has been observed, remains challenging due to the syntactic structure difference and simultaneity requirements.
Word-level Textual Adversarial Attacking as Combinatorial Optimization
Also, further experiments show our model has higher transferability and can bring more robustness enhancement to victim models by adversarial training.
Improving Sequence Modeling Ability of Recurrent Neural Networks via Sememes
Sememes, the minimum semantic units of human languages, have been successfully utilized in various natural language processing applications.
TinyBERT: Distilling BERT for Natural Language Understanding
To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models.
Ranked #1 on
Natural Language Inference
on MultiNLI Dev
NEZHA: Neural Contextualized Representation for Chinese Language Understanding
The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora.
Dialog State Tracking with Reinforced Data Augmentation
Neural dialog state trackers are generally limited due to the lack of quantity and diversity of annotated training data.
PCGAN-CHAR: Progressively Trained Classifier Generative Adversarial Networks for Classification of Noisy Handwritten Bangla Characters
We show that by learning the features at each resolution independently a trained model is able to accurately classify characters even in the presence of noise.
Ranked #1 on
Image Classification
on Noisy MNIST (AWGN)
Huawei's NMT Systems for the WMT 2019 Biomedical Translation Task
This paper describes Huawei{'}s neural machine translation systems for the WMT 2019 biomedical translation shared task.
Modeling Semantic Compositionality with Sememe Knowledge
In this paper, we verify the effectiveness of sememes, the minimum semantic units of human languages, in modeling SC by a confirmatory experiment.
multi-word expression embedding
multi-word expression sememe prediction
GPT-based Generation for Classical Chinese Poetry
We present a simple yet effective method for generating high quality classical Chinese poetry with Generative Pre-trained Language Model (GPT).
