Search Results for author:
Found 112 papers, 33 papers with code
Controlled Text Generation Using Dictionary Prior in Variational Autoencoders
While variational autoencoders (VAEs) have been widely applied in text generation tasks, they are troubled by two challenges: insufficient representation capacity and poor controllability.
ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer
Other sparse methods use clustering patterns to select words, but the clustering process is separate from the training process of the target task, which causes a decrease in effectiveness.
MTRec: Multi-Task Learning over BERT for News Recommendation
With the adoption of large pre-trained models like BERT in news recommendation, the above way to incorporate multi-field information may encounter challenges: the shallow feature encoding to compress the category and entity information is not compatible with the deep BERT encoding.
MINER: Multi-Interest Matching Network for News Recommendation
Accurately matching user’s interests and candidate news is the key to news recommendation.
PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing
In this work, we develop a system that trained a trillion-parameter language model on a cluster of Ascend 910 AI processors and MindSpore framework, and present the language model with 1. 085T parameters named PanGu-{\Sigma}.
DialogPaint: A Dialog-based Image Editing Model
The framework comprises a pretrained dialogue model (Blenderbot) and a diffusion model (Stable Diffusion).
Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding
While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far.
Retrieval-based Disentanglement with Distant Supervision
Disentangled representation learning remains challenging as ground truth factors of variation do not naturally exist.
G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks
Recently, domain-specific PLMs have been proposed to boost the task performance of specific domains (e. g., biomedical and computer science) by continuing to pre-train general PLMs with domain-specific corpora.
KPT: Keyword-guided Pre-training for Grounded Dialog Generation
For the former, the grounding knowledge consists of keywords extracted from the response.
Lexicon-injected Semantic Parsing for Task-Oriented Dialog
Then we present a novel lexicon-injected semantic parser, which collects slot labels of tree representation as a lexicon, and injects lexical features to the span representation of parser.
Label Mask AutoEncoder(L-MAE): A Pure Transformer Method to Augment Semantic Segmentation Datasets
Semantic segmentation models based on the conventional neural network can achieve remarkable performance in such tasks, while the dataset is crucial to the training model process.
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling
Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks.
Pre-training Language Models with Deterministic Factual Knowledge
Further experiments on question-answering datasets show that trying to learn a deterministic relationship with the proposed methods can also help other knowledge-intensive tasks.
PanGu-Coder: Program Synthesis with Function-Level Language Modeling
We present PanGu-Coder, a pretrained decoder-only language model adopting the PanGu-Alpha architecture for text-to-code generation, i. e. the synthesis of programming language solutions given a natural language problem description.
Boosting Graph Structure Learning with Dummy Nodes
We extend graph kernels and graph neural networks with dummy nodes and conduct experiments on graph classification and subgraph isomorphism matching tasks.
FreeTransfer-X: Safe and Label-Free Cross-Lingual Transfer from Off-the-Shelf Models
However, labeled cross-lingual corpus is expensive or even inaccessible, especially in the fields where labels are private, such as diagnostic results of symptoms in medicine and user profiles in business.
PERT: A New Solution to Pinyin to Character Conversion Task
Pinyin to Character conversion (P2C) task is the key task of Input Method Engine (IME) in commercial input software for Asian languages, such as Chinese, Japanese, Thai language and so on.
Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding
There is a growing body of work in recent years to develop pre-trained language models (PLMs) for the Arabic language.
Deeper vs Wider: A Revisit of Transformer Configuration
In this paper, we revisit these conventional configurations.
Ranked #88 on
Image Classification
on ImageNet
Exploring Extreme Parameter Compression for Pre-trained Language Models
A tiny version achieves $96. 7\%$ performance of BERT-base with $ {1}/{48} $ encoder parameters (i. e., less than 2M parameters excluding the embedding layer) and $2. 7 \times$ faster on inference.
UTC: A Unified Transformer with Inter-Task Contrastive Learning for Visual Dialog
In this paper, we propose a contrastive learning-based framework UTC to unify and facilitate both discriminative and generative tasks in visual dialog with a single model.
CorrectSpeech: A Fully Automated System for Speech Correction and Accent Reduction
This study propose a fully automated system for speech correction and accent reduction.
PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model
We investigate different aspects of responses generated by PanGu-Bot, including response quality, knowledge, and safety.
How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis
We check the words that have three typical associations with the missing words: knowledge-dependent, positionally close, and highly co-occurred.
Compression of Generative Pre-trained Language Models via Quantization
We find that previous quantization methods fail on generative tasks due to the \textit{homogeneous word embeddings} caused by reduced capacity, and \textit{varied distribution of weights}.
Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering
To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR).
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.
Compilable Neural Code Generation with Compiler Feedback
Automatically generating compilable programs with (or without) natural language descriptions has always been a touchstone problem for computational linguistics and automated software engineering.
HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks
In this paper, we design a novel unified parameter-efficient transfer learning framework that works effectively on both pure language and V&L tasks.
Read before Generate! Faithful Long Form Question Answering with Machine Reading
Long-form question answering (LFQA) aims to generate a paragraph-length answer for a given question.
Ranked #1 on
Question Answering
on KILT: ELI5
Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarks
The research of open-domain dialog systems has been greatly prospered by neural models trained on large-scale corpora, however, such corpora often introduce various safety problems (e. g., offensive languages, biases, and toxic behaviors) that significantly hinder the deployment of dialog systems in practice.
Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark
Experiments show that Wukong can serve as a promising Chinese pre-training dataset and benchmark for different cross-modal learning methods.
Ranked #6 on
Zero-shot Image Retrieval
on COCO-CN
Pan More Gold from the Sand: Refining Open-domain Dialogue Training with Noisy Self-Retrieval Generation
Experiments over publicly available datasets demonstrate that our method can help models generate better responses, even such training data are usually impressed as low-quality data.
SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training
The student network is trained to output representation resembling that of the teacher.
LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework
We propose LMTurk, a novel approach that treats few-shot learners as crowdsourcing workers.
JABER and SABER: Junior and Senior Arabic BERt
Language-specific pre-trained models have proven to be more accurate than multilingual ones in a monolingual evaluation setting, Arabic is no exception.
CoCA-MDD: A Coupled Cross-Attention based Framework for Streaming Mispronunciation Detection and Diagnosis
We utilize conv-transformer structure to encode input speech in a streaming manner.
FILIP: Fine-grained Interactive Language-Image Pre-Training
In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective.
Robust Multi-view Registration of Point Sets with Laplacian Mixture Model
The proposed method assumes each data point is generated by a Laplacian Mixture Model (LMM), where its centers are determined by the corresponding points in other point sets.
UniDS: A Unified Dialogue System for Chit-Chat and Task-oriented Dialogues
With the advances in deep learning, tremendous progress has been made with chit-chat dialogue systems and task-oriented dialogue systems.
bert2BERT: Towards Reusable Pretrained Language Models
However, large language model pre-training costs intensive computational resources and most of the models are trained from scratch without reusing the existing pre-trained models, which is wasteful.
Towards Efficient Post-training Quantization of Pre-trained Language Models
Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.
DyLex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling
Incorporating lexical knowledge into deep learning models has been proved to be very effective for sequence labeling tasks.
Improving Unsupervised Question Answering via Summarization-Informed Question Generation
Template-based QG uses linguistically-informed heuristics to transform declarative sentences into interrogatives, whereas supervised QG uses existing Question Answering (QA) datasets to train a system to generate a question given a passage and an answer.
UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation
Specially, we adopt knowledge distillation from a vision-language pretrained model to improve image selection, which avoids any requirement on the existence and quality of image captions.
CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems
As labeling cost for different modules in task-oriented dialog (ToD) systems is high, a major challenge in practice is to learn different tasks with the least amount of labeled data.
Integrating Regular Expressions with Neural Networks via DFA
Human-designed rules are widely used to build industry applications.
Generate & Rank: A Multi-task Framework for Math Word Problems
Math word problem (MWP) is a challenging and critical task in natural language processing.
Ranked #1 on
Math Word Problem Solving
on Math23K
NumGPT: Improving Numeracy Ability of Generative Pre-trained Models
However, those models do not consider the numerical properties of numbers and cannot perform robustly on numerical reasoning tasks (e. g., math word problems and measurement estimation).
SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation
Code representation learning, which aims to encode the semantics of source code into distributed vectors, plays an important role in recent deep-learning-based models for code intelligence.
GhostBERT: Generate More Features with Cheap Operations for BERT
Transformer-based pre-trained language models like BERT, though powerful in many tasks, are expensive in both memory and computation, due to their large number of parameters.
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
Specifically, we carefully design the techniques of one-shot learning and the search space to provide an adaptive and efficient development way of tiny PLMs for various latency constraints.
AutoBERT-Zero: Evolving BERT Backbone from Scratch
Transformer-based pre-trained language models like BERT and its variants have recently achieved promising performance in various natural language processing (NLP) tasks.
Ranked #10 on
Semantic Textual Similarity
on MRPC
EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusion
This paper presents the design, implementation and evaluation of a speech editing system, named EditSpeech, which allows a user to perform deletion, insertion and replacement of words in a given speech utterance, without causing audible degradation in speech quality and naturalness.
Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision
Recently, pre-training multilingual language models has shown great potential in learning multilingual representation, a crucial topic of natural language processing.
Exploring Discourse Structures for Argument Impact Classification
Discourse relations among arguments reveal logical structures of a debate conversation.
Improved OOD Generalization via Adversarial Training and Pre-training
In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data.
A novel feed rate scheduling method based on Sigmoid function with chord error and kinematics constraints
With the consideration of kinematic limitation and machining efficiency, a time-optimal feed rate adjustment algorithm is proposed to further adjust feed rate value at breaking points.
PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
To enhance the generalization ability of PanGu-$\alpha$, we collect 1. 1TB high-quality Chinese data from a wide range of domains to pretrain the model.
Ranked #1 on
Reading Comprehension (Zero-Shot)
on CMRC 2018
Cloze (multi-choices) (Few-Shot)
Cloze (multi-choices) (One-Shot)
+18
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
Task-agnostic knowledge distillation, a teacher-student framework, has been proved effective for BERT compression.
A novel S-shape based NURBS interpolation with acc-jerk- Continuity and round-off error elimination
The algorithm consists of three modules: bidirectional scanning module, velocity scheduling module and round-off error elimination module.
An Approach to Improve Robustness of NLP Systems against ASR Errors
Speech-enabled systems typically first convert audio to text through an automatic speech recognition (ASR) model and then feed the text to downstream natural language processing (NLP) modules.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Reweighting Augmented Samples by Minimizing the Maximal Expected Loss
Inspired by adversarial training, we minimize this maximal expected loss (MMEL) and obtain a simple and interpretable closed-form solution: more attention should be paid to augmented samples with large loss values (i. e., harder examples).
LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
The multilingual pre-trained language models (e. g, mBERT, XLM and XLM-R) have shown impressive performance on cross-lingual natural language understanding tasks.
Training Multilingual Pre-trained Language Model with Byte-level Subwords
The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora.
Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks
In this work, we demonstrate the universal vulnerability of PTMs, where fine-tuned PTMs can be easily controlled by backdoor attacks in arbitrary downstream tasks.
On Position Embeddings in BERT
Various Position Embeddings (PEs) have been proposed in Transformer based architectures~(e. g. BERT) to model word order.
HopRetriever: Retrieve Hops over Wikipedia to Answer Complex Questions
In this paper, we propose a new retrieval target, hop, to collect the hidden reasoning evidence from Wikipedia for complex question answering.
Ranked #7 on
Question Answering
on HotpotQA
BinaryBERT: Pushing the Limit of BERT Quantization
In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization.
Blindfolded Attackers Still Threatening: Strict Black-Box Adversarial Attacks on Graphs
To bridge the gap between theoretical graph attacks and real-world scenarios, in this work, we propose a novel and more realistic setting: strict black-box graph attack, in which the attacker has no knowledge about the victim model at all and is not allowed to send any queries.
Improving Task-Agnostic BERT Distillation with Layer Mapping Search
Comprehensive experiments on the evaluation benchmarks demonstrate that 1) layer mapping strategy has a significant effect on task-agnostic BERT distillation and different layer mappings can result in quite different performances; 2) the optimal layer mapping strategy from the proposed search process consistently outperforms the other heuristic ones; 3) with the optimal layer mapping, our student model achieves state-of-the-art performance on the GLUE tasks.
KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
Recent studies on pre-trained language models have demonstrated their ability to capture factual knowledge and applications in knowledge-aware downstream tasks.
PPKE: Knowledge Representation Learning by Path-based Pre-training
Entities may have complex interactions in a knowledge graph (KG), such as multi-step relationships, which can be viewed as graph contextual information of the entities.
Unsupervised Adversarially-Robust Representation Learning on Graphs
Additionally, we explore a provable connection between the robustness of the unsupervised graph encoder and that of models on downstream tasks.
BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models
Complex node interactions are common in knowledge graphs (KGs), and these interactions can be considered as contextualized knowledge exists in the topological structure of KGs.
HyperText: Endowing FastText with Hyperbolic Geometry
Natural language data exhibit tree-like hierarchical structures such as the hypernym-hyponym relations in WordNet.
A Large Scale Benchmark and an Inclusion-Based Algorithm for Continuous Collision Detection
We introduce a large scale benchmark for continuous collision detection (CCD) algorithms, composed of queries manually constructed to highlight challenging degenerate cases and automatically generated using existing simulators to cover common cases.
Graphics
TernaryBERT: Distillation-aware Ultra-low Bit BERT
Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks. However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices.
Unsupervised Text Generation by Learning from Search
In this work, we present TGLS, a novel framework to unsupervised Text Generation by Learning from Search.
Learning to Detect Unacceptable Machine Translations for Downstream Tasks
In this work, we put machine translation in a cross-lingual pipeline and introduce downstream tasks to define task-specific acceptability of machine translations.
Accurate Word Alignment Induction from Neural Machine Translation
Shift-Att is an interpretation method that induces alignments from the attention weights of Transformer and does not require parameter update or architecture change.
On the Importance of Word and Sentence Representation Learning in Implicit Discourse Relation Classification
Implicit discourse relation classification is one of the most difficult parts in shallow discourse parsing as the relation prediction without explicit connectives requires the language understanding at both the text span level and the sentence level.
Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
Masked language model and autoregressive language model are two types of language models.
DynaBERT: Dynamic BERT with Adaptive Width and Depth
The pre-trained language models like BERT, though powerful in many natural language processing tasks, are both computation and memory expensive.
Neural Subgraph Isomorphism Counting
In this paper, we study a new graph learning problem: learning to count subgraph isomorphisms.
Integrating Graph Contextualized Knowledge into Pre-trained Language Models
Complex node interactions are common in knowledge graphs, and these interactions also contain rich knowledge information.
HMTNet:3D Hand Pose Estimation from Single Depth Image Based on Hand Morphological Topology
In this paper, we propose a method that takes advantage of human hand morphological topology (HMT) structure to improve the pose estimation performance.
Robotics Human-Computer Interaction
Zero-Shot Paraphrase Generation with Multilingual Language Models
Leveraging multilingual parallel texts to automatically generate paraphrases has drawn much attention as size of high-quality paraphrase corpus is limited.
A General Framework for Adaptation of Neural Machine Translation to Simultaneous Translation
Despite the success of neural machine translation (NMT), simultaneous neural machine translation (SNMT), the task of translating in real time before a full sentence has been observed, remains challenging due to the syntactic structure difference and simultaneity requirements.
Pretrained Language Models for Document-Level Neural Machine Translation
Previous work on document-level NMT usually focuses on limited contexts because of degraded performance on larger contexts.
Exploring Diverse Expressions for Paraphrase Generation
Paraphrasing plays an important role in various natural language processing (NLP) tasks, such as question answering, information retrieval and sentence simplification.
Personalized Graph Neural Networks with Attention Mechanism for Session-Aware Recommendation
The other is Dot-Product Attention mechanism, which draws on the Transformer net to explicitly model the effect of historical sessions on the current session.
TinyBERT: Distilling BERT for Natural Language Understanding
To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models.
Ranked #1 on
Natural Language Inference
on MultiNLI Dev
Assembly of randomly placed parts realized by using only one robot arm with a general parallel-jaw gripper
The proposal in this paper is verified by a simulated assembly in which a robot arm completed the assembly process including parts picking from bin and a subsequent peg-in-hole assembly.
NEZHA: Neural Contextualized Representation for Chinese Language Understanding
The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora.
Dialog State Tracking with Reinforced Data Augmentation
Neural dialog state trackers are generally limited due to the lack of quantity and diversity of annotated training data.
GPT-based Generation for Classical Chinese Poetry
We present a simple yet effective method for generating high quality classical Chinese poetry with Generative Pre-trained Language Model (GPT).
Decomposable Neural Paraphrase Generation
Paraphrasing exists at different granularity levels, such as lexical level, phrasal level and sentential level.
Triple-to-Text: Converting RDF Triples into High-Quality Natural Languages via Optimizing an Inverse KL Divergence
Knowledge base is one of the main forms to represent information in a structured way.
ERNIE: Enhanced Language Representation with Informative Entities
Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks.
Ranked #1 on
Relation Extraction
on FewRel
Efficient Fully Convolution Neural Network for Generating Pixel Wise Robotic Grasps With High Resolution Images
This paper presents an efficient neural network model to generate robotic grasps with high resolution images.
Robotics
An Investigation of Few-Shot Learning in Spoken Term Classification
In this paper, we investigate the feasibility of applying few-shot learning algorithms to a speech task.
Progressive Memory Banks for Incremental Domain Adaptation
The memory bank provides a natural way of IDA: when adapting our model to a new domain, we progressively add new slots to the memory bank, which increases the number of parameters, and thus the model capacity.
Interpretable Rationale Augmented Charge Prediction System
This paper proposes a neural based system to solve the essential interpretability problem existing in text classification, especially in charge prediction task.
CRST: a Claim Retrieval System in Twitter
For controversial topics, collecting argumentation-containing tweets which tend to be more convincing will help researchers analyze public opinions.
Interpretable Charge Predictions for Criminal Cases: Learning to Generate Court Views from Fact Descriptions
In this paper, we propose to study the problem of COURT VIEW GENeration from the fact description in a criminal case.
Paraphrase Generation with Deep Reinforcement Learning
The generator, built as a sequence-to-sequence learning model, can produce paraphrases given a sentence.
Affective Neural Response Generation
Existing neural conversational models process natural language primarily on a lexico-syntactic level, thereby ignoring one of the most crucial components of human-to-human dialogue: its affective content.
Deep Active Learning for Dialogue Generation
We propose an online, end-to-end, neural generative conversational model for open-domain dialogue.
Online Data Thinning via Multi-Subspace Tracking
At the heart of this proposed approach is an online anomaly detection method based on dynamic, low-rank Gaussian mixture models.
Neural Generative Question Answering
Empirical study shows the proposed model can effectively deal with the variations of questions and answers, and generate right and natural answers by referring to the facts in the knowledge-base.
