Search Results for author:
Found 129 papers, 59 papers with code
Tencent AI Lab Machine Translation Systems for WMT20 Chat Translation Task
This paper describes the Tencent AI Lab’s submission of the WMT 2020 shared task on chat translation in English-German.
The USYD-JD Speech Translation System for IWSLT2021
Our constrained system is based on a pipeline framework, i. e. ASR and NMT.
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation
In this work, we provide an appealing alternative for NAT – monolingual KD, which trains NAT student on external monolingual data with AT teacher trained on the original bilingual data.
VisuoThink: Empowering LVLM Reasoning with Multimodal Tree Search
Recent advancements in Large Vision-Language Models have showcased remarkable capabilities.
AgentDropout: Dynamic Agent Elimination for Token-Efficient and High-Performance LLM-Based Multi-Agent Collaboration
Multi-agent systems (MAS) based on large language models (LLMs) have demonstrated significant potential in collaborative problem-solving.
Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
High-resolution (HR) image perception remains a key challenge in multimodal large language models (MLLMs).
Edit Once, Update Everywhere: A Simple Framework for Cross-Lingual Knowledge Synchronization in LLMs
Knowledge editing allows for efficient adaptation of large language models (LLMs) to new information or corrections without requiring full retraining.
Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding
Large language models (LLMs) excel at a range of tasks through in-context learning (ICL), where only a few task examples guide their predictions.
"Short-length" Adversarial Training Helps LLMs Defend "Long-length" Jailbreak Attacks: Theoretical and Empirical Evidence
This paper focuses on adversarial suffix jailbreak attacks and unveils that to defend against a jailbreak attack with an adversarial suffix of length $\Theta(M)$, it is enough to align LLMs on prompts with adversarial suffixes of length $\Theta(\sqrt{M})$.
TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
Zeroth-order optimization (ZO) has demonstrated remarkable promise in efficient fine-tuning tasks for Large Language Models (LLMs).
The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking
This work identifies the Energy Loss Phenomenon in Reinforcement Learning from Human Feedback (RLHF) and its connection to reward hacking.
Leveraging Metamemory Mechanisms for Enhanced Data-Free Code Generation in LLMs
Automated code generation using large language models (LLMs) has gained attention due to its efficiency and adaptability.
Self-Evolution Knowledge Distillation for LLM-based Machine Translation
Knowledge distillation (KD) has shown great promise in transferring knowledge from larger teacher models to smaller student models.
DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs
Efficient KV cache management in LLMs is crucial for long-context tasks like RAG and summarization.
CogSteer: Cognition-Inspired Selective Layer Intervention for Efficient Semantic Steering in Large Language Models
Despite their impressive capabilities, large language models (LLMs) often lack interpretability and can generate toxic content.
Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQL
Large Language Models (LLMs) have shown promising performance in text-to-SQL, which involves translating natural language questions into SQL queries.
Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models
By revisiting the Memory-efficient ZO (MeZO) optimizer, we discover that the full-parameter perturbation and updating processes consume over 50% of its overall fine-tuning time cost.
Self-Powered LLM Modality Expansion for Large Speech-Text Models
This study aims to refine the use of speech datasets for LSM training by addressing the limitations of vanilla instruction tuning.
End-to-End Graph Flattening Method for Large Language Models
In recent years, the breakthrough of Large Language Models (LLMs) offers new ideas for achieving universal methods on graph data.
MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators
To enhance the quality of error annotations predicted by LLM evaluators, we introduce a universal and training-free framework, $\textbf{MQM-APE}$, based on the idea of filtering out non-impactful errors by Automatically Post-Editing (APE) the original translation based on each error, leaving only those errors that contribute to quality improvement.
Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models
Knowledge distillation (KD) is a technique that compresses large teacher models by training smaller student models to mimic them.
$\mathbb{USCD}$: Improving Code Generation of LLMs by Uncertainty-Aware Selective Contrastive Decoding
Then, we selectively eliminate output noise induced by lame prompts based on the uncertainty of the prediction distribution from the standard prompt.
Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models
Building upon this insight, we propose Divide, Conquer and Combine (DC$^2$), a novel training-free framework for enhancing MLLM perception of HR images.
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
In aligning large language models (LLMs), utilizing feedback from existing advanced AI rather than humans is an important method to scale supervisory signals.
Revisiting Catastrophic Forgetting in Large Language Model Tuning
Catastrophic Forgetting (CF) means models forgetting previously acquired knowledge when learning new data.
Uncertainty Aware Learning for Language Model Alignment
As instruction-tuned large language models (LLMs) evolve, aligning pretrained foundation models presents increasing challenges.
Demystifying the Compression of Mixture-of-Experts Through a Unified Framework
Extensive experimental results demonstrate the effectiveness of the compression methods under our framework and the proposed recipe, achieving a 6. 05x speedup and only 20. 0GB memory usage while maintaining over 92% of performance on Mixtral-8x7B.
Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuning
Specifically, we augment the arbitrary first-order logical queries via binary tree decomposition, to stimulate the reasoning capability of LLMs.
3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset
Experimental results show that MMT models trained on our dataset exhibit a greater ability to exploit visual information than those trained on other MMT datasets.
Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems
To this end, we propose a simple-yet-effective method, namely Deeply Understanding the Problems (DUP), to improve the LLMs' math problem-solving ability by addressing semantic misunderstanding errors.
Ranked #1 on
Math Word Problem Solving
on SVAMP
(Accuracy metric)
Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
Our method is inspired by our observation that what we call disturbance instructions significantly exacerbate hallucinations in multimodal fusion modules.
Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning
In this work, we design a two-stage fine-tuning algorithm to improve the instruction-following ability (especially the translation direction) of LLMs.
Kernel Multigrid: Accelerate Back-fitting via Sparse Gaussian Process Regression
By utilizing a technique called Kernel Packets (KP), we prove that the convergence rate of Back-fitting is no faster than $(1-\mathcal{O}(\frac{1}{n}))^t$, where $n$ and $t$ denote the data size and the iteration number, respectively.
Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extraction
Experiments across various prompts, PLMs, and benchmarks show that our approach can not only correct the overfitted performance caused by prompt bias, but also significantly improve the prompt retrieval capability (up to 10% absolute performance gain).
Towards Training A Chinese Large Language Model for Anesthesiology
3) We introduce a standardized benchmark for evaluating medical LLM in Anesthesiology.
Healthcare Copilot: Eliciting the Power of General LLMs for Medical Consultation
The copilot framework, which aims to enhance and tailor large language models (LLMs) for specific complex tasks without requiring fine-tuning, is gaining increasing attention from the community.
DB-LLM: Accurate Dual-Binarization for Efficient LLMs
Large language models (LLMs) have significantly advanced the field of natural language processing, while the expensive memory and computation consumption impede their practical deployment.
ROSE Doesn't Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding
With the development of instruction-tuned large language models (LLMs), improving the safety of LLMs has become more critical.
Revisiting Knowledge Distillation for Autoregressive Language Models
Knowledge distillation (KD) is a common approach to compress a teacher model to reduce its inference cost and memory footprint, by training a smaller student model.
InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling
Inspired by this finding, we propose the Cluster Separation Index (CSI), which quantifies deviations in the IB latent space, as an indicator of reward overoptimization to facilitate the development of online mitigation strategies.
A General Theory for Kernel Packets: from state space model to compactly supported basis
It is well known that the state space (SS) model formulation of a Gaussian process (GP) can lower its training and prediction time both to $\CalO(n)$ for $n$ data points.
Revisiting Demonstration Selection Strategies in In-Context Learning
In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent.
WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge
Sentiment analysis is rapidly advancing by utilizing various data modalities (e. g., text, image).
Intention Analysis Makes LLMs A Good Jailbreak Defender
Extensive experiments on varying jailbreak benchmarks across a wide range of LLMs show that $\mathbb{IA}$ could consistently and significantly reduce the harmfulness in responses (averagely -48. 2% attack success rate).
OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models
Advancing automated programming necessitates robust and comprehensive code generation benchmarks, yet current evaluation frameworks largely neglect object-oriented programming (OOP) in favor of functional programming (FP), e. g., HumanEval and MBPP.
POMP: Probability-driven Meta-graph Prompter for LLMs in Low-resource Unsupervised Neural Machine Translation
POMP involves constructing a directed acyclic meta-graph for each source language, from which we dynamically sample multiple paths to prompt LLMs to mitigate the linguistic noise and improve translations during training.
Sheared Backpropagation for Fine-tuning Foundation Models
To address these challenges we introduce PreBackRazor a novel activation pruning scheme offering both computational and memory efficiency through a sparsified backpropagation strategy which judiciously avoids unnecessary activation pruning and storage and gradient computation.
Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion
At the upper level, we focus on learning a shared Concrete mask to identify the subspace, while at the inner level, model merging is performed to maximize the performance of the merged model.
Exploring Sparsity in Graph Transformers
Graph Transformers (GTs) have achieved impressive results on various graph-related tasks.
SpliceMix: A Cross-scale and Semantic Blending Augmentation Strategy for Multi-label Image Classification
The "splice" in our method is two-fold: 1) Each mixed image is a splice of several downsampled images in the form of a grid, where the semantics of images attending to mixing are blended without object deficiencies for alleviating co-occurred bias; 2) We splice mixed images and the original mini-batch to form a new SpliceMixed mini-batch, which allows an image with different scales to contribute to training together.
Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models
The key algorithm in solving ZSAQ is the SAM-SGA optimization, which aims to improve the quantization accuracy and model generalization via optimizing a minimax problem.
Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
The Mixture of Experts (MoE) has emerged as a highly successful technique in deep learning, based on the principle of divide-and-conquer to maximize model capacity without significant additional computational cost.
Merging Experts into One: Improving Computational Efficiency of Mixture of Experts
Although a sparse Mixture of Experts (MoE) can reduce the cost by activating a small subset of parameters (e. g., one expert) for each input, its computation escalates significantly if increasing the number of activated experts, limiting its practical utility.
Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation
Zero-shot translation (ZST), which is generally based on a multilingual neural machine translation model, aims to translate between unseen language pairs in training data.
Deep Model Fusion: A Survey
Specifically, we categorize existing deep model fusion methods as four-fold: (1) "Mode connectivity", which connects the solutions in weight space via a path of non-increasing loss, in order to obtain better initialization for model fusion; (2) "Alignment" matches units between neural networks to create better conditions for fusion; (3) "Weight average", a classical model fusion method, averages the weights of multiple models to obtain more accurate results closer to the optimal solution; (4) "Ensemble learning" combines the outputs of diverse models, which is a foundational technique for improving the accuracy and robustness of the final model.
MerA: Merging Pretrained Adapters For Few-Shot Learning
Adapter tuning, which updates only a few parameters, has become a mainstream method for fine-tuning pretrained language models to downstream tasks.
Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models
We evaluate our method on both open and closed LLMs, and the experiments on the widely-used public dataset show that our method can generate more consistent responses in a long-context conversation.
Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?
The multimedia community has shown a significant interest in perceiving and representing the physical world with multimodal pretrained neural network models, and among them, the visual-language pertaining (VLP) is, currently, the most captivating topic.
Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup
Adaptive optimization has achieved notable success for distributed learning while extending adaptive optimizer to federated Learning (FL) suffers from severe inefficiency, including (i) rugged convergence due to inaccurate gradient estimation in global adaptive optimizer; (ii) client drifts exacerbated by local over-fitting with the local adaptive optimizer.
Free-Form Composition Networks for Egocentric Action Recognition
We evaluated our method on three popular egocentric action recognition datasets, Something-Something V2, H2O, and EPIC-KITCHENS-100, and the experimental results demonstrate the effectiveness of the proposed method for handling data scarcity problems, including long-tailed and few-shot egocentric action recognition.
Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training
Dense retrievers have achieved impressive performance, but their demand for abundant training data limits their application scenarios.
Divide, Conquer, and Combine: Mixture of Semantic-Independent Experts for Zero-Shot Dialogue State Tracking
Zero-shot transfer learning for Dialogue State Tracking (DST) helps to handle a variety of task-oriented dialogue domains without the cost of collecting in-domain data.
Self-Evolution Learning for Discriminative Language Model Pretraining
Masked language modeling, widely used in discriminative language model (e. g., BERT) pretraining, commonly adopts a random masking strategy.
Revisiting Token Dropping Strategy in Efficient BERT Pretraining
Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT, by skipping the computation of a subset of the input tokens at several middle layers.
Self-Evolution Learning for Mixup: Enhance Data Augmentation on Few-Shot Text Classification Tasks
However, most of the mixup methods do not consider the varying degree of learning difficulty in different stages of training and generate new samples with one hot labels, resulting in the model over confidence.
Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape
In federated learning (FL), a cluster of local clients are chaired under the coordination of the global server and cooperatively train one model with privacy protection.
Random Smoothing Regularization in Kernel Gradient Descent Learning
In this paper, we aim to bridge this gap by presenting a framework for random smoothing regularization that can adaptively and effectively learn a wide range of ground truth functions belonging to the classical Sobolev spaces.
Representing Additive Gaussian Processes by Sparse Matrices
We show how to use these sparse formulas to generalize back-fitting-based algorithms to efficiently compute the posterior mean, posterior variance, log-likelihood, and gradient of these three functions for additive GPs, all in $O(n \log n)$ time.
Prompt-Learning for Cross-Lingual Relation Extraction
Experiments on WMT17-EnZh XRE also show the effectiveness of our Prompt-XRE against other competitive baselines.
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review
The field of deep learning has witnessed significant progress, particularly in computer vision (CV), natural language processing (NLP), and speech.
Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
To further improve the performance of LLMs on MT quality assessment, we investigate several prompting designs, and propose a new prompting method called \textbf{\texttt{Error Analysis Prompting}} (EAPrompt) by combining Chain-of-Thoughts (Wei et al., 2022) and Error Analysis (Lu et al., 2023).
Towards Making the Most of ChatGPT for Machine Translation
We show that: 1) The performance of ChatGPT depends largely on temperature, and a lower temperature usually can achieve better performance; 2) Emphasizing the task information can further improve ChatGPT's performance, particularly in complex MT tasks; 3) Introducing domain information can elicit ChatGPT's generalization ability and improve its performance in the specific domain; 4) ChatGPT tends to generate hallucinations for non-English-centric MT tasks, which can be partially addressed by our proposed prompts but still need to be highlighted for the MT/NLP community.
AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks
Integrating SAM with adaptive learning rate and momentum acceleration, dubbed AdaSAM, has already been explored empirically to train large-scale deep neural networks without theoretical guarantee due to the triple difficulties in analyzing the coupled perturbation step, adaptive learning rate and momentum step.
OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML System
Automated machine learning (AutoML) seeks to build ML models with minimal human effort.
FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracy
Federated learning is an emerging distributed machine learning framework which jointly trains a global model via a large number of local devices with data privacy protections.
Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT
Recently, ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries.
Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE
This technical report briefly describes our JDExplore d-team's submission Vega v1 on the General Language Understanding Evaluation (GLUE) leaderboard, where GLUE is a collection of nine natural language understanding tasks, including question answering, linguistic acceptability, sentiment analysis, text similarity, paraphrase detection, and natural language inference.
Original or Translated? On the Use of Parallel Data for Translation Quality Estimation
Machine Translation Quality Estimation (QE) is the task of evaluating translation output in the absence of human-written references.
Toward Human-Like Evaluation for Natural Language Generation with Error Analysis
To this end, we augment BARTScore by incorporating the human-like error analysis strategies, namely BARTScore++, where the final score consists of both the evaluations of major errors and minor errors.
Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE
This technical report briefly describes our JDExplore d-team's Vega v2 submission on the SuperGLUE leaderboard.
Ranked #1 on
Common Sense Reasoning
on ReCoRD
Improving Simultaneous Machine Translation with Monolingual Data
Preliminary experiments on En-Zh and En-Ja news domain corpora demonstrate that monolingual data can significantly improve translation quality (e. g., +3. 15 BLEU on En-Zh).
PAD-Net: An Efficient Framework for Dynamic Networks
The main contributions of our work are challenging the basic commonsense in dynamic networks and proposing a partially dynamic network, namely PAD-Net, to transform the redundant dynamic parameters into static ones.
Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models
Fine-tuning large pretrained language models on a limited training corpus usually suffers from poor generalization.
SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters
Adapter Tuning, which freezes the pretrained language models (PLMs) and only fine-tunes a few extra modules, becomes an appealing efficient alternative to the full model fine-tuning.
Vega-MT: The JD Explore Academy Translation System for WMT22
As for model sizes, we scale the Transformer-Big up to the extremely large model that owns nearly 4. 7 Billion parameters, to fully enhance the model capacity for our Vega-MT.
Ranked #1 on
Machine Translation
on WMT 2022 English-Russian
On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation
Pre-Training (PT) of text representations has been successfully applied to low-resource Neural Machine Translation (NMT).
PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation
Prompt Transfer (PoT) is a recently-proposed approach to improve prompt-tuning, by initializing the target prompt with the existing prompt trained on similar source tasks.
Comprehensive Graph Gradual Pruning for Sparse Training in Graph Neural Networks
However, the LTH-based methods suffer from two major drawbacks: 1) they require exhaustive and iterative training of dense models, resulting in an extremely large training computation cost, and 2) they only trim graph structures and model parameters but ignore the node feature dimension, where significant redundancy exists.
Dynamic Contrastive Distillation for Image-Text Retrieval
Although the vision-and-language pretraining (VLP) equipped cross-modal image-text retrieval (ITR) has achieved remarkable progress in the past two years, it suffers from a major drawback: the ever-increasing size of VLP models restricts its deployment to real-world search scenarios (where the high latency is unacceptable).
E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation
To verify our hypothesis, we first empirically study the functionalities of the encoder and decoder in seq2seq pretrained language models, and find that the encoder takes an important but under-exploitation role than the decoder regarding the downstream performance and neuron activation.
Parameter-Efficient and Student-Friendly Knowledge Distillation
In this paper, we present a parameter-efficient and student-friendly knowledge distillation method, namely PESF-KD, to achieve efficient and sufficient knowledge transfer by updating relatively few partial parameters.
Interpretable Proof Generation via Iterative Backward Reasoning
We present IBR, an Iterative Backward Reasoning model to solve the proof generation tasks on rule-based Question Answering (QA), where models are required to reason over a series of textual rules and facts to find out the related proof path and derive the final answer.
BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation
Data augmentations (DA) are the cores to achieving robust sequence-to-sequence learning on various natural language processing (NLP) tasks.
A Contrastive Cross-Channel Data Augmentation Framework for Aspect-based Sentiment Analysis
Aspect-based sentiment analysis (ABSA) is a fine-grained sentiment analysis task, which focuses on detecting the sentiment polarity towards the aspect in a sentence.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
+4
Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understanding
For multilingual sequence-to-sequence pretrained language models (multilingual Seq2Seq PLMs), e. g. mBART, the self-supervised pretraining task is trained on a wide range of monolingual languages, e. g. 25 languages from CommonCrawl, while the downstream cross-lingual tasks generally progress on a bilingual language subset, e. g. English-German, making there exists the data discrepancy, namely domain discrepancy, and cross-lingual learning objective discrepancy, namely task discrepancy, between the pretraining and finetuning stages.
SD-Conv: Towards the Parameter-Efficiency of Dynamic Convolution
Dynamic convolution achieves better performance for efficient CNNs at the cost of negligible FLOPs increase.
Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning
Instead, we propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG), which relieves the issue of direct model aggregation.
Where Does the Performance Improvement Come From? -- A Reproducibility Concern about Image-Text Retrieval
In contrast to previous works, we focus on the reproducibility of the approaches and the examination of the elements that lead to improved performance by pretrained and nonpretrained models in retrieving images and text.
Kernel Packet: An Exact and Scalable Algorithm for Gaussian Process Regression with Matérn Correlations
We develop an exact and scalable algorithm for one-dimensional Gaussian process regression with Mat\'ern correlations whose smoothness parameter $\nu$ is a half-integer.
Improving Neural Machine Translation by Denoising Training
We present a simple and effective pretraining strategy {D}en{o}ising {T}raining DoT for neural machine translation.
Knowledge Graph Augmented Network Towards Multiview Representation Learning for Aspect-based Sentiment Analysis
To this end, we propose a knowledge graph augmented network KGAN, which aims to effectively incorporate external knowledge with explicitly syntactic and contextual information.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
+2
A Sparse Expansion For Deep Gaussian Processes
In this work, we use Deep Gaussian Processes (DGPs) as statistical surrogates for stochastic processes with complex distributions.
Unified Instance and Knowledge Alignment Pretraining for Aspect-based Sentiment Analysis
In practice, we formulate the model pretrained on the sampled instances into a knowledge guidance model and a learner model, respectively.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
+2
On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation
Pre-training (PT) and back-translation (BT) are two simple and powerful methods to utilize monolingual data for improving the model performance of neural machine translation (NMT).
FLBoost: On-the-Fly Fine-tuning Boosts Federated Learning via Data-free Distillation
On the contrary, we propose a new solution: on-the-fly fine-tuning the global model in server via data-free distillation to boost its performance, dubbed FLBoost to relieve the issue of direct model aggregation.
Improving Neural Machine Translation by Bidirectional Training
We present a simple and effective pretraining strategy -- bidirectional training (BiT) for neural machine translation.
The USYD-JD Speech Translation System for IWSLT 2021
Our constrained system is based on a pipeline framework, i. e. ASR and NMT.
High-Dimensional Simulation Optimization via Brownian Fields and Sparse Grids
High-dimensional simulation optimization is notoriously challenging.
On the Copying Behaviors of Pre-Training for Neural Machine Translation
In response to this problem, we propose a simple and effective method named copying penalty to control the copying behaviors in decoding.
Progressive Multi-Granularity Training for Non-Autoregressive Translation
Non-autoregressive translation (NAT) significantly accelerates the inference process via predicting the entire target sequence.
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Results demonstrate that the proposed approach can significantly and universally improve translation quality by reducing translation errors on low-frequency words.
Self-Guided Curriculum Learning for Neural Machine Translation
In the field of machine learning, the well-trained model is assumed to be able to recover the training labels, i. e. the synthetic labels predicted by the model should be as close to the ground-truth labels as possible.
Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding
Spoken language understanding (SLU) system usually consists of various pipeline components, where each component heavily relies on the results of its upstream ones.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+7
Towards Efficiently Diversifying Dialogue Generation via Embedding Augmentation
Dialogue generation models face the challenge of producing generic and repetitive responses.
MirrorAlign: A Super Lightweight Unsupervised Word Alignment Model via Cross-Lingual Contrastive Learning
Recently, the performance of the neural word alignment models has exceeded that of statistical models.
Unsupervised Word Alignment via Cross-Lingual Contrastive Learning
Recently, the performance of the neural word alignment models has exceeded that of statistical models.
Understanding and Improving Lexical Choice in Non-Autoregressive Translation
To this end, we introduce an extra Kullback-Leibler divergence term derived by comparing the lexical choice of NAT model and that embedded in the raw data.
Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
Encoder layer fusion (EncoderFusion) is a technique to fuse all the encoder layers (instead of the uppermost layer) for sequence-to-sequence (Seq2Seq) models, which has proven effective on various NLP tasks.
Context-Aware Cross-Attention for Non-Autoregressive Translation
Non-autoregressive translation (NAT) significantly accelerates the inference process by predicting the entire target sequence.
Sample and Computationally Efficient Stochastic Kriging in High Dimensions
However, its use is limited to cases where the design space is low-dimensional because, in general, the sample complexity (i. e., the number of design points required for stochastic kriging to produce an accurate prediction) grows exponentially in the dimensionality of the design space.
Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns
In response to this issue, we propose to expose explicit cross-lingual patterns, \textit{e. g.} word alignments and generation score, to our proposed zero-shot models.
SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Filling
Slot filling and intent detection are two main tasks in spoken language understanding (SLU) system.
High-Dimensional Non-Parametric Density Estimation in Mixed Smooth Sobolev Spaces
Density estimation plays a key role in many tasks in machine learning, statistical inference, and visualization.
Self-Attention with Cross-Lingual Position Representation
Position encoding (PE), an essential part of self-attention networks (SANs), is used to preserve the word order information for natural language processing tasks, generating fixed position indices for input sequences.
Generalization Guarantees for Sparse Kernel Approximation with Entropic Optimal Features
Despite their success, kernel methods suffer from a massive computational cost in practice.
Recurrent Graph Syntax Encoder for Neural Machine Translation
Syntax-incorporated machine translation models have been proven successful in improving the model's reasoning and meaning preservation ability.
The University of Sydney's Machine Translation System for WMT19
This paper describes the University of Sydney's submission of the WMT 2019 shared news translation task.
Ranked #1 on
Machine Translation
on WMT 2018 Finnish-English
Efficient Learning of Optimal Markov Network Topology with k-Tree Modeling
The seminal work of Chow and Liu (1968) shows that approximation of a finite probabilistic system by Markov trees can achieve the minimum information loss with the topology of a maximum spanning tree.
