Search Results for author:
Found 85 papers, 29 papers with code
Unified Language Model Pre-training for Natural Language Understanding and Generation
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks.
Ranked #2 on
Generative Question Answering
on CoQA
(using extra training data)
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher).
Ranked #8 on
Zero-shot Text Search
on BEIR
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Ranked #4 on
Question Generation
on SQuAD1.1
(using extra training data)
Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Pretrained bidirectional Transformers, such as BERT, have achieved significant improvements in a wide variety of language understanding tasks, while it is not straightforward to directly apply them for natural language generation.
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training.
Text Embeddings by Weakly-Supervised Contrastive Pre-training
This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks.
Ranked #11 on
Only Connect Walls Dataset Task 1 (Grouping)
on OCW
(using extra training data)
Multilingual E5 Text Embeddings: A Technical Report
This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.
Ranked #16 on
Zero-Shot Cross-Lingual Transfer
on XTREME
Inference with Reference: Lossless Acceleration of Large Language Models
We propose LLMA, an LLM accelerator to losslessly speed up Large Language Model (LLM) inference with references.
Learning to Retrieve In-Context Examples for Large Language Models
Our framework initially trains a reward model based on LLM feedback to evaluate the quality of candidate examples, followed by knowledge distillation to train a bi-encoder based dense retriever.
TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo
In this paper, we present TANDEM a real-time monocular tracking and dense mapping framework.
Generative Representational Instruction Tuning
Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss.
MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera
Unlike other multi-view stereo methods, MonoRec is able to reconstruct both static and moving objects by leveraging the predicted masks.
Fine-Tuning LLaMA for Multi-Stage Text Retrieval
Our findings demonstrate that the effectiveness of large language models indeed surpasses that of smaller models.
Behind the Scenes: Density Fields for Single View Reconstruction
By directly sampling color from the available views instead of storing color in the density field, our scene representation becomes significantly less complex compared to NeRFs, and a neural network can predict it in a single forward pass.
Neural Document Summarization by Jointly Learning to Score and Select Sentences
In this paper, we present a novel end-to-end neural network framework for extractive document summarization by jointly learning to score and select sentences.
Ranked #9 on
Extractive Text Summarization
on CNN / Daily Mail
Neural Question Generation from Text: A Preliminary Study
Automatic question generation aims to generate questions from a text passage where the generated questions can be answered by certain sub-spans of the given passage.
Ranked #13 on
Question Generation
on SQuAD1.1
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
To decouple train length from target length for efficient context window extension, we propose Positional Skip-wisE (PoSE) training that smartly simulates long inputs using a fixed context window.
Improving Text Embeddings with Large Language Models
In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps.
Selective Encoding for Abstractive Sentence Summarization
We propose a selective encoding model to extend the sequence-to-sequence framework for abstractive sentence summarization.
Ranked #8 on
Text Summarization
on DUC 2004 Task 1
Multiview Identifiers Enhanced Generative Retrieval
Instead of simply matching a query to pre-existing passages, generative retrieval generates identifier strings of passages as the retrieval target.
Learning to Rank in Generative Retrieval
However, only learning to generate is insufficient for generative retrieval.
Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval
In this paper, we propose a new dense retrieval model which learns diverse document representations with deep query interactions.
Confidence Estimation Transformer for Long-term Renewable Energy Forecasting in Reinforcement Learning-based Power Grid Dispatching
The expansion of renewable energy could help realizing the goals of peaking carbon dioxide emissions and carbon neutralization.
An Efficient Method for Detecting Asphalt Pavement Cracks and Sealed Cracks Based on a Deep Data-Driven Model
Second, we develop a dense and redundant crack annotation method based on the characteristics of the crack images.
Sequential Copying Networks
Copying mechanism shows effectiveness in sequence-to-sequence based neural network models for text generation tasks, such as abstractive sentence summarization and question generation.
Challenges in Monocular Visual Odometry: Photometric Calibration, Motion Bias and Rolling Shutter Effect
Monocular visual odometry (VO) and simultaneous localization and mapping (SLAM) have seen tremendous improvements in accuracy, robustness and efficiency, and have gained increasing popularity over recent years.
Monocular Visual Odometry
Simultaneous Localization and Mapping
Relaxed Wasserstein with Applications to GANs
Wasserstein Generative Adversarial Networks (WGANs) provide a versatile class of models, which have attracted great attention in various applications.
S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
We build the answer extraction model with state-of-the-art neural networks for single passage reading comprehension, and propose an additional task of passage ranking to help answer extraction in multiple passages.
Ambiguity set and learning via Bregman and Wasserstein
Construction of ambiguity set in robust optimization relies on the choice of divergences between probability distributions.
Brains and pseudorandom generators
In a pioneering classic, Warren McCulloch and Walter Pitts proposed a model of the central nervous system; motivated by EEG recordings of normal brain activity, Chv\' atal and Goldsmith asked whether or not this model can be engineered to provide pseudorandom number generators.
Jointly Modeling Topics and Intents with Global Order Structure
Modeling document structure is of great importance for discourse analysis and related applications.
Radical-Enhanced Chinese Character Embedding
We present a method to leverage radical for learning Chinese character embedding.
Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry
To this end, we incorporate deep depth predictions into Direct Sparse Odometry (DSO) as direct virtual stereo measurements.
Read + Verify: Machine Reading Comprehension with Unanswerable Questions
Machine reading comprehension with unanswerable questions aims to abstain from answering when no answer can be inferred.
Ranked #11 on
Question Answering
on SQuAD2.0 dev
Attention-Guided Answer Distillation for Machine Reading Comprehension
Despite that current reading comprehension systems have achieved significant advancements, their promising performances are often obtained at the cost of making an ensemble of numerous models.
A Disease Diagnosis and Treatment Recommendation System Based on Big Data Mining and Cloud Computing
It is crucial to provide compatible treatment schemes for a disease according to various symptoms at different stages.
Gated Self-Matching Networks for Reading Comprehension and Question Answering
We first match the question and passage with gated attention-based recurrent networks to obtain the question-aware passage representation.
Ranked #35 on
Question Answering
on SQuAD1.1 dev
Sequence-to-Dependency Neural Machine Translation
Nowadays a typical Neural Machine Translation (NMT) model generates translations from left to right as a linear sequence, during which latent syntactic structures of the target sentences are not explicitly concerned.
Improving Attention Modeling with Implicit Distortion and Fertility for Machine Translation
In neural machine translation, the attention mechanism facilitates the translation process by producing a soft alignment between the source sentence and the target sentence.
DirectShape: Direct Photometric Alignment of Shape Priors for Visual Vehicle Pose and Shape Estimation
Scene understanding from images is a challenging problem encountered in autonomous driving.
Multi-Frame GAN: Image Enhancement for Stereo Visual Odometry in Low Light
We propose the concept of a multi-frame GAN (MFGAN) and demonstrate its potential as an image sequence enhancement for stereo visual odometry in low light conditions.
Inspecting Unification of Encoding and Matching with Transformer: A Case Study of Machine Reading Comprehension
Most machine reading comprehension (MRC) models separately handle encoding and matching with different network architectures.
D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry
We propose D3VO as a novel framework for monocular visual odometry that exploits deep networks on three levels -- deep depth, pose and uncertainty estimation.
4Seasons: A Cross-Season Dataset for Multi-Weather SLAM in Autonomous Driving
We present a novel dataset covering seasonal and challenging perceptual conditions for autonomous driving.
LM-Reloc: Levenberg-Marquardt Based Direct Visual Relocalization
The learned features significantly improve the robustness of direct image alignment, especially for relocalization across different conditions.
Coverage Analysis for 3D Terahertz Communication Systems with Blockage and Directional Antennas
We further show that the coverage performance improvement brought by increasing the antenna directivity at APs is higher than that brought by increasing the antenna directivity at UEs.
Expected Density of Cooperative Bacteria in a 2D Quorum Sensing Based Molecular Communication System
The number of molecules observed at each randomly-distributed bacterium is first derived by characterizing the diffusion and degradation of signaling molecules within the population.
Hybrid Beamforming for Terahertz Multi-Carrier Systems over Frequency Selective Fading
We consider a three-dimensional wideband THz channel by incorporating the joint effect of molecular absorption, high sparsity, and multi-path fading, and consider the carrier frequency offset in multi-carrier systems.
Directional Modulation-Enabled Secure Transmission with Intelligent Reflecting Surface
We propose a new secure transmission scheme which uses directional modulation (DM) with artificial noise and is aided by the intelligent reflecting surface (IRS).
Coverage Analysis for 3D Terahertz Communication Systems
We conduct novel coverage probability analysis of downlink transmission in a three-dimensional (3D) terahertz (THz) communication (THzCom) system.
SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis
Specifically, in terms of usability, it is demanding for non-experts to implement deep learning models, obtain the right settings for the entire machine learning pipeline, manage models and datasets, and exploit external data sources all together.
xMoCo: Cross Momentum Contrastive Learning for Open-Domain Question Answering
Dense passage retrieval has been shown to be an effective approach for information retrieval tasks such as open domain question answering.
Spectrum Allocation with Adaptive Sub-band Bandwidth for Terahertz Communication Systems
Aided by numerical results, we show that by enabling and optimizing ASB, significantly higher throughput can be achieved as compared to adopting equal sub-band bandwidth, and this throughput gain is most profound when the power budget constraint is more stringent.
FIRe: Fast Inverse Rendering using Directional and Signed Distance Functions
To this end, we learn a signed distance function (SDF) along with our DDF model to represent a class of shapes.
Novel Spectrum Allocation Among Multiple Transmission Windows for Terahertz Communication Systems
We also show that a further data rate gain can be obtained by optimally determining the unused spectra at the edges of TWs, as compared to avoiding using pre-defined spectra at the edges of TWs.
Terahertz Communications for 6G and Beyond Wireless Networks: Challenges, Key Advancements, and Opportunities
The unprecedented increase in wireless data traffic, predicted to occur within the next decade, is motivating academia and industries to look beyond contemporary wireless standards and conceptualize the sixth-generation (6G) wireless networks.
An Unsupervised Learning Approach for Spectrum Allocation in Terahertz Communication Systems
Numerical results demonstrate that comparing to existing approaches, our proposed unsupervised learning-based approach achieves a higher data rate, especially when the molecular absorption coefficient within the spectrum of interest varies in a highly non-linear manner.
CCR: Facial Image Editing with Continuity, Consistency and Reversibility
In this work, we put forward three concepts and corresponding definitions: editing continuity, consistency, and reversibility.
Adversarial Transformer for Repairing Human Airway Segmentation
Discontinuity in the delineation of peripheral bronchioles hinders the potential clinical application of automated airway segmentation models.
Terahertz Communications for Massive Connectivity and Security in 6G and Beyond Era
Terahertz (THz) communications (THzCom) has experienced a meteoric rise of interest, due to its benefits for ultra-high data rate transmission in the sixth generation (6G) and beyond era.
Title2Event: Benchmarking Open Event Extraction with a Large-scale Chinese Title Dataset
To the best of our knowledge, it is currently the largest manually-annotated Chinese dataset for open event extraction.
4Seasons: Benchmarking Visual SLAM and Long-Term Localization for Autonomous Driving in Challenging Conditions
In this paper, we present a novel visual SLAM and long-term localization benchmark for autonomous driving in challenging conditions based on the large-scale 4Seasons dataset.
Random Padding Data Augmentation
Random Padding is parameter-free, simple to construct, and compatible with the majority of CNN-based recognition models.
FedIL: Federated Incremental Learning from Decentralized Unlabeled Data with Convergence Analysis
Most existing federated learning methods assume that clients have fully labeled data to train on, while in reality, it is hard for the clients to get task-specific labels due to users' privacy concerns, high labeling costs, or lack of expertise.
Real-time scheduling of renewable power systems through planning-based reinforcement learning
The well-trained scheduling agent significantly reduces renewable curtailment and load shedding, which are issues arising from traditional scheduling's reliance on inaccurate day-ahead forecasts.
Query2doc: Query Expansion with Large Language Models
This paper introduces a simple yet effective query expansion approach, denoted as query2doc, to improve both sparse and dense retrieval systems.
FedMAE: Federated Self-Supervised Learning with One-Block Masked Auto-Encoder
Latest federated learning (FL) methods started to focus on how to use unlabeled data in clients for training due to users' privacy concerns, high labeling costs, or lack of expertise.
Combining Adversaries with Anti-adversaries in Training
On the basis of our theoretical findings, a more general learning objective that combines adversaries and anti-adversaries with varied bounds on each training sample is presented.
UAV-assisted IoT Monitoring Network: Adaptive Multiuser Access for Low-Latency and High-Reliability Under Bursty Traffic
In this work, we propose an adaptive system design for an Internet of Things (IoT) monitoring network with latency and reliability requirements, where IoT devices generate time-critical and event-triggered bursty traffic, and an unmanned aerial vehicle (UAV) aggregates and relays sensed data to the base station.
Incremental Dense Reconstruction from Monocular Video with Guided Sparse Feature Volume Fusion
Incrementally recovering 3D dense structures from monocular videos is of paramount importance since it enables various robotics and AR applications.
Project Aria: A New Tool for Egocentric Multi-Modal AI Research
Egocentric, multi-modal data as available on future augmented reality (AR) devices provides unique challenges and opportunities for machine perception.
Large Search Model: Redefining Search Stack in the Era of LLMs
Modern search engines are built on a stack of different components, including query understanding, retrieval, multi-stage ranking, and question answering, among others.
Enhancing Traffic Object Detection in Variable Illumination with RGB-Event Fusion
To address this issue, we introduce bio-inspired event cameras and propose a novel Structure-aware Fusion Network (SFNet) that extracts sharp and complete object structures from the event stream to compensate for the lost information in images through cross-modality fusion, enabling the network to obtain illumination-robust representations for traffic object detection.
Time-Frequency Localization Characteristics of the Delay-Doppler Plane Orthogonal Pulse
Furthermore, we determine the TFA for the recently proposed generalized design of the DDOP.
Event-driven Real-time Retrieval in Web Search
The Event information is then integrated with the query through a cross-attention mechanism, resulting in a time-context query representation.
Data is all you need: Finetuning LLMs for Chip Design via an Automated design-data augmentation framework
Our 13B model (ChipGPT-FT) has a pass rate improvement compared with GPT-3. 5 in Verilog generation and outperforms in EDA script (i. e., SiliconCompiler) generation with only 200 EDA script data.
