Search Results for author:
Found 92 papers, 24 papers with code
Show, Don’t Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue
Building universal dialogue systems that operate across multiple domains/APIs and generalize to new ones with minimal overhead is a critical challenge.
Retrieval Augmented End-to-End Spoken Dialog Models
We recently developed SLM, a joint speech and language model, which fuses a pretrained foundational speech model and a large language model (LLM), while preserving the in-context learning capability intrinsic to the pretrained LLM.
Can overfitted deep neural networks in adversarial training generalize? -- An approximation viewpoint
Adversarial training is a widely used method to improve the robustness of deep neural networks (DNNs) over adversarial perturbations.
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Multi-task Language Understanding
on MMLU
(using extra training data)
IG Captioner: Information Gain Captioners are Strong Zero-shot Classifiers
To tackle this problem, we redesign the scoring objective for the captioner to alleviate the distributional bias and focus on measuring the gain of information brought by the visual inputs.
ViP-Mixer: A Convolutional Mixer for Video Prediction
Video prediction aims to predict future frames from a video's previous content.
Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Data
Modern deep learning models are usually highly over-parameterized so that they can overfit the training data.
SLM: Bridge the thin gap between speech and text foundation models
We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models.
Graph Neural Networks Provably Benefit from Structural Information: A Feature Learning Perspective
Graph neural networks (GNNs) have pioneered advancements in graph representation learning, exhibiting superior feature learning and performance over multilayer perceptrons (MLPs) when handling graph inputs.
The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks
We show that when learning a linear model with batch normalization for binary classification, gradient descent converges to a uniform margin classifier on the training data with an $\exp(-\Omega(\log^2 t))$ convergence rate.
Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding
Large Language Models (LLMs) have been applied in the speech domain, often incurring a performance drop due to misaligned between speech and language representations.
Grammar Prompting for Domain-Specific Language Generation with Large Language Models
Large language models (LLMs) can learn to perform a wide range of natural language tasks from just a handful of in-context examples.
Can Public Large Language Models Help Private Cross-device Federated Learning?
We study (differentially) private federated learning (FL) of language models.
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference.
Learn to Cluster Faces with Better Subgraphs
Face clustering can provide pseudo-labels to the massive unlabeled face data and improve the performance of different face recognition models.
Per-Example Gradient Regularization Improves Learning Signals from Noisy Data
In this paper, we explore the per-example gradient regularization (PEGR) and present a theoretical analysis that demonstrates its effectiveness in improving both test error and robustness against noise perturbations.
The Benefits of Mixup for Feature Learning
We consider a feature-noise data model and show that Mixup training can effectively learn the rare features (appearing in a small fraction of data) from its mixture with the common features (appearing in a large fraction of data).
MUX-PLMs: Data Multiplexing for High-throughput Language Models
The widespread adoption of large language models such as ChatGPT and Bard has led to unprecedented demand for these technologies.
Binarized Neural Machine Translation
In this work, we propose a novel binarization technique for Transformers applied to machine translation (BMT), the first of its kind.
SimCGNN: Simple Contrastive Graph Neural Network for Session-based Recommendation
Session-based recommendation (SBR) problem, which focuses on next-item prediction for anonymous users, has received increasingly more attention from researchers.
AnyTOD: A Programmable Task-Oriented Dialog System
We propose AnyTOD, an end-to-end, zero-shot task-oriented dialog (TOD) system capable of handling unseen tasks without task-specific training.
Speech Aware Dialog System Technology Challenge (DSTC11)
The research on this topic is stymied by the lack of a public corpus.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners
We explore an efficient approach to establish a foundational video-text model.
Ranked #1 on
Video Captioning
on ActivityNet Captions
(using extra training data)
Fast Online Hashing with Multi-Label Projection
When a new query arrives, only the binary codes of the corresponding potential neighbors are updated.
Knowledge-grounded Dialog State Tracking
Knowledge (including structured knowledge such as schema and ontology, and unstructured knowledge such as web corpus) is a critical part of dialog understanding, especially for unseen tasks and domains.
ReAct: Synergizing Reasoning and Acting in Language Models
While large language models (LLMs) have demonstrated impressive capabilities across tasks in language understanding and interactive decision making, their abilities for reasoning (e. g. chain-of-thought prompting) and acting (e. g. action plan generation) have primarily been studied as separate topics.
Multiple Descent in the Multiple Random Feature Model
Recent works have demonstrated a double descent phenomenon in over-parameterized learning.
Unsupervised Slot Schema Induction for Task-oriented Dialog
Carefully-designed schemas describing how to collect and annotate dialog corpora are a prerequisite towards building task-oriented dialog systems.
Building Machine Translation Systems for the Next Thousand Languages
In this paper we share findings from our effort to build practical machine translation (MT) systems capable of translating across over one thousand languages.
The Implicit Length Bias of Label Smoothing on Beam Search Decoding
Label smoothing is ubiquitously applied in Neural Machine Translation (NMT) training.
Show, Don't Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue
Building universal dialogue systems that operate across multiple domains/APIs and generalize to new ones with minimal overhead is a critical challenge.
Nearest Neighbor Classifier with Margin Penalty for Active Learning
As a result, informative samples in the margin area can not be discovered and AL performance are damaged.
Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation
The dominant inductive bias applied to these models is a shared vocabulary and a shared set of parameters across languages; the inputs and labels corresponding to examples drawn from different language pairs might still reside in distinct sub-spaces.
Benign Overfitting in Two-layer Convolutional Neural Networks
In this paper, we study the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN).
Description-Driven Task-Oriented Dialog Modeling
Task-oriented dialogue (TOD) systems are required to identify key information from conversations for the completion of given tasks.
Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learning
While recent progress in massively multilingual MT is one step closer to reaching this goal, it is becoming evident that extending a multilingual MT system simply by training on more parallel data is unscalable, since the availability of labeled data for low-resource and non-English-centric language pairs is forbiddingly limited.
Benign Overfitting in Adversarially Robust Linear Classification
Our result suggests that under moderate perturbations, adversarially trained linear classifiers can achieve the near-optimal standard and adversarial risks, despite overfitting the noisy training data.
Understanding How Encoder-Decoder Architectures Attend
Moreover, how these mechanisms vary depending on the particular architecture used for the encoder and decoder (recurrent, feed-forward, etc.)
SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems
Zero/few-shot transfer to unseen services is a critical challenge in task-oriented dialogue research.
Efficient and Private Federated Learning with Partially Trainable Networks
Through extensive experiments, we empirically show that Federated learning of Partially Trainable neural networks (FedPT) can result in superior communication-accuracy trade-offs, with up to $46\times$ reduction in communication cost, at a small accuracy cost.
Towards Zero-Label Language Learning
This paper explores zero-label learning in Natural Language Processing (NLP), whereby no human-annotated data is used anywhere during training and models are trained purely on synthetic data.
Effective Sequence-to-Sequence Dialogue State Tracking
We also explore using Pegasus, a span prediction-based pre-training objective for text summarization, for the state tracking model.
Ranked #1 on
Multi-domain Dialogue State Tracking
on MULTIWOZ 2.4
Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for a class of data distributions (inspired from image data), Adam and gradient descent (GD) can converge to different global solutions of the training objective with provably different generalization errors, even with weight decay regularization.
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks.
Ranked #4 on
Visual Entailment
on SNLI-VE val
A Comprehensive Survey and Taxonomy on Single Image Dehazing Based on Deep Learning
With the development of convolutional neural networks, hundreds of deep learning based dehazing methods have been proposed.
Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures
In this paper, we study this "benign overfitting" phenomenon of the maximum margin classifier for linear classification problems.
Single-photon imaging over 200 km
Long-range active imaging has widespread applications in remote sensing and target recognition.
Improving Longer-range Dialogue State Tracking
Dialogue state tracking (DST) is a pivotal component in task-oriented dialogue systems.
Echo State Speech Recognition
We propose automatic speech recognition (ASR) models inspired by echo state network (ESN), in which a subset of recurrent neural networks (RNN) layers in the models are randomly initialized and untrained.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noise
We consider a one-hidden-layer leaky ReLU network of arbitrary width trained by stochastic gradient descent (SGD) following an arbitrary initialization.
Towards NNGP-guided Neural Architecture Search
Since NNGP inference provides a cheap measure of performance of a network architecture, we investigate its potential as a signal for neural architecture search (NAS).
The geometry of integration in text classification RNNs
Using tools from dynamical systems analysis, we study recurrent networks trained on a battery of both natural and synthetic text classification tasks.
Rapid Domain Adaptation for Machine Translation with Monolingual Data
In this paper, we propose an approach that enables rapid domain adaptation from the perspective of unsupervised translation.
Deciphering Undersegmented Ancient Scripts Using Phonetic Prior
We evaluate the model on both deciphered languages (Gothic, Ugaritic) and an undeciphered one (Iberian).
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
Massively multilingual models subsuming tens or even hundreds of languages pose great challenges to multi-task optimization.
Agnostic Learning of Halfspaces with Gradient Descent via Soft Margins
We analyze the properties of gradient descent on convex surrogates for the zero-one loss for the agnostic learning of linear halfspaces.
Agnostic Learning of a Single Neuron with Gradient Descent
In the agnostic PAC learning setting, where no assumption on the relationship between the labels $y$ and the input $x$ is made, if the optimal population risk is $\mathsf{OPT}$, we show that gradient descent achieves population risk $O(\mathsf{OPT})+\epsilon$ in polynomial time and sample complexity when $\sigma$ is strictly increasing.
Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation
Over the last few years two promising research directions in low-resource neural machine translation (NMT) have emerged.
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling
To make that practical, we show that sampling from this modified density can be achieved by sampling in latent space according to an energy-based model induced by the sum of the latent prior log-density and the discriminator output score.
Echo State Neural Machine Translation
We present neural machine translation (NMT) models inspired by echo state network (ESN), named Echo State NMT (ESNMT), in which the encoder and decoder layer weights are randomly generated then fixed throughout training.
A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
In this paper, we provide a generalized neural tangent kernel analysis and show that noisy gradient descent with weight decay can still exhibit a "kernel-like" behavior.
Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody prior
Recent neural text-to-speech (TTS) models with fine-grained latent features enable precise control of the prosody of synthesized speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis
This paper proposes a hierarchical, fine-grained and interpretable latent variable model for prosody based on the Tacotron 2 text-to-speech model.
Towards Understanding the Spectral Bias of Deep Learning
An intriguing phenomenon observed during training neural networks is the spectral bias, which states that neural networks are biased towards learning less complex functions.
How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
A recent line of research on deep learning focuses on the extremely over-parameterized setting, and shows that when the network width is larger than a high degree polynomial of the training sample size $n$ and the inverse of the target error $\epsilon^{-1}$, deep neural networks learned by (stochastic) gradient descent enjoy nice optimization and generalization guarantees.
Tight Sample Complexity of Learning One-hidden-layer Convolutional Neural Networks
We study the sample complexity of learning one-hidden-layer convolutional neural networks (CNNs) with non-overlapping filters.
Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networks
The skip-connections used in residual networks have become a standard architecture choice in deep learning due to the increased training stability and generalization performance with this architecture, although there has been limited theoretical understanding for this improvement.
Training Deep Neural Networks with Partially Adaptive Momentum
Experiments on standard benchmarks show that our proposed algorithm can maintain fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks.
Precipitation Nowcasting with Star-Bridge Networks
Precipitation nowcasting, which aims to precisely predict the short-term rainfall intensity of a local region, is gaining increasing attention in the artificial intelligence community.
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
We introduce our efforts towards building a universal neural machine translation (NMT) system capable of translating between any language pair.
Neural Decipherment via Minimum-Cost Flow: from Ugaritic to Linear B
In this paper we propose a novel neural approach for automatic decipherment of lost languages.
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
We study the training and generalization of deep neural networks (DNNs) in the over-parameterized regime, where the network width (i. e., number of hidden nodes per layer) is much larger than the number of training data points.
Gmail Smart Compose: Real-Time Assisted Writing
In this paper, we present Smart Compose, a novel system for generating interactive, real-time suggestions in Gmail that assists users in writing mails by reducing repetitive typing.
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
However, existing generalization error bounds are unable to explain the good generalization performance of over-parameterized DNNs.
Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
In particular, we study the binary classification problem and show that for a broad family of loss functions, with proper random weight initialization, both gradient descent and stochastic gradient descent can find the global minima of the training loss for an over-parameterized deep ReLU network, under mild assumption on the training data.
Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation
In this paper, we demonstrate that using pre-trained MT or text-to-speech (TTS) synthesis models to convert weakly supervised data into speech-to-translation pairs for ST training can be more effective than multi-task learning.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Hierarchical Generative Modeling for Controllable Speech Synthesis
This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions.
Towards Decomposed Linguistic Representation with Holographic Reduced Representation
The vast majority of neural models in Natural Language Processing adopt a form of structureless distributed representations.
High-Temperature Structure Detection in Ferromagnets
The goal is to distinguish whether the underlying graph is empty, i. e., the model consists of independent Rademacher variables, versus the alternative that the underlying graph contains a subgraph of a certain structure.
Training Deeper Neural Machine Translation Models with Transparent Attention
While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications.
On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
In this paper, we provide a fine-grained convergence analysis for a general class of adaptive gradient methods including AMSGrad, RMSProp and AdaGrad.
The Edge Density Barrier: Computational-Statistical Tradeoffs in Combinatorial Inference
We study the hypothesis testing problem of inferring the existence of combinatorial structures in undirected graphical models.
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
Experiments on standard benchmarks show that our proposed algorithm can maintain a fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks.
Satellite-relayed intercontinental quantum network
This was on the one hand the transmission of images in a one-time pad configuration from China to Austria as well as from Austria to China.
Quantum Physics
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder.
Ranked #35 on
Machine Translation
on WMT2014 English-French
Training Conditional Random Fields with Natural Gradient Descent
We propose a novel parameter estimation procedure that works efficiently for conditional random fields (CRF).
Local and Global Inference for High Dimensional Nonparanormal Graphical Models
Due to the presence of unknown marginal transformations, we propose a pseudo likelihood based inferential approach.
