Search Results for author:
Found 64 papers, 23 papers with code
Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models
Recent advancements in long-context language models (LCLMs) promise to transform Retrieval-Augmented Generation (RAG) by simplifying pipelines.
Training Software Engineering Agents and Verifiers with SWE-Gym
When combined with our fine-tuned SWE agents, we achieve 32. 0% and 26. 0% on SWE-Bench Verified and Lite, respectively, reflecting a new state-of-the-art for open-weight SWE agents.
Normalizing Flows are Capable Generative Models
Normalizing Flows (NFs) are likelihood-based models for continuous inputs.
Ranked #12 on
Image Generation
on ImageNet 128x128
Visatronic: A Multimodal Decoder-Only Model for Speech Synthesis
In this paper, we propose a new task -- generating speech from videos of people and their transcripts (VTTS) -- to motivate new techniques for multimodal speech generation.
TypeScore: A Text Fidelity Metric for Text-to-Image Generative Models
In this work, we focus on the text rendering aspect of these models, which provides a lens for evaluating a generative model's fine-grained instruction-following capabilities.
Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
Then we jointly train a prompt generator, optimized to produce a prompt embedding that stays close to the aggregated summary while minimizing task loss at the same time.
DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation
Furthermore, DART seamlessly trains with both text and image data in a unified model.
dMel: Speech Tokenization made Simple
Large language models have revolutionized natural language processing by leveraging self-supervised pretraining on vast textual data.
Improving GFlowNets for Text-to-Image Diffusion Alignment
Diffusion models have become the de-facto approach for generating visual data, which are trained to match the distribution of the training dataset.
Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling
Furthermore, we show that Kaleido adheres closely to the guidance provided by the generated latent variables, demonstrating its capability to effectively control and direct the image generation process.
Denoising LM: Pushing the Limits of Error Correction Models for Speech Recognition
Language models (LMs) have long been used to improve results of automatic speech recognition (ASR) systems, but they are unaware of the errors that ASR systems make.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
How Far Are We from Intelligent Visual Deductive Reasoning?
Vision-Language Models (VLMs) have recently demonstrated incredible strides on diverse vision language tasks.
Divide-or-Conquer? Which Part Should You Distill Your LLM?
Recent methods have demonstrated that Large Language Models (LLMs) can solve reasoning tasks better when they are encouraged to solve subtasks of the main task first.
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased.
KGLens: Towards Efficient and Effective Knowledge Probing of Large Language Models with Knowledge Graphs
Large Language Models (LLMs) might hallucinate facts, while curated Knowledge Graph (KGs) are typically factually reliable especially with domain-specific knowledge.
Swallowing the Bitter Pill: Simplified Scalable Conformer Generation
We present a novel way to predict molecular conformers through a simple formulation that sidesteps many of the heuristics of prior works and achieves state of the art results by using the advantages of scale.
Matryoshka Diffusion Models
Diffusion models are the de facto approach for generating high-quality images and videos, but learning high-dimensional models remains a formidable task due to computational and optimization challenges.
Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games
In this paper, we offer a surrogate problem which assesses an LLMs's capability to deduce an entity unknown to itself, but revealed to a judge, by asking the judge a series of queries.
Construction of Paired Knowledge Graph-Text Datasets Informed by Cyclic Evaluation
Guided by these observations, we construct a new, improved dataset called LAGRANGE using heuristics meant to improve equivalence between KG and text and show the impact of each of the heuristics on cyclic evaluation.
REALM: Robust Entropy Adaptive Loss Minimization for Improved Single-Sample Test-Time Adaptation
In online F-TTA, a pre-trained model is adapted using a stream of test samples by minimizing a self-supervised objective, such as entropy minimization.
Robotic Table Tennis: A Case Study into a High Speed Learning System
We present a deep-dive into a real-world robotic learning system that, in previous work, was shown to be capable of hundreds of table tennis rallies with a human and has the ability to precisely return the ball to desired targets.
PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model
Autoregressive models for text sometimes generate repetitive and low-quality output because errors accumulate during the steps of generation.
Understanding the Robustness of Multi-Exit Models under Common Corruptions
Furthermore, the lack of calibration increases the inconsistency in the predictions of the model across exits, leading to both inefficient inference and more misclassifications compared with evaluation on in-distribution data.
Continuous Soft Pseudo-Labeling in ASR
Continuous pseudo-labeling (PL) algorithms such as slimIPL have recently emerged as a powerful strategy for semi-supervised learning in speech recognition.
More Speaking or More Speakers?
We perform a systematic analysis on both labeled and unlabeled data by varying the number of speakers while keeping the number of hours fixed and vice versa.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Continuous Pseudo-Labeling from the Start
Nevertheless, these approaches still rely on bootstrapping the ST using an initial supervised learning phase where the model is trained on labeled data alone.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Position Prediction as an Effective Pretraining Strategy
This pretraining strategy which has been used in BERT models in NLP, Wav2Vec models in Speech and, recently, in MAE models in Vision, forces the model to learn about relationships between the content in different parts of the input using autoencoding related objectives.
Efficient Representation Learning via Adaptive Context Pooling
Inspired by the success of ConvNets that are combined with pooling to capture long-range dependencies, we learn to pool neighboring features for each token before computing attention in a given attention layer.
RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions
On a long-form YouTube test set, when the nonstreaming RNN-T model is trained with shorter segments of data, the proposed combination improves word error rate (WER) from 22. 3% to 14. 8%; when the streaming RNN-T model trained on short Search queries, the proposed techniques improve WER on the YouTube set from 67. 0% to 25. 3%.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Robotic Table Tennis with Model-Free Reinforcement Learning
We propose a model-free algorithm for learning efficient policies capable of returning table tennis balls by controlling robot joints at a rate of 100Hz.
Imputer: Sequence Modelling via Imputation and Dynamic Programming
This paper presents the Imputer, a neural sequence model that generates output sequences iteratively via imputations.
SPIN: A High Speed, High Resolution Vision Dataset for Tracking and Action Recognition in Ping Pong
The corpus consists of ping pong play with three main annotation streams that can be used to learn tracking and action recognition models -- tracking of the ping pong ball and poses of humans in the videos and the spin of the ball being hit by humans.
Policy Optimization by Local Improvement through Search
On the other end of the spectrum, approaches rooted in Policy Iteration, such as Dual Policy Iteration do not choose next step actions based on an expert, but instead use planning or search over the policy to choose an action distribution to train towards.
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
Peptide-Spectra Matching from Weak Supervision
As in many other scientific domains, we face a fundamental problem when using machine learning to identify proteins from mass spectrometry data: large ground truth datasets mapping inputs to correct outputs are extremely difficult to obtain.
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
This paper describes Tacotron 2, a neural network architecture for speech synthesis directly from text.
Ranked #2 on
Speech Synthesis
on North American English
State-of-the-art Speech Recognition With Sequence-to-Sequence Models
Attention-based encoder-decoder architectures such as Listen, Attend, and Spell (LAS), subsume the acoustic, pronunciation and language model components of a traditional automatic speech recognition (ASR) system into a single neural network.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Speech recognition for medical conversations
We explored both CTC and LAS systems for building speech recognition models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
An online sequence-to-sequence model for noisy speech recognition
This is because the models require that the entirety of the input sequence be available at the beginning of inference, an assumption that is not valid for instantaneous speech recognition.
Learning Hard Alignments with Variational Inference
There has recently been significant interest in hard attention models for tasks such as object recognition, visual captioning and speech recognition.
Discrete Sequential Prediction of Continuous Actions for Deep RL
Specifically, we show how Q-values and policies over continuous spaces can be modeled using a next step prediction model over discretized dimensions.
Tacotron: Towards End-to-End Speech Synthesis
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.
Ranked #5 on
Speech Synthesis
on North American English
Sequence-to-Sequence Models Can Directly Translate Foreign Speech
We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another.
Next-Step Conditioned Deep Convolutional Neural Networks Improve Protein Secondary Structure Prediction
This sequential model achieves 70. 3% Q8 accuracy on CB513 with a single model; an ensemble of these models produces 71. 4% Q8 accuracy on the same test set, improving upon the previous overall state of the art for the eight-class secondary structure problem.
Towards better decoding and language model integration in sequence to sequence models
The recently proposed Sequence-to-Sequence (seq2seq) framework advocates replacing complex data processing pipelines, such as an entire automatic speech recognition system, with a single neural network trained in an end-to-end fashion.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
An Online Sequence-to-Sequence Model Using Partial Conditioning
However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences.
Protein Secondary Structure Prediction Using Deep Multi-scale Convolutional Neural Networks and Next-Step Conditioning
We first train a series of deep neural networks to predict eight-class secondary structure labels given a protein's amino acid sequence information and find that using recent methods for regularization, such as dropout and weight-norm constraining, leads to measurable gains in accuracy.
Protein Secondary Structure Prediction
Protein Structure Prediction
RNN Approaches to Text Normalization: A Challenge
Though our conclusions are largely negative on this point, we are actually not arguing that the text normalization problem is intractable using an pure RNN approach, merely that it is not going to be something that can be solved merely by having huge amounts of annotated text data and feeding that to a general RNN model.
Very Deep Convolutional Networks for End-to-End Speech Recognition
Sequence-to-sequence models have shown success in end-to-end speech recognition.
Reward Augmented Maximum Likelihood for Neural Structured Prediction
A key problem in structured output prediction is direct optimization of the task reward function that matters for test evaluation.
Learning Online Alignments with Continuous Rewards Policy Gradient
Though capable and easy to use, they require that the entirety of the input sequence is available at the beginning of inference, an assumption that is not valid for instantaneous translation and speech recognition.
Chained Predictions Using Convolutional Neural Networks
In this model the output variables for a given input are predicted sequentially using neural networks.
Adversarial Autoencoders
In this paper, we propose the "adversarial autoencoder" (AAE), which is a probabilistic autoencoder that uses the recently proposed generative adversarial networks (GAN) to perform variational inference by matching the aggregated posterior of the hidden code vector of the autoencoder with an arbitrary prior distribution.
Ranked #7 on
Unsupervised MNIST
on MNIST
A Neural Transducer
However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences.
Object Recognition from Short Videos for Robotic Perception
Videos, unlike still images, are temporally coherent which makes the application of deep networks non-trivial.
Listen, Attend and Spell
Unlike traditional DNN-HMM models, this model learns all the components of a speech recognizer jointly.
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Recurrent Neural Networks can be trained to produce sequences of tokens given some input, as exemplified by recent results in machine translation and image captioning.
Pointer Networks
It differs from the previous attention attempts in that, instead of using attention to blend hidden units of an encoder to a context vector at each decoder step, it uses attention as a pointer to select a member of the input sequence as the output.
Ranked #9 on
Point Cloud Completion
on ShapeNet
(using extra training data)
A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
Learning long term dependencies in recurrent networks is difficult due to vanishing and exploding gradients.
Ranked #27 on
Sequential Image Classification
on Sequential MNIST
Occlusion Edge Detection in RGB-D Frames using Deep Convolutional Networks
Occlusion edges in images which correspond to range discontinuity in the scene from the point of view of the observer are an important prerequisite for many vision and mobile robot tasks.
Multi-task Neural Networks for QSAR Predictions
Although artificial neural networks have occasionally been used for Quantitative Structure-Activity/Property Relationship (QSAR/QSPR) studies in the past, the literature has of late been dominated by other machine learning techniques such as random forests.
Towards End-To-End Speech Recognition with Recurrent Neural Networks
A modification to the objective function is introduced that trains the network to minimise the expectation of an arbitrary transcription loss function.
Deep Neural Networks for Acoustic Modeling in Speech Recognition
Most current speech recognition systems use hidden Markov models (HMMs) to deal with the temporal variability of speech and Gaussian mixture models to determine how well each state of each HMM fits a frame or a short window of frames of coefficients that represents the acoustic input.
