Search Results for author:
Found 29 papers, 15 papers with code
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq Sˆ2, a fairseq extension for speech synthesis.
Video Editing via Factorized Diffusion Distillation
We introduce Emu Video Edit (EVE), a model that establishes a new state-of-the art in video editing without relying on any supervised video editing data.
Emu Edit: Precise Image Editing via Recognition and Generation Tasks
Lastly, to facilitate a more rigorous and informed assessment of instructable image editing models, we release a new challenging and versatile benchmark that includes seven different image editing tasks.
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning
It is also a general-purpose model that can do both text-to-image and image-to-text generation, allowing us to introduce self-contained contrastive decoding methods that produce high-quality outputs.
Ranked #2 on
Text-to-Image Generation
on MS COCO
Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users' preferences over generated images.
X&Fuse: Fusing Visual Information in Text-to-Image Generation
We introduce X&Fuse, a general approach for conditioning on visual information when generating images from text.
Text-To-4D Dynamic Scene Generation
We present MAV3D (Make-A-Video3D), a method for generating three-dimensional dynamic scenes from text descriptions.
AudioGen: Textually Guided Audio Generation
Finally, we explore the ability of the proposed method to generate audio continuation conditionally and unconditionally.
Ranked #13 on
Audio Generation
on AudioCaps
Make-A-Video: Text-to-Video Generation without Text-Video Data
We propose Make-A-Video -- an approach for directly translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V).
Ranked #3 on
Text-to-Video Generation
on MSR-VTT
(CLIP-FID metric)
KNN-Diffusion: Image Generation via Large-Scale Retrieval
Recent text-to-image models have achieved impressive results.
Ranked #34 on
Text-to-Image Generation
on MS COCO
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors
Recent text-to-image generation methods provide a simple yet exciting conversion capability between text and image domains.
Ranked #20 on
Text-to-Image Generation
on MS COCO
(using extra training data)
Locally Shifted Attention With Early Global Integration
The separation of the attention layer into local and global counterparts allows for a low computational cost in the number of patches, while still supporting data-dependent localization already at the first layer, as opposed to the static positioning in other visual transformers.
Textless Speech Emotion Conversion using Decomposed & Discrete Representations
We decompose speech into discrete and disentangled learned representations, consisting of content units, F0, speaker, and emotion.
Textless Speech Emotion Conversion using Discrete and Decomposed Representations
We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion.
Local-Global Shifting Vision Transformers
Due to the expensive quadratic cost of the attention mechanism, either a large patch size is used, resulting in coarse-grained global interactions, or alternatively, attention is applied only on a local region of the image at the expense of long-range interactions.
fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq S^2, a fairseq extension for speech synthesis.
Text-Free Prosody-Aware Generative Spoken Language Modeling
Generative Spoken Language Modeling (GSLM) \cite{Lakhotia2021} is the only prior work addressing the generative aspects of speech pre-training, which replaces text with discovered phone-like units for language modeling and shows the ability to generate meaningful novel sentences.
Ranked #6 on
Language Modelling
on SALMon
(using extra training data)
Direct speech-to-speech translation with discrete units
When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass.
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations
We propose using self-supervised discrete representations for the task of speech resynthesis.
Generative Spoken Language Modeling from Raw Audio
We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation.
Ranked #1 on
Resynthesis
on LibriSpeech
High Fidelity Speech Regeneration with Application to Speech Enhancement
Speech enhancement has seen great improvement in recent years mainly through contributions in denoising, speaker separation, and dereverberation methods that mostly deal with environmental effects on vocal audio.
Unsupervised Cross-Domain Singing Voice Conversion
We present a wav-to-wav generative model for the task of singing voice conversion from any identity.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Autoencoder-based Music Translation
We present a method for translating music across musical instruments and styles.
TTS Skins: Speaker Conversion via ASR
We present a fully convolutional wav-to-wav network for converting between speakers' voices, without relying on text.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
A Universal Music Translation Network
We present a method for translating music across musical instruments, genres, and styles.
Fitting New Speakers Based on a Short Untranscribed Sample
Learning-based Text To Speech systems have the potential to generalize from one speaker to the next and thus require a relatively short sample of any new voice.
VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop
We present a new neural text to speech (TTS) method that is able to transform text to speech in voices that are sampled in the wild.
Unsupervised Creation of Parameterized Avatars
We study the problem of mapping an input image to a tied pair consisting of a vector of parameters and an image that is created using a graphical engine from the vector of parameters.
Unsupervised Cross-Domain Image Generation
We study the problem of transferring a sample in one domain to an analog sample in another domain.
