Search Results for author:
Found 102 papers, 58 papers with code
Mix and Match: Learning-free Controllable Text Generationusing Energy Language Models
Recent work on controlled text generation has either required attribute-based fine-tuning of the base language model (LM), or has restricted the parameterization of the attribute discriminator to be compatible with the base autoregressive LM.
Truth-Conditional Captions for Time Series Data
In this paper, we explore the task of automatically generating natural language descriptions of salient patterns in a time series, such as stock prices of a company over a week.
HOLM: Hallucinating Objects with Language Models for Referring Expression Recognition in Partially-Observed Scenes
AI systems embodied in the physical world face a fundamental challenge of partial observability; operating with only a limited view and knowledge of the environment.
MORL-Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimization
RL-based techniques can be used to search for prompts that when fed into a target language model maximize a set of user-specified reward functions.
DITTO: Diffusion Inference-Time T-Optimization for Music Generation
We propose Diffusion Inference-Time T-Optimization (DITTO), a general-purpose frame-work for controlling pre-trained text-to-music diffusion models at inference-time via optimizing initial noise latents.
A Block Metropolis-Hastings Sampler for Controllable Energy-based Text Generation
Recent work has shown that energy-based language modeling is an effective framework for controllable text generation because it enables flexible integration of arbitrary discriminators.
URL-BERT: Training Webpage Representations via Social Media Engagements
Common language model (LM) encoders such as BERT can be used to understand and represent the textual content of webpages.
Unsupervised Lead Sheet Generation via Semantic Compression
Lead sheets have become commonplace in generative music research, being used as an initial compressed representation for downstream tasks like multitrack music generation and automatic arrangement.
Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability
We also measure each architecture's predisposition towards in-context learning when presented with the option to memorize rather than leverage in-context examples.
Misusing Tools in Large Language Models With Visual Adversarial Examples
Large Language Models (LLMs) are being enhanced with the ability to use tools and to process multiple modalities.
Towards Improving Harmonic Sensitivity and Prediction Stability for Singing Melody Extraction
In deep learning research, many melody extraction models rely on redesigning neural network architectures to improve performance.
MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies
Diffusion models have shown promising results in cross-modal generation tasks, including text-to-image and text-to-audio generation.
CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models
Our results show the effectiveness of the proposed method, and that the pretrained diffusion prior can reduce the modality transfer gap.
Contrastive Attention Networks for Attribution of Early Modern Print
Specifically, we focus on matching uniquely damaged character type-imprints in anonymously printed books to works with known printers in order to provide evidence of their origins.
Membership Inference Attacks against Language Models via Neighbourhood Comparison
To investigate whether this fragility provides a layer of safety, we propose and evaluate neighbourhood attacks, which compare model scores for a given sample to scores of synthetically generated neighbour texts and therefore eliminate the need for access to the training data distribution.
Alt-Text with Context: Improving Accessibility for Images on Twitter
In this work we present an approach for generating alternative text (or alt-text) descriptions for images shared on social media, specifically Twitter.
Smaller Language Models are Better Black-box Machine-Generated Text Detectors
With the advent of fluent generative language models that can produce convincing utterances very similar to those written by humans, distinguishing whether a piece of text is machine-generated or human-written becomes more challenging and more important, as such models could be used to spread misinformation, fake news, fake reviews and to mimic certain authors and figures.
Towards Answering Climate Questionnaires from Unstructured Climate Reports
The topic of Climate Change (CC) has received limited attention in NLP despite its urgency.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Contrastive learning has been successfully used for retrieval of semantically aligned sentences, but it often requires large batch sizes or careful engineering to work well.
CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos
Further, videos in the wild often contain off-screen sounds and background noise that may hinder the model from learning the desired audio-textual correspondence.
Non-Parametric Temporal Adaptation for Social Media Topic Classification
User-generated social media data is constantly changing as new trends influence online discussion and personal information is deleted due to privacy concerns.
Checklist Models for Improved Output Fluency in Piano Fingering Prediction
In this work we present a new approach for the task of predicting fingerings for piano music.
Multitrack Music Transformer
Existing approaches for generating multitrack music with transformer models have been limited in terms of the number of instruments, the length of the music segments and slow inference.
Memorization in NLP Fine-tuning Methods
Large language models are shown to present privacy risks through memorization of training data, and several recent works have studied such risks for the pre-training phase.
Prompt Consistency for Zero-Shot Task Generalization
One of the most impressive results of recent NLP history is the ability of pre-trained language models to solve new tasks in a zero-shot setting.
Mix and Match: Learning-free Controllable Text Generation using Energy Language Models
Recent work on controlled text generation has either required attribute-based fine-tuning of the base language model (LM), or has restricted the parameterization of the attribute discriminator to be compatible with the base autoregressive LM.
Achieving Conversational Goals with Unsupervised Post-hoc Knowledge Injection
In this paper, we propose a post-hoc knowledge-injection technique where we first retrieve a diverse set of relevant knowledge snippets conditioned on both the dialog history and an initial response from an existing dialog model.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
The wide adoption and application of Masked language models~(MLMs) on sensitive data (from legal to medical) necessitates a thorough quantitative investigation into their privacy vulnerabilities -- to what extent do MLMs leak information about their training data?
Deep Performer: Score-to-Audio Music Performance Synthesis
Music performance synthesis aims to synthesize a musical score into a natural performance.
HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection
To combat these problems, we introduce HTS-AT: an audio transformer with a hierarchical structure to reduce the model size and training time.
Ranked #4 on
Sound Event Detection
on DESED
TONet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music
In this paper, we propose TONet, a plug-and-play model that improves both tone and octave perceptions by leveraging a novel input representation and a novel network architecture.
An Unsupervised Masking Objective for Abstractive Multi-Document News Summarization
We show that a simple unsupervised masking objective can approach near supervised performance on abstractive multi-document news summarization.
Masked Measurement Prediction: Learning to Jointly Predict Quantities and Units from Textual Context
Physical measurements constitute a large portion of numbers in academic papers, engineering reports, and web tables.
Lacuna Reconstruction: Self-supervised Pre-training for Low-Resource Historical Document Transcription
We present a self-supervised pre-training approach for learning rich visual language representations for both handwritten and printed historical document transcription.
Zero-shot Audio Source Separation through Query-based Learningfrom Weakly-labeled Data
Our approach uses a single model for source separation of multiple sound types, and relies solely on weakly-labeled data for training.
Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Data
Our approach uses a single model for source separation of multiple sound types, and relies solely on weakly-labeled data for training.
Ranked #1 on
Audio Source Separation
on AudioSet
Towards a Unified View of Parameter-Efficient Transfer Learning
Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks.
Truth-Conditional Captioning of Time Series Data
In this paper, we explore the task of automatically generating natural language descriptions of salient patterns in a time series, such as stock prices of a company over a week.
UserIdentifier: Implicit User Representations for Simple and Effective Personalized Sentiment Analysis
As such, these models are often unable to produce personalized responses for individual users, based on their data.
Investigating Robustness of Dialog Models to Popular Figurative Language Constructs
Humans often employ figurative language use in communication, including during interactions with dialog systems.
Scalable Font Reconstruction with Dual Latent Manifolds
We propose a deep generative model that performs typography analysis and font reconstruction by learning disentangled manifolds of both font style and character shape.
Style Pooling: Automatic Text Style Obfuscation for Improved Classification Fairness
Text style can reveal sensitive attributes of the author (e. g. race or age) to the reader, which can, in turn, lead to privacy violations and bias in both human and algorithmic decisions based on text.
Efficient Nearest Neighbor Language Models
Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore, which allows them to learn through explicitly memorizing the training datapoints.
An Empirical Evaluation of End-to-End Polyphonic Optical Music Recognition
Monophonic and homophonic music can be described as homorhythmic, or having a single musical rhythm.
Neural Representation Learning for Scribal Hands of Linear B
In this work, we present an investigation into the use of neural feature extraction in performing scribal hand analysis of the Linear B writing system.
Towards Automatic Instrumentation by Learning to Separate Parts in Symbolic Multitrack Music
In this paper, we aim to further extend this idea and examine the feasibility of automatic instrumentation -- dynamically assigning instruments to notes in solo music during performance.
Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transduction
Traditionally, character-level transduction problems have been solved with finite-state models designed to encode structural and linguistic knowledge of the underlying process, whereas recent approaches rely on the power and flexibility of sequence-to-sequence models with attention.
Unsupervised Enrichment of Persona-grounded Dialog with Background Stories
Humans often refer to personal narratives, life experiences, and events to make a conversation more engaging and rich.
Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentation
Multiple different responses are often plausible for a given open domain dialog context.
Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings
While recent work has shown that scores from models trained by the ubiquitous masked language modeling (MLM) objective effectively discriminate probable from improbable sequences, it is still an open question if these MLMs specify a principled probability distribution over the space of possible sequences.
Privacy Regularization: Joint Privacy-Utility Optimization in LanguageModels
In this work, we introduce two privacy-preserving regularization methods for training language models that enable joint optimization of utility and privacy through (1) the use of a discriminator and (2) the inclusion of a novel triplet-loss term.
Paraphrastic Representations at Scale
We train these models on large amounts of data, achieving significantly improved performance from the original papers proposing the methods on a suite of monolingual semantic similarity, cross-lingual semantic similarity, and bitext mining tasks.
An Empirical Study of Extrapolation in Text Generation with Scalar Control
We conduct an empirical evaluation of extrapolation performance when conditioning on scalar control inputs like desired output length, desired edit from an input sentence, and desired sentiment across three text generation tasks.
Privacy Regularization: Joint Privacy-Utility Optimization in Language Models
In this work, we introduce two privacy-preserving regularization methods for training language models that enable joint optimization of utility and privacy through (1) the use of a discriminator and (2) the inclusion of a triplet-loss term.
An Empirical Investigation of Contextualized Number Prediction
We conduct a large scale empirical investigation of contextualized number prediction in running text.
Like hiking? You probably enjoy nature: Persona-grounded Dialog with Commonsense Expansions
Existing persona-grounded dialog models often fail to capture simple implications of given persona descriptions, something which humans are able to do seamlessly.
Narrative Text Generation with a Latent Discrete Plan
Past work on story generation has demonstrated the usefulness of conditioning on a generation plan to generate coherent stories.
MusPy: A Toolkit for Symbolic Music Generation
MusPy provides easy-to-use tools for essential components in a music generation system, including dataset management, data I/O, data preprocessing and model evaluation.
Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythm
Drawing an analogy with automatic image completion systems, we propose Music SketchNet, a neural network framework that allows users to specify partial musical ideas guiding automatic music generation.
Refer360$^\circ$: A Referring Expression Recognition Dataset in 360$^\circ$ Images
We propose a novel large-scale referring expression recognition dataset, Refer360{\mbox{$^\circ$}}, consisting of 17, 137 instruction sequences and ground-truth actions for completing these instructions in 360{\mbox{$^\circ$}} scenes.
Learning Sparse Prototypes for Text Generation
While effective, these methods are inefficient at test time as a result of needing to store and index the entire training corpus.
Phonetic and Visual Priors for Decipherment of Informal Romanization
Informal romanization is an idiosyncratic process used by humans in informal digital communication to encode non-Latin script languages into Latin character sets found on common keyboards.
A Probabilistic Generative Model for Typographical Analysis of Early Modern Printing
We show that our approach outperforms rigid interpretable clustering baselines (Ocular) and overly-flexible deep generative models (VAE) alike on the task of completely unsupervised discovery of typefaces in mixed-font documents.
A Probabilistic Formulation of Unsupervised Text Style Transfer
Across all style transfer tasks, our approach yields substantial gains over state-of-the-art non-generative baselines, including the state-of-the-art unsupervised machine translation techniques that our approach generalizes.
Where New Words Are Born: Distributional Semantic Analysis of Neologisms and Their Semantic Neighborhoods
Besides presenting a new linguistic application of distributional semantics, this study tackles the linguistic question of the role of language-internal factors (in our case, sparsity) in language change motivated by language-external factors (reflected in frequency growth).
A Bilingual Generative Transformer for Semantic Sentence Embedding
Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences.
A Deep Factorization of Style and Structure in Fonts
We propose a deep factorization model for typographic analysis that disentangles content from style.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We present a model and methodology for learning paraphrastic sentence embeddings directly from bitext, removing the time-consuming intermediate step of creating paraphrase corpora.
Learning Rhyming Constraints using Structured Adversaries
Existing recurrent neural language models often fail to capture higher-level structure present in text: for example, rhyming patterns present in poetry.
Beyond BLEU: Training Neural Machine Translation with Semantic Similarity
While most neural machine translation (NMT) systems are still trained using maximum likelihood estimation, recent work has demonstrated that optimizing systems to directly improve evaluation metrics such as BLEU can substantially improve final translation accuracy.
A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text
In this paper, we investigate a simple fix for posterior collapse which yields surprisingly effective results.
Beyond BLEU:Training Neural Machine Translation with Semantic Similarity
While most neural machine translation (NMT)systems are still trained using maximum likelihood estimation, recent work has demonstrated that optimizing systems to directly improve evaluation metrics such as BLEU can significantly improve final translation accuracy.
Cross-Lingual Syntactic Transfer through Unsupervised Adaptation of Invertible Projections
The parameters of source model and target model are softly shared through a regularized log likelihood objective.
An Empirical Investigation of Global and Local Normalization for Recurrent Neural Sequence Models Using a Continuous Relaxation to Beam Search
Globally normalized neural sequence models are considered superior to their locally normalized equivalents because they may ameliorate the effects of label bias.
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
The variational autoencoder (VAE) is a popular combination of deep latent variable model and accompanying variational learning technique.
Ranked #1 on
Text Generation
on Yahoo Questions
Modeling Online Discourse with Coupled Distributed Topics
In this paper, we propose a deep, globally normalized topic model that incorporates structural relationships connecting documents in socially generated corpora, such as online forums.
Learning to Describe Differences Between Pairs of Similar Images
We propose a model that captures visual salience by using a latent variable to align clusters of differing pixels with output sentences.
Unsupervised Learning of Syntactic Structure with Invertible Neural Projections
In this work, we propose a novel generative model that jointly learns discrete syntactic structure and continuous word representations in an unsupervised fashion by cascading an invertible neural network with a structured generative prior.
Learning to Generate Move-by-Move Commentary for Chess Games from Large-Scale Social Forum Data
This paper examines the problem of generating natural language descriptions of chess games.
Speaker-Follower Models for Vision-and-Language Navigation
We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction.
Unsupervised Text Style Transfer using Language Models as Discriminators
Binary classifiers are often employed as discriminators in GAN-based unsupervised style transfer systems to ensure that transferred sentences are similar to sentences in the target domain.
Visual Referring Expression Recognition: What Do Systems Actually Learn?
We present an empirical analysis of the state-of-the-art systems for referring expression recognition -- the task of identifying the object in an image referred to by a natural language expression -- with the goal of gaining insight into how these systems reason about language and vision.
Using Syntax to Ground Referring Expressions in Natural Images
We introduce GroundNet, a neural network for referring expression recognition -- the task of localizing (or grounding) in an image the object referred to by a natural language expression.
SPINE: SParse Interpretable Neural Embeddings
We propose a novel variant of denoising k-sparse autoencoders that generates highly efficient and interpretable distributed word representations (word embeddings), beginning with existing word representations from state-of-the-art methods like GloVe and word2vec.
Identifying Products in Online Cybercrime Marketplaces: A Dataset for Fine-grained Domain Adaptation
One weakness of machine-learned NLP models is that they typically perform poorly on out-of-domain data.
A Continuous Relaxation of Beam Search for End-to-end Training of Neural Sequence Models
In experiments, we show that optimizing this new training objective yields substantially better results on two sequence tasks (Named Entity Recognition and CCG Supertagging) when compared with both cross entropy trained greedy decoding and cross entropy trained beam decoding baselines.
Ranked #3 on
Motion Segmentation
on Hopkins155
Efficient Correlated Topic Modeling with Topic Embedding
Correlated topic modeling has been limited to small model and problem sizes due to their high computational cost and poor scaling.
Automatic Compositor Attribution in the First Folio of Shakespeare
Compositor attribution, the clustering of pages in a historical printed document by the individual who set the type, is a bibliographic task that relies on analysis of orthographic variation and inspection of visual details of the printed page.
Differentiable Scheduled Sampling for Credit Assignment
We demonstrate that a continuous relaxation of the argmax operation can be used to create a differentiable approximation to greedy decoding for sequence-to-sequence (seq2seq) models.
Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
Recent work on generative modeling of text has found that variational auto-encoders (VAE) incorporating LSTM decoders perform worse than simpler LSTM language models (Bowman et al., 2015).
Ranked #3 on
Text Generation
on Yahoo Questions
Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints
We present a discriminative model for single-document summarization that integrally combines compression and anaphoricity constraints.
Unsupervised Transcription of Piano Music
We present a new probabilistic model for transcribing piano music from audio to a symbolic form.
