Search Results for author:
Found 102 papers, 79 papers with code
NetFlowGen: Leveraging Generative Pre-training for Network Traffic Dynamics
Our presented NetFlowGen framework goes beyond a proof-of-concept for network traffic pre-training and addresses specific challenges such as unifying network feature representations, learning from large unlabeled traffic data volume, and testing on real downstream tasks in DDoS attack detection.
Compute-Constrained Data Selection
Data selection can reduce the amount of training data needed to finetune LLMs; however, the efficacy of data selection scales directly with its compute.
Contextual Document Embeddings
In this work, we argue that these embeddings, while effective, are implicitly out-of-context for targeted use cases of retrieval, and that a contextualized document embedding should take into account both the document and neighboring documents in context - analogous to contextualized word embeddings.
A Controlled Study on Long Context Extension and Generalization in LLMs
Broad textual understanding and in-context learning require language models that utilize full document contexts.
The Mamba in the Llama: Distilling and Accelerating Hybrid Models
The resulting hybrid model, which incorporates a quarter of the attention layers, achieves performance comparable to the original Transformer in chat benchmarks and outperforms open-source hybrid Mamba models trained from scratch with trillions of tokens in both chat benchmarks and general benchmarks.
I Could've Asked That: Reformulating Unanswerable Questions
We curate CouldAsk, an evaluation benchmark composed of existing and new datasets for document-grounded question answering, specifically designed to study reformulating unanswerable questions.
Entity Disambiguation via Fusion Entity Decoding
Existing generative approaches demonstrate improved accuracy compared to classification approaches under the standardized ZELDA benchmark.
Ranked #1 on
Entity Linking
on KORE50
(Micro-F1 strong metric)
MambaByte: Token-free Selective State Space Model
We propose MambaByte, a token-free adaptation of the Mamba SSM trained autoregressively on byte sequences.
Diffusion Models Without Attention
In recent advancements in high-fidelity image generation, Denoising Diffusion Probabilistic Models (DDPMs) have emerged as a key player.
Language Model Inversion
We consider the problem of language model inversion and show that next-token probabilities contain a surprising amount of information about the preceding text.
Predicting Text Preference Via Structured Comparative Reasoning
Comparative reasoning plays a crucial role in text preference prediction; however, large language models (LLMs) often demonstrate inconsistencies in their reasoning.
Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling
As the size of pre-trained speech recognition models increases, running these large models in low-latency or resource-constrained environments becomes challenging.
Symbolic Planning and Code Generation for Grounded Dialogue
Large language models (LLMs) excel at processing and generating both text and code.
Zephyr: Direct Distillation of LM Alignment
Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment.
Ranked #7 on
Zero-Shot Learning
on MedConceptsQA
Tree Prompting: Efficient Task Adaptation without Fine-Tuning
Prompting language models (LMs) is the main interface for applying them to new tasks.
Text Embeddings Reveal (Almost) As Much As Text
How much private information do text embeddings reveal about the original text?
Guess & Sketch: Language Model Guided Transpilation
In this work, we leverage the strengths of LMs and symbolic solvers in a neurosymbolic approach to learned transpilation for assembly code.
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
Large multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks.
Ranked #14 on
MMR total
on MRR-Benchmark
(using extra training data)
Scaling Data-Constrained Language Models
We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.
Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanations
Instead of using direct supervision, this work proposes an approach for abductive commonsense reasoning that exploits the fact that only a subset of explanations is correct for a given context.
HOP, UNION, GENERATE: Explainable Multi-hop Reasoning without Rationale Supervision
Explainable multi-hop question answering (QA) not only predicts answers but also identifies rationales, i. e. subsets of input sentences used to derive the answers.
Pretraining Without Attention
Even so, BiGS is able to match BERT pretraining accuracy on GLUE and can be extended to long-form pretraining of 4096 tokens without approximation.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Teal: Learning-Accelerated Optimization of WAN Traffic Engineering
The rapid expansion of global cloud wide-area networks (WANs) has posed a challenge for commercial optimization engines to efficiently solve network traffic engineering (TE) problems at scale.
ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition
To promote the development of multi-domain speech systems, we introduce the End-to-end Speech Benchmark (ESB) for evaluating the performance of a single automatic speech recognition (ASR) system across a broad set of speech datasets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Unsupervised Text Deidentification
We propose an unsupervised deidentification method that masks words that leak personally-identifying information.
Model Criticism for Long-Form Text Generation
Language models have demonstrated the ability to generate highly fluent text; however, it remains unclear whether their output retains coherent high-level structure (e. g., story progression).
Markup-to-Image Diffusion Models with Scheduled Sampling
These experiments each verify the effectiveness of the diffusion process and the use of scheduled sampling to fix generation issues.
Explaining Patterns in Data with Language Models via Interpretable Autoprompting
Large language models (LLMs) have displayed an impressive ability to harness natural language to perform complex tasks.
Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements
We introduce Evaluate and Evaluation on the Hub --a set of tools to facilitate the evaluation of models and datasets in ML.
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
State-of-the-art neural language models can now be used to solve ad-hoc language tasks through zero-shot prompting without the need for supervised training.
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
PromptSource is a system for creating, sharing, and using natural language prompts.
Low-Rank Constraints for Fast Inference in Structured Models
This work demonstrates a simple approach to reduce the computational and memory complexity of a large class of structured models.
GenNI: Human-AI Collaboration for Data-Backed Text Generation
These controls allow users to globally constrain model generations, without sacrificing the representation power of the deep learning models.
Multitask Prompted Training Enables Zero-Shot Task Generalization
Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020).
Rationales for Sequential Predictions
Compared to existing baselines, greedy rationalization is best at optimizing the combinatorial objective and provides the most faithful rationales.
Block Pruning For Faster Transformers
Pre-training has improved model accuracy for both classification and generation tasks at the cost of introducing much larger and slower models.
Datasets: A Community Library for Natural Language Processing
The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks.
Low-Complexity Probing via Finding Subnetworks
The dominant approach in probing neural networks for linguistic properties is to train a new shallow multi-layer perceptron (MLP) on top of the model's internal representations.
How Many Data Points is a Prompt Worth?
When fine-tuning pretrained models for classification, researchers either use a generic model head or a task-specific prompt for prediction.
Named Tensor Notation
We propose a notation for tensors with named axes, which relieves the author, reader, and future implementers of machine learning models from the burden of keeping track of the order of axes and the purpose of each.
Parameter-Efficient Transfer Learning with Diff Pruning
This approach views finetuning as learning a task-specific diff vector that is applied on top of the pretrained parameter vector, which remains fixed and is shared across different tasks.
Learning from others' mistakes: Avoiding dataset biases without modeling them
State-of-the-art natural language processing (NLP) models often learn to model dataset biases and surface form correlations instead of features that target the intended underlying task.
Latent Template Induction with Gumbel-CRFs
Learning to control the structure of sentences is a challenging problem in text generation.
EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference
Transformer-based language models such as BERT provide significant accuracy improvement for a multitude of natural language processing (NLP) tasks.
Sequence-Level Mixed Sample Data Augmentation
Despite their empirical success, neural networks still have difficulty capturing compositional aspects of natural language.
Scaling Hidden Markov Language Models
The hidden Markov model (HMM) is a fundamental tool for sequence modeling that cleanly separates the hidden state from the emission structure.
Adversarial Semantic Collisions
We study semantic collisions: texts that are semantically unrelated but judged as similar by NLP models.
Pre-trained Summarization Distillation
A third, simpler approach is to 'shrink and fine-tune' (SFT), which avoids any explicit distillation by copying parameters to a smaller student model and then fine-tuning.
GRIT: Generative Role-filler Transformers for Document-level Event Entity Extraction
We revisit the classic problem of document-level role-filler entity extraction (REE) for template filling.
Cascaded Text Generation with Markov Transformers
The two dominant approaches to neural text generation are fully autoregressive models, using serial beam search decoding, and non-autoregressive models, using parallel decoding with no output dependencies.
Movement Pruning: Adaptive Sparsity by Fine-Tuning
Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications.
Posterior Control of Blackbox Generation
In this work, we consider augmenting neural generation models with discrete control states learned through a structured latent-variable approach.
What is Learned in Visually Grounded Neural Syntax Acquisition
Visual features are a promising signal for learning bootstrap textual models.
Automating Botnet Detection with Graph Neural Networks
Botnets are now a major source for many network attacks, such as DDoS attacks and spam.
Torch-Struct: Deep Structured Prediction Library
The literature on structured prediction for NLP describes a rich collection of distributions and algorithms over sequences, segmentations, alignments, and trees; however, these algorithms are difficult to utilize in deep learning frameworks.
A Hierarchy of Graph Neural Networks Based on Learnable Local Features
We derive theoretical results about the discriminative power and feature representation capabilities of each class.
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks.
LAN -- A materials notation for 2D layered assemblies
Two-dimensional (2D) layered materials offer intriguing possibilities for novel physics and applications.
Materials Science
Neural Linguistic Steganography
Whereas traditional cryptography encrypts a secret message into an unintelligible form, steganography conceals that communication is taking place by encoding a secret message into a cover signal.
Commonsense Knowledge Mining from Pretrained Models
Inferring commonsense knowledge is a key challenge in natural language processing, but due to the sparsity of training data, previous work has shown that supervised methods for commonsense knowledge mining underperform when evaluated on novel data.
MASR: A Modular Accelerator for Sparse RNNs
The architecture is enhanced by a series of dynamic activation optimizations that enable compact storage, ensure no energy is wasted computing null operations, and maintain high MAC utilization for highly parallel accelerator designs.
Encoder-Agnostic Adaptation for Conditional Language Generation
Large pretrained language models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that adapt a pretrained model for arbitrary downstream tasks.
Simple Unsupervised Summarization by Contextual Matching
We propose an unsupervised method for sentence summarization using only language modeling.
Ranked #39 on
Text Summarization
on GigaWord
Visual Interaction with Deep Learning Models through Collaborative Semantic Inference
Automation of tasks can have critical consequences when humans lose agency over decision processes.
Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference
In contrast to standard approaches to NLI, our methods predict the probability of a premise given a hypothesis and NLI label, discouraging models from ignoring the premise.
On Adversarial Removal of Hypothesis-only Bias in Natural Language Inference
Popular Natural Language Inference (NLI) datasets have been shown to be tainted by hypothesis-only biases.
Compound Probabilistic Context-Free Grammars for Grammar Induction
We study a formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context-free grammar.
GLTR: Statistical Detection and Visualization of Generated Text
The rapid improvement of language models has raised the specter of abuse of text generation systems.
Unsupervised Recurrent Neural Network Grammars
On language modeling, unsupervised RNNGs perform as well their supervised counterparts on benchmarks in English and Chinese.
Ranked #11 on
Constituency Grammar Induction
on PTB Diagnostic ECG Database
(Max F1 (WSJ) metric)
Latent Normalizing Flows for Discrete Sequences
Normalizing flows are a powerful class of generative models for continuous random variables, showing both strong model flexibility and the potential for non-autoregressive generation.
A Tutorial on Deep Latent Variable Models of Natural Language
There has been much recent, exciting work on combining the complementary strengths of latent variable models and deep learning.
End-to-End Content and Plan Selection for Data-to-Text Generation
Learning to generate fluent natural language from structured data with neural networks has become an common approach for NLG.
Entity Tracking Improves Cloze-style Reading Comprehension
Reading comprehension tasks test the ability of models to process long-term context and remember salient information.
Bottom-Up Abstractive Summarization
We use this selector as a bottom-up attention step to constrain the model to likely phrases.
Ranked #4 on
Multi-Document Summarization
on Multi-News
Learning Neural Templates for Text Generation
While neural, encoder-decoder models have had significant empirical success in text generation, there remain several unaddressed problems with this style of generation.
Avoiding Latent Variable Collapse With Generative Skip Models
VAEs can capture complex distributions, but they can also suffer from an issue known as "latent variable collapse," especially if the likelihood model is powerful.
Latent Alignment and Variational Attention
This work considers variational attention networks, alternatives to soft and hard attention for learning latent variable alignment models, with tighter approximation bounds based on amortized variational inference.
Ranked #29 on
Machine Translation
on IWSLT2014 German-English
OpenNMT: Neural Machine Translation Toolkit
OpenNMT is an open-source toolkit for neural machine translation (NMT).
Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models
In this work, we present a visual analysis tool that allows interaction with a trained sequence-to-sequence model through each stage of the translation process.
Semi-Amortized Variational Autoencoders
Amortized variational inference (AVI) replaces instance-specific local inference with a global inference network.
Ranked #2 on
Text Generation
on Yahoo Questions
Weightless: Lossy Weight Encoding For Deep Neural Network Compression
This results in up to a 1. 51x improvement over the state-of-the-art.
Dilated Convolutions for Modeling Long-Distance Genomic Dependencies
We consider the task of detecting regulatory elements in the human genome directly from raw DNA.
OpenNMT: Open-source Toolkit for Neural Machine Translation
We introduce an open-source toolkit for neural machine translation (NMT) to support research into model architectures, feature representations, and source modalities, while maintaining competitive performance, modularity and reasonable training requirements.
Adapting Sequence Models for Sentence Correction
In a controlled experiment of sequence-to-sequence approaches for the task of sentence correction, we find that character-based models are generally more effective than word-based models and models that encode subword information via convolutions, and that modeling the output data as a series of diffs improves effectiveness over standard approaches.
Challenges in Data-to-Document Generation
Recent neural models have shown significant progress on the problem of generating short descriptive texts conditioned on a small number of database records.
Adversarially Regularized Autoencoders
This adversarially regularized autoencoder (ARAE) allows us to generate natural textual outputs as well as perform manipulations in the latent space to induce change in the output space.
Structured Attention Networks
Attention networks have proven to be an effective approach for embedding categorical inference within a deep neural network.
OpenNMT: Open-Source Toolkit for Neural Machine Translation
We describe an open-source toolkit for neural machine translation (NMT).
Lie-Access Neural Turing Machines
The head is moved via Lie group actions, such as shifts or rotations, generated by a controller, and memory access is performed by linear smoothing in key space.
Image-to-Markup Generation with Coarse-to-Fine Attention
We present a neural encoder-decoder model to convert images into presentational markup based on a scalable coarse-to-fine attention mechanism.
Sequence-Level Knowledge Distillation
We demonstrate that standard knowledge distillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to eliminate the need for beam search (even when applied on the original teacher model).
Ranked #1 on
Machine Translation
on IWSLT2015 Thai-English
LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
In this work, we present LSTMVIS, a visual analysis tool for recurrent neural networks with a focus on understanding these hidden state dynamics.
Sequence-to-Sequence Learning as Beam-Search Optimization
In this work, we introduce a model and beam-search training scheme, based on the work of Daume III and Marcu (2005), that extends seq2seq to learn global sequence scores.
Ranked #13 on
Machine Translation
on IWSLT2015 German-English
Word Ordering Without Syntax
Recent work on word ordering has argued that syntactic structure is important, or even required, for effectively recovering the order of a sentence.
Sentence-Level Grammatical Error Identification as Sequence-to-Sequence Correction
We demonstrate that an attention-based encoder-decoder model can be used for sentence-level grammatical error identification for the Automated Evaluation of Scientific Writing (AESW) Shared Task 2016.
Learning Global Features for Coreference Resolution
There is compelling evidence that coreference prediction would benefit from modeling global information about entity-clusters.
Ranked #26 on
Coreference Resolution
on OntoNotes
A Neural Attention Model for Abstractive Sentence Summarization
Summarization based on text extraction is inherently limited, but generation-style abstractive methods have proven challenging to build.
Ranked #1 on
Extractive Text Summarization
on DUC 2004 Task 1
Character-Aware Neural Language Models
We describe a simple neural language model that relies only on character-level inputs.
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
One long-term goal of machine learning research is to produce methods that are applicable to reasoning and natural language, in particular building an intelligent dialogue agent.
A Tutorial on Dual Decomposition and Lagrangian Relaxation for Inference in Natural Language Processing
Dual decomposition, and more generally Lagrangian relaxation, is a classical method for combinatorial optimization; it has recently been applied to several inference problems in natural language processing (NLP).
