Search Results for author:
Found 47 papers, 20 papers with code
Evaluating Gender Bias Transfer from Film Data
We extend our analysis to a longitudinal study of bias in film dialogue over the last 110 years and find that continued pre-training on OpenSubtitles encodes additional bias into BERT.
On the Benefit of Syntactic Supervision for Cross-lingual Transfer in Semantic Role Labeling
Although recent developments in neural architectures and pre-trained representations have greatly increased state-of-the-art model performance on fully-supervised semantic role labeling (SRL), the task remains challenging for languages where supervised SRL training data are not abundant.
Comparing Span Extraction Methods for Semantic Role Labeling
In this work, we empirically compare span extraction methods for the task of semantic role labeling (SRL).
Carbon Connect: An Ecosystem for Sustainable Computing
For embodied carbon, we must re-think conventional design strategies -- over-provisioned monolithic servers, frequent hardware refresh cycles, custom silicon -- and adopt life-cycle design strategies that more effectively reduce, reuse and recycle hardware at scale.
What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions
Large language models (LLMs) are trained on a vast amount of human-written data, but data providers often remain uncredited.
Source-Aware Training Enables Knowledge Attribution in Language Models
We investigate the problem of intrinsic source citation, where LLMs are required to cite the pretraining source supporting a generated response.
OLMo: Accelerating the Science of Language Models
Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs.
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations.
AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters
Large language models' (LLMs) abilities are drawn from their pretraining data, and model development begins with data curation.
Understanding the Effect of Model Compression on Social Bias in Large Language Models
Large Language Models (LLMs) trained with self-supervision on vast corpora of web text fit to the social biases of that text.
Power Hungry Processing: Watts Driving the Cost of AI Deployment?
Recent years have seen a surge in the popularity of commercial AI products based on generative, multi-purpose AI systems promising a unified approach to building machine learning (ML) models into technology.
Energy and Carbon Considerations of Fine-Tuning BERT
Despite the popularity of the `pre-train then fine-tune' paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on language model pre-training.
To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing
NLP is in a period of disruptive change that is impacting our methodologies, funding sources, and public perception.
Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation
In response, we introduce Pentathlon, a benchmark for holistic and realistic evaluation of model efficiency.
Queer People are People First: Deconstructing Sexual Identity Stereotypes in Large Language Models
Large Language Models (LLMs) are trained primarily on minimally processed web text, which exhibits the same wide range of social biases held by the humans who created that content.
Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research
Many recent improvements in NLP stem from the development and use of large pre-trained language models (PLMs) with billions of parameters.
Just CHOP: Embarrassingly Simple LLM Compression
Large language models (LLMs) enable unparalleled few- and zero-shot reasoning capabilities but at a high computational footprint.
Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training
To address this challenge, we adopt an error estimator to adaptively decide the partial selection ratio according to the current model's capability.
Regularizing Self-training for Unsupervised Domain Adaptation via Structural Constraints
Self-training based on pseudo-labels has emerged as a dominant approach for addressing conditional distribution shifts in unsupervised domain adaptation (UDA) for semantic segmentation problems.
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment
In this work, we examine this phenomenon through a series of case studies analyzing the effects of model design decisions, framework paradigms, and hardware platforms on total model latency.
To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering
Recent advances in open-domain question answering (ODQA) have demonstrated impressive accuracy on standard Wikipedia style benchmarks.
DSI++: Updating Transformer Memory with New Documents
In this work, we introduce DSI++, a continual learning challenge for DSI to incrementally index new documents while being able to answer queries related to both previously and newly indexed documents.
Error-aware Quantization through Noise Tempering
In this work, we incorporate exponentially decaying quantization-error-aware noise together with a learnable scale of task loss gradient to approximate the effect of a quantization operator.
Bridging Fairness and Environmental Sustainability in Natural Language Processing
Fairness and environmental impact are important research directions for the sustainable development of artificial intelligence.
A Survey of Active Learning for Natural Language Processing
In this work, we provide a survey of active learning (AL) for its applications in natural language processing (NLP).
Mention Annotations Alone Enable Efficient Domain Adaptation for Coreference Resolution
Although recent neural models for coreference resolution have led to substantial improvements on benchmark datasets, transferring these models to new target domains containing out-of-vocabulary spans and requiring differing annotation schemes remains challenging.
SQuAT: Sharpness- and Quantization-Aware Training for BERT
Quantization is an effective technique to reduce memory footprint, inference latency, and power consumption of deep learning models.
Efficient Methods for Natural Language Processing: A Survey
Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows.
Measuring the Carbon Intensity of AI in Cloud Instances
By providing unprecedented access to computational resources, cloud computing has enabled rapid growth in technologies such as machine learning, the computational demands of which incur a high energy cost and a commensurate carbon footprint.
Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models
Model compression by way of parameter pruning, quantization, or distillation has recently gained popularity as an approach for reducing the computational requirements of modern deep neural network models for NLP.
An Empirical Investigation of the Role of Pre-training in Lifelong Learning
The lifelong learning paradigm in machine learning is an attractive alternative to the more prominent isolated learning scheme not only due to its resemblance to biological learning but also its potential to reduce energy waste by obviating excessive model re-training.
Improving Compositional Generalization with Self-Training for Data-to-Text Generation
Data-to-text generation focuses on generating fluent natural language responses from structured meaning representations (MRs).
Unsupervised Domain Adaptation Via Pseudo-labels And Objectness Constraints
Crucially, the objectness constraint is agnostic to the ground-truth semantic segmentation labels and, therefore, remains appropriate for unsupervised adaptation settings.
End-to-end Quantized Training via Log-Barrier Extensions
Quantization of neural network parameters and activations has emerged as a successful approach to reducing the model size and inference time on hardware that sup-ports native low-precision arithmetic.
Energy and Policy Considerations for Deep Learning in NLP
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data.
The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structures
Materials science literature contains millions of materials synthesis procedures described in unstructured natural language text.
Inorganic Materials Synthesis Planning with Literature-Trained Neural Networks
Leveraging new data sources is a key step in accelerating the pace of materials design and discovery.
Syntax Helps ELMo Understand Semantics: Is Syntax Still Relevant in a Deep Neural Architecture for SRL?
Do unsupervised methods for learning rich, contextualized token representations obviate the need for explicit modeling of linguistic structure in neural network models for semantic role labeling (SRL)?
Linguistically-Informed Self-Attention for Semantic Role Labeling
Unlike previous models which require significant pre-processing to prepare linguistic features, LISA can incorporate syntax using merely raw tokens as input, encoding the sequence only once to simultaneously perform parsing, predicate detection and role labeling for all predicates.
Simultaneously Self-Attending to All Mentions for Full-Abstract Biological Relation Extraction
Most work in relation extraction forms a prediction by looking at a short span of text within a single sentence containing a single entity pair mention.
Automatically Extracting Action Graphs from Materials Science Synthesis Procedures
In this work, we present a system for automatically extracting structured representations of synthesis procedures from the texts of materials science journal articles that describe explicit, experimental syntheses of inorganic compounds.
Attending to All Mention Pairs for Full Abstract Biological Relation Extraction
We propose a model to consider all mention and entity pairs simultaneously in order to make a prediction.
Dependency Parsing with Dilated Iterated Graph CNNs
Dependency parses are an effective way to inject linguistic knowledge into many downstream tasks, and many practitioners wish to efficiently parse sentences at scale.
Fast and Accurate Entity Recognition with Iterated Dilated Convolutions
Today when many practitioners run basic NLP on the entire web and large-volume traffic, faster methods are paramount to saving time and energy costs.
Ranked #25 on
Named Entity Recognition (NER)
on Ontonotes v5 (English)
Multilingual Relation Extraction using Compositional Universal Schema
In response, this paper introduces significant further improvements to the coverage and flexibility of universal schema relation extraction: predictions for entities unseen in training and multilingual transfer learning to domains with no annotation.
Learning Dynamic Feature Selection for Fast Sequential Prediction
We present paired learning and inference algorithms for significantly reducing computation and increasing speed of the vector dot products in the classifiers that are at the heart of many NLP components.
Training for Fast Sequential Prediction Using Dynamic Feature Selection
We present paired learning and inference algorithms for significantly reducing computation and increasing speed of the vector dot products in the classifiers that are at the heart of many NLP components.
