Search Results for author:
Found 81 papers, 41 papers with code
Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare
In this work we adopt tools from mechanistic interpretability to unveil sociodemographic representations and biases within LLMs in the context of healthcare.
Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?
They might also propagate spin into their outputs: We find evidence, e. g., that LLMs implicitly incorporate spin into plain language summaries that they generate.
Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation
Measuring the occurrence of these often subtle ``hallucinations'' automatically has proved to be challenging.
NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals
We introduce NNsight and NDIF, technologies that work in tandem to enable scientific study of the representations and computations learned by very large neural networks.
Open (Clinical) LLMs are Sensitive to Instruction Phrasings
Instruction-tuned Large Language Models (LLMs) can perform a wide range of tasks given natural language instructions to do so, but they are sensitive to how such instructions are phrased.
Detection and Measurement of Syntactic Templates in Generated Text
Recent work on evaluating the diversity of text generated by LLMs has focused on word-level features.
Investigating Mysteries of CoT-Augmented Distillation
In fact, (3) a small number of key tokens are sufficient to achieve improvements equivalent to those observed when full rationales are used in model distillation.
Learning from Natural Language Explanations for Generalizable Entity Matching
Entity matching is the task of linking records from different sources that refer to the same real-world entity.
Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models
Using this dataset, we evaluate the performance of seven LLMs applied zero-shot for the task of conditionally extracting numerical findings from trial reports.
Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores
The applicability of scores extends beyond analysis of generative models; for example, we highlight applications on instruction-tuning datasets and human-produced texts.
How Much Annotation is Needed to Compare Summarization Models?
Modern instruction-tuned models have become highly capable in text generation tasks such as summarization, and are expected to be released at a steady pace.
Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study
With the advent of large language models (LLMs), there has been growing interest in exploring their potential for medical applications.
GenAudit: Fixing Factual Errors in Language Model Outputs with Evidence
GenAudit suggests edits to the LLM response by revising or removing claims that are not supported by the reference document, and also presents evidence from the reference for facts that do appear to have support.
Towards Reducing Diagnostic Errors with Interpretable Risk Prediction
In this work we propose a method to use LLMs to identify pieces of evidence in patient EHR data that indicate increased or decreased risk of specific diagnoses; our ultimate aim is to increase access to evidence and reduce diagnostic errors.
InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification
Text simplification aims to make technical texts more accessible to laypeople but often results in deletion of information and vagueness.
Leveraging Generative AI for Clinical Evidence Summarization Needs to Ensure Trustworthiness
Evidence-based medicine promises to improve the quality of healthcare by empowering medical decisions and practices with the best available evidence.
Future Lens: Anticipating Subsequent Tokens from a Single Hidden State
More concretely, in this paper we ask: Given a hidden (internal) representation of a single token at position $t$ in an input, can we reliably anticipate the tokens that will appear at positions $\geq t + 2$?
Function Vectors in Large Language Models
Using causal mediation analysis on a diverse range of in-context-learning (ICL) tasks, we find that a small number attention heads transport a compact representation of the demonstrated task, which we call a function vector (FV).
Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges
Our method entails tasking an LLM to infer whether a patient has, or is at risk of, a particular condition on the basis of associated notes; if so, we ask the model to summarize the supporting evidence.
Evaluating the Zero-shot Robustness of Instruction-tuned Language Models
To answer the former, we collect a set of 319 instructions manually written by NLP practitioners for over 80 unique tasks included in widely used benchmarks, and we evaluate the variance and average performance of these instructions as compared to instruction phrasings observed during instruction fine-tuning.
USB: A Unified Summarization Benchmark Across Tasks and Domains
While the NLP community has produced numerous summarization benchmarks, none provide the rich annotations required to simultaneously address many important problems related to control and reliability.
Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations
We analyze how automated summarization evaluation metrics correlate with lexical features of generated summaries, to other automated metrics including several we propose in this work, and to aspects of human-assessed summary quality.
Multilingual Simplification of Medical Texts
This work addresses this limitation via multilingual simplification, i. e., directly simplifying complex texts into simplified texts in multiple languages.
Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews
We conducted 16 interviews with international systematic review experts to characterize the perceived utility and risks of LLMs in the specific context of medical evidence reviews.
Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)
Large language models, particularly GPT-3, are able to produce high quality summaries of general domain news articles in few- and zero-shot settings.
Revisiting Relation Extraction in the era of Large Language Models
Relation extraction (RE) is the core NLP task of inferring semantic relationships between entities from text.
Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs
However, results from RCTs are presented in (often unstructured) natural language articles describing the design, execution, and outcomes of trials; clinicians must manually extract findings pertaining to interventions and outcomes of interest from such articles.
Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges
We present TrialsSummarizer, a system that aims to automatically summarize evidence presented in the set of randomized controlled trials most relevant to a given query.
CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models
We propose CHiLL (Crafting High-Level Latents), an approach for natural-language specification of features for linear models.
NapSS: Paragraph-level Medical Text Simplification via Narrative Prompting and Sentence-matching Summarization
In this work, we propose a summarize-then-simplify two-stage strategy, which we call NapSS, identifying the relevant content to simplify while ensuring that the original narrative flow is preserved.
How Many and Which Training Points Would Need to be Removed to Flip this Prediction?
We consider the problem of identifying a minimal subset of training data $\mathcal{S}_t$ such that if the instances comprising $\mathcal{S}_t$ had been removed prior to training, the categorization of a given test point $x_t$ would have been different.
Do Multi-Document Summarization Models Synthesize?
Multi-document summarization entails producing concise synopses of collections of inputs.
Intermediate Entity-based Sparse Interpretable Representation Learning
However, while fine-tuning sparse, interpretable representations improves accuracy on downstream tasks, it destroys the semantics of the dimensions which were enforced in pre-training.
Influence Functions for Sequence Tagging Models
We show the practical utility of segment influence by using the method to identify systematic annotation errors in two named entity recognition corpora.
PHEE: A Dataset for Pharmacovigilance Event Extraction from Text
The primary goal of drug safety researchers and regulators is to promptly identify adverse drug reactions.
Self-Repetition in Abstractive Neural Summarizers
In a regression analysis, we find that the three architectures have different propensities for repeating content across output summaries for inputs, with BART being particularly prone to self-repetition.
That's the Wrong Lung! Evaluating and Improving the Interpretability of Unsupervised Multimodal Encoders for Medical Data
We compare alignments from a state-of-the-art multimodal (image and text) model for EHR with human annotations that link image regions to sentences.
Learning to Ask Like a Physician
The questions are generated by medical experts from 100+ MIMIC-III discharge summaries.
Evaluating Factuality in Text Simplification
We find that errors often appear in both that are not captured by existing evaluation metrics, motivating a need for research into ensuring the factual accuracy of automated simplification models.
What Would it Take to get Biomedical QA Systems into Practice?
Medical question answering (QA) systems have the potential to answer clinicians uncertainties about treatment and diagnosis on demand, informed by the latest evidence.
Combining Feature and Instance Attribution to Detect Artifacts
In this paper we evaluate use of different attribution methods for aiding identification of training data artifacts.
Biomedical Interpretable Entity Representations
Pre-trained language models induce dense entity representations that offer strong performance on entity-centric NLP tasks, but such representations are not immediately interpretable.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
The cost of training such models (and the necessity of data access to do so) coupled with their utility motivates parameter sharing, i. e., the release of pretrained models such as ClinicalBERT.
Disentangling Representations of Text by Masking Transformers
Representations from large pretrained models such as BERT encode a range of features into monolithic vectors, affording strong predictive accuracy across a multitude of downstream tasks.
On the Impact of Random Seeds on the Fairness of Clinical Classifiers
Recent work has shown that fine-tuning large networks is surprisingly sensitive to changes in random seed(s).
Paragraph-level Simplification of Medical Texts
In this work we introduce a new corpus of parallel texts in English comprising technical and lay summaries of all published evidence pertaining to different clinical topics.
An Empirical Comparison of Instance Attribution Methods for NLP
Instance attribution methods constitute one means of accomplishing these goals by retrieving training instances that (may have) led to a particular prediction.
Unsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data
Unsupervised Data Augmentation (UDA) is a semi-supervised technique that applies a consistency loss to penalize differences between a model's predictions on (a) observed (unlabeled) examples; and (b) corresponding 'noised' examples produced via data augmentation.
Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations
Here we consider the end-to-end task of both (a) extracting treatments and outcomes from full-text articles describing clinical trials (entity identification) and, (b) inferring the reported results for the former with respect to the latter (relation extraction).
Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization
We enlist medical professionals to evaluate generated summaries, and we find that modern summarization systems yield consistently fluent and relevant synopses, but that they are not always factual.
Trialstreamer: Mapping and Browsing Medical Evidence in Real-Time
We apply the system at scale to all reports of randomized controlled trials indexed in MEDLINE, powering the automatic generation of evidence maps, which provide a global view of the efficacy of different interventions combining data from all relevant clinical trials on a topic.
Semi-Automating Knowledge Base Construction for Cancer Genetics
We propose and evaluate several model variants, including a transformer-based joint entity and relation extraction model to extract <germline mutation, risk-estimate>} pairs.
Joint Entity and Relation Extraction
Knowledge Base Construction
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
In this work, we investigate the use of influence functions for NLP, providing an alternative approach to interpreting neural text classifiers.
Evidence Inference 2.0: More Data, Better Models
Ideally, we could consult a database of evidence gleaned from clinical trials to answer such questions.
Learning to Faithfully Rationalize by Construction
In NLP this often entails extracting snippets of an input text `responsible for' corresponding model output; when such a snippet comprises tokens that indeed informed the model's prediction, it is a faithful explanation.
Interpretability Analysis for Named Entity Recognition to Understand System Predictions and How They Can Improve
We examine these questions by contrasting the performance of several variants of LSTM-CRF architectures for named entity recognition, with some provided only representations of the context as features.
Query-Focused EHR Summarization to Aid Imaging Diagnosis
We propose and evaluate models that extract relevant text snippets from patient records to provide a rough case summary intended to aid physicians considering one or more diagnoses.
Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Models
We propose a method for auditing the in-domain robustness of systems, focusing specifically on differences in performance due to the national origin of entities.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
We propose several metrics that aim to capture how well the rationales provided by models align with human rationales, and also how faithful these rationales are (i. e., the degree to which provided rationales influenced the corresponding predictions).
Learning to Identify Patients at Risk of Uncontrolled Hypertension Using Electronic Health Records Data
We develop machine learning models (logistic regression and recurrent neural networks) to stratify patients with respect to the risk of exhibiting uncontrolled hypertension within the coming three-month period.
Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction
Experiments on a complex biomedical information extraction task using expert and lay annotators show that: (i) simply excluding from the training data instances predicted to be difficult yields a small boost in performance; (ii) using difficulty scores to weight instances during training provides further, consistent gains; (iii) assigning instances predicted to be difficult to domain experts is an effective strategy for task routing.
An Analysis of Attention over Clinical Notes for Predictive Tasks
In this work we perform experiments to explore this question using two EMR corpora and four different predictive tasks, that: (i) inclusion of attention mechanisms is critical for neural encoder modules that operate over notes fields in order to yield competitive performance, but, (ii) unfortunately, while these boost predictive performance, it is decidedly less clear whether they provide meaningful support for predictions.
Inferring Which Medical Treatments Work from Reports of Clinical Trials
In this paper, we present a new task and corpus for making this unstructured evidence actionable.
Structured Neural Topic Models for Reviews
We present Variational Aspect-based Latent Topic Allocation (VALTA), a family of autoencoding topic models that learn aspect-based representations of reviews.
Structured Multi-Label Biomedical Text Tagging via Attentive Neural Tree Decoding
We propose a model for tagging unstructured texts with an arbitrary number of terms drawn from a tree-structured vocabulary (i. e., an ontology).
Practical Obstacles to Deploying Active Learning
Active learning (AL) is a widely-used training strategy for maximizing predictive performance subject to a fixed annotation budget.
A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature
We present a corpus of 5, 000 richly annotated abstracts of medical articles describing clinical randomized controlled trials.
Learning Disentangled Representations of Texts with Application to Biomedical Abstracts
We propose a method for learning disentangled representations of texts that code for distinct and complementary aspects, with the aim of affording efficient model transfer and interpretability.
Retrofitting Concept Vector Representations of Medical Concepts to Improve Estimates of Semantic Similarity and Relatedness
In this paper, we present a method that retrofits distributional context vector representations of biomedical concepts using structural information from the UMLS Metathesaurus, such that the similarity between vector representations of linked concepts is augmented.
Quantifying Mental Health from Social Media with Neural User Embeddings
Our experimental results demonstrate that the user embeddings capture similarities between users with respect to mental conditions, and are predictive of mental health.
Exploiting Domain Knowledge via Grouped Weight Sharing with Application to Text Categorization
A fundamental advantage of neural models for NLP is their ability to learn representations from scratch.
Neural Information Retrieval: A Literature Review
A recent "third wave" of Neural Network (NN) approaches now delivers state-of-the-art performance in many machine learning tasks, spanning speech recognition, computer vision, and natural language processing.
Modelling Context with User Embeddings for Sarcasm Detection in Social Media
We introduce a deep neural network for automated sarcasm detection.
Active Discriminative Text Representation Learning
We also show that, as expected, the method quickly learns discriminative word embeddings.
Rationale-Augmented Convolutional Neural Networks for Text Classification
We present a new Convolutional Neural Network (CNN) model for text classification that jointly exploits labels on documents and their component sentences.
