Search Results for author: Zachary C. Lipton

Found 133 papers, 60 papers with code

LLM-Select: Feature Selection with Large Language Models

no code implementations2 Jul 2024 Daniel P. Jeong, Zachary C. Lipton, Pradeep Ravikumar

In particular, we find that the latest models, such as GPT-4, can consistently identify the most predictive features regardless of the query mechanism and across various prompting strategies.

feature selection

Understanding Hallucinations in Diffusion Models through Mode Interpolation

1 code implementation13 Jun 2024 Sumukh K Aithal, Pratyush Maini, Zachary C. Lipton, J. Zico Kolter

Specifically, we find that diffusion models smoothly "interpolate" between nearby data modes in the training set, to generate samples that are completely outside the support of the original training distribution; this phenomenon leads diffusion models to generate artifacts that never existed in real data (i. e., hallucinations).

Hallucination Image Generation

Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends

no code implementations5 Jun 2024 Sanjana Ramprasad, Elisa Ferracane, Zachary C. Lipton

Our work benchmarks the faithfulness of LLMs for dialogue summarization, using human annotations and focusing on identifying and categorizing span-level inconsistencies.

Hallucination

Rethinking LLM Memorization through the Lens of Adversarial Compression

no code implementations23 Apr 2024 Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C. Lipton, J. Zico Kolter

The ACR overcomes the limitations of existing notions of memorization by (i) offering an adversarial view of measuring memorization, especially for monitoring unlearning and compliance; and (ii) allowing for the flexibility to measure memorization for arbitrary strings at a reasonably low compute.

Memorization

Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic

1 code implementation10 Apr 2024 Sachin Goyal, Pratyush Maini, Zachary C. Lipton, aditi raghunathan, J. Zico Kolter

Vision-language models (VLMs) are trained for thousands of GPU hours on carefully curated web datasets.

Auditing Fairness under Unobserved Confounding

1 code implementation18 Mar 2024 Yewon Byun, Dylan Sam, Michael Oberst, Zachary C. Lipton, Bryan Wilder

The presence of inequity is a fundamental problem in the outcomes of decision-making systems, especially when human lives are at stake.

Decision Making Fairness

Contrastive Multiple Instance Learning for Weakly Supervised Person ReID

no code implementations12 Feb 2024 Jacob Tyo, Zachary C. Lipton

Through extensive experiments and analysis across three datasets, CMIL not only matches state-of-the-art performance on the large-scale SYSU-30k dataset with fewer assumptions but also consistently outperforms all baselines on the WL-market1501 and Weakly Labeled MUddy racer re-iDentification dataset (WL-MUDD) datasets.

Multiple Instance Learning Person Re-Identification

Beyond the Mud: Datasets and Benchmarks for Computer Vision in Off-Road Racing

no code implementations12 Feb 2024 Jacob Tyo, Motolani Olarinre, Youngseog Chung, Zachary C. Lipton

With these datasets and analysis of model limitations, we aim to foster innovations in handling real-world conditions like mud and complex poses to drive progress in robust computer vision.

Optical Character Recognition Optical Character Recognition (OCR) +2

Personalized Language Modeling from Personalized Human Feedback

no code implementations6 Feb 2024 Xinyu Li, Zachary C. Lipton, Liu Leqi

We then propose a general Personalized-RLHF (P-RLHF) framework, including a user model that maps user information to user representations and can flexibly encode our assumptions on user preferences.

Instruction Following Language Modelling +1

Red-Teaming for Generative AI: Silver Bullet or Security Theater?

no code implementations29 Jan 2024 Michael Feffer, Anusha Sinha, Wesley Hanwen Deng, Zachary C. Lipton, Hoda Heidari

In response to rising concerns surrounding the safety, security, and trustworthiness of Generative AI (GenAI) models, practitioners and regulators alike have pointed to AI red-teaming as a key component of their strategies for identifying and mitigating these risks.

The Impact of Differential Feature Under-reporting on Algorithmic Fairness

no code implementations16 Jan 2024 Nil-Jana Akpinar, Zachary C. Lipton, Alexandra Chouldechova

Predictive risk models in the public sector are commonly developed using administrative data that is more complete for subpopulations that more greatly rely on public services.

Decision Making Fairness

TOFU: A Task of Fictitious Unlearning for LLMs

2 code implementations11 Jan 2024 Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, J. Zico Kolter

Large language models trained on massive corpora of data from the web can memorize and reproduce sensitive or private data raising both legal and ethical concerns.

Scaling Laws for Data Filtering-- Data Curation cannot be Compute Agnostic

no code implementations CVPR 2024 Sachin Goyal, Pratyush Maini, Zachary C. Lipton, aditi raghunathan, J. Zico Kolter

Our work bridges this important gap in the literature by developing scaling laws that characterize the differing utility of various data subsets and accounting for how this diminishes for a data point at its nth repetition.

Deep Equilibrium Based Neural Operators for Steady-State PDEs

no code implementations NeurIPS 2023 Tanya Marwah, Ashwini Pokle, J. Zico Kolter, Zachary C. Lipton, Jianfeng Lu, Andrej Risteski

Motivated by this observation, we propose FNO-DEQ, a deep equilibrium variant of the FNO architecture that directly solves for the solution of a steady-state PDE as the infinite-depth fixed point of an implicit operator layer using a black-box root solver and differentiates analytically through this fixed point resulting in $\mathcal{O}(1)$ training memory.

MoCo-Transfer: Investigating out-of-distribution contrastive learning for limited-data domains

no code implementations15 Nov 2023 YuWen Chen, Helen Zhou, Zachary C. Lipton

Medical imaging data is often siloed within hospitals, limiting the amount of data available for specialized model development.

Contrastive Learning

Goodhart's Law Applies to NLP's Explanation Benchmarks

no code implementations28 Aug 2023 Jennifer Hsia, Danish Pruthi, Aarti Singh, Zachary C. Lipton

First, we show that we can inflate a model's comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution test inputs.

Can Neural Network Memorization Be Localized?

1 code implementation18 Jul 2023 Pratyush Maini, Michael C. Mozer, Hanie Sedghi, Zachary C. Lipton, J. Zico Kolter, Chiyuan Zhang

Recent efforts at explaining the interplay of memorization and generalization in deep overparametrized networks have posited that neural networks $\textit{memorize}$ "hard" examples in the final few layers of the model.

Memorization

T-MARS: Improving Visual Representations by Circumventing Text Feature Learning

1 code implementation6 Jul 2023 Pratyush Maini, Sachin Goyal, Zachary C. Lipton, J. Zico Kolter, aditi raghunathan

However, naively removing all such data could also be wasteful, as it throws away images that contain visual features (in addition to overlapping text).

Optical Character Recognition

Moral Machine or Tyranny of the Majority?

no code implementations27 May 2023 Michael Feffer, Hoda Heidari, Zachary C. Lipton

With Artificial Intelligence systems increasingly applied in consequential domains, researchers have begun to ask how these systems ought to act in ethically charged situations where even humans lack consensus.

Autonomous Vehicles Fairness

PromptNER: Prompting For Named Entity Recognition

no code implementations24 May 2023 Dhananjay Ashok, Zachary C. Lipton

In a surprising turn, Large Language Models (LLMs) together with a growing arsenal of prompt-based heuristics now offer powerful off-the-shelf approaches providing few-shot solutions to myriad classic NLP problems.

Ranked #4 on Zero-shot Named Entity Recognition (NER) on CrossNER (using extra training data)

few-shot-ner Few-shot NER +5

USB: A Unified Summarization Benchmark Across Tasks and Domains

1 code implementation23 May 2023 Kundan Krishna, Prakhar Gupta, Sanjana Ramprasad, Byron C. Wallace, Jeffrey P. Bigham, Zachary C. Lipton

While the NLP community has produced numerous summarization benchmarks, none provide the rich annotations required to simultaneously address many important problems related to control and reliability.

Abstractive Text Summarization Extractive Summarization +1

Evaluating Model Performance in Medical Datasets Over Time

1 code implementation22 May 2023 Helen Zhou, YuWen Chen, Zachary C. Lipton

Machine learning (ML) models deployed in healthcare systems must face data drawn from continually evolving environments.

Risk-limiting Financial Audits via Weighted Sampling without Replacement

no code implementations8 May 2023 Shubhanshu Shekhar, Ziyu Xu, Zachary C. Lipton, Pierre J. Liang, Aaditya Ramdas

Next, we develop methods to improve the quality of CSs by incorporating side information about the unknown values associated with each item.

Model-tuning Via Prompts Makes NLP Models Adversarially Robust

1 code implementation13 Mar 2023 Mrigank Raman, Pratyush Maini, J. Zico Kolter, Zachary C. Lipton, Danish Pruthi

Across 5 NLP datasets, 4 adversarial attacks, and 3 different models, MVP improves performance against adversarial substitutions by an average of 8% over standard methods and even outperforms adversarial training-based state-of-art defenses by 3. 5%.

Adversarial Robustness Language Modelling +1

Users are the North Star for AI Transparency

no code implementations9 Mar 2023 Alex Mei, Michael Saxon, Shiyu Chang, Zachary C. Lipton, William Yang Wang

We conduct a broad literature survey, identifying many clusters of similar conceptions of transparency, tying each back to our north star with analysis of how it furthers or hinders our ideal AI transparency goals.

Local Causal Discovery for Estimating Causal Effects

1 code implementation16 Feb 2023 Shantanu Gupta, David Childers, Zachary C. Lipton

Even when the causal graph underlying our data is unknown, we can use observational data to narrow down the possible values that an average treatment effect (ATE) can take by (1) identifying the graph up to a Markov equivalence class; and (2) estimating that ATE for each graph in the class.

Causal Discovery Computational Efficiency

RLSbench: Domain Adaptation Under Relaxed Label Shift

1 code implementation6 Feb 2023 Saurabh Garg, Nick Erickson, James Sharpnack, Alex Smola, Sivaraman Balakrishnan, Zachary C. Lipton

Despite the emergence of principled methods for domain adaptation under label shift, their sensitivity to shifts in class conditional distributions is precariously under explored.

Domain Adaptation

Meta-Learning Mini-Batch Risk Functionals

no code implementations27 Jan 2023 Jacob Tyo, Zachary C. Lipton

Supervised learning typically optimizes the expected value risk functional of the loss, but in many cases, we want to optimize for other risk functionals.

Meta-Learning

Disentangling the Mechanisms Behind Implicit Regularization in SGD

1 code implementation29 Nov 2022 Zachary Novack, Simran Kaur, Tanya Marwah, Saurabh Garg, Zachary C. Lipton

A number of competing hypotheses have been proposed to explain why small-batch Stochastic Gradient Descent (SGD)leads to improved generalization over the full-batch regime, with recent work crediting the implicit regularization of various quantities throughout training.

Model Evaluation in Medical Datasets Over Time

no code implementations14 Nov 2022 Helen Zhou, YuWen Chen, Zachary C. Lipton

Machine learning models deployed in healthcare systems face data drawn from continually evolving environments.

Domain Adaptation under Missingness Shift

1 code implementation3 Nov 2022 Helen Zhou, Sivaraman Balakrishnan, Zachary C. Lipton

Rates of missing data often depend on record-keeping policies and thus may change across times and locations, even when the underlying features are comparatively stable.

Domain Adaptation

Characterizing Datapoints via Second-Split Forgetting

1 code implementation26 Oct 2022 Pratyush Maini, Saurabh Garg, Zachary C. Lipton, J. Zico Kolter

Popular metrics derived from these dynamics include (i) the epoch at which examples are first correctly classified; (ii) the number of times their predictions flip during training; and (iii) whether their prediction flips if they are held out.

Neural Network Approximations of PDEs Beyond Linearity: A Representational Perspective

no code implementations21 Oct 2022 Tanya Marwah, Zachary C. Lipton, Jianfeng Lu, Andrej Risteski

We show that if composing a function with Barron norm $b$ with partial derivatives of $L$ produces a function of Barron norm at most $B_L b^p$, the solution to the PDE can be $\epsilon$-approximated in the $L^2$ sense by a function with Barron norm $O\left(\left(dB_L\right)^{\max\{p \log(1/ \epsilon), p^{\log(1/\epsilon)}\}}\right)$.

Downstream Datasets Make Surprisingly Good Pretraining Corpora

1 code implementation28 Sep 2022 Kundan Krishna, Saurabh Garg, Jeffrey P. Bigham, Zachary C. Lipton

In experiments addressing both ELECTRA and RoBERTa models and 10 distinct downstream classification datasets, we observe that self-pretraining rivals standard pretraining on the BookWiki corpus (despite using around $10\times$--$500\times$ less data), outperforming the latter on $7$ and $5$ datasets, respectively.

Question Answering

On the State of the Art in Authorship Attribution and Authorship Verification

1 code implementation14 Sep 2022 Jacob Tyo, Bhuwan Dhingra, Zachary C. Lipton

Despite decades of research on authorship attribution (AA) and authorship verification (AV), inconsistent dataset splits/filtering and mismatched evaluation methods make it difficult to assess the state of the art.

Authorship Attribution Authorship Verification

Unsupervised Learning under Latent Label Shift

2 code implementations26 Jul 2022 Manley Roberts, Pranav Mani, Saurabh Garg, Zachary C. Lipton

Thus motivated, we introduce a practical algorithm that leverages domain-discriminative models as follows: (i) push examples through domain discriminator $p(d|\mathbf{x})$; (ii) discretize the data by clustering examples in $p(d|\mathbf{x})$ space; (iii) perform non-negative matrix factorization on the discrete data; (iv) combine the recovered $p(y|d)$ with the discriminator outputs $p(d|\mathbf{x})$ to compute $p_d(y|x) \; \forall d$.

Domain Adaptation under Open Set Label Shift

1 code implementation26 Jul 2022 Saurabh Garg, Sivaraman Balakrishnan, Zachary C. Lipton

We introduce the problem of domain adaptation under Open Set Label Shift (OSLS) where the label distribution can change arbitrarily and a new class may arrive during deployment, but the class-conditional distributions p(x|y) are domain-invariant.

Domain Adaptation

Supervised Learning with General Risk Functionals

no code implementations27 Jun 2022 Liu Leqi, Audrey Huang, Zachary C. Lipton, Kamyar Azizzadenesheli

Standard uniform convergence results bound the generalization gap of the expected loss over a hypothesis class.

On the Maximum Hessian Eigenvalue and Generalization

no code implementations21 Jun 2022 Simran Kaur, Jeremy Cohen, Zachary C. Lipton

The mechanisms by which certain training interventions, such as increasing learning rates and applying batch normalization, improve the generalization of deep networks remains a mystery.

Resolving the Human Subjects Status of Machine Learning's Crowdworkers

no code implementations8 Jun 2022 Divyansh Kaushik, Zachary C. Lipton, Alex John London

We highlight two challenges posed by ML: the same set of workers can serve multiple roles and provide many sorts of information; and ML research tends to embrace a dynamic workflow, where research questions are seldom stated ex ante and data sharing opens the door for future studies to aim questions at different targets.

Ethics

Modeling Attrition in Recommender Systems with Departing Bandits

no code implementations25 Mar 2022 Omer Ben-Porat, Lee Cohen, Liu Leqi, Zachary C. Lipton, Yishay Mansour

We first address the case where all users share the same type, demonstrating that a recent UCB-based algorithm is optimal.

Multi-Armed Bandits Recommendation Systems

Exploring Transformer Backbones for Heterogeneous Treatment Effect Estimation

1 code implementation2 Feb 2022 Yi-Fan Zhang, HANLIN ZHANG, Zachary C. Lipton, Li Erran Li, Eric P. Xing

Previous works on Treatment Effect Estimation (TEE) are not in widespread use because they are predominantly theoretical, where strong parametric assumptions are made but untractable for practical application.

POS Selection bias

Leveraging Unlabeled Data to Predict Out-of-Distribution Performance

1 code implementation ICLR 2022 Saurabh Garg, Sivaraman Balakrishnan, Zachary C. Lipton, Behnam Neyshabur, Hanie Sedghi

Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions that may cause performance drops.

Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations

1 code implementation17 Dec 2021 Siddhant Arora, Danish Pruthi, Norman Sadeh, William W. Cohen, Zachary C. Lipton, Graham Neubig

Through our evaluation, we observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.

Deception Detection

Mixture Proportion Estimation and PU Learning: A Modern Approach

2 code implementations NeurIPS 2021 Saurabh Garg, Yifan Wu, Alex Smola, Sivaraman Balakrishnan, Zachary C. Lipton

Formally, this task is broken down into two subtasks: (i) Mixture Proportion Estimation (MPE) -- determining the fraction of positive examples in the unlabeled data; and (ii) PU-learning -- given such an estimate, learning the desired positive-versus-negative classifier.

Practical Benefits of Feature Feedback Under Distribution Shift

no code implementations14 Oct 2021 Anurag Katakkar, Clay H. Yoo, Weiqin Wang, Zachary C. Lipton, Divyansh Kaushik

In attempts to develop sample-efficient and interpretable algorithms, researcher have explored myriad mechanisms for collecting and exploiting feature feedback (or rationales) auxiliary annotations provided for training (but not test) instances that highlight salient evidence.

Natural Language Inference Sentiment Analysis

Efficient Online Estimation of Causal Effects by Deciding What to Observe

1 code implementation NeurIPS 2021 Shantanu Gupta, Zachary C. Lipton, David Childers

Researchers often face data fusion problems, where multiple data sources are available, each capturing a distinct subset of variables.

Dive into Deep Learning

1 code implementation21 Jun 2021 Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola

This open-source book represents our attempt to make deep learning approachable, teaching readers the concepts, the context, and the code.

Math Multi-Domain Recommender Systems

Correcting Exposure Bias for Link Recommendation

1 code implementation13 Jun 2021 Shantanu Gupta, Hao Wang, Zachary C. Lipton, Yuyang Wang

Link prediction methods are frequently applied in recommender systems, e. g., to suggest citations for academic papers or friends in social networks.

Diversity Link Prediction +1

On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study

1 code implementation ACL 2021 Divyansh Kaushik, Douwe Kiela, Zachary C. Lipton, Wen-tau Yih

In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions.

Question Answering

RATT: Leveraging Unlabeled Data to Guarantee Generalization

1 code implementation1 May 2021 Saurabh Garg, Sivaraman Balakrishnan, J. Zico Kolter, Zachary C. Lipton

To assess generalization, machine learning scientists typically either (i) bound the generalization gap and then (after training) plug in the empirical risk to obtain a bound on the true risk; or (ii) validate empirically on holdout data.

Generalization Bounds Holdout Set +1

On the Convergence and Optimality of Policy Gradient for Markov Coherent Risk

no code implementations4 Mar 2021 Audrey Huang, Liu Leqi, Zachary C. Lipton, Kamyar Azizzadenesheli

Because optimizing the coherent risk is difficult in Markov decision processes, recent work tends to focus on the Markov coherent risk (MCR), a time-consistent surrogate.

Parametric Complexity Bounds for Approximating PDEs with Neural Networks

no code implementations NeurIPS 2021 Tanya Marwah, Zachary C. Lipton, Andrej Risteski

Recent experiments have shown that deep networks can approximate solutions to high-dimensional PDEs, seemingly escaping the curse of dimensionality.

On Proximal Policy Optimization's Heavy-tailed Gradients

no code implementations20 Feb 2021 Saurabh Garg, Joshua Zhanson, Emilio Parisotto, Adarsh Prasad, J. Zico Kolter, Zachary C. Lipton, Sivaraman Balakrishnan, Ruslan Salakhutdinov, Pradeep Ravikumar

In this paper, we present a detailed empirical study to characterize the heavy-tailed nature of the gradients of the PPO surrogate reward function.

Continuous Control

Evaluating Explanations: How much do explanations from the teacher aid students?

1 code implementation1 Dec 2020 Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, William W. Cohen

While many methods purport to explain predictions by highlighting salient features, what aims these explanations serve and how they ought to be evaluated often go unstated.

Question Answering text-classification +1

Decoding and Diversity in Machine Translation

no code implementations26 Nov 2020 Nicholas Roberts, Davis Liang, Graham Neubig, Zachary C. Lipton

This makes human-level BLEU a misleading benchmark in that modern MT systems cannot approach human-level BLEU while simultaneously maintaining human-level translation diversity.

Diversity Machine Translation +2

Rebounding Bandits for Modeling Satiation Effects

no code implementations NeurIPS 2021 Liu Leqi, Fatma Kilinc-Karzan, Zachary C. Lipton, Alan L. Montgomery

Psychological research shows that enjoyment of many goods is subject to satiation, with short-term satisfaction declining after repeated exposures to the same item.

Recommendation Systems

Fair Machine Learning Under Partial Compliance

no code implementations7 Nov 2020 Jessica Dai, Sina Fazelpour, Zachary C. Lipton

If k% of employers were to voluntarily adopt a fairness-promoting intervention, should we expect k% progress (in aggregate) towards the benefits of universal adoption, or will the dynamics of partial compliance wash out the hoped-for benefits?

BIG-bench Machine Learning Fairness

Weakly- and Semi-supervised Evidence Extraction

1 code implementation Findings of the Association for Computational Linguistics 2020 Danish Pruthi, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton

For many prediction tasks, stakeholders desire not only predictions but also supporting evidence that a human can use to verify its correctness.

Unsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data

no code implementations EMNLP 2021 David Lowell, Brian E. Howard, Zachary C. Lipton, Byron C. Wallace

Unsupervised Data Augmentation (UDA) is a semi-supervised technique that applies a consistency loss to penalize differences between a model's predictions on (a) observed (unlabeled) examples; and (b) corresponding 'noised' examples produced via data augmentation.

Data Augmentation text-classification +2

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

1 code implementation EMNLP 2020 ZiRui Wang, Zachary C. Lipton, Yulia Tsvetkov

Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages.

Meta-Learning

Explaining The Efficacy of Counterfactually Augmented Data

no code implementations ICLR 2021 Divyansh Kaushik, Amrith Setlur, Eduard Hovy, Zachary C. Lipton

In attempts to produce ML models less reliant on spurious patterns in NLP datasets, researchers have recently proposed curating counterfactually augmented data (CAD) via a human-in-the-loop process in which given some documents and their (initial) labels, humans must revise the text to make a counterfactual label applicable.

counterfactual Domain Generalization

Extracting Structured Data from Physician-Patient Conversations By Predicting Noteworthy Utterances

no code implementations14 Jul 2020 Kundan Krishna, Amy Pavel, Benjamin Schloss, Jeffrey P. Bigham, Zachary C. Lipton

In this exploratory study, we describe a new dataset consisting of conversation transcripts, post-visit summaries, corresponding supporting evidence (in the transcript), and structured labels.

Sentence

Uncertainty-Aware Lookahead Factor Models for Quantitative Investing

no code implementations7 Jul 2020 Lakshay Chauhan, John Alberg, Zachary C. Lipton

On a periodic basis, publicly traded companies report fundamentals, financial data including revenue, earnings, debt, among others.

Predicting Mortality Risk in Viral and Unspecified Pneumonia to Assist Clinicians with COVID-19 ECMO Planning

1 code implementation2 Jun 2020 Helen Zhou, Cheng Cheng, Zachary C. Lipton, George H. Chen, Jeremy C. Weiss

Finally, the PEER score is provided in the form of a nomogram for direct calculation of patient risk, and can be used to highlight at-risk patients among critical care patients eligible for ECMO.

Decompensation

Estimating Treatment Effects with Observed Confounders and Mediators

no code implementations26 Mar 2020 Shantanu Gupta, Zachary C. Lipton, David Childers

We show that it strictly outperforms the backdoor and frontdoor estimators and that this improvement can be unbounded.

valid

A Unified View of Label Shift Estimation

no code implementations NeurIPS 2020 Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, Zachary C. Lipton

Our contributions include (i) consistency conditions for MLLS, which include calibration of the classifier and a confusion matrix invertibility condition that BBSE also requires; (ii) a unified framework, casting BBSE as roughly equivalent to MLLS for a particular choice of calibration method; and (iii) a decomposition of MLLS's finite-sample error into terms reflecting miscalibration and estimation error.

Causal Inference With Selectively Deconfounded Data

no code implementations25 Feb 2020 Kyra Gan, Andrew A. Li, Zachary C. Lipton, Sridhar Tayur

In this paper, we consider the benefit of incorporating a large confounded observational dataset (confounder unobserved) alongside a small deconfounded observational dataset (confounder revealed) when estimating the ATE.

Causal Inference

Algorithmic Fairness from a Non-ideal Perspective

no code implementations8 Jan 2020 Sina Fazelpour, Zachary C. Lipton

Inspired by recent breakthroughs in predictive modeling, practitioners in both industry and government have turned to machine learning with hopes of operationalizing predictions to drive automated decisions.

BIG-bench Machine Learning Fairness +1

Game Design for Eliciting Distinguishable Behavior

no code implementations NeurIPS 2019 Fan Yang, Liu Leqi, Yifan Wu, Zachary C. Lipton, Pradeep Ravikumar, William W. Cohen, Tom Mitchell

The ability to inferring latent psychological traits from human behavior is key to developing personalized human-interacting machine learning systems.

Game Design

Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?

no code implementations18 Oct 2019 Simran Kaur, Jeremy Cohen, Zachary C. Lipton

For a standard convolutional neural network, optimizing over the input pixels to maximize the score of some target class will generally produce a grainy-looking version of the original image.

Adversarial Robustness

Accelerating Deep Learning by Focusing on the Biggest Losers

2 code implementations2 Oct 2019 Angela H. Jiang, Daniel L. -K. Wong, Giulio Zhou, David G. Andersen, Jeffrey Dean, Gregory R. Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C. Lipton, Padmanabhan Pillai

This paper introduces Selective-Backprop, a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration.

Learning the Difference that Makes a Difference with Counterfactually-Augmented Data

2 code implementations ICLR 2020 Divyansh Kaushik, Eduard Hovy, Zachary C. Lipton

While classifiers trained on either original or manipulated data alone are sensitive to spurious features (e. g., mentions of genre), models trained on the combined data are less sensitive to this signal.

counterfactual Data Augmentation +2

Learning to Deceive with Attention-Based Explanations

3 code implementations ACL 2020 Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton

Attention mechanisms are ubiquitous components in neural architectures applied to natural language processing.

Fairness

Entity Projection via Machine Translation for Cross-Lingual NER

1 code implementation IJCNLP 2019 Alankar Jain, Bhargavi Paranjape, Zachary C. Lipton

Although over 100 languages are supported by strong off-the-shelf machine translation systems, only a subset of them possess large annotated corpora for named entity recognition.

Cross-Lingual NER Machine Translation +4

AmazonQA: A Review-Based Question Answering Task

1 code implementation12 Aug 2019 Mansi Gupta, Nitish Kulkarni, Raghuveer Chanda, Anirudha Rayasam, Zachary C. Lipton

Observing that many questions can be answered based upon the available product reviews, we propose the task of review-based QA.

Answer Generation Information Retrieval +3

Estimating brain age based on a healthy population with deep learning and structural MRI

no code implementations1 Jul 2019 Xinyang Feng, Zachary C. Lipton, Jie Yang, Scott A. Small, Frank A. Provenzano

Numerous studies have established that estimated brain age, as derived from statistical models trained on healthy populations, constitutes a valuable biomarker that is predictive of cognitive decline and various neurological diseases.

Age Estimation

Learning Causal State Representations of Partially Observable Environments

no code implementations25 Jun 2019 Amy Zhang, Zachary C. Lipton, Luis Pineda, Kamyar Azizzadenesheli, Anima Anandkumar, Laurent Itti, Joelle Pineau, Tommaso Furlanello

In this paper, we propose an algorithm to approximate causal states, which are the coarsest partition of the joint history of actions and observations in partially-observable Markov decision processes (POMDP).

Causal Inference

Efficient candidate screening under multiple tests and implications for fairness

no code implementations27 May 2019 Lee Cohen, Zachary C. Lipton, Yishay Mansour

We analyze the optimal employer policy both when the employer sets a fixed number of tests per candidate and when the employer can set a dynamic policy, assigning further tests adaptively based on results from the previous tests.

Fairness

Combating Adversarial Misspellings with Robust Word Recognition

3 code implementations ACL 2019 Danish Pruthi, Bhuwan Dhingra, Zachary C. Lipton

To combat adversarial spelling mistakes, we propose placing a word recognition model in front of the downstream classifier.

Sentiment Analysis

Embryo staging with weakly-supervised region selection and dynamically-decoded predictions

no code implementations9 Apr 2019 Tingfung Lau, Nathan Ng, Julian Gingold, Nina Desai, Julian McAuley, Zachary C. Lipton

First, noting that in each image the embryo occupies a small subregion, we jointly train a region proposal network with the downstream classifier to isolate the embryo.

Decoder Region Proposal

Learning Robust Representations by Projecting Superficial Statistics Out

no code implementations ICLR 2019 Haohan Wang, Zexue He, Zachary C. Lipton, Eric P. Xing

We test our method on the battery of standard domain generalization data sets and, interestingly, achieve comparable or better performance as compared to other domain generalization methods that explicitly require samples from the target distribution for training.

Domain Generalization

What is the Effect of Importance Weighting in Deep Learning?

1 code implementation8 Dec 2018 Jonathon Byrd, Zachary C. Lipton

Importance-weighted risk minimization is a key ingredient in many machine learning algorithms for causal inference, domain adaptation, class imbalance, and off-policy reinforcement learning.

Causal Inference Domain Adaptation +1

Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study

no code implementations EMNLP 2018 Aditya Siddhant, Zachary C. Lipton

This paper provides a large scale empirical study of deep active learning, addressing multiple tasks and, for each, multiple datasets, multiple models, and a full suite of acquisition functions.

Active Learning Open-Ended Question Answering

Learning Noise-Invariant Representations for Robust Speech Recognition

no code implementations17 Jul 2018 Davis Liang, Zhiheng Huang, Zachary C. Lipton

Despite rapid advances in speech recognition, current models remain brittle to superficial perturbations to their inputs.

Data Augmentation Representation Learning +2

Practical Obstacles to Deploying Active Learning

no code implementations IJCNLP 2019 David Lowell, Zachary C. Lipton, Byron C. Wallace

Active learning (AL) is a widely-used training strategy for maximizing predictive performance subject to a fixed annotation budget.

Active Learning Text Classification

Troubling Trends in Machine Learning Scholarship

no code implementations9 Jul 2018 Zachary C. Lipton, Jacob Steinhardt

Collectively, machine learning (ML) researchers are engaged in the creation and dissemination of knowledge about data-driven algorithms.

BIG-bench Machine Learning

Surprising Negative Results for Generative Adversarial Tree Search

3 code implementations ICLR 2019 Kamyar Azizzadenesheli, Brandon Yang, Weitang Liu, Zachary C. Lipton, Animashree Anandkumar

We deploy this model and propose generative adversarial tree search (GATS) a deep RL algorithm that learns the environment model and implements Monte Carlo tree search (MCTS) on the learned model for planning.

Atari Games Reinforcement Learning (RL)

Born Again Neural Networks

2 code implementations ICML 2018 Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, Anima Anandkumar

Knowledge distillation (KD) consists of transferring knowledge from one machine learning model (the teacher}) to another (the student).

Image Classification Knowledge Distillation

Correction by Projection: Denoising Images with Generative Adversarial Networks

no code implementations12 Mar 2018 Subarna Tripathi, Zachary C. Lipton, Truong Q. Nguyen

In this paper, we propose to denoise corrupted images by finding the nearest point on the GAN manifold, recovering latent vectors by minimizing distances in image space.

Denoising

Active Learning with Partial Feedback

1 code implementation ICLR 2019 Peiyun Hu, Zachary C. Lipton, Anima Anandkumar, Deva Ramanan

While many active learning papers assume that the learner can simply ask for a label and receive it, real annotation often presents a mismatch between the form of a label (say, one among many classes), and the form of an annotation (typically yes/no binary feedback).

Active Learning

Detecting and Correcting for Label Shift with Black Box Predictors

1 code implementation ICML 2018 Zachary C. Lipton, Yu-Xiang Wang, Alex Smola

Faced with distribution shift between training and test set, we wish to detect and quantify the shift, and to correct our classifiers without test set labels.

Medical Diagnosis

Learning From Noisy Singly-labeled Data

1 code implementation ICLR 2018 Ashish Khetan, Zachary C. Lipton, Anima Anandkumar

We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data.

The Doctor Just Won't Accept That!

no code implementations20 Nov 2017 Zachary C. Lipton

For the field of interpretable machine learning to advance, we must ask the following questions: What precisely won't various stakeholders accept?

BIG-bench Machine Learning Interpretable Machine Learning

Does mitigating ML's impact disparity require treatment disparity?

1 code implementation NeurIPS 2018 Zachary C. Lipton, Alexandra Chouldechova, Julian McAuley

Following related work in law and policy, two notions of disparity have come to shape the study of fairness in algorithmic decision-making.

Decision Making Fairness

Improving Factor-Based Quantitative Investing by Forecasting Company Fundamentals

no code implementations13 Nov 2017 John Alberg, Zachary C. Lipton

Academic research has identified some factors, i. e. computed features of the reported data, that are known through retrospective analysis to outperform the market average.

Estimating Reactions and Recommending Products with Generative Models of Reviews

no code implementations IJCNLP 2017 Jianmo Ni, Zachary C. Lipton, Sharad Vikram, Julian McAuley

Natural language approaches that model information like product reviews have proved to be incredibly useful in improving the performance of such methods, as reviews provide valuable auxiliary information that can be used to better estimate latent user preferences and item properties.

Collaborative Filtering Language Modelling +2

Tensor Regression Networks

no code implementations26 Jul 2017 Jean Kossaifi, Zachary C. Lipton, Arinbjorn Kolbeinsson, Aran Khanna, Tommaso Furlanello, Anima Anandkumar

First, we introduce Tensor Contraction Layers (TCLs) that reduce the dimensionality of their input while preserving their multilinear structure using tensor contraction.

regression

Deep Active Learning for Named Entity Recognition

2 code implementations WS 2017 Yanyao Shen, Hyokun Yun, Zachary C. Lipton, Yakov Kronrod, Animashree Anandkumar

In this work, we demonstrate that the amount of labeled training data can be drastically reduced when deep learning is combined with active learning.

Active Learning Decoder +4

Tensor Contraction Layers for Parsimonious Deep Nets

no code implementations1 Jun 2017 Jean Kossaifi, Aran Khanna, Zachary C. Lipton, Tommaso Furlanello, Anima Anandkumar

Specifically, we propose the Tensor Contraction Layer (TCL), the first attempt to incorporate tensor contractions as end-to-end trainable neural network layers.

Model Compression

Dance Dance Convolution

1 code implementation ICML 2017 Chris Donahue, Zachary C. Lipton, Julian McAuley

For the step placement task, we combine recurrent and convolutional neural networks to ingest spectrograms of low-level audio features to predict steps, conditioned on chart difficulty.

Predicting Surgery Duration with Neural Heteroscedastic Regression

no code implementations17 Feb 2017 Nathan Ng, Rodney A Gabriel, Julian McAuley, Charles Elkan, Zachary C. Lipton

Scheduling surgeries is a challenging task due to the fundamental uncertainty of the clinical environment, as well as the risks and costs associated with under- and over-booking.

regression Scheduling

Precise Recovery of Latent Vectors from Generative Adversarial Networks

1 code implementation15 Feb 2017 Zachary C. Lipton, Subarna Tripathi

Generative adversarial networks (GANs) transform latent vectors into visually plausible images.

Context Matters: Refining Object Detection in Video with Recurrent Neural Networks

no code implementations15 Jul 2016 Subarna Tripathi, Zachary C. Lipton, Serge Belongie, Truong Nguyen

Then we train a recurrent neural network that takes as input sequences of pseudo-labeled frames and optimizes an objective that encourages both accuracy on the target frame and consistency across consecutive frames.

Object object-detection +1

The Mythos of Model Interpretability

2 code implementations10 Jun 2016 Zachary C. Lipton

First, we examine the motivations underlying interest in interpretability, finding them to be diverse and occasionally discordant.

Stuck in a What? Adventures in Weight Space

no code implementations23 Feb 2016 Zachary C. Lipton

As neural networks are typically over-complete, it's easy to show the existence of vast continuous regions through weight space with equal loss.

Generative Concatenative Nets Jointly Learn to Write and Classify Reviews

1 code implementation11 Nov 2015 Zachary C. Lipton, Sharad Vikram, Julian McAuley

A recommender system's basic task is to estimate how users will respond to unseen items.

Negation

Learning to Diagnose with LSTM Recurrent Neural Networks

no code implementations11 Nov 2015 Zachary C. Lipton, David C. Kale, Charles Elkan, Randall Wetzel

We present the first study to empirically evaluate the ability of LSTMs to recognize patterns in multivariate time series of clinical measurements.

Time Series Time Series Analysis

Phenotyping of Clinical Time Series with LSTM Recurrent Neural Networks

no code implementations26 Oct 2015 Zachary C. Lipton, David C. Kale, Randall C. Wetzel

We present a novel application of LSTM recurrent neural networks to multilabel classification of diagnoses given variable-length time series of clinical measurements.

General Classification Time Series +1

A Critical Review of Recurrent Neural Networks for Sequence Learning

3 code implementations29 May 2015 Zachary C. Lipton, John Berkowitz, Charles Elkan

Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes.

Handwriting Recognition Image Captioning +5

Efficient Elastic Net Regularization for Sparse Linear Models

no code implementations24 May 2015 Zachary C. Lipton, Charles Elkan

This paper provides closed-form updates for the popular squared norm $\ell^2_2$ and elastic net regularizers.

Differential Privacy and Machine Learning: a Survey and Review

no code implementations24 Dec 2014 Zhanglong Ji, Zachary C. Lipton, Charles Elkan

The objective of machine learning is to extract useful information from data, while privacy is preserved by concealing information.

BIG-bench Machine Learning Privacy Preserving

Cannot find the paper you are looking for? You can Submit a new open access paper.