Search Results for author: Dawn Song

Found 139 papers, 72 papers with code

RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content

no code implementations19 Mar 2024 Zhuowen Yuan, Zidi Xiong, Yi Zeng, Ning Yu, Ruoxi Jia, Dawn Song, Bo Li

The innovative use of constrained optimization and a fusion-based guardrail approach represents a significant step forward in developing more secure and reliable LLMs, setting a new standard for content moderation frameworks in the face of evolving digital threats.

Data Augmentation

Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression

no code implementations18 Mar 2024 Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian Bartoldson, Ajay Jaiswal, Kaidi Xu, Bhavya Kailkhura, Dan Hendrycks, Dawn Song, Zhangyang Wang, Bo Li

While state-of-the-art (SoTA) compression methods boast impressive advancements in preserving benign task performance, the potential risks of compression in terms of safety and trustworthiness have been largely neglected.

Ethics Fairness +1

Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Study

1 code implementation15 Mar 2024 Chenguang Wang, Ruoxi Jia, Xin Liu, Dawn Song

We show that CLIP leads to a significant robustness drop compared to supervised ImageNet models on our benchmark, especially under synthetic distribution shift and adversarial attacks.

Benchmarking

C-RAG: Certified Generation Risks for Retrieval-Augmented Language Models

1 code implementation5 Feb 2024 Mintong Kang, Nezihe Merve Gürel, Ning Yu, Dawn Song, Bo Li

Specifically, we provide conformal risk analysis for RAG models and certify an upper confidence bound of generation risks, which we refer to as conformal generation risk.

Retrieval

GRATH: Gradual Self-Truthifying for Large Language Models

no code implementations22 Jan 2024 Weixin Chen, Dawn Song, Bo Li

GRATH iteratively refines truthfulness data and updates the model, leading to a gradual improvement in model truthfulness in a self-supervised manner.

TextGuard: Provable Defense against Backdoor Attacks on Text Classification

1 code implementation19 Nov 2023 Hengzhi Pei, Jinyuan Jia, Wenbo Guo, Bo Li, Dawn Song

In this work, we propose TextGuard, the first provable defense against backdoor attacks on text classification.

Sentence text-classification +1

DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification

1 code implementation NeurIPS 2023 Mintong Kang, Dawn Song, Bo Li

In particular, we propose a deviated-reconstruction loss at intermediate diffusion steps to induce inaccurate density gradient estimation to tackle the problem of vanishing/exploding gradients.

Effective and Efficient Federated Tree Learning on Hybrid Data

no code implementations18 Oct 2023 Qinbin Li, Chulin Xie, Xiaojun Xu, Xiaoyuan Liu, Ce Zhang, Bo Li, Bingsheng He, Dawn Song

To address this, we propose HybridTree, a novel federated learning approach that enables federated tree learning on hybrid data.

Federated Learning

Agent Instructs Large Language Models to be General Zero-Shot Reasoners

1 code implementation5 Oct 2023 Nicholas Crispino, Kyle Montgomery, Fankun Zeng, Dawn Song, Chenguang Wang

For instance, our method boosts the performance of state-of-the-art large language models by a large margin, including Vicuna-13b (13. 3%), Llama-2-70b-chat (23. 2%), and GPT-3. 5 Turbo (17. 0%).

Representation Engineering: A Top-Down Approach to AI Transparency

1 code implementation2 Oct 2023 Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, Dan Hendrycks

In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience.

Question Answering

DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models

no code implementations NeurIPS 2023 Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, Bo Li

Yet, while the literature on the trustworthiness of GPT models remains limited, practitioners have proposed employing capable GPT models for sensitive applications such as healthcare and finance -- where mistakes can be costly.

Adversarial Robustness Ethics +1

The False Promise of Imitating Proprietary LLMs

2 code implementations25 May 2023 Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao liu, Pieter Abbeel, Sergey Levine, Dawn Song

This approach looks to cheaply imitate the proprietary model's capabilities using a weaker open-source model.

Language Modelling

Blockchain Large Language Models

no code implementations25 Apr 2023 Yu Gai, Liyi Zhou, Kaihua Qin, Dawn Song, Arthur Gervais

This paper presents a dynamic, real-time approach to detecting anomalous blockchain transactions.

Anomaly Detection Intrusion Detection +2

TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets

3 code implementations CVPR 2023 Weixin Chen, Dawn Song, Bo Li

To answer these questions, we propose an effective Trojan attack against diffusion models, TrojDiff, which optimizes the Trojan diffusion and generative processes during training.

Image Generation

Unique Identification of 50,000+ Virtual Reality Users from Head & Hand Motion Data

1 code implementation17 Feb 2023 Vivek Nair, Wenbo Guo, Justus Mattern, Rui Wang, James F. O'Brien, Louis Rosenberg, Dawn Song

With the recent explosive growth of interest and investment in virtual reality (VR) and the so-called "metaverse," public attention has rightly shifted toward the unique security and privacy threats that these platforms may pose.

DensePure: Understanding Diffusion Models towards Adversarial Robustness

no code implementations1 Nov 2022 Chaowei Xiao, Zhongzhu Chen, Kun Jin, Jiongxiao Wang, Weili Nie, Mingyan Liu, Anima Anandkumar, Bo Li, Dawn Song

By using the highest density point in the conditional distribution as the reversed sample, we identify the robust region of a given instance under the diffusion model's reverse process.

Adversarial Robustness Denoising

Benchmarking Language Models for Code Syntax Understanding

1 code implementation26 Oct 2022 Da Shen, Xinyun Chen, Chenguang Wang, Koushik Sen, Dawn Song

Our key observation is that existing language models pretrained on code still lack the understanding of code syntax.

Benchmarking

IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models

no code implementations25 Oct 2022 Chenguang Wang, Xiao Liu, Dawn Song

Instead of focusing on pre-defined relations, we create an OIE benchmark aiming to fully examine the open relational information present in the pre-trained LMs.

Open Information Extraction

How Would The Viewer Feel? Estimating Wellbeing From Video Scenarios

1 code implementation18 Oct 2022 Mantas Mazeika, Eric Tang, Andy Zou, Steven Basart, Jun Shern Chan, Dawn Song, David Forsyth, Jacob Steinhardt, Dan Hendrycks

In experiments, we show how video models that are primarily trained to recognize actions and find contours of objects can be repurposed to understand human preferences and the emotional content of videos.

Video Understanding

Joint Language Semantic and Structure Embedding for Knowledge Graph Completion

1 code implementation COLING 2022 Jianhao Shen, Chenguang Wang, Linyuan Gong, Dawn Song

Unlike previous approaches that rely on either the structures or semantics of the knowledge graphs, we propose to jointly embed the semantics in the natural language description of the knowledge triplets with their structure information.

Link Prediction

UniFed: All-In-One Federated Learning Platform to Unify Open-Source Frameworks

1 code implementation21 Jul 2022 Xiaoyuan Liu, Tianneng Shi, Chulin Xie, Qinbin Li, Kangping Hu, Haoyu Kim, Xiaojun Xu, The-Anh Vu-Le, Zhen Huang, Arash Nourian, Bo Li, Dawn Song

The platform streamlines the end-to-end workflow for distributed experimentation and deployment, encompassing 11 popular open-source FL frameworks.

Federated Learning

Is Vertical Logistic Regression Privacy-Preserving? A Comprehensive Privacy Analysis and Beyond

no code implementations19 Jul 2022 Yuzheng Hu, Tianle Cai, Jinyong Shan, Shange Tang, Chaochao Cai, Ethan Song, Bo Li, Dawn Song

We provide a comprehensive and rigorous privacy analysis of VLR in a class of open-source Federated Learning frameworks, where the protocols might differ between one another, yet a procedure of obtaining local gradients is implicitly shared.

Philosophy Privacy Preserving +2

Byzantine-Robust Federated Learning with Optimal Statistical Rates and Privacy Guarantees

2 code implementations24 May 2022 Banghua Zhu, Lun Wang, Qi Pang, Shuai Wang, Jiantao Jiao, Dawn Song, Michael I. Jordan

In contrast to prior work, our proposed protocols improve the dimension dependence and achieve a tight statistical rate in terms of all the parameters for strongly convex losses.

Federated Learning

PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures

2 code implementations CVPR 2022 Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, Jacob Steinhardt

In real-world applications of machine learning, reliable and safe systems must consider measures of performance beyond standard test set accuracy.

Adversarial Robustness Anomaly Detection +1

Grounded Graph Decoding Improves Compositional Generalization in Question Answering

1 code implementation Findings (EMNLP) 2021 Yu Gai, Paras Jain, Wendi Zhang, Joseph E. Gonzalez, Dawn Song, Ion Stoica

Grounding enables the model to retain syntax information from the input in thereby significantly improving generalization over complex inputs.

Question Answering

What Would Jiminy Cricket Do? Towards Agents That Behave Morally

1 code implementation25 Oct 2021 Dan Hendrycks, Mantas Mazeika, Andy Zou, Sahil Patel, Christine Zhu, Jesus Navarro, Dawn Song, Bo Li, Jacob Steinhardt

When making everyday decisions, people are guided by their conscience, an internal sense of right and wrong.

Adversarial Collaborative Learning on Non-IID Features

1 code implementation29 Sep 2021 Qinbin Li, Bingsheng He, Dawn Song

Federated learning has been a popular approach to enable collaborative learning on multiple parties without exchanging raw data.

Federated Learning

Secure Byzantine-Robust Federated Learning with Dimension-free Error

no code implementations29 Sep 2021 Lun Wang, Qi Pang, Shuai Wang, Dawn Song

In the present work, we propose a federated learning protocol with bi-directional security guarantees.

Federated Learning

FED-$\chi^2$: Secure Federated Correlation Test

no code implementations29 Sep 2021 Lun Wang, Qi Pang, Shuai Wang, Dawn Song

In this paper, we propose the first secure federated $\chi^2$-test protocol, FED-$\chi^2$.

RobustART: Benchmarking Robustness on Architecture Design and Training Techniques

1 code implementation11 Sep 2021 Shiyu Tang, Ruihao Gong, Yan Wang, Aishan Liu, Jiakai Wang, Xinyun Chen, Fengwei Yu, Xianglong Liu, Dawn Song, Alan Yuille, Philip H. S. Torr, DaCheng Tao

Thus, we propose RobustART, the first comprehensive Robustness investigation benchmark on ImageNet regarding ARchitecture design (49 human-designed off-the-shelf architectures and 1200+ networks from neural architecture search) and Training techniques (10+ techniques, e. g., data augmentation) towards diverse noises (adversarial, natural, and system noises).

Adversarial Robustness Benchmarking +2

Latent Execution for Neural Program Synthesis

1 code implementation NeurIPS 2021 Xinyun Chen, Dawn Song, Yuandong Tian

While recent works demonstrated limited success on domain-specific languages (DSL), it remains highly challenging to apply them to real-world programming languages, such as C. Due to complicated syntax and token variation, there are three major challenges: (1) unlike many DSLs, programs in languages like C need to compile first and are not executed via interpreters; (2) the program search space grows exponentially when the syntax and semantics of the programming language become more complex; and (3) collecting a large-scale dataset of real-world programs is non-trivial.

C++ code Program Synthesis

Adversarial Examples for k-Nearest Neighbor Classifiers Based on Higher-Order Voronoi Diagrams

no code implementations NeurIPS 2021 Chawin Sitawarin, Evgenios M. Kornaropoulos, Dawn Song, David Wagner

On a high level, the search radius expands to the nearby higher-order Voronoi cells until we find a cell that classifies differently from the input point.

Adversarial Robustness

Measuring Coding Challenge Competence With APPS

3 code implementations20 May 2021 Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, Jacob Steinhardt

Recent models such as GPT-Neo can pass approximately 20% of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code.

BIG-bench Machine Learning Code Generation

BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning

no code implementations2 May 2021 Lun Wang, Zaynah Javed, Xian Wu, Wenbo Guo, Xinyu Xing, Dawn Song

Recent research has confirmed the feasibility of backdoor attacks in deep reinforcement learning (RL) systems.

Atari Games Backdoor Attack +2

Model-Contrastive Federated Learning

6 code implementations CVPR 2021 Qinbin Li, Bingsheng He, Dawn Song

A key challenge in federated learning is to handle the heterogeneity of local data distribution across parties.

Contrastive Learning Federated Learning +1

Measuring Mathematical Problem Solving With the MATH Dataset

4 code implementations5 Mar 2021 Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt

To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics.

Math Math Word Problem Solving +1

TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models

1 code implementation16 Feb 2021 Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, Ion Stoica

With this key idea, we design TeraPipe, a high-performance token-level pipeline parallel algorithm for synchronous model-parallel training of Transformer-based language models.

A System for Automated Open-Source Threat Intelligence Gathering and Management

no code implementations19 Jan 2021 Peng Gao, Xiaoyuan Liu, Edward Choi, Bhavna Soman, Chinmaya Mishra, Kate Farris, Dawn Song

SecurityKG collects OSCTI reports from various sources, uses a combination of AI and NLP techniques to extract high-fidelity knowledge about threat behaviors, and constructs a security knowledge graph.

Management

Perturbation Type Categorization for Multiple $\ell_p$ Bounded Adversarial Robustness

no code implementations1 Jan 2021 Pratyush Maini, Xinyun Chen, Bo Li, Dawn Song

In addition, we demonstrate the realization of this trade-off in deep networks by adding random noise to the model input at test time, enabling enhanced robustness against strong adaptive attacks.

Adversarial Robustness Vocal Bursts Type Prediction

D2p-fed:Differentially Private Federated Learning with Efficient Communication

no code implementations1 Jan 2021 Lun Wang, Ruoxi Jia, Dawn Song

We provide complete analysis of the privacy guarantee, communication cost and convergence rate of D2p-fed.

Federated Learning

F^2ed-Learning: Good Fences Make Good Neighbors

no code implementations1 Jan 2021 Lun Wang, Qi Pang, Shuai Wang, Dawn Song

In this paper, we present F^2ed-Learning, the first federated learning protocol simultaneously defending against both semi-honest server and Byzantine malicious clients.

Federated Learning

How Multipurpose Are Language Models?

no code implementations ICLR 2021 Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt

By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.

Elementary Mathematics World Knowledge

A Rigorous Evaluation of Real-World Distribution Shifts

no code implementations1 Jan 2021 Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, Justin Gilmer

Motivated by this, we introduce a new data augmentation method which advances the state-of-the-art and outperforms models pretrained with 1000x more labeled data.

Data Augmentation

Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses

no code implementations18 Dec 2020 Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, Tom Goldstein

As machine learning systems grow in scale, so do their training data requirements, forcing practitioners to automate and outsource the curation of training data in order to achieve state-of-the-art performance.

BIG-bench Machine Learning Data Poisoning

Extracting Training Data from Large Language Models

4 code implementations14 Dec 2020 Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel

We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data.

Language Modelling

Adversarial Examples for $k$-Nearest Neighbor Classifiers Based on Higher-Order Voronoi Diagrams

1 code implementation NeurIPS 2021 Chawin Sitawarin, Evgenios M. Kornaropoulos, Dawn Song, David Wagner

On a high level, the search radius expands to the nearby Voronoi cells until we find a cell that classifies differently from the input point.

Adversarial Robustness

Enabling Efficient Cyber Threat Hunting With Cyber Threat Intelligence

1 code implementation26 Oct 2020 Peng Gao, Fei Shao, Xiaoyuan Liu, Xusheng Xiao, Zheng Qin, Fengyuan Xu, Prateek Mittal, Sanjeev R. Kulkarni, Dawn Song

Log-based cyber threat hunting has emerged as an important solution to counter sophisticated attacks.

Language Models are Open Knowledge Graphs

2 code implementations22 Oct 2020 Chenguang Wang, Xiao Liu, Dawn Song

This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e. g., BERT, GPT-2/3), without human supervision.

Knowledge Graphs

Towards Bidirectional Protection in Federated Learning

no code implementations2 Oct 2020 Lun Wang, Qi Pang, Shuai Wang, Dawn Song

At one end of the spectrum, some work uses secure aggregation techniques to hide the individual client's updates and only reveal the aggregated global update to a malicious server that strives to infer the clients' privacy from their updates.

Federated Learning

Practical One-Shot Federated Learning for Cross-Silo Setting

1 code implementation2 Oct 2020 Qinbin Li, Bingsheng He, Dawn Song

Federated learning enables multiple parties to collaboratively learn a model without exchanging their data.

Federated Learning Transfer Learning

Model-Agnostic Round-Optimal Federated Learning via Knowledge Transfer

no code implementations28 Sep 2020 Qinbin Li, Bingsheng He, Dawn Song

In this paper, we propose a novel federated learning algorithm FedKT that needs only a single communication round (i. e., round-optimal).

Federated Learning Transfer Learning

A Principled Approach to Data Valuation for Federated Learning

no code implementations14 Sep 2020 Tianhao Wang, Johannes Rausch, Ce Zhang, Ruoxi Jia, Dawn Song

The federated SV preserves the desirable properties of the canonical SV while it can be calculated without incurring extra communication cost and is also able to capture the effect of participation order on data value.

Data Summarization Data Valuation +1

Measuring Massive Multitask Language Understanding

11 code implementations7 Sep 2020 Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt

By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.

Elementary Mathematics Multi-task Language Understanding +1

Compositional Generalization via Neural-Symbolic Stack Machines

no code implementations NeurIPS 2020 Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, Denny Zhou

Despite achieving tremendous success, existing deep learning models have exposed limitations in compositional generalization, the capability to learn compositional rules and apply them to unseen cases in a systematic manner.

Few-Shot Learning Machine Translation +1

Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis

1 code implementation NeurIPS 2020 Kavi Gupta, Peter Ebert Christensen, Xinyun Chen, Dawn Song

The use of deep learning techniques has achieved significant progress for program synthesis from input-output examples.

Program Synthesis

D2P-Fed: Differentially Private Federated Learning With Efficient Communication

no code implementations22 Jun 2020 Lun Wang, Ruoxi Jia, Dawn Song

In this paper, we propose the discrete Gaussian based differentially private federated learning (D2P-Fed), a unified scheme to achieve both differential privacy (DP) and communication efficiency in federated learning (FL).

Federated Learning

Towards practical differentially private causal graph discovery

no code implementations NeurIPS 2020 Lun Wang, Qi Pang, Dawn Song

Causal graph discovery refers to the process of discovering causal relation graphs from purely observational data.

Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension

no code implementations ICLR 2020 Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, Quoc V. Le

Integrating distributed representations with symbolic operations is essential for reading comprehension requiring complex reasoning, such as counting, sorting and arithmetics, but most existing approaches are hard to scale to more domains or more complex reasoning.

Data Augmentation Math +2

Imitation Attacks and Defenses for Black-box Machine Translation Systems

1 code implementation EMNLP 2020 Eric Wallace, Mitchell Stern, Dawn Song

To mitigate these vulnerabilities, we propose a defense that modifies translation outputs in order to misdirect the optimization of imitation models.

Machine Translation Translation

Pretrained Transformers Improve Out-of-Distribution Robustness

1 code implementation ACL 2020 Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, Dawn Song

Although pretrained Transformers such as BERT achieve high accuracy on in-distribution examples, do they generalize to new distributions?

Anomalous Example Detection in Deep Learning: A Survey

no code implementations16 Mar 2020 Saikiran Bulusu, Bhavya Kailkhura, Bo Li, Pramod K. Varshney, Dawn Song

This survey tries to provide a structured and comprehensive overview of the research on anomaly detection for DL based applications.

Anomaly Detection

Proceedings of the AAAI-20 Workshop on Intelligent Process Automation (IPA-20)

no code implementations15 Jan 2020 Dell Zhang, Andre Freitas, DaCheng Tao, Dawn Song

This is the Proceedings of the AAAI-20 Workshop on Intelligent Process Automation (IPA-20) which took place in New York, NY, USA on February 7th 2020.

Synthetic Datasets for Neural Program Synthesis

no code implementations ICLR 2019 Richard Shin, Neel Kant, Kavi Gupta, Christopher Bender, Brandon Trabucco, Rishabh Singh, Dawn Song

The goal of program synthesis is to automatically generate programs in a particular language from corresponding specifications, e. g. input-output behavior.

Program Synthesis

Scaling Out-of-Distribution Detection for Real-World Settings

3 code implementations25 Nov 2019 Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, Dawn Song

We conduct extensive experiments in these more realistic settings for out-of-distribution detection and find that a surprisingly simple detector based on the maximum logit outperforms prior methods in all the large-scale multi-class, multi-label, and segmentation tasks, establishing a simple new baseline for future work.

Out-of-Distribution Detection Segmentation +2

The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks

1 code implementation CVPR 2020 Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, Dawn Song

This paper studies model-inversion attacks, in which the access to a model is abused to infer information about the training data.

Face Recognition regression

REFIT: A Unified Watermark Removal Framework For Deep Learning Systems With Limited Data

1 code implementation17 Nov 2019 Xinyun Chen, Wenxiao Wang, Chris Bender, Yiming Ding, Ruoxi Jia, Bo Li, Dawn Song

The experimental results demonstrate that our fine-tuning based watermark removal attacks could pose real threats to the copyright of pre-trained models, and thus highlight the importance of further investigating the watermarking problem and proposing more robust watermark embedding schemes against the attacks.

Scalability vs. Utility: Do We Have to Sacrifice One for the Other in Data Importance Quantification?

1 code implementation CVPR 2021 Ruoxi Jia, Fan Wu, Xuehui Sun, Jiacen Xu, David Dao, Bhavya Kailkhura, Ce Zhang, Bo Li, Dawn Song

Quantifying the importance of each training point to a learning task is a fundamental problem in machine learning and the estimated importance scores have been leveraged to guide a range of data workflows such as data summarization and domain adaption.

Data Summarization Domain Adaptation

Robust Anomaly Detection and Backdoor Attack Detection Via Differential Privacy

no code implementations ICLR 2020 Min Du, Ruoxi Jia, Dawn Song

In this paper, we demonstrate that applying differential privacy can improve the utility of outlier detection and novelty detection, with an extension to detect poisoning samples in backdoor attacks.

Anomaly Detection Backdoor Attack +2

An Empirical and Comparative Analysis of Data Valuation with Scalable Algorithms

no code implementations25 Sep 2019 Ruoxi Jia, Xuehui Sun, Jiacen Xu, Ce Zhang, Bo Li, Dawn Song

Existing approximation algorithms, although achieving great improvement over the exact algorithm, relies on retraining models for multiple times, thus remaining limited when applied to larger-scale learning tasks and real-world datasets.

Data Summarization Data Valuation +1

Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms

3 code implementations22 Aug 2019 Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J. Spanos, Dawn Song

The most surprising result is that for unweighted $K$NN classifiers and regressors, the Shapley value of all $N$ data points can be computed, exactly, in $O(N\log N)$ time -- an exponential improvement on computational complexity!

Data Valuation Fairness

TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systems

1 code implementation2 Aug 2019 Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, Dawn Song

As such, given a deep neural network model and clean input samples, it is very challenging to inspect and determine the existence of a trojan backdoor.

Anomaly Detection

Natural Adversarial Examples

3 code implementations CVPR 2021 Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, Dawn Song

We also curate an adversarial out-of-distribution detection dataset called ImageNet-O, which is the first out-of-distribution detection dataset created for ImageNet models.

Adversarial Attack Data Augmentation +2

Execution-Guided Neural Program Synthesis

no code implementations ICLR 2019 Xinyun Chen, Chang Liu, Dawn Song

Most existing neural program synthesis approaches employ an encoder-decoder architecture, which uses an encoder to compute the embedding of the given input-output examples, as well as a decoder to generate the program from the embedding following a given syntax.

Program Synthesis

How You Act Tells a Lot: Privacy-Leakage Attack on Deep Reinforcement Learning

no code implementations24 Apr 2019 Xinlei Pan, Wei-Yao Wang, Xiaoshuai Zhang, Bo Li, Jin-Feng Yi, Dawn Song

To the best of our knowledge, this is the first work to investigate privacy leakage in DRL settings and we show that DRL-based agents do potentially leak privacy-sensitive information from the trained policies.

Autonomous Driving Continuous Control +3

Towards Efficient Data Valuation Based on the Shapley Value

1 code implementation27 Feb 2019 Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gurel, Bo Li, Ce Zhang, Dawn Song, Costas Spanos

In this paper, we study the problem of data valuation by utilizing the Shapley value, a popular notion of value which originated in cooperative game theory.

Data Valuation

Improving Neural Program Synthesis with Inferred Execution Traces

no code implementations NeurIPS 2018 Richard Shin, Illia Polosukhin, Dawn Song

The task of program synthesis, or automatically generating programs that are consistent with a provided specification, remains a challenging task in artificial intelligence.

Program Synthesis

Data Poisoning Attack against Unsupervised Node Embedding Methods

no code implementations30 Oct 2018 Mingjie Sun, Jian Tang, Huichen Li, Bo Li, Chaowei Xiao, Yao Chen, Dawn Song

In this paper, we take the task of link prediction as an example, which is one of the most fundamental problems for graph analysis, and introduce a data positioning attack to node embedding methods.

Data Poisoning Link Prediction

Characterizing Audio Adversarial Examples Using Temporal Dependency

no code implementations ICLR 2019 Zhuolin Yang, Bo Li, Pin-Yu Chen, Dawn Song

In particular, our results reveal the importance of using the temporal dependency in audio data to gain discriminate power against adversarial examples.

Adversarial Defense Automatic Speech Recognition +2

Chorus: a Programming Framework for Building Scalable Differential Privacy Mechanisms

1 code implementation20 Sep 2018 Noah Johnson, Joseph P. Near, Joseph M. Hellerstein, Dawn Song

Differential privacy is fast becoming the gold standard in enabling statistical analysis of data while protecting the privacy of individuals.

Cryptography and Security

Practical Black-box Attacks on Deep Neural Networks using Efficient Query Mechanisms

no code implementations ECCV 2018 Arjun Nitin Bhagoji, Warren He, Bo Li, Dawn Song

An iterative variant of our attack achieves close to 100% attack success rates for both targeted and untargeted attacks on DNNs.

Physical Adversarial Examples for Object Detectors

no code implementations20 Jul 2018 Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Florian Tramer, Atul Prakash, Tadayoshi Kohno, Dawn Song

In this work, we extend physical attacks to more challenging object detection models, a broader class of deep learning algorithms widely used to detect and label multiple objects within a scene.

Object object-detection +1

GamePad: A Learning Environment for Theorem Proving

1 code implementation ICLR 2019 Daniel Huang, Prafulla Dhariwal, Dawn Song, Ilya Sutskever

In this paper, we introduce a system called GamePad that can be used to explore the application of machine learning methods to theorem proving in the Coq proof assistant.

Automated Theorem Proving Position

Robust Physical-World Attacks on Deep Learning Visual Classification

no code implementations CVPR 2018 Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, Dawn Song

Recent studies show that the state-of-the-art deep neural networks (DNNs) are vulnerable to adversarial examples, resulting from small-magnitude perturbations added to the input.

Classification General Classification

Curriculum Adversarial Training

2 code implementations13 May 2018 Qi-Zhi Cai, Min Du, Chang Liu, Dawn Song

The existence of adversarial examples hinders such applications.

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

1 code implementation22 Feb 2018 Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, Dawn Song

This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models---a common type of machine-learning model.

Memorization

Tree-to-tree Neural Networks for Program Translation

no code implementations ICLR 2018 Xinyun Chen, Chang Liu, Dawn Song

We observe that program translation is a modular procedure, in which a sub-tree of the source tree is translated into the corresponding target sub-tree at each step.

Translation

Adversarial Texts with Gradient Methods

1 code implementation22 Jan 2018 Zhitao Gong, Wenlu Wang, Bo Li, Dawn Song, Wei-Shinn Ku

In addition, we empirically show that WMD is closely related to the quality of adversarial texts.

Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality

1 code implementation ICLR 2018 Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E. Houle, James Bailey

Deep Neural Networks (DNNs) have recently been shown to be vulnerable against adversarial examples, which are carefully crafted instances that can mislead DNNs to make errors during prediction.

Adversarial Defense

Spatially Transformed Adversarial Examples

3 code implementations ICLR 2018 Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, Dawn Song

Perturbations generated through spatial transformation could result in large $\mathcal{L}_p$ distance measures, but our extensive experiments show that such spatially transformed adversarial examples are perceptually realistic and more difficult to defend against with existing defense systems.

Decision Boundary Analysis of Adversarial Examples

1 code implementation ICLR 2018 Warren He, Bo Li, Dawn Song

We find that the boundaries around these adversarial examples do not resemble the boundaries around benign examples.

Gradients explode - Deep Networks are shallow - ResNet explained

no code implementations ICLR 2018 George Philipp, Dawn Song, Jaime G. Carbonell

Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities ``solve'' the exploding gradient problem, we show that this is not the case and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice.

Learning what to learn in a neural program

no code implementations ICLR 2018 Richard Shin, Dawn Song

Recent work has shown that it is possible to address these issues by using recursion in the Neural Programmer-Interpreter, but this technique requires a verification set which is difficult to construct without knowledge of the internals of the oracle used to generate training data.

Parametrized Hierarchical Procedures for Neural Programming

no code implementations ICLR 2018 Roy Fox, Richard Shin, Sanjay Krishnan, Ken Goldberg, Dawn Song, Ion Stoica

Neural programs are highly accurate and structured policies that perform algorithmic tasks by controlling the behavior of a computation mechanism.

Imitation Learning

Exploring the Space of Black-box Attacks on Deep Neural Networks

1 code implementation ICLR 2018 Arjun Nitin Bhagoji, Warren He, Bo Li, Dawn Song

An iterative variant of our attack achieves close to 100% adversarial success rates for both targeted and untargeted attacks on DNNs.

Note on Attacking Object Detectors with Adversarial Stickers

no code implementations21 Dec 2017 Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Dawn Song, Tadayoshi Kohno, Amir Rahmati, Atul Prakash, Florian Tramer

Given the fact that state-of-the-art objection detection algorithms are harder to be fooled by the same set of adversarial examples, here we show that these detectors can also be attacked by physical adversarial examples.

Object

A Berkeley View of Systems Challenges for AI

no code implementations15 Dec 2017 Ion Stoica, Dawn Song, Raluca Ada Popa, David Patterson, Michael W. Mahoney, Randy Katz, Anthony D. Joseph, Michael Jordan, Joseph M. Hellerstein, Joseph E. Gonzalez, Ken Goldberg, Ali Ghodsi, David Culler, Pieter Abbeel

With the increasing commoditization of computer vision, speech recognition and machine translation systems and the widespread deployment of learning-based back-end technologies such as digital advertising and intelligent infrastructures, AI (Artificial Intelligence) has moved from research labs to production.

Machine Translation speech-recognition +1

Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning

2 code implementations15 Dec 2017 Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, Dawn Song

In this work, we consider a new type of attacks, called backdoor attacks, where the attacker's goal is to create a backdoor into a learning-based authentication system, so that he can easily circumvent the system by leveraging the backdoor.

Data Poisoning Face Recognition

The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions

no code implementations15 Dec 2017 George Philipp, Dawn Song, Jaime G. Carbonell

Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice.

JPEG-resistant Adversarial Images

no code implementations NIPS 2017 Workshop on Machine Learning and Computer Security 2017 Richard Shin, Dawn Song

Several papers have explored the use of JPEG compression as a defense against adversarial images.

SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning

13 code implementations ICLR 2018 Xiaojun Xu, Chang Liu, Dawn Song

Existing state-of-the-art approaches rely on reinforcement learning to reward the decoder when it generates any of the equivalent serializations.

reinforcement-learning Reinforcement Learning (RL) +1

Fooling Vision and Language Models Despite Localization and Attention Mechanism

no code implementations CVPR 2018 Xiaojun Xu, Xinyun Chen, Chang Liu, Anna Rohrbach, Trevor Darrell, Dawn Song

Our work sheds new light on understanding adversarial attacks on vision systems which have a language component and shows that attention, bounding box localization, and compositional internal structures are vulnerable to adversarial attacks.

Dense Captioning Natural Language Understanding +2

Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection

1 code implementation22 Aug 2017 Xiaojun Xu, Chang Liu, Qian Feng, Heng Yin, Le Song, Dawn Song

The problem of cross-platform binary code similarity detection aims at detecting whether two binary functions coming from different platforms are similar or not.

Computer Security Graph Embedding +2

Robust Physical-World Attacks on Deep Learning Models

1 code implementation27 Jul 2017 Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, Dawn Song

We propose a general attack algorithm, Robust Physical Perturbations (RP2), to generate robust visual adversarial perturbations under different physical conditions.

Towards Practical Differential Privacy for SQL Queries

2 code implementations28 Jun 2017 Noah Johnson, Joseph P. Near, Dawn Song

To meet these requirements we propose elastic sensitivity, a novel method for approximating the local sensitivity of queries with general equijoins.

Cryptography and Security Databases

Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong

no code implementations15 Jun 2017 Warren He, James Wei, Xinyun Chen, Nicholas Carlini, Dawn Song

We ask whether a strong defense can be created by combining multiple (possibly weak) defenses.

Towards Synthesizing Complex Programs from Input-Output Examples

no code implementations ICLR 2018 Xinyun Chen, Chang Liu, Dawn Song

In our evaluation, we show that using our novel approach, neural parsing programs can be learned to achieve 100% test accuracy on test inputs that are 500x longer than the training samples.

Program Synthesis reinforcement-learning +1

Delving into adversarial attacks on deep policies

no code implementations18 May 2017 Jernej Kos, Dawn Song

Adversarial examples have been shown to exist for a variety of deep learning architectures.

reinforcement-learning Reinforcement Learning (RL)

Making Neural Programming Architectures Generalize via Recursion

no code implementations21 Apr 2017 Jonathon Cai, Richard Shin, Dawn Song

Empirically, neural networks that attempt to learn programs from data have exhibited poor generalizability.

Adversarial examples for generative models

1 code implementation22 Feb 2017 Jernej Kos, Ian Fischer, Dawn Song

We explore methods of producing adversarial examples on deep generative models such as the variational autoencoder (VAE) and the VAE-GAN.

Classification General Classification

Delving into Transferable Adversarial Examples and Black-box Attacks

2 code implementations8 Nov 2016 Yanpei Liu, Xinyun Chen, Chang Liu, Dawn Song

In this work, we are the first to conduct an extensive study of the transferability over large models and a large scale dataset, and we are also the first to study the transferability of targeted adversarial examples with their target labels.

Adversarial Attack Adversarial Defense +1

ExploreKit: Automatic Feature Generation and Selection

1 code implementation ICDM 2016 2016 Gilad Katz, Eui Chul Richard Shin, Dawn Song

To overcome the exponential growth of the feature space, ExploreKit uses a novel machine learning-based feature selection approach to predict the usefulness of new candidate features.

Automated Feature Engineering BIG-bench Machine Learning +3

Cannot find the paper you are looking for? You can Submit a new open access paper.