no code implementations • ICLR 2019 • Xinyun Chen, Yuandong Tian
For problem solving, making reactive decisions based on problem description is fast but inaccurate, while search-based planning using heuristics gives better solutions but could be exponentially slow.
1 code implementation • 7 Sep 2023 • Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen
Optimization is ubiquitous.
1 code implementation • 31 Aug 2023 • Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, Mark Gerstein
BioCoder incorporates a fuzz-testing framework for evaluation, and we have applied it to evaluate many models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, and ChatGPT.
1 code implementation • 26 May 2023 • Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, Denny Zhou
Our approach consists of two key phases: 1) tool making: an LLM acts as the tool maker that crafts tools for given tasks, where a tool is implemented as a Python utility function.
no code implementations • 24 May 2023 • Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, Denny Zhou
Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost.
no code implementations • 15 May 2023 • Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, Quoc V. Le
We present symbol tuning - finetuning language models on in-context input-label pairs where natural language labels (e. g., "positive/negative sentiment") are replaced with arbitrary symbols (e. g., "foo/bar").
no code implementations • 11 Apr 2023 • Xinyun Chen, Maxwell Lin, Nathanael Schärli, Denny Zhou
In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i. e., without any feedback on the code correctness or error messages, the model is able to identify its mistakes by explaining the generated code in natural language.
1 code implementation • 1 Apr 2023 • Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, David Wagner
Combining our new dataset with previous datasets, we present an analysis of the challenges and promising research directions of using deep learning for detecting software vulnerabilities.
no code implementations • 7 Mar 2023 • Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, Tengyu Ma
We next study semantically-unrelated label ICL (SUL-ICL), in which labels are semantically unrelated to their inputs (e. g., foo/bar instead of negative/positive), thereby forcing language models to learn the input-label mappings shown in in-context exemplars in order to perform the task.
no code implementations • 6 Mar 2023 • Xinyun Chen, Yunan Liu, Guiyu Hong
A major drawback of PTO is that its solution accuracy can often be highly sensitive to the parameter estimation errors because PTO is unable to properly link these errors (step 1) to the quality of the optimized solutions (step 2).
1 code implementation • 31 Jan 2023 • Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, Denny Zhou
We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included.
no code implementations • 16 Nov 2022 • Xinyun Chen, Pengyi Shi, Shanwen Pu
Motivated by the emerging needs of personalized preventative intervention in many healthcare applications, we consider a multi-stage, dynamic decision-making problem in the online setting with unknown model parameters.
1 code implementation • 26 Oct 2022 • Da Shen, Xinyun Chen, Chenguang Wang, Koushik Sen, Dawn Song
Our key observation is that existing language models pretrained on code still lack the understanding of code syntax.
4 code implementations • 20 Oct 2022 • Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, Jason Wei
We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation).
Ranked #1 on
Coreference Resolution
on WSC
Cross-Lingual Question Answering
Multi-task Language Understanding
+1
no code implementations • 29 Sep 2022 • Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, Denny Zhou
Humans can reason compositionally when presented with new tasks.
Ranked #1 on
Semantic Parsing
on CFQ
1 code implementation • Findings (NAACL) 2022 • Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver
To deal with this problem, we modify a number of state-of-the-art models to train on the segmented data of Spider-SS, and we show that this method improves the generalization performance.
1 code implementation • DeepMind 2022 • Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, Oriol Vinyals
Programming is a powerful and ubiquitous problem-solving tool.
Ranked #1 on
Code Generation
on CodeContests
1 code implementation • 16 Dec 2021 • Hui Shi, Sicun Gao, Yuandong Tian, Xinyun Chen, Jishen Zhao
With the forced decomposition, we show that the performance upper bounds of LSTM and Transformer in learning CFL are close: both of them can simulate a stack and perform stack operation along with state transitions.
1 code implementation • 28 Oct 2021 • Hongyu Ren, Hanjun Dai, Bo Dai, Xinyun Chen, Denny Zhou, Jure Leskovec, Dale Schuurmans
There are two important reasoning tasks on KGs: (1) single-hop knowledge graph completion, which involves predicting individual links in the KG; and (2), multi-hop reasoning, where the goal is to predict which KG entities satisfy a given logical query.
1 code implementation • Findings (EMNLP) 2021 • Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John Drake, Qiaofu Zhang
Addressing the mismatch between natural language descriptions and the corresponding SQL queries is a key challenge for text-to-SQL translation.
1 code implementation • EMNLP 2021 • Yujian Gan, Xinyun Chen, Matthew Purver
Recently, there has been significant progress in studying neural networks for translating text descriptions into SQL queries under the zero-shot cross-domain setting.
1 code implementation • 11 Sep 2021 • Shiyu Tang, Ruihao Gong, Yan Wang, Aishan Liu, Jiakai Wang, Xinyun Chen, Fengwei Yu, Xianglong Liu, Dawn Song, Alan Yuille, Philip H. S. Torr, DaCheng Tao
Thus, we propose RobustART, the first comprehensive Robustness investigation benchmark on ImageNet regarding ARchitecture design (49 human-designed off-the-shelf architectures and 1200+ networks from neural architecture search) and Training techniques (10+ techniques, e. g., data augmentation) towards diverse noises (adversarial, natural, and system noises).
1 code implementation • ACL 2021 • Xinyun Chen, Linyuan Gong, Alvin Cheung, Dawn Song
Creating effective visualization is an important part of data analytics.
1 code implementation • NeurIPS 2021 • Xinyun Chen, Dawn Song, Yuandong Tian
While recent works demonstrated limited success on domain-specific languages (DSL), it remains highly challenging to apply them to real-world programming languages, such as C. Due to complicated syntax and token variation, there are three major challenges: (1) unlike many DSLs, programs in languages like C need to compile first and are not executed via interpreters; (2) the program search space grows exponentially when the syntax and semantics of the programming language become more complex; and (3) collecting a large-scale dataset of real-world programs is non-trivial.
1 code implementation • 26 Jun 2021 • Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, Denny Zhou
In this work, we present the first approach for synthesizing spreadsheet formulas from tabular context, which includes both headers and semi-structured tabular data.
no code implementations • 24 Jun 2021 • Shuang Li, Lu Wang, Xinyun Chen, Yixiang Fang, Yan Song
In this paper, we model the propagation of the COVID-19 as spatio-temporal point processes and propose a generative and intensity-free model to track the spread of the disease.
1 code implementation • ACL 2021 • Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, Pengsheng Huang
We observe that the accuracy dramatically drops by eliminating such explicit correspondence between NL questions and table schemas, even if the synonyms are not adversarially selected to conduct worst-case adversarial attacks.
no code implementations • NeurIPS 2021 • Xinyun Chen, Dawn Song, Yuandong Tian
Program synthesis from input-output (IO) examples has been a long-standing challenge.
no code implementations • 12 Apr 2021 • Yihan Pan, Zhenghang Xu, Jin Guang, Jingjing Sun, Chengwenjian Wang, Xuanming Zhang, Xinyun Chen, J. G. Dai, Yichuan Ding, Pengyi Shi, Hongxin Pan, Kai Yang, Song Wu
To address the issue, we propose a novel two-level routing component to the queueing network model.
no code implementations • 25 Feb 2021 • Zhuolin Yang, Zhaoxi Chen, Tiffany Cai, Xinyun Chen, Bo Li, Yuandong Tian
Extensive experiments show that student specialization correlates strongly with model robustness in different scenarios, including student trained via standard training, adversarial training, confidence-calibrated adversarial training, and training with robust feature dataset.
1 code implementation • 1 Jan 2021 • Cheng Fu, Kunlin Yang, Xinyun Chen, Yuandong Tian, Jishen Zhao
In software development, decompilation aims to reverse engineer binary executables.
no code implementations • 1 Jan 2021 • Pratyush Maini, Xinyun Chen, Bo Li, Dawn Song
In addition, we demonstrate the realization of this trade-off in deep networks by adding random noise to the model input at test time, enabling enhanced robustness against strong adaptive attacks.
no code implementations • 18 Dec 2020 • Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, Tom Goldstein
As machine learning systems grow in scale, so do their training data requirements, forcing practitioners to automate and outsource the curation of training data in order to achieve state-of-the-art performance.
1 code implementation • 3 Dec 2020 • Aishan Liu, Shiyu Tang, Xinyun Chen, Lei Huang, Haotong Qin, Xianglong Liu, DaCheng Tao
In this paper, we observe that different $\ell_p$ bounded adversarial perturbations induce different statistical properties that can be separated and characterized by the statistics of Batch Normalization (BN).
no code implementations • 7 Sep 2020 • Xinyun Chen, Yunan Liu, Guiyu Hong
In this work we propose an online learning framework designed for solving this problem which does not require the system's scale to increase.
no code implementations • NeurIPS 2020 • Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, Denny Zhou
Despite achieving tremendous success, existing deep learning models have exposed limitations in compositional generalization, the capability to learn compositional rules and apply them to unseen cases in a systematic manner.
1 code implementation • NeurIPS 2020 • Kavi Gupta, Peter Ebert Christensen, Xinyun Chen, Dawn Song
The use of deep learning techniques has achieved significant progress for program synthesis from input-output examples.
1 code implementation • ECCV 2020 • Aishan Liu, Tairan Huang, Xianglong Liu, Yitao Xu, Yuqing Ma, Xinyun Chen, Stephen J. Maybank, DaCheng Tao
Adversarial attacks are valuable for providing insights into the blind-spots of deep learning models and help improve their robustness.
1 code implementation • ICLR 2020 • Hui Shi, Yang Zhang, Xinyun Chen, Yuandong Tian, Jishen Zhao
Deep symbolic superoptimization refers to the task of applying deep learning methods to simplify symbolic expressions.
no code implementations • ICLR 2020 • Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, Quoc V. Le
Integrating distributed representations with symbolic operations is essential for reading comprehension requiring complex reasoning, such as counting, sorting and arithmetics, but most existing approaches are hard to scale to more domains or more complex reasoning.
Ranked #5 on
Question Answering
on DROP Test
no code implementations • NeurIPS 2019 • Cheng Fu, Huili Chen, Haolan Liu, Xinyun Chen, Yuandong Tian, Farinaz Koushanfar, Jishen Zhao
Furthermore, Coda outperforms the sequence-to-sequence model with attention by a margin of 70% program accuracy.
1 code implementation • 17 Nov 2019 • Xinyun Chen, Wenxiao Wang, Chris Bender, Yiming Ding, Ruoxi Jia, Bo Li, Dawn Song
The experimental results demonstrate that our fine-tuning based watermark removal attacks could pose real threats to the copyright of pre-trained models, and thus highlight the importance of further investigating the watermarking problem and proposing more robust watermark embedding schemes against the attacks.
no code implementations • ICLR 2020 • Xinyun Chen, Lu Wang, Yizhe Hang, Heng Ge, Hongyuan Zha
We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies.
no code implementations • 28 Jun 2019 • Cheng Fu, Huili Chen, Haolan Liu, Xinyun Chen, Yuandong Tian, Farinaz Koushanfar, Jishen Zhao
Reverse engineering of binary executables is a critical problem in the computer security domain.
no code implementations • ICLR 2019 • Xinyun Chen, Chang Liu, Dawn Song
Most existing neural program synthesis approaches employ an encoder-decoder architecture, which uses an encoder to compute the embedding of the given input-output examples, as well as a decoder to generate the program from the embedding following a given syntax.
1 code implementation • NeurIPS 2019 • Xinyun Chen, Yuandong Tian
Search-based methods for hard combinatorial optimization are often guided by heuristics.
no code implementations • ICLR 2018 • Xinyun Chen, Chang Liu, Dawn Song
We observe that program translation is a modular procedure, in which a sub-tree of the source tree is translated into the corresponding target sub-tree at each step.
1 code implementation • 15 Dec 2017 • Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, Dawn Song
In this work, we consider a new type of attacks, called backdoor attacks, where the attacker's goal is to create a backdoor into a learning-based authentication system, so that he can easily circumvent the system by leveraging the backdoor.
no code implementations • CVPR 2018 • Xiaojun Xu, Xinyun Chen, Chang Liu, Anna Rohrbach, Trevor Darrell, Dawn Song
Our work sheds new light on understanding adversarial attacks on vision systems which have a language component and shows that attention, bounding box localization, and compositional internal structures are vulnerable to adversarial attacks.
no code implementations • 15 Jun 2017 • Warren He, James Wei, Xinyun Chen, Nicholas Carlini, Dawn Song
We ask whether a strong defense can be created by combining multiple (possibly weak) defenses.
no code implementations • ICLR 2018 • Xinyun Chen, Chang Liu, Dawn Song
In our evaluation, we show that using our novel approach, neural parsing programs can be learned to achieve 100% test accuracy on test inputs that are 500x longer than the training samples.
1 code implementation • 8 Nov 2016 • Yanpei Liu, Xinyun Chen, Chang Liu, Dawn Song
In this work, we are the first to conduct an extensive study of the transferability over large models and a large scale dataset, and we are also the first to study the transferability of targeted adversarial examples with their target labels.
no code implementations • NeurIPS 2016 • Xinyun Chen, Chang Liu, Richard Shin, Dawn Song, Mingcheng Chen
Automatic translation from natural language descriptions into programs is a longstanding challenging problem.
no code implementations • 9 Apr 2016 • Bo Li, Yevgeniy Vorobeychik, Xinyun Chen
We propose the first systematic and general-purpose retraining framework which can: a) boost robustness of an \emph{arbitrary} learning algorithm, in the face of b) a broader class of adversarial models than any prior methods.