Search Results for author:
Found 117 papers, 78 papers with code
Advancing the Robustness of Large Language Models through Self-Denoised Smoothing
Although large language models (LLMs) have achieved significant success, their vulnerability to adversarial perturbations, including recent jailbreak attacks, has raised considerable concerns.
A Survey on Data Selection for Language Models
A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training.
Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing
Aligned large language models (LLMs) are vulnerable to jailbreaking attacks, which bypass the safeguards of targeted LLMs and fool them into generating objectionable content.
Augment before You Try: Knowledge-Enhanced Table Question Answering via Table Expansion
Table question answering is a popular task that assesses a model's ability to understand and interact with structured data.
Correcting Diffusion Generation through Resampling
Despite diffusion models' superior capabilities in modeling complex distributions, there are still non-trivial distributional discrepancies between generated and ground-truth images, which has resulted in several notable problems in image generation, including missing object errors in text-to-image generation and low image quality.
Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling
Uncertainty decomposition refers to the task of decomposing the total uncertainty of a model into data (aleatoric) uncertainty, resulting from the inherent complexity or ambiguity of the data, and model (epistemic) uncertainty, resulting from the lack of knowledge in the model.
GDL-DS: A Benchmark for Geometric Deep Learning under Distribution Shifts
Geometric deep learning (GDL) has gained significant attention in various scientific fields, chiefly for its proficiency in modeling data with intricate geometric structures.
Audio-Visual Neural Syntax Acquisition
We study phrase structure induction from visually-grounded speech.
Robust Mixture-of-Expert Training for Convolutional Neural Networks
Since the lack of robustness has become one of the main hurdles for CNNs, in this paper we ask: How to adversarially robustify a CNN-based MoE model?
Certified Robustness for Large Language Models with Self-Denoising
This largely falls into the study of certified robust LLMs, i. e., all predictions of LLM are certified to be correct in a local region around the input.
Improving Diffusion Models for Scene Text Editing with Dual Encoders
Scene text editing is a challenging task that involves modifying or inserting specified texts in an image while maintaining its natural and realistic appearance.
Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis
We then impose spatial attention control by combining the attention over the entire text description and that over the local description of the particular object in the corresponding pixel region of that object.
Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models
COPAINT also uses the Bayesian framework to jointly modify both revealed and unrevealed regions, but approximates the posterior distribution in a way that allows the errors to gradually drop to zero throughout the denoising steps, thus strongly penalizing any mismatches with the reference image.
Users are the North Star for AI Transparency
We conduct a broad literature survey, identifying many clusters of similar conceptions of transparency, tying each back to our north star with analysis of how it furthers or hinders our ideal AI transparency goals.
TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimization
Robustness evaluation against adversarial examples has become increasingly important to unveil the trustworthiness of the prevailing deep models in natural language processing (NLP).
PromptBoosting: Black-Box Text Classification with Ten Forward Passes
Instead of directly optimizing in prompt space, PromptBoosting obtains a small pool of prompts via a gradient-free approach and then constructs a large pool of weak learners by pairing these prompts with different elements of the LM's output distribution.
Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models
Based on this finding, we further propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation.
CLAWSAT: Towards Both Robust and Accurate Code Models
We integrate contrastive learning (CL) with adversarial learning to co-optimize the robustness and accuracy of code models.
Data-Model-Circuit Tri-Design for Ultra-Light Video Intelligence on Edge Devices
In this paper, we propose a data-model-hardware tri-design framework for high-throughput, low-cost, and high-accuracy multi-object tracking (MOT) on High-Definition (HD) video stream.
Fairness Reprogramming
Specifically, FairReprogram considers the case where models can not be changed and appends to the input a set of perturbations, called the fairness trigger, which is tuned towards the fairness criteria under a min-max formulation.
Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning
Inspired by the recent success of learning robust models with unlabeled data, we explore a new robustness-aware CIL setting, where the learned adversarial robustness has to resist forgetting and be transferred as new tasks come in continually.
Linearity Grafting: Relaxed Neuron Pruning Helps Certifiable Robustness
Certifiable robustness is a highly desirable property for adopting deep neural networks (DNNs) in safety-critical scenarios, but often demands tedious computations to establish.
Data-Efficient Double-Win Lottery Tickets from Robust Pre-training
For example, on downstream CIFAR-10/100 datasets, we identify double-win matching subnetworks with the standard, fast adversarial, and adversarial pre-training from ImageNet, at 89. 26%/73. 79%, 89. 26%/79. 03%, and 91. 41%/83. 22% sparsity, respectively.
Quarantine: Sparsity Can Uncover the Trojan Attack Trigger for Free
Trojan attacks threaten deep neural networks (DNNs) by poisoning them to behave normally on most samples, yet to produce manipulated results for inputs attached with a particular trigger.
Grasping the Arrow of Time from the Singularity: Decoding Micromotion in Low-dimensional Latent Spaces from StyleGAN
To study the motion features in the latent space of StyleGAN, in this paper, we hypothesize and demonstrate that a series of meaningful, natural, and versatile small, local movements (referred to as "micromotion", such as expression, head movement, and aging effect) can be represented in low-rank spaces extracted from the latent space of a conventionally pre-trained StyleGAN-v2 model for face generation, with the guidance of proper "anchors" in the form of either short text or video clips.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings.
Ranked #13 on
Semantic Textual Similarity
on STS16
ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers
Self-supervised learning in speech involves training a speech representation network on a large-scale unannotated speech corpus, and then applying the learned representations to downstream tasks.
Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection
Lifelong event detection aims to incrementally update a model with new event types and data while retaining the capability on previously learned old types.
WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Models
We show that WavPrompt is a few-shot learner that can perform speech understanding tasks better than a naive text baseline.
Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognition
An unsupervised text-to-speech synthesis (TTS) system learns to generate speech waveforms corresponding to any written sentence in a language by observing: 1) a collection of untranscribed speech waveforms in that language; 2) a collection of texts written in that language without access to any transcribed speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspective
To tackle this problem, we next propose to prepend an autoencoder (AE) to a given (black-box) model so that DS can be trained using variance-reduced ZO optimization.
Adversarial Support Alignment
Furthermore, we show that our approach can be viewed as a limit of existing notions of alignment by increasing transportation assignment tolerance.
Optimizer Amalgamation
Selecting an appropriate optimizer for a given problem is of major interest for researchers and practitioners.
Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimization
We first show that the commonly-used Fast-AT is equivalent to using a stochastic gradient algorithm to solve a linearized BLO problem involving a sign operation.
Understanding Interlocking Dynamics of Cooperative Rationalization
The selection mechanism is commonly integrated into the model itself by specifying a two-component cascaded system consisting of a rationale generator, which makes a binary selection of the input features (which is the rationale), and a predictor, which predicts the output based only on the selected features.
Query and Extract: Refining Event Extraction as Type-oriented Binary Decoding
Event extraction is typically modeled as a multi-class classification problem where event types and argument roles are treated as atomic symbols.
On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis
Are end-to-end text-to-speech (TTS) models over-parametrized?
Global Rhythm Style Transfer Without Text Transcriptions
In this paper, we propose AutoPST, which can disentangle global prosody style from speech without relying on any text transcriptions.
Learning Stable Classifiers by Transferring Unstable Features
Empirical results demonstrate that our algorithm is able to maintain robustness on the target task for both synthetically generated environments and real-world environments.
PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
We investigate the existence of sparse subnetworks in pre-trained speech SSL models that achieve even better low-resource ASR results.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Predict then Interpolate: A Simple Algorithm to Learn Stable Classifiers
In this work, we prove that by interpolating the distributions of the correct predictions and the wrong predictions, we can uncover an oracle distribution where the unstable correlation vanishes.
Complementary Evidence Identification in Open-Domain Question Answering
This paper proposes a new problem of complementary evidence identification for open-domain question answering (QA).
Generating Adversarial Computer Programs using Optimized Obfuscations
We further show that our formulation is better at training models that are robust to adversarial attacks.
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
Our vanilla GAN architecture, dubbed TransGAN, consists of a memory-friendly transformer-based generator that progressively increases feature resolution, and correspondingly a multi-scale discriminator to capture simultaneously semantic contexts and low-level textures.
Ranked #8 on
Image Generation
on STL-10
Robust Overfitting may be mitigated by properly learned smoothening
A recent study (Rice et al., 2020) revealed overfitting to be a dominant phenomenon in adversarially robust training of deep networks, and that appropriate early-stopping of adversarial training (AT) could match the performance gains of most recent algorithmic improvements.
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning
In view of those, we introduce two pruning options, e. g., top-down and bottom-up, for finding lifelong tickets.
Self-Progressing Robust Training
Enhancing model robustness under new and even adversarial environments is a crucial milestone toward building trustworthy machine learning systems.
The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
We extend the scope of LTH and question whether matching subnetworks still exist in pre-trained computer vision models, that enjoy the same downstream transfer performance.
Training Stronger Baselines for Learning to Optimize
Learning to optimize (L2O) has gained increasing attention since classical optimizers require laborious problem-specific design and hyperparameter tuning.
Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning
Interactive Fiction (IF) games with real human-written natural language texts provide a new natural evaluation for language understanding techniques.
Lifelong Object Detection
Recent advances in object detection have benefited significantly from rapid developments in deep neural networks.
The Lottery Ticket Hypothesis for Pre-trained BERT Networks
For a range of downstream tasks, we indeed find matching subnetworks at 40% to 90% sparsity.
Proper Network Interpretability Helps Adversarial Robustness in Classification
Recent works have empirically shown that there exist adversarial examples that can be hidden from neural network interpretability (namely, making network interpretation maps visually similar), or interpretability is itself susceptible to adversarial attacks.
Can 3D Adversarial Logos Cloak Humans?
Contrary to the traditional adversarial patch, this new form of attack is mapped into the 3D object world and back-propagates to the 2D image domain through differentiable rendering.
Unsupervised Speech Decomposition via Triple Information Bottleneck
Speech information can be roughly decomposed into four components: language content, timbre, pitch, and rhythm.
Learning to Recover Reasoning Chains for Multi-Hop Question Answering via Cooperative Games
We propose the new problem of learning to recover reasoning chains from weakly supervised signals, i. e., the question-answer pairs.
Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins (eg, 3. 83% on robust accuracy and 1. 3% on standard accuracy, on the CIFAR-10 dataset), compared with the conventional end-to-end adversarial training baseline.
Invariant Rationalization
Selective rationalization improves neural network interpretability by identifying a small subset of input features -- the rationale -- that best explains or supports the prediction.
Context-Aware Conversation Thread Detection in Multi-Party Chat
In multi-party chat, it is common for multiple conversations to occur concurrently, leading to intermingled conversation threads in chat logs.
Rethinking Cooperative Rationalization: Introspective Extraction and Complement Control
Moreover, we explicitly control the rationale complement via an adversary so as not to leave any useful information out of the selection.
A Game Theoretic Approach to Class-wise Selective Rationalization
Selection of input features such as relevant pieces of text has become a common technique of highlighting how complex neural predictors operate.
An Efficient and Margin-Approaching Zero-Confidence Adversarial Attack
The second paradigm, called the zero-confidence attack, finds the smallest perturbation needed to cause mis-classification, also known as the margin of an input feature.
Continuous Convolutional Neural Network forNonuniform Time Series
Convolutional neural network (CNN) for time series data implicitly assumes that the data are uniformly sampled, whereas many event-based and multi-modal data are nonuniform or have heterogeneous sampling rates.
SPROUT: Self-Progressing Robust Training
Enhancing model robustness under new and even adversarial environments is a crucial milestone toward building trustworthy and reliable machine learning systems.
Visual Interpretability Alone Helps Adversarial Robustness
Recent works have empirically shown that there exist adversarial examples that can be hidden from neural network interpretability, and interpretability is itself susceptible to adversarial attacks.
Simple yet Effective Bridge Reasoning for Open-Domain Multi-Hop Question Answering
To resolve this issue, we introduce a new sub-problem of open-domain multi-hop QA, which aims to recognize the bridge (\emph{i. e.}, the anchor that links to the answer passage) from the context of a set of start passages with a reading comprehension model.
Out-of-Domain Detection for Low-Resource Text Classification Tasks
Out-of-domain (OOD) detection for low-resource text classification is a realistic but understudied task.
Few-shot Text Classification with Distributional Signatures
In this paper, we explore meta-learning for few-shot text classification.
Meta Reasoning over Knowledge Graphs
The ability to reason over learned knowledge is an innate ability for humans and humans can easily master new reasoning rules with only a few demonstrations.
AutoGAN: Neural Architecture Search for Generative Adversarial Networks
Neural architecture search (NAS) has witnessed prevailing success in image classification and (very recently) segmentation tasks.
Ranked #16 on
Image Generation
on STL-10
TWEETQA: A Social Media Focused Question Answering Dataset
With social media becoming increasingly pop-ular on which lots of news and real-time eventsare reported, developing automated questionanswering systems is critical to the effective-ness of many applications that rely on real-time knowledge.
Tight Certificates of Adversarial Robustness for Randomly Smoothed Classifiers
Specifically, an $\ell_2$ bounded adversary cannot alter the ensemble prediction generated by an additive isotropic Gaussian noise, where the radius for the adversary depends on both the variance of the distribution as well as the ensemble margin at the point of interest.
Self-Supervised Learning for Contextualized Extractive Summarization
Existing models for extractive summarization are usually trained from scratch with a cross-entropy loss, which does not explicitly capture the global context at the document level.
Coupled Variational Recurrent Collaborative Filtering
To bridge the gap, in this paper, we propose a Coupled Variational Recurrent Collaborative Filtering (CVRCF) framework based on the idea of Deep Bayesian Learning to handle the streaming recommendation problem.
Improving Question Answering over Incomplete KBs with Knowledge-Aware Reader
We propose a new end-to-end question answering model, which learns to aggregate answer evidence from an incomplete knowledge base (KB) and a set of retrieved text snippets.
Additive Adversarial Learning for Unbiased Authentication
Authentication is a task aiming to confirm the truth between data instances and personal identities.
Selection Bias Explorations and Debias Methods for Natural Language Sentence Matching Datasets
Natural Language Sentence Matching (NLSM) has gained substantial attention from both academics and the industry, and rich public datasets contribute a lot to this process.
AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
On the other hand, CVAE training is simple but does not come with the distribution-matching property of a GAN.
Hybrid Reinforcement Learning with Expert State Sequences
The empirical results show that (1) the agents are able to leverage state expert sequences to learn faster than pure reinforcement learning baselines, (2) our tensor-based action inference model is advantageous compared to standard deep neural networks in inferring expert actions, and (3) the hybrid policy optimization objective is robust against noise in expert state sequences.
Sentence Embedding Alignment for Lifelong Relation Extraction
We formulate such a challenging problem as lifelong relation extraction and investigate memory-efficient incremental learning methods without catastrophically forgetting knowledge learned from previous tasks.
Imposing Label-Relational Inductive Bias for Extremely Fine-Grained Entity Typing
Existing entity typing systems usually exploit the type hierarchy provided by knowledge base (KB) schema to model label correlations and thus improve the overall performance.
Ranked #3 on
Entity Typing
on Ontonotes v5 (English)
Extracting Multiple-Relations in One-Pass with Pre-Trained Transformers
Most approaches to extraction multiple relations from a paragraph require multiple passes over the paragraph.
Ranked #19 on
Relation Extraction
on SemEval-2010 Task-8
A Simple Non-i.i.d. Sampling Approach for Efficient Training and Better Generalization
While training on samples drawn from independent and identical distribution has been a de facto paradigm for optimizing image classification networks, humans learn new concepts in an easy-to-hard manner and on the selected examples progressively.
Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge
This study assesses the state-of-the-art machine learning (ML) methods used for brain tumor image analysis in mpMRI scans, during the last seven instances of the International Brain Tumor Segmentation (BraTS) challenge, i. e., 2012-2018.
Learning Corresponded Rationales for Text Matching
The ability to predict matches between two sources of text has a number of applications including natural language inference (NLI) and question answering (QA).
Improving Reinforcement Learning Based Image Captioning with Natural Language Prior
Recently, Reinforcement Learning (RL) approaches have demonstrated advanced performance in image captioning by directly optimizing the metric used for testing.
Deriving Machine Attention from Human Rationales
Attention-based models are successful when trained on large amounts of data.
One-Shot Relational Learning for Knowledge Graphs
Knowledge graphs (KGs) are the key components of various natural language processing applications.
Matrix Factorization on GPUs with Memory Optimization and Approximate Computing
Current MF implementations are either optimized for a single machine or with a need of a large computer cluster but still are insufficient.
Scheduled Policy Optimization for Natural Language Communication with Intelligent Agents
We investigate the task of learning to follow natural language instructions by jointly reasoning with visual observations and language inputs.
A Co-Matching Model for Multi-choice Reading Comprehension
Multi-choice reading comprehension is a challenging task, which involves the matching between a passage and a question-answer pair.
Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization
As application demands for zeroth-order (gradient-free) optimization accelerate, the need for variance reduced and faster converging approaches is also intensifying.
Diverse Few-Shot Text Classification with Multiple Metrics
We study few-shot learning in natural language domains.
Image Super-Resolution via Dual-State Recurrent Networks
Advances in image super-resolution (SR) have recently benefited significantly from rapid developments in deep neural networks.
Ranked #42 on
Image Super-Resolution
on BSD100 - 4x upscaling
Deep Learning Based Speech Beamforming
On the other hand, deep learning based enhancement approaches are able to learn complicated speech distributions and perform efficient inference, but they are unable to deal with variable number of input channels.
Faster Reinforcement Learning with Expert State Sequences
In this paper, we consider a realistic and more difficult sce- nario where a reinforcement learning agent only has access to the state sequences of an expert, while the expert actions are not available.
Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering
We propose two methods, namely, strength-based re-ranking and coverage-based re-ranking, to make use of the aggregated evidence from different passages to better determine the answer.
Ranked #1 on
Open-Domain Question Answering
on Quasar
Dilated Recurrent Neural Networks
To provide a theory-based quantification of the architecture's advantages, we introduce a memory capacity measure, the mean recurrent length, which is more suitable for RNNs with long skip connections than existing measures.
Ranked #24 on
Sequential Image Classification
on Sequential MNIST
Robust Video Super-Resolution With Learned Temporal Dynamics
Second, we reduce the complexity of motion between neighboring frames using a spatial alignment network that is much more robust and efficient than competing alignment methods and can be jointly trained with the temporal adaptive network in an end-to-end manner.
R$^3$: Reinforced Reader-Ranker for Open-Domain Question Answering
Second, we propose a novel method that jointly trains the Ranker along with an answer-generation Reader model, based on reinforcement learning.
Ranked #4 on
Open-Domain Question Answering
on Quasar
Robust Task Clustering for Deep Many-Task Learning
We propose a new method to measure task similarities with cross-task transfer performance matrix for the deep learning scenario.
Jointly Attentive Spatial-Temporal Pooling Networks for Video-based Person Re-Identification
Person Re-Identification (person re-id) is a crucial task as its applications in visual surveillance and human-computer interaction.
Fast Generation for Convolutional Autoregressive Models
In this work, we describe a method to speed up generation in convolutional autoregressive models.
Fast Wavenet Generation Algorithm
This paper presents an efficient implementation of the Wavenet generation process called Fast Wavenet.
Stacked Approximated Regression Machine: A Simple Deep Learning Approach
With the agreement of my coauthors, I Zhangyang Wang would like to withdraw the manuscript "Stacked Approximated Regression Machine: A Simple Deep Learning Approach".
Streaming Recommender Systems
The increasing popularity of real-world recommender systems produces data continuously and rapidly, and it becomes more realistic to study recommender systems under streaming scenarios.
D3: Deep Dual-Domain Based Fast Restoration of JPEG-Compressed Images
In this paper, we design a Deep Dual-Domain (D3) based fast restoration model to remove artifacts of JPEG compressed images.
Learning A Deep $\ell_\infty$ Encoder for Hashing
We investigate the $\ell_\infty$-constrained representation which demonstrates robustness to quantization errors, utilizing the tool of deep learning.
Brain-Inspired Deep Networks for Image Aesthetics Assessment
Image aesthetics assessment has been challenging due to its subjective nature.
Studying Very Low Resolution Recognition Using Deep Networks
Visual recognition research often assumes a sufficient resolution of the region of interest (ROI).
$\mathbf{D^3}$: Deep Dual-Domain Based Fast Restoration of JPEG-Compressed Images
In this paper, we design a Deep Dual-Domain ($\mathbf{D^3}$) based fast restoration model to remove artifacts of JPEG compressed images.
Learning A Task-Specific Deep Architecture For Clustering
In this paper, we propose to emulate the sparse coding-based clustering pipeline in the context of deep learning, leading to a carefully crafted deep model benefiting from both.
Self-Tuned Deep Super Resolution
Deep learning has been successfully applied to image super resolution (SR).
Learning Super-Resolution Jointly from External and Internal Examples
Single image super-resolution (SR) aims to estimate a high-resolution (HR) image from a lowresolution (LR) input.
Scalable Similarity Learning using Large Margin Neighborhood Embedding
In this paper, we present an image similarity learning method that can scale well in both the number of images and the dimensionality of image descriptors.
Learning Locally-Adaptive Decision Functions for Person Verification
This paper proposes to learn a decision function for verification that can be viewed as a joint model of a distance metric and a locally adaptive thresholding rule.
Blind Image Deblurring by Spectral Properties of Convolution Operators
We show that the minimizer of this regularizer guarantees to give good approximation to the blur kernel if the original image is sharp enough.
