Search Results for author:
Found 120 papers, 35 papers with code
Rethinking Sentiment Style Transfer
To overcome this issue, we propose a graph-based method to extract attribute content and attribute-independent content from input sentences in the YELP dataset and IMDB dataset.
A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation
Large multimodal models still struggle with text-rich images because of inadequate training data.
Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements
Diffusion models have emerged as a powerful foundation model for visual generation.
LoRA-Contextualizing Adaptation of Large Multimodal Models for Long Document Understanding
In this work, we present a novel framework named LoRA-Contextualizing Adaptation of Large multimodal models (LoCAL), which broadens the capabilities of any LMM to support long-document understanding.
TextLap: Customizing Language Models for Text-to-Layout Planning
Automatic generation of graphical layouts is crucial for many real-world applications, including designing posters, flyers, advertisements, and graphical user interfaces.
MMR: Evaluating Reading Ability of Large Multimodal Models
Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images.
LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models
Large multimodal language models have demonstrated impressive capabilities in understanding and manipulating images.
Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning
The experiment on four different prompt tuning structures consistently shows the improvement of our method, with increases of up to $6. 1\%$ in the Base-to-Novel generalization task, $5. 8\%$ in the group robustness task, and $2. 7\%$ in the out-of-distribution tasks.
TRINS: Towards Multimodal Language Models that Can Read
In this work, we introduce TRINS: a Text-Rich image INStruction dataset, with the objective of enhancing the reading ability of the multimodal large language model.
Teaching Human Behavior Improves Content Understanding Abilities Of LLMs
Specifically, we show that training LLMs to predict the receiver behavior of likes and comments improves the LLM's performance on a wide variety of downstream content understanding tasks.
Discriminative Adversarial Unlearning
We consider the scenario of two networks, the attacker $\mathbf{A}$ and the trained defender $\mathbf{D}$ pitted against each other in an adversarial objective, wherein the attacker aims at teasing out the information of the data to be unlearned in order to infer membership, and the defender unlearns to defend the network against the attack, whilst preserving its general performance.
Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints
Controllable layout generation refers to the process of creating a plausible visual arrangement of elements within a graphic design (e. g., document and web designs) with constraints representing design intentions.
Behavior Optimized Image Generation
We show that BoigLLM outperforms 13x larger models such as GPT-3. 5 and GPT-4 in this task, demonstrating that while these state-of-the-art models can understand images, they lack information on how these images perform in the real world.
Low-Resource Named Entity Recognition: Can One-vs-All AUC Maximization Help?
Named entity recognition (NER), a task that identifies and categorizes named entities such as persons or organizations from text, is traditionally framed as a multi-class classification problem.
Long-Term Ad Memorability: Understanding & Generating Memorable Ads
Finally, with the intent of memorable ad generation, we present a scalable method to build a high-quality memorable ad generation model by leveraging automatically annotated data.
Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior
We call these models Large Content and Behavior Models (LCBMs).
Learning Unnormalized Statistical Models via Compositional Optimization
Learning unnormalized statistical models (e. g., energy-based models) is computationally challenging due to the complexity of handling the partition function.
Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels
Remarkably, by incorporating conditional information from the powerful CLIP model, our method can boost the current SOTA accuracy by 10-20 absolute points in many cases.
Ranked #1 on
Image Classification
on Food-101N
(using extra training data)
A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot
Multimedia content, such as advertisements and story videos, exhibit a rich blend of creativity and multiple modalities.
Prompt Tuning based Adapter for Vision-Language Model Adaption
To address this issue, efficient adaptation methods such as prompt tuning have been proposed.
Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning
Hence we advocate that the key of better performance lies in meaningful latent modality structures instead of perfect modality alignment.
ALCAP: Alignment-Augmented Music Captioner
Music captioning has gained significant attention in the wake of the rising prominence of streaming media platforms.
AUC Maximization for Low-Resource Named Entity Recognition
To the best of our knowledge, this is the first work that brings AUC maximization to the NER setting.
Low Resource Named Entity Recognition
named-entity-recognition
+2
Shifted Diffusion for Text-to-image Generation
Unlike the baseline diffusion model used in DALL-E 2, our method seamlessly encodes prior knowledge of the pre-trained CLIP model in its diffusion process by designing a new initialization distribution and a new transition step of the diffusion.
Ranked #3 on
Text-to-Image Generation
on Multi-Modal-CelebA-HQ
Hardness-guided domain adaptation to recognise biomedical named entities under low-resource scenarios
Domain adaptation is an effective solution to data scarcity in low-resource scenarios.
Lafite2: Few-shot Text-to-Image Generation
The low requirement of the proposed method yields high flexibility and usability: it can be beneficial to a wide range of settings, including the few-shot, semi-supervised and fully-supervised learning; it can be applied on different models including generative adversarial networks (GANs) and diffusion models.
Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reduction
Based on this measure, we also design a computation-efficient client sampling strategy, such that the actively selected clients will generate a more class-balanced grouped dataset with theoretical guarantees.
Persuasion Strategies in Advertisements
Motivated by persuasion literature in social psychology and marketing, we introduce an extensive vocabulary of persuasion strategies and build the first ad image corpus annotated with persuasion strategies.
STT: Soft Template Tuning for Few-Shot Adaptation
Significantly, it can even outperform the time- and resource-consuming fine-tuning method on sentiment classification tasks.
Towards Language-Free Training for Text-to-Image Generation
One of the major challenges in training text-to-image generation models is the need of a large number of high-quality text-image pairs.
A Generic Approach for Enhancing GANs by Regularized Latent Optimization
With the rapidly growing model complexity and data volume, training deep generative models (DGMs) for better performance has becoming an increasingly more important challenge.
LAFITE: Towards Language-Free Training for Text-to-Image Generation
One of the major challenges in training text-to-image generation models is the need of a large number of high-quality image-text pairs.
Ranked #2 on
Text-to-Image Generation
on Multi-Modal-CelebA-HQ
Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees
Automated Scoring (AS), the natural language processing task of scoring essays and speeches in an educational testing setting, is growing in popularity and being deployed across contexts from government examinations to companies providing language proficiency services.
Perception Point: Identifying Critical Learning Periods in Speech for Bilingual Networks
Recent studies in speech perception have been closely linked to fields of cognitive psychology, phonology, and phonetics in linguistics.
Rethinking Deep Face Restoration
While existing approaches for face restoration make significant progress in generating high-quality faces, they often fail to preserve facial features and cannot authentically reconstruct the faces.
MINIMAL: Mining Models for Data Free Universal Adversarial Triggers
It is well known that natural language models are vulnerable to adversarial attacks, which are mostly input-specific in nature.
AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses
This is in stark contrast to recent probing studies on pre-trained representation learning models, which show that rich linguistic features such as parts-of-speech and morphology are encoded by them.
Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoring
In our technique, we take advantage of the fact that oral proficiency tests rate multiple responses for a candidate.
Integrating Semantics and Neighborhood Information with Graph-Driven Generative Models for Document Retrieval
With the need of fast retrieval speed and small memory footprint, document hashing has been playing a crucial role in large-scale information retrieval.
Unsupervised Hashing with Contrastive Information Bottleneck
Many unsupervised hashing methods are implicitly established on the idea of reconstructing the input data, which basically encourages the hashing codes to retain as much information of original data as possible.
Learning High-Dimensional Distributions with Latent Neural Fokker-Planck Kernels
Learning high-dimensional distributions is an important yet challenging problem in machine learning with applications in various domains.
Towards Fair Federated Learning with Zero-Shot Data Augmentation
Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data.
Meta-Learning with Neural Tangent Kernels
We achieve this goal by 1) replacing the adaptation with a fast-adaptive regularizer in the RKHS; and 2) solving the adaptation analytically based on the NTK theory.
Outline to Story: Fine-grained Controllable Story Generation from Cascaded Events
Our paper is among the first ones by our knowledge to propose a model and to create datasets for the task of "outline to story".
Transformer-based Conditional Variational Autoencoder for Controllable Story Generation
In this paper, we advocate to revive latent variable modeling, essentially the power of representation learning, in the era of Transformers to enhance controllability without hurting state-of-the-art generation effectiveness.
SDA: Improving Text Generation with Self Data Augmentation
Data augmentation has been widely used to improve deep neural networks in many research fields, such as computer vision.
What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure
In recent times, BERT based transformer models have become an inseparable part of the 'tech stack' of text processing models.
Meta-Learning in Reproducing Kernel Hilbert Space
Within this paradigm, we introduce two meta learning algorithms in RKHS, which no longer need an explicit inner-loop adaptation as in the MAML framework.
Towards Learning to Remember in Meta Learning of Sequential Domains
However, a natural generalization to the sequential domain setting to avoid catastrophe forgetting has not been well investigated.
My Teacher Thinks The World Is Flat! Interpreting Automatic Essay Scoring Mechanism
However, little research has been put to understand and interpret the black-box nature of these deep-learning based scoring models.
ReMP: Rectified Metric Propagation for Few-Shot Learning
Few-shot learning features the capability of generalizing from a few examples.
Bayesian Multi-type Mean Field Multi-agent Imitation Learning
We demonstrate the performance of our algorithm through benchmarking with three state-of-the-art multi-agent imitation learning algorithms on several tasks, including solving a multi-agent traffic optimization problem in a real-world transportation network.
Unpaired Image-to-Image Translation via Latent Energy Transport
Instead of explicitly extracting the two codes and applying adaptive instance normalization to combine them, our latent EBM can implicitly learn to transport the source style code to the target style code while preserving the content code, an advantage over existing image translation methods.
Semantic Matching for Sequence-to-Sequence Learning
In sequence-to-sequence models, classical optimal transport (OT) can be applied to semantically match generated sentences with target sentences.
MixKD: Towards Efficient Distillation of Large-scale Language Models
Large-scale language models have recently demonstrated impressive empirical performance.
Learning Manifold Implicitly via Explicit Heat-Kernel Learning
Manifold learning is a fundamental problem in machine learning with numerous applications.
Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference
The neural attention mechanism plays an important role in many natural language processing applications.
Structure-Aware Human-Action Generation
Generating long-range skeleton-based human actions has been a challenging problem since small deviations of one frame can cause a malformed action sequence.
Ranked #2 on
Human action generation
on NTU RGB+D 2D
Generative Semantic Hashing Enhanced via Boltzmann Machines
Generative semantic hashing is a promising technique for large-scale information retrieval thanks to its fast retrieval speed and small memory footprint.
Graph Neural Networks with Composite Kernels
We then propose feature aggregation as the composition of the original neighbor-based kernel and a learnable kernel to encode feature similarities in a feature space.
Towards Understanding the Adversarial Vulnerability of Skeleton-based Action Recognition
We first formulate generation of adversarial skeleton actions as a constrained optimization problem by representing or approximating the physiological and physical constraints with mathematical formulations.
Reward Constrained Interactive Recommendation with Natural Language Feedback
Text-based interactive recommendation provides richer user feedback and has demonstrated advantages over traditional interactive recommender systems.
Improving Adversarial Text Generation by Modeling the Distant Future
Auto-regressive text generation models usually focus on local fluency, and may cause inconsistent semantic meaning in long text generation.
Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints
Text generation from a knowledge base aims to translate knowledge triples to natural language descriptions.
Bayesian Meta Sampling for Fast Uncertainty Adaptation
Specifically, we propose a Bayesian meta sampling framework consisting of two main components: a meta sampler and a sample adapter.
Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck
Variational autoencoders (VAEs) are essential tools in end-to-end representation learning.
Decomposed Adversarial Learned Inference
Effective inference for a generative adversarial model remains an important and challenging problem.
Feature Quantization Improves GAN Training
The instability in GAN training has been a long-standing problem despite remarkable research efforts.
Ranked #1 on
Image-to-Image Translation
on anime-to-selfie
Nested-Wasserstein Self-Imitation Learning for Sequence Generation
Reinforcement learning (RL) has been widely studied for improving sequence-generation models.
Learning Diverse Stochastic Human-Action Generators by Learning Smooth Latent Transitions
In this paper, we focus on skeleton-based action generation and propose to model smooth and diverse transitions on a latent space of action sequences with much lower dimensionality.
Ranked #4 on
Human action generation
on NTU RGB+D 2D
KernelNet: A Data-Dependent Kernel Parameterization for Deep Generative Modeling
Learning with kernels is an important concept in machine learning.
Text-Based Interactive Recommendation via Constraint-Augmented Reinforcement Learning
Text-based interactive recommendation provides richer user preferences and has demonstrated advantages over traditional interactive recommender systems.
Fine-grained Attention and Feature-sharing Generative Adversarial Networks for Single Image Super-Resolution
Firstly, instead of producing a single score to discriminate images between real and fake, we propose a variant, called Fine-grained Attention Generative Adversarial Network for image super-resolution (FASRGAN), to discriminate each pixel between real and fake.
Implicit Deep Latent Variable Models for Text Generation
Deep latent variable models (LVM) such as variational auto-encoder (VAE) have recently played an important role in text generation.
Document Hashing with Mixture-Prior Generative Models
Hashing is promising for large-scale information retrieval tasks thanks to the efficiency of distance evaluation between binary codes.
Bayesian Uncertainty Matching for Unsupervised Domain Adaptation
By imposing distribution matching on both features and labels (via uncertainty), label distribution mismatching in source and target data is effectively alleviated, encouraging the classifier to produce consistent predictions across domains.
Topic-Guided Variational Auto-Encoder for Text Generation
We propose a topic-guided variational auto-encoder (TGVAE) model for text generation.
On Norm-Agnostic Robustness of Adversarial Training
Adversarial examples are carefully perturbed in-puts for fooling machine learning models.
Second-Order Adversarial Attack and Certifiable Robustness
In this paper, we propose a powerful second-order attack method that reduces the accuracy of the defense model by Madry et al. (2017).
Topic-Guided Variational Autoencoders for Text Generation
We propose a topic-guided variational autoencoder (TGVAE) model for text generation.
Scalable Thompson Sampling via Optimal Transport
Thompson sampling (TS) is a class of algorithms for sequential decision-making, which requires maintaining a posterior distribution over a model.
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
The posteriors over neural network weights are high dimensional and multimodal.
Improving Sequence-to-Sequence Learning via Optimal Transport
Sequence-to-sequence models are commonly trained via maximum likelihood estimation (MLE).
Adversarial Learning of a Sampler Based on an Unnormalized Distribution
We investigate adversarial learning in the case when only an unnormalized form of the density can be accessed, rather than samples.
PointCloud Saliency Maps
Our motivation for constructing a saliency map is by point dropping, which is a non-differentiable operator.
Self-Adversarially Learned Bayesian Sampling
Scalable Bayesian sampling is playing an important role in modern machine learning, especially in the fast-developed unsupervised-(deep)-learning models.
Variance Reduction in Stochastic Particle-Optimization Sampling
Stochastic particle-optimization sampling (SPOS) is a recently-developed scalable Bayesian sampling framework that unifies stochastic gradient MCMC (SG-MCMC) and Stein variational gradient descent (SVGD) algorithms based on Wasserstein gradient flows.
Sequence Generation with Guider Network
Sequence generation with reinforcement learning (RL) has received significant attention recently.
Is PGD-Adversarial Training Necessary? Alternative Training via a Soft-Quantization Network with Noisy-Natural Samples Only
In this paper, we give a negative answer by proposing a training paradigm that is comparable to PGD adversarial training on several standard datasets, while only using noisy-natural samples.
Towards More Theoretically-Grounded Particle Optimization Sampling for Deep Learning
With such theoretical guarantees, SPOS can be safely and effectively applied on both Bayesian DL and deep RL tasks.
AIM: Adversarial Inference by Matching Priors and Conditionals
We propose a novel approach, Adversarial Inference by Matching priors and conditionals (AIM), which explicitly matches prior and conditional distributions in both data and code spaces, and puts a direct constraint on the dependency structure of the generative model.
Certified Adversarial Robustness with Additive Noise
The existence of adversarial data examples has drawn significant attention in the deep-learning community; such data are seemingly minimally perturbed relative to the original data, but lead to very different outputs from a deep-learning algorithm.
Stochastic Particle-Optimization Sampling and the Non-Asymptotic Convergence Theory
Particle-optimization-based sampling (POS) is a recently developed effective sampling technique that interactively updates a set of particles.
Distributionally Adversarial Attack
Recent work on adversarial attack has shown that Projected Gradient Descent (PGD) Adversary is a universal first-order adversary, and the classifier adversarially trained by PGD is robust against a wide range of first-order attacks.
Policy Optimization as Wasserstein Gradient Flows
Policy optimization is a core component of reinforcement learning (RL), and most existing RL methods directly optimize parameters of a policy based on maximizing the expected total reward, or its surrogate.
A Unified Particle-Optimization Framework for Scalable Bayesian Sampling
There has been recent interest in developing scalable Bayesian sampling methods such as stochastic gradient MCMC (SG-MCMC) and Stein variational gradient descent (SVGD) for big-data analysis.
Learning Structural Weight Uncertainty for Sequential Decision-Making
Learning probability distributions on the weights of neural networks (NNs) has recently proven beneficial in many applications.
On Connecting Stochastic Gradient MCMC and Differential Privacy
Significant success has been realized recently on applying machine learning to real-world applications.
Particle Optimization in Stochastic Gradient MCMC
Stochastic gradient Markov chain Monte Carlo (SG-MCMC) has been increasingly popular in Bayesian learning due to its ability to deal with large data.
Zero-Shot Learning via Class-Conditioned Deep Generative Models
We present a deep generative model for learning to predict classes not seen at training time.
ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching
We investigate the non-identifiability issues associated with bidirectional adversarial training for joint distribution matching.
Continuous-Time Flows for Efficient Inference and Density Estimation
Distinct from normalizing flows and GANs, CTFs can be adopted to achieve the above two goals in one framework, with theoretical guarantees.
A Convergence Analysis for A Class of Practical Variance-Reduction Stochastic Gradient MCMC
However, there has been little theoretical analysis of the impact of minibatch size to the algorithm's convergence rate.
Stochastic Gradient Monomial Gamma Sampler
A framework is proposed to improve the sampling efficiency of stochastic gradient MCMC, based on Hamiltonian Monte Carlo.
Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling
Recurrent neural networks (RNNs) have shown promising performance for language modeling.
Earliness-Aware Deep Convolutional Networks for Early Time Series Classification
Unlike most existing methods for early classification of time series data, that are designed to solve this problem under the assumption of the availability of a good set of pre-defined (often hand-crafted) features, our framework can jointly perform feature learning (by learning a deep hierarchy of \emph{shapelets} capturing the salient characteristics in each time series), along with a dynamic truncation model to help our deep feature learning architecture focus on the early parts of each time series.
On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
Our theoretical results show faster convergence rates and more accurate invariant measures for SG-MCMCs with higher-order integrators.
Stochastic Gradient MCMC with Stale Gradients
In this paper we develop theory to show that while the bias and MSE of an SG-MCMC algorithm depend on the staleness of stochastic gradients, its estimation variance (relative to the expected estimate, based on a prescribed number of samples) is independent of it.
Nonparametric Bayesian Topic Modelling with the Hierarchical Pitman-Yor Processes
In this article, we present efficient methods for the use of these processes in this hierarchical context, and apply them to latent variable models for text analytics.
Twitter-Network Topic Model: A Full Bayesian Treatment for Social Network and Text Modeling
Exploiting this additional information, we propose the Twitter-Network (TN) topic model to jointly model the text and the social network in a full Bayesian nonparametric way.
Nonlinear Statistical Learning with Truncated Gaussian Graphical Models
We introduce the truncated Gaussian graphical model (TGGM) as a novel framework for designing statistical models for nonlinear learning.
Learning Weight Uncertainty With Stochastic Gradient MCMC for Shape Classification
Learning the representation of shape cues in 2D & 3D objects for recognition is a fundamental task in computer vision.
Towards Unifying Hamiltonian Monte Carlo and Slice Sampling
We unify slice sampling and Hamiltonian Monte Carlo (HMC) sampling, demonstrating their connection via the Hamiltonian-Jacobi equation from Hamiltonian mechanics.
Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization
Stochastic gradient Markov chain Monte Carlo (SG-MCMC) methods are Bayesian analogs to popular stochastic optimization methods; however, this connection is not well studied.
Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
Pytorch implementations of Bayes By Backprop, MC Dropout, SGLD, the Local Reparametrization Trick, KF-Laplace and more
High-Order Stochastic Gradient Thermostats for Bayesian Learning of Deep Models
Stochastic gradient MCMC algorithms (SG-MCMC) are a family of diffusion-based sampling methods for large-scale Bayesian learning.
Scalable Bayesian Non-Negative Tensor Factorization for Massive Count Data
We present a Bayesian non-negative tensor factorization model for count-valued tensor data, and develop scalable inference algorithms (both batch and online) for dealing with massive tensors.
Bayesian Sampling Using Stochastic Gradient Thermostats
To remedy this problem, we show that one can leverage a small number of additional variables in order to stabilize momentum fluctuations induced by the unknown noise.
Robust Bayesian Max-Margin Clustering
We present max-margin Bayesian clustering (BMC), a general and robust framework that incorporates the max-margin criterion into Bayesian clustering models, as well as two concrete models of BMC to demonstrate its flexibility and effectiveness in dealing with different clustering tasks.
