Search Results for author:
Found 18 papers, 14 papers with code
Adversarial Attacks on Multimodal Agents
Our attacks use adversarial text strings to guide gradient-based perturbation over one trigger image in the environment: (1) our captioner attack attacks white-box captioners if they are used to process images into captions as additional inputs to the VLM; (2) our CLIP attack attacks a set of CLIP models jointly, which can transfer to proprietary VLMs.
Domain Gap Embeddings for Generative Dataset Augmentation
The performance of deep learning models is intrinsically tied to the quality volume and relevance of their training data.
Personalized Face Inpainting with Diffusion Models by Parallel Visual Attention
Specifically, we insert parallel attention matrices to each cross-attention module in the denoising network, which attends to features extracted from reference images by an identity encoder.
Text2Reward: Reward Shaping with Language Models for Reinforcement Learning
Unlike inverse RL and recent work that uses LLMs to write sparse reward codes or unshaped dense rewards with a constant function across timesteps, Text2Reward produces interpretable, free-form dense reward codes that cover a wide range of tasks, utilize existing packages, and allow iterative refinement with human feedback.
ITI-GEN: Inclusive Text-to-Image Generation
We show that, for some attributes, images can represent concepts more expressively than text.
PATMAT: Person Aware Tuning of Mask-Aware Transformer for Face Inpainting
By using ~40 reference images, PATMAT creates anchor points in MAT's style module, and tunes the model using the fixed anchors to adapt the model to a new face identity.
Zero-shot Model Diagnosis
Extensive experiments demonstrate that our method is capable of producing counterfactual images and offering sensitivity analysis for model diagnosis without the need for a test set.
Semantic Image Attack for Visual Model Diagnosis
Rather than relying on a carefully designed test set to assess ML models' failures, fairness, or robustness, this paper proposes Semantic Image Attack (SIA), a method based on the adversarial attack that provides semantic adversarial images to allow model diagnosis, interpretability, and robustness.
A Latent Space of Stochastic Diffusion Models for Zero-Shot Image Editing and Guidance
We demonstrate that this latent space of stochastic diffusion models can be used in the same way as that of deterministic diffusion models in two applications.
Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance
The commonly-adopted formulation of the latent code of diffusion models is a sequence of gradually denoised samples, as opposed to the simpler (e. g., Gaussian) latent space of GANs, VAEs, and normalizing flows.
Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models
Our experiments demonstrate how PromptGen can efficiently sample from several unconditional generative models (e. g., StyleGAN2, StyleNeRF, diffusion autoencoder, NVAE) in a controlled or/and de-biased manner using various off-the-shelf models: (1) with the CLIP model as control, PromptGen can sample images guided by text, (2) with image classifiers as control, PromptGen can de-bias generative models across a set of attributes or attribute combinations, and (3) with inverse graphics models as control, PromptGen can sample images of the same identity in different poses.
Selective Annotation Makes Language Models Better Few-Shot Learners
Departing from recent in-context learning methods, we formulate an annotation-efficient, two-step framework: selective annotation that chooses a pool of examples to annotate from unlabeled data in advance, followed by prompt retrieval that retrieves task examples from the annotated pool at test time.
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models
Structured knowledge grounding (SKG) leverages structured knowledge to complete user requests, such as semantic parsing over databases and question answering over knowledge bases.
Ranked #1 on
Task-Oriented Dialogue Systems
on KVRET
Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents
To the best of our knowledge, Summ$^N$ is the first multi-stage split-then-summarize framework for long input summarization.
DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization
Transformer-based models have achieved state-of-the-art performance on short-input summarization.
Transferable Persona-Grounded Dialogues via Grounded Minimal Edits
Grounded dialogue models generate responses that are grounded on certain concepts.
EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training
Although pre-trained language models have remarkably enhanced the generation ability of dialogue systems, open-domain Chinese dialogue systems are still limited by the dialogue data and the model size compared with English ones.
NAST: A Non-Autoregressive Generator with Word Alignment for Unsupervised Text Style Transfer
First, we observe that most words in the transferred sentence can be aligned with related words in the source sentence, so we explicitly model word alignments to suppress irrelevant words.
