Search Results for author:
Found 63 papers, 23 papers with code
SwapAnyone: Consistent and Realistic Video Synthesis for Swapping Any Person into Any Video
Video body-swapping aims to replace the body in an existing video with a new body from arbitrary sources, which has garnered more attention in recent years.
WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation
Text-to-Image (T2I) models are capable of generating high-quality artistic creations and visual content.
OSCAR: Object Status and Contextual Awareness for Recipes to Support Non-Visual Cooking
OSCAR leverages both Large-Language Models (LLMs) and Vision-Language Models (VLMs) to manipulate recipe steps, extract object status information, align visual frames with object status, and provide cooking progress tracking log.
HD-EPIC: A Highly-Detailed Egocentric Video Dataset
We present a validation dataset of newly-collected kitchen-based egocentric videos, manually annotated with highly detailed and interconnected ground-truth labels covering: recipe steps, fine-grained actions, ingredients with nutritional values, moving objects, and audio annotations.
Calling a Spade a Heart: Gaslighting Multimodal Large Language Models via Negation
Multimodal Large Language Models (MLLMs) have exhibited remarkable advancements in integrating different modalities, excelling in complex understanding and generation tasks.
CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion
These prompts encompass text prompts (representing cooking steps), image prompts (corresponding to cooking images), and multi-modal prompts (mixing cooking steps and images), ensuring the consistent generation of cooking procedural images.
Next Patch Prediction for Autoregressive Visual Generation
Autoregressive models, built based on the Next Token Prediction (NTP) paradigm, show great potential in developing a unified framework that integrates both language and vision tasks.
Open-Sora Plan: Open-Source Large Video Generation Model
We introduce Open-Sora Plan, an open-source project that aims to contribute a large generation model for generating desired high-resolution videos with long durations based on various user inputs.
Visual Cue Enhancement and Dual Low-Rank Adaptation for Efficient Visual Instruction Fine-Tuning
Parameter-efficient fine-tuning multimodal large language models (MLLMs) presents significant challenges, including reliance on high-level visual features that limit fine-grained detail comprehension, and data conflicts that arise from task complexity.
Retrieval Augmented Recipe Generation
Existing works for recipe generation primarily utilize a two-stage training method, first generating ingredients and then obtaining instructions from both the image and ingredients.
$\ell_0$ factor analysis
Factor Analysis is about finding a low-rank plus sparse additive decomposition from a noisy estimate of the signal covariance matrix.
IterSelectTune: An Iterative Training Framework for Efficient Instruction-Tuning Data Selection
As large language models (LLMs) continue to advance, instruction tuning has become critical for improving their ability to generate accurate and contextually appropriate responses.
When atomic norm meets the G-filter: A general framework for line spectral estimation
This paper proposes a novel approach for line spectral estimation which combines Georgiou's filter bank (G-filter) with atomic norm minimization (ANM).
Line Spectral Analysis Using the G-Filter: An Atomic Norm Minimization Approach
In this paper, we develop a novel approach for line spectral estimation which combines ideas of Georgiou's filter banks (G-filters) and atomic norm minimization (ANM), a mainstream method for line spectral analysis in the last decade following the theory of compressed sensing.
OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video Diffusion Model
With the same reconstruction quality, the more sufficient the VAE's compression for videos is, the more efficient the LVDMs are.
Hand1000: Generating Realistic Hands from Text with Only 1,000 Images
The training of Hand1000 is divided into three stages with the first stage aiming to enhance the model's understanding of hand anatomy by using a pre-trained hand gesture recognition model to extract gesture representation.
RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models
Uni-Food is designed to provide a more holistic approach to food data analysis, thereby enhancing the performance and capabilities of LMMs in this domain.
Model Inversion Attacks Through Target-Specific Conditional Diffusion Models
To alleviate these issues, leveraging on diffusion models' remarkable synthesis capabilities, we propose Diffusion-based Model Inversion (Diff-MI) attacks.
EfficientGS: Streamlining Gaussian Splatting for Large-Scale High-Resolution Scene Representation
In the domain of 3D scene representation, 3D Gaussian Splatting (3DGS) has emerged as a pivotal technology.
AURORA: Navigating UI Tarpits via Automated Neural Screen Understanding
We evaluated AURORA both on a set of 12 apps with known tarpits from prior work, and on a new set of five of the most popular apps from the Google Play store.
From Canteen Food to Daily Meals: Generalizing Food Recognition to More Practical Scenarios
These two datasets are used to evaluate the transferability of approaches from the well-curated food image domain to the everyday-life food image domain.
LLMBind: A Unified Modality-Task Integration Framework
In the multi-modal domain, the dependence of various models on specific input formats leads to user confusion and hinders progress.
Video Editing for Video Retrieval
The teacher model is employed to edit the clips in the training set whereas the student model trains on the edited clips.
FoodLMM: A Versatile Food Assistant using Large Multi-modal Model
In the second stage, we construct a multi-round conversation dataset and a reasoning segmentation dataset to fine-tune the model, enabling it to conduct professional dialogues and generate segmentation masks based on complex reasoning in the food domain.
Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models
Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content.
Enhancing Recipe Retrieval with Foundation Models: A Data Augmentation Perspective
Leveraging on the remarkable capabilities of foundation models (i. e., Llama2 and SAM), we propose to augment recipe and food image by extracting alignable information related to the counterpart.
Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models
Video-based large language models (Video-LLMs) have been recently introduced, targeting both fundamental improvements in perception and comprehension, and a diverse range of user inquiries.
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM.
Ranked #9 on
Zero-Shot Video Question Answer
on TGIF-QA
X-Transfer: A Transfer Learning-Based Framework for GAN-Generated Fake Image Detection
Generative adversarial networks (GANs) have remarkably advanced in diverse domains, especially image generation and editing.
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
We thus propose VIDAL-10M with Video, Infrared, Depth, Audio and their corresponding Language, naming as VIDAL-10M.
Ranked #1 on
Zero-shot Audio Classification
on VGG-Sound
(using extra training data)
Controlling Neural Style Transfer with Deep Reinforcement Learning
Controlling the degree of stylization in the Neural Style Transfer (NST) is a little tricky since it usually needs hand-engineering on hyper-parameters.
CrossDF: Improving Cross-Domain Deepfake Detection with Deep Information Decomposition
Deepfake technology poses a significant threat to security and social trust.
RL-I2IT: Image-to-Image Translation with Deep Reinforcement Learning
The key feature in the RL-I2IT framework is to decompose a monolithic learning process into small steps with a lightweight model to progressively transform a source image successively to a target image.
MKL-$L_{0/1}$-SVM
This paper presents a Multiple Kernel Learning (abbreviated as MKL) framework for the Support Vector Machine (SVM) with the $(0, 1)$ loss function.
CgT-GAN: CLIP-guided Text GAN for Image Captioning
Particularly, we use adversarial training to teach CgT-GAN to mimic the phrases of an external text corpus and CLIP-based reward to provide semantic guidance.
Towards Attack-tolerant Federated Learning via Critical Parameter Analysis
Federated learning is used to train a shared model in a decentralized way without clients sharing private data with each other.
FedDefender: Client-Side Attack-Tolerant Federated Learning
Evaluations of real-world scenarios across multiple datasets show that the proposed method enhances the robustness of federated learning against model poisoning attacks.
Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark
Companies have begun to offer Embedding as a Service (EaaS) based on these LLMs, which can benefit various natural language processing (NLP) tasks for customers.
Harnessing the Power of Text-image Contrastive Models for Automatic Detection of Online Misinformation
As growing usage of social media websites in the recent decades, the amount of news articles spreading online rapidly, resulting in an unprecedented scale of potentially fraudulent information.
An ADMM Solver for the MKL-$L_{0/1}$-SVM
We formulate the Multiple Kernel Learning (abbreviated as MKL) problem for the support vector machine with the infamous $(0, 1)$-loss function.
Attacking Important Pixels for Anchor-free Detectors
Existing adversarial attacks on object detection focus on attacking anchor-based detectors, which may not work well for anchor-free detectors.
On the Statistical Consistency of a Generalized Cepstral Estimator
We consider the problem to estimate the generalized cepstral coefficients of a stationary stochastic process or stationary multidimensional random field.
RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Control
Reinforcement learning is applied to solve actual complex tasks from high-dimensional, sensory inputs.
Text-driven Video Prediction
Specifically, appearance and motion components are provided by the image and caption separately.
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations
VISOR annotates videos from EPIC-KITCHENS, which comes with a new set of challenges not encountered in current video segmentation datasets.
Robust Quantity-Aware Aggregation for Federated Learning
Federated learning (FL) enables multiple clients to collaboratively train models without sharing their local data, and becomes an important privacy-preserving machine learning framework.
Cross-lingual Adaptation for Recipe Retrieval with Mixup
To bridge the domain gap, recipe mixup loss is proposed to enforce the intermediate domain to locate in the shortest geodesic path between source and target domains in the recipe embedding space.
Improving robustness of language models from a geometry-aware perspective
On top of FADA, we propose geometry-aware adversarial training (GAT) to perform adversarial training on friendly adversarial data so that we can save a large number of search steps.
OneLabeler: A Flexible System for Building Data Labeling Tools
In this paper, we propose a conceptual framework for data labeling and OneLabeler based on the conceptual framework to support easy building of labeling tools for diverse usage scenarios.
UA-FedRec: Untargeted Attack on Federated News Recommendation
Our study reveals a critical security issue in existing federated news recommendation systems and calls for research efforts to address the issue.
TREATED:Towards Universal Defense against Textual Adversarial Attacks
Recent work shows that deep neural networks are vulnerable to adversarial examples.
Imperceptible Adversarial Examples for Fake Image Detection
Fooling people with highly realistic fake images generated with Deepfake or GANs brings a great social disturbance to our society.
Transferable Adversarial Examples for Anchor Free Object Detection
Deep neural networks have been demonstrated to be vulnerable to adversarial attacks: subtle perturbation can completely change prediction result.
Pyramid Fusion Dark Channel Prior for Single Image Dehazing
In this paper, we propose the pyramid fusion dark channel prior (PF-DCP) for single image dehazing.
An Optimized H.266/VVC Software Decoder On Mobile Platform
As the successor of H. 265/HEVC, the new versatile video coding standard (H. 266/VVC) can provide up to 50% bitrate saving with the same subjective quality, at the cost of increased decoding complexity.
New Strong Bounds on sub-GeV Dark Matter from Boosted and Migdal Effects
Due to the low nuclear recoils, sub-GeV dark matter (DM) is usually beyond the sensitivity of the conventional DM direct detection experiments.
High Energy Physics - Phenomenology Cosmology and Nongalactic Astrophysics
Line Spectrum Representation for Vector Processes With Application to Frequency Estimation
A positive semidefinite Toeplitz matrix, which often arises as the finite covariance matrix of a stationary random process, can be decomposed as the sum of a nonnegative multiple of the identity corresponding to a white noise, and a singular term corresponding to a purely deterministic process.
CookGAN: Causality Based Text-to-Image Synthesis
Particularly, a cooking simulator sub-network is proposed to incrementally make changes to food images based on the interaction between ingredients and cooking methods over a series of steps.
CPM R-CNN: Calibrating Point-guided Misalignment in Object Detection
In object detection, offset-guided and point-guided regression dominate anchor-based and anchor-free method separately.
An Empirical Bayes Approach to Frequency Estimation
In this paper we show that the classical problem of frequency estimation can be formulated and solved efficiently in an empirical Bayesian framework by assigning a uniform a priori probability distribution to the unknown frequency.
Automatic Group Cohesiveness Detection With Multi-modal Features
The task is to predict the cohesive level for a group of people in images.
Graph Neural Networks for Image Understanding Based on Multiple Cues: Group Emotion Recognition and Event Recognition as Use Cases
A graph neural network (GNN) for image understanding based on multiple cues is proposed in this paper.
R2GAN: Cross-Modal Recipe Retrieval With Generative Adversarial Network
Representing procedure text such as recipe for crossmodal retrieval is inherently a difficult problem, not mentioning to generate image from recipe for visualization.
