Search Results for author:
Found 98 papers, 52 papers with code
Identifying Worry in Twitter: Beyond Emotion Analysis
Identifying the worries of individuals and societies plays a crucial role in providing social support and enhancing policy decision-making.
Inferring DQN structure for high-dimensional continuous control
We propose a novel approach to infer the network structure for DQN models operating with high-dimensional continuous actions.
VidHal: Benchmarking Temporal Hallucinations in Vision LLMs
Through our benchmark, we aim to inspire further research on 1) holistic understanding of VLLM capabilities, particularly regarding hallucination, and 2) extensive development of advanced VLLMs to alleviate this problem.
VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation
To further streamline our evaluation, we introduce VideoAutoBench as an auxiliary benchmark, where human annotators label winners in a subset of VideoAutoArena battles.
Joint Vision-Language Social Bias Removal for CLIP
Additionally, we advocate a new evaluation protocol that can 1) holistically quantify the model debiasing and V-L alignment ability, and 2) evaluate the generalization of social bias removal models.
SCAN: Bootstrapping Contrastive Pre-training for Data Efficiency
In particular, we individually pre-train seven CLIP models on two large-scale image-text pair datasets, and two MoCo models on the ImageNet dataset, resulting in a total of 16 pre-trained models.
The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense
The vulnerability of Vision Large Language Models (VLLMs) to jailbreak attacks appears as no surprise.
UnStar: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs
The key components of machine learning are data samples for training, model for learning patterns, and loss function for optimizing accuracy.
Strong Preferences Affect the Robustness of Value Alignment
A key component of value alignment is the modeling of human preferences as a representation of human values.
STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion Retargeting
The creation of 4D avatars (i. e., animated 3D avatars) from text description typically uses text-to-image (T2I) diffusion models to synthesize 3D avatars in the canonical space and subsequently applies animation with target motions.
Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCR
Visual Commonsense Reasoning (VCR) calls for explanatory reasoning behind question answering over visual scenes.
Multi-Modal Recommendation Unlearning
This is particularly true in case of multi-modal recommender systems (MMRS), which aim to accommodate the growing influence of multi-modal information on user preferences.
TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment
Recent advancements in image understanding have benefited from the extensive use of web image-text pairs.
Bridging the Intent Gap: Knowledge-Enhanced Visual Generation
To address these challenges, we propose a knowledge-enhanced iterative refinement framework for visual content generation.
MCM: Multi-condition Motion Synthesis Framework
Conditional human motion synthesis (HMS) aims to generate human motion sequences that conform to specific conditions.
Ranked #5 on
Motion Synthesis
on HumanML3D
Cluster-based Graph Collaborative Filtering
This model performs high-order graph convolution on cluster-specific graphs, which are constructed by capturing the multiple interests of users and identifying the common interests among them.
S3Editor: A Sparse Semantic-Disentangled Self-Training Framework for Face Video Editing
S3Editor is model-agnostic and compatible with various editing approaches.
How to Understand Named Entities: Using Common Sense for News Captioning
Our approach consists of three modules: (a) Filter Module aims to clarify the common sense concerning a named entity from two aspects: what does it mean?
EcoVal: An Efficient Data Valuation Framework for Machine Learning
In this paper, we introduce an efficient data valuation framework EcoVal, to estimate the value of data for machine learning models in a fast and practical manner.
Diffusion Facial Forgery Detection
In particular, this dataset leverages 30, 000 carefully collected textual and visual prompts, ensuring the synthesis of images with both high fidelity and semantic consistency.
Hallucination is Inevitable: An Innate Limitation of Large Language Models
Since the formal world is a part of the real world which is much more complicated, hallucinations are also inevitable for real world LLMs.
Bilateral Adaptation for Human-Object Interaction Detection with Occlusion-Robustness
Human-Object Interaction (HOI) Detection constitutes an important aspect of human-centric scene understanding which requires precise object detection and interaction recognition.
Understanding Before Recommendation: Semantic Aspect-Aware Review Exploitation via Large Language Models
Recommendation systems harness user-item interactions like clicks and reviews to learn their representations.
Attribute-driven Disentangled Representation Learning for Multimodal Recommendation
To obtain robust and independent representations for each factor associated with a specific attribute, we first disentangle the representations of features both within and across different modalities.
Generating Human-Centric Visual Cues for Human-Object Interaction Detection via Large Vision-Language Models
To capitalize on these rich Human-Centric Visual Cues, we propose a novel approach named HCVC for HOI detection.
Finetuning Text-to-Image Diffusion Models for Fairness
The rapid adoption of text-to-image diffusion models in society underscores an urgent need to address their biases.
Image-Based Virtual Try-On: A Survey
Image-based virtual try-on aims to synthesize a naturally dressed person image with a clothing image, which revolutionizes online shopping and inspires related topics within image generation, showing both research significance and commercial potential.
UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models
Teaching Visual Question Answering (VQA) models to refrain from answering unanswerable questions is necessary for building a trustworthy AI system.
PELA: Learning Parameter-Efficient Models with Low-Rank Approximation
This allows for direct and efficient utilization of the low-rank model for downstream fine-tuning tasks.
Prior-Free Continual Learning with Unlabeled Data in the Wild
Existing CL methods usually reduce forgetting with task priors, \ie using task identity or a subset of previously seen samples for model training.
AutoLoRa: A Parameter-Free Automated Robust Fine-Tuning Framework
To mitigate this issue, we propose a low-rank (LoRa) branch that disentangles RFT into two distinct components: optimizing natural objectives via the LoRa branch and adversarial objectives via the FE.
Distill to Delete: Unlearning in Graph Networks with Knowledge Distillation
Our work takes a novel approach to address these challenges in graph unlearning through knowledge distillation, as it distills to delete in GNN (D2DGN).
ELIP: Efficient Language-Image Pre-training with Fewer Vision Tokens
Learning a versatile language-image model is computationally prohibitive under a limited computing budget.
Semantic-Guided Feature Distillation for Multimodal Recommendation
The teacher model first extracts rich modality features from the generic modality feature by considering both the semantic information of items and the complementary information of multiple modalities.
DPMix: Mixture of Depth and Point Cloud Video Experts for 4D Action Segmentation
The proposed method, named Mixture of Depth and Point cloud video experts (DPMix), achieved the first place in the 4D Action Segmentation Track of the HOI4D Challenge 2023.
Sample Less, Learn More: Efficient Action Recognition via Frame Feature Restoration
Training an effective video action recognition model poses significant computational challenges, particularly under limited resource budgets.
Keyword-Aware Relative Spatio-Temporal Graph Networks for Video Question Answering
The main challenge in video question answering (VideoQA) is to capture and understand the complex spatial and temporal relations between objects based on given questions.
Towards Generalizable Deepfake Detection by Primary Region Regularization
The existing deepfake detection methods have reached a bottleneck in generalizing to unseen forgeries and manipulation approaches.
Mining Conditional Part Semantics with Occluded Extrapolation for Human-Object Interaction Detection
Human-Object Interaction Detection is a crucial aspect of human-centric scene understanding, with important applications in various domains.
A Study on Differentiable Logic and LLMs for EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition 2023
Our research focuses on the innovative application of a differentiable logic loss in the training to leverage the co-occurrence relations between verb and noun, as well as the pre-trained Large Language Models (LLMs) to generate the logic rules for the adaptation to unseen action labels.
Continual Learning with Strong Experience Replay
Experience Replay (ER) is a simple and effective rehearsal-based strategy, which optimizes the model with current training data and a subset of old samples stored in a memory buffer.
What Makes for Good Visual Tokenizers for Large Language Models?
In our benchmark, which is curated to evaluate MLLMs visual semantic understanding and fine-grained perception capabilities, we discussed different visual tokenizers pre-trained with dominant methods (i. e., DeiT, CLIP, MAE, DINO), and observe that: i) Fully/weakly supervised models capture more semantics than self-supervised models, but the gap is narrowed by scaling up the pre-training dataset.
DSFNet: Dual Space Fusion Network for Occlusion-Robust 3D Dense Face Alignment
Thanks to the proposed fusion module, our method is robust not only to occlusion and large pitch and roll view angles, which is the benefit of our image space approach, but also to noise and large yaw angles, which is the benefit of our model space method.
Ranked #1 on
3D Face Reconstruction
on AFLW2000-3D
(Mean NME metric)
Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization
To improve transferability, the existing work introduced the standard invariant regularization (SIR) to impose style-independence property to SCL, which can exempt the impact of nuisance style factors in the standard representation.
Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection
Adversarial contrastive learning (ACL) does not require expensive data annotations but outputs a robust representation that withstands adversarial attacks and also generalizes to a wide range of downstream tasks.
Learning to Agree on Vision Attention for Visual Commonsense Reasoning
Visual Commonsense Reasoning (VCR) remains a significant yet challenging research problem in the realm of visual reasoning.
Text to Point Cloud Localization with Relation-Enhanced Transformer
To overcome these two challenges, we propose a unified Relation-Enhanced Transformer (RET) to improve representation discriminability for both point cloud and natural language queries.
PointListNet: Deep Learning on 3D Point Lists
Deep neural networks on regular 1D lists (e. g., natural languages) and irregular 3D sets (e. g., point clouds) have made tremendous achievements.
Deep Regression Unlearning
In the last few years, there have been notable developments in machine unlearning to remove the information of certain training data efficiently and effectively from ML models.
Privacy-Preserving Synthetic Data Generation for Recommendation Systems
At the item level, a synthetic data generation module is proposed to generate a synthetic item corresponding to the selected item based on the user's preferences.
Chairs Can be Stood on: Overcoming Object Bias in Human-Object Interaction Detection
To quantitatively study the object bias problem, we advocate a new protocol for evaluating model performance.
Distance Matters in Human-Object Interaction Detection
2) Insufficient number of distant interactions in benchmark datasets results in under-fitting on these instances.
A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQA
On the other hand, pertaining to the implicit knowledge, the multi-modal implicit knowledge for knowledge-based VQA still remains largely unexplored.
PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences
Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension.
Ranked #3 on
3D Action Recognition
on NTU RGB+D
Effective Abstract Reasoning with Dual-Contrast Network
As a step towards improving the abstract reasoning capability of machines, we aim to solve Raven's Progressive Matrices (RPM) with neural networks, since solving RPM puzzles is highly correlated with human intelligence.
Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher
It facilitates the provision for removal of certain set or class of data from an already trained ML model without requiring retraining from scratch.
Disentangled Multimodal Representation Learning for Recommendation
However, existing methods ignore the fact that different modalities contribute differently towards a user's preference on various factors of an item.
Joint Answering and Explanation for Visual Commonsense Reasoning
Given that our framework is model-agnostic, we apply it to the existing popular baselines and validate its effectiveness on the benchmark dataset.
On Modality Bias Recognition and Reduction
From the results on four datasets regarding the above three tasks, our method yields remarkable performance improvements compared with the baselines, demonstrating its superiority on reducing the modality bias problem.
Adversarial Attack and Defense for Non-Parametric Two-Sample Tests
Furthermore, we theoretically find that the adversary can also degrade the lower bound of a TST's test power, which enables us to iteratively minimize the test criterion in order to search for adversarial pairs.
Learning to Predict Gradients for Semi-Supervised Continual Learning
To explore these issues, we formulate a new semi-supervised continual learning method, which can be generically applied to existing continual learning models.
Zero-Shot Machine Unlearning
In case of machine learning (ML) applications, this necessitates deletion of data not only from storage archives but also from ML models.
Self-Supervised Global-Local Structure Modeling for Point Cloud Domain Adaptation With Reliable Voted Pseudo Labels
In this paper, we propose an unsupervised domain adaptation method for deep point cloud representation learning.
Fast Yet Effective Machine Unlearning
In the impair step, the noise matrix along with a very high learning rate is used to induce sharp unlearning in the model.
Unsupervised Abstract Reasoning for Raven's Problem Matrices
Raven's Progressive Matrices (RPM) is highly correlated with human intelligence, and it has been widely used to measure the abstract reasoning ability of humans.
Fair Representation: Guaranteeing Approximate Multiple Group Fairness for Unknown Tasks
Motivated by scenarios where data is used for diverse prediction tasks, we study whether fair representation can be used to guarantee fairness for unknown tasks and for multiple fairness notions simultaneously.
Relation-aware Compositional Zero-shot Learning for Attribute-Object Pair Recognition
The concept module generates semantically meaningful features for primitive concepts, whereas the visual module extracts visual features for attributes and objects from input images.
Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos
To capture the dynamics in point cloud videos, point tracking is usually employed.
Ranked #4 on
3D Action Recognition
on NTU RGB+D
Understanding the Interaction of Adversarial Training with Noisy Labels
A recent adversarial training (AT) study showed that the number of projected gradient descent (PGD) steps to successfully attack a point (i. e., find an adversarial example in its proximity) is an effective measure of the robustness of this point.
Learning Causal Representation for Training Cross-Domain Pose Estimator via Generative Interventions
In this work, we take a step towards training robust models for cross-domain pose estimation task, which brings together ideas from causal representation learning and generative adversarial networks.
Unsupervised Spatial-spectral Network Learning for Hyperspectral Compressive Snapshot Reconstruction
Hyperspectral compressive imaging takes advantage of compressive sensing theory to achieve coded aperture snapshot measurement without temporal scanning, and the entire three-dimensional spatial-spectral data is captured by a two-dimensional projection during a single integration period.
Geometry-aware Instance-reweighted Adversarial Training
The belief was challenged by recent studies where we can maintain the robustness and improve the accuracy.
Using Under-trained Deep Ensembles to Learn Under Extreme Label Noise
A new model is trained with these labels to generalize reliably despite the label noise.
My Health Sensor, my Classifier: Adapting a Trained Classifier to Unlabeled End-User Data
In this work, we present an approach for unsupervised domain adaptation (DA) with the constraint, that the labeled source data are not directly available, and instead only access to a classifier trained on the source data is provided.
Learning Realistic Patterns from Unrealistic Stimuli: Generalization and Data Anonymization
We use sleep monitoring data from both an open and a large closed clinical study and evaluate whether (1) end-users can create and successfully use customized classification models for sleep apnea detection, and (2) the identity of participants in the study is protected.
Robust Federated Recommendation System
We argue that the key to Byzantine detection is monitoring of gradients of the model parameters of clients.
Hierarchically Fair Federated Learning
When the federated learning is adopted among competitive agents with siloed datasets, agents are self-interested and participate only if they are fairly rewarded.
Adversarial Learning for Personalized Tag Recommendation
We demonstrate the effectiveness of the proposed model on two different large-scale and publicly available datasets, YFCC100M and NUS-WIDE.
Evaluating Salient Object Detection in Natural Images with Multiple Objects having Multi-level Saliency
In this paper, we corroborate based on three subjective experiments on a novel image dataset that objects in natural images are inherently perceived to have varying levels of importance.
Attacks Which Do Not Kill Training Make Adversarial Learning Stronger
Adversarial training based on the minimax formulation is necessary for obtaining adversarial robustness of trained models.
Weakly-Supervised Multi-Person Action Recognition in 360$^{\circ}$ Videos
To enable research in this direction, we introduce 360Action, the first omnidirectional video dataset for multi-person action recognition.
GradMix: Multi-source Transfer across Domains and Tasks
Therefore, it is important to develop algorithms that can leverage off-the-shelf labeled dataset to learn useful knowledge for the target task.
Solving Raven's Progressive Matrices with Neural Networks
Based on the design of the pseudo target, MCPT converts the unsupervised learning problem to a supervised task.
Fast Video Object Segmentation via Mask Transfer Network
To overcome this limitation, we propose a novel mask transfer network (MTN), which can greatly boost the processing speed of VOS and also achieve a reasonable accuracy.
Explainable Video Action Reasoning via Prior Knowledge and State Transitions
Finally, by sequentially examining each state transition in the video graph, our method can detect and explain how those actions are executed with prior knowledge, just like the logical manner of thinking by humans.
User Diverse Preference Modeling by Multimodal Attentive Metric Learning
To tackle this problem, in this paper, we propose a novel Multimodal Attentive Metric Learning (MAML) method to model user diverse preferences for various items.
Augmenting Physiological Time Series Data: A Case Study for Sleep Apnea Detection
Supervised machine learning applications in the health domain often face the problem of insufficient training datasets.
Quantifying and Alleviating the Language Prior Problem in Visual Question Answering
Benefiting from the advancement of computer vision, natural language processing and information retrieval techniques, visual question answering (VQA), which aims to answer questions about an image or a video, has received lots of attentions over the past few years.
$\mathcal{G}$-softmax: Improving Intra-class Compactness and Inter-class Separability of Features
In addition, analysis of the intra-class compactness and inter-class separability demonstrates the advantages of the proposed function over the softmax function, which is consistent with the performance improvement.
Recognition of Advertisement Emotions with Application to Computational Advertising
Advertisements (ads) often contain strong affective content to capture viewer attention and convey an effective message to the audience.
Towards Robust ResNet: A Small Step but A Giant Leap
Our analytical studies reveal that the step factor h in the Euler method is able to control the robustness of ResNet in both its training and generalization.
Learning to Learn from Noisy Labeled Data
Despite the success of deep neural networks (DNNs) in image classification tasks, the human-level performance relies on massive training data with high-quality manual annotations, which are expensive and time-consuming to collect.
Ranked #26 on
Image Classification
on Clothing1M
(using extra training data)
MMALFM: Explainable Recommendation by Leveraging Reviews and Images
Then the aspect importance is integrated into a novel aspect-aware latent factor model (ALFM), which learns user's and item's latent factors based on ratings.
Unsupervised Online Video Object Segmentation with Motion Property Understanding
Moreover, our method achieves better performance than the best unsupervised offline algorithm on the DAVIS-2016 benchmark dataset.
Looking Beyond a Clever Narrative: Visual Context and Attention are Primary Drivers of Affect in Video Advertisements
Contrary to the popular notion that ad affect hinges on the narrative and the clever use of linguistic and social cues, we find that actively attended objects and the coarse scene structure better encode affective information as compared to individual scene objects or conspicuous background elements.
Attention Transfer from Web Images for Video Recognition
However, due to the domain shift problem, the performance of Web images trained deep classifiers tend to degrade when directly deployed to videos.
Multi-Camera Action Dataset for Cross-Camera Action Recognition Benchmarking
To enable the study of this problem, there exist a vast number of action datasets, which are recorded under controlled laboratory settings, real-world surveillance environments, or crawled from the Internet.
Near-Optimal Active Learning of Multi-Output Gaussian Processes
This paper addresses the problem of active learning of a multi-output Gaussian process (MOGP) model representing multiple types of coexisting correlated environmental phenomena.
