Search Results for author:
Found 79 papers, 45 papers with code
MMEKG: Multi-modal Event Knowledge Graph towards Universal Representation across Modalities
Events are fundamental building blocks of real-world happenings.
RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents
The ultimate goals of robotic learning is to acquire a comprehensive and generalizable robotic system capable of performing both seen skills within the training distribution and unseen skills in novel environments.
Assessment of Multimodal Large Language Models in Alignment with Human Values
Large Language Models (LLMs) aim to serve as versatile assistants aligned with human values, as defined by the principles of being helpful, honest, and harmless (hhh).
EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models
This paper introduces EasyJailbreak, a unified framework simplifying the construction and evaluation of jailbreak attacks against LLMs.
MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control
It is a long-lasting goal to design a generalist-embodied agent that can follow diverse instructions in human-like ways.
Endora: Video Generation Models as Endoscopy Simulators
In a nutshell, Endora marks a notable breakthrough in the deployment of generative AI for clinical endoscopy research, setting a substantial stage for further advances in medical content generation.
Exploring Safety Generalization Challenges of Large Language Models via Code
The rapid advancement of Large Language Models (LLMs) has brought about remarkable generative capabilities but also raised concerns about their potential misuse.
Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models
This research provides an initial exploration of trustworthiness modeling during LLM pre-training, seeking to unveil new insights and spur further developments in the field.
Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey
Large Language Models (LLMs) are now commonplace in conversation applications.
SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
In the rapidly evolving landscape of Large Language Models (LLMs), ensuring robust safety measures is paramount.
From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities
Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents.
PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety
In this paper, we explore these concerns through the innovative lens of agent psychology, revealing that the dark psychological states of agents constitute a significant threat to safety.
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception
It is a long-lasting goal to design an embodied system that can solve long-horizon open-world tasks in human-like ways.
Self Model for Embodied Intelligence: Modeling Full-Body Human Musculoskeletal System and Locomotion Control with Hierarchical Low-Dimensional Representation
Modeling and control of the human musculoskeletal system is important for understanding human motor functions, developing embodied intelligence, and optimizing human-robot interaction systems.
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
While this phenomenon has been overlooked in previous work, we propose a novel and extensible framework, called Octavius, for comprehensive studies and experimentation on multimodal learning with Multimodal Large Language Models (MLLMs).
ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
We will publicly release all the detailed implementations for further analysis, as well as an easy-to-use modular toolkit for the integration of new recipes and models, so that ChEF can be a growing evaluation framework for the MLLM community.
Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization
A single language model (LM), despite aligning well with an average labeler through reinforcement learning from human feedback (RLHF), may not universally suit diverse human preferences.
UniG3D: A Unified 3D Object Generation Dataset
However, the quality and diversity of existing 3D object generation methods are constrained by the inadequacies of existing 3D object datasets, including issues related to text quality, the incompleteness of multi-modal data representation encompassing 2D rendered images and 3D assets, as well as the size of the dataset.
LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark
To the best of our knowledge, we present one of the very first open-source endeavors in the field, LAMM, encompassing a Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark.
Latent Distribution Adjusting for Face Anti-Spoofing
In this work, we propose a unified framework called Latent Distribution Adjusting (LDA) with properties of latent, discriminative, adaptive, generic to improve the robustness of the FAS model by adjusting complex data distribution with multiple prototypes.
Mask Hierarchical Features For Self-Supervised Learning
We mask part of patches in the representation space and then utilize sparse visible patches to reconstruct high semantic image representation.
Siamese DETR
In this work, we present Siamese DETR, a Siamese self-supervised pretraining approach for the Transformer architecture in DETR.
Fast-BEV: A Fast and Strong Bird's-Eye View Perception Baseline
Our Fast-BEV consists of five parts, We novelly propose (1) a lightweight deployment-friendly view transformation which fast transfers 2D image feature to 3D voxel space, (2) an multi-scale image encoder which leverages multi-scale information for better performance, (3) an efficient BEV encoder which is particularly designed to speed up on-vehicle inference.
Fast-BEV: Towards Real-time On-vehicle Bird's-Eye View Perception
Recently, the pure camera-based Bird's-Eye-View (BEV) perception removes expensive Lidar sensors, making it a feasible solution for economical autonomous driving.
Parallel Reasoning Network for Human-Object Interaction Detection
Human-Object Interaction (HOI) detection aims to learn how human interacts with surrounding objects.
BEVBert: Multimodal Map Pre-training for Language-guided Navigation
Concretely, we build a local metric map to explicitly aggregate incomplete observations and remove duplicates, while modeling navigation dependency in a global topological map.
Ranked #2 on
Visual Navigation
on R2R
R$^2$F: A General Retrieval, Reading and Fusion Framework for Document-level Natural Language Inference
Document-level natural language inference (DOCNLI) is a new challenging task in natural language processing, aiming at judging the entailment relationship between a pair of hypothesis and premise documents.
PalGAN: Image Colorization with Palette Generative Adversarial Networks
Multimodal ambiguity and color bleeding remain challenging in colorization.
Towards Accurate Binary Neural Networks via Modeling Contextual Dependencies
Second, to improve the robustness of binary models with contextual dependencies, we compute the contextual dynamic embeddings to determine the binarization thresholds in general binary convolutional blocks.
Task-Balanced Distillation for Object Detection
To alleviate this problem, a novel Task-decoupled Feature Distillation (TFD) is proposed by flexibly balancing the contributions of classification and regression tasks.
Benchmarking Omni-Vision Representation through the Lens of Visual Realms
We benchmark ReCo and other advances in omni-vision representation studies that are different in architectures (from CNNs to transformers) and in learning paradigms (from supervised learning to self-supervised learning) on OmniBenchmark.
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning
This has led to a new research direction in parameter-efficient transfer learning.
Ranked #23 on
Action Recognition
on Something-Something V2
(using extra training data)
1st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)
Our model consists of three modules: the candidate waypoints predictor (CWP), the history enhanced planner and the tryout controller.
Robust Face Anti-Spoofing with Dual Probabilistic Modeling
In this work, we attempt to fill this gap by automatically addressing the noise problem from both label and data perspectives in a probabilistic manner.
ERGO: Event Relational Graph Transformer for Document-level Event Causality Identification
In this paper, we propose a novel Event Relational Graph TransfOrmer (ERGO) framework for DECI, which improves existing state-of-the-art (SOTA) methods upon two aspects.
Few-shot Forgery Detection via Guided Adversarial Interpolation
Notably, our method is also validated to be robust to choices of majority and minority forgery approaches.
X-Learner: Learning Cross Sources and Tasks for Universal Visual Representation
2) Squeeze Stage: X-Learner condenses the model to a reasonable size and learns the universal and generalizable representation for various tasks transferring.
Bamboo: Building Mega-Scale Vision Dataset Continually with Human-Machine Synergy
This work thus proposes a novel active learning framework for realistic dataset annotation.
Ranked #1 on
Image Classification
on Food-101
(using extra training data)
Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision
This is because researchers do not choose consistent training recipes and even use different data, hampering the fair comparison between different methods.
Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction
We have conducted extensive experiments on three benchmarks, including both sentence- and document-level EAE.
RePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-training
Recently, self-supervised vision transformers have attracted unprecedented attention for their impressive representation learning ability.
SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples
To alleviate feature suppression, we propose contrastive learning for unsupervised sentence embedding with soft negative samples (SNCSE).
ForgeryNet -- Face Forgery Analysis Challenge 2021: Methods and Results
The rapid progress of photorealistic synthesis techniques has reached a critical point where the boundary between real and manipulated images starts to blur.
A Simple Long-Tailed Recognition Baseline via Vision-Language Model
Recent advances in large-scale contrastive visual-language pretraining shed light on a new pathway for visual recognition.
Ranked #4 on
Long-tail Learning
on Places-LT
(using extra training data)
INTERN: A New Learning Paradigm Towards General Vision
Enormous waves of technological innovations over the past several years, marked by the advances in AI technologies, are profoundly reshaping the industry and the society.
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
Recently, large-scale Contrastive Language-Image Pre-training (CLIP) has attracted unprecedented attention for its impressive zero-shot recognition ability and excellent transferability to downstream tasks.
Few-Shot Domain Expansion for Face Anti-Spoofing
In practice, given a handful of labeled samples from a new deployment scenario (target domain) and abundant labeled face images in the existing source domain, the FAS system is expected to perform well in the new scenario without sacrificing the performance on the original domain.
ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis
To counter this emerging threat, we construct the ForgeryNet dataset, an extremely large face forgery dataset with unified annotations in image- and video-level data across four tasks: 1) Image Forgery Classification, including two-way (real / fake), three-way (real / fake with identity-replaced forgery approaches / fake with identity-remained forgery approaches), and n-way (real and 15 respective forgery approaches) classification.
CelebA-Spoof Challenge 2020 on Face Anti-Spoofing: Methods and Results
It is the largest face anti-spoofing dataset in terms of the numbers of the data and the subjects.
PV-NAS: Practical Neural Architecture Search for Video Recognition
Recently, deep learning has been utilized to solve video recognition problem due to its prominent representation ability.
Learning Connectivity of Neural Networks from a Topological Perspective
In this paper, we attempt to optimize the connectivity in neural networks.
CelebA-Spoof: Large-Scale Face Anti-Spoofing Dataset with Rich Annotations
The main reason is that current face anti-spoofing datasets are limited in both quantity and diversity.
Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues
As realistic facial manipulation technologies have achieved remarkable progress, social concerns about potential malicious abuse of these technologies bring out an emerging research topic of face forgery detection.
1st place solution for AVA-Kinetics Crossover in AcitivityNet Challenge 2020
This technical report introduces our winning solution to the spatio-temporal action localization track, AVA-Kinetics Crossover, in ActivityNet Challenge 2020.
Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization
We propose to explicitly model the Actor-Context-Actor Relation, which is the relation between two actors based on their interactions with the context.
Ranked #2 on
Action Recognition
on AVA v2.1
Morphing and Sampling Network for Dense Point Cloud Completion
3D point cloud completion, the task of inferring the complete geometric shape from a partial point cloud, has been attracting attention in the community.
Ranked #8 on
Point Cloud Completion
on ShapeNet
Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
Semantic image synthesis aims at generating photorealistic images from semantic layouts.
Diving into Optimization of Topology in Neural Networks
Seeking effective networks has become one of the most crucial and practical areas in deep learning.
CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
Text-image cross-modal retrieval is a challenging task in the field of language and vision.
Ranked #9 on
Image Retrieval
on Flickr30K 1K test
Semantics Disentangling for Text-to-Image Generation
Synthesizing photo-realistic images from text descriptions is a challenging problem.
Context and Attribute Grounded Dense Captioning
Dense captioning aims at simultaneously localizing semantic regions and describing these regions-of-interest (ROIs) with short phrases or sentences in natural language.
Ranked #3 on
Dense Captioning
on Visual Genome
Video Generation from Single Semantic Label Map
This paper proposes the novel task of video generation conditioned on a SINGLE semantic label map, which provides a good balance between flexibility and quality in the generation process.
Unsupervised Bi-directional Flow-based Video Generation from one Snapshot
Imagining multiple consecutive frames given one single snapshot is challenging, since it is difficult to simultaneously predict diverse motions from a single image and faithfully generate novel frames without visual distortions.
Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing
Referring expression grounding aims at locating certain objects or persons in an image with a referring expression, where the key challenge is to comprehend and align various types of information from visual and textual domain, such as visual attributes, location and interactions with surrounding regions.
Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection
Specifically, given the image-level annotations, (1) we first develop a weakly-supervised detection (WSD) model, and then (2) construct an end-to-end multi-label image classification framework augmented by a knowledge distillation module that guides the classification model by the WSD model according to the class-level predictions for the whole image and the object-level visual features for object RoIs.
Ranked #9 on
Multi-Label Classification
on NUS-WIDE
Transductive Centroid Projection for Semi-supervised Large-scale Recognition
It is inspired by the observation of the weights in classification layer (called extit{anchors}) converge to the central direction of each class in hyperspace.
Localization Guided Learning for Pedestrian Attribute Recognition
Pedestrian attribute recognition has attracted many attentions due to its wide applications in scene understanding and person analysis from surveillance videos.
BlockQNN: Efficient Block-wise Neural Network Architecture Generation
The block-wise generation brings unique advantages: (1) it yields state-of-the-art results in comparison to the hand-crafted networks on image classification, particularly, the best network generated by BlockQNN achieves 2. 35% top-1 error rate on CIFAR-10.
Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
We show that by encouraging deep message propagation and interactions between local object features and global predicate features, one can achieve compelling performance in recognizing complex relationships without using any linguistic priors.
Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
Zero-shot artistic style transfer is an important image synthesis problem aiming at transferring arbitrary style into content images.
Exploring Disentangled Feature Representation Beyond Face Identification
This paper proposes learning disentangled but complementary face features with minimal supervision by face identification.
Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data
The aim of image captioning is to generate captions by machine to describe image contents.
Orientation Invariant Feature Embedding and Spatial Temporal Regularization for Vehicle Re-Identification
In our vehicle ReID framework, an orientation invariant feature embedding module and a spatial-temporal regularization module are proposed.
HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis
Pedestrian analysis plays a vital role in intelligent video surveillance and is a key component for security-centric computer vision systems.
Ranked #2 on
Pedestrian Attribute Recognition
on RAP
Practical Block-wise Neural Network Architecture Generation
Convolutional neural networks have gained a remarkable success in computer vision.
Spindle Net: Person Re-Identification With Human Body Region Guided Feature Decomposition and Fusion
Person re-identification (ReID) is an important task in video surveillance and has various applications.
Slicing Convolutional Neural Network for Crowd Video Understanding
Learning and capturing both appearance and dynamic representations are pivotal for crowd video understanding.
Deeply Learned Attributes for Crowded Scene Understanding
We further measure user study performance on WWW and compare this with the proposed deep models.
