Search Results for author:
Found 55 papers, 19 papers with code
Visual Prompt Tuning for Few-Shot Text Classification
In few-shot text classification, a feasible paradigm for deploying VL-PTMs is to align the input samples and their category names via the text encoders.
Leveraging Large Vision-Language Model as User Intent-aware Encoder for Composed Image Retrieval
Composed Image Retrieval (CIR) aims to retrieve target images from candidate set using a hybrid-modality query consisting of a reference image and a relative caption that describes the user intent.
Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of Experts
As the research of Multimodal Large Language Models (MLLMs) becomes popular, an advancing MLLM model is typically required to handle various textual and visual tasks (e. g., VQA, Detection, OCR, and ChartQA) simultaneously for real-world applications.
Optical Character Recognition (OCR)
Visual Question Answering (VQA)
MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents
To bridge this gap, we introduce the concept of Multimodal Role-Playing Agents (MRPAs), and propose a comprehensive framework, MMRole, for their development and evaluation, which comprises a personalized multimodal dataset and a robust evaluation method.
CoTBal: Comprehensive Task Balancing for Multi-Task Visual Instruction Tuning
To mitigate this issue, we propose a novel Comprehensive Task Balancing (CoTBal) algorithm for multi-task visual instruction tuning of LMMs.
Dual-Enhanced Coreset Selection with Class-wise Collaboration for Online Blurry Class Incremental Learning
To tackle these challenges we introduce DECO (Dual-Enhanced Coreset Selection with Class-wise Collaboration) an approach that starts by establishing a class-wise balanced memory to address data imbalances followed by a tailored class-wise gradient-based similarity scoring system for refined coreset selection strategies with reasonable score guidance to all classes.
Improvable Gap Balancing for Multi-Task Learning
In this paper, following the loss balancing framework, we propose two novel improvable gap balancing (IGB) algorithms for MTL: one takes a simple heuristic, and the other (for the first time) deploys deep reinforcement learning for MTL.
VDT: General-purpose Video Diffusion Transformers via Mask Modeling
We also propose a unified spatial-temporal mask modeling mechanism, seamlessly integrated with the model, to cater to diverse video generation scenarios.
UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling
Particularly, on the MSRVTT retrieval task, UniAdapter achieves 49. 7% recall@1 with 2. 2% model parameters, outperforming the latest competitors by 2. 0%.
TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World
Experimental results indicate that the models incorporating large language models (LLM) can generate more diverse responses, while the model utilizing knowledge graphs to introduce external knowledge performs the best overall.
Text2Poster: Laying out Stylized Texts on Retrieved Images
Poster generation is a significant task for a wide range of applications, which is often time-consuming and requires lots of manual editing and artistic experience.
LGDN: Language-Guided Denoising Network for Video-Language Modeling
However, this hypothesis often fails for two reasons: (1) With the rich semantics of video contents, it is difficult to cover all frames with a single video-level description; (2) A raw video typically has noisy/meaningless information (e. g., scenery shot, transition or teaser).
A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language
Although artificial intelligence (AI) has made significant progress in understanding molecules in a wide range of fields, existing models generally acquire the single cognitive ability from the single molecular modality.
Ranked #12 on
Molecule Captioning
on ChEBI-20
Multimodal foundation models are better simulators of the human brain
Further, from the perspective of neural encoding (based on our foundation model), we find that both visual and lingual encoders trained multimodally are more brain-like compared with unimodal ones.
COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval
Under a fair comparison setting, our COTS achieves the highest performance among all two-stream methods and comparable performance (but with 10, 800X faster in inference) w. r. t.
Ranked #24 on
Video Retrieval
on MSR-VTT
A Roadmap for Big Model
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm.
Compressed Video Contrastive Learning
To enhance the representation ability of the motion vectors, hence the effectiveness of our method, we design a cross guidance contrastive learning algorithm based on multi-instance InfoNCE loss, where motion vectors can take supervision signals from RGB frames and vice versa.
Towards artificial general intelligence via a multimodal foundation model
To overcome this limitation and take a solid step towards artificial general intelligence (AGI), we develop a foundation model pre-trained with huge multimodal data, which can be quickly adapted for various downstream cognitive tasks.
Task Relatedness-Based Generalization Bounds for Meta Learning
Supposing the $n$ training tasks and the new task are sampled from the same environment, traditional meta learning theory derives an error bound on the expected loss over the new task in terms of the empirical training loss, uniformly over the set of all hypothesis spaces.
Improved Generalization Risk Bounds for Meta-Learning with PAC-Bayes-kl Analysis
In particular, we identify that when the number of training task is large, utilizing a prior generated from an informative hyperposterior can achieve the same order of PAC-Bayes-kl bound as that obtained through setting a localized distribution-dependent prior for a novel task.
L2M-GAN: Learning To Manipulate Latent Space Semantics for Facial Attribute Editing
To overcome these limitations, we propose a novel latent space factorization model, called L2M-GAN, which is learned end-to-end and effective for editing both local and global attributes.
Pre-Trained Models: Past, Present and Future
Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI).
HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
Last, we proposed an efficient fine-grained search strategy to train HR-NAS, which effectively explores the search space, and finds optimal architectures given various tasks and computation resources.
SVT-Net: Super Light-Weight Sparse Voxel Transformer for Large Scale Place Recognition
Point cloud-based large scale place recognition is fundamental for many applications like Simultaneous Localization and Mapping (SLAM).
Ranked #2 on
3D Place Recognition
on Oxford RobotCar Dataset
Learning Versatile Neural Architectures by Propagating Network Codes
(4) Thorough studies of NCP on inter-, cross-, and intra-tasks highlight the importance of cross-task neural architecture design, i. e., multitask neural architectures and architecture transferring between different tasks.
WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
We further construct a large Chinese multi-source image-text dataset called RUC-CAS-WenLan for pre-training our BriVL model.
Ranked #1 on
Image Retrieval
on RUC-CAS-WenLan
Contrastive Prototype Learning with Augmented Embeddings for Few-Shot Learning
First, data augmentations are introduced to both the support and query sets with each sample now being represented as an augmented embedding (AE) composed of concatenated embeddings of both the original and augmented versions.
Z-Score Normalization, Hubness, and Few-Shot Learning
This means that these methods are prone to the hubness problem, that is, a certain class prototype becomes the nearest neighbor of many test instances regardless which classes they belong to.
Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw
With the constrained jigsaw puzzles, instead of solving them directly, which could still be extremely hard, we carefully design four surrogate tasks that are more solvable but meanwhile still ensure that the learned representation is sensitive to spatiotemporal continuity at both the local and global levels.
IEPT: Instance-Level and Episode-Level Pretext Tasks for Few-Shot Learning
Importantly, at the episode-level, two SSL-FSL hybrid learning objectives are devised: (1) The consistency across the predictions of an FSL classifier from different extended episodes is maximized as an episode-level pretext task.
MELR: Meta-Learning via Modeling Episode-Level Relationships for Few-Shot Learning
Most recent few-shot learning (FSL) approaches are based on episodic training whereby each episode samples few training instances (shots) per class to imitate the test condition.
Margin-Based Transfer Bounds for Meta Learning with Deep Feature Embedding
By transferring knowledge learned from seen/previous tasks, meta learning aims to generalize well to unseen/future tasks.
Counterfactual VQA: A Cause-Effect Look at Language Bias
VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language.
Domain-Adaptive Few-Shot Learning
Existing few-shot learning (FSL) methods make the implicit assumption that the few target class samples are from the same domain as the source class samples.
AdarGCN: Adaptive Aggregation GCN for Few-Shot Learning
To address this problem, we propose a graph convolutional network (GCN)-based label denoising (LDN) method to remove the irrelevant images.
Meta-Learning across Meta-Tasks for Few-Shot Learning
In this paper, we argue that the inter-meta-task relationships should be exploited and those tasks are sampled strategically to assist in meta-learning.
Few-Shot Learning as Domain Adaptation: Algorithm and Analysis
Specifically, armed with a set transformer based attention module, we construct each episode with two sub-episodes without class overlap on the seen classes to simulate the domain shift between the seen and unseen classes.
Learning Depth-Guided Convolutions for Monocular 3D Object Detection
3D object detection from a single image without LiDAR is a challenging task due to the lack of accurate depth information.
Ranked #17 on
Vehicle Pose Estimation
on KITTI Cars Hard
Every Frame Counts: Joint Learning of Video Segmentation and Optical Flow
Moreover, our framework is able to utilize both labeled and unlabeled frames in the video through joint training, while no additional calculation is required in inference.
Mobile Video Action Recognition
In addition to motion vectors, we also provide a temporal fusion method to explicitly induce the temporal context.
Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions
Specifically, our framework exploits the reciprocal relation between the referent and context, i. e., either of them influences estimation of the posterior distribution of the other, and thereby the search space of context can be greatly reduced.
Face-Focused Cross-Stream Network for Deception Detection in Videos
To address the training data scarcity problem, our FFCSN model is trained with both meta learning and adversarial learning.
Zero-Shot Learning with Sparse Attribute Propagation
The standard approach to ZSL requires a set of training images annotated with seen class labels and a semantic descriptor for seen/unseen classes (attribute vector is the most widely used).
Recursive Visual Attention in Visual Dialog
Visual dialog is a challenging vision-language task, which requires the agent to answer multi-round questions about an image.
Ranked #13 on
Visual Dialog
on VisDial v0.9 val
Zero and Few Shot Learning with Semantic Feature Synthesis and Competitive Learning
Specifically, we assume that each synthesised data point can belong to any unseen class; and the most likely two class candidates are exploited to learn a robust projection function in a competitive fashion.
Domain-Invariant Projection Learning for Zero-Shot Recognition
This is made possible by learning a projection between a feature space and a semantic space (e. g. attribute space).
Transferrable Feature and Projection Learning with Class Hierarchy for Zero-Shot Learning
Inspired by the fact that an unseen class is not exactly `unseen' if it belongs to the same superclass as a seen class, we propose a novel inductive ZSL model that leverages superclasses as the bridge between seen and unseen classes to narrow the domain gap.
RUM: network Representation learning throUgh Multi-level structural information preservation
We propose a novel network representations learning model framework called RUM (network Representation learning throUgh Multi-level structural information preservation).
Multi-Modal Multi-Scale Deep Learning for Large-Scale Image Annotation
In this paper, we address two main issues in large-scale image annotation: 1) how to learn a rich feature representation suitable for predicting a diverse set of visual concepts ranging from object, scene to abstract concept; 2) how to annotate an image with the optimal number of class labels.
Zero-Shot Fine-Grained Classification by Deep Feature Learning with Semantics
In this paper, to address the two issues, we propose a two-phase framework for recognizing images from unseen fine-grained classes, i. e. zero-shot fine-grained classification.
Pairwise Constraint Propagation: A Survey
As one of the most important types of (weaker) supervised information in machine learning and pattern recognition, pairwise constraint, which specifies whether a pair of data points occur together, has recently received significant attention, especially the problem of pairwise constraint propagation.
Image classification by visual bag-of-words refinement and reduction
This paper presents a new framework for visual bag-of-words (BOW) refinement and reduction to overcome the drawbacks associated with the visual BOW model which has been widely used for image classification.
Pairwise Constraint Propagation on Multi-View Data
This paper presents a graph-based learning approach to pairwise constraint propagation on multi-view data.
Can Image-Level Labels Replace Pixel-Level Labels for Image Parsing
By oversegmenting all the images into regions, we formulate noisily tagged image parsing as a weakly supervised sparse learning problem over all the regions, where the initial labels of each region are inferred from image-level labels.
Exhaustive and Efficient Constraint Propagation: A Semi-Supervised Learning Perspective and Its Applications
This paper presents a novel pairwise constraint propagation approach by decomposing the challenging constraint propagation problem into a set of independent semi-supervised learning subproblems which can be solved in quadratic time using label propagation based on k-nearest neighbor graphs.
