Search Results for author:
Found 16 papers, 6 papers with code
EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation
Remarkably, EACO also shows the potential critical ability in open-source MLLMs, demonstrating that EACO is a viable path to boost the competence of MLLMs.
Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs
To address this problem, we propose $\texttt{Web2Code}$, a benchmark consisting of a new large-scale webpage-to-code dataset for instruction tuning and an evaluation framework for the webpage understanding and HTML code translation abilities of MLLMs.
Prototype-Based Layered Federated Cross-Modal Hashing
And due to the problems of statistical heterogeneity, model heterogeneity, and forcing each client to accept the same parameters, applying federated learning to cross-modal hash learning becomes very tricky.
Three-Stream Joint Network for Zero-Shot Sketch-Based Image Retrieval
To narrow the domain differences between sketches and images, we extract edge maps for natural images and treat them as a bridge between images and sketches, which have similar content to images and similar style to sketches.
ViT-FOD: A Vision Transformer based Fine-grained Object Discriminator
Recently, several Vision Transformer (ViT) based methods have been proposed for Fine-Grained Visual Classification (FGVC). These methods significantly surpass existing CNN-based ones, demonstrating the effectiveness of ViT in FGVC tasks. However, there are some limitations when applying ViT directly to FGVC. First, ViT needs to split images into patches and calculate the attention of every pair, which may result in heavy redundant calculation and unsatisfying performance when handling fine-grained images with complex background and small objects. Second, a standard ViT only utilizes the class token in the final layer for classification, which is not enough to extract comprehensive fine-grained information.
Semi-TCL: Semi-Supervised Track Contrastive Representation Learning
We design a new instance-to-track matching objective to learn appearance embedding that compares a candidate detection to the embedding of the tracks persisted in the tracker.
Learning Hierarchical Graph Neural Networks for Image Clustering
Our hierarchical GNN uses a novel approach to merge connected components predicted at each level of the hierarchy to form a new graph at the next level.
MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences
Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed.
Weakly-Supervised Online Hashing
However, existing hashing methods for social image retrieval are based on batch mode which violates the nature of social images, i. e., social images are usually generated periodically or collected in a stream fashion.
Graph Neural Networks for 3D Multi-Object Tracking
3D Multi-object tracking (MOT) is crucial to autonomous systems.
What Gives the Answer Away? Question Answering Bias Analysis on Video QA Datasets
In this paper, we analyze QA biases in popular video question answering datasets and discover pretrained language models can answer 37-48% questions correctly without using any multimodal context information, far exceeding the 20% random guess baseline for 5-choose-1 multiple-choice questions.
Joint Object Detection and Multi-Object Tracking with Graph Neural Networks
Despite the fact that the two components are dependent on each other, prior works often design detection and data association modules separately which are trained with separate objectives.
Ranked #1 on
Multi-Object Tracking
on 2D MOT 2015
Pixel Invisibility: Detecting Objects Invisible in Color Images
Despite recent success of object detectors using deep neural networks, their deployment on safety-critical applications such as self-driving cars remains questionable.
GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning
As a result, the feature of one object is informed of the features of other objects so that the object feature can lean towards the object with similar feature (i. e., object probably with a same ID) and deviate from objects with dissimilar features (i. e., object probably with different IDs), leading to a more discriminative feature for each object; (2) instead of obtaining the feature from either 2D or 3D space in prior work, we propose a novel joint feature extractor to learn appearance and motion features from 2D and 3D space simultaneously.
Detecting Attended Visual Targets in Video
We address the problem of detecting attention targets in video.
Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency
This paper addresses the challenging problem of estimating the general visual attention of people in images.
