Search Results for author:
Found 17 papers, 12 papers with code
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
In this work, we develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations: (i) UI-Guided Visual Token Selection to reduce computational costs by formulating screenshots as an UI connected graph, adaptively identifying their redundant relationship and serve as the criteria for token selection during self-attention blocks; (ii) Interleaved Vision-Language-Action Streaming that flexibly unifies diverse needs within GUI tasks, enabling effective management of visual-action history in navigation or pairing multi-turn query-action sequences per screenshot to enhance training efficiency; (iii) Small-scale High-quality GUI Instruction-following Datasets by careful data curation and employing a resampling strategy to address significant data type imbalances.
Factorized Visual Tokenization and Generation
Visual tokenizers are fundamental to image generation.
One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos
We introduce VideoLISA, a video-based multimodal large language model designed to tackle the problem of language-instructed reasoning segmentation in videos.
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
We present a unified transformer, i. e., Show-o, that unifies multimodal understanding and generation.
GQE: Generalized Query Expansion for Enhanced Text-Video Retrieval
This paper introduces a novel data-centric approach, Generalized Query Expansion (GQE), to address the inherent information imbalance between text and video, enhancing the effectiveness of text-video retrieval systems.
Adaptive Slot Attention: Object Discovery with Dynamic Slot Number
Moreover, our analysis substantiates that our method exhibits the capability to dynamically adapt the slot number according to each instance's complexity, offering the potential for further exploration in slot attention research.
LOVA3: Learning to Visual Question Answering, Asking and Assessment
Our results demonstrate consistent performance gains, underscoring the critical role of these additional tasks in fostering comprehensive intelligence in MLLMs.
Ranked #125 on
Visual Question Answering
on MM-Vet
Hallucination of Multimodal Large Language Models: A Survey
By drawing the granular classification and landscapes of hallucination causes, evaluation benchmarks, and mitigation methods, this survey aims to deepen the understanding of hallucinations in MLLMs and inspire further advancements in the field.
Bring Your Own Character: A Holistic Solution for Automatic Facial Animation Generation of Customized Characters
In our solution, a deep learning model was first trained to retarget the facial expression from input face images to virtual human faces by estimating the blendshape coefficients.
Skip \n: A Simple Method to Reduce Hallucination in Large Vision-Language Models
Recent advancements in large vision-language models (LVLMs) have demonstrated impressive capability in visual information understanding with human language.
AssistGUI: Task-Oriented PC Graphical User Interface Automation
Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks thereby boosting human productivity.
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks, thereby boosting human productivity.
Unsupervised Open-Vocabulary Object Localization in Videos
In this paper, we show that recent advances in video representation learning and pre-trained vision-language models allow for substantial improvements in self-supervised video object localization.
Object-Centric Multiple Object Tracking
Unsupervised object-centric learning methods allow the partitioning of scenes into entities without additional localization information and are excellent candidates for reducing the annotation burden of multiple-object tracking (MOT) pipelines.
Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation
Have you ever looked at a painting and wondered what is the story behind it?
Unsupervised Multi-Source Domain Adaptation for Person Re-Identification
Although achieving great success, most of them only use limited data from a single-source domain for model pre-training, making the rich labeled data insufficiently exploited.
Show, Recall, and Tell: Image Captioning with Recall Mechanism
SG and RWS are de-signed for the best use of recalled words.
