Search Results for author:
Found 16 papers, 7 papers with code
VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework
With the emergence of large language models (LLMs) and vision foundation models, how to combine the intelligence and capacity of these open-sourced or API-available models to achieve open-world visual perception remains an open question.
VisionGPT-3D: A Generalized Multimodal Agent for Enhanced 3D Vision Understanding
It seamlessly integrates various SOTA vision models and brings the automation in the selection of SOTA vision models, identifies the suitable 3D mesh creation algorithms corresponding to 2D depth maps analysis, generates optimal results based on diverse multimodal inputs such as text prompts.
WorldGPT: A Sora-Inspired Video AI Agent as Rich World Models from Text and Image Inputs
Several text-to-video diffusion models have demonstrated commendable capabilities in synthesizing high-quality video content.
Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning
To alleviate CF raised by covariate shift and lexical overlap, we further propose a novel approach that ensures the identical distribution of all token embeddings during initialization and regularizes token embedding learning during training.
Improving Medical Report Generation with Adapter Tuning and Knowledge Enhancement in Vision-Language Foundation Models
Medical report generation demands automatic creation of coherent and precise descriptions for medical images.
UnifiedVisionGPT: Streamlining Vision-Oriented AI through Generalized Multimodal Framework
In the current landscape of artificial intelligence, foundation models serve as the bedrock for advancements in both language and vision domains.
MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning
To deal with the label shortage problem, we present a simple yet effective zero-shot approach MultiCapCLIP that can generate visual captions for different scenarios and languages without any labeled vision-caption pairs of downstream datasets.
Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labels
We observe that the core challenges of novel product title generation are the understanding of novel product characteristics and the generation of titles in a novel writing style.
Customizing General-Purpose Foundation Models for Medical Report Generation
In this work, we propose customizing off-the-shelf general-purpose large-scale pre-trained models, i. e., foundation models (FMs), in computer vision and natural language processing with a specific focus on medical report generation.
Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation
Automatic radiology report generation has attracted enormous research interest due to its practical value in reducing the workload of radiologists.
ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation
We present the results of extensive experiments on twelve NLG tasks, showing that, without using any labeled downstream pairs for training, ZeroNLG generates high-quality and believable outputs and significantly outperforms existing zero-shot methods.
Generating Accurate and Faithful Discharge Instructions: Task, Dataset, and Model
We build a benchmark clinical dataset and propose the Re3Writer, which imitates the working patterns of physicians to first retrieve related working experience from historical PIs written by physicians, then reason related medical knowledge.
Graph-in-Graph Network for Automatic Gene Ontology Description Generation
This task aims to automatically generate a sentence that describes the function of a GO term belonging to one of the three categories, i. e., molecular function, biological process, and cellular component.
CLIP Meets Video Captioning: Concept-Aware Representation Learning Does Matter
DCD is an auxiliary task that requires a caption model to learn the correspondence between video content and concepts and the co-occurrence relations between concepts.
Ranked #16 on
Video Captioning
on MSR-VTT
O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning
Video captioning combines video understanding and language generation.
Non-Autoregressive Coarse-to-Fine Video Captioning
However, mainstream video captioning methods suffer from slow inference speed due to the sequential manner of autoregressive decoding, and prefer generating generic descriptions due to the insufficient training of visual words (e. g., nouns and verbs) and inadequate decoding paradigm.
