Search Results for author:
Found 70 papers, 55 papers with code
VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?
Recent studies have shown that long chain-of-thought (CoT) reasoning can significantly enhance the performance of large language models (LLMs) on complex tasks.
SpatialScore: Towards Unified Evaluation for Multimodal Spatial Understanding
Multimodal large language models (MLLMs) have achieved impressive success in question-answering tasks, yet their capabilities for spatial understanding are less explored.
Multi-Agent System for Comprehensive Soccer Understanding
Recent advancements in AI-driven soccer understanding have demonstrated rapid progress, yet existing research predominantly focuses on isolated or narrow tasks.
Kimi-VL Technical Report
We present Kimi-VL, an efficient open-source Mixture-of-Experts (MoE) vision-language model (VLM) that offers advanced multimodal reasoning, long-context understanding, and strong agent capabilities - all while activating only 2. 8B parameters in its language decoder (Kimi-VL-A3B).
Q-Adapt: Adapting LMM for Visual Quality Assessment with Progressive Instruction Tuning
To mitigate this, we propose a new paradigm for perception-oriented instruction tuning, i. e., Q-Adapt, which aims to eliminate the conflicts and achieve the synergy between these two EIQA tasks when adapting LMM, resulting in enhanced multi-faceted explanations of IQA.
SpaceR: Reinforcing MLLMs in Video Spatial Reasoning
Motivated by the success of Reinforcement Learning with Verifiable Reward (RLVR) in unlocking LLM reasoning abilities, this work aims to improve MLLMs in video spatial reasoning through the RLVR paradigm.
Image Quality Assessment: From Human to Machine Preference
Considering the huge gap between human and machine visual systems, this paper proposes the topic: Image Quality Assessment for Machine Vision for the first time.
Generative Frame Sampler for Long Video Understanding
Despite recent advances in Video Large Language Models (VideoLLMs), effectively understanding long-form videos remains a significant challenge.
Teaching LMMs for Image Quality Scoring and Interpreting
With this joint learning framework and corresponding training strategy, we develop Q-SiT, the first model capable of simultaneously performing image quality scoring and interpreting tasks, along with its lightweight variant, Q-SiT-mini.
ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks
Solving expert-level multimodal tasks is a key milestone towards general intelligence.
Q-Bench-Video: Benchmark the Video Quality Understanding of LMMs
With the rising interest in research on Large Multi-modal Models (LMMs) for video understanding, many studies have emphasized general video comprehension capabilities, neglecting the systematic exploration into video quality understanding.
MRGen: Diffusion-based Controllable Data Engine for MRI Segmentation towards Unannotated Modalities
Medical image segmentation has recently demonstrated impressive progress with deep neural networks, yet the heterogeneous modalities and scarcity of mask annotations limit the development of segmentation models on unannotated modalities.
Towards Universal Soccer Video Understanding
As a globally celebrated sport, soccer has attracted widespread interest from fans all over the world.
VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation
To further streamline our evaluation, we introduce VideoAutoBench as an auxiliary benchmark, where human annotators label winners in a subset of VideoAutoArena battles.
VQA$^2$: Visual Question Answering for Video Quality Assessment
To address this gap, we introduce the VQA2 Instruction Dataset - the first visual question answering instruction dataset that focuses on video quality assessment.
Aria: An Open Multimodal Native Mixture-of-Experts Model
Information comes in diverse modalities.
Ranked #11 on
Video Question Answering
on TVBench
R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?
However, various corruptions in the real world mean that images will not be as ideal as in simulations, presenting significant challenges for the practical application of LMMs.
Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMs
With the rising interest in research on Large Multi-modal Models (LMMs) for video understanding, many studies have emphasized general video comprehension capabilities, neglecting the systematic exploration into video quality understanding.
Explore the Hallucination on Low-level Perception for MLLMs
However, these models also exhibit hallucinations, which limit their reliability as AI systems, especially in tasks involving low-level visual perception and understanding.
LIME: Less Is More for MLLM Evaluation
However, many of these benchmarks include overly simple or uninformative samples, complicating the effective distinction of different MLLMs' performance.
MegaFusion: Extend Diffusion Models towards Higher-resolution Image Generation without Further Tuning
Diffusion models have emerged as frontrunners in text-to-image generation, but their fixed image resolution during training often leads to challenges in high-resolution image generation, such as semantic deviations and object replication.
Q-Ground: Image Quality Grounding with Large Multi-modality Models
Recent advances of large multi-modality models (LMM) have greatly improved the ability of image quality assessment (IQA) method to evaluate and explain the quality of visual content.
MMRA: A Benchmark for Evaluating Multi-Granularity and Multi-Image Relational Association Capabilities in Large Visual Language Models
Moreover, we explored the ability of LVLMs to perceive image sequences within the context of our multi-image association task.
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
In addition, our results indicate that model performance on the benchmark improves only when they are capable of processing more frames, positioning LongVideoBench as a valuable benchmark for evaluating future-generation long-context LMMs.
MatchTime: Towards Automatic Soccer Game Commentary Generation
Soccer is a globally popular sport with a vast audience, in this paper, we consider constructing an automatic soccer game commentary model to improve the audiences' viewing experience.
CMC-Bench: Towards a New Paradigm of Visual Signal Compression
With the development of Large Multimodal Models (LMMs), a Cross Modality Compression (CMC) paradigm of Image-Text-Image has emerged.
A-Bench: Are LMMs Masters at Evaluating AI-generated Images?
How to accurately and efficiently assess AI-generated images (AIGIs) remains a critical challenge for generative models.
Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare
Extensive experiments on nine IQA datasets validate that the Compare2Score effectively bridges text-defined comparative levels during training with converted single image quality score for inference, surpassing state-of-the-art IQA models across diverse scenarios.
Dual-Branch Network for Portrait Image Quality Assessment
We employ the fidelity loss to train the model via a learning-to-rank manner to mitigate inconsistencies in quality scores in the portrait image quality assessment dataset PIQ.
Ranked #1 on
Face Image Quality Assessment
on PIQ23
Enhancing Blind Video Quality Assessment with Rich Quality-aware Features
Motivated by previous researches that leverage pre-trained features extracted from various computer vision models as the feature representation for BVQA, we further explore rich quality-aware features from pre-trained blind image quality assessment (BIQA) and BVQA models as auxiliary features to help the BVQA model to handle complex distortions and diverse content of social media videos.
Light-VQA+: A Video Quality Assessment Model for Exposure Correction with Vision-Language Guidance
Equally important to the VEC is the Video Quality Assessment (VQA).
G-Refine: A General Quality Refiner for Text-to-Image Generation
Based on the mechanisms of the Human Visual System (HVS) and syntax trees, the first two indicators can respectively identify the perception and alignment deficiencies, and the last module can apply targeted quality enhancement accordingly.
LMM-PCQA: Assisting Point Cloud Quality Assessment with LMM
Although large multi-modality models (LMMs) have seen extensive exploration and application in various quality assessment studies, their integration into Point Cloud Quality Assessment (PCQA) remains unexplored.
NTIRE 2024 Quality Assessment of AI-Generated Content Challenge
A total of 196 participants have registered in the video track.
AIS 2024 Challenge on Video Quality Assessment of User-Generated Content: Methods and Results
The performance of the top-5 submissions is reviewed and provided here as a survey of diverse deep models for efficient video quality assessment of user-generated content.
NTIRE 2024 Challenge on Short-form UGC Video Quality Assessment: Methods and Results
This paper reviews the NTIRE 2024 Challenge on Shortform UGC Video Quality Assessment (S-UGC VQA), where various excellent solutions are submitted and evaluated on the collected dataset KVQ from popular short-form video platform, i. e., Kuaishou/Kwai Platform.
AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessment
With the rapid advancements in AI-Generated Content (AIGC), AI-Generated Images (AIGIs) have been widely applied in entertainment, education, and social media.
Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment
Based on T2VQA-DB, we propose a novel transformer-based model for subjective-aligned Text-to-Video Quality Assessment (T2VQA).
Towards Open-ended Visual Quality Comparison
Comparative settings (e. g. pairwise choice, listwise ranking) have been adopted by a wide range of subjective studies for image quality assessment (IQA), as it inherently standardizes the evaluation criteria across different observers and offer more clear-cut responses.
MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model
With the evolution of storage and communication protocols, ultra-low bitrate image compression has become a highly demanding topic.
Q-Bench+: A Benchmark for Multi-modal Foundation Models on Low-level Vision from Single Images to Pairs
To this end, we design benchmark settings to emulate human language responses related to low-level vision: the low-level visual perception (A1) via visual question answering related to low-level attributes (e. g. clarity, lighting); and the low-level visual description (A2), on evaluating MLLMs for low-level text descriptions.
AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception
An obvious obstacle lies in the absence of a specific benchmark to evaluate the effectiveness of MLLMs on aesthetic perception.
Towards A Better Metric for Text-to-Video Generation
Experiments on the TVGE dataset demonstrate the superiority of the proposed T2VScore on offering a better metric for text-to-video generation.
Q-Refine: A Perceptual Quality Refiner for AI-Generated Image
With the rapid evolution of the Text-to-Image (T2I) model in recent years, their unsatisfactory generation result has become a challenge.
Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models
Generative models have recently exhibited exceptional capabilities in text-to-image generation but still struggle to generate image sequences coherently.
Boosting Image Quality Assessment through Efficient Transformer Adaptation with Local Feature Enhancement
Further we propose a local distortion extractor to obtain local distortion features from the pretrained CNNs and a local distortion injector to inject the local distortion features into ViT.
Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels
The explosion of visual content available online underscores the requirement for an accurate machine assessor to robustly evaluate scores across diverse types of visual contents.
Ranked #1 on
Video Quality Assessment
on LIVE-FB LSVQ
Q-Boost: On Visual Quality Assessment Ability of Low-level Multi-Modality Foundation Models
Recent advancements in Multi-modality Large Language Models (MLLMs) have demonstrated remarkable capabilities in complex high-level vision tasks.
Exploring the Naturalness of AI-Generated Images
In this paper, we take the first step to benchmark and assess the visual naturalness of AI-generated images.
Iterative Token Evaluation and Refinement for Real-World Super-Resolution
Distortion removal involves simple HQ token prediction with LQ images, while texture generation uses a discrete diffusion model to iteratively refine the distortion removal output with a token refinement network.
Enhancing Diffusion Models with Text-Encoder Reinforcement Learning
While fine-tuning the U-Net can partially improve performance, it remains suffering from the suboptimal text encoder.
Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models
Multi-modality foundation models, as represented by GPT-4V, have brought a new paradigm for low-level visual perception and understanding tasks, that can respond to a broad range of natural human instructions in a model.
Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision
To address this gap, we present Q-Bench, a holistic benchmark crafted to systematically evaluate potential abilities of MLLMs on three realms: low-level visual perception, low-level visual description, and overall visual quality assessment.
Local Distortion Aware Efficient Transformer Adaptation for Image Quality Assessment
Further, we propose a local distortion extractor to obtain local distortion features from the pretrained CNN and a local distortion injector to inject the local distortion features into ViT.
TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessment
Our approach to IQA involves the design of a heuristic coarse-to-fine network (CFANet) that leverages multi-scale features and progressively propagates multi-level semantic information to low-level representations in a top-down manner.
Ranked #11 on
Video Quality Assessment
on MSU SR-QA Dataset
NTIRE 2023 Quality Assessment of Video Enhancement Challenge
61 participating teams submitted their prediction results during the development phase, with a total of 3168 submissions.
Advancing Zero-Shot Digital Human Quality Assessment through Text-Prompted Evaluation
To address this gap, we propose SJTU-H3D, a subjective quality assessment database specifically designed for full-body digital humans.
Boost Video Frame Interpolation via Motion Adaptation
Video frame interpolation (VFI) is a challenging task that aims to generate intermediate frames between two consecutive frames in a video.
AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment
With the rapid advancements of the text-to-image generative model, AI-generated images (AGIs) have been widely applied to entertainment, education, social media, etc.
Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models
Generative models have recently exhibited exceptional capabilities in text-to-image generation, but still struggle to generate image sequences coherently.
Towards Explainable In-the-Wild Video Quality Assessment: A Database and a Language-Prompted Approach
Though subjective studies have collected overall quality scores for these videos, how the abstract quality scores relate with specific factors is still obscure, hindering VQA methods from more concrete quality evaluations (e. g. sharpness of a video).
Towards Robust Text-Prompted Semantic Criterion for In-the-Wild Video Quality Assessment
The proliferation of videos collected during in-the-wild natural settings has pushed the development of effective Video Quality Assessment (VQA) methodologies.
Exploring Opinion-unaware Video Quality Assessment with Semantic Affinity Criterion
Recent learning-based video quality assessment (VQA) algorithms are expensive to implement due to the cost of data collection of human quality opinions, and are less robust across various scenarios due to the biases of these opinions.
Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives
In light of this, we propose the Disentangled Objective Video Quality Evaluator (DOVER) to learn the quality of UGC videos based on the two perspectives.
Ranked #1 on
Video Quality Assessment
on YouTube-UGC
Neighbourhood Representative Sampling for Efficient End-to-end Video Quality Assessment
On the other hand, existing practices, such as resizing and cropping, will change the quality of original videos due to the loss of details and contents, and are therefore harmful to quality assessment.
Ranked #2 on
Video Quality Assessment
on KoNViD-1k
(using extra training data)
Exploring the Effectiveness of Video Perceptual Representation in Blind Video Quality Assessment
Experiments show that the perceptual representation in the HVS is an effective way of predicting subjective temporal quality, and thus TPQI can, for the first time, achieve comparable performance to the spatial quality metric and be even more effective in assessing videos with large temporal variations.
FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling
Consisting of fragments and FANet, the proposed FrAgment Sample Transformer for VQA (FAST-VQA) enables efficient end-to-end deep VQA and learns effective video-quality-related representations.
Ranked #4 on
Video Quality Assessment
on LIVE-VQC
(using extra training data)
DisCoVQA: Temporal Distortion-Content Transformers for Video Quality Assessment
Based on prominent time-series modeling ability of transformers, we propose a novel and effective transformer-based VQA method to tackle these two issues.
Ranked #6 on
Video Quality Assessment
on KoNViD-1k
LAR-SR: A Local Autoregressive Model for Image Super-Resolution
Previous super-resolution (SR) approaches often formulate SR as a regression problem and pixel wise restoration, which leads to a blurry and unreal SR output.
AIM 2020 Challenge on Real Image Super-Resolution: Methods and Results
This paper introduces the real image Super-Resolution (SR) challenge that was part of the Advances in Image Manipulation (AIM) workshop, held in conjunction with ECCV 2020.
