Search Results for author:
Found 196 papers, 101 papers with code
Self-supervised Motion Representation via Scattering Local Motion Cues
Motion representation is key to many computer vision problems but has never been well studied in the literature.
Variational Bayesian Personalized Ranking
However, the training data available typically comprises implicit feedback, manifested as user clicks and purchase behaviors, rather than explicit declarations of user preferences.
Information Density Principle for MLLM Benchmarks
Therefore, we propose a critical principle of Information Density, which examines how much insight a benchmark can provide for the development of MLLMs.
Image Quality Assessment: From Human to Machine Preference
Considering the huge gap between human and machine visual systems, this paper proposes the topic: Image Quality Assessment for Machine Vision for the first time.
Teaching LMMs for Image Quality Scoring and Interpreting
With this joint learning framework and corresponding training strategy, we develop Q-SiT, the first model capable of simultaneously performing image quality scoring and interpreting tasks, along with its lightweight variant, Q-SiT-mini.
Towards All-in-One Medical Image Re-Identification
ComPA condenses medical content into a continuous modality representation and dynamically adjusts the modality-agnostic model with modality-specific parameters at runtime.
Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content
While significant progress has been made in developing objective models to assess these dimensions, the performance of such models heavily relies on the scale and quality of human annotations.
Perceptual Visual Quality Assessment: Principles, Methods, and Future Directions
Thus, perceptual visual quality assessment (PVQA), which focuses on evaluating the quality of multimedia content based on human perception, is essential for optimizing user experiences in advanced communication systems.
OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference
Recent advancements in open-source multi-modal large language models (MLLMs) have primarily focused on enhancing foundational capabilities, leaving a significant gap in human preference alignment.
Multi-Dimensional Quality Assessment for Text-to-3D Assets: Dataset and Model
This database encompasses 969 validated 3D assets generated from 170 prompts via 6 popular text-to-3D asset generation models, and corresponding subjective quality ratings for these assets from the perspectives of quality, authenticity, and text-asset correspondence, respectively.
AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment
Many video-to-audio (VTA) methods have been proposed for dubbing silent AI-generated videos.
Ultrasound-QBench: Can LLMs Aid in Quality Assessment of Ultrasound Imaging?
With the dramatic upsurge in the volume of ultrasound examinations, low-quality ultrasound imaging has gradually increased due to variations in operator proficiency and imaging circumstances, imposing a severe burden on diagnosis accuracy and even entailing the risk of restarting the diagnosis in critical cases.
Facial Attractiveness Prediction in Live Streaming: A New Benchmark and Multi-modal Method
Furthermore, a multi-modal FAP method is proposed to measure the facial attractiveness in live streaming.
HarmonyIQA: Pioneering Benchmark and Model for Image Harmonization Quality Assessment
To address the issue and facilitate the advancement of IHAs, we introduce the first Image Quality Assessment Database for image Harmony evaluation (HarmonyIQAD), which consists of 1, 350 harmonized images generated by 9 different IHAs, and the corresponding human visual preference scores.
IllusionBench: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Models
To address this gap, we introduce IllusionBench, a comprehensive visual illusion dataset that encompasses not only classic cognitive illusions but also real-world scene illusions.
Low-Light Image Enhancement via Generative Perceptual Priors
Although significant progress has been made in enhancing visibility, retrieving texture details, and mitigating noise in Low-Light (LL) images, the challenge persists in applying current Low-Light Image Enhancement (LLIE) methods to real-world scenarios, primarily due to the diverse illumination conditions encountered.
ESVQA: Perceptual Quality Assessment of Egocentric Spatial Videos
With the rapid development of eXtended Reality (XR), egocentric spatial shooting and display technologies have further enhanced immersion and engagement for users.
FineVQ: Fine-Grained User Generated Content Video Quality Assessment
The rapid growth of user-generated content (UGC) videos has produced an urgent need for effective video quality assessment (VQA) algorithms to monitor video quality and guide optimization and recommendation procedures.
Embodied Image Quality Assessment for Robotic Intelligence
Image quality assessment (IQA) of user-generated content (UGC) is a critical technique for human quality of experience (QoE).
F-Bench: Rethinking Human Preference Evaluation Metrics for Benchmarking Face Generation, Customization, and Restoration
The FaceQ database comprises 12, 255 images generated by 29 models across three tasks: (1) face generation, (2) face customization, and (3) face restoration.
VRVVC: Variable-Rate NeRF-Based Volumetric Video Compression
In this paper, we propose VRVVC, a novel end-to-end joint optimization variable-rate framework for volumetric video compression that achieves variable bitrates using a single model while maintaining superior RD performance.
Medical Manifestation-Aware De-Identification
Face de-identification (DeID) has been widely studied for common scenes, but remains under-researched for medical scenes, mostly due to the lack of large-scale patient face datasets.
Textured Mesh Saliency: Bridging Geometry and Texture for Human Perception in 3D Graphics
Textured meshes significantly enhance the realism and detail of objects by mapping intricate texture details onto the geometric structure of 3D models.
OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?
We introduce OBI-Bench, a holistic benchmark crafted to systematically evaluate large multi-modal models (LMMs) on whole-process oracle bone inscriptions (OBI) processing tasks demanding expert-level domain knowledge and deliberate cognition.
Face2QR: A Unified Framework for Aesthetic, Face-Preserving, and Scannable QR Code Generation
Existing methods to generate aesthetic QR codes, such as image and style transfer techniques, tend to compromise either the visual appeal or the scannability of QR codes when they incorporate human face identity.
AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM
The rapid advancement of large multimodal models (LMMs) has led to the rapid expansion of artificial intelligence generated videos (AIGVs), which highlights the pressing need for effective video quality assessment (VQA) models designed specifically for AIGVs.
Human-Activity AGV Quality Assessment: A Benchmark Dataset and an Objective Evaluation Metric
To address this challenge, we conduct a pioneering study on human activity AGV quality assessment, focusing on visual quality evaluation and the identification of semantic distortions.
MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysis
Artificial Intelligence (AI) has demonstrated significant capabilities in various fields, and in areas such as human-computer interaction (HCI), embodied intelligence, and the design and animation of virtual digital humans, both practitioners and users are increasingly concerned with AI's ability to understand and express emotion.
No-Reference Point Cloud Quality Assessment via Graph Convolutional Network
Finally, reasoning on the constructed graph is performed by GCN to characterize the mutual dependencies and interactions between different projected images, and aggregate feature information of multi-view projected images for final quality prediction.
VQA$^2$: Visual Question Answering for Video Quality Assessment
To address this gap, we introduce the VQA2 Instruction Dataset - the first visual question answering instruction dataset that focuses on video quality assessment.
On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detection
Large numbers of synthesized videos from diffusion models pose threats to information security and authenticity, leading to an increasing demand for generated content detection.
ResAD: A Simple Framework for Class Generalizable Anomaly Detection
In this work, we propose a simple but effective framework (called ResAD) that can be directly applied to detect anomalies in new classes.
MMHead: Towards Fine-grained Multi-modal 3D Facial Animation
To fill this gap, we first construct a large-scale multi-modal 3D facial animation dataset, MMHead, which consists of 49 hours of 3D facial motion sequences, speech audios, and rich hierarchical text annotations.
R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?
However, various corruptions in the real world mean that images will not be as ideal as in simulations, presenting significant challenges for the practical application of LMMs.
AIM 2024 Challenge on Video Super-Resolution Quality Assessment: Methods and Results
This paper presents the Video Super-Resolution (SR) Quality Assessment (QA) Challenge that was part of the Advances in Image Manipulation (AIM) workshop, held in conjunction with ECCV 2024.
Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMs
With the rising interest in research on Large Multi-modal Models (LMMs) for video understanding, many studies have emphasized general video comprehension capabilities, neglecting the systematic exploration into video quality understanding.
Subjective and Objective Quality-of-Experience Evaluation Study for Live Video Streaming
For the subjective QoE study, we introduce the first live video streaming QoE dataset, TaoLive QoE, which consists of $42$ source videos collected from real live broadcasts and $1, 155$ corresponding distorted ones degraded due to a variety of streaming distortions, including conventional streaming distortions such as compression, stalling, as well as live streaming-specific distortions like frame skipping, variable frame rate, etc.
AIM 2024 Challenge on Video Saliency Prediction: Methods and Results
The goal of the participants was to develop a method for predicting accurate saliency maps for the provided set of video sequences.
Free-VSC: Free Semantics from Visual Foundation Models for Unsupervised Video Semantic Compression
Unsupervised video semantic compression (UVSC), i. e., compressing videos to better support various analysis tasks, has recently garnered attention.
Towards Effective User Attribution for Latent Diffusion Models via Watermark-Informed Blending
Rapid advancements in multimodal large language models have enabled the creation of hyper-realistic images from textual descriptions.
Explore the Hallucination on Low-level Perception for MLLMs
However, these models also exhibit hallucinations, which limit their reliability as AI systems, especially in tasks involving low-level visual perception and understanding.
3DGCQA: A Quality Assessment Database for 3D AI-Generated Contents
To provide a valuable resource for future research and development in 3D content generation and quality assessment, the dataset has been open-sourced in https://github. com/zyj-2000/3DGCQA.
Exploring Rich Subjective Quality Information for Image Quality Assessment in the Wild
RichIQA is characterized by two key novel designs: (1) a three-stage image quality prediction network which exploits the powerful feature representation capability of the Convolutional vision Transformer (CvT) and mimics the short-term and long-term memory mechanisms of human brain; (2) a multi-label training strategy in which rich subjective quality information like MOS, SOS and DOS are concurrently used to train the quality prediction network.
Enhancing Test Time Adaptation with Few-shot Guidance
Furthermore, we propose a two-stage framework to tackle FS-TTA, including (i) fine-tuning the pre-trained source model with few-shot support set, along with using feature diversity augmentation module to avoid overfitting, (ii) implementing test time adaptation based on prototype memory bank guidance to produce high quality pseudo-label for model adaptation.
Assessing UHD Image Quality from Aesthetics, Distortions, and Saliency
To address this problem, we design a multi-branch deep neural network (DNN) to assess the quality of UHD images from three perspectives: global aesthetic characteristics, local technical distortions, and salient content perception.
LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models
Then, we design a spatiotemporal vision encoder to extract spatial and temporal features to represent the quality characteristics of videos, which are subsequently mapped into the language space by the spatiotemporal projector for modality alignment.
AIM 2024 Challenge on Compressed Video Quality Assessment: Methods and Results
The challenge aimed to evaluate the performance of VQA methods on a diverse dataset of 459 videos, encoded with 14 codecs of various compression standards (AVC/H. 264, HEVC/H. 265, AV1, and VVC/H. 266) and containing a comprehensive collection of compression artifacts.
Quality Assessment in the Era of Large Models: A Survey
Quality assessment, which evaluates the visual quality level of multimedia experiences, has garnered significant attention from researchers and has evolved substantially through dedicated efforts.
How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model
To advance the research on audio-visual saliency prediction for ODVs, we further establish a new benchmark based on the AVS-ODV database by testing numerous state-of-the-art saliency models, including visual-only models and audio-visual models.
SG-JND: Semantic-Guided Just Noticeable Distortion Predictor For Image Compression
Just noticeable distortion (JND), representing the threshold of distortion in an image that is minimally perceptible to the human visual system (HVS), is crucial for image compression algorithms to achieve a trade-off between transmission bit rate and image quality.
ESIQA: Perceptual Quality Assessment of Vision-Pro-based Egocentric Spatial Images
In this paper, we establish the Egocentric Spatial Images Quality Assessment Database (ESIQAD), the first IQA database dedicated for egocentric spatial images as far as we know.
Benchmarking Multi-dimensional AIGC Video Quality Assessment: A Dataset and Unified Model
To bridge this gap, we propose the Unify Generated Video Quality assessment (UGVQ) model, designed to accurately evaluate the multi-dimensional quality of AIGC videos.
UniProcessor: A Text-induced Unified Low-level Image Processor
In this paper, we propose a text-induced unified image processor for low-level vision tasks, termed UniProcessor, which can effectively process various degradation types and levels, and support multimodal control.
UNQA: Unified No-Reference Quality Assessment for Audio, Image, Video, and Audio-Visual Content
In this paper, we propose the Unified No-reference Quality Assessment model (UNQA) for audio, image, video, and A/V content, which tries to train a single QA model across different media modalities.
HazeCLIP: Towards Language Guided Real-World Image Dehazing
Combined with a region-specific dehazing technique and tailored prompt sets, the CLIP model accurately identifies hazy areas, providing a high-quality, human-like prior that guides the fine-tuning process of pre-trained networks.
GLARE: Low Light Image Enhancement via Generative Latent Feature based Codebook Retrieval
Most existing Low-light Image Enhancement (LLIE) methods either directly map Low-Light (LL) to Normal-Light (NL) images or use semantic or illumination maps as guides.
Ranked #2 on
Low-Light Image Enhancement
on LOLv2
DiffStega: Towards Universal Training-Free Coverless Image Steganography with Diffusion Models
Traditional image steganography focuses on concealing one image within another, aiming to avoid steganalysis by unauthorized entities.
Vision-Language Consistency Guided Multi-modal Prompt Learning for Blind AI Generated Image Quality Assessment
This is not enough for adapting CLIP models to AI generated image quality assessment (AGIQA) since AGIs visually differ from natural images.
Quality-guided Skin Tone Enhancement for Portrait Photography
In this paper, we propose a quality-guided image enhancement paradigm that enables image enhancement models to learn the distribution of images with various quality ratings.
CMC-Bench: Towards a New Paradigm of Visual Signal Compression
With the development of Large Multimodal Models (LMMs), a Cross Modality Compression (CMC) paradigm of Image-Text-Image has emerged.
DehazeDCT: Towards Effective Non-Homogeneous Dehazing via Deformable Convolutional Transformer
Image dehazing, a pivotal task in low-level vision, aims to restore the visibility and detail from hazy images.
Ranked #1 on
Nonhomogeneous Image Dehazing
on DNH-HAZE
GAIA: Rethinking Action Quality Assessment for AI-Generated Videos
Assessing action quality is both imperative and challenging due to its significant impact on the quality of AI-generated videos, further complicated by the inherently ambiguous nature of actions within AI-generated video (AIGV).
SMC++: Masked Learning of Unsupervised Video Semantic Compression
In this paper, we propose a Masked Video Modeling (MVM)-powered compression framework that particularly preserves video semantics, by jointly mining and compressing the semantics in a self-supervised manner.
A-Bench: Are LMMs Masters at Evaluating AI-generated Images?
How to accurately and efficiently assess AI-generated images (AIGIs) remains a critical challenge for generative models.
Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare
Extensive experiments on nine IQA datasets validate that the Compare2Score effectively bridges text-defined comparative levels during training with converted single image quality score for inference, surpassing state-of-the-art IQA models across diverse scenarios.
MVBIND: Self-Supervised Music Recommendation For Videos Via Embedding Space Binding
However, at present, the background music of short videos is generally chosen by the video producer, and there is a lack of automatic music recommendation methods for short videos.
Dual-Branch Network for Portrait Image Quality Assessment
We employ the fidelity loss to train the model via a learning-to-rank manner to mitigate inconsistencies in quality scores in the portrait image quality assessment dataset PIQ.
Enhancing Blind Video Quality Assessment with Rich Quality-aware Features
Motivated by previous researches that leverage pre-trained features extracted from various computer vision models as the feature representation for BVQA, we further explore rich quality-aware features from pre-trained blind image quality assessment (BIQA) and BVQA models as auxiliary features to help the BVQA model to handle complex distortions and diverse content of social media videos.
Quality Assessment for AI Generated Images with Instruction Tuning
Artificial Intelligence Generated Content (AIGC) has grown rapidly in recent years, among which AI-based image generation has gained widespread attention due to its efficient and imaginative image creation ability.
Compression-Realized Deep Structural Network for Video Quality Enhancement
This paper focuses on the task of quality enhancement for compressed videos.
Light-VQA+: A Video Quality Assessment Model for Exposure Correction with Vision-Language Guidance
Equally important to the VEC is the Video Quality Assessment (VQA).
Charting the Path Forward: CT Image Quality Assessment -- An In-Depth Review
Computed Tomography (CT) is a frequently utilized imaging technology that is employed in the clinical diagnosis of many disorders.
G-Refine: A General Quality Refiner for Text-to-Image Generation
Based on the mechanisms of the Human Visual System (HVS) and syntax trees, the first two indicators can respectively identify the perception and alignment deficiencies, and the last module can apply targeted quality enhancement accordingly.
LMM-PCQA: Assisting Point Cloud Quality Assessment with LMM
Although large multi-modality models (LMMs) have seen extensive exploration and application in various quality assessment studies, their integration into Point Cloud Quality Assessment (PCQA) remains unexplored.
Large Multi-modality Model Assisted AI-Generated Image Quality Assessment
Traditional deep neural network (DNN)-based image quality assessment (IQA) models leverage convolutional neural networks (CNN) or Transformer to learn the quality-aware feature representation, achieving commendable performance on natural scene images.
NTIRE 2024 Quality Assessment of AI-Generated Content Challenge
A total of 196 participants have registered in the video track.
AIS 2024 Challenge on Video Quality Assessment of User-Generated Content: Methods and Results
The performance of the top-5 submissions is reviewed and provided here as a survey of diverse deep models for efficient video quality assessment of user-generated content.
ShadowRefiner: Towards Mask-free Shadow Removal via Fast Fourier Transformer
Shadow-affected images often exhibit pronounced spatial discrepancies in color and illumination, consequently degrading various vision applications including object detection and segmentation systems.
Ranked #1 on
Shadow Removal
on WSRD+
NTIRE 2024 Challenge on Short-form UGC Video Quality Assessment: Methods and Results
This paper reviews the NTIRE 2024 Challenge on Shortform UGC Video Quality Assessment (S-UGC VQA), where various excellent solutions are submitted and evaluated on the collected dataset KVQ from popular short-form video platform, i. e., Kuaishou/Kwai Platform.
THQA: A Perceptual Quality Assessment Database for Talking Heads
In the realm of media technology, digital humans have gained prominence due to rapid advancements in computer technology.
How is Visual Attention Influenced by Text Guidance? Database and Model
Finally, considering the effect of text descriptions on visual attention, while most existing saliency models ignore this impact, we further propose a text-guided saliency (TGSal) prediction model, which extracts and integrates both image features and text features to predict the image saliency under various text-description conditions.
AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessment
With the rapid advancements in AI-Generated Content (AIGC), AI-Generated Images (AIGIs) have been widely applied in entertainment, education, and social media.
AIGCOIQA2024: Perceptual Quality Assessment of AI Generated Omnidirectional Images
Finally, we conduct a benchmark experiment to evaluate the performance of state-of-the-art IQA models on our database.
Deepfake Generation and Detection: A Benchmark and Survey
Deepfake is a technology dedicated to creating highly realistic facial images and videos under specific conditions, which has significant application potential in fields such as entertainment, movie production, digital human creation, to name a few.
Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment
Based on T2VQA-DB, we propose a novel transformer-based model for subjective-aligned Text-to-Video Quality Assessment (T2VQA).
When No-Reference Image Quality Models Meet MAP Estimation in Diffusion Latents
In this work, we show -- for the first time -- that NR-IQA models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement.
A Comparative Study of Perceptual Quality Metrics for Audio-driven Talking Head Videos
However, performance evaluation research lags behind the development of talking head generation techniques.
Text2QR: Harmonizing Aesthetic Customization and Scanning Robustness for Text-Guided QR Code Generation
This approach harnesses the potent generation capabilities of stable-diffusion models, navigating the trade-off between image aesthetics and QR code scannability.
MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model
With the evolution of storage and communication protocols, ultra-low bitrate image compression has become a highly demanding topic.
Resolution-Agnostic Neural Compression for High-Fidelity Portrait Video Conferencing via Implicit Radiance Fields
In this paper, we propose a novel low bandwidth neural compression approach for high-fidelity portrait video conferencing using implicit radiance fields to achieve both major objectives.
Towards Open-ended Visual Quality Comparison
Comparative settings (e. g. pairwise choice, listwise ranking) have been adopted by a wide range of subjective studies for image quality assessment (IQA), as it inherently standardizes the evaluation criteria across different observers and offer more clear-cut responses.
Q-Bench+: A Benchmark for Multi-modal Foundation Models on Low-level Vision from Single Images to Pairs
To this end, we design benchmark settings to emulate human language responses related to low-level vision: the low-level visual perception (A1) via visual question answering related to low-level attributes (e. g. clarity, lighting); and the low-level visual description (A2), on evaluating MLLMs for low-level text descriptions.
Perceptual Video Quality Assessment: A Survey
Perceptual video quality assessment plays a vital role in the field of video processing due to the existence of quality degradations introduced in various stages of video signal acquisition, compression, transmission and display.
Few-Shot Class-Incremental Learning with Prior Knowledge
To tackle the issues of catastrophic forgetting and overfitting in few-shot class-incremental learning (FSCIL), previous work has primarily concentrated on preserving the memory of old knowledge during the incremental phase.
class-incremental learning
Few-Shot Class-Incremental Learning
+1
Uncertainty-aware Sampling for Long-tailed Semi-supervised Learning
This approach allows the model to perceive the uncertainty of pseudo-labels at different training stages, thereby adaptively adjusting the selection thresholds for different classes.
AttentionLut: Attention Fusion-based Canonical Polyadic LUT for Real-time Image Enhancement
Subsequently, the attention fusion module integrates the image feature with the priori attention feature obtained during training to generate image-adaptive canonical polyadic tensors.
Q-Refine: A Perceptual Quality Refiner for AI-Generated Image
With the rapid evolution of the Text-to-Image (T2I) model in recent years, their unsatisfactory generation result has become a challenge.
Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels
The explosion of visual content available online underscores the requirement for an accurate machine assessor to robustly evaluate scores across diverse types of visual contents.
Ranked #1 on
Video Quality Assessment
on LIVE-FB LSVQ
Perceptual Quality Assessment for Video Frame Interpolation
To evaluate the quality of VFI frames without reference videos, a no-reference perceptual quality assessment method is proposed in this paper.
Q-Boost: On Visual Quality Assessment Ability of Low-level Multi-Modality Foundation Models
Recent advancements in Multi-modality Large Language Models (MLLMs) have demonstrated remarkable capabilities in complex high-level vision tasks.
Exploring the Naturalness of AI-Generated Images
In this paper, we take the first step to benchmark and assess the visual naturalness of AI-generated images.
SingingHead: A Large-scale 4D Dataset for Singing Head Animation
Singing, as a common facial movement second only to talking, can be regarded as a universal language across ethnicities and cultures, plays an important role in emotional communication, art, and entertainment.
FS-BAND: A Frequency-Sensitive Banding Detector
Banding artifact, as known as staircase-like contour, is a common quality annoyance that happens in compression, transmission, etc.
BAND-2k: Banding Artifact Noticeable Database for Banding Detection and Quality Assessment
The quality score of a banding image is generated by pooling the banding detection maps masked by the spatial frequency filters.
Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models
Multi-modality foundation models, as represented by GPT-4V, have brought a new paradigm for low-level visual perception and understanding tasks, that can respond to a broad range of natural human instructions in a model.
Audio-visual Saliency for Omnidirectional Videos
Visual saliency prediction for omnidirectional videos (ODVs) has shown great significance and necessity for omnidirectional videos to help ODV coding, ODV transmission, ODV rendering, etc..
Simple Baselines for Projection-based Full-reference and No-reference Point Cloud Quality Assessment
Point clouds are widely used in 3D content representation and have various applications in multimedia.
A No-Reference Quality Assessment Method for Digital Human Head
In this paper, we develop a novel no-reference (NR) method based on Transformer to deal with DHQA in a multi-task manner.
Geometry-Aware Video Quality Assessment for Dynamic Digital Human
Usually, DDHs are displayed as 2D rendered animation videos and it is natural to adapt video quality assessment (VQA) methods to DDH quality assessment (DDH-QA) tasks.
Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision
To address this gap, we present Q-Bench, a holistic benchmark crafted to systematically evaluate potential abilities of MLLMs on three realms: low-level visual perception, low-level visual description, and overall visual quality assessment.
Joint Gaze-Location and Gaze-Object Detection
In contrast, we reframe the gaze following detection task as detecting human head locations and their gaze followings simultaneously, aiming at jointly detect human gaze location and gaze object in a unified and single-stage pipeline.
AccFlow: Backward Accumulation for Long-Range Optical Flow
To train and evaluate the proposed AccFlow, we have constructed a large-scale high-quality dataset named CVO, which provides ground-truth optical flow labels between adjacent and distant frames.
Agglomerative Transformer for Human-Object Interaction Detection
We propose an agglomerative Transformer (AGER) that enables Transformer-based human-object interaction (HOI) detectors to flexibly exploit extra instance-level cues in a single-stage and end-to-end manner for the first time.
StableVQA: A Deep No-Reference Quality Assessment Model for Video Stability
Indeed, most existing quality assessment models evaluate video quality as a whole without specifically taking the subjective experience of video stability into consideration.
RAWIW: RAW Image Watermarking Robust to ISP Pipeline
To achieve end-to-end training of the framework, we integrate a neural network that simulates the ISP pipeline to handle the RAW-to-RGB conversion process.
Analysis of Video Quality Datasets via Design of Minimalistic Video Quality Models
By minimalistic, we restrict our family of BVQA models to build only upon basic blocks: a video preprocessor (for aggressive spatiotemporal downsampling), a spatial quality analyzer, an optional temporal quality analyzer, and a quality regressor, all with the simplest possible instantiations.
Perceptual Quality Assessment of Omnidirectional Audio-visual Signals
Omnidirectional videos (ODVs) play an increasingly important role in the application fields of medical, education, advertising, tourism, etc.
NTIRE 2023 Quality Assessment of Video Enhancement Challenge
61 participating teams submitted their prediction results during the development phase, with a total of 3168 submissions.
Blind Image Quality Assessment: A Fuzzy Neural Network for Opinion Score Distribution Prediction
On the other hand, we also prove the feasibility of the proposed method in predicting the MOS of image quality on several popular IQA databases, including CSIQ, TID2013, LIVE MD, and LIVE Challenge.
Subjective and Objective Audio-Visual Quality Assessment for User Generated Content
Then, to facilitate the development of AVQA fields, we construct a benchmark of AVQA models on the proposed SJTU-UAV database and other two AVQA databases, of which the benchmark models consist of AVQA models designed for synthetically distorted A/V sequences and AVQA models built through combining the popular VQA methods and audio features via support vector regressor (SVR).
Advancing Zero-Shot Digital Human Quality Assessment through Text-Prompted Evaluation
To address this gap, we propose SJTU-H3D, a subjective quality assessment database specifically designed for full-body digital humans.
AIGCIQA2023: A Large-scale Image Quality Assessment Database for AI Generated Images: from the Perspectives of Quality, Authenticity and Correspondence
In this paper, in order to get a better understanding of the human visual preferences for AIGIs, a large-scale IQA database for AIGC is established, which is named as AIGCIQA2023.
GMS-3DQA: Projection-based Grid Mini-patch Sampling for 3D Model Quality Assessment
Model-based 3DQA methods extract features directly from the 3D models, which are characterized by their high degree of complexity.
AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment
With the rapid advancements of the text-to-image generative model, AI-generated images (AGIs) have been widely applied to entertainment, education, social media, etc.
Light-VQA: A Multi-Dimensional Quality Assessment Model for Low-Light Video Enhancement
To this end, we first construct a Low-Light Video Enhancement Quality Assessment (LLVE-QA) dataset in which 254 original low-light videos are collected and then enhanced by leveraging 8 LLVE algorithms to obtain 2, 060 videos in total.
GANHead: Towards Generative Animatable Neural Head Avatars
To bring digital avatars into people's lives, it is highly demanded to efficiently generate complete, realistic, and animatable head avatars.
Masked Autoencoders as Image Processors
Recently, masked autoencoders (MAE) for feature pre-training have further unleashed the potential of Transformers, leading to state-of-the-art performances on various high-level vision tasks.
Ranked #4 on
Image Defocus Deblurring
on DPD (Dual-view)
Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective
We aim at advancing blind image quality assessment (BIQA), which predicts the human perception of image quality without any reference information.
MD-VQA: Multi-Dimensional Quality Assessment for UGC Live Videos
User-generated content (UGC) live videos are often bothered by various distortions during capture procedures and thus exhibit diverse visual qualities.
A Perceptual Quality Assessment Exploration for AIGC Images
\underline{AI} \underline{G}enerated \underline{C}ontent (\textbf{AIGC}) has gained widespread attention with the increasing efficiency of deep learning in content creation.
VDPVE: VQA Dataset for Perceptual Video Enhancement
Few researchers have specifically proposed a video quality assessment method for video enhancement, and there is also no comprehensive video quality assessment dataset available in public.
Subjective and Objective Quality Assessment for in-the-Wild Computer Graphics Images
Computer graphics images (CGIs) are artificially generated by means of computer programs and are widely perceived under various scenarios, such as games, streaming media, etc.
CASP-Net: Rethinking Video Saliency Prediction from an Audio-VisualConsistency Perceptual Perspective
Incorporating the audio stream enables Video Saliency Prediction (VSP) to imitate the selective attention mechanism of human brain.
Audio-Visual Quality Assessment for User Generated Content: Database and Method
Specifically, we construct the first UGC AVQA database named the SJTU-UAV database, which includes 520 in-the-wild UGC audio and video (A/V) sequences, and conduct a user study to obtain the mean opinion scores of the A/V sequences.
EEP-3DQA: Efficient and Effective Projection-based 3D Model Quality Assessment
Currently, great numbers of efforts have been put into improving the effectiveness of 3D model quality assessment (3DQA) methods.
CASP-Net: Rethinking Video Saliency Prediction From an Audio-Visual Consistency Perceptual Perspective
Incorporating the audio stream enables Video Saliency Prediction (VSP) to imitate the selective attention mechanism of human brain.
Non-Semantics Suppressed Mask Learning for Unsupervised Video Semantic Compression
Most video compression methods aim to improve the decoded video visual quality, instead of particularly guaranteeing the semantic-completeness, which deteriorates downstream video analysis tasks, e. g., action recognition.
DDH-QA: A Dynamic Digital Humans Quality Assessment Database
In recent years, large amounts of effort have been put into pushing forward the real-world application of dynamic digital human (DDH).
Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening
Stage I introduces a test time adaptation strategy, which improves the physical plausibility of synthesized human skeleton motions by optimizing skeleton joint locations.
Perceptual Attacks of No-Reference Image Quality Models with Human-in-the-Loop
No-reference image quality assessment (NR-IQA) aims to quantify how humans perceive visual distortions of digital images without access to their undistorted references.
Perceptual Quality Assessment for Digital Human Heads
Digital humans are attracting more and more research interest during the last decade, the generation, representation, rendering, and animation of which have been put into large amounts of effort.
MM-PCQA: Multi-Modal Learning for No-reference Point Cloud Quality Assessment
In specific, we split the point clouds into sub-models to represent local geometry distortions such as point shift and down-sampling.
Ranked #1 on
Point Cloud Quality Assessment
on SJTU-PCQA
Blind Quality Assessment of 3D Dense Point Clouds with Structure Guided Resampling
Objective quality assessment of 3D point clouds is essential for the development of immersive multimedia systems in real-world applications.
Evaluating Point Cloud from Moving Camera Videos: A No-Reference Metric
To tackle the challenge of point cloud quality assessment (PCQA), many PCQA methods have been proposed to evaluate the visual quality levels of point clouds by assessing the rendered static 2D projections.
Perceptual Quality Assessment of Omnidirectional Images
The original and distorted omnidirectional images, subjective quality ratings, and the head and eye movement data together constitute the OIQA database.
Enhanced Deep Animation Video Interpolation
With AutoFI and SktFI, the interpolated animation frames show high perceptual quality.
Subjective Quality Assessment for Images Generated by Computer Graphics
However, limited work has been put forward to tackle the problem of computer graphics generated images' quality assessment (CG-IQA).
Blind Surveillance Image Quality Assessment via Deep Neural Network Combined with the Visual Saliency
In this paper, we first conduct an example experiment (i. e. the face detection task) to demonstrate that the quality of the SIs has a crucial impact on the performance of the IVSS, and then propose a saliency-based deep neural network for the blind quality assessment of the SIs, which helps IVSS to filter the low-quality SIs and improve the detection and recognition performance.
A No-Reference Deep Learning Quality Assessment Method for Super-resolution Images Based on Frequency Maps
Therefore, in this paper, we propose a no-reference deep-learning image quality assessment method based on frequency maps because the artifacts caused by SISR algorithms are quite sensitive to frequency information.
Deep Neural Network for Blind Visual Quality Assessment of 4K Content
In this paper, we propose a deep learning-based BIQA model for 4K content, which on one hand can recognize true and pseudo 4K content and on the other hand can evaluate their perceptual visual quality.
A No-reference Quality Assessment Metric for Point Cloud Based on Captured Video Sequences
Point cloud is one of the most widely used digital formats of 3D models, the visual quality of which is quite sensitive to distortions such as downsampling, noise, and compression.
Perceptual Quality Assessment for Fine-Grained Compressed Images
Nowadays, the mainstream full-reference (FR) metrics are effective to predict the quality of compressed images at coarse-grained levels (the bit rates differences of compressed images are obvious), however, they may perform poorly for fine-grained compressed images whose bit rates differences are quite subtle.
Video-based Human-Object Interaction Detection from Tubelet Tokens
We present a novel vision Transformer, named TUTOR, which is able to learn tubelet tokens, served as highly-abstracted spatiotemporal representations, for video-based human-object interaction (V-HOI) detection.
Deep Decomposition and Bilinear Pooling Network for Blind Night-Time Image Quality Evaluation
The DDB-Net contains three modules, i. e., an image decomposition module, a feature encoding module, and a bilinear pooling module.
A Deep Learning based No-reference Quality Assessment Model for UGC Videos
The proposed model utilizes very sparse frames to extract spatial features and dense frames (i. e. the video chunk) with a very low spatial resolution to extract motion features, which thereby has low computational complexity.
Ranked #5 on
Video Quality Assessment
on YouTube-UGC
Saliency in Augmented Reality
Therefore, in this paper, we mainly analyze the interaction effect between background (BG) scenes and AR contents, and study the saliency prediction problem in AR.
Confusing Image Quality Assessment: Towards Better Augmented Reality Experience
An objective metric termed CFIQA is also proposed to better evaluate the confusing image quality.
End-to-End Human-Gaze-Target Detection with Transformers
In contrast, we redefine the HGT detection task as detecting human head locations and their gaze targets, simultaneously.
Iwin: Human-Object Interaction Detection via Transformer with Irregular Windows
Iwin Transformer is a hierarchical Transformer which progressively performs token representation learning and token agglomeration within irregular windows.
Parameterized Image Quality Score Distribution Prediction
Experimental results verifythe feasibility of using alpha stable model to describe the IQSD, and prove the effectiveness of objective alpha stable model basedIQSD prediction method.
A Coding Framework and Benchmark towards Low-Bitrate Video Understanding
The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved w. r. t.
DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering
Specifically, neural radiance field takes lip movements features and personalized attributes as two disentangled conditions, where lip movements are directly predicted from the audio inputs to achieve lip-synchronized generation.
Learning Invisible Markers for Hidden Codes in Offline-to-Online Photography
In addition, the automatic localization of hidden codes significantly reduces the time of manually correcting geometric distortions for photos, which is a revolutionary innovation for information hiding in mobile applications.
Frequency-Aware Physics-Inspired Degradation Model for Real-World Image Super-Resolution
In particular, we propose to use a convolutional neural network (CNN) to learn the cutoff frequency of real-world degradation process.
Unfolding Projection-free SDP Relaxation of Binary Graph Classifier via GDPA Linearization
Algorithm unfolding creates an interpretable and parsimonious neural network architecture by implementing each iteration of a model-based algorithm as a neural layer.
Blindly Assess Quality of In-the-Wild Videos via Quality-aware Pre-training and Motion Perception
The inaccessibility of reference videos with pristine quality and the complexity of authentic distortions pose great challenges for this kind of blind video quality assessment (BVQA) task.
Ranked #4 on
Video Quality Assessment
on MSU NR VQA Database
Cough Detection Using Selected Informative Features from Audio Signals
Cough is a common symptom of respiratory and lung diseases.
Task-Specific Normalization for Continual Learning of Blind Image Quality Models
In this paper, we present a simple yet effective continual learning method for blind image quality assessment (BIQA) with improved quality prediction accuracy, plasticity-stability trade-off, and task-order/-length robustness.
Angel's Girl for Blind Painters: an Efficient Painting Navigation System Validated by Multimodal Evaluation Approach
This work demonstrates that it is practicable for the blind people to feel the world through the brush in their hands.
Self-Conditioned Probabilistic Learning of Video Rescaling
After optimization, the downscaled video by our framework preserves more meaningful information, which is beneficial for both the upscaling step and the downstream tasks, e. g., video action recognition task.
EAN: Event Adaptive Network for Enhanced Action Recognition
In this paper, we propose a unified action recognition framework to investigate the dynamic nature of video content by introducing the following designs.
Ranked #15 on
Action Recognition
on Something-Something V1
No-Reference Quality Assessment for 3D Colored Point Cloud and Mesh Models
Therefore, many related studies such as point cloud quality assessment (PCQA) and mesh quality assessment (MQA) have been carried out to measure the visual quality degradations of 3D models.
Ranked #3 on
Point Cloud Quality Assessment
on WPC
Dual Attention Guided Gaze Target Detection in the Wild
In the third stage, we use the generated dual attention as guidance to perform two sub-tasks: (1) identifying whether the gaze target is inside or out of the image; (2) locating the target if inside.
Projection-free Graph-based Classifier Learning using Gershgorin Disc Perfect Alignment
We repose the SDR dual for solution $\bar{\mathbf{H}}$, then replace the PSD cone constraint $\bar{\mathbf{H}} \succeq 0$ with linear constraints derived from GDPA -- sufficient conditions to ensure $\bar{\mathbf{H}}$ is PSD -- so that the optimization becomes an LP per iteration.
Deep Learning based Full-reference and No-reference Quality Assessment Models for Compressed UGC Videos
The proposed VQA framework consists of three modules, the feature extraction module, the quality regression module, and the quality pooling module.
Ranked #12 on
Video Quality Assessment
on MSU NR VQA Database
Prediction-assistant Frame Super-Resolution for Video Streaming
An effective approach is to transmit frames in lower-quality under poor bandwidth conditions, such as using scalable video coding.
Continual Learning for Blind Image Quality Assessment
In this paper, we formulate continual learning for BIQA, where a model learns continually from a stream of IQA datasets, building on what was learned from previously seen data.
Looking Here or There? Gaze Following in 360-Degree Images
Specifically, the local region is obtained as a 2D cone-shaped field along the 2D projection of the sight line starting at the human subject's head position, and the distant region is obtained by searching along the sight line in 3D sphere space.
Identification of deep breath while moving forward based on multiple body regions and graph signal analysis
This paper presents an unobtrusive solution that can automatically identify deep breath when a person is walking past the global depth camera.
Face Mask Assistant: Detection of Face Mask Service Stage Based on Mobile Phone
Aimed at solving the problem that we don't know which service stage of the mask belongs to, we propose a detection system based on the mobile phone.
Perceptron Synthesis Network: Rethinking the Action Scale Variances in Videos
More specifically, an \textit{action perceptron synthesizer} is proposed to generate the kernels from a bag of fixed-size kernels that are interacted by dense routing paths.
Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild
Nevertheless, due to the distributional shift between images simulated in the laboratory and captured in the wild, models trained on databases with synthetic distortions remain particularly weak at handling realistic distortions (and vice versa).
Permutation Matters: Anisotropic Convolutional Layer for Learning on Point Clouds
Various point neural networks have been developed with isotropic filters or using weighting matrices to overcome the structure inconsistency on point clouds.
Learning Local Neighboring Structure for Robust 3D Shape Representation
Mesh is a powerful data structure for 3D shapes.
Combining Visible Light and Infrared Imaging for Efficient Detection of Respiratory Infections such as COVID-19 on Portable Device
In this work, we perform the health screening through the combination of the RGB and thermal videos obtained from the dual-mode camera and deep learning architecture. We first accomplish a respiratory data capture technique for people wearing masks by using face recognition.
Blurry Video Frame Interpolation
Existing works reduce motion blur and up-convert frame rate through two separate ways, including frame deblurring and frame interpolation.
Abnormal respiratory patterns classifier may contribute to large-scale screening of people infected with COVID-19 in an accurate and unobtrusive manner
During the epidemic prevention and control period, our study can be helpful in prognosis, diagnosis and screening for the patients infected with COVID-19 (the novel coronavirus) based on breathing characteristics.
Toward Better Understanding of Saliency Prediction in Augmented 360 Degree Videos
The saliency annotations of head and eye movements for both original and augmented videos are collected and together constitute the ARVR dataset.
Robust Invisible Hyperlinks in Physical Photographs Based on 3D Rendering Attacks
To train a robust decoder against the physical distortion from the real world, a distortion network based on 3D rendering is inserted between the encoder and the decoder to simulate the camera imaging process.
A New Ensemble Adversarial Attack Powered by Long-term Gradient Memories
Deep neural networks are vulnerable to adversarial attacks.
Semi-supervised 3D Face Reconstruction with Nonlinear Disentangled Representations
Moreover, the identity and expression representations are entangled in these models, which hurdles many facial editing applications.
Learning to Blindly Assess Image Quality in the Laboratory and Wild
Computational models for blind image quality assessment (BIQA) are typically trained in well-controlled laboratory environments with limited generalizability to realistically distorted images.
How is Gaze Influenced by Image Transformations? Dataset and Model
Data size is the bottleneck for developing deep saliency models, because collecting eye-movement data is very time consuming and expensive.
Adversarial Attacks against Deep Saliency Models
The proposed attack only requires a part of the model information, and is able to generate a sparser and more insidious adversarial perturbation, compared to traditional image-space attacks.
Invariance Analysis of Saliency Models versus Human Gaze During Scene Free Viewing
Most of current studies on human gaze and saliency modeling have used high-quality stimuli.
Terahertz Security Image Quality Assessment by No-reference Model Observers
Subsequently, the existing no-reference IQA algorithms, which were 5 opinion-aware approaches viz., NFERM, GMLF, DIIVINE, BRISQUE and BLIINDS2, and 8 opinion-unaware approaches viz., QAC, SISBLIM, NIQE, FISBLIM, CPBD, S3 and Fish_bb, were executed for the evaluation of the THz security image quality.
