Search Results for author:
Found 91 papers, 41 papers with code
Self-supervised Motion Representation via Scattering Local Motion Cues
Motion representation is key to many computer vision problems but has never been well studied in the literature.
Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models
Multi-modality foundation models, as represented by GPT-4V, have brought a new paradigm for low-level visual perception and understanding tasks, that can respond to a broad range of natural human instructions in a model.
Audio-visual Saliency for Omnidirectional Videos
Visual saliency prediction for omnidirectional videos (ODVs) has shown great significance and necessity for omnidirectional videos to help ODV coding, ODV transmission, ODV rendering, etc..
Simple Baselines for Projection-based Full-reference and No-reference Point Cloud Quality Assessment
Point clouds are widely used in 3D content representation and have various applications in multimedia.
A No-Reference Quality Assessment Method for Digital Human Head
In this paper, we develop a novel no-reference (NR) method based on Transformer to deal with DHQA in a multi-task manner.
Geometry-Aware Video Quality Assessment for Dynamic Digital Human
Usually, DDHs are displayed as 2D rendered animation videos and it is natural to adapt video quality assessment (VQA) methods to DDH quality assessment (DDH-QA) tasks.
Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision
To address this gap, we present Q-Bench, a holistic benchmark crafted to systematically evaluate potential abilities of MLLMs on three realms: low-level visual perception, low-level visual description, and overall visual quality assessment.
Joint Gaze-Location and Gaze-Object Detection
In contrast, we reframe the gaze following detection task as detecting human head locations and their gaze followings simultaneously, aiming at jointly detect human gaze location and gaze object in a unified and single-stage pipeline.
AccFlow: Backward Accumulation for Long-Range Optical Flow
To train and evaluate the proposed AccFlow, we have constructed a large-scale high-quality dataset named CVO, which provides ground-truth optical flow labels between adjacent and distant frames.
Agglomerative Transformer for Human-Object Interaction Detection
We propose an agglomerative Transformer (AGER) that enables Transformer-based human-object interaction (HOI) detectors to flexibly exploit extra instance-level cues in a single-stage and end-to-end manner for the first time.
StableVQA: A Deep No-Reference Quality Assessment Model for Video Stability
Indeed, most existing quality assessment models evaluate video quality as a whole without specifically taking the subjective experience of video stability into consideration.
RAWIW: RAW Image Watermarking Robust to ISP Pipeline
To achieve end-to-end training of the framework, we integrate a neural network that simulates the ISP pipeline to handle the RAW-to-RGB conversion process.
Analysis of Video Quality Datasets via Design of Minimalistic Video Quality Models
By minimalistic, we restrict our family of BVQA models to build only upon basic blocks: a video preprocessor (for aggressive spatiotemporal downsampling), a spatial quality analyzer, an optional temporal quality analyzer, and a quality regressor, all with the simplest possible instantiations.
Perceptual Quality Assessment of Omnidirectional Audio-visual Signals
Omnidirectional videos (ODVs) play an increasingly important role in the application fields of medical, education, advertising, tourism, etc.
NTIRE 2023 Quality Assessment of Video Enhancement Challenge
61 participating teams submitted their prediction results during the development phase, with a total of 3168 submissions.
Advancing Zero-Shot Digital Human Quality Assessment through Text-Prompted Evaluation
To address this gap, we propose SJTU-H3D, a subjective quality assessment database specifically designed for full-body digital humans.
AIGCIQA2023: A Large-scale Image Quality Assessment Database for AI Generated Images: from the Perspectives of Quality, Authenticity and Correspondence
In this paper, in order to get a better understanding of the human visual preferences for AIGIs, a large-scale IQA database for AIGC is established, which is named as AIGCIQA2023.
GMS-3DQA: Projection-based Grid Mini-patch Sampling for 3D Model Quality Assessment
Model-based 3DQA methods extract features directly from the 3D models, which are characterized by their high degree of complexity.
AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment
With the rapid advancements of the text-to-image generative model, AI-generated images (AGIs) have been widely applied to entertainment, education, social media, etc.
Light-VQA: A Multi-Dimensional Quality Assessment Model for Low-Light Video Enhancement
To this end, we first construct a Low-Light Video Enhancement Quality Assessment (LLVE-QA) dataset in which 254 original low-light videos are collected and then enhanced by leveraging 8 LLVE algorithms to obtain 2, 060 videos in total.
GANHead: Towards Generative Animatable Neural Head Avatars
To bring digital avatars into people's lives, it is highly demanded to efficiently generate complete, realistic, and animatable head avatars.
Masked Autoencoders as Image Processors
Recently, masked autoencoders (MAE) for feature pre-training have further unleashed the potential of Transformers, leading to state-of-the-art performances on various high-level vision tasks.
Ranked #4 on
Image Defocus Deblurring
on DPD (Dual-view)
Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective
We aim at advancing blind image quality assessment (BIQA), which predicts the human perception of image quality without any reference information.
MD-VQA: Multi-Dimensional Quality Assessment for UGC Live Videos
User-generated content (UGC) live videos are often bothered by various distortions during capture procedures and thus exhibit diverse visual qualities.
A Perceptual Quality Assessment Exploration for AIGC Images
\underline{AI} \underline{G}enerated \underline{C}ontent (\textbf{AIGC}) has gained widespread attention with the increasing efficiency of deep learning in content creation.
VDPVE: VQA Dataset for Perceptual Video Enhancement
Few researchers have specifically proposed a video quality assessment method for video enhancement, and there is also no comprehensive video quality assessment dataset available in public.
Subjective and Objective Quality Assessment for in-the-Wild Computer Graphics Images
Computer graphics images (CGIs) are artificially generated by means of computer programs and are widely perceived under various scenarios, such as games, streaming media, etc.
CASP-Net: Rethinking Video Saliency Prediction from an Audio-VisualConsistency Perceptual Perspective
Incorporating the audio stream enables Video Saliency Prediction (VSP) to imitate the selective attention mechanism of human brain.
Audio-Visual Quality Assessment for User Generated Content: Database and Method
Specifically, we construct the first UGC AVQA database named the SJTU-UAV database, which includes 520 in-the-wild UGC audio and video (A/V) sequences, and conduct a user study to obtain the mean opinion scores of the A/V sequences.
EEP-3DQA: Efficient and Effective Projection-based 3D Model Quality Assessment
Currently, great numbers of efforts have been put into improving the effectiveness of 3D model quality assessment (3DQA) methods.
CASP-Net: Rethinking Video Saliency Prediction From an Audio-Visual Consistency Perceptual Perspective
Incorporating the audio stream enables Video Saliency Prediction (VSP) to imitate the selective attention mechanism of human brain.
Non-Semantics Suppressed Mask Learning for Unsupervised Video Semantic Compression
Most video compression methods aim to improve the decoded video visual quality, instead of particularly guaranteeing the semantic-completeness, which deteriorates downstream video analysis tasks, e. g., action recognition.
DDH-QA: A Dynamic Digital Humans Quality Assessment Database
In recent years, large amounts of effort have been put into pushing forward the real-world application of dynamic digital human (DDH).
Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening
Stage I introduces a test time adaptation strategy, which improves the physical plausibility of synthesized human skeleton motions by optimizing skeleton joint locations.
Perceptual Attacks of No-Reference Image Quality Models with Human-in-the-Loop
No-reference image quality assessment (NR-IQA) aims to quantify how humans perceive visual distortions of digital images without access to their undistorted references.
Perceptual Quality Assessment for Digital Human Heads
Digital humans are attracting more and more research interest during the last decade, the generation, representation, rendering, and animation of which have been put into large amounts of effort.
MM-PCQA: Multi-Modal Learning for No-reference Point Cloud Quality Assessment
In specific, we split the point clouds into sub-models to represent local geometry distortions such as point shift and down-sampling.
Ranked #1 on
Point Cloud Quality Assessment
on SJTU-PCQA
Blind Quality Assessment of 3D Dense Point Clouds with Structure Guided Resampling
Objective quality assessment of 3D point clouds is essential for the development of immersive multimedia systems in real-world applications.
Treating Point Cloud as Moving Camera Videos: A No-Reference Quality Assessment Metric
Point cloud is one of the most widely used digital representation formats for three-dimensional (3D) contents, the visual quality of which may suffer from noise and geometric shift distortions during the production procedure as well as compression and downsampling distortions during the transmission process.
Perceptual Quality Assessment of Omnidirectional Images
The original and distorted omnidirectional images, subjective quality ratings, and the head and eye movement data together constitute the OIQA database.
Enhanced Deep Animation Video Interpolation
With AutoFI and SktFI, the interpolated animation frames show high perceptual quality.
Subjective Quality Assessment for Images Generated by Computer Graphics
However, limited work has been put forward to tackle the problem of computer graphics generated images' quality assessment (CG-IQA).
A No-reference Quality Assessment Metric for Point Cloud Based on Captured Video Sequences
Point cloud is one of the most widely used digital formats of 3D models, the visual quality of which is quite sensitive to distortions such as downsampling, noise, and compression.
Blind Surveillance Image Quality Assessment via Deep Neural Network Combined with the Visual Saliency
In this paper, we first conduct an example experiment (i. e. the face detection task) to demonstrate that the quality of the SIs has a crucial impact on the performance of the IVSS, and then propose a saliency-based deep neural network for the blind quality assessment of the SIs, which helps IVSS to filter the low-quality SIs and improve the detection and recognition performance.
A No-Reference Deep Learning Quality Assessment Method for Super-resolution Images Based on Frequency Maps
Therefore, in this paper, we propose a no-reference deep-learning image quality assessment method based on frequency maps because the artifacts caused by SISR algorithms are quite sensitive to frequency information.
Deep Neural Network for Blind Visual Quality Assessment of 4K Content
In this paper, we propose a deep learning-based BIQA model for 4K content, which on one hand can recognize true and pseudo 4K content and on the other hand can evaluate their perceptual visual quality.
Perceptual Quality Assessment for Fine-Grained Compressed Images
Nowadays, the mainstream full-reference (FR) metrics are effective to predict the quality of compressed images at coarse-grained levels (the bit rates differences of compressed images are obvious), however, they may perform poorly for fine-grained compressed images whose bit rates differences are quite subtle.
Video-based Human-Object Interaction Detection from Tubelet Tokens
We present a novel vision Transformer, named TUTOR, which is able to learn tubelet tokens, served as highly-abstracted spatiotemporal representations, for video-based human-object interaction (V-HOI) detection.
Deep Decomposition and Bilinear Pooling Network for Blind Night-Time Image Quality Evaluation
The DDB-Net contains three modules, i. e., an image decomposition module, a feature encoding module, and a bilinear pooling module.
A Deep Learning based No-reference Quality Assessment Model for UGC Videos
The proposed model utilizes very sparse frames to extract spatial features and dense frames (i. e. the video chunk) with a very low spatial resolution to extract motion features, which thereby has low computational complexity.
Ranked #4 on
Video Quality Assessment
on YouTube-UGC
Saliency in Augmented Reality
Therefore, in this paper, we mainly analyze the interaction effect between background (BG) scenes and AR contents, and study the saliency prediction problem in AR.
Confusing Image Quality Assessment: Towards Better Augmented Reality Experience
An objective metric termed CFIQA is also proposed to better evaluate the confusing image quality.
End-to-End Human-Gaze-Target Detection with Transformers
In contrast, we redefine the HGT detection task as detecting human head locations and their gaze targets, simultaneously.
Iwin: Human-Object Interaction Detection via Transformer with Irregular Windows
Iwin Transformer is a hierarchical Transformer which progressively performs token representation learning and token agglomeration within irregular windows.
Parameterized Image Quality Score Distribution Prediction
Experimental results verifythe feasibility of using alpha stable model to describe the IQSD, and prove the effectiveness of objective alpha stable model basedIQSD prediction method.
A Coding Framework and Benchmark towards Compressed Video Understanding
However, in real-world scenarios, the videos are first compressed before the transportation and then decompressed for understanding.
DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering
Specifically, neural radiance field takes lip movements features and personalized attributes as two disentangled conditions, where lip movements are directly predicted from the audio inputs to achieve lip-synchronized generation.
Learning Invisible Markers for Hidden Codes in Offline-to-Online Photography
In addition, the automatic localization of hidden codes significantly reduces the time of manually correcting geometric distortions for photos, which is a revolutionary innovation for information hiding in mobile applications.
Frequency-Aware Physics-Inspired Degradation Model for Real-World Image Super-Resolution
In particular, we propose to use a convolutional neural network (CNN) to learn the cutoff frequency of real-world degradation process.
Unfolding Projection-free SDP Relaxation of Binary Graph Classifier via GDPA Linearization
Algorithm unfolding creates an interpretable and parsimonious neural network architecture by implementing each iteration of a model-based algorithm as a neural layer.
Blindly Assess Quality of In-the-Wild Videos via Quality-aware Pre-training and Motion Perception
The inaccessibility of reference videos with pristine quality and the complexity of authentic distortions pose great challenges for this kind of blind video quality assessment (BVQA) task.
Ranked #4 on
Video Quality Assessment
on MSU NR VQA Database
Cough Detection Using Selected Informative Features from Audio Signals
Cough is a common symptom of respiratory and lung diseases.
Task-Specific Normalization for Continual Learning of Blind Image Quality Models
In this paper, we present a simple yet effective continual learning method for BIQA with improved quality prediction accuracy, plasticity-stability trade-off, and task-order/-length robustness.
Angel's Girl for Blind Painters: an Efficient Painting Navigation System Validated by Multimodal Evaluation Approach
This work demonstrates that it is practicable for the blind people to feel the world through the brush in their hands.
Self-Conditioned Probabilistic Learning of Video Rescaling
After optimization, the downscaled video by our framework preserves more meaningful information, which is beneficial for both the upscaling step and the downstream tasks, e. g., video action recognition task.
EAN: Event Adaptive Network for Enhanced Action Recognition
In this paper, we propose a unified action recognition framework to investigate the dynamic nature of video content by introducing the following designs.
Ranked #13 on
Action Recognition
on Something-Something V1
No-Reference Quality Assessment for 3D Colored Point Cloud and Mesh Models
Therefore, many related studies such as point cloud quality assessment (PCQA) and mesh quality assessment (MQA) have been carried out to measure the visual quality degradations of 3D models.
Ranked #3 on
Point Cloud Quality Assessment
on WPC
Dual Attention Guided Gaze Target Detection in the Wild
In the third stage, we use the generated dual attention as guidance to perform two sub-tasks: (1) identifying whether the gaze target is inside or out of the image; (2) locating the target if inside.
Projection-free Graph-based Classifier Learning using Gershgorin Disc Perfect Alignment
We repose the SDR dual for solution $\bar{\mathbf{H}}$, then replace the PSD cone constraint $\bar{\mathbf{H}} \succeq 0$ with linear constraints derived from GDPA -- sufficient conditions to ensure $\bar{\mathbf{H}}$ is PSD -- so that the optimization becomes an LP per iteration.
Deep Learning based Full-reference and No-reference Quality Assessment Models for Compressed UGC Videos
The proposed VQA framework consists of three modules, the feature extraction module, the quality regression module, and the quality pooling module.
Ranked #12 on
Video Quality Assessment
on MSU NR VQA Database
Prediction-assistant Frame Super-Resolution for Video Streaming
An effective approach is to transmit frames in lower-quality under poor bandwidth conditions, such as using scalable video coding.
Continual Learning for Blind Image Quality Assessment
In this paper, we formulate continual learning for BIQA, where a model learns continually from a stream of IQA datasets, building on what was learned from previously seen data.
Looking Here or There? Gaze Following in 360-Degree Images
Specifically, the local region is obtained as a 2D cone-shaped field along the 2D projection of the sight line starting at the human subject's head position, and the distant region is obtained by searching along the sight line in 3D sphere space.
Identification of deep breath while moving forward based on multiple body regions and graph signal analysis
This paper presents an unobtrusive solution that can automatically identify deep breath when a person is walking past the global depth camera.
Face Mask Assistant: Detection of Face Mask Service Stage Based on Mobile Phone
Aimed at solving the problem that we don't know which service stage of the mask belongs to, we propose a detection system based on the mobile phone.
Perceptron Synthesis Network: Rethinking the Action Scale Variances in Videos
More specifically, an \textit{action perceptron synthesizer} is proposed to generate the kernels from a bag of fixed-size kernels that are interacted by dense routing paths.
Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild
Nevertheless, due to the distributional shift between images simulated in the laboratory and captured in the wild, models trained on databases with synthetic distortions remain particularly weak at handling realistic distortions (and vice versa).
Permutation Matters: Anisotropic Convolutional Layer for Learning on Point Clouds
Various point neural networks have been developed with isotropic filters or using weighting matrices to overcome the structure inconsistency on point clouds.
Learning Local Neighboring Structure for Robust 3D Shape Representation
Mesh is a powerful data structure for 3D shapes.
Combining Visible Light and Infrared Imaging for Efficient Detection of Respiratory Infections such as COVID-19 on Portable Device
In this work, we perform the health screening through the combination of the RGB and thermal videos obtained from the dual-mode camera and deep learning architecture. We first accomplish a respiratory data capture technique for people wearing masks by using face recognition.
Blurry Video Frame Interpolation
Existing works reduce motion blur and up-convert frame rate through two separate ways, including frame deblurring and frame interpolation.
Abnormal respiratory patterns classifier may contribute to large-scale screening of people infected with COVID-19 in an accurate and unobtrusive manner
During the epidemic prevention and control period, our study can be helpful in prognosis, diagnosis and screening for the patients infected with COVID-19 (the novel coronavirus) based on breathing characteristics.
Toward Better Understanding of Saliency Prediction in Augmented 360 Degree Videos
The saliency annotations of head and eye movements for both original and augmented videos are collected and together constitute the ARVR dataset.
Robust Invisible Hyperlinks in Physical Photographs Based on 3D Rendering Attacks
To train a robust decoder against the physical distortion from the real world, a distortion network based on 3D rendering is inserted between the encoder and the decoder to simulate the camera imaging process.
A New Ensemble Adversarial Attack Powered by Long-term Gradient Memories
Deep neural networks are vulnerable to adversarial attacks.
Semi-supervised 3D Face Reconstruction with Nonlinear Disentangled Representations
Moreover, the identity and expression representations are entangled in these models, which hurdles many facial editing applications.
Learning to Blindly Assess Image Quality in the Laboratory and Wild
Computational models for blind image quality assessment (BIQA) are typically trained in well-controlled laboratory environments with limited generalizability to realistically distorted images.
How is Gaze Influenced by Image Transformations? Dataset and Model
Data size is the bottleneck for developing deep saliency models, because collecting eye-movement data is very time consuming and expensive.
Adversarial Attacks against Deep Saliency Models
The proposed attack only requires a part of the model information, and is able to generate a sparser and more insidious adversarial perturbation, compared to traditional image-space attacks.
Invariance Analysis of Saliency Models versus Human Gaze During Scene Free Viewing
Most of current studies on human gaze and saliency modeling have used high-quality stimuli.
Terahertz Security Image Quality Assessment by No-reference Model Observers
Subsequently, the existing no-reference IQA algorithms, which were 5 opinion-aware approaches viz., NFERM, GMLF, DIIVINE, BRISQUE and BLIINDS2, and 8 opinion-unaware approaches viz., QAC, SISBLIM, NIQE, FISBLIM, CPBD, S3 and Fish_bb, were executed for the evaluation of the THz security image quality.
