Search Results for author:
Found 99 papers, 49 papers with code
BBS-Net: RGB-D Salient Object Detection with a Bifurcated Backbone Strategy Network
In particular, we 1) propose a bifurcated backbone strategy (BBS) to split the multi-level features into teacher and student features, and 2) utilize a depth-enhanced module (DEM) to excavate informative parts of depth cues from the channel and spatial views.
Addressing a fundamental limitation in deep vision models: lack of spatial attention
The primary aim of this manuscript is to underscore a significant limitation in current deep learning models, particularly vision models.
SalFoM: Dynamic Saliency Prediction with Video Foundation Models
Recent advancements in video saliency prediction (VSP) have shown promising performance compared to the human visual system, whose emulation is the primary goal of VSP.
FLORIDA: Fake-looking Real Images Dataset
Although extensive research has been carried out to evaluate the effectiveness of AI tools and models in detecting deep fakes, the question remains unanswered regarding whether these models can accurately identify genuine images that appear artificial.
Key-Value Transformer
Transformers have emerged as the prevailing standard solution for various AI tasks, including computer vision and natural language processing.
Qualitative Failures of Image Generation Models and Their Application in Detecting Deepfakes
The ability of image and video generation models to create photorealistic images has reached unprecedented heights, making it difficult to distinguish between real and fake images in many cases.
A Categorical Archive of ChatGPT Failures
The goal of this study is to assist researchers and developers in enhancing future language models and chatbots.
Diverse, Difficult, and Odd Instances (D2O): A New Test Set for Object Classification
Our dataset also comes with a ``miscellaneous'' category, over which we test the image tagging models.
BinaryVQA: A Versatile Test Set to Evaluate the Out-of-Distribution Generalization of VQA Models
Questions and answers are formulated and verified carefully and manually.
Logits are predictive of network type
A classifier is then trained on the logit vectors of the trained set of this dataset to map the logit vector to the network index that has generated it.
Generated Faces in the Wild: Quantitative Comparison of Stable Diffusion, Midjourney and DALL-E 2
Here, we conduct a quantitative comparison of three popular systems including Stable Diffusion, Midjourney, and DALL-E 2 in their ability to generate photorealistic faces in the wild.
How good are deep models in understanding the generated images?
On VQA, the OFA model scores 77. 3\% on answering 241 binary questions across 50 images.
A New Kind of Adversarial Example
Almost all adversarial attacks are formulated to add an imperceptible perturbation to an image in order to fool a model.
Is current research on adversarial robustness addressing the right problem?
Short answer: Yes, Long answer: No!
SplitMixer: Fat Trimmed From MLP-like Models
We show, both theoretically and experimentally, that SplitMixer performs on par with the state-of-the-art MLP-like models while having a significantly lower number of parameters and FLOPS.
Complementary datasets to COCO for object detection
For nearly a decade, the COCO dataset has been the central test bed of research in object detection.
Sensitivity of Average Precision to Bounding Box Perturbations
Here, we quantify the sensitivity of AP to bounding box perturbations and show that AP is very sensitive to small translations.
CNNs and Transformers Perceive Hybrid Images Similar to Humans
Hybrid images is a technique to generate images with two interpretations that change as a function of viewing distance.
Overparametrization improves robustness against adversarial attacks: A replication study
Overparametrization has become a de facto standard in machine learning.
Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos
In this paper, we propose a multitask learning method for visual-audio saliency prediction and sound source localization on multi-face video by leveraging visual, audio and face information.
Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model
Inspired by the findings of our investigation, we propose a novel multi-modal video saliency model consisting of three branches: visual, audio and face.
Pros and Cons of GAN Evaluation Measures: New Developments
This work is an update of a previous paper on the same topic published a few years ago.
Enhancing sensor resolution improves CNN accuracy given the same number of parameters or FLOPS
High image resolution is critical to obtain a good performance in many computer vision applications.
Contemplating real-world object classification
They showed a dramatic performance drop of the state of the art object recognition models on this dataset.
EMPIRICAL UPPER BOUND IN OBJECT DETECTION
In this work, by employing two popular state-of-the-art object detection benchmarks, MMDetection and Detectron2, and analyzing more than 15 models over 4 large-scale datasets, we systematically determine the upper bound in AP, which is 91. 6% on PASCAL VOC (test2007), 78. 2% on MS COCO (val2017), and 58. 9% on OpenImages (V4 validation set), regardless of the IOU.
CONTEMPLATING REAL-WORLDOBJECT RECOGNITION
Relative to the numbers reported in Barbu et al., around 10-15% of the performance loss is recovered, without any test time data augmentation.
Shape Defense
Further, we show that edge information can a) benefit other adversarial training methods, b) be even more effective in conjunction with background subtraction, c) be used to defend against poisoning attacks, and d) make CNNs more robust against natural image corruptions such as motion blur, impulse noise, and JPEG compression, than CNNs trained solely on RGB images.
Shape Defense Against Adversarial Attacks
In the first one, a classifier is adversarially trained on images with the edge map as an additional channel.
A Compact Deep Architecture for Real-time Saliency Prediction
Our proposed model consists of a modified U-net architecture, a novel fully connected layer, and central difference convolutional layers.
Bifurcated backbone strategy for RGB-D salient object detection
In particular, first, we propose to regroup the multi-level features into teacher and student features using a bifurcated backbone strategy (BBS).
Ranked #2 on
RGB-D Salient Object Detection
on RGBD135
Adversarial examples are useful too!
Deep learning has come a long way and has enjoyed an unprecedented success.
Harnessing adversarial examples with a surprisingly simple defense
I introduce a very simple method to defend against adversarial examples.
Empirical Upper Bound, Error Diagnosis and Invariance Analysis of Modern Object Detectors
Object detection remains as one of the most notorious open problems in computer vision.
ObjectNet Dataset: Reanalysis and Correction
They showed a dramatic performance drop of the state of the art object recognition models on this dataset.
Ranked #4 on
Image Classification
on ObjectNet (Bounding Box)
(using extra training data)
White Noise Analysis of Neural Networks
A white noise analysis of modern deep neural networks is presented to unveil their biases at the whole network level or the single neuron level.
Empirical Upper Bound in Object Detection and More
Object detection remains as one of the most notorious open problems in computer vision.
A New Ensemble Adversarial Attack Powered by Long-term Gradient Memories
Deep neural networks are vulnerable to adversarial attacks.
DAVE: A Deep Audio-Visual Embedding for Dynamic Saliency Prediction
Our results suggest that (1) audio is a strong contributing cue for saliency prediction, (2) salient visible sound-source is the natural cause of the superiority of our Audio-Visual model, (3) richer feature representations for the input space leads to more powerful predictions even in absence of more sophisticated saliency decoders, and (4) Audio-Visual model improves over 53. 54\% of the frames predicted by the best Visual model (our baseline).
How is Gaze Influenced by Image Transformations? Dataset and Model
Data size is the bottleneck for developing deep saliency models, because collecting eye-movement data is very time consuming and expensive.
Digging Deeper into Egocentric Gaze Prediction
Extensive experiments over multiple datasets reveal that (1) spatial biases are strong in egocentric videos, (2) bottom-up saliency models perform poorly in predicting gaze and underperform spatial biases, (3) deep features perform better compared to traditional features, (4) as opposed to hand regions, the manipulation point is a strong influential cue for gaze prediction, (5) combining the proposed recurrent model with bottom-up cues, vanishing points and, in particular, manipulation point results in the best gaze prediction accuracy over egocentric videos, (6) the knowledge transfer works best for cases where the tasks or sequences are similar, and (7) task and activity recognition can benefit from gaze prediction.
Adversarial Attacks against Deep Saliency Models
The proposed attack only requires a part of the model information, and is able to generate a sparser and more insidious adversarial perturbation, compared to traditional image-space attacks.
Human Attention in Image Captioning: Dataset and Analysis
In this work, we present a novel dataset consisting of eye movements and verbal descriptions recorded synchronously over images.
Understanding and Visualizing Deep Visual Saliency Models
Our analyses reveal that: 1) some visual regions (e. g. head, text, symbol, vehicle) are already encoded within various layers of the network pre-trained for object recognition, 2) using modern datasets, we find that fine-tuning pre-trained models for saliency prediction makes them favor some categories (e. g. head) over some others (e. g. text), 3) although deep models of saliency outperform classical models on natural images, the converse is true for synthetic stimuli (e. g. pop-out search arrays), an evidence of significant difference between human and data-driven saliency models, and 4) we confirm that, after-fine tuning, the change in inner-representations is mostly due to the task and not the domain shift in the data.
Video Summarization via Actionness Ranking
Prior work proposed supervised and unsupervised algorithms to train models for learning the underlying behavior of humans by increasing modeling complexity or craft-designing better heuristics to simulate human summary generation process.
From Third Person to First Person: Dataset and Baselines for Synthesis and Retrieval
In this work, we introduce two datasets (synthetic and natural/real) containing simultaneously recorded egocentric and exocentric videos.
Multi-Stream Dynamic Video Summarization
With vast amounts of video content being uploaded to the Internet every minute, video summarization becomes critical for efficient browsing, searching, and indexing of visual content.
Bottom-up Attention, Models of
In this review, we examine the recent progress in saliency prediction and proposed several avenues for future research.
Invariance Analysis of Saliency Models versus Human Gaze During Scene Free Viewing
Most of current studies on human gaze and saliency modeling have used high-quality stimuli.
Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges
Visual saliency models have enjoyed a big leap in performance in recent years, thanks to advances in deep learning and large scale annotated data.
Integrating Egocentric Videos in Top-view Surveillance Videos: Joint Identification and Temporal Alignment
Videos recorded from first person (egocentric) perspective have little visual appearance in common with those from third person perspective, especially with videos captured by top-view surveillance cameras.
Understanding Perceptual and Conceptual Fluency at a Large Scale
We create a dataset of 543, 758 logo designs spanning 39 industrial categories and 216 countries.
Cross-view image synthesis using geometry-guided conditional GANs
For this, we propose to use homography as a guide to map the images between the views based on the common field of view to preserve the details in the input image.
Improving Sequential Determinantal Point Processes for Supervised Video Summarization
In terms of modeling, we design a new probabilistic distribution such that, when it is integrated into SeqDPP, the resulting model accepts user input about the expected length of the summary.
Learning a Saliency Evaluation Metric Using Crowdsourced Perceptual Judgments
Due to the effectiveness of the learned metric, it also can be used to facilitate the development of new models for fixation prediction.
Learning to Promote Saliency Detectors
The categories and appearance of salient objects vary from image to image, therefore, saliency detection is an image-specific task.
Detect Globally, Refine Locally: A Novel Approach to Saliency Detection
Moreover, to effectively recover object boundaries, we propose a local Boundary Refinement Network (BRN) to adaptively learn the local contextual information for each spatial position.
Ranked #15 on
RGB Salient Object Detection
on DUTS-TE
Salient Object Detection Driven by Fixation Prediction
Salient object detection is then viewed as fine-grained object-level saliency segmentation and is progressively optimized with the guidance of the fixation map in a top-down manner.
Enhanced-alignment Measure for Binary Foreground Map Evaluation
The existing binary foreground map (FM) measures to address various types of errors in either pixel-wise or structural ways.
Three Birds One Stone: A General Architecture for Salient Object Segmentation, Edge Detection and Skeleton Extraction
Although these tasks are inherently very different, we show that our unified approach performs very well on all of them and works far better than current single-purpose state-of-the-art methods.
What Catches the Eye? Visualizing and Understanding Deep Saliency Models
Deep convolutional neural networks have demonstrated high performances for fixation prediction in recent years.
Cross-View Image Synthesis using Conditional GANs
X-Fork architecture has a single discriminator and a single generator.
Analysis of Hand Segmentation in the Wild
In the quest for robust hand segmentation methods, we evaluated the performance of the state of the art semantic segmentation methods, off the shelf and fine-tuned, on existing datasets.
Pros and Cons of GAN Evaluation Measures
Generative models, in particular generative adversarial networks (GANs), have received significant attention recently.
Visual Weather Temperature Prediction
In this paper, we attempt to employ convolutional recurrent neural networks for weather temperature estimation using only image data.
Revisiting Video Saliency: A Large-scale Benchmark and a New Model
Existing video saliency datasets lack variety and generality of common dynamic scenes and fall short in covering challenging situations in unconstrained environments.
Segmenting Sky Pixels in Images
As a result of our efforts, we have seen an improvement of 10-15% in the average MCR compared to the prior methods on SkyFinder dataset.
A Stagewise Refinement Model for Detecting Salient Objects in Images
To remedy this problem, here we propose to augment feedforward neural networks with a novel pyramid pooling module and a multi-stage refinement mechanism for saliency detection.
Ranked #16 on
RGB Salient Object Detection
on DUTS-TE
(max F-measure metric)
Statistics of Deep Generated Images
Here, we explore the low-level statistics of images generated by state-of-the-art deep generative models.
An Unsupervised Game-Theoretic Approach to Saliency Detection
Saliency maps are generated according to each region's strategy in the Nash equilibrium of the proposed Saliency Game.
Structure-measure: A New Way to Evaluate Foreground Maps
Our new measure simultaneously evaluates region-aware and object-aware structural similarity between a SM and a GT map.
Human-like Clustering with Deep Convolutional Neural Networks
We do not dwell much on the learning mechanisms in these frameworks as they are still a matter of debate, with respect to biological constraints.
Saliency Revisited: Analysis of Mouse Movements versus Fixations
This paper revisits visual saliency prediction by evaluating the recent advancements in this field such as crowd-sourced mouse tracking-based databases and contextual annotations.
What are the Receptive, Effective Receptive, and Projective Fields of Neurons in Convolutional Neural Networks?
In this work, we explain in detail how receptive fields, effective receptive fields, and projective fields of neurons in different layers, convolution or pooling, of a Convolutional Neural Network (CNN) are calculated.
Negative Results in Computer Vision: A Perspective
A negative result is when the outcome of an experiment or a model is not what is expected or when a hypothesis does not hold.
Cultural Vocal Bursts Intensity Prediction
Experimental Design
+1
Paying Attention to Descriptions Generated by Image Captioning Models
To bridge the gap between humans and machines in image understanding and describing, we need further insight into how people describe a perceived scene.
Learning to predict where to look in interactive environments using deep recurrent q-learning
Bottom-Up (BU) saliency models do not perform well in complex interactive environments where humans are actively engaged in tasks (e. g., sandwich making and playing the video games).
EgoTransfer: Transferring Motion Across Egocentric and Exocentric Domains using Deep Neural Networks
On one hand, the abundance of egocentric cameras in the past few years has offered the opportunity to study a lot of vision problems from the first-person perspective.
Deeply supervised salient object detection with short connections
Recent progress on saliency detection is substantial, benefiting mostly from the explosive development of Convolutional Neural Networks (CNNs).
Ranked #4 on
RGB Salient Object Detection
on SBU / SBU-Refine
Exploiting inter-image similarity and ensemble of extreme learners for fixation prediction using deep features
This paper presents a novel fixation prediction and saliency modeling framework based on inter-image similarities and ensemble of Extreme Learning Machines (ELM).
Egocentric Height Estimation
We used both traditional computer vision approaches and deep learning in order to determine the visual cues that results in best height estimation.
Vanishing point detection with convolutional neural networks
Inspired by the finding that vanishing point (road tangent) guides driver's gaze, in our previous work we showed that vanishing point attracts gaze during free viewing of natural scenes as well as in visual search (Borji et al., Journal of Vision 2016).
Egocentric Meets Top-view
First, having a set of egocentric videos and a top-view video, can we verify if the top-view video contains all, or some of the egocentric viewers present in the egocentric set?
Ego2Top: Matching Viewers in Egocentric and Top-view Videos
At the same time, surveillance cameras and drones offer an abundance of visual information, often captured from top-view.
iLab-20M: A Large-Scale Controlled Object Dataset to Investigate Deep Learning
Tolerance to image variations (e. g. translation, scale, pose, illumination, background) is an important desired property of any object recognition system, be it human or machine.
A Review of Co-saliency Detection Technique: Fundamentals, Applications, and Challenges
Co-saliency detection is a newly emerging and rapidly growing research area in computer vision community.
Vanishing point attracts gaze in free-viewing and visual search tasks
In the second experiment, we asked 14 subjects (4 female, mean age 23. 07, SD=1. 26) to search for a target character (T or L) placed randomly on a 3x3 imaginary grid overlaid on top of an image.
Fixation prediction with a combined model of bottom-up saliency and vanishing point
By predicting where humans look in natural scenes, we can understand how they perceive complex natural scenes and prioritize information for further high-level visual processing.
What can we learn about CNNs from a large scale controlled object dataset?
Tolerance to image variations (e. g. translation, scale, pose, illumination) is an important desired property of any object recognition system, be it human or machine.
Computational models: Bottom-up and top-down aspects
We focus on {\em computational models of attention} as defined by Tsotsos \& Rothenstein \shortcite{Tsotsos_Rothenstein11}: Models which can process any visual stimulus (typically, an image or video clip), which can possibly also be given some task definition, and which make predictions that can be compared to human or animal behavioral or physiological responses elicited by the same stimulus and task.
Computational models of attention
This chapter reviews recent computational models of visual attention.
Vanishing Point Attracts Eye Movements in Scene Free-viewing
Eye movements are crucial in understanding complex scenes.
CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research
Saliency modeling has been an active research area in computer vision for about two decades.
Reconciling saliency and object center-bias hypotheses in explaining free-viewing fixations
Predicting where people look in natural scenes has attracted a lot of interest in computer vision and computational neuroscience over the past two decades.
Salient Object Detection: A Benchmark
We extensively compare, qualitatively and quantitatively, 40 state-of-the-art models (28 salient object detection, 10 fixation prediction, 1 objectness, and 1 baseline) over 6 challenging datasets for the purpose of benchmarking salient object detection and segmentation methods.
What is a salient object? A dataset and a baseline model for salient object detection
While the notion of most salient object is sensible when multiple objects exist in a scene, current datasets for evaluation of saliency detection approaches often have scenes with only one single object.
Salient Object Detection: A Survey
Detecting and segmenting salient objects from natural scenes, often referred to as salient object detection, has attracted great interest in computer vision.
Human vs. Computer in Scene and Object Recognition
Several decades of research in computer and primate vision have resulted in many models (some specialized for one problem, others more general) and invaluable experimental data.
Bayesian optimization explains human active search
Many real-world problems have complicated objective functions.
