Search Results for author:
Found 39 papers, 18 papers with code
Sequential Compositional Generalization in Multimodal Models
The rise of large-scale multimodal models has paved the pathway for groundbreaking advances in generative modeling and reasoning, unlocking transformative applications in a variety of complex tasks.
ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models
With the ever-increasing popularity of pretrained Video-Language Models (VidLMs), there is a pressing need to develop robust evaluation methodologies that delve deeper into their visio-linguistic capabilities.
Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers
Neural networks have revolutionized language modeling and excelled in various downstream tasks.
Hyperspectral Image Denoising via Self-Modulating Convolutional Neural Networks
At the core of the model lies a novel block, which we call spectral self-modulating residual block (SSMRB), that allows the network to transform the features in an adaptive manner based on the adjacent spectral data, enhancing the network's ability to handle complex noise.
Spherical Vision Transformer for 360-degree Video Saliency Prediction
The growing interest in omnidirectional videos (ODVs) that capture the full field-of-view (FOV) has gained 360-degree saliency prediction importance in computer vision.
CLIP-Guided StyleGAN Inversion for Text-Driven Real Image Editing
Existing approaches for editing images using language either resort to instance-level latent code optimization or map predefined text prompts to some editing directions in the latent space.
HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks
Event-based cameras are becoming increasingly popular for their ability to capture high-speed motion with low latency and high dynamic range.
Ranked #1 on
Video Reconstruction
on MVSEC
EVREAL: Towards a Comprehensive Benchmark and Analysis Suite for Event-based Video Reconstruction
Event cameras are a new type of vision sensor that incorporates asynchronous and independent pixels, offering advantages over traditional frame-based cameras such as high dynamic range and minimal motion blur.
VidStyleODE: Disentangled Video Editing via StyleGAN and NeuralODEs
However, the applicability of such advancements to the video domain has been hindered by the difficulty of representing and controlling videos in the latent space of GANs.
Inst-Inpaint: Instructing to Remove Objects with Diffusion Models
From the application point of view, a user needs to generate the masks for the objects they would like to remove which can be time-consuming and prone to errors.
ST360IQ: No-Reference Omnidirectional Image Quality Assessment with Spherical Vision Transformers
As their popularity has increased dramatically in recent years, evaluating the quality of 360 images has become a problem of interest since it provides insights for capturing, transmitting, and consuming this new media.
Detecting Euphemisms with Literal Descriptions and Visual Imagery
In the second stage, we integrate visual supervision into our system using visual imageries, two sets of images generated by a text-to-image model by taking terms and descriptions as input.
Disentangling Content and Motion for Text-Based Neural Video Manipulation
Giving machines the ability to imagine possible new objects or scenes from linguistic descriptions and produce their realistic renderings is arguably one of the most challenging problems in computer vision.
Perception-Distortion Trade-off in the SR Space Spanned by Flow Models
We achieve this by benefiting from a diverse set of feasible photo-realistic solutions in the SR space spanned by flow models.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
SLAMP: Stochastic Latent Appearance and Motion Prediction
Motion is an important cue for video prediction and often utilized by separating video content into static and dynamic components.
Ranked #1 on
Video Prediction
on Cityscapes 128x128
(PSNR metric)
A Gated Fusion Network for Dynamic Saliency Prediction
Predicting saliency in videos is a challenging problem due to complex modeling of interactions between spatial and temporal information, especially when ever-changing, dynamic nature of videos is considered.
Object and Relation Centric Representations for Push Effect Prediction
Our approach enables the robot to predict and adapt the effect of a pushing action as it observes the scene.
Cross-lingual Visual Pre-training for Multimodal Machine Translation
Pre-trained language models have been shown to improve performance in many natural language tasks substantially.
MSVD-Turkish: A Comprehensive Multimodal Dataset for Integrated Vision and Language Research in Turkish
We hope that the MSVD-Turkish dataset and the results reported in this work will lead to better video captioning and multimodal machine translation models for Turkish and other morphology rich and agglutinative languages.
CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions
Humans are able to perceive, understand and reason about causal events.
Burst Photography for Learning to Enhance Extremely Dark Images
Capturing images under extremely low-light conditions poses significant challenges for the standard camera pipeline.
Modulating Bottom-Up and Top-Down Visual Processing via Language-Conditional Filters
Our experiments reveal that using language to control the filters for bottom-up visual processing in addition to top-down attention leads to better results on both tasks and achieves competitive performance.
Burst Denoising of Dark Images
Capturing images under extremely low-light conditions poses significant challenges for the standard camera pipeline.
mustGAN: Multi-Stream Generative Adversarial Networks for MR Image Synthesis
Multi-contrast MRI protocols increase the level of morphological information available for diagnosis.
Procedural Reasoning Networks for Understanding Multimodal Procedures
Our model learns to dynamically update entity states in relation to each other while reading the text instructions.
Belief Regulated Dual Propagation Nets for Learning Action Effects on Articulated Multi-Part Objects
In this paper, we introduce Belief Regulated Dual Propagation Networks (BRDPN), a general purpose learnable physics engine, which enables a robot to predict the effects of its actions in scenes containing groups of articulated multi-part objects.
Robotics
RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes
With over 36K automatically generated question-answer pairs, we design a set of comprehension and reasoning tasks that require joint understanding of images and text, capturing the temporal flow of events and making sense of procedural knowledge.
Manipulating Attributes of Natural Scenes via Hallucination
In this study, we explore building a two-stage framework for enabling users to directly manipulate high-level attributes of a natural scene.
Language Guided Fashion Image Manipulation with Feature-wise Transformations
Developing techniques for editing an outfit image through natural sentences and accordingly generating new outfits has promising applications for art, fashion and design.
Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks
The proposed approach preserves high-frequency details via an adversarial loss; and it offers enhanced synthesis performance via a pixel-wise loss for registered multi-contrast images and a cycle-consistency loss for unregistered images.
Re-evaluating Automatic Metrics for Image Captioning
The task of generating natural language descriptions from images has received a lot of attention in recent years.
Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
Automatic image synthesis research has been rapidly growing with deep networks getting more and more expressive.
Spatio-Temporal Saliency Networks for Dynamic Saliency Prediction
We also carry out some experiments on a number of still images from the MIT300 dataset by exploiting the optical flow maps predicted from these images.
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities.
Image Matting With KL-Divergence Based Sparse Sampling
Previous sampling-based image matting methods typically rely on certain heuristics in collecting representative samples from known regions, and thus their performance deteriorates if the underlying assumptions are not satisfied.
Visual saliency estimation by integrating features using multiple kernel learning
In the last few decades, significant achievements have been attained in predicting where humans look at images through different computational models.
