Search Results for author:
Found 27 papers, 19 papers with code
From Unimodal to Multimodal: Scaling up Projectors to Align Modalities
However, this practice has left powerful unimodal encoders for both vision and language underutilized in multimodal applications which raises a key question: Is there a plausible way to connect unimodal backbones for zero-shot vision-language tasks?
Falcon2-11B Technical Report
We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model.
Open-Vocabulary Temporal Action Localization using Multimodal Guidance
Open-Vocabulary Temporal Action Localization (OVTAL) enables a model to recognize any desired action category in videos without the need to explicitly curate training data for all categories.
Efficient 3D-Aware Facial Image Editing via Attribute-Specific Prompt Learning
The LAE harnesses a pre-trained vision-language model to find text-guided attribute-specific editing direction in the latent space of any pre-trained 3D-aware GAN.
Multi-modal Generation via Cross-Modal In-Context Learning
Moreover, they often result in misaligned image generation for prompt sequences featuring multiple objects.
Do Vision and Language Encoders Represent the World Similarly?
In the absence of statistical similarity in aligned encoders like CLIP, we show that a possible matching of unaligned encoders exists without any training.
Do VSR Models Generalize Beyond LRS3?
The Lip Reading Sentences-3 (LRS3) benchmark has primarily been the focus of intense research in visual speech recognition (VSR) during the last few years.
Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mapping
Visual Speech Recognition (VSR) differs from the common perception tasks as it requires deeper reasoning over the video sequence, even by human experts.
Remote Sensing Change Detection With Transformers Trained from Scratch
Current transformer-based change detection (CD) approaches either employ a pre-trained model trained on large-scale image classification ImageNet dataset or rely on first pre-training on another CD dataset and then fine-tuning on the target benchmark.
Cross-modulated Few-shot Image Generation for Colorectal Tissue Classification
In this work, we propose a few-shot colorectal tissue image generation method for addressing the scarcity of histopathological training data for rare cancer tissues.
Generative Multiplane Neural Radiance for 3D-Aware Image Generation
We present a method to efficiently generate 3D-aware high-resolution images that are view-consistent across multiple target views.
Video Instance Segmentation in an Open-World
Open-world formulation relaxes the close-world static-learning assumption as follows: (a) first, it distinguishes a set of known categories as well as labels an unknown object as `unknown' and then (b) it incrementally learns the class of an unknown as and when the corresponding semantic labels become available.
PS-ARM: An End-to-End Attention-aware Relation Mixer Network for Person Search
Our PS-ARM achieves state-of-the-art performance on both datasets.
Video Instance Segmentation via Multi-scale Spatio-temporal Split Attention Transformer
When using the ResNet50 backbone, our MS-STS achieves a mask AP of 50. 1 %, outperforming the best reported results in literature by 2. 7 % and by 4. 8 % at higher overlap threshold of AP_75, while being comparable in model size and speed on Youtube-VIS 2019 val.
Spatio-temporal Relation Modeling for Few-shot Action Recognition
Experiments are performed on four few-shot action recognition benchmarks: Kinetics, SSv2, HMDB51 and UCF101.
Ranked #1 on
Few Shot Action Recognition
on UCF101
(using extra training data)
OW-DETR: Open-world Detection Transformer
In the case of incremental object detection, OW-DETR outperforms the state-of-the-art for all settings on PASCAL VOC.
Discriminative Region-based Multi-Label Zero-Shot Learning
We note that the best existing multi-label ZSL method takes a shared approach towards attending to region features with a common set of attention maps for all the classes.
Ranked #2 on
Multi-label zero-shot learning
on Open Images V4
Structured Latent Embeddings for Recognizing Unseen Classes in Unseen Domains
The need to address the scarcity of task-specific annotated data has resulted in concerted efforts in recent years for specific settings such as zero-shot learning (ZSL) and domain generalization (DG), to separately address the issues of semantic shift and domain shift, respectively.
Generative Multi-Label Zero-Shot Learning
Nevertheless, computing reliable attention maps for unseen classes during inference in a multi-label setting is still a challenge.
Ranked #8 on
Multi-label zero-shot learning
on NUS-WIDE
D2-Net: Weakly-Supervised Action Localization via Discriminative Embeddings and Denoised Activations
The proposed formulation comprises a discriminative and a denoising loss term for enhancing temporal action localization.
Ranked #3 on
Weakly Supervised Action Localization
on THUMOS’14
Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification
We propose to enforce semantic consistency at all stages of (generalized) zero-shot learning: training, feature synthesis and classification.
Ranked #2 on
Generalized Zero-Shot Learning
on Oxford 102 Flower
3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization
Our joint formulation has three terms: a classification term to ensure the separability of learned action features, an adapted multi-label center loss term to enhance the action feature discriminability and a counting loss term to delineate adjacent action sequences, leading to improved localization.
Ranked #1 on
Action Classification
on THUMOS'14
Action Classification
Weakly Supervised Action Localization
+2
Out-of-Distribution Detection for Generalized Zero-Shot Action Recognition
We introduce an out-of-distribution detector that determines whether the video features belong to a seen or unseen action category.
Action Recognition In Videos
Out-of-Distribution Detection
+2
A Large Dataset for Improving Patch Matching
Similarly on the Strecha dataset, we see an improvement of 3-5% for the matching task in non-planar scenes.
Improved Descriptors for Patch Matching and Reconstruction
Scenes from the Oxford ACRD, MVS and Synthetic datasets are used for evaluating the patch matching performance of the learnt descriptors while the Strecha dataset is used to evaluate the 3D reconstruction task.
Hyper-Fisher Vectors for Action Recognition
We also perform experiments to show that the performance of the Hyper-Fisher Vector is robust to the dictionary size of the BoW encoding.
A Cause and Effect Analysis of Motion Trajectories for Modeling Actions
Different object-parts have varying degrees of interactions with the other parts during an action cycle.
