Search Results for author:
Found 43 papers, 13 papers with code
Multi-scale Image Super Resolution with a Single Auto-Regressive Model
We address these limitations through two novel components: a) a Hierarchical Image Tokenization approach with a multi-scale image tokenizer that progressively represents images at different scales while simultaneously enforcing token overlap across scales, and b) a Direct Preference Optimization (DPO) regularization term that, relying solely on the LR and HR tokenizations, encourages the transformer to produce the latter over the former.
Edge-SD-SR: Low Latency and Parameter Efficient On-device Super-Resolution with Stable Diffusion via Bidirectional Conditioning
To maintain a high visual quality on such low compute budget, we introduce a number of training strategies: (i) A novel conditioning mechanism on the low resolution input, coined bidirectional conditioning, which tailors the SD model for the SR task.
VladVA: Discriminative Fine-tuning of LVLMs
Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the de facto approach for discriminative vision-language representation learning.
FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion
We introduce a Frequency Modulation (FM) module that leverages the Fourier domain to improve the global structure consistency, and an Attention Modulation (AM) module which improves the consistency of local texture patterns, a problem largely ignored in prior works.
A Bayesian Approach to Data Point Selection
Data point selection (DPS) is becoming a critical topic in deep learning due to the ease of acquiring uncurated training data compared to the difficulty of obtaining curated or processed data.
MobileQuant: Mobile-friendly Quantization for On-device Language Models
We first investigate the limitations of existing quantization methods for on-device deployment, with a special focus on activation quantization.
CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs
Despite recent successes, LVLMs or Large Vision Language Models are prone to hallucinating details like objects and their properties or relations, limiting their real-world deployment.
You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillation
We show that the combination of spatially distilled U-Net and fine-tuned decoder outperforms state-of-the-art methods requiring 200 steps with only one single step.
Graph Guided Question Answer Generation for Procedural Question-Answering
The key technological enabler is a novel mechanism for automatic question-answer generation from procedural text which can ingest large amounts of textual instructions and produce exhaustive in-domain QA training data.
Aligned Unsupervised Pretraining of Object Detectors with Self-training
This results in complex pipelines and a task gap between the pretraining and the downstream task.
Black Box Few-Shot Adaptation for Vision-Language models
Vision-Language (V-L) models trained with contrastive learning to align the visual and language modalities have been shown to be strong few-shot learners.
ReGen: A good Generative Zero-Shot Video Classifier Should be Rewarded
Specifically, we propose ReGen, a novel reinforcement learning based framework with a three-fold objective and reward functions: (1) a class-level discrimination reward that enforces the generated caption to be correctly classified into the corresponding action class, (2) a CLIP reward that encourages the generated caption to continue to be descriptive of the input video (i. e. video-specific), and (3) a grammar reward that preserves the grammatical correctness of the caption.
Graph2Vid: Flow graph to Video Grounding for Weakly-supervised Multi-Step Localization
In this setup, we seek the optimal step ordering consistent with the procedure flow graph and a given video.
FS-DETR: Few-Shot DEtection TRansformer with prompting and without re-training
Importantly, we show that our system is not only more flexible than existing methods, but also, it makes a step towards satisfying desideratum (c).
Effective Self-supervised Pre-training on Low-compute Networks without Distillation
This hints at view sampling being one of the performance bottlenecks for SSL on low-capacity networks.
Bayesian Prompt Learning for Image-Language Model Generalization
Our approach regularizes the prompt space, reduces overfitting to the seen prompts and improves the prompt generalization on unseen prompts.
Ranked #1 on
Few-Shot Learning
on food101
REST: REtrieve & Self-Train for generative action recognition
We evaluate REST on the problem of zero-shot action recognition where we show that our approach is very competitive when compared to contrastive learning-based methods.
Efficient Attention-free Video Shift Transformers
To address this gap, in this paper, we make the following contributions: (a) we construct a highly efficient \& accurate attention-free block based on the shift operator, coined Affine-Shift block, specifically designed to approximate as closely as possible the operations in the MHSA block of a Transformer layer.
iBoot: Image-bootstrapped Self-Supervised Video Representation Learning
Video self-supervised learning (SSL) suffers from added challenges: video datasets are typically not as large as image datasets, compute is an order of magnitude larger, and the amount of spurious patterns the optimizer has to sieve through is multiplied several fold.
Knowledge Distillation Meets Open-Set Semi-Supervised Learning
The key idea is that we leverage the teacher's classifier as a semantic critic for evaluating the representations of both teacher and student and distilling the semantic knowledge with high-order structured information over all feature dimensions.
EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers
In this work, pushing further along this under-studied direction we introduce EdgeViTs, a new family of light-weight ViTs that, for the first time, enable attention-based vision models to compete with the best light-weight CNNs in the tradeoff between accuracy and on-device efficiency.
SOS! Self-supervised Learning Over Sets Of Handled Objects In Egocentric Action Recognition
To overcome both limitations, we introduce Self-Supervised Learning Over Sets (SOS), an approach to pre-train a generic Objects In Contact (OIC) representation model from video object regions detected by an off-the-shelf hand-object contact detector.
Low-Fidelity Video Encoder Optimization for Temporal Action Localization
This results in a task discrepancy problem for the video encoder – trained for action classification, but used for TAL.
Ranked #11 on
Temporal Action Localization
on HACS
SAIC_Cambridge-HuPBA-FBK Submission to the EPIC-Kitchens-100 Action Recognition Challenge 2021
This report presents the technical details of our submission to the EPIC-Kitchens-100 Action Recognition Challenge 2021.
Space-time Mixing Attention for Video Transformer
In this work, we propose a Video Transformer model the complexity of which scales linearly with the number of frames in the video sequence and hence induces no overhead compared to an image-based Transformer model.
Ranked #32 on
Action Classification
on Kinetics-600
Low-Fidelity End-to-End Video Encoder Pre-training for Temporal Action Localization
This results in a task discrepancy problem for the video encoder -- trained for action classification, but used for TAL.
Few-shot Action Recognition with Prototype-centered Attentive Learning
Extensive experiments on four standard few-shot action benchmarks show that our method clearly outperforms previous state-of-the-art methods, with the improvement particularly significant (10+\%) on the most challenging fine-grained action recognition benchmark.
Knowledge distillation via softmax regression representation learning
We advocate for a method that optimizes the output feature of the penultimate layer of the student network and hence is directly related to representation learning.
Boundary-sensitive Pre-training for Temporal Localization in Videos
However, most existing models developed for these tasks are pre-trained on general video action classification tasks.
Ranked #24 on
Temporal Action Localization
on ActivityNet-1.3
High-Capacity Expert Binary Networks
Network binarization is a promising hardware-aware direction for creating efficient deep models.
Towards Practical Lipreading with Distilled and Efficient Models
However, our most promising lightweight models are on par with the current state-of-the-art while showing a reduction of 8. 2x and 3. 9x in terms of computational cost and number of parameters, respectively, which we hope will enable the deployment of lipreading models in practical applications.
Ranked #7 on
Lipreading
on Lip Reading in the Wild
Egocentric Action Recognition by Video Attention and Temporal Context
In this challenge, action recognition is posed as the problem of simultaneously predicting a single `verb' and `noun' class label given an input trimmed video clip.
Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention
Departing from existing alternatives, our W3 module models all three facets of video attention jointly.
Ranked #1 on
Action Recognition
on EgoGesture
Training Binary Neural Networks with Real-to-Binary Convolutions
This paper shows how to train binary networks to within a few percent points ($\sim 3-5 \%$) of the full precision counterpart.
Knowledge distillation via adaptive instance normalization
To this end, we propose a new knowledge distillation method based on transferring feature statistics, specifically the channel-wise mean and variance, from the teacher to the student.
BATS: Binary ArchitecTure Search
We show that directly applying NAS to the binary domain provides very poor results.
Ranked #1 on
Classification with Binary Neural Network
on CIFAR-10
(using extra training data)
Lipreading using Temporal Convolutional Networks
We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively.
Ranked #1 on
Lipreading
on LRW-1000
Action recognition with spatial-temporal discriminative filter banks
In this work we focus on how to improve the representation capacity of the network, but rather than altering the backbone, we focus on improving the last layers of the network, where changes have low impact in terms of computational cost.
Ranked #37 on
Action Recognition
on Something-Something V1
(using extra training data)
Fusing Deep Learned and Hand-Crafted Features of Appearance, Shape, and Dynamics for Automatic Pain Estimation
Automatic continuous time, continuous value assessment of a patient's pain from face video is highly sought after by the medical profession.
A Functional Regression approach to Facial Landmark Tracking
This paper presents a Functional Regression solution to the least squares problem, which we coin Continuous Regression, resulting in the first real-time incremental face tracker.
Cascaded Continuous Regression for Real-time Incremental Face Tracking
We then derive the incremental learning updates for CCR (iCCR) and show that it is an order of magnitude faster than standard incremental learning for cascaded regression, bringing the time required for the update from seconds down to a fraction of a second, thus enabling real-time tracking.
Learning to Transfer: Transferring Latent Task Structures and Its Application to Person-Specific Facial Action Unit Detection
We thus consider a novel problem: all AU models for the target subject are to be learnt using person-specific annotated data for a reference AU (AU12 in our case), and no data or little data regarding the target AU.
TRIC-track: Tracking by Regression With Incrementally Learned Cascades
This paper proposes a novel approach to part-based tracking by replacing local matching of an appearance model by direct prediction of the displacement between local image patches and part locations.
