Search Results for author:
Found 48 papers, 19 papers with code
CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion
These prompts encompass text prompts (representing cooking steps), image prompts (corresponding to cooking images), and multi-modal prompts (mixing cooking steps and images), ensuring the consistent generation of cooking procedural images.
PolySmart and VIREO @ TRECVid 2024 Ad-hoc Video Search
The generated queries retrieve different rank lists from the original query.
Visual Cue Enhancement and Dual Low-Rank Adaptation for Efficient Visual Instruction Fine-Tuning
Parameter-efficient fine-tuning multimodal large language models (MLLMs) presents significant challenges, including reliance on high-level visual features that limit fine-grained detail comprehension, and data conflicts that arise from task complexity.
Retrieval Augmented Recipe Generation
Existing works for recipe generation primarily utilize a two-stage training method, first generating ingredients and then obtaining instructions from both the image and ingredients.
WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines
This benchmark includes a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects, spanning 9 language families and featuring over 1 million data points, making it the largest multicultural VQA benchmark to date.
Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models
Despite having tremendous progress in image-to-3D generation, existing methods still struggle to produce multi-view consistent images with high-resolution textures in detail, especially in the paradigm of 2D diffusion that lacks 3D awareness.
RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models
Uni-Food is designed to provide a more holistic approach to food data analysis, thereby enhancing the performance and capabilities of LMMs in this domain.
PosMLP-Video: Spatial and Temporal Relative Position Encoding for Efficient Video Recognition
In recent years, vision Transformers and MLPs have demonstrated remarkable performance in image understanding tasks.
Improving Interpretable Embeddings for Ad-hoc Video Search with Generative Captions and Multi-word Concept Bank
Experimental results show that the integration of the above-proposed elements doubles the R@1 performance of the AVS method on the MSRVTT dataset and improves the xinfAP on the TRECVid AVS query sets for 2016-2023 (eight years) by a margin from 2% to 77%, with an average about 20%.
OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation
The pre-training phase equips FoodLearner with the capability to align visual information with corresponding textual representations that are specifically related to food, while the second phase adapts both the FoodLearner and the Image-Informed Text Encoder for the segmentation task.
Interpretable Embedding for Ad-hoc Video Search
Answering query with semantic concepts has long been the mainstream approach for video search.
FoodLMM: A Versatile Food Assistant using Large Multi-modal Model
In the second stage, we construct a multi-round conversation dataset and a reasoning segmentation dataset to fine-tune the model, enabling it to conduct professional dialogues and generate segmentation masks based on complex reasoning in the food domain.
Incremental Learning on Food Instance Segmentation
The power of the framework is a novel difficulty assessment model, which forecasts how challenging an unlabelled sample is to the latest trained instance segmentation model.
GroundNLQ @ Ego4D Natural Language Queries Challenge 2023
Motivated by this, we leverage a two-stage pre-training strategy to train egocentric feature extractors and the grounding model on video narrations, and further fine-tune the model on annotated data.
Interactive Video Corpus Moment Retrieval using Reinforcement Learning
Nevertheless, when the first few pages of results are swamped with visually similar items, or the search target is hidden deep in the ranked list, finding the know-item target usually requires a long duration of browsing and result inspection.
ObjectFusion: Multi-modal 3D Object Detection with Object-Centric Fusion
Recent progress on multi-modal 3D object detection has featured BEV (Bird-Eye-View) based fusion, which effectively unifies both LiDAR point clouds and camera images in a shared BEV space.
An Efficient COarse-to-fiNE Alignment Framework @ Ego4D Natural Language Queries Challenge 2022
This technical report describes the CONE approach for Ego4D Natural Language Queries (NLQ) Challenge in ECCV 2022.
Dynamic Temporal Filtering in Video Models
The pre-determined kernel size severely limits the temporal receptive fields and the fixed weights treat each spatial location across frames equally, resulting in sub-optimal solution for long-range temporal modeling in natural scenes.
CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding
This paper tackles an emerging and challenging problem of long video temporal grounding~(VTG) that localizes video moments related to a natural language (NL) query.
Long-term Leap Attention, Short-term Periodic Shift for Video Classification
By replacing a vanilla 2D attention with the LAPS, we could adapt a static transformer into a video one, with zero extra parameters and neglectable computation overhead ($\sim$2. 6\%).
Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning
Motivated by the wavelet theory, we construct a new Wavelet Vision Transformer (\textbf{Wave-ViT}) that formulates the invertible down-sampling with wavelet transforms and self-attention learning in a unified way.
Ranked #226 on
Image Classification
on ImageNet
(Un)likelihood Training for Interpretable Embedding
Cross-modal representation learning has become a new normal for bridging the semantic gap between text and visual data.
MLP-3D: A MLP-like 3D Architecture with Grouped Time Mixing
By deriving the novel grouped time mixing (GTM) operations, we equip the basic token-mixing MLP with the ability of temporal modeling.
Ranked #22 on
Action Recognition
on Something-Something V1
Cross-lingual Adaptation for Recipe Retrieval with Mixup
To bridge the domain gap, recipe mixup loss is proposed to enforce the intermediate domain to locate in the shortest geodesic path between source and target domains in the recipe embedding space.
Adaptive Split-Fusion Transformer
Neural networks for visual content understanding have recently evolved from convolutional ones (CNNs) to transformers.
Group Contextualization for Video Recognition
By utilizing calibrators to embed feature with four different kinds of contexts in parallel, the learnt representation is expected to be more resilient to diverse types of activities.
Ranked #3 on
Egocentric Activity Recognition
on EGTEA
Condensing a Sequence to One Informative Frame for Video Recognition
This paper studies a two-step alternative that first condenses the video sequence to an informative "frame" and then exploits off-the-shelf image recognition system on the synthetic frame.
Boosting Video Representation Learning with Multi-Faceted Integration
Video content is multifaceted, consisting of objects, scenes, interactions or actions.
Optimization Planning for 3D ConvNets
In this paper, we decompose the path into a series of training "states" and specify the hyper-parameters, e. g., learning rate and the length of input clips, in each state.
CONQUER: Contextual Query-aware Ranking for Video Corpus Moment Retrieval
This task is essential because advanced video retrieval applications should enable users to retrieve a precise moment from a large video corpus.
Ranked #1 on
Video Corpus Moment Retrieval
on TVR
Token Shift Transformer for Video Classification
It is worth noticing that our TokShift transformer is a pure convolutional-free video transformer pilot with computational efficiency for video understanding.
Pyramid Fusion Dark Channel Prior for Single Image Dehazing
In this paper, we propose the pyramid fusion dark channel prior (PF-DCP) for single image dehazing.
Multi-modal Cooking Workflow Construction for Food Recipes
Understanding food recipe requires anticipating the implicit causal effects of cooking actions, such that the recipe can be converted into a graph describing the temporal workflow of the recipe.
Transferring and Regularizing Prediction for Semantic Segmentation
In the view of extremely expensive expert labeling, recent research has shown that the models trained on photo-realistic synthetic data (e. g., computer games) with computer-generated annotations can be adapted to real images.
Ranked #19 on
Domain Adaptation
on SYNTHIA-to-Cityscapes
Exploring Category-Agnostic Clusters for Open-Set Domain Adaptation
A clustering branch is capitalized on to ensure that the learnt representation preserves such underlying structure by matching the estimated assignment distribution over clusters to the inherent cluster distribution for each target sample.
k-sums: another side of k-means
This optimization procedure converges faster to a better local minimum over k-means and many of its variants.
Deeply Activated Salient Region for Instance Search
Due to the lack of proper mechanism in locating instances and deriving feature representation, instance search is generally only effective for retrieving instances of known object categories.
On the Merge of k-NN Graph
Unfortunately, a closely related issue of how to merge two existing k-NN graphs has been overlooked.
vireoJD-MM at Activity Detection in Extended Videos
This notebook paper presents an overview and comparative analysis of our system designed for activity detection in extended videos (ActEV-PC) in ActivityNet Challenge 2019.
Learning Spatio-Temporal Representation with Local and Global Diffusion
Diffusions effectively interact two aspects of information, i. e., localized and holistic, for more powerful way of representation learning.
Ranked #10 on
Action Recognition
on UCF101
Transferrable Prototypical Networks for Unsupervised Domain Adaptation
Specifically, we present Transferrable Prototypical Networks (TPN) for adaptation such that the prototypes for each class in source and target domains are close in the embedding space and the score distributions predicted by prototypes separately on source and target data are similar.
Exploring Object Relation in Mean Teacher for Cross-Domain Detection
The whole architecture is then optimized with three consistency regularizations: 1) region-level consistency to align the region-level predictions between teacher and student, 2) inter-graph consistency for matching the graph structures between teacher and student, and 3) intra-graph consistency to enhance the similarity between regions of same class within the graph of student.
On the Selection of Anchors and Targets for Video Hyperlinking
A principle way of hyperlinking can be carried out by picking centers of clusters as anchors and from there reach out to targets within or outside of clusters with consideration of neighborhood complexity.
Approximate k-NN Graph Construction: a Generic Online Approach
On the one hand, the approximate k-nearest neighbor graph construction is treated as a search task.
McKernel: A Library for Approximate Kernel Expansions in Log-linear Time
The algorithm requires to compute the product of Walsh Hadamard Transform (WHT) matrices.
Boost K-Means
The performance of k-means has been enhanced from different perspectives over the years.
Learning Query and Image Similarities With Ranking Canonical Correlation Analysis
One of the fundamental problems in image search is to learn the ranking functions, i. e., similarity between the query and image.
Semi-Supervised Domain Adaptation With Subspace Learning for Visual Recognition
In many real-world applications, we are often facing the problem of cross domain learning, i. e., to borrow the labeled data or transfer the already learnt knowledge from a source domain to a target domain.
