Search Results for author:
Found 28 papers, 18 papers with code
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision.
Ranked #3 on
Open Vocabulary Attribute Detection
on OVAD-Box benchmark
(using extra training data)
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models.
Ranked #1 on
Image Retrieval
on Flickr30k
LAVIS: A Library for Language-Vision Intelligence
We introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications.
From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models
To address this issue, we propose \emph{Img2Prompt}, a plug-and-play module that provides the prompts that can bridge the aforementioned modality and task disconnections, so that LLMs can perform zero-shot VQA tasks without end-to-end training.
BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing
Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions.
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence.
Ranked #5 on
visual instruction following
on LLaVA-Bench
Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions
This work presents Moonshot, a new video generation model that conditions simultaneously on multimodal inputs of image and text.
Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison
Based on this new large-scale dataset, we are able to experiment with several deep learning methods for word-level sign recognition and evaluate their performances in large scale scenarios.
Ranked #3 on
Sign Language Recognition
on WLASL100
TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation
Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences.
Align and Prompt: Video-and-Language Pre-training with Entity Prompts
To achieve this, we first introduce an entity prompter module, which is trained with VTC to produce the similarity between a video crop and text prompts instantiated with entity names.
Ranked #19 on
Zero-Shot Video Retrieval
on DiDeMo
cosFormer: Rethinking Softmax in Attention
As one of its core components, the softmax attention helps to capture long-range dependencies yet prohibits its scale-up due to the quadratic space and time complexity to the sequence length.
Ranked #4 on
Offline RL
on D4RL
EDFace-Celeb-1M: Benchmarking Face Hallucination with a Million-scale Dataset
It is thus unclear how these algorithms perform on public face hallucination datasets.
Ranked #1 on
Image Super-Resolution
on WLFW
Toeplitz Neural Network for Sequence Modeling
Sequence modeling has important applications in natural language processing and computer vision.
The Devil in Linear Transformer
In this paper, we examine existing kernel-based linear transformers and identify two key issues that lead to such performance gaps: 1) unbounded gradients in the attention computation adversely impact the convergence of linear transformer models; 2) attention dilution which trivially distributes attention scores over long sequences while neglecting neighbouring structures.
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning
Vision-language pre-training and instruction tuning have demonstrated general-purpose capabilities in 2D visual reasoning tasks by aligning visual encoders with state-of-the-art large language models (LLMs).
TODE-Trans: Transparent Object Depth Estimation with Transformer
We observe that the global characteristics of the transformer make it easier to extract contextual information to perform depth estimation of transparent areas.
Enhanced Spatio-Temporal Interaction Learning for Video Deraining: A Faster and Better Framework
Video deraining is an important task in computer vision as the unwanted rain hampers the visibility of videos and deteriorates the robustness of most outdoor vision systems.
Transcribing Natural Languages for The Deaf via Neural Editing Programs
This work studies the task of glossification, of which the aim is to em transcribe natural spoken language sentences for the Deaf (hard-of-hearing) community to ordered sign language glosses.
Transferring Cross-domain Knowledge for Video Sign Language Recognition
To this end, we extract news signs using a base WSLR model, and then design a classifier jointly trained on news and isolated signs to coarsely align these two domain features.
ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring
Video deblurring models exploit consecutive frames to remove blurs from camera shakes and object motions.
Dual Attention-in-Attention Model for Joint Rain Streak and Raindrop Removal
In addition, to further refine the result, a Differential-driven Dual Attention-in-Attention Model (D-DAiAM) is proposed with a "heavy-to-light" scheme to remove rain via addressing the unsatisfying deraining regions.
Automatic Gloss Dictionary for Sign Language Learners
A multi-language dictionary is a fundamental tool for language learning, allowing the learner to look up unfamiliar words.
PCRED: Zero-shot Relation Triplet Extraction with Potential Candidate Relation Selection and Entity Boundary Detection
To address the above issues, we propose a novel method named PCRED for ZeroRTE with Potential Candidate Relation Selection and Entity Boundary Detection.
Joint Task and Data Oriented Semantic Communications: A Deep Separate Source-channel Coding Scheme
This paper proposes a deep separate source-channel coding (DSSCC) framework for the joint task and data oriented semantic communications (JTD-SC) and utilizes the variational autoencoder approach to solve the rate-distortion problem with semantic distortion.
From Images to Textual Prompts: Zero-Shot Visual Question Answering With Frozen Large Language Models
To address this issue, we propose Img2Prompt, a plug-and-play module that provides the prompts that can bridge the aforementioned modality and task disconnections, so that LLMs can perform zero-shot VQA tasks without end-to-end training.
Linearized Relative Positional Encoding
Meanwhile, it emphasizes a general paradigm for designing broadly more relative positional encoding methods that are applicable to linear transformers.
Fundamental Limitation of Semantic Communications: Neural Estimation for Rate-Distortion
For the case with unknown semantic source distribution, while only a set of the source samples is available, we propose a neural-network-based method by leveraging the generative networks to learn the semantic source distribution.
