Search Results for author:
Found 41 papers, 27 papers with code
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons.
Ranked #18 on
Visual Question Answering
on MM-Vet
Improving Language Understanding from Screenshots
An emerging family of language models (LMs), capable of processing both text and images within a single visual view, has the promise to unlock complex tasks such as chart understanding and UI navigation.
Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response
Robust locomotion control depends on accurate state estimations.
Ferret: Refer and Ground Anything Anywhere at Any Granularity
We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions.
A Read Margin Enhancement Circuit with Dynamic Bias Optimization for MRAM
This brief introduces a read bias circuit to improve readout yield of magnetic random access memories (MRAMs).
Guiding Image Captioning Models Toward More Specific Captions
We further explore the use of language models to guide the decoding process, obtaining small improvements over the Pareto frontier of reference-free vs. reference-based captioning metrics that arises from classifier-free guidance, and substantially improving the quality of captions generated from a model trained only on minimally curated web data.
SHISRCNet: Super-resolution And Classification Network For Low-resolution Breast Cancer Histopathology Image
CF module extracts and fuses the multi-scale features of SR images for classification.
Histopathological Image Classification
Image Classification
+1
Language Models Meet World Models: Embodied Experiences Enhance Language Models
While large language models (LMs) have shown remarkable capabilities across numerous tasks, they often struggle with simple reasoning and planning in physical environments, such as understanding object permanence or planning household activities.
MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts
Vision-language pre-training (VLP) models have been demonstrated to be effective in many computer vision applications.
PaLM 2 Technical Report
Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.
Ranked #1 on
Question Answering
on StrategyQA
Neural Refinement for Absolute Pose Regression with Feature Synthesis
To enhance the robustness of our model, we introduce a feature fusion module and a progressive training strategy.
NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior
Recent advances in this direction demonstrate the possibility of jointly optimising a NeRF and camera poses in forward-facing scenes.
REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory
REVEAL consists of four key components: the memory, the encoder, the retriever and the generator.
Ranked #9 on
Visual Question Answering (VQA)
on OK-VQA
VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners
We explore an efficient approach to establish a foundational video-text model.
Ranked #1 on
Video Captioning
on ActivityNet Captions
(using extra training data)
Generative Data Augmentation for Non-IID Problem in Decentralized Clinical Machine Learning
To address this problem, we propose a generative augmentation framework in swarm learning called SL-GAN, which augments the non-IID data by generating the synthetic data from participants.
Exploiting Category Names for Few-Shot Classification with Vision-Language Models
Vision-language foundation models pretrained on large-scale data provide a powerful tool for many visual understanding tasks.
On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning
Reinforcement Learning (RL) algorithms can solve challenging control problems directly from image observations, but they often require millions of environment interactions to do so.
AS-IntroVAE: Adversarial Similarity Distance Makes Robust IntroVAE
However, due to the unavailability of an effective metric to evaluate the difference between the real and the fake images, the posterior collapse and the vanishing gradient problem still exist, reducing the fidelity of the synthesized images.
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge.
Ranked #1 on
Text-to-Image Generation
on LAION COCO
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
CoCa: Contrastive Captioners are Image-Text Foundation Models
We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively.
Ranked #1 on
Visual Question Answering
on VQA v2 test-dev
DFNet: Enhance Absolute Pose Regression with Direct Feature Matching
We introduce a camera relocalization pipeline that combines absolute pose regression (APR) and direct feature matching.
HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Images
Multiple medical institutions collaboratively training a model using federated learning (FL) has become a promising solution for maximizing the potential of data-driven models, yet the non-independent and identically distributed (non-iid) data in medical images is still an outstanding challenge in real-world practice.
Towards Zero-Label Language Learning
This paper explores zero-label learning in Natural Language Processing (NLP), whereby no human-annotated data is used anywhere during training and models are trained purely on synthetic data.
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks.
Ranked #4 on
Visual Entailment
on SNLI-VE val
Ray-ONet: Efficient 3D Reconstruction From A Single RGB Image
We propose Ray-ONet to reconstruct detailed 3D models from monocular images efficiently.
NTIRE 2021 Challenge on Perceptual Image Quality Assessment
This paper reports on the NTIRE 2021 challenge on perceptual image quality assessment (IQA), held in conjunction with the New Trends in Image Restoration and Enhancement workshop (NTIRE) workshop at CVPR 2021.
Direct-PoseNet: Absolute Pose Regression with Photometric Consistency
We present a relocalization pipeline, which combines an absolute pose regression (APR) network with a novel view synthesis based direct matching module, offering superior accuracy while maintaining low inference time.
NeRF--: Neural Radiance Fields Without Known Camera Parameters
Considering the problem of novel view synthesis (NVS) from only a set of 2D images, we simplify the training process of Neural Radiance Field (NeRF) on forward-facing scenes by removing the requirement of known or pre-computed camera parameters, including both intrinsics and 6DoF poses.
Temporal Cue Guided Video Highlight Detection With Low-Rank Audio-Visual Fusion
Video highlight detection plays an increasingly important role in social media content filtering, however, it remains highly challenging to develop automated video highlight detection methods because of the lack of temporal annotations (i. e., where the highlight moments are in long videos) for supervised learning.
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
Massively multilingual models subsuming tens or even hundreds of languages pose great challenges to multi-task optimization.
On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages.
Efficient Meta Lifelong-Learning with Limited Memory
State-of-the-art lifelong language learning methods store past examples in episodic memory and replay them at both training and inference time.
Neighbourhood-Insensitive Point Cloud Normal Estimation Network
We introduce a novel self-attention-based normal estimation network that is able to focus softly on relevant points and adjust the softness by learning a temperature parameter, making it able to work naturally and effectively within a large neighbourhood range.
FlowNet3D++: Geometric Losses For Deep Scene Flow Estimation
We present FlowNet3D++, a deep scene flow estimation network.
Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework
Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks.
Characterizing and Avoiding Negative Transfer
When labeled data is scarce for a specific target task, transfer learning often offers an effective solution by utilizing data from a related source task.
Theoretical Guarantees of Transfer Learning
Transfer learning has been proven effective when within-target labeled data is scarce.
Towards more Reliable Transfer Learning
Multi-source transfer learning has been proven effective when within-target labeled data is scarce.
