Search Results for author:
Found 29 papers, 15 papers with code
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
We present MM1. 5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning.
Ranked #55 on
Visual Question Answering
on MM-Vet
Apple Intelligence Foundation Language Models
We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large server-based language model designed for Private Cloud Compute.
Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training
In this work, we revisit the settings by adopting step time as a more accurate measure of model complexity, and by determining the total compute budget under the Chinchilla compute-optimal settings.
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons.
Ranked #72 on
Visual Question Answering
on MM-Vet
Ferret: Refer and Ground Anything Anywhere at Any Granularity
We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions.
VeCLIP: Improving CLIP Training via Visual-enriched Captions
For example, VeCLIP achieves up to +25. 2% gain in COCO and Flickr30k retrieval tasks under the 12M setting.
Compressing LLMs: The Truth is Rarely Pure and Never Simple
Recently, several works have shown significant success in training-free and data-free compression (pruning and quantization) of LLMs that achieve 50 - 60% sparsity and reduce the bit width to 3 or 4 bits per weight, with negligible degradation of perplexity over the uncompressed baseline.
Guiding Instruction-based Image Editing via Multimodal Large Language Models
Extensive experimental results demonstrate that expressive instructions are crucial to instruction-based image editing, and our MGIE can lead to a notable improvement in automatic metrics and human evaluation while maintaining competitive inference efficiency.
Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts
We empirically show that our sparse Mobile Vision MoEs (V-MoEs) can achieve a better trade-off between performance and efficiency than the corresponding dense ViTs.
MOFI: Learning Image Representations from Noisy Entity Annotated Images
Our approach involves employing a named entity recognition model to extract entities from the alt-text, and then using a CLIP model to select the correct entities as labels of the paired image.
Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustness
In this paper, we discuss two effective approaches to improve the efficiency and robustness of CLIP training: (1) augmenting the training dataset while maintaining the same number of optimization steps, and (2) filtering out samples that contain text regions in the image.
AdaMV-MoE: Adaptive Multi-Task Vision Mixture-of-Experts
Instead of compressing multiple tasks' knowledge into a single model, MoE separates the parameter space and only utilizes the relevant model pieces given task type and its input, which provides stabilized MTL training and ultra-efficient inference.
Optimizing Anchor-based Detectors for Autonomous Driving Scenes
This paper summarizes model improvements and inference-time optimizations for the popular anchor-based detectors in the scenes of autonomous driving.
Auto-scaling Vision Transformers without Training
The motivation comes from two pain spots: 1) the lack of efficient and principled methods for designing and scaling ViTs; 2) the tremendous computational cost of training ViT that is much heavier than its convolution counterpart.
Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance
In this paper, we study contrastive learning from an optimization perspective, aiming to analyze and address a fundamental issue of existing contrastive learning methods that either rely on a large batch size or a large dictionary of feature vectors.
A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation
In this paper, we comprehensively study three architecture design choices on ViT -- spatial reduction, doubled channels, and multiscale features -- and demonstrate that a vanilla ViT architecture can fulfill this goal without handcrafting multiscale features, maintaining the original ViT design philosophy.
Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text
The experiments show that the resultant unified foundation transformer works surprisingly well on both the vision-only and text-only tasks, and the proposed knowledge distillation and gradient masking strategy can effectively lift the performance to approach the level of separately-trained models.
Revisiting 3D ResNets for Video Recognition
A recent work from Bello shows that training and scaling strategies may be more significant than model architectures for visual recognition.
Ranked #40 on
Action Classification
on Kinetics-600
Simple Training Strategies and Model Scaling for Object Detection
We benchmark these improvements on the vanilla ResNet-FPN backbone with RetinaNet and RCNN detectors.
Ranked #57 on
Object Detection
on COCO minival
Dilated SpineNet for Semantic Segmentation
Scale-permuted networks have shown promising results on object bounding box detection and instance segmentation.
Ranked #5 on
Semantic Segmentation
on PASCAL VOC 2012 val
Revisiting ResNets: Improved Training and Scaling Strategies
Using improved training and scaling strategies, we design a family of ResNet architectures, ResNet-RS, which are 1. 7x - 2. 7x faster than EfficientNets on TPUs, while achieving similar accuracies on ImageNet.
Efficient Scale-Permuted Backbone with Learned Resource Distribution
Furthermore, SpineNet is built with a uniform resource distribution over operations.
SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization
We propose SpineNet, a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search.
Ranked #14 on
Image Classification
on iNaturalist
TW-SMNet: Deep Multitask Learning of Tele-Wide Stereo Matching
In this paper, we introduce the problem of estimating the real world depth of elements in a scene captured by two cameras with different field of views, where the first field of view (FOV) is a Wide FOV (WFOV) captured by a wide angle lens, and the second FOV is contained in the first FOV and is captured by a tele zoom lens.
AMNet: Deep Atrous Multiscale Stereo Disparity Estimation Networks
A stacked atrous multiscale network is proposed to aggregate rich multiscale contextual information from the cost volume which allows for estimating the disparity with high accuracy at multiple scales.
Fused Deep Neural Networks for Efficient Pedestrian Detection
The classification system further classifies the generated candidates based on opinions of multiple deep verification networks and a fusion network which utilizes a novel soft-rejection fusion method to adjust the confidence in the detection results.
Boundary-sensitive Network for Portrait Segmentation
Compared to the general semantic segmentation problem, portrait segmentation has higher precision requirement on boundary area.
Fused DNN: A deep neural network fusion approach to fast and robust pedestrian detection
A single shot deep convolutional network is trained as a object detector to generate all possible pedestrian candidates of different sizes and occlusions.
Ranked #21 on
Pedestrian Detection
on Caltech
A Graphical Model Approach for Matching Partial Signatures
In this paper, we present a novel partial signature matching method using graphical models.
