Search Results for author:
Found 25 papers, 14 papers with code
ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models
Our multi-image instruction data leads to an 8% improvement on Mantis-Eval.
Ranked #105 on
Visual Question Answering
on MM-Vet
TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action
While open-source multi-modal language models perform well on simple question answering tasks, they often fail on complex questions that require multiple capabilities, such as fine-grained recognition, visual grounding, and reasoning, and that demand multi-step solutions.
Ranked #61 on
Visual Question Answering
on MM-Vet
BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions
We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that bridges the gap between descriptive synthetic captions and factual web-scale alt-text.
xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs
We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for videos, particularly designed to efficiently capture temporal information over multiple frames.
xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations
We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions.
xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs.
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
Multimodal interleaved datasets featuring free-form interleaved sequences of images and text are crucial for training frontier large multimodal models (LMMs).
Coercing LLMs to do and reveal (almost) anything
It has recently been shown that adversarial attacks on large language models (LLMs) can "jailbreak" the model into making harmful statements.
Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models
Vision-Language Models (VLMs) excel in generating textual responses from visual inputs, but their versatility raises security concerns.
Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks
Battle of the Backbones (BoB) makes this choice easier by benchmarking a diverse suite of pretrained models, including vision-language models, those trained via self-supervised learning, and the Stable Diffusion backbone, across a diverse set of computer vision tasks ranging from classification to object detection to OOD generalization and more.
On the Exploitability of Instruction Tuning
In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavior.
Bring Your Own Data! Self-Supervised Evaluation for Large Language Models
With the rise of Large Language Models (LLMs) and their ubiquitous deployment in diverse domains, measuring language model behavior on realistic data is imperative.
On the Reliability of Watermarks for Large Language Models
We also consider a range of new detection schemes that are sensitive to short spans of watermarked text embedded inside a large document, and we compare the robustness of watermarking to other kinds of detectors.
Hierarchical Point Attention for Indoor 3D Object Detection
By plugging our proposed modules into the state-of-the-art transformer-based 3D detectors, we improve the previous best results on both benchmarks, with more significant improvements on smaller objects.
What do Vision Transformers Learn? A Visual Exploration
In addition, we show that ViTs maintain spatial information in all layers except the final layer.
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data.
Gradient-Free Adversarial Training Against Image Corruption for Learning-based Steering
We introduce a simple yet effective framework for improving the robustness of learning algorithms against image corruptions for autonomous driving.
Contrastive Learning is Just Meta-Learning
Contrastive learning has recently taken off as a paradigm for learning from unlabeled data.
Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability
We find that samples which cause similar parameters to malfunction are semantically similar.
Improving Robustness of Learning-based Autonomous Steering Using Adversarial Images
For safety of autonomous driving, vehicles need to be able to drive under various lighting, weather, and visibility conditions in different environments.
Driving through the Lens: Improving Generalization of Learning-based Steering using Simulated Adversarial Examples
To ensure the wide adoption and safety of autonomous driving, the vehicles need to be able to drive under various lighting, weather, and visibility conditions in different environments.
Towards Accurate Quantization and Pruning via Data-free Knowledge Transfer
Further, we demonstrate that the compact structure and corresponding initialization from the Lottery Ticket Hypothesis can also help in data-free training.
Prepare for the Worst: Generalizing across Domain Shifts with Adversarial Batch Normalization
Adversarial training is the industry standard for producing models that are robust to small adversarial perturbations.
Encoding Robustness to Image Style via Adversarial Feature Perturbations
We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce models that are robust to various unseen distributional shifts.
Headless Horseman: Adversarial Attacks on Transfer Learning Models
We first demonstrate successful transfer attacks against a victim network using \textit{only} its feature extractor.
