Search Results for author:
Found 18 papers, 13 papers with code
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark.
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach.
Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
Finally, our framework uses these task rewards to fine-tune the entire VLM with RL.
Ctrl123: Consistent Novel View Synthesis via Closed-Loop Transcription
This consequently limits the performance of downstream tasks, such as image-to-multiview generation and 3D reconstruction.
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
To understand the roots of these errors, we explore the gap between the visual embedding space of CLIP and vision-only self-supervised learning.
White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?
This leads to a family of white-box transformer-like deep network architectures, named CRATE, which are mathematically fully interpretable.
Investigating the Catastrophic Forgetting in Multimodal Large Language Models
However, catastrophic forgetting, a notorious phenomenon where the fine-tuned model fails to retain similar performance compared to the pre-trained model, still remains an inherent problem in multimodal LLMs (MLLM).
Emergence of Segmentation with Minimalistic White-Box Transformers
Transformer-like models for vision tasks have recently proven effective for a wide range of downstream applications such as segmentation and detection.
Mass-Producing Failures of Multimodal Systems with Language Models
Because CLIP is the backbone for most state-of-the-art multimodal systems, these inputs produce failures in Midjourney 5. 1, DALL-E, VideoFusion, and others.
Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models
In this paper, we propose a novel image clustering pipeline that leverages the powerful feature representation of large pre-trained models such as CLIP and cluster images effectively and efficiently at scale.
White-Box Transformers via Sparse Rate Reduction
Particularly, we show that the standard transformer block can be derived from alternating optimization on complementary parts of this objective: the multi-head self-attention operator can be viewed as a gradient descent step to compress the token sets by minimizing their lossy coding rate, and the subsequent multi-layer perceptron can be viewed as attempting to sparsify the representation of the tokens.
EMP-SSL: Towards Self-Supervised Learning in One Training Epoch
Recently, self-supervised learning (SSL) has achieved tremendous success in learning image representation.
Closed-Loop Transcription via Convolutional Sparse Coding
Our method is arguably the first to demonstrate that a concatenation of multiple convolution sparse coding/decoding layers leads to an interpretable and effective autoencoder for modeling the distribution of large-scale natural image datasets.
Unsupervised Manifold Linearizing and Clustering
We consider the problem of simultaneously clustering and learning a linear representation of data lying close to a union of low-dimensional manifolds, a fundamental task in machine learning and computer vision.
Unsupervised Learning of Structured Representations via Closed-Loop Transcription
This paper proposes an unsupervised method for learning a unified representation that serves both discriminative and generative purposes.
Revisiting Sparse Convolutional Model for Visual Recognition
We show that such models have equally strong empirical performance on CIFAR-10, CIFAR-100, and ImageNet datasets when compared to conventional neural networks.
Incremental Learning of Structured Memory via Closed-Loop Transcription
Our method is simpler than existing approaches for incremental learning, and more efficient in terms of model size, storage, and computation: it requires only a single, fixed-capacity autoencoding network with a feature space that is used for both discriminative and generative purposes.
Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction
In particular, we propose to learn a closed-loop transcription between a multi-class multi-dimensional data distribution and a linear discriminative representation (LDR) in the feature space that consists of multiple independent multi-dimensional linear subspaces.
