Search Results for author:
Found 24 papers, 12 papers with code
FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations
Visual understanding is inherently contextual -- what we focus on in an image depends on the task at hand.
TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
Large Language Models (LLMs) trained on historical web data inevitably become outdated.
Mutual Reinforcement of LLM Dialogue Synthesis and Summarization Capabilities for Few-Shot Dialogue Summarization
In this work, we propose Mutual Reinforcing Data Synthesis (MRDS) within LLMs to improve few-shot dialogue summarization task.
FastVLM: Efficient Vision Encoding for Vision Language Models
At different operational resolutions, the vision encoder of a VLM can be optimized along two axes: reducing encoding latency and minimizing the number of visual tokens passed to the LLM, thereby lowering overall latency.
Promoting cross-modal representations to improve multimodal foundation models for physiological signals
However, methods for developing foundation models in healthcare are still in early exploration and it is unclear which pretraining strategies are most effective given the diversity of physiological signals.
Generalizable autoregressive modeling of time series through functional narratives
In this work, we propose a novel objective for transformers that learn time series by re-interpreting them as temporal functions.
MUSCLE: A Model Update Strategy for Compatible LLM Evolution
We propose a training strategy to minimize the extent of instance regression in model updates, involving training of a compatibility adapter that can enhance task fine-tuned language models.
DataComp-LM: In search of the next generation of training sets for language models
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models.
CLIP with Quality Captions: A Strong Pretraining for Vision Tasks
In this work, we find that simply improving the quality of captions in image-text datasets improves the quality of CLIP's visual representations, resulting in significant improvement on downstream dense prediction vision tasks.
Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models
Motivated by this, we ask the following important question, "How can we leverage the knowledge from a large VFM to train a small task-specific model for a new target task with limited labeled training data?
MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
We further demonstrate the effectiveness of our multi-modal reinforced training by training a CLIP model based on ViT-B/16 image backbone and achieving +2. 9% average performance improvement on 38 evaluation benchmarks compared to the previous best.
TiC-CLIP: Continual Training of CLIP Models
We introduce the first set of web-scale Time-Continual (TiC) benchmarks for training vision-language models: TiC-DataComp, TiC-YFCC, and TiC-Redcaps.
SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding
By applying our method to SAM and CLIP, we obtain SAM-CLIP: a unified model that combines the capabilities of SAM and CLIP into a single vision transformer.
CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
While CLIP is scalable, promptable, and robust to distribution shifts on image classification tasks, it lacks object localization capabilities.
Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals
To achieve effective pretraining in the presence of potential distributional shifts, we propose a frequency-aware masked autoencoder ($\texttt{bio}$FAME) that learns to parameterize the representation of biosignals in the frequency space.
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement
Models pretrained on ImageNet+ and fine-tuned on CIFAR-100+, Flowers-102+, and Food-101+, reach up to 3. 4% improved accuracy.
FastFill: Efficient Compatible Model Update
The task of retrieving the most similar data from a gallery set to a given query data is performed through a similarity comparison on features.
APE: Aligning Pretrained Encoders to Quickly Learn Aligned Multimodal Representations
Recent advances in learning aligned multimodal representations have been primarily driven by training large neural networks on massive, noisy paired-modality datasets.
Forward Compatible Training for Large-Scale Embedding Retrieval Systems
To avoid the cost of backfilling, BCT modifies training of the new model to make its representations compatible with those of the old model.
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution
We propose extracurricular learning, a novel knowledge distillation method, that bridges this gap by (1) modeling student and teacher output distributions; (2) sampling examples from an approximation to the underlying data distribution; and (3) matching student and teacher output distributions over this extended set including uncertain samples.
Least squares binary quantization of neural networks
We conduct experiments on the ImageNet dataset and show a reduced accuracy gap when using the proposed least squares quantization algorithms.
OPTIMAL BINARY QUANTIZATION FOR DEEP NEURAL NETWORKS
We conduct experiments on the ImageNet dataset and show a reduced accuracy gap when using the proposed optimal quantization algorithms.
Democratizing Production-Scale Distributed Deep Learning
The interest and demand for training deep neural networks have been experiencing rapid growth, spanning a wide range of applications in both academia and industry.
