Search Results for author:
Found 46 papers, 19 papers with code
NVILA: Efficient Frontier Visual Language Models
This paper introduces NVILA, a family of open VLMs designed to optimize both efficiency and accuracy.
Ranked #5 on
Video Question Answering
on NExT-QA
NaVILA: Legged Robot Vision-Language-Action Model for Navigation
This paper proposes to solve the problem of Vision-and-Language Navigation with legged robots, which not only provides a flexible way for humans to command but also allows the robot to navigate through more challenging and cluttered scenes.
VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge
In contrast, we propose that for medical VLMs, a fourth stage of specialized IFT is necessary, which focuses on medical data and includes information from domain expert models.
EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation
In this work, we re-formulate the model compression problem into the customized compensation problem: Given a compressed model, we aim to introduce residual low-rank paths to compensate for compression errors under customized requirements from users (e. g., tasks, compression ratios), resulting in greater flexibility in adjusting overall capacity without being constrained by specific compression formats.
MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models
This approach facilitates end-to-end training on large-scale datasets and offers two notable advantages: 1) High-quality Masks - our method effectively scales to large datasets and learns accurate masks; 2) Transferability - the probabilistic modeling of mask distribution enables the transfer learning of sparsity across domains or tasks.
VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation
VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation.
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs).
LongVILA: Scaling Long-Context Visual Language Models for Long Videos
We introduce the long-context Multi-Modal Sequence Parallelism (MM-SP) system that efficiently parallelizes long video training and inference, enabling 2M context length training on 256 GPUs without any gradient checkpointing.
Ranked #8 on
Video Question Answering
on NExT-QA
VILA$^2$: VILA Augmented VILA
In the self-augment step, the instruction-finetuned VLM recaptions its pretraining caption datasets and then retrains from scratch leveraging refined data.
Ranked #63 on
Visual Question Answering
on MM-Vet
Flextron: Many-in-One Flexible Large Language Model
Training modern LLMs is extremely resource intensive, and customizing them for various deployment scenarios characterized by limited compute and memory resources through repeated training is impractical.
Step Out and Seek Around: On Warm-Start Training with Incremental Data
First, a novel feature regularization (FeatReg) to retain and refine knowledge from existing checkpoints; Second, we propose adaptive knowledge distillation (AdaKD), a novel approach to forget mitigation and knowledge transfer.
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models
Vision Language Models (VLMs) have demonstrated remarkable performance in 2D vision and language tasks.
X-VILA: Cross-Modality Alignment for Large Language Model
We introduce X-VILA, an omni-modality model designed to extend the capabilities of large language models (LLMs) by incorporating image, video, and audio modalities.
LITA: Language Instructed Temporal-Localization Assistant
In addition to leveraging existing video datasets with timestamps, we propose a new task, Reasoning Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for learning and evaluating this task.
RegionGPT: Towards Region Understanding Vision Language Model
Vision language models (VLMs) have experienced rapid advancements through the integration of large language models (LLMs) with image-text pairs, yet they struggle with detailed regional visual understanding due to limited spatial awareness of the vision encoder, and the use of coarse-grained training data that lacks detailed, region-specific captions.
DoRA: Weight-Decomposed Low-Rank Adaptation
By employing \ours, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead.
Ranked #2 on
parameter-efficient fine-tuning
on WinoGrande
(using extra training data)
VILA: On Pre-training for Visual Language Models
Visual language models (VLMs) rapidly progressed with the recent success of large language models.
Ranked #4 on
Zero-Shot Video Question Answer
on MSVD-QA
FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models
However, applying FL to finetune PLMs is hampered by challenges, including restricted model parameter access, high computational requirements, and communication overheads.
Online Overexposed Pixels Hallucination in Videos with Adaptive Reference Frame Selection
We present a learning-based system to reduce these artifacts without resorting to complex acquisition mechanisms like alternating exposures or costly processing that are typical of high dynamic range (HDR) imaging.
Adaptive Sharpness-Aware Pruning for Robust Sparse Networks
Robustness and compactness are two essential attributes of deep learning models that are deployed in the real world.
Heterogeneous Continual Learning
We propose a novel framework and a solution to tackle the continual learning (CL) problem with changing network architectures.
FasterViT: Fast Vision Transformers with Hierarchical Attention
At a high level, global self-attentions enable the efficient cross-window communication at lower costs.
Ranked #188 on
Image Classification
on ImageNet
Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Models
Our Landmark DEQ (LDEQ) achieves state-of-the-art performance on the challenging WFLW facial landmark dataset, reaching $3. 92$ NME with fewer parameters and a training memory cost of $\mathcal{O}(1)$ in the number of recurrent modules.
Ranked #2 on
Face Alignment
on WFLW
Structural Pruning via Latency-Saliency Knapsack
We propose Hardware-Aware Latency Pruning (HALP) that formulates structural pruning as a global resource allocation optimization problem, aiming at maximizing the accuracy while constraining latency under a predefined budget on targeting device.
Global Context Vision Transformers
Pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation using MS COCO and ADE20K datasets outperform prior work consistently.
Ranked #135 on
Semantic Segmentation
on ADE20K
GradViT: Gradient Inversion of Vision Transformers
In this work we demonstrate the vulnerability of vision transformers (ViTs) to gradient-based inversion attacks.
Do Gradient Inversion Attacks Make Federated Learning Unsafe?
Federated learning (FL) allows the collaborative training of AI models without needing to share raw data.
AdaViT: Adaptive Tokens for Efficient Vision Transformer
A-ViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds.
Ranked #34 on
Efficient ViTs
on ImageNet-1K (with DeiT-S)
When to Prune? A Policy towards Early Structural Pruning
Through extensive experiments on ImageNet, we show that EPI empowers a quick tracking of early training epochs suitable for pruning, offering same efficacy as an otherwise ``oracle'' grid-search that scans through epochs and requires orders of magnitude more compute.
HALP: Hardware-Aware Latency Pruning
We propose Hardware-Aware Latency Pruning (HALP) that formulates structural pruning as a global resource allocation optimization problem, aiming at maximizing the accuracy while constraining latency under a predefined budget.
Global Vision Transformer Pruning with Hessian-Aware Saliency
This work aims on challenging the common design philosophy of the Vision Transformer (ViT) model with uniform dimension across all the stacked blocks in a model stage, where we redistribute the parameters both across transformer blocks and between different structures within the block via the first systematic attempt on global structural pruning.
Hardware-Aware Network Transformation
In the second phase, it solves the combinatorial selection of efficient operations using a novel constrained integer linear optimization approach.
Privacy Vulnerability of Split Computing to Data-Free Model Inversion Attacks
Prior works usually assume that SC offers privacy benefits as only intermediate features, instead of private data, are shared from devices to the cloud.
LANA: Latency Aware Network Acceleration
We analyze three popular network architectures: EfficientNetV1, EfficientNetV2 and ResNeST, and achieve accuracy improvement for all models (up to $3. 0\%$) when compressing larger models to the latency level of smaller models.
Optimal Quantization Using Scaled Codebook
We study the problem of quantizing N sorted, scalar datapoints with a fixed codebook containing K entries that are allowed to be rescaled.
See through Gradients: Image Batch Recovery via GradInversion
In this work, we introduce GradInversion, using which input images from a larger batch (8 - 48 images) can also be recovered for large networks such as ResNets (50 layers), on complex datasets such as ImageNet (1000 classes, 224x224 px).
MHDeep: Mental Health Disorder Detection System based on Body-Area and Deep Neural Networks
At the patient level, MHDeep DNNs achieve an accuracy of 100%, 100%, and 90. 0% for the three mental health disorders, respectively.
Fully Dynamic Inference with Deep Neural Networks
Modern deep neural networks are powerful and widely applicable models that extract task-relevant information through multi-level abstraction.
Efficient Synthesis of Compact Deep Neural Networks
These large, deep models are often unsuitable for real-world applications, due to their massive computational cost, high memory bandwidth, and long latency.
Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
We introduce DeepInversion, a new method for synthesizing images from the image distribution used to train a deep neural network.
DiabDeep: Pervasive Diabetes Diagnosis based on Wearable Medical Sensors and Efficient Neural Networks
For server (edge) side inference, we achieve a 96. 3% (95. 3%) accuracy in classifying diabetics against healthy individuals, and a 95. 7% (94. 6%) accuracy in distinguishing among type-1/type-2 diabetic, and healthy individuals.
Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks
Deep neural networks (DNNs) have become a widely deployed model for numerous machine learning applications.
Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM
In this work, we propose a hardware-guided symbiotic training methodology for compact, accurate, yet execution-efficient inference models.
ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
We formulate platform-aware NN architecture search in an optimization framework and propose a novel algorithm to search for optimal architectures aided by efficient accuracy and resource (latency and/or energy) predictors.
Grow and Prune Compact, Fast, and Accurate LSTMs
To address these problems, we propose a hidden-layer LSTM (H-LSTM) that adds hidden layers to LSTM's original one level non-linear control gates.
NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
To address these problems, we introduce a network growth algorithm that complements network pruning to learn both weights and compact DNN architectures during training.
