Search Results for author:
Found 74 papers, 30 papers with code
On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention
Large language models (LLMs) excel at capturing global token dependencies via self-attention but face prohibitive compute and memory costs on lengthy inputs.
Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection
In this paper, we propose an efficient method to perform PEFT within the FL framework for Multi-Head Attention (MHA) based language models.
Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits
To demonstrate real-world applicability, we deploy our method on the Unitree Go2 quadruped robot using Vision-Language Model (VLM) based control, achieving a 21% speedup over traditional cloud-based autoregressive decoding.
Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression
Despite their remarkable progress in multimodal understanding tasks, large vision language models (LVLMs) often suffer from "hallucinations", generating texts misaligned with the visual context.
Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator
FLOW enables the identification of optimal layer-wise N and M values (from a given range) by simultaneously accounting for the presence and distribution of outliers, allowing a higher degree of representational freedom.
SEAL: Steerable Reasoning Calibration of Large Language Models for Free
To address this issue, we investigate the internal reasoning structures of LLMs and categorize them into three primary thought types: execution, reflection, and transition thoughts.
OuroMamba: A Data-Free Quantization Framework for Vision Mamba Models
We identify two key challenges in enabling DFQ for VMMs, (1) VMM's recurrent state transitions restricts capturing of long-range interactions and leads to semantically weak synthetic data, (2) VMM activations exhibit dynamic outlier variations across time-steps, rendering existing static PTQ techniques ineffective.
Enhancing Large Language Models for Hardware Verification: A Novel SystemVerilog Assertion Dataset
However, proprietary SOTA models like GPT-4o often generate inaccurate assertions and require expensive licenses, while smaller open-source LLMs need fine-tuning to manage HDL code complexities.
LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression
Despite recent efforts in understanding the compression impact on large language models (LLMs) in terms of their downstream task performance and trustworthiness on relatively simpler uni-modal benchmarks (for example, question answering, common sense reasoning), their detailed study on multi-modal Large Vision-Language Models (LVLMs) is yet to be unveiled.
LANTERN++: Enhancing Relaxed Speculative Decoding with Static Tree Drafting for Visual Auto-regressive Models
Recently, relaxed speculative decoding with dynamic tree drafting was proposed to mitigate this ambiguity, demonstrating promising results in accelerating visual AR models.
CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing
This allows the router to learn to predict token-level routing scores and make routing decisions based on both the current token and the future impact of its decisions.
Unraveling Zeroth-Order Optimization through the Lens of Low-Dimensional Structured Perturbations
In this work, we develop a unified theoretical framework that analyzes both the convergence and generalization properties of ZO optimization under structured perturbations.
GenBFA: An Evolutionary Optimization Approach to Bit-Flip Attacks on LLMs
Current BFA techniques are inadequate for exploiting this vulnerability due to the difficulty of efficiently identifying critical parameters within the immense parameter space.
MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantization
Quantization of foundational models (FMs) is significantly more challenging than traditional DNNs due to the emergence of large magnitude values called outliers.
LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding
To overcome this challenge, we propose a relaxed acceptance condition referred to as LANTERN that leverages the interchangeability of tokens in latent space.
Understanding the Performance and Estimating the Cost of LLM Fine-Tuning
This model, based on parameters of the model and GPU architecture, estimates LLM throughput and the cost of training, aiding practitioners in industry and academia to budget the cost of fine-tuning a specific model.
MaskVD: Region Masking for Efficient Video Object Detection
Video tasks are compute-heavy and thus pose a challenge when deploying in real-time applications, particularly for tasks that require state-of-the-art Vision Transformers (ViTs).
Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems
However, these metrics fail to fully capture the nuances of LLM inference, leading to an incomplete assessment of user-facing performance crucial for real-time applications such as chat and translation.
CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTs
We present CLAMP-ViT, a data-free post-training quantization method for vision transformers (ViTs).
LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation
LaMDA freezes a first projection matrix (PMA) in the adaptation path while introducing a low-dimensional trainable square matrix, resulting in substantial reductions in trainable parameters and peak GPU memory usage.
ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization
Experiments on five LLM families and eight tasks consistently validate the effectiveness of ShiftAddLLM, achieving average perplexity improvements of 5. 6 and 22. 7 points at comparable or lower latency compared to the most competitive quantized LLMs at 3 and 2 bits, respectively, and more than 80% memory and energy reductions over the original LLMs.
Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
We have validated against real hardware platforms running various different LLM models, achieving a max geomean error of 5. 82. We use GenZ to identify compute, memory capacity, memory bandwidth, network latency, and network bandwidth requirements across diverse LLM inference use cases.
AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models
While compared in terms of runtime, AFLoRA can yield up to $1. 86\times$ improvement as opposed to similar PEFT alternatives.
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference.
CiMNet: Towards Joint Optimization for DNN Architecture and Configuration for Compute-In-Memory Hardware
With the recent growth in demand for large-scale deep neural networks, compute in-memory (CiM) has come up as a prominent solution to alleviate bandwidth and on-chip interconnect bottlenecks that constrain Von-Neuman architectures.
Linearizing Models for Efficient yet Robust Private Inference
Our experimental evaluations show that RLNet can yield models with up to 11. 14x fewer ReLUs, with accuracy close to the all-ReLU models, on clean, naturally perturbed, and gradient-based perturbed images.
Sparse but Strong: Crafting Adversarially Robust Graph Lottery Tickets
By iteratively applying ARGS to prune both the perturbed graph adjacency matrix and the GNN model weights, we can find adversarially robust graph lottery tickets that are highly sparse yet achieve competitive performance under different untargeted training-time structure attacks.
GenQ: Quantization in Low Data Regimes with Generative Synthetic Data
In the realm of deep neural network deployment, low-bit quantization presents a promising avenue for enhancing computational efficiency.
Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technology
Neuromorphic computing and, in particular, spiking neural networks (SNNs) have become an attractive alternative to deep neural networks for a broad range of signal processing applications, processing static and/or temporal inputs from different sensory modalities, including audio and vision sensors.
Fusing Models with Complementary Expertise
Training AI models that generalize across tasks and domains has long been among the open problems driving AI research.
Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs
Contrary to this belief, this paper presents a counter-argument: small-magnitude weights of pre-trained model weights encode vital knowledge essential for tackling difficult downstream tasks - manifested as the monotonic relationship between the performance drop of downstream tasks across the difficulty spectrum, as we prune more pre-trained weights by magnitude.
InstaTune: Instantaneous Neural Architecture Search During Fine-Tuning
However, training super-networks from scratch can be extremely time consuming and compute intensive especially for large models that rely on a two-stage training process of pre-training and fine-tuning.
FireFly A Synthetic Dataset for Ember Detection in Wildfire
This paper presents "FireFly", a synthetic dataset for ember detection created using Unreal Engine 4 (UE4), designed to overcome the current lack of ember-specific training resources.
Sensi-BERT: Towards Sensitivity Driven Fine-Tuning for Parameter-Efficient BERT
In this paper, we present Sensi-BERT, a sensitivity driven efficient fine-tuning of BERT models that can take an off-the-shelf pre-trained BERT model and yield highly parameter-efficient models for downstream tasks.
NeRFool: Uncovering the Vulnerability of Generalizable Neural Radiance Fields against Adversarial Perturbations
Generalizable Neural Radiance Fields (GNeRF) are one of the most promising real-world solutions for novel view synthesis, thanks to their cross-scene generalization capability and thus the possibility of instant rendering on new scenes.
Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inference
Large number of ReLU and MAC operations of Deep neural networks make them ill-suited for latency and compute-efficient private inference.
Technology-Circuit-Algorithm Tri-Design for Processing-in-Pixel-in-Memory (P2M)
The massive amounts of data generated by camera sensors motivate data processing inside pixel arrays, i. e., at the extreme-edge.
ViTA: A Vision Transformer Inference Accelerator for Edge Applications
Vision Transformer models, such as ViT, Swin Transformer, and Transformer-in-Transformer, have recently gained significant traction in computer vision tasks due to their ability to capture the global relation between features which leads to superior performance.
Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inference
For a similar ReLU budget SENet can yield models with ~2. 32% improved classification accuracy, evaluated on CIFAR-100.
SAL-ViT: Towards Latency Efficient Private Inference on ViT using Selective Attention Search with a Learnable Softmax Approximation
Then, given our observation that external attention (EA) presents lower PI latency than widely-adopted self-attention (SA) at the cost of accuracy, we present a selective attention search (SAS) method to integrate the strength of EA and SA.
Vision HGNN: An Image is More than a Graph of Nodes
In this paper, we enhance ViG by transcending conventional "pairwise" linkages and harnessing the power of the hypergraph to encapsulate image information.
Sparse Mixture Once-for-all Adversarial Training for Efficient In-Situ Trade-Off Between Accuracy and Robustness of DNNs
Existing deep neural networks (DNNs) that achieve state-of-the-art (SOTA) performance on both clean and adversarially-perturbed images rely on either activation or weight conditioned convolution operations.
In-Sensor & Neuromorphic Computing are all you need for Energy Efficient Computer Vision
Although the overhead for the first layer MACs with direct encoding is negligible for deep SNNs and the CV processing is efficient using SNNs, the data transfer between the image sensors and the downstream processing costs significant bandwidth and may dominate the total energy.
Self-Attentive Pooling for Efficient Deep Learning
With the aggressive down-sampling of the activation maps in the initial layers (providing up to 22x reduction in memory consumption), our approach achieves 1. 43% higher test accuracy compared to SOTA techniques with iso-memory footprints.
Dynamic Calibration of Nonlinear Sensors with Time-Drifts and Delays by Bayesian Inference
Most sensor calibrations rely on the linearity and steadiness of their response characteristics, but practical sensors are nonlinear, and their response drifts with time, restricting their choices for adoption.
Federated Learning of Large Models at the Edge via Principal Sub-Model Training
However, the heterogeneous-client setting requires some clients to train full model, which is not aligned with the resource-constrained setting; while the latter ones break privacy promises in FL when sharing intermediate representations or labels with the server.
Lottery Aware Sparsity Hunting: Enabling Federated Learning on Resource-Limited Edge
A possible solution to this problem is to utilize off-the-shelf sparse learning algorithms at the clients to meet their resource budget.
Implementation of fast ICA using memristor crossbar arrays for blind image source separations
The experimental results demonstrate that the proposed approach is very effective to separate image sources, and also the contrast of the images are improved with an improvement factor in terms of percentage of structural similarity as 67. 27% when compared with the software-based implementation of conventional ACY ICA and Fast ICA algorithms.
P2M-DeTrack: Processing-in-Pixel-in-Memory for Energy-efficient and Real-Time Multi-Object Detection and Tracking
The designs also reduce the sensor and total energy (obtained from in-house circuit simulations at Globalfoundries 22nm technology node) per frame by 5. 7x and 1. 14x, respectively.
Evaluation Benchmarks for Spanish Sentence Representations
Due to the success of pre-trained language models, versions of languages other than English have been released in recent years.
A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustness
In this paper, we present a fast learnable once-for-all adversarial training (FLOAT) algorithm, which instead of the existing FiLM-based conditioning, presents a unique weight conditioned learning that requires no additional layer, thereby incurring no significant increase in parameter count, training time, or network latency compared to standard adversarial training.
P2M: A Processing-in-Pixel-in-Memory Paradigm for Resource-Constrained TinyML Applications
Visual data in such cameras are usually captured in the form of analog voltages by a sensor pixel array, and then converted to the digital domain for subsequent AI processing using analog-to-digital converters (ADC).
BMPQ: Bit-Gradient Sensitivity Driven Mixed-Precision Quantization of DNNs from Scratch
Compared to the baseline FP-32 models, BMPQ can yield models that have 15. 4x fewer parameter bits with a negligible drop in accuracy.
Analyzing the Confidentiality of Undistillable Teachers in Knowledge Distillation
Knowledge distillation (KD) has recently been identified as a method that can unintentionally leak private information regarding the details of a teacher model to an unauthorized student.
Pipeline Parallelism for Inference on Heterogeneous Edge Computing
We propose EdgePipe, a distributed framework for edge systems that uses pipeline parallelism to both speed up inference and enable running larger (and more accurate) models that otherwise cannot fit on single edge devices.
Understanding of Emotion Perception from Art
Computational modeling of the emotions evoked by art in humans is a challenging problem because of the subjective and nuanced nature of art and affective signals.
HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noise
Low-latency deep spiking neural networks (SNNs) have become a promising alternative to conventional artificial neural networks (ANNs) because of their potential for increased energy efficiency on event-driven neuromorphic hardware.
FLOAT: FAST LEARNABLE ONCE-FOR-ALL ADVERSARIAL TRAINING FOR TUNABLE TRADE-OFF BETWEEN ACCURACY AND ROBUSTNESS
In this paper, we present Fast Learnable Once-for-all Adversarial Training (FLOAT) which transforms the weight tensors without using extra layers, thereby incurring no significant increase in parameter count, training time, or network latency compared to a standard adversarial training.
Training Energy-Efficient Deep Spiking Neural Networks with Single-Spike Hybrid Input Encoding
This paper presents a training framework for low-latency energy-efficient SNNs that uses a hybrid encoding scheme at the input layer in which the analog pixel values of an image are directly applied during the first timestep and a novel variant of spike temporal coding is used during subsequent timesteps.
HYPER-SNN: Towards Energy-efficient Quantized Deep Spiking Neural Networks for Hyperspectral Image Classification
However, the accurate processing of the spectral and spatial correlation between the bands requires the use of energy-expensive 3-D Convolutional Neural Networks (CNNs).
Computational Efficiency
Hyperspectral Image Classification
+2
Towards Low-Latency Energy-Efficient Deep SNNs via Attention-Guided Compression
To evaluate the merits of our approach, we performed experiments with variants of VGG and ResNet, on both CIFAR-10 and CIFAR-100, and VGG16 on Tiny-ImageNet. The SNN models generated through the proposed technique yield SOTA compression ratios of up to 33. 4x with no significant drops in accuracy compared to baseline unpruned counterparts.
AttentionLite: Towards Efficient Self-Attention Models for Vision
We propose a novel framework for producing a class of parameter and compute efficient models called AttentionLitesuitable for resource-constrained applications.
Attention-based Image Upsampling
Convolutional layers are an integral part of many deep neural network solutions in computer vision.
A Co-Attentive Cross-Lingual Neural Model for Dialogue Breakdown Detection
One of the major challenges is that a dialogue system may generate an undesired utterance leading to a dialogue breakdown, which degrades the overall interaction quality.
A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs
This paper presents a dynamic network rewiring (DNR) method to generate pruned deep neural network (DNN) models that are robust against adversarial attacks yet maintain high accuracy on clean images.
Hybrid Ranking Network for Text-to-SQL
In this paper, we study how to leverage pre-trained language models in Text-to-SQL.
Learning to Identify Follow-Up Questions in Conversational Question Answering
Despite recent progress in conversational question answering, most prior work does not focus on follow-up questions.
Pre-defined Sparsity for Low-Complexity Convolutional Neural Networks
We also compared the performance of our proposed architectures with that of ShuffleNet andMobileNetV2.
A Pre-defined Sparse Kernel Based Convolution for Deep CNNs
To explore the potential of this approach, we have experimented with two widely accepted datasets, CIFAR-10 and Tiny ImageNet, in sparse variants of both the ResNet18 and VGG16 architectures.
Exploiting Explicit Paths for Multi-hop Reading Comprehension
To capture additional context, PathNet also composes the passage representations along each path to compute a passage-based representation.
A Nil-Aware Answer Extraction Framework for Question Answering
However, current approaches suffer from an impractical assumption that every question has a valid answer in the associated passage.
A Highly Parallel FPGA Implementation of Sparse Neural Network Training
We demonstrate an FPGA implementation of a parallel and reconfigurable architecture for sparse neural networks, capable of on-chip training and inference.
A Question-Focused Multi-Factor Attention Network for Question Answering
Neural network models recently proposed for question answering (QA) primarily focus on capturing the passage-question relation.
Ranked #10 on
Open-Domain Question Answering
on SearchQA
