Search Results for author:
Found 28 papers, 6 papers with code
Feature-based Graph Attention Networks Improve Online Continual Learning
Online continual learning for image classification is crucial for models to adapt to new data while retaining knowledge of previously learned tasks.
Parallel Sequence Modeling via Generalized Spatial Propagation Network
We present the Generalized Spatial Propagation Network (GSPN), a new attention mechanism optimized for vision tasks that inherently captures 2D spatial structures.
Hymba: A Hybrid-head Architecture for Small Language Models
We propose Hymba, a family of small language models featuring a hybrid-head parallel architecture that integrates transformer attention mechanisms with state space models (SSMs) for enhanced efficiency.
An Empirical Study of Mamba-based Language Models
On an additional 23 long-context tasks, the hybrid model continues to closely match or exceed the Transformer on average.
RegionGPT: Towards Region Understanding Vision Language Model
Vision language models (VLMs) have experienced rapid advancements through the integration of large language models (LLMs) with image-text pairs, yet they struggle with detailed regional visual understanding due to limited spatial awareness of the vision encoder, and the use of coarse-grained training data that lacks detailed, region-specific captions.
MEVG: Multi-event Video Generation with Text-to-Video Models
We introduce a novel diffusion-based video generation method, generating a video showing multiple events given multiple individual sentences from the user.
The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion
Hence, we propose The Power of Sound (TPoS) model to incorporate audio input that includes both changeable temporal semantics and magnitude.
Heterogeneous Continual Learning
We propose a novel framework and a solution to tackle the continual learning (CL) problem with changing network architectures.
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
Our approach outperforms the previous state of the art by significant margins on both open-vocabulary panoptic and semantic segmentation tasks.
Ranked #2 on
Open-World Instance Segmentation
on UVO
(using extra training data)
Open Vocabulary Panoptic Segmentation
Open Vocabulary Semantic Segmentation
+4
LISA: Localized Image Stylization with Audio via Implicit Neural Representation
We present a novel framework, Localized Image Stylization with Audio (LISA) which performs audio-driven localized image stylization.
Robust Sound-Guided Image Manipulation
Our extensive experiments show that our sound-guided image manipulation approach produces semantically and visually more plausible manipulation results than the state-of-the-art text and sound-guided image manipulation methods, which are further confirmed by our human evaluations.
Sound-Guided Semantic Video Generation
The recent success in StyleGAN demonstrates that pre-trained StyleGAN latent space is useful for realistic video generation.
Physics Informed RNN-DCT Networks for Time-Dependent Partial Differential Equations
This efficiently and flexibly produces a compressed representation which is used for additional conditioning of physics-informed models.
GroupViT: Semantic Segmentation Emerges from Text Supervision
With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i. e., without any further fine-tuning.
Coupled Segmentation and Edge Learning via Dynamic Graph Propagation
It is therefore interesting to study how these two tasks can be coupled to benefit each other.
Sound-Guided Semantic Image Manipulation
Our audio encoder is trained to produce a latent representation from an audio input, which is forced to be aligned with image and text representations in the multi-modal embedding space.
Scaling-up Diverse Orthogonal Convolutional Networks by a Paraunitary Framework
Some of these designs are not exactly orthogonal, while others only consider standard convolutional layers and propose specific classes of their realizations.
Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Framework
To address this problem, we propose a theoretical framework for orthogonal convolutional layers, which establishes the equivalence between various orthogonal convolutional layers in the spatial domain and the paraunitary systems in the spectral domain.
Weakly-Supervised Physically Unconstrained Gaze Estimation
A major challenge for physically unconstrained gaze estimation is acquiring training data with 3D gaze annotations for in-the-wild and outdoor scenarios.
Ranked #3 on
Gaze Estimation
on Gaze360
NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
We present real-world use cases that range from challenging forward multi-physics simulations with turbulence and complex 3D geometries, to industrial design optimization and inverse problems that are not addressed efficiently by the traditional solvers.
Displacement-Invariant Cost Computation for Efficient Stereo Matching
A common way to speed up the computation is to downsample the feature volume, but this loses high-frequency details.
Convolutional Tensor-Train LSTM for Spatio-temporal Learning
Learning from spatio-temporal data has numerous applications such as human-behavior analysis, object tracking, video compression, and physics simulation. However, existing methods still perform poorly on challenging video tasks such as long-term forecasting.
Ranked #1 on
Video Prediction
on KTH
(Cond metric)
Convolutional Tensor-Train LSTM for Long-Term Video Prediction
Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across a long range of image frames. Standard recurrent models are ineffective since they are prone to error propagation and cannot effectively capture higher-order correlations.
Long History Short-Term Memory for Long-Term Video Prediction
While video prediction approaches have advanced considerably in recent years, learning to predict long-term future is challenging — ambiguous future or error propagation over time yield blurry predictions.
Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks
We introduce a data-driven forecasting method for high-dimensional chaotic systems using long short-term memory (LSTM) recurrent neural networks.
ContextVP: Fully Context-Aware Video Prediction
Video prediction models based on convolutional networks, recurrent networks, and their combinations often result in blurry predictions.
Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation
In contrast, Multi-Dimensional Recurrent NNs (MD-RNNs) can perceive the entire spatio-temporal context of each pixel in a few sweeps through all pixels, especially when the RNN is a Long Short-Term Memory (LSTM).
Scene Labeling With LSTM Recurrent Neural Networks
This paper addresses the problem of pixel-level segmentation and classification of scene images with an entirely learning-based approach using Long Short Term Memory (LSTM) recurrent neural networks, which are commonly used for sequence classification.
