Search Results for author:
Found 32 papers, 7 papers with code
Brain-to-Text Decoding with Context-Aware Neural Representations and Large Language Models
Decoding attempted speech from neural activity offers a promising avenue for restoring communication abilities in individuals with speech impairments.
Tell What You Hear From What You See -- Video to Audio Generation Through Text
Experiments show that when VATT is compared to existing video-to-audio generation methods in objective metrics, it achieves competitive performance when the audio caption is not provided.
Transferable polychromatic optical encoder for neural networks
Our results evidence the potential of hybrid optical/digital computer vision system in which the optical frontend can pre-process an ambient scene to reduce the energy and latency of the whole computer vision system.
CaloChallenge 2022: A Community Challenge for Fast Calorimeter Simulation
We present the results of the "Fast Calorimeter Simulation Challenge 2022" - the CaloChallenge.
From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation
Previous studies of audio-visual modalities primarily focused on either audio-visual representation learning or generative modeling of a modality conditioned on the other, creating a disconnect between these two branches.
Calo-VQ: Vector-Quantized Two-Stage Generative Model in Calorimeter Simulation
We introduce a novel machine learning method developed for the fast simulation of calorimeter detector response, adapting vector-quantized variational autoencoder (VQ-VAE).
Compressed Meta-Optical Encoder for Image Classification
Optical and hybrid convolutional neural networks (CNNs) recently have become of increasing interest to achieve low-latency, low-power image classification and computer vision tasks.
Evolutionary algorithms as an alternative to backpropagation for supervised training of Biophysical Neural Networks and Neural ODEs
Training networks consisting of biophysically accurate neuron models could allow for new insights into how brain circuits can organize and solve tasks.
Learning Time-Invariant Representations for Individual Neurons from Population Dynamics
This suggests that neuronal activity is a combination of its time-invariant identity and the inputs the neuron receives from the rest of the circuit.
The time is ripe to reverse engineer an entire nervous system: simulating behavior from neural interactions
Here, we argue that such completeness is possible in C. elegans; a complete description that goes all the way from the activity of every neuron to predict behavior.
Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos
Traditional methods of impact sound synthesis use physics simulation to obtain a set of physics parameters that could represent and synthesize the sound.
Be Everywhere - Hear Everything (BEE): Audio Scene Reconstruction by Sparse Audio-Visual Samples
The audio at the listener location is a complex outcome of sound propagation through the scene geometry and interacting with surfaces and also the locations of the emitters and the sounds they emit.
TKIL: Tangent Kernel Approach for Class Balanced Incremental Learning
In our work, we propose to address these challenges with the introduction of a novel methodology of Tangent Kernel for Incremental Learning (TKIL) that achieves class-balanced performance.
STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer
Modeling neural population dynamics underlying noisy single-trial spiking activities is essential for relating neural observation and behavior.
Lyapunov-Guided Representation of Recurrent Neural Network Performance
To address the fact that RNN features go beyond the existing Lyapunov spectral analysis, we propose to infer relevant features from the Lyapunov spectrum with an Autoencoder and an embedding of its latent representation (AeLLE).
Statistical Perspective on Functional and Causal Neural Connectomics: The Time-Aware PC Algorithm
In this work, we propose to model and estimate the causal functional connectivity from neural time series using a novel approach that adapts directed probabilistic graphical modeling to the time series scenario.
How Does it Sound?
It is often the case that the experience of watching the video can be enhanced by adding a musical soundtrack that is in-sync with the rhythmic features of these activities.
Statistical Perspective on Functional and Causal Neural Connectomics: A Comparative Study
Additional approaches have been proposed to incorporate aspects of causality to turn functional connectomes into causal functional connectomes, however, these methodologies typically focus on specific aspects of causality.
Knowledge Distillation Circumvents Nonlinearity for Optical Convolutional Neural Networks
However, a major challenge in using this spectral approach, as well as in an optical implementation of CNNs, is the inclusion of a nonlinearity between each convolutional layer, without which CNN performance drops dramatically.
Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
We propose a novel system that takes as an input body movements of a musician playing a musical instrument and generates music in an unsupervised setting.
Sparse Semi-Supervised Action Recognition with Active Learning
Current state-of-the-art methods for skeleton-based action recognition are supervised and rely on labels.
On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters.
Audeo: Audio Generation for a Silent Performance Video
We present a novel system that gets as an input video frames of a musician playing the piano and generates the music for that video.
BI-MAML: Balanced Incremental Approach for Meta Learning
The balanced learning strategy enables BI-MAML to both outperform other state-of-the-art models in terms of classification accuracy for existing tasks and also accomplish efficient adaption to similar new tasks with less required shots.
Iterate & Cluster: Iterative Semi-Supervised Action Recognition
The method utilizes latent space embedding and clustering of the unsupervised encoder-decoder to guide the selection of sequences to be annotated in each iteration.
Deep Reinforcement Learning for Neural Control
To infer candidate control policies, our approach maps neural circuits and their connectome into a grid-world like setting and infers the actions needed to achieve aimed behavior.
R-FORCE: Robust Learning for Random Recurrent Neural Networks
Random Recurrent Neural Networks (RRNN) are the simplest recurrent networks to model and extract features from sequential data.
PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition
Given inputs of body keypoints sequences obtained during various movements, our system associates the sequences with actions.
Clustering and Recognition of Spatiotemporal Features through Interpretable Embedding of Sequence to Sequence Recurrent Neural Networks
We show that the methodology provides a high-quality unsupervised categorization of movements.
An Optical Frontend for a Convolutional Neural Network
The parallelism of optics and the miniaturization of optical components using nanophotonic structures, such as metasurfaces present a compelling alternative to electronic implementations of convolutional neural networks.
Audio to Body Dynamics
We present a method that gets as input an audio of violin or piano playing, and outputs a video of skeleton predictions which are further used to animate an avatar.
Low-dimensional functionality of complex network dynamics: Neuro-sensory integration in the Caenorhabditis elegans connectome
We develop a biophysical model of neuro-sensory integration in the model organism Caenorhabditis elegans.
