Search Results for author: Yiannis Aloimonos

Found 61 papers, 15 papers with code

CodedEvents: Optimal Point-Spread-Function Engineering for 3D-Tracking with Event Cameras

no code implementations CVPR 2024 Sachin Shah, Matthew Albert Chan, Haoming Cai, Jingxi Chen, Sakshum Kulshrestha, Chahat Deep Singh, Yiannis Aloimonos, Christopher Metzler

We then demonstrate that existing designs are sub-optimal for tracking moving point sources and proceed to use our theory to design optimal phase masks and binary amplitude masks for this task.

Diving Deep into the Motion Representation of Video-Text Models

no code implementations7 Jun 2024 Chinmaya Devaraj, Cornelia Fermuller, Yiannis Aloimonos

This method proves to be effective on two action datasets for the motion description retrieval task.

Retrieval Text Retrieval +1

Event3DGS: Event-Based 3D Gaussian Splatting for High-Speed Robot Egomotion

no code implementations5 Jun 2024 Tianyi Xiong, Jiayi Wu, Botao He, Cornelia Fermuller, Yiannis Aloimonos, Heng Huang, Christopher A. Metzler

By combining differentiable rendering with explicit point-based scene representations, 3D Gaussian Splatting (3DGS) has demonstrated breakthrough 3D reconstruction capabilities.

3D Reconstruction 3D Scene Reconstruction +1

TimeRewind: Rewinding Time with Image-and-Events Video Diffusion

no code implementations20 Mar 2024 Jingxi Chen, Brandon Y. Feng, Haoming Cai, Mingyang Xie, Christopher Metzler, Cornelia Fermuller, Yiannis Aloimonos

Through extensive experimentation, we demonstrate the capability of our approach to synthesize high-quality videos that effectively ``rewind'' time, showcasing the potential of combining event camera technology with generative models.

MARVIS: Motion & Geometry Aware Real and Virtual Image Segmentation

1 code implementation14 Mar 2024 Jiayi Wu, Xiaomin Lin, Shahriar Negahdaripour, Cornelia Fermüller, Yiannis Aloimonos

By creating realistic synthetic images that mimic the complexities of the water surface, we provide fine-grained training data for our network (MARVIS) to discern between real and virtual images effectively.

3D Reconstruction Autonomous Navigation +4

NatSGD: A Dataset with Speech, Gestures, and Demonstrations for Robot Learning in Natural Human-Robot Interaction

no code implementations4 Mar 2024 Snehesh Shrestha, Yantian Zha, Saketh Banagiri, Ge Gao, Yiannis Aloimonos, Cornelia Fermuller

NatSGD serves as a foundational resource at the intersection of machine learning and HRI research, and we demonstrate its effectiveness in training robots to understand tasks through multimodal human commands, emphasizing the significance of jointly considering speech and gestures.

LEAP: LLM-Generation of Egocentric Action Programs

no code implementations29 Nov 2023 Eadom Dessalene, Michael Maynord, Cornelia Fermüller, Yiannis Aloimonos

We apply LEAP over a majority (87\%) of the training set of the EPIC Kitchens dataset, and release the resulting action programs as a publicly available dataset here (https://drive. google. com/drive/folders/1Cpkw_TI1IIxXdzor0pOXG3rWJWuKU5Ex? usp=drive_link).

Action Recognition Language Modelling +1

Decodable and Sample Invariant Continuous Object Encoder

1 code implementation31 Oct 2023 Dehao Yuan, Furong Huang, Cornelia Fermüller, Yiannis Aloimonos

In addition, the encoding is decodable, which enables neural networks to regress continuous objects by regressing their encodings.

Object Surface Normal Estimation

AcTExplore: Active Tactile Exploration on Unknown Objects

no code implementations12 Oct 2023 Amir-Hossein Shahidzadeh, Seong Jong Yoo, Pavan Mantripragada, Chahat Deep Singh, Cornelia Fermüller, Yiannis Aloimonos

Tactile exploration plays a crucial role in understanding object structures for fundamental robotics tasks such as grasping and manipulation.

Object Object Reconstruction

Detecting Olives with Synthetic or Real Data? Olive the Above

no code implementations16 Aug 2023 Yianni Karabatis, Xiaomin Lin, Nitin J. Sanket, Michail G. Lagoudakis, Yiannis Aloimonos

When access to real, human-labeled data is limited, a combination of mostly synthetic data and a small amount of real data can enhance olive detection.

Whale Detection Enhancement through Synthetic Satellite Images

1 code implementation15 Aug 2023 Akshaj Gaur, Cheng Liu, Xiaomin Lin, Nare Karapetyan, Yiannis Aloimonos

With a number of marine populations in rapid decline, collecting and analyzing data about marine populations has become increasingly important to develop effective conservation policies for a wide range of marine animals, including whales.

WorldGen: A Large Scale Generative Simulator

no code implementations3 Oct 2022 Chahat Deep Singh, Riya Kumari, Cornelia Fermüller, Nitin J. Sanket, Yiannis Aloimonos

In the era of deep learning, data is the critical determining factor in the performance of neural network models.

Object Optical Flow Estimation

OysterNet: Enhanced Oyster Detection Using Simulation

no code implementations16 Sep 2022 Xiaomin Lin, Nitin J. Sanket, Nare Karapetyan, Yiannis Aloimonos

However, systems for accurate oyster detection require large datasets obtaining which is an expensive and labor-intensive task in underwater environments.

Gluing Neural Networks Symbolically Through Hyperdimensional Computing

no code implementations31 May 2022 Peter Sutor, Dehao Yuan, Douglas Summers-Stay, Cornelia Fermuller, Yiannis Aloimonos

This process can be performed iteratively and even on single neural networks by instead making a consensus of multiple classification hypervectors.

AIMusicGuru: Music Assisted Human Pose Correction

no code implementations24 Mar 2022 Snehesh Shrestha, Cornelia Fermüller, Tianyu Huang, Pyone Thant Win, Adam Zukerman, Chethan M. Parameshwara, Yiannis Aloimonos

Pose Estimation techniques rely on visual cues available through observations represented in the form of pixels.

Pose Estimation

TTCDist: Fast Distance Estimation From an Active Monocular Camera Using Time-to-Contact

no code implementations14 Mar 2022 Levi Burner, Nitin J. Sanket, Cornelia Fermüller, Yiannis Aloimonos

Distance estimation from vision is fundamental for a myriad of robotic applications such as navigation, manipulation, and planning.

Sensor Fusion

NudgeSeg: Zero-Shot Object Segmentation by Repeated Physical Interaction

no code implementations22 Sep 2021 Chahat Deep Singh, Nitin J. Sanket, Chethan M. Parameshwara, Cornelia Fermüller, Yiannis Aloimonos

In this paper, we present the first framework to segment unknown objects in a cluttered scene by repeatedly 'nudging' at the objects and moving them to obtain additional motion cues at every step using only a monochrome monocular camera.

Motion Segmentation Object +2

EVPropNet: Detecting Drones By Finding Propellers For Mid-Air Landing And Following

no code implementations29 Jun 2021 Nitin J. Sanket, Chahat Deep Singh, Chethan M. Parameshwara, Cornelia Fermüller, Guido C. H. E. de Croon, Yiannis Aloimonos

Our network can detect propellers at a rate of 85. 1% even when 60% of the propeller is occluded and can run at upto 35Hz on a 2W power budget.

SpikeMS: Deep Spiking Neural Network for Motion Segmentation

no code implementations13 May 2021 Chethan M. Parameshwara, Simin Li, Cornelia Fermüller, Nitin J. Sanket, Matthew S. Evanusa, Yiannis Aloimonos

Spiking Neural Networks (SNN) are the so-called third generation of neural networks which attempt to more closely match the functioning of the biological brain.

Motion Segmentation

Forecasting Action through Contact Representations from First Person Video

no code implementations1 Feb 2021 Eadom Dessalene, Chinmaya Devaraj, Michael Maynord, Cornelia Fermuller, Yiannis Aloimonos

Human actions involving hand manipulations are structured according to the making and breaking of hand-object contact, and human visual understanding of action is reliant on anticipation of contact as is demonstrated by pioneering work in cognitive science.

Action Anticipation Object

MorphEyes: Variable Baseline Stereo For Quadrotor Navigation

1 code implementation5 Nov 2020 Nitin J. Sanket, Chahat Deep Singh, Varun Asthana, Cornelia Fermüller, Yiannis Aloimonos

To our knowledge, this is the first work that applies the concept of morphable design to achieve a variable baseline stereo vision system on a quadrotor.

Depth Estimation

Grasping in the Dark: Zero-Shot Object Grasping Using Tactile Feedback

1 code implementation2 Nov 2020 Kanishka Ganguly, Behzad Sadrfaridpour, Krishna Bhavithavya Kidambi, Cornelia Fermüller, Yiannis Aloimonos

Several end-effector designs for robust manipulation have been proposed but they mostly work when provided with prior information about the objects or equipped with external sensors for estimating object shape or size.

Robotics

Hybrid Backpropagation Parallel Reservoir Networks

no code implementations27 Oct 2020 Matthew Evanusa, Snehesh Shrestha, Michelle Girvan, Cornelia Fermüller, Yiannis Aloimonos

In many real-world applications, fully-differentiable RNNs such as LSTMs and GRUs have been widely deployed to solve time series learning tasks.

EEG Emotion Recognition +4

Deep Reservoir Networks with Learned Hidden Reservoir Weights using Direct Feedback Alignment

no code implementations13 Oct 2020 Matthew Evanusa, Cornelia Fermüller, Yiannis Aloimonos

Deep Reservoir Computing has emerged as a new paradigm for deep learning, which is based around the reservoir computing principle of maintaining random pools of neurons combined with hierarchical deep learning.

Time Series Time Series Prediction

A Deep 2-Dimensional Dynamical Spiking Neuronal Network for Temporal Encoding trained with STDP

no code implementations1 Sep 2020 Matthew Evanusa, Cornelia Fermuller, Yiannis Aloimonos

Here we show that a large, deep layered SNN with dynamical, chaotic activity mimicking the mammalian cortex with biologically-inspired learning rules, such as STDP, is capable of encoding information from temporal data.

PRGFlow: Benchmarking SWAP-Aware Unified Deep Visual Inertial Odometry

1 code implementation11 Jun 2020 Nitin J. Sanket, Chahat Deep Singh, Cornelia Fermüller, Yiannis Aloimonos

Odometry on aerial robots has to be of low latency and high robustness whilst also respecting the Size, Weight, Area and Power (SWAP) constraints as demanded by the size of the robot.

Benchmarking Translation

Egocentric Object Manipulation Graphs

no code implementations5 Jun 2020 Eadom Dessalene, Michael Maynord, Chinmaya Devaraj, Cornelia Fermuller, Yiannis Aloimonos

We introduce Egocentric Object Manipulation Graphs (Ego-OMG) - a novel representation for activity modeling and anticipation of near future actions integrating three components: 1) semantic temporal structure of activities, 2) short-term dynamics, and 3) representations for appearance.

Action Anticipation Attribute +1

Following Instructions by Imagining and Reaching Visual Goals

no code implementations25 Jan 2020 John Kanu, Eadom Dessalene, Xiaomin Lin, Cornelia Fermuller, Yiannis Aloimonos

While traditional methods for instruction-following typically assume prior linguistic and perceptual knowledge, many recent works in reinforcement learning (RL) have proposed learning policies end-to-end, typically by training neural networks to map joint representations of observations and instructions directly to actions.

Instruction Following Reinforcement Learning (RL)

EVDodgeNet: Deep Dynamic Obstacle Dodging with Event Cameras

2 code implementations7 Jun 2019 Nitin J. Sanket, Chethan M. Parameshwara, Chahat Deep Singh, Ashwin V. Kuruttukulam, Cornelia Fermüller, Davide Scaramuzza, Yiannis Aloimonos

To our knowledge, this is the first deep learning -- based solution to the problem of dynamic obstacle avoidance using event cameras on a quadrotor.

Motion Estimation

Network Deconvolution

5 code implementations ICLR 2020 Chengxi Ye, Matthew Evanusa, Hua He, Anton Mitrokhin, Tom Goldstein, James A. Yorke, Cornelia Fermüller, Yiannis Aloimonos

Convolution is a central operation in Convolutional Neural Networks (CNNs), which applies a kernel to overlapping regions shifted across the image.

Image Classification

EV-IMO: Motion Segmentation Dataset and Learning Pipeline for Event Cameras

no code implementations18 Mar 2019 Anton Mitrokhin, Chengxi Ye, Cornelia Fermuller, Yiannis Aloimonos, Tobi Delbruck

In addition to camera egomotion and a dense depth map, the network estimates pixel-wise independently moving object segmentation and computes per-object 3D translational velocities for moving objects.

Motion Segmentation Object +1

Topology-Aware Non-Rigid Point Cloud Registration

no code implementations16 Nov 2018 Konstantinos Zampogiannis, Cornelia Fermuller, Yiannis Aloimonos

In this paper, we introduce a non-rigid registration pipeline for pairs of unorganized point clouds that may be topologically different.

Dynamic Reconstruction Event Detection +2

Unsupervised Learning of Dense Optical Flow, Depth and Egomotion from Sparse Event Data

no code implementations23 Sep 2018 Chengxi Ye, Anton Mitrokhin, Cornelia Fermüller, James A. Yorke, Yiannis Aloimonos

In this work we present a lightweight, unsupervised learning pipeline for \textit{dense} depth, optical flow and egomotion estimation from sparse event output of the Dynamic Vision Sensor (DVS).

Decoder Optical Flow Estimation

Extracting Contact and Motion from Manipulation Videos

no code implementations13 Jul 2018 Konstantinos Zampogiannis, Kanishka Ganguly, Cornelia Fermuller, Yiannis Aloimonos

When we physically interact with our environment using our hands, we touch objects and force them to move: contact and motion are defining properties of manipulation.

Clustering Imitation Learning +2

Evenly Cascaded Convolutional Networks

no code implementations2 Jul 2018 Chengxi Ye, Chinmaya Devaraj, Michael Maynord, Cornelia Fermüller, Yiannis Aloimonos

We introduce Evenly Cascaded convolutional Network (ECN), a neural network taking inspiration from the cascade algorithm of wavelet analysis.

A Computational Theory for Life-Long Learning of Semantics

no code implementations28 Jun 2018 Peter Sutor Jr., Douglas Summers-Stay, Yiannis Aloimonos

Semantic vectors are learned from data to express semantic relationships between elements of information, for the purpose of solving and informing downstream tasks.

Joint direct estimation of 3D geometry and 3D motion using spatio temporal gradients

no code implementations17 May 2018 Francisco Barranco, Cornelia Fermüller, Yiannis Aloimonos, Eduardo Ros

Conventional image motion based structure from motion methods first compute optical flow, then solve for the 3D motion parameters based on the epipolar constraint, and finally recover the 3D geometry of the scene.

Motion Estimation Optical Flow Estimation

Event-based Moving Object Detection and Tracking

no code implementations12 Mar 2018 Anton Mitrokhin, Cornelia Fermuller, Chethan Parameshwara, Yiannis Aloimonos

Event-based vision sensors, such as the Dynamic Vision Sensor (DVS), are ideally suited for real-time motion analysis.

Event-based vision Motion Detection +5

SalientDSO: Bringing Attention to Direct Sparse Odometry

1 code implementation28 Feb 2018 Huai-Jen Liang, Nitin J. Sanket, Cornelia Fermüller, Yiannis Aloimonos

We merge the successes of these two communities and present a way to incorporate semantic information in the form of visual saliency to Direct Sparse Odometry - a highly successful direct sparse VO algorithm.

feature selection Scene Parsing +1

GapFlyt: Active Vision Based Minimalist Structure-less Gap Detection For Quadrotor Flight

1 code implementation14 Feb 2018 Nitin J. Sanket, Chahat Deep Singh, Kanishka Ganguly, Cornelia Fermüller, Yiannis Aloimonos

We use this philosophy to design a minimalist sensori-motor framework for a quadrotor to fly though unknown gaps without a 3D reconstruction of the scene using only a monocular camera and onboard sensing.

Robotics

Generalization of Learning using Reservoir Computing

no code implementations ICLR 2018 Sanjukta Krishnagopal, Yiannis Aloimonos, Michelle Girvan

Thus, as opposed to training the entire high dimensional reservoir state, the reservoir only needs to train on these unique relationships, allowing the reservoir to perform well with very few training examples.

Clustering

On the Importance of Consistency in Training Deep Neural Networks

no code implementations2 Aug 2017 Chengxi Ye, Yezhou Yang, Cornelia Fermuller, Yiannis Aloimonos

We conclude this paper with the construction of a novel contractive neural network.

LightNet: A Versatile, Standalone Matlab-based Environment for Deep Learning

1 code implementation9 May 2016 Chengxi Ye, Chen Zhao, Yezhou Yang, Cornelia Fermuller, Yiannis Aloimonos

LightNet is a lightweight, versatile and purely Matlab-based deep learning framework.

Revisiting Active Perception

no code implementations8 Mar 2016 Ruzena Bajcsy, Yiannis Aloimonos, John K. Tsotsos

Despite the recent successes in robotics, artificial intelligence and computer vision, a complete artificial agent necessarily must include active perception.

What Can I Do Around Here? Deep Functional Scene Understanding for Cognitive Robots

no code implementations29 Jan 2016 Chengxi Ye, Yezhou Yang, Cornelia Fermuller, Yiannis Aloimonos

For robots that have the capability to interact with the physical environment through their end effectors, understanding the surrounding scenes is not merely a task of image classification or object recognition.

Image Classification Object Recognition +1

Neural Self Talk: Image Understanding via Continuous Questioning and Answering

no code implementations10 Dec 2015 Yezhou Yang, Yi Li, Cornelia Fermuller, Yiannis Aloimonos

In this paper we consider the problem of continuously discovering image contents by actively asking image based questions and subsequently answering the questions being asked.

Question Answering Question Generation +2

Contour Detection and Characterization for Asynchronous Event Sensors

no code implementations ICCV 2015 Francisco Barranco, Ching L. Teo, Cornelia Fermuller, Yiannis Aloimonos

The bio-inspired, asynchronous event-based dynamic vision sensor records temporal changes in the luminance of the scene at high temporal resolution.

Boundary Detection Contour Detection +2

Detection and Segmentation of 2D Curved Reflection Symmetric Structures

no code implementations ICCV 2015 Ching L. Teo, Cornelia Fermuller, Yiannis Aloimonos

Symmetry, as one of the key components of Gestalt theory, provides an important mid-level cue that serves as input to higher visual processes such as segmentation.

Segmentation Symmetry Detection

From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge

no code implementations10 Nov 2015 Somak Aditya, Yezhou Yang, Chitta Baral, Cornelia Fermuller, Yiannis Aloimonos

Specifically, commonsense reasoning is applied on (a) detections obtained from existing perception methods on given images, (b) a "commonsense" knowledge base constructed using natural language processing of image annotations and (c) lexical ontological knowledge from resources such as WordNet.

image-sentence alignment Sentence

Fast 2D Border Ownership Assignment

no code implementations CVPR 2015 Ching Teo, Cornelia Fermuller, Yiannis Aloimonos

To this end, we use several different local cues: shape, spectral properties of boundary patches, and semi-global grouping cues that are indicative of perceived depth.

Boundary Detection

Detection of Manipulation Action Consequences (MAC)

no code implementations CVPR 2013 Yezhou Yang, Cornelia Fermuller, Yiannis Aloimonos

There is a small set of fundamental primitive action consequences that provides a systematic high-level classification of manipulation actions.

Action Recognition Temporal Action Localization

Cannot find the paper you are looking for? You can Submit a new open access paper.