Search Results for author:
Found 100 papers, 48 papers with code
Multi-Agent Embodied Question Answering in Interactive Environments
We investigate a new AI task --- Multi-Agent Interactive Question Answering --- where several agents explore the scene jointly in interactive environments to answer a question.
Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learning
In this paper, we introduce a new video2reward method, which directly generates reward functions from videos depicting the behaviors to be mimicked and learned.
Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions
Recent advancements in deep learning have significantly revolutionized the field of clinical diagnosis and treatment, offering novel approaches to improve diagnostic precision and treatment efficacy across diverse clinical domains, thus driving the pursuit of precision medicine.
Rethinking Generalizability and Discriminability of Self-Supervised Learning from Evolutionary Game Theory Perspective
Thus, to enhance the methodological generalization, we propose a novel self-supervised learning method that leverages advancements in reinforcement learning to jointly benefit from the general guidance of EGT and sequentially optimize the model to chase the consistent improvement of generalizability and discriminability for specific target domains during pre-training.
DiffDesign: Controllable Diffusion with Meta Prior for Efficient Interior Design Generation
To address these challenges, we propose DiffDesign, a controllable diffusion model with meta priors for efficient interior design generation.
On the Generalization and Causal Explanation in Self-Supervised Learning
Self-supervised learning (SSL) methods learn from unlabeled data and achieve high generalization performance on downstream tasks.
On the Discriminability of Self-Supervised Representation Learning
We provide a theoretical framework showing that the performance gap between SSL and SL mainly stems from the inability of SSL methods to capture the aggregation of similar augmentations and the separation of dissimilar augmentations.
Subequivariant Reinforcement Learning in 3D Multi-Entity Physical Environments
In particular, SHNN first dynamically decouples the global space into local entity-level graphs via task assignment.
A Physical Model-Guided Framework for Underwater Image Enhancement and Depth Estimation
DDM includes three well-designed sub-networks to accurately estimate various imaging parameters: a veiling light estimation sub-network, a factors estimation sub-network, and a depth estimation sub-network.
Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policies
Hierarchical reinforcement learning (HRL) addresses complex long-horizon tasks by skillfully decomposing them into subgoals.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
RoboGolf: Mastering Real-World Minigolf with a Reflective Multi-Modality Vision-Language Model
Minigolf is an exemplary real-world game for examining embodied intelligence, requiring challenging spatial and kinodynamic understanding to putt the ball.
Small Scale Data-Free Knowledge Distillation
In this line of research, existing methods typically follow an inversion-and-distillation paradigm in which a generative adversarial network on-the-fly trained with the guidance of the pre-trained teacher network is used to synthesize a large-scale sample set for knowledge distillation.
Data-free Knowledge Distillation
Generative Adversarial Network
+2
OMPO: A Unified Framework for RL under Policy and Dynamics Shifts
Training reinforcement learning policies using environment interaction data collected from varying policies or dynamics presents a fundamental challenge.
Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
Based on this insight, we present Offline-Boosted Actor-Critic (OBAC), a model-free online RL framework that elegantly identifies the outperforming offline policy through value comparison, and uses it as an adaptive constraint to guarantee stronger policy learning performance.
Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels
Vidu4D also contains a novel initialization state that provides a proper start for the warping fields in DGS.
Equivariant Local Reference Frames for Unsupervised Non-rigid Point Cloud Shape Correspondence
Based on the assumption of local rigidity, one solution for reducing complexity is to decompose the overall shape into independent local regions using Local Reference Frames (LRFs) that are invariant to SE(3) transformations.
Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding
As a result, with a single image CLIP embedding, Isotropic3D is capable of generating multi-view mutually consistent images and also a 3D model with more symmetrical and neat content, well-proportioned geometry, rich colored texture, and less distortion compared with existing image-to-3D methods while still preserving the similarity to the reference image to a large extent.
Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learning
Language-conditioned robot behavior plays a vital role in executing complex tasks by associating human commands or instructions with perception and actions.
ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization
The varying significance of distinct primitive behaviors during the policy learning process has been overlooked by prior model-free RL algorithms.
Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy
As a key component to intuitive cognition and reasoning solutions in human intelligence, causal knowledge provides great potential for reinforcement learning (RL) agents' interpretability towards decision-making by helping reduce the searching space.
BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction
From the perspective of distribution analyses, we disclose that the intrinsic issues behind the phenomenon are the over-multitudinous conceptual knowledge contained in PLMs and the abridged knowledge for target downstream domains, which jointly result in that PLMs mis-locate the knowledge distributions corresponding to the target domains in the universal knowledge embedding space.
Hierarchical Topology Isomorphism Expertise Embedded Graph Contrastive Learning
To this end, we propose a novel hierarchical topology isomorphism expertise embedded graph contrastive learning, which introduces knowledge distillations to empower GCL models to learn the hierarchical topology isomorphism expertise, including the graph-tier and subgraph-tier.
UIEDP:Underwater Image Enhancement with Diffusion Prior
To address this issue, we propose UIE with Diffusion Prior (UIEDP), a novel framework treating UIE as a posterior distribution sampling process of clear images conditioned on degraded underwater inputs.
AnimatableDreamer: Text-Guided Non-rigid 3D Model Generation and Reconstruction with Canonical Score Distillation
Extensive experiments demonstrate the capability of our method in generating high-flexibility text-guided 3D models from the monocular video, while also showing improved reconstruction performance over existing non-rigid reconstruction methods.
Measuring Acoustics with Collaborative Multiple Agents
As humans, we hear sound every second of our life.
PrototypeFormer: Learning to Explore Prototype Relationships for Few-shot Image Classification
In particular, our method achieves 97. 07% and 90. 88% on 5-way 5-shot and 5-way 1-shot tasks of miniImageNet, which surpasses the state-of-the-art results with accuracy of 7. 27% and 8. 72%, respectively.
Root Pose Decomposition Towards Generic Non-rigid 3D Reconstruction with Monocular Videos
The key idea of our method, Root Pose Decomposition (RPD), is to maintain a per-frame root pose transformation, meanwhile building a dense field with local transformations to rectify the root pose.
IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation
Recent research has made significant progress in designing fusion modules for audio-visual speech separation.
Ranked #1 on
Speech Separation
on LRS3
Towards Task Sampler Learning for Meta-Learning
We obtain three conclusions: (i) there is no universal task sampling strategy that can guarantee the optimal performance of meta-learning models; (ii) over-constraining task diversity may incur the risk of under-fitting or over-fitting during training; and (iii) the generalization performance of meta-learning models are affected by task diversity, task entropy, and task difficulty.
Towards the Sparseness of Projection Head in Self-Supervised Learning
In recent years, self-supervised learning (SSL) has emerged as a promising approach for extracting valuable representations from unlabeled data.
A Dimensional Structure based Knowledge Distillation Method for Cross-Modal Learning
To address this issue, in this paper, we discover the correlation between feature discriminability and dimensional structure (DS) by analyzing and observing features extracted from simple and hard tasks.
Structure-Aware DropEdge Towards Deep Graph Convolutional Networks
Regarding the layer-dependent sampler, we interestingly find that increasingly sampling edges from the bottom layer yields superior performance than the decreasing counterpart as well as DropEdge.
Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic
We find that such a long-neglected phenomenon is often related to the use of inferior actions from the current policy in Bellman updates as compared to the more optimal action samples in the replay buffer.
Manifold Constraint Regularization for Remote Sensing Image Generation
This study identifies a previously overlooked issue: GANs exhibit a heightened susceptibility to overfitting on remote sensing images. To address this challenge, this paper analyzes the characteristics of remote sensing images and proposes manifold constraint regularization, a novel approach that tackles overfitting of GANs on remote sensing images for the first time.
Subequivariant Graph Reinforcement Learning in 3D Environments
Learning a shared policy that guides the locomotion of different agents is of core interest in Reinforcement Learning (RL), which leads to the study of morphology-agnostic RL.
SE-Bridge: Speech Enhancement with Consistent Brownian Bridge
Our experiments on multiple datasets demonstrate the effectiveness of SE-Bridge in SE.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Compacting Binary Neural Networks by Sparse Kernel Selection
Binary Neural Network (BNN) represents convolution weights with 1-bit values, which enhances the efficiency of storage and computation.
Intriguing Property and Counterfactual Explanation of GAN for Remote Sensing Image Generation
Based on this SCM, we theoretically prove that the quality of generated images is positively correlated with the amount of feature information.
Deep Reinforcement Learning for Robotic Pushing and Picking in Cluttered Environment
A composite robotic hand composed of a suction cup and a gripper is designed for grasping the object stably.
A Survey on Causal Reinforcement Learning
While Reinforcement Learning (RL) achieves tremendous success in sequential decision-making problems of many domains, it still faces key challenges of data inefficiency and the lack of interpretability.
Introducing Expertise Logic into Graph Representation Learning from A Causal Perspective
By further observing the ramifications of introducing expertise logic into graph representation learning, we conclude that leading the GNNs to learn human expertise can improve the model performance.
Timestamp-Supervised Action Segmentation from the Perspective of Clustering
Most existing methods generate pseudo-labels for all frames in each video to train the segmentation model.
A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodal
According to the graph types, existing KGR models can be roughly divided into three categories, i. e., static models, temporal models, and multi-modal models.
Recognizing Object by Components with Human Prior Knowledge Enhances Adversarial Robustness of Deep Neural Networks
The first stage of ROCK corresponds to the process of decomposing objects into parts in human vision.
When to Update Your Model: Constrained Model-based Reinforcement Learning
Our follow-up derived bounds reveal the relationship between model shifts and performance improvement.
Bridged Transformer for Vision and Point Cloud 3D Object Detection
Recently, there is a trend of leveraging multiple sources of input data, such as complementing the 3D point cloud with 2D images that often have richer color and fewer noises.
Pay Self-Attention to Audio-Visual Navigation
Audio-visual embodied navigation, as a hot research topic, aims training a robot to reach an audio target using egocentric visual (from the sensors mounted on the robot) and audio (emitted from the target) input.
Disentangle and Remerge: Interventional Knowledge Distillation for Few-Shot Object Detection from A Conditional Causal Perspective
To understand the reasons behind this phenomenon, we revisit the learning paradigm of knowledge distillation on the few-shot object detection task from the causal theoretic standpoint, and accordingly, develop a Structural Causal Model.
Robust Causal Graph Representation Learning against Confounding Effects
This observation reveals that there exist confounders in graphs, which may interfere with the model learning semantic information, and current graph representation learning methods have not eliminated their influence.
Similarity-aware Positive Instance Sampling for Graph Contrastive Pre-training
To remedy this issue, we propose to select positive graph instances directly from existing graphs in the training set, which ultimately maintains the legality and similarity to the target graphs.
SNAKE: Shape-aware Neural 3D Keypoint Field
Detecting 3D keypoints from point clouds is important for shape reconstruction, while this work investigates the dual question: can shape reconstruction benefit 3D keypoint detection?
Multimodal Token Fusion for Vision Transformers
Many adaptations of transformers have emerged to address the single-modal vision tasks, where self-attention modules are stacked to handle input sources like images.
Ranked #3 on
Semantic Segmentation
on SUN-RGBD
(using extra training data)
Smoothing Matters: Momentum Transformer for Domain Adaptive Semantic Segmentation
After the great success of Vision Transformer variants (ViTs) in computer vision, it has also demonstrated great potential in domain adaptive semantic segmentation.
Ranked #7 on
Semantic Segmentation
on SYNTHIA-to-Cityscapes
Equivariant Graph Mechanics Networks with Constraints
The core of GMN is that it represents, by generalized coordinates, the forward kinematics information (positions and velocities) of a structural object.
Adversarial Texture for Fooling Person Detectors in the Physical World
Experiments showed that these clothes could fool person detectors in the physical world.
Sound Adversarial Audio-Visual Navigation
In this work, we design an acoustically complex environment in which, besides the target sound, there exists a sound attacker playing a zero-sum game with the agent.
Sim2Real Object-Centric Keypoint Detection and Description
Keypoint detection and description play a central role in computer vision.
Self-supervised 3D Semantic Representation Learning for Vision-and-Language Navigation
In the Vision-and-Language Navigation task, the embodied agent follows linguistic instructions and navigates to a specific goal.
Bootstrapping Informative Graph Augmentation via A Meta Learning Approach
To this end, we propose a novel approach to learning a graph augmenter that can generate an augmentation with uniformity and informativeness.
Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction
For the application of dense image prediction, the validity of CEN is tested by four different scenarios: multimodal fusion, cycle multimodal fusion, multitask learning, and multimodal multitask learning.
Ranked #7 on
Semantic Segmentation
on LLRGBD-synthetic
Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks
In the low-bit quantization field, training Binary Neural Networks (BNNs) is the extreme solution to ease the deployment of deep models on resource-constrained devices, having the lowest storage cost and significantly cheaper bit-wise operations compared to 32-bit floating-point counterparts.
Constrained Graph Mechanics Networks
In this manner, the geometrical constraints are implicitly and naturally encoded in the forward kinematics.
Multi-Agent Embodied Visual Semantic Navigation with Scene Prior Knowledge
In this paper, we propose the multi-agent visual semantic navigation, in which multiple agents collaborate with others to find multiple target objects.
Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion
We propose a compact and effective framework to fuse multimodal features at multiple layers in a single network.
Ranked #48 on
Semantic Segmentation
on NYU Depth v2
Elastic Tactile Simulation Towards Tactile-Visual Perception
By contrast, EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact.
Adversarial Option-Aware Hierarchical Imitation Learning
In particular, we propose an Expectation-Maximization(EM)-style algorithm: an E-step that samples the options of expert conditioned on the current learned policy, and an M-step that updates the low- and high-level policies of agent simultaneously to minimize the newly proposed option-occupancy measurement between the expert and the agent.
Elastic Interaction of Particles for Robotic Tactile Simulation
At its core, EIP models the tactile sensor as a group of coordinated particles, and the elastic theory is applied to regulate the deformation of particles during the contact process.
Fault-Aware Robust Control via Adversarial Reinforcement Learning
Robots have limited adaptation ability compared to humans and animals in the case of damage.
Deep Multimodal Fusion by Channel Exchanging
Deep multimodal fusion by using multiple sources of data for classification or regression has exhibited a clear advantage over the unimodal counterpart on various applications.
Unsupervised Representation Learning by InvariancePropagation
In this paper, we propose Invariance Propagation to focus on learning representations invariant to category-level variations, which are provided by different instances from the same category.
Tackling Over-Smoothing for General Graph Convolutional Networks
Increasing the depth of GCN, which is expected to permit more expressivity, is shown to incur performance detriment especially on node classification.
A Boundary Based Out-of-Distribution Classifier for Generalized Zero-Shot Learning
Using a gating mechanism that discriminates the unseen samples from the seen samples can decompose the GZSL problem to a conventional Zero-Shot Learning (ZSL) problem and a supervised classification problem.
Resolution Switchable Networks for Runtime Efficient Image Recognition
We propose a general method to train a single convolutional neural network which is capable of switching image resolutions at inference.
Towards Embodied Scene Description
Embodiment is an important characteristic for all intelligent agents (creatures and robots), while existing scene description tasks mainly focus on analyzing images passively and the semantic understanding of the scenario is separated from the interaction between the agent and the environment.
A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU
In this paper, we present a multimodal mobile teleoperation system that consists of a novel vision-based hand pose regression network (Transteleop) and an IMU-based arm tracking method.
MQA: Answering the Question via Robotic Manipulation
In this paper, we propose a novel task, Manipulation Question Answering (MQA), where the robot performs manipulation actions to change the environment in order to answer a given question.
Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation
The proposed architecture, termed as NICE-GAN, exhibits two advantageous patterns over previous approaches: First, it is more compact since no independent encoding component is required; Second, this plug-in encoder is directly trained by the adversary loss, making it more informative and trained more effectively if a multi-scale discriminator is applied.
Robust Robotic Pouring using Audition and Haptics
Both network training results and robot experiments demonstrate that MP-Net is robust against noise and changes to the task and environment.
FoveaBox: Beyound Anchor-based Object Detection
While almost all state-of-the-art object detectors utilize predefined anchors to enumerate possible locations, scales and aspect ratios for the search of the objects, their performance and generalization ability are also limited to the design of anchors.
Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance
In this paper, we study Reinforcement Learning from Demonstrations (RLfD) that improves the exploration efficiency of Reinforcement Learning (RL) by providing expert demonstrations.
Regularized Adversarial Sampling and Deep Time-aware Attention for Click-Through Rate Prediction
In deep CTR models, exploiting users' historical data is essential for learning users' behaviors and interests.
Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement
This paper studies Learning from Observations (LfO) for imitation learning with access to state-only demonstrations.
Deep Point-wise Prediction for Action Temporal Proposal
Detecting actions in videos is an important yet challenging task.
Attention-based Transfer Learning for Brain-computer Interface
Different functional areas of the human brain play different roles in brain activity, which has not been paid sufficient research attention in the brain-computer interface (BCI) field.
FoveaBox: Beyond Anchor-based Object Detector
In FoveaBox, an instance is assigned to adjacent feature levels to make the model more accurate. We demonstrate its effectiveness on standard benchmarks and report extensive experimental analysis.
Ranked #91 on
Object Detection
on COCO test-dev
(APM metric)
Consistent Optimization for Single-Shot Object Detection
We present consistent optimization for single stage object detection.
PointNetGPD: Detecting Grasp Configurations from Point Sets
In this paper, we propose an end-to-end grasp evaluation model to address the challenging problem of localizing robot grasp configurations directly from the point cloud.
Robotics
Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network
In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands.
Robotics
Deep Feature Pyramid Reconfiguration for Object Detection
In this paper, we begin by investigating current feature pyramids solutions, and then reformulate the feature pyramid construction as the feature reconfiguration process.
A Survey on Deep Transfer Learning
As a new classification platform, deep learning has recently received increasing attention from researchers and has been successfully applied to many domains.
Deep Transfer Learning for EEG-based Brain Computer Interface
First, we model cognitive events based on EEG data by characterizing the data using EEG optical flow, which is designed to preserve multimodal EEG information in a uniform representation.
Multimodal Classification with Deep Convolutional-Recurrent Neural Networks for Electroencephalography
Herein, we propose a novel approach to modeling cognitive events from EEG data by reducing it to a video classification problem, which is designed to preserve the multimodal information of EEG.
Learning and Inferring Movement with Deep Generative Model
Learning and inference movement is a very challenging problem due to its high dimensionality and dependency to varied environments or tasks.
Task Transfer by Preference-Based Cost Learning
The goal of task transfer in reinforcement learning is migrating the action policy of an agent to the target task from the source task.
Efficient Optimization for Linear Dynamical Systems with Applications to Clustering and Sparse Coding
Linear Dynamical Systems (LDSs) are fundamental tools for modeling spatio-temporal data in various disciplines.
RON: Reverse Connection with Objectness Prior Networks for Object Detection
To address (a), we design the reverse connection, which enables the network to detect objects on multi-levels of CNNs.
MAT: A Multimodal Attentive Translator for Image Captioning
In this way, the sequential representation of an image can be naturally translated to a sequence of words, as the target sequence of the RNN model.
Analyzing Linear Dynamical Systems: From Modeling to Coding and Learning
We then devise efficient algorithms to perform sparse coding and dictionary learning on the space of infinite-dimensional subspaces.
Sparse Coding and Dictionary Learning With Linear Dynamical Systems
To enhance the performance of LDSs, in this paper, we address the challenging issue of performing sparse coding on the space of LDSs, where both data and dictionary atoms are LDSs.
HyperNet: Towards Accurate Region Proposal Generation and Joint Object Detection
Almost all of the current top-performing object detection networks employ region proposals to guide the search for object instances.
