Search Results for author:
Found 19 papers, 11 papers with code
3D-VLA: A 3D Vision-Language-Action Generative World Model
Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world.
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World
Human beings possess the capability to multiply a melange of multisensory cues while actively exploring and interacting with the 3D world.
SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-world Object Detector
Ablation experiments demonstrate that both of them are effective in mitigating the impact of open-world knowledge distillation on the learning of known objects.
DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning
In this paper, we propose a new approach that enables agents to learn whether their behaviors should be consistent with that of other agents by utilizing intrinsic rewards to learn the optimal policy for each agent.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
A communication token is generated by the LLM following a visual entity or a relation, to inform the detection network to propose regions that are relevant to the sentence generated so far.
$A^2$Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models
We study the task of zero-shot vision-and-language navigation (ZS-VLN), a practical yet challenging problem in which an agent learns to navigate following a path described by language instructions without requiring any path-instruction annotation data.
3D-LLM: Injecting the 3D World into Large Language Models
Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs.
Learning Vision-and-Language Navigation from YouTube Videos
In this paper, we propose to learn an agent from these videos by creating a large-scale dataset which comprises reasonable path-instruction pairs from house tour videos and pre-training the agent on it.
Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition
Although PTMs shed new light on artificial general intelligence, they are constructed with general tasks in mind, and thus, their efficacy for specific tasks can be further improved.
Detecting the open-world objects with the help of the Brain
We propose leveraging the VL as the ``Brain'' of the open-world detector by simply generating unknown labels.
Learning Active Camera for Multi-Object Navigation
How to make robots perceive the environment as efficiently as humans is a fundamental problem in robotics.
Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation
To achieve accurate and efficient navigation, it is critical to build a map that accurately represents both spatial location and the semantic information of the environment objects.
Masked Motion Encoding for Self-Supervised Video Representation Learning
The latest attempts seek to learn a representation model by predicting the appearance contents in the masked regions.
Ranked #2 on
Self-Supervised Action Recognition
on HMDB51
RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning
We study unsupervised video representation learning that seeks to learn both motion and appearance features from unlabeled video only, which can be reused for downstream tasks such as action recognition.
Ranked #11 on
Self-Supervised Action Recognition
on UCF101
Location-aware Graph Convolutional Networks for Video Question Answering
In this work, we propose to represent the contents in the video as a location-aware graph by incorporating the location information of an object into the graph construction.
Foley Music: Learning to Generate Music from Videos
In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments.
Generating Visually Aligned Sound from Videos
During testing, the audio forwarding regularizer is removed to ensure that REGNET can produce purely aligned sound only from visual features.
Dense Regression Network for Video Grounding
The key idea of this paper is to use the distances between the frame within the ground truth and the starting (ending) frame as dense supervisions to improve the video grounding accuracy.
Natural Language Moment Retrieval
Natural Language Queries
+2
Self-supervised Moving Vehicle Tracking with Stereo Sound
At test time, the stereo-sound student network can work independently to perform object localization us-ing just stereo audio and camera meta-data, without any visual input.
