Search Results for author:
Found 198 papers, 87 papers with code
DataMix: Efficient Privacy-Preserving Edge-Cloud Inference
Third, our solution is extit{efficient} on the edge since the majority of the workload is delegated to the cloud, and our mixing and de-mixing processes introduce very few extra computations.
Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering
Language models (LMs) perform well on standardized coding benchmarks but struggle with real-world software engineering tasks such as resolving GitHub issues in SWE-Bench, especially when model parameters are less than 100B.
Learning 3D Persistent Embodied World Models
By conditioning the video model on this 3D spatial map, we illustrate how this enables video world models to faithfully simulate both seen and unseen parts of the world.
TesserAct: Learning 4D Embodied World Models
This paper presents an effective approach for learning novel 4D embodied world models, which predict the dynamic evolution of 3D scenes over time in response to an embodied agent's actions, providing both spatial and temporal consistency.
Towards Understanding Camera Motions in Any Video
We introduce CameraBench, a large-scale dataset and benchmark designed to assess and improve camera motion understanding.
AdaWorld: Learning Adaptable World Models with Latent Actions
To overcome this limitation, we propose AdaWorld, an innovative world model learning approach that enables efficient adaptation.
LuciBot: Automated Robot Policy Learning from Generated Videos
While prior works use large language models (LLMs) or vision-language models (VLMs) to generate rewards, these approaches are largely limited to simple tasks with well-defined rewards, such as pick-and-place.
Your Language Model May Think Too Rigidly: Achieving Reasoning Consistency with Symmetry-Enhanced Training
Large Language Models (LLMs) have demonstrated strong reasoning capabilities across various tasks.
Scaling Autonomous Agents via Automatic Reward Modeling And Planning
This model can be used to evaluate the action trajectories of LLM agents and provide heuristics for task planning.
Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
Despite external guidance, the effectiveness of this system demonstrates the potential of a single LLM to tackle complex tasks.
Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modeling
3D articulated objects modeling has long been a challenging problem, since it requires to capture both accurate surface geometries and semantically meaningful and spatially precise structures, parts, and joints.
VCA: Video Curious Agent for Long Video Understanding
Long video understanding poses unique challenges due to their temporal complexity and low information density.
LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences
Experiments show that our method surpasses existing methods on both large scene understanding and existing scene understanding benchmarks.
3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning
To support lifelong memory in active exploration settings, we present an incremental construction pipeline for 3D-Mem, as well as a memory retrieval technique for memory management.
Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting
Creating large-scale interactive 3D environments is essential for the development of Robotics and Embodied AI research.
Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge
In CHAIC, the goal is for an embodied agent equipped with egocentric observations to assist a human who may be operating under physical constraints -- e. g., unable to reach high places or confined to a wheelchair -- in performing common household or outdoor tasks as efficiently as possible.
DELTA: Dense Efficient Long-range 3D Tracking for any video
Our method provides a robust solution for applications requiring fine-grained, long-term motion tracking in 3D space.
SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization
The manual prompt design process for LLMs at the reasoning phase is tedious and an automated prompt optimization method is desired.
Ranked #1 on
Visual Social Relationship Recognition
on PIPA
UniMuMo: Unified Text, Music and Motion Generation
We introduce a music-motion parallel generation scheme that unifies all music and motion generation tasks into a single transformer decoder architecture with a single training task of music-motion joint generation.
Compositional Physical Reasoning of Objects and Events from Videos
The model is evaluated based on its capability to unravel the compositional hidden properties, such as mass and charge, and use this knowledge to answer a set of questions.
FlexAttention for Efficient High-Resolution Vision-Language Models
Specifically, a high-resolution image is encoded both as high-resolution tokens and low-resolution tokens, where only the low-resolution tokens and a few selected high-resolution tokens are utilized to calculate the attention map, which greatly shrinks the computational cost.
ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs
Compositional Reasoning (CR) entails grasping the significance of attributes, relations, and word order.
CoNav: A Benchmark for Human-Centered Collaborative Navigation
To achieve this goal, the agent needs to be equipped with a fundamental collaborative navigation ability, where the agent should reason human intention by observing human activities and then navigate to the human's intended destination in advance of the human.
Physically Compatible 3D Object Modeling from a Single Image
We present a computational framework that transforms single images into 3D physical objects.
RapVerse: Coherent Vocals and Whole-Body Motions Generations from Text
In this work, we introduce a challenging task for simultaneously generating 3D holistic body motions and singing vocals directly from textual lyrics inputs, advancing beyond existing works that typically address these two modalities in isolation.
LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery
Large Language Models have recently gained significant attention in scientific discovery for their extensive knowledge and advanced reasoning capabilities.
SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge
Learning commonsense reasoning from visual contexts and scenes in real-world is a crucial step toward advanced artificial intelligence.
STAR: A Benchmark for Situated Reasoning in Real-World Videos
This paper introduces a new benchmark that evaluates the situated reasoning ability via situation abstraction and logic-grounded question answering for real-world videos, called Situated Reasoning in Real-World Videos (STAR Benchmark).
QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
The key insight driving QServe is that the efficiency of LLM serving on GPUs is critically influenced by operations on low-throughput CUDA cores.
Virtual Foundry Graphnet for Metal Sintering Deformation Prediction
Metal Sintering is a necessary step for Metal Injection Molded parts and binder jet such as HP's metal 3D printer.
COMBO: Compositional World Models for Embodied Multi-Agent Cooperation
To effectively plan in this setting, in contrast to learning world dynamics in a single-agent scenario, we must simulate world dynamics conditioned on an arbitrary number of agents' actions given only partial egocentric visual observations of the world.
Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision
This paper answers this question in the context of tackling hard reasoning tasks (e. g., level 4-5 MATH problems) via learning from human annotations on easier tasks (e. g., level 1-3 MATH problems), which we term as easy-to-hard generalization.
3D-VLA: A 3D Vision-Language-Action Generative World Model
Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world.
ContPhy: Continuum Physical Concept Learning and Reasoning from Videos
We evaluated a range of AI models and found that they still struggle to achieve satisfactory performance on ContPhy, which shows that the current AI models still lack physical commonsense for the continuum, especially soft-bodies, and illustrates the value of the proposed dataset.
Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble
Reinforcement Learning from Human Feedback (RLHF) is a widely adopted approach for aligning large language models with human values.
HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments
Recent advances in high-fidelity virtual environments serve as one of the major driving forces for building intelligent embodied agents to perceive, reason and interact with the physical world.
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World
Human beings possess the capability to multiply a melange of multisensory cues while actively exploring and interacting with the 3D world.
RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation
We leverage Large Language Models (LLM) for zeroshot Semantic Audio Visual Navigation (SAVN).
Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance
We introduce Open3DIS, a novel solution designed to tackle the problem of Open-Vocabulary Instance Segmentation within 3D scenes.
Ranked #1 on
3D Open-Vocabulary Instance Segmentation
on S3DIS
3D Instance Segmentation
3D Open-Vocabulary Instance Segmentation
+4
DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
Combining gradient-based trajectory optimization with differentiable physics simulation is an efficient technique for solving soft-body manipulation problems.
DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning
In this paper, we propose a new approach that enables agents to learn whether their behaviors should be consistent with that of other agents by utilizing intrinsic rewards to learn the optimal policy for each agent.
Multi-agent Reinforcement Learning
reinforcement-learning
+3
GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs
If not, we initialize a new module needed by the task and specify the inputs and outputs of this new module.
CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
A communication token is generated by the LLM following a visual entity or a relation, to inform the detection network to propose regions that are relevant to the sentence generated so far.
RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation
We present RoboGen, a generative robotic agent that automatically learns diverse robotic skills at scale via generative simulation.
PockEngine: Sparse and Efficient Fine-tuning in a Pocket
On-device learning and efficient fine-tuning enable continuous and privacy-preserving customization (e. g., locally fine-tuning large language models on personalized data).
Autonomous Tree-search Ability of Large Language Models
This approach yield a greater improvement compared to the ones fine-tuned on CoT data.
Sparse Universal Transformer
The Universal Transformer (UT) is a variant of the Transformer that shares parameters across its layers.
TextPSG: Panoptic Scene Graph Generation from Textual Descriptions
To tackle this problem, we propose a new framework TextPSG consisting of four modules, i. e., a region grouper, an entity grounder, a segment merger, and a label generator, with several novel techniques.
SALMON: Self-Alignment with Instructable Reward Models
Supervised Fine-Tuning (SFT) on response demonstrations combined with Reinforcement Learning from Human Feedback (RLHF) constitutes a powerful paradigm for aligning LLM-based AI agents.
Generalizable Long-Horizon Manipulations with Large Language Models
This work introduces a framework harnessing the capabilities of Large Language Models (LLMs) to generate primitive task conditions for generalizable long-horizon manipulations with novel objects and unseen tasks.
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning
We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts.
Aligning Large Multimodal Models with Factually Augmented RLHF
Large Multimodal Models (LMM) are built across modalities and the misalignment between two modalities can result in "hallucination", generating textual outputs that are not grounded by the multimodal information in context.
$A^2$Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models
We study the task of zero-shot vision-and-language navigation (ZS-VLN), a practical yet challenging problem in which an agent learns to navigate following a path described by language instructions without requiring any path-instruction annotation data.
3D-LLM: Injecting the 3D World into Large Language Models
Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs.
Ranked #6 on
3D Object Captioning
on Objaverse
Learning Vision-and-Language Navigation from YouTube Videos
In this paper, we propose to learn an agent from these videos by creating a large-scale dataset which comprises reasonable path-instruction pairs from house tour videos and pre-training the agent on it.
Reparameterized Policy Learning for Multimodal Trajectory Optimization
We then present a practical model-based RL method, called Reparameterized Policy Gradient (RPG), which leverages the multimodal policy parameterization and learned world model to achieve strong exploration capabilities and high data efficiency.
Building Cooperative Embodied Agents Modularly with Large Language Models
In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments.
An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training
We present a model that can perform multiple vision tasks and can be adapted to other downstream tasks efficiently.
ModuleFormer: Modularity Emerges from Mixture-of-Experts
In our experiment, we found that the modular architecture enables three important abilities for large pre-trained language models: 1) Efficiency, since ModuleFormer only activates a subset of its modules for each input token, thus it could achieve the same performance as dense LLMs with more than two times throughput; 2) Extendability, ModuleFormer is more immune to catastrophic forgetting than dense LLMs and can be easily extended with new modules to learn new knowledge that is not included in the training data; 3) Specialisation, finetuning ModuleFormer could specialize a subset of modules to the finetuning task and the task-unrelated modules could be easily pruned for a lightweight deployment.
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
We propose Activation-aware Weight Quantization (AWQ), a hardware-friendly approach for LLM low-bit weight-only quantization.
SafeDiffuser: Safe Planning with Diffusion Probabilistic Models
Diffusion model-based approaches have shown promise in data-driven planning, but there are no safety guarantees, thus making it hard to be applied for safety-critical applications.
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to align the output of large language models (LLMs) with human intentions, ensuring they are helpful, ethical, and reliable.
Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics
Many NN approaches learn an end-to-end model that implicitly models both the governing PDE and constitutive models (or material models).
EC^2: Emergent Communication for Embodied Control
We learn embodied representations of video trajectories, emergent language, and natural language using a language model, which is then used to finetune a lightweight policy network for downstream control.
Learning Situation Hyper-Graphs for Video Question Answering
The proposed method is trained in an end-to-end manner and optimized by a VQA loss with the cross-entropy function and a Hungarian matching loss for the situation graph prediction.
Ranked #6 on
Video Question Answering
on AGQA 2.0 balanced
Hyper-Decision Transformer for Efficient Online Policy Adaptation
To address this challenge, we propose a new framework, called Hyper-Decision Transformer (HDT), that can generalize to novel tasks from a handful of demonstrations in a data- and parameter-efficient manner.
Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following
ECL consists of: (i) an instruction parser that translates the natural languages into executable programs; (ii) an embodied concept learner that grounds visual concepts based on language descriptions; (iii) a map constructor that estimates depth and constructs semantic maps by leveraging the learned concepts; and (iv) a program executor with deterministic policies to execute each program.
Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention
When looking at an image, we can decompose the scene into entities and their parts as well as obtain the dependencies between them.
Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos
Traditional methods of impact sound synthesis use physics simulation to obtain a set of physics parameters that could represent and synthesize the sound.
DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics
Reinforcement learning approaches for dexterous rigid object manipulation would struggle in this setting due to the complexity of physics interaction with deformable objects.
3D Concept Learning and Reasoning from Multi-View Images
We suggest that a principled approach for 3D reasoning from multi-view images should be to infer a compact 3D representation of the world from the multi-view images, which is further grounded on open-vocabulary semantic concepts, and then to execute reasoning on these 3D representations.
SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environments
Existing work has typically been tailored for particular environments or representations.
Planning with Large Language Models for Code Generation
Existing large language model-based code generation pipelines typically use beam search or sampling algorithms during the decoding process.
PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification
In this work, we aim to identify parameters characterizing a physical system from a set of multi-view videos without any assumption on object geometry or topology.
FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation
We identify several challenges for fluid manipulation learning by evaluating a set of reinforcement learning and trajectory optimization methods on our platform.
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning
The see stage scans the image and grounds the visual concept candidates with a visual perception model.
EfficientViT: Lightweight Multi-Scale Attention for High-Resolution Dense Prediction
Without performance loss on Cityscapes, our EfficientViT provides up to 8. 8x and 3. 8x GPU latency reduction over SegFormer and SegNeXt, respectively.
Mod-Squad: Designing Mixtures of Experts As Modular Multi-Task Learners
To address the MTL challenge, we propose Mod-Squad, a new model that is Modularized into groups of experts (a 'Squad').
EC2: Emergent Communication for Embodied Control
We learn embodied representations of video trajectories, emergent language, and natural language using a language model, which is then used to finetune a lightweight policy network for downstream control.
Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners
To address the MTL challenge, we propose Mod-Squad, a new model that is Modularized into groups of experts (a 'Squad').
CLAWSAT: Towards Both Robust and Accurate Code Models
We integrate contrastive learning (CL) with adversarial learning to co-optimize the robustness and accuracy of code models.
Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation
Effective planning of long-horizon deformable object manipulation requires suitable abstractions at both the spatial and temporal levels.
JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions
Hence, in order to achieve higher performance on our tasks, models need to effectively utilize such functional knowledge to infer the outcomes of actions, rather than relying solely on memorizing facts.
Revisiting the Roles of "Text" in Text Games
Text games present opportunities for natural language understanding (NLU) methods to tackle reinforcement learning (RL) challenges.
Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation
To achieve accurate and efficient navigation, it is critical to build a map that accurately represents both spatial location and the semantic information of the environment objects.
Learning Active Camera for Multi-Object Navigation
How to make robots perceive the environment as efficiently as humans is a fundamental problem in robotics.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
Learning Physical Dynamics with Subequivariant Graph Neural Networks
Graph Neural Networks (GNNs) have become a prevailing tool for learning physical dynamics.
Masked Motion Encoding for Self-Supervised Video Representation Learning
The latest attempts seek to learn a representation model by predicting the appearance contents in the masked regions.
Ranked #2 on
Self-Supervised Action Recognition
on HMDB51
On the Forward Invariance of Neural ODEs
We propose a new method to ensure neural ordinary differential equations (ODEs) satisfy output specifications by using invariance set propagation.
MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning
Decentralized learning has shown great promise for cooperative multi-agent reinforcement learning (MARL).
Gait Recognition in the Wild with Multi-hop Temporal Switch
Current methods that obtain state-of-the-art performance on in-the-lab benchmarks achieve much worse accuracy on the recently proposed in-the-wild datasets because these methods can hardly model the varied temporal dynamics of gait sequences in unconstrained scenes.
Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation
2) Prototype-based alignment and replay: based on the identified label prototypes, we align both domains and enforce the model to retain previous knowledge.
3D Concept Grounding on Neural Fields
Experimental results show that our proposed framework outperforms unsupervised/language-mediated segmentation models on semantic and instance segmentation tasks, as well as outperforms existing models on the challenging 3D aware visual reasoning tasks.
Finding Fallen Objects Via Asynchronous Audio-Visual Integration
The way an object looks and sounds provide complementary reflections of its physical properties.
Weakly Supervised Grounding for VQA in Vision-Language Transformers
Transformers for visual-language representation learning have been getting a lot of interest and shown tremendous performance on visual question answering (VQA) and grounding.
On-Device Training Under 256KB Memory
To reduce the memory footprint, we propose Sparse Update to skip the gradient computation of less important layers and sub-tensors.
Prompting Decision Transformer for Few-Shot Policy Generalization
Humans can leverage prior experience and learn novel tasks from a handful of demonstrations.
SNAKE: Shape-aware Neural 3D Keypoint Field
Detecting 3D keypoints from point clouds is important for shape reconstruction, while this work investigates the dual question: can shape reconstruction benefit 3D keypoint detection?
EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction
Without performance loss on Cityscapes, our EfficientViT provides up to 13. 9$\times$ and 6. 2$\times$ GPU latency reduction over SegFormer and SegNeXt, respectively.
Ranked #27 on
Semantic Segmentation
on Cityscapes val
RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation
To train this predictor, we formulate a new loss on rendering variances using gradients from differentiable rendering.
Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics
Extensive experimental results suggest that: 1) on multi-stage tasks that are infeasible for the vanilla differentiable physics solver, our approach discovers contact points that efficiently guide the solver to completion; 2) on tasks where the vanilla solver performs sub-optimally or near-optimally, our contact point discovery method performs better than or on par with the manipulation performance obtained with handcrafted contact points.
Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction
Specifically, FixNet consists of a perception module to extract the structured representation from the 3D point cloud, a physical dynamics prediction module to simulate the results of interactions on 3D objects, and a functionality prediction module to evaluate the functionality and choose the correct fix.
ComPhy: Compositional Physical Reasoning of Objects and Events from Videos
In this paper, we take an initial step to highlight the importance of inferring the hidden physical properties not directly observable from visual appearances, by introducing the Compositional Physical Reasoning (ComPhy) dataset.
Learning Neural Acoustic Fields
By modeling acoustic propagation in a scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds.
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools
We consider the problem of sequential robotic manipulation of deformable objects using tools.
FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations
We present a meta-learning framework for learning new visual concepts quickly, from just one or a few examples, guided by multiple naturally occurring data streams: simultaneously looking at images, reading sentences that describe the objects in the scene, and interpreting supplemental sentences that relate the novel concept with other concepts.
Linking Emergent and Natural Languages via Corpus Transfer
In this work, we propose a novel way to establish such a link by corpus transfer, i. e. pretraining on a corpus of emergent language for downstream natural language tasks, which is in contrast to prior work that directly transfers speaker and listener parameters.
AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation
To solve the above issues, we propose AutoGPart, a generic method enabling training generalizable 3D part segmentation networks with the task prior considered.
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierarchies.
Graph Convolutional Module for Temporal Action Localization in Videos
To this end, we propose a general graph convolutional module (GCM) that can be easily plugged into existing action localization methods, including two-stage and one-stage paradigms.
Ranked #2 on
Temporal Action Localization
on THUMOS’14
(mAP IOU@0.1 metric)
Memory-efficient Patch-based Inference for Tiny Deep Learning
We further propose receptive field redistribution to shift the receptive field and FLOPs to the later stage and reduce the computation overhead.
When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?
We show that AdvCL is able to enhance cross-task robustness transferability without loss of model accuracy and finetuning efficiency.
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language
This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine.
MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
We further propose network redistribution to shift the receptive field and FLOPs to the later stage and reduce the computation overhead.
Counterfactual Debiasing Inference for Compositional Action Recognition
By doing that, our proposed CDN method can better recognize unseen action instances by debiasing the effect of appearances.
Network Augmentation for Tiny Deep Learning
We introduce Network Augmentation (NetAug), a new training method for improving the performance of tiny neural networks.
OPEn: An Open-ended Physics Environment for Learning Without a Task
We test several existing RL-based exploration methods on this benchmark and find that an agent using unsupervised contrastive learning for representation learning, and impact-driven learning for exploration, achieved the best results.
Inducing Reusable Skills From Demonstrations with Option-Controller Network
With the isolation of information and the synchronous calling mechanism, we can impose a division of works between the controller and options in an end-to-end training regime.
TSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Device
Secondly, TSM has high efficiency; it achieves a high frame rate of 74fps and 29fps for online video recognition on Jetson Nano and Galaxy Note8.
Self-supervised Audiovisual Representation Learning for Remote Sensing Data
By fine-tuning the models on a number of commonly used remote sensing datasets, we show that our approach outperforms existing pre-training strategies for remote sensing imagery.
Ranked #2 on
Cross-Modal Retrieval
on SoundingEarth
Certifiably Robust Interpretation via Renyi Differential Privacy
The advantages of our Renyi-Robust-Smooth (RDP-based interpretation method) are three-folds.
Global Rhythm Style Transfer Without Text Transcriptions
In this paper, we propose AutoPST, which can disentangle global prosody style from speech without relying on any text transcriptions.
Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning
Cross-modal correlation provides an inherent supervision for video unsupervised representation learning.
Adversarial Option-Aware Hierarchical Imitation Learning
In particular, we propose an Expectation-Maximization(EM)-style algorithm: an E-step that samples the options of expert conditioned on the current learned policy, and an M-step that updates the low- and high-level policies of agent simultaneously to minimize the newly proposed option-occupancy measurement between the expert and the agent.
Temporal and Object Quantification Networks
We present Temporal and Object Quantification Networks (TOQ-Nets), a new class of neuro-symbolic networks with a structural bias that enables them to learn to recognize complex relational-temporal events.
Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules
In this paper, we focus on a more relaxed setting: the grounding of relevant visual entities in a weakly supervised manner by training on the VQA task alone.
Curious Representation Learning for Embodied Intelligence
Instead, it must explore its environment to acquire the data it will learn from.
PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics
Experimental results suggest that 1) RL-based approaches struggle to solve most of the tasks efficiently; 2) gradient-based approaches, by optimizing open-loop control sequences with the built-in differentiable physics engine, can rapidly find a solution within tens of iterations, but still fall short on multi-stage tasks that require long-term planning.
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos.
TransCenter: Transformers with Dense Representations for Multiple-Object Tracking
Methodologically, we propose the use of image-related dense detection queries and efficient sparse tracking queries produced by our carefully designed query learning networks (QLN).
Ranked #18 on
Multi-Object Tracking
on MOT20
(MOTA metric, using extra
training data)
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI
To complete the task, an embodied agent must plan a sequence of actions to change the state of a large number of objects in the face of realistic physical constraints.
Learning Task Decomposition with Ordered Memory Policy Network
The discovered subtask hierarchy could be used to perform task decomposition, recovering the subtask boundaries in an unstruc-tured demonstration.
AGENT: A Benchmark for Core Psychological Reasoning
For machine agents to successfully interact with humans in real-world settings, they will need to develop an understanding of human mental life.
Ranked #1 on
Core Psychological Reasoning
on AGENT
On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learning
Despite the generalization power of the meta-model, it remains elusive that how adversarial robustness can be maintained by MAML in few-shot learning.
Grounding Physical Object and Event Concepts Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos.
Learning Task Decomposition with Order-Memory Policy Network
Many complex real-world tasks are composed of several levels of sub-tasks.
Temporal and Object Quantification Nets
We aim to learn generalizable representations for complex activities by quantifying over both entities and time, as in “the kicker is behind all the other players,” or “the player controls the ball until it moves toward the goal.” Such a structural inductive bias of object relations, object quantification, and temporal orders will enable the learned representation to generalize to situations with varying numbers of agents, objects, and time courses.
Augmenting Policy Learning with Routines Discovered from a Single Demonstration
Humans can abstract prior knowledge from very little data and use it to boost skill learning.
Object-Centric Diagnosis of Visual Reasoning
In contrast, symbolic and modular models have a relatively better grounding and robustness, though at the cost of accuracy.
MVFNet: Multi-View Fusion Network for Efficient Video Recognition
Existing state-of-the-art methods have achieved excellent accuracy regardless of the complexity meanwhile efficient spatiotemporal modeling solutions are slightly inferior in performance.
Ranked #34 on
Action Recognition
on Something-Something V1
Synthetic Training for Monocular Human Mesh Recovery
To solve this problem, we design a multi-branch framework to disentangle the regression of different body properties, enabling us to separate each component's training in a synthetic training manner using unpaired data available.
RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning
We study unsupervised video representation learning that seeks to learn both motion and appearance features from unlabeled video only, which can be reused for downstream tasks such as action recognition.
Ranked #11 on
Self-Supervised Action Recognition
on UCF101
Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning
Interactive Fiction (IF) games with real human-written natural language texts provide a new natural evaluation for language understanding techniques.
Location-aware Graph Convolutional Networks for Video Question Answering
In this work, we propose to represent the contents in the video as a location-aware graph by incorporating the location information of an object into the graph construction.
Noisy Agents: Self-supervised Exploration by Predicting Auditory Events
Humans integrate multiple sensory modalities (e. g. visual and audio) to build a causal understanding of the physical world.
TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
Furthermore, combined with feature extractor adaptation, TinyTL provides 7. 3-12. 9x memory saving without sacrificing accuracy compared to fine-tuning the full Inception-V3.
Foley Music: Learning to Generate Music from Videos
In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments.
MCUNet: Tiny Deep Learning on IoT Devices
Machine learning on tiny IoT devices based on microcontroller units (MCU) is appealing but challenging: the memory of microcontrollers is 2-3 orders of magnitude smaller even than mobile phones.
Generating Visually Aligned Sound from Videos
During testing, the audio forwarding regularizer is removed to ensure that REGNET can produce purely aligned sound only from visual features.
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation.
Language Guided Networks for Cross-modal Moment Retrieval
In this paper, we present Language Guided Networks (LGN), a new framework that leverages the sentence embedding to guide the whole process of moment retrieval.
A Real-time Action Representation with Temporal Encoding and Deep Compression
Our method achieves clear improvements on UCF101 action recognition benchmark against state-of-the-art real-time methods by 5. 4% in terms of accuracy and 2 times faster in terms of inference speed with a less than 5MB storage model.
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
To enable low-latency inference on resource-constrained hardware platforms, we propose to design Hardware-Aware Transformers (HAT) with neural architecture search.
Ranked #21 on
Machine Translation
on WMT2014 English-French
Deep Audio Priors Emerge From Harmonic Convolutional Networks
We show that networks using Harmonic Convolution can reliably model audio priors and achieve high performance in unsupervised audio restoration tasks.
Once for All: Train One Network and Specialize it for Efficient Deployment
Most of the traditional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally expensive and unscalable.
Music Gesture for Visual Sound Separation
Recent deep learning approaches have achieved impressive performance on visual sound separation tasks.
Dense Regression Network for Video Grounding
The key idea of this paper is to use the distances between the frame within the ground truth and the starting (ending) frame as dense supervisions to improve the video grounding accuracy.
Natural Language Moment Retrieval
Natural Language Queries
+2
Visual Concept-Metaconcept Learning
Humans reason with concepts and metaconcepts: we recognize red and green from visual input; we also understand that they describe the same property of objects (i. e., the color).
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation
In this paper, we attempt to approach the problem of Audio-Visual Embodied Navigation, the task of planning the shortest path from a random starting location in a scene to the sound source in an indoor environment, given only raw egocentric visual and audio sensory data.
Cross-channel Communication Networks
Convolutional neural networks process input data by sending channel-wise feature response maps to subsequent layers.
Self-supervised Moving Vehicle Tracking with Stereo Sound
At test time, the stereo-sound student network can work independently to perform object localization us-ing just stereo audio and camera meta-data, without any visual input.
TruNet: Short Videos Generation from Long Videos via Story-Preserving Truncation
In this work, we introduce a new problem, named as {\em story-preserving long video truncation}, that requires an algorithm to automatically truncate a long-duration video into multiple short and attractive sub-videos with each one containing an unbroken story.
Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement
This paper studies Learning from Observations (LfO) for imitation learning with access to state-only demonstrations.
CLEVRER: CoLlision Events for Video REpresentation and Reasoning
While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations.
Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos
With such hardware-aware model design, we are able to scale up the training on Summit supercomputer and reduce the training time on Kinetics dataset from 49 hours 55 minutes to 14 minutes 13 seconds, achieving a top-1 accuracy of 74. 0%, which is 1. 6x and 2. 9x faster than previous 3D video models with higher accuracy.
Graph Convolutional Networks for Temporal Action Localization
Then we apply the GCNs over the graph to model the relations among different proposals and learn powerful representations for the action classification and localization.
Ranked #4 on
Temporal Action Localization
on THUMOS’14
(mAP IOU@0.1 metric)
Deep Concept-wise Temporal Convolutional Networks for Action Localization
In this paper, we empirically find that stacking more conventional temporal convolution layers actually deteriorates action classification performance, possibly ascribing to that all channels of 1D feature map, which generally are highly abstract and can be regarded as latent concepts, are excessively recombined in temporal convolution.
Once-for-All: Train One Network and Specialize it for Efficient Deployment
On diverse edge devices, OFA consistently outperforms state-of-the-art (SOTA) NAS methods (up to 4. 0% ImageNet top1 accuracy improvement over MobileNetV3, or same accuracy but 1. 5x faster than MobileNetV3, 2. 6x faster than EfficientNet w. r. t measured latency) while reducing many orders of magnitude GPU hours and $CO_2$ emission.
Ranked #76 on
Neural Architecture Search
on ImageNet
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation.
Ranked #6 on
Visual Question Answering (VQA)
on CLEVR
Self-Supervised Audio-Visual Co-Segmentation
Segmenting objects in images and separating sound sources in audio are challenging tasks, in part because traditional approaches require large amounts of labeled data.
Defensive Quantization: When Efficiency Meets Robustness
This paper aims to raise people's awareness about the security of the quantized models, and we designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models.
Interpreting Adversarial Examples by Activation Promotion and Suppression
It is widely known that convolutional neural networks (CNNs) are vulnerable to adversarial examples: images with imperceptible perturbations crafted to fool classifiers.
Weakly Supervised Dense Event Captioning in Videos
Dense event captioning aims to detect and describe all events of interest contained in a video.
TSM: Temporal Shift Module for Efficient Video Understanding
The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost.
Ranked #2 on
3D Action Recognition
on Assembly101
StNet: Local and Global Spatial-Temporal Modeling for Action Recognition
In this paper, in contrast to the existing CNN+RNN or pure 3D convolution based approaches, we explore a novel spatial temporal network (StNet) architecture for both local and global spatial-temporal modeling in videos.
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
Second, the model is more data- and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering.
Ranked #1 on
Visual Question Answering (VQA)
on CLEVR
Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation
The recent advances in deep learning have made it possible to generate photo-realistic images by using neural networks and even to extrapolate video frames from an input video clip.
Geometry Guided Convolutional Neural Networks for Self-Supervised Video Representation Learning
In addition, we also find that a progressive training strategy can foster a better neural network for the video recognition task than blindly pooling the distinct sources of geometry cues together.
Sparse, Smart Contours to Represent and Edit Images
We study the problem of reconstructing an image from information stored at contour locations.
The Sound of Pixels
We introduce PixelPlayer, a system that, by leveraging large amounts of unlabeled videos, learns to locate image regions which produce sounds and separate the input sounds into a set of components that represents the sound from each pixel.
End-to-End Learning of Motion Representation for Video Understanding
Despite the recent success of end-to-end learned representations, hand-crafted optical flow features are still widely used in video analysis tasks.
Ranked #45 on
Action Recognition
on UCF101
Smart, Sparse Contours to Represent and Edit Images
We study the problem of reconstructing an image from information stored at contour locations.
Unsupervised Domain Adaptation for 3D Keypoint Estimation via View Consistency
In this paper, we introduce a novel unsupervised domain adaptation technique for the task of 3D keypoint prediction from a single depth scan or image.
Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification
In this paper, however, we show that temporal information, especially longer-term patterns, may not be necessary to achieve competitive results on common video classification datasets.
VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
Many seemingly distant annotations (e. g., semantic segmentation and visual question answering (VQA)) are inherently connected in that they reveal different levels and perspectives of human understandings about the same visual scenes --- and even the same set of images (e. g., of COCO).
Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification
Experiment results on the challenging Kinetics dataset demonstrate that our proposed temporal modeling approaches can significantly improve existing approaches in the large-scale video recognition tasks.
Ranked #169 on
Action Classification
on Kinetics-400
Temporal Modeling Approaches for Large-scale Youtube-8M Video Understanding
This paper describes our solution for the video recognition task of the Google Cloud and YouTube-8M Video Understanding Challenge that ranked the 3rd place.
StyleNet: Generating Attractive Visual Captions With Styles
We propose a novel framework named StyleNet to address the task of generating attractive captions for images and videos with different styles.
Recurrent Topic-Transition GAN for Visual Paragraph Generation
The proposed Recurrent Topic-Transition Generative Adversarial Network (RTT-GAN) builds an adversarial framework between a structured paragraph generator and multi-level paragraph discriminators.
Generative Adversarial Network
Image Paragraph Captioning
+1
Video Captioning with Multi-Faceted Attention
Recently, video captioning has been attracting an increasing amount of interest, due to its potential for improving accessibility and information retrieval.
Semantic Compositional Networks for Visual Captioning
The degree to which each member of the ensemble is used to generate an image caption is tied to the image-dependent probability of the corresponding tag.
Strategies for Searching Video Content with Text Queries or Video Examples
The large number of user-generated videos uploaded on to the Internet everyday has led to many commercial video search engines, which mainly rely on text metadata for search.
You Lead, We Exceed: Labor-Free Video Concept Learning by Jointly Exploiting Web Videos and Images
The Web images are then filtered by the learnt network and the selected images are additionally fed into the network to enhance the architecture and further trim the videos.
Learning Attributes Equals Multi-Source Domain Generalization
Attributes possess appealing properties and benefit many computer vision problems, such as object recognition, learning with humans in the loop, and image retrieval.
DevNet: A Deep Event Network for Multimedia Event Detection and Evidence Recounting
Taking key frames of videos as input, we first detect the event of interest at the video level by aggregating the CNN features of the key frames.
