Search Results for author:
Found 171 papers, 62 papers with code
Test-Time Training for Generalization under Distribution Shifts
We introduce a general approach, called test-time training, for improving the performance of predictive models when training and test data come from different distributions.
TD-MPC2: Scalable, Robust World Models for Continuous Control
TD-MPC is a model-based reinforcement learning (RL) algorithm that performs local trajectory optimization in the latent space of a learned implicit (decoder-free) world model.
Finetuning Offline World Models in the Real World
In this work, we seek to get the best of both worlds: we consider the problem of pretraining a world model with offline data collected on a real robot, and then finetuning the model on online data collected by planning with the learned model.
Learning to (Learn at Test Time)
Our inner loop turns out to be equivalent to linear attention when the inner-loop learner is only a linear model, and to self-attention when it is a kernel estimator.
GenSim: Generating Robotic Simulation Tasks via Large Language Models
Collecting large amounts of real-world interaction data to train general robotic policies is often prohibitively expensive, thus motivating the use of simulation data.
Generalized Animal Imitator: Agile Locomotion with Versatile Motion Prior
To our understanding, this is the first work that allows a robot to concurrently learn diverse agile locomotion tasks using a singular controller.
3D Reconstruction with Generalizable Neural Fields using Scene Priors
NFP not only demonstrates SOTA scene reconstruction performance and efficiency, but it also supports single-image novel-view synthesis, which is underexplored in neural fields.
SHAPE: A Sample-adaptive Hierarchical Prediction Network for Medication Recommendation
Effectively medication recommendation with complex multimorbidity conditions is a critical task in healthcare.
Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf
Communication games, which we refer to as incomplete information games that heavily depend on natural language communication, hold significant research value in fields such as economics, social science, and artificial intelligence.
PointLLM: Empowering Large Language Models to Understand Point Clouds
The unprecedented advancements in Large Language Models (LLMs) have created a profound impact on natural language processing but are yet to fully embrace the realm of 3D understanding.
GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields
To incorporate semantics in 3D, the reconstruction module utilizes a vision-language foundation model ($\textit{e. g.}$, Stable Diffusion) to distill rich semantic information into the deep 3D voxel.
Learning Dense Correspondences between Photos and Sketches
Humans effortlessly grasp the connection between sketches and real-world objects, even when these sketches are far from realistic.
Pluggable Neural Machine Translation Models via Memory-augmented Adapters
Although neural machine translation (NMT) models perform well in the general domain, it remains rather challenging to control their generation behavior to satisfy the requirement of different users.
Causal Kripke Models
This work extends Halpern and Pearl's causal models for actual causality to a possible world semantics environment.
Test-Time Training on Video Streams
Before making a prediction on each test instance, the model is trained on the same instance using a self-supervised task, such as image reconstruction with masked autoencoders.
AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System
For real-world experiments, AnyTeleop can outperform a previous system that was designed for a specific robot hardware with a higher success rate, using the same robot.
Elastic Decision Transformer
This paper introduces Elastic Decision Transformer (EDT), a significant advancement over the existing Decision Transformer (DT) and its variants.
Zero-shot Pose Transfer for Unrigged Stylized 3D Characters
We present a zero-shot approach that requires only the widely available deformed non-stylized avatars in training, and deforms stylized characters of significantly different shapes at inference.
Medication Recommendation via Domain Knowledge Informed Deep Learning
However, most of them neglect incorporating domain knowledge according to the clinical manifestations in the EHR of the patient.
DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects
On the other hand, operating with a multi-finger robot hand will allow better approximation to human behavior and enable the robot to operate on diverse articulated objects.
TUVF: Learning Generalizable Texture UV Radiance Fields
This allows the texture to be disentangled from the underlying shape and transferable to other shapes that share the same UV space, i. e., from the same category.
ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimation
We propose a new dataset and a novel approach to learning hand-object interaction priors for hand and articulated object pose estimation.
ActorsNeRF: Animatable Few-shot Human Rendering with Generalizable NeRFs
While NeRF-based human representations have shown impressive novel view synthesis results, most methods still rely on a large number of images / views for training.
Neural Volumetric Memory for Visual Locomotion Control
To solve this problem, we follow the paradigm in computer vision that explicitly models the 3D geometry of the scene and propose Neural Volumetric Memory (NVM), a geometric memory architecture that explicitly accounts for the SE(3) equivariance of the 3D world.
Investigating and Mitigating the Side Effects of Noisy Views for Self-Supervised Clustering Algorithms in Practical Multi-View Scenarios
Multi-view clustering (MVC) aims at exploring category structures among multi-view data in self-supervised manners.
FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models
Recent works on generalizable NeRFs have shown promising results on novel view synthesis from single or few images.
Rotating without Seeing: Towards In-hand Dexterity through Touch
Relying on touch-only sensing, we can directly deploy the policy in a real robot hand and rotate novel objects that are not presented in training.
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
Our approach outperforms the previous state of the art by significant margins on both open-vocabulary panoptic and semantic segmentation tasks.
Ranked #2 on
Open Vocabulary Panoptic Segmentation
on ADE20K
Open Vocabulary Panoptic Segmentation
Open Vocabulary Semantic Segmentation
+3
Dynamic Inference With Grounding Based Vision and Language Models
For example, recent image and language models with more than 200M parameters have been proposed to learn visual grounding in the pre-training step and show impressive results on downstream vision and language tasks.
Policy Adaptation from Foundation Model Feedback
When deploying the trained policy to a new task or a new environment, we first let the policy play with randomly generated instructions to record the demonstrations.
GPViT: A High Resolution Non-Hierarchical Vision Transformer with Group Propagation
In each GP Block, features are first grouped together by a fixed number of learnable group tokens; we then perform Group Propagation where global information is exchanged between the grouped features; finally, global information in the updated grouped features is returned back to the image features through a transformer decoder.
On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline
In this paper, we examine the effectiveness of pre-training for visuo-motor control tasks.
MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations
We identify key ingredients for leveraging demonstrations in model learning -- policy pretraining, targeted exploration, and oversampling of demonstration data -- which forms the three phases of our model-based RL framework.
Model-based Reinforcement Learning
reinforcement-learning
+1
DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation
We empirically evaluate our method using an Allegro Hand to grasp novel objects in both simulation and real world.
Visual Reinforcement Learning with Self-Supervised 3D Representations
A prominent approach to visual Reinforcement Learning (RL) is to learn an internal state representation using self-supervised methods, which has the potential benefit of improved sample-efficiency and generalization through additional learning signal and inductive biases.
MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Pose
We propose a generalizable neural radiance fields - MonoNeRF, that can be trained on large-scale monocular videos of moving in static scenes without any ground-truth annotations of depth and camera poses.
Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild
While 6D object pose estimation has wide applications across computer vision and robotics, it remains far from being solved due to the lack of annotations.
Transformers as Meta-Learners for Implicit Neural Representations
Motivated by a generalized formulation of gradient-based meta-learning, we propose a formulation that uses Transformers as hypernetworks for INRs, where it can directly build the whole set of INR weights with Transformers specialized as set-to-set mapping.
Graph Inverse Reinforcement Learning from Diverse Videos
Research on Inverse Reinforcement Learning (IRL) from third-person videos has shown encouraging results on removing the need for manual reward design for robotic tasks.
Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations
We will first convert the large-scale human-object interaction trajectories to robot demonstrations via motion retargeting, and then use these demonstrations to train CGF.
Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset
6D object pose estimation is one of the fundamental problems in computer vision and robotics research.
Learning Implicit Feature Alignment Function for Semantic Segmentation
As such, IFA implicitly aligns the feature maps at different levels and is capable of producing segmentation maps in arbitrary resolutions.
Medical Dialogue Response Generation with Pivotal Information Recalling
To mitigate this problem, we propose a medical response generation model with Pivotal Information Recalling (MedPIR), which is built on two components, i. e., knowledge-aware dialogue graph encoder and recall-enhanced generator.
MSDF: A General Open-Domain Multi-Skill Dialog Framework
In this paper, we propose a general Multi-Skill Dialog Framework, namely MSDF, which can be applied in different dialog tasks (e. g. knowledge grounded dialog and persona based dialog).
VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution
The learned implicit neural representation can be decoded to videos of arbitrary spatial resolution and frame rate.
Space-time Video Super-resolution
Video Frame Interpolation
+1
CATNet: Cross-event Attention-based Time-aware Network for Medical Event Prediction
It is a time-aware, event-aware and task-adaptive method with the following advantages: 1) modeling heterogeneous information and temporal information in a unified way and considering temporal irregular characteristics locally and globally respectively, 2) taking full advantage of correlations among different types of events via cross-event attention.
From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation
We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand.
GIFS: Neural Implicit Function for General Shape Representation
This limitation leads to tedious data processing (converting non-watertight raw data to watertight) as well as the incapability of representing general object shapes in the real world.
Learning Generalizable Dexterous Manipulation from Human Grasp Affordance
In this paper, we propose to learn dexterous manipulation using large-scale demonstrations with diverse 3D objects in a category, which are generated from a human grasp affordance model.
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos
To tackle this task, we first provide an automatic way to collect trajectory and hotspots labels on large-scale data.
CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs
We represent the correspondence maps of different images as warped coordinate frames transformed from a canonical coordinate frame, i. e., the correspondence map, which describes the structure (e. g., the shape of a face), is controlled via a transformation.
Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Image
Novel view synthesis from a single image has recently attracted a lot of attention, and it has been primarily advanced by 3D deep learning and rendering techniques.
Temporal Difference Learning for Model Predictive Control
Data-driven model predictive control has two key advantages over model-free methods: a potential for improved sample efficiency through model learning, and better performance as computational budget for planning increases.
GroupViT: Semantic Segmentation Emerges from Text Supervision
With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i. e., without any further fine-tuning.
Multimodal data matters: language model pre-training over structured and unstructured electronic health records
As two important textual modalities in electronic health records (EHR), both structured data (clinical codes) and unstructured data (clinical narratives) have recently been increasingly applied to the healthcare domain.
Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation
We propose a setting for robotic manipulation in which the agent receives visual feedback from both a third-person camera and an egocentric camera mounted on the robot's wrist.
NovelD: A Simple yet Effective Exploration Criterion
We analyze NovelD thoroughly in MiniGrid and found that empirically it helps the agent explore the environment more uniformly with a focus on exploring beyond the boundary.
Learning Continuous Environment Fields via Implicit Functions
We propose a novel scene representation that encodes reaching distance -- the distance between any position in the scene to a goal along a feasible trajectory.
Online Adaptation for Implicit Object Tracking and Shape Reconstruction in the Wild
Tracking and reconstructing 3D objects from cluttered scenes are the key components for computer vision, robotics and autonomous driving systems.
Multi-Person 3D Motion Prediction with Multi-Range Transformers
Thus, instead of predicting each human pose trajectory in isolation, we introduce a Multi-Range Transformers model which contains of a local-range encoder for individual motion and a global-range encoder for social interactions.
Video Autoencoder: self-supervised disentanglement of static 3D structure and motion
Relying on temporal continuity in videos, our work assumes that the 3D scene structure in nearby video frames remains static.
Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization
While Reinforcement Learning (RL) provides a promising paradigm for agile locomotion skills with vision inputs in simulation, it is still very challenging to deploy the RL policy in the real world.
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos
While significant progress has been made on understanding hand-object interactions in computer vision, it is still very challenging for robots to perform complex dexterous manipulation.
Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
Our key insight is that proprioceptive states only offer contact measurements for immediate reaction, whereas an agent equipped with visual sensory observations can learn to proactively maneuver environments with obstacles and uneven terrain by anticipating changes in the environment many steps ahead.
Test-Time Personalization with a Transformer for Human Pose Estimation
During test time, we personalize and adapt our model by fine-tuning with the self-supervised objective.
Sentence-level Online Handwritten Chinese Character Recognition
Single online handwritten Chinese character recognition~(single OLHCCR) has achieved prominent performance.
Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
Our method greatly improves stability and sample efficiency of ConvNets under augmentation, and achieves generalization results competitive with state-of-the-art methods for image-based RL in environments with unseen visuals.
GlyphCRM: Bidirectional Encoder Representation for Chinese Character with its Glyph
Previous works indicate that the glyph of Chinese characters contains rich semantic information and has the potential to enhance the representation of Chinese characters.
Single RGB-D Camera Teleoperation for General Robotic Manipulation
We propose a teleoperation system that uses a single RGB-D camera as the human motion capture device.
DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation
We address the problem of safely solving complex bimanual robot manipulation tasks with sparse rewards.
Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time
Estimating 3D hand and object pose from a single image is an extremely challenging problem: hands and objects are often self-occluded during interactions, and the 3D annotations are scarce as even humans cannot directly label the ground-truths from a single image perfectly.
Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency
In this paper, we propose a novel contrastive learning method which explores the cross-video relation by using cycle-consistency for general image representation learning.
Robust Object Detection via Instance-Level Temporal Cycle Confusion
Building reliable object detectors that are robust to domain shifts, such as various changes in context, viewpoint, and object appearances, is critical for real-world applications.
A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation
To deal with the large shape variance, we introduce Articulated Signed Distance Functions (A-SDF) to represent articulated shapes with a disentangled latent space, where we have separate codes for encoding shape and articulation.
Hand-Object Contact Consistency Reasoning for Human Grasps Generation
Based on the hand-object contact consistency, we design novel objectives in training the human grasp generation model and also a new self-supervised task which allows the grasp generation network to be adjusted even during test time.
Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective
To learn generalizable representation for correspondence in large-scale, a variety of self-supervised pretext tasks are proposed to explicitly perform object-level or patch-level similarity learning.
Region Similarity Representation Learning
We present Region Similarity Representation Learning (ReSim), a new approach to self-supervised representation learning for localization-based tasks such as object detection and segmentation.
Solving Compositional Reinforcement Learning Problems via Task Reduction
We propose a novel learning paradigm, Self-Imitation via Reduction (SIR), for solving compositional reinforcement learning problems.
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization
We propose a simple, general and effective technique, Reward Randomization for discovering diverse strategic policies in complex multi-agent games.
Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency
We propose \textit{dynamics cycles} that align dynamic robot behavior across two domains using a cycle-consistency constraint.
Learning Continuous Image Representation with Local Implicit Image Function
How to represent an image?
Ranked #2 on
Image Super-Resolution
on DIV2K val - 4x upscaling
(SSIM metric)
BeBold: Exploration Beyond the Boundary of Explored Regions
In this paper, we analyze the pros and cons of each method and propose the regulated difference of inverse visitation counts as a simple but effective criterion for IR.
Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes
Synthesizing 3D human motion plays an important role in many graphics applications as well as understanding human activity.
Online Adaptation for Consistent Mesh Reconstruction in the Wild
This paper presents an algorithm to reconstruct temporally consistent 3D meshes of deformable object instances from videos in the wild.
MedWriter: Knowledge-Aware Medical Text Generation
To exploit the domain knowledge to guarantee the correctness of generated text has been a hot topic in recent years, especially for high professional domains such as medical.
Generalization in Reinforcement Learning by Soft Data Augmentation
Extensive efforts have been made to improve the generalization ability of Reinforcement Learning (RL) methods via domain randomization and data augmentation.
Continual Learning Long Short Term Memory
Catastrophic forgetting in neural networks indicates the performance decreasing of deep learning models on previous tasks while learning new tasks.
Multi-Agent Collaboration via Reward Attribution Decomposition
In this work, we propose Collaborative Q-learning (CollaQ) that achieves state-of-the-art performance in the StarCraft multi-agent challenge and supports ad hoc team play.
Reducing Class Collapse in Metric Learning with Easy Positive Sampling
We theoretically prove and empirically show that under reasonable noise assumptions, prevalent embedding losses in metric learning, e. g., triplet loss, tend to project all samples of a class with various modes onto a single point in the embedding space, resulting in a class collapse that usually renders the space ill-sorted for classification or retrieval.
Hierarchical Style-based Networks for Motion Synthesis
Generating diverse and natural human motion is one of the long-standing goals for creating intelligent characters in the animated world.
Fast ORB-SLAM without Keypoint Descriptors
ORB-SLAM2 \cite{orbslam2} is a benchmark method in this domain, however, it consumes significant time for computing descriptors that never get reused unless a frame is selected as a keyframe.
Robotics Computational Geometry I.4.0; I.4.9
What Should Not Be Contrastive in Contrastive Learning
Recent self-supervised contrastive methods have been able to produce impressive transferable visual representations by learning to be invariant to different data augmentations.
Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
Learning long-term dynamics models is the key to understanding physical common sense.
Ranked #1 on
Visual Reasoning
on PHYRE-1B-Within
Self-Supervised Policy Adaptation during Deployment
A natural solution would be to keep training after deployment in the new environment, but this cannot be done if the new environment offers no reward signal.
Deep Isometric Learning for Visual Recognition
Initialization, normalization, and skip connections are believed to be three indispensable techniques for training very deep convolutional neural networks and obtaining state-of-the-art performance.
Compositional Video Synthesis with Action Graphs
Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation.
Rethinking preventing class-collapsing in metric learning with margin-based losses
We theoretically prove and empirically show that under reasonable noise assumptions, margin-based losses tend to project all samples of a class with various modes onto a single point in the embedding space, resulting in a class collapse that usually renders the space ill-sorted for classification or retrieval.
Class-Aware Domain Adaptation for Improving Adversarial Robustness
To improve the adversarial robustness of neural networks, adversarial training has been proposed to train networks by injecting adversarial examples into the training data.
State-Only Imitation Learning for Dexterous Manipulation
To tackle this setting, we train an inverse dynamics model and use it to predict actions for state-only demonstrations.
Multi-Task Reinforcement Learning with Soft Modularization
While training multiple tasks jointly allow the policies to share parameters across different tasks, the optimization problem becomes non-trivial: It remains unclear what parameters in the network should be reused across tasks, and how the gradients from different tasks may interfere with each other.
Ranked #1 on
Meta-Learning
on MT50
Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so it is particularly challenging to learn good policies when the agent population is large.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning
The edge between these two lines of works has yet been underexplored, and the effectiveness of meta-learning in few-shot learning remains unclear.
Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks
Human action is naturally compositional: humans can easily recognize and perform actions with objects that are different from those used in training demonstrations.
A Deep Learning-Based System for PharmaCoNER
The Biological Text Mining Unit at BSC and CNIO organized the first shared task on chemical {\&} drug mention recognition from Spanish medical texts called PharmaCoNER (Pharmacological Substances, Compounds and proteins and Named Entity Recognition track) in 2019, which includes two tracks: one for NER offset and entity classification (track 1) and the other one for concept indexing (track 2).
Family history information extraction via deep joint learning
For this task, we propose a system based on deep joint learning methods to extract FH information.
Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
In this paper, we propose Test-Time Training, a general approach for improving the performance of predictive models when training and test data come from different distributions.
Building change detection for remote sensing images
CARLA MAP Leaderboard
+5
Joint-task Self-supervised Learning for Temporal Correspondence
Our learning process integrates two highly related tasks: tracking large image regions \emph{and} establishing fine-grained pixel-level associations between consecutive video frames.
IS THE LABEL TRUSTFUL: TRAINING BETTER DEEP LEARNING MODEL VIA UNCERTAINTY MINING NET
During the training stage, we utilize all the available data (labeled and unlabeled) to train the classifier via a semi-supervised generative framework.
Test-Time Training for Out-of-Distribution Generalization
We introduce a general approach, called test-time training, for improving the performance of predictive models when test and training data come from different distributions.
Deep Kinship Verification via Appearance-shape Joint Prediction and Adaptation-based Approach
Kinship verification aims to identify the kin relation between two given face images.
Accelerating Proposal Generation Network for \\Fast Face Detection on Mobile Devices
In this work, we present a proposal generation acceleration framework for real-time face detection.
Learning Correspondence from the Cycle-Consistency of Time
We introduce a self-supervised method for learning visual correspondence from unlabeled video.
Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments
In order to predict valid affordances and learn possible 3D human poses in indoor scenes, we need to understand the semantic and geometric structure of a scene as well as its potential interactions with a human.
MICIK: MIning Cross-Layer Inherent Similarity Knowledge for Deep Model Compression
State-of-the-art deep model compression methods exploit the low-rank approximation and sparsity pruning to remove redundant parameters from a learned hidden layer.
Spatio-Temporal Action Graph Networks
Events defined by the interaction of objects in a scene are often of critical importance; yet important events may have insufficient labeled examples to train a conventional deep model to generalize to future object appearance.
Visual Semantic Navigation using Scene Priors
Do we use the semantic/functional priors we have built over years to efficiently search and navigate?
Interpretable Intuitive Physics Model
In order to demonstrate that our system models these underlying physical properties, we train our model on collisions of different shapes (cube, cone, cylinder, spheres etc.)
Videos as Space-Time Region Graphs
These nodes are connected by two types of relations: (i) similarity relations capturing the long range dependencies between correlated objects and (ii) spatial-temporal relations capturing the interactions between nearby objects.
Ranked #31 on
Action Classification
on Charades
(using extra training data)
Dynamically Hierarchy Revolution: DirNet for Compressing Recurrent Neural Network on Mobile Devices
Recurrent neural networks (RNNs) achieve cutting-edge performance on a variety of problems.
LSDSCC: a Large Scale Domain-Specific Conversational Corpus for Response Generation with Diversity Oriented Evaluation Metrics
The experimental results have shown that our proposed corpus can be taken as a new benchmark dataset for the NRG task, and the presented metrics are promising to guide the optimization of NRG models by quantifying the diversity of the generated responses reasonably.
Talking Face Generation by Conditional Recurrent Adversarial Network
Given an arbitrary face image and an arbitrary speech clip, the proposed work attempts to generating the talking face video with accurate lip synchronization while maintaining smooth transition of both lip and facial movement over the entire video clip.
Binge Watching: Scaling Affordance Learning from Sitcoms
In this paper, we tackle the challenge of creating one of the biggest dataset for learning affordances.
3D Human Pose Estimation in the Wild by Adversarial Learning
Instead of defining hard-coded rules to constrain the pose estimation results, we design a novel multi-source discriminator to distinguish the predicted 3D poses from the ground-truth, which helps to enforce the pose estimator to generate anthropometrically valid poses even with images in the wild.
Ranked #1 on
Monocular 3D Human Pose Estimation
on Human3.6M
(Use Video Sequence metric)
Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs
Given a learned knowledge graph (KG), our approach takes as input semantic embeddings for each node (representing visual category).
LCANet: End-to-End Lipreading with Cascaded Attention-CTC
In this paper, we propose LCANet, an end-to-end deep neural network based lipreading system.
Ranked #2 on
Lipreading
on GRID corpus (mixed-speech)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Boundary-sensitive Network for Portrait Segmentation
Compared to the general semantic segmentation problem, portrait segmentation has higher precision requirement on boundary area.
Non-local Neural Networks
Both convolutional and recurrent operations are building blocks that process one local neighborhood at a time.
Ranked #8 on
Action Classification
on Toyota Smarthome dataset
(using extra training data)
Predicting Users' Negative Feedbacks in Multi-Turn Human-Computer Dialogues
Besides, strategies of obtaining distance supervision data for pre-training are also discussed in this work.
Neural Response Generation via GAN with an Approximate Embedding Layer
This paper presents a Generative Adversarial Network (GAN) to model single-turn short-text conversations, which trains a sequence-to-sequence (Seq2Seq) network for response generation simultaneously with a discriminative classifier that measures the differences between human-produced responses and machine-generated ones.
DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices
As observed in the experiment, DeepRebirth achieves more than 3x speed-up and 2. 5x run-time memory saving on GoogLeNet with only 0. 4% drop of top-5 accuracy on ImageNet.
Transitive Invariance for Self-supervised Visual Representation Learning
The objects are connected by two types of edges which correspond to two types of invariance: "different instances but a similar viewpoint and category" and "different viewpoints of the same instance".
Temporal Dynamic Graph LSTM for Action-driven Video Object Detection
A common issue, however, is that objects of interest that are not involved in human actions are often absent in global action descriptions known as "missing label".
Ranked #3 on
Weakly Supervised Object Detection
on Charades
A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detection
We propose to learn an adversarial network that generates examples with occlusions and deformations.
Ranked #19 on
Object Detection
on PASCAL VOC 2007
(using extra training data)
Incorporating Label Dependency for Answer Quality Tagging in Community Question Answering via CNN-LSTM-CRF
In community question answering (cQA), the quality of answers are determined by the matching degree between question-answer pairs and the correlation among the answers.
Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
In our approach, the encoding of sentence is a two-stage process.
Ranked #74 on
Natural Language Inference
on SNLI
Incorporating Loose-Structured Knowledge into Conversation Modeling via Recall-Gate LSTM
Modeling human conversations is the essence for building satisfying chat-bots with multi-turn dialog ability.
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
Each video is annotated by multiple free-text descriptions, action labels, action intervals and classes of interacted objects.
Generative Image Modeling using Style and Structure Adversarial Networks
Current generative frameworks use end-to-end learning and generate images by sampling from uniform noise distribution.
Actions ~ Transformations
In this paper, we propose a novel representation for actions by modeling an action as a transformation which changes the state of the environment before the action happens (precondition) to the state after the action (effect).
Answer Sequence Learning with Neural Networks for Answer Selection in Community Question Answering
In this paper, the answer selection problem in community question answering (CQA) is regarded as an answer sequence labeling task, and a novel approach is proposed based on the recurrent architecture for this problem.
In Defense of the Direct Perception of Affordances
The field of functional recognition or affordance estimation from images has seen a revival in recent years.
Unsupervised Learning of Visual Representations using Videos
Is strong supervision necessary for learning a good visual representation?
Incorporating Structural Alternatives and Sharing into Hierarchy for Multiclass Object Recognition and Detection
This paper proposes a reconfigurable model to recognize and detect multiclass (or multiview) objects with large variation in appearance.
Deep Joint Task Learning for Generic Object Extraction
We present a general joint task learning framework, in which each task (either object localization or object segmentation) is tackled via a multi-layer convolutional neural network, and the two networks work collaboratively to boost performance.
Dynamical And-Or Graph Learning for Object Shape Modeling and Detection
A discriminative learning algorithm, extended from the CCCP [23], is proposed to train the model in a dynamical manner: the model structure (e. g., the configuration of the leaf-nodes associated with the or-nodes) is automatically determined with optimizing the multi-layer parameters during the iteration.
Learning Contour-Fragment-based Shape Model with And-Or Tree Representation
This paper proposes a simple yet effective method to learn the hierarchical object shape model consisting of local contour fragments, which represents a category of shapes in the form of an And-Or tree.
Discriminatively Trained And-Or Graph Models for Object Shape Detection
In this paper, we investigate a novel reconfigurable part-based model, namely And-Or graph model, to recognize object shapes in images.
An Expressive Deep Model for Human Action Parsing from A Single Image
This paper aims at one newly raising task in vision and multimedia research: recognizing human actions from still images.
3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks
Our model thus advances existing approaches in two aspects: (i) it acts directly on the raw inputs (grayscale-depth data) to conduct recognition instead of relying on hand-crafted features, and (ii) the model structure can be dynamically adjusted accounting for the temporal variations of human activities, i. e. the network configuration is allowed to be partially activated during inference.
Designing Deep Networks for Surface Normal Estimation
We show by incorporating several constraints (man-made, manhattan world) and meaningful intermediate representations (room layout, edge labels) in the architecture leads to state of the art performance on surface normal estimation.
Radical-Enhanced Chinese Character Embedding
We present a method to leverage radical for learning Chinese character embedding.
