Search Results for author:
Found 78 papers, 24 papers with code
Scene Graph Parsing via Abstract Meaning Representation in Pre-trained Language Models
In this work, we propose the application of abstract meaning representation (AMR) based semantic parsing models to parse textual descriptions of a visual scene into scene graphs, which is the first work to the best of our knowledge.
Language-agnostic Semantic Consistent Text-to-Image Generation
Recent GAN-based text-to-image generation models have advanced that they can generate photo-realistic images matching semantically with descriptions.
Generative Adversarial Network
Multi-lingual Text-to-Image Generation
+2
Devil’s Advocate: Novel Boosting Ensemble Method from Psychological Findings for Text Classification
We present a new form of ensemble method–Devil’s Advocate, which uses a deliberately dissenting model to force other submodels within the ensemble to better collaborate.
Continual Vision-and-Language Navigation
For the training and evaluation of CVLN agents, we re-arrange existing VLN datasets to propose two datasets: CVLN-I, focused on navigation via initial-instruction interpretation, and CVLN-D, aimed at navigation through dialogue with other agents.
Unveiling the Significance of Toddler-Inspired Reward Transition in Goal-Oriented Reinforcement Learning
Toddlers evolve from free exploration with sparse feedback to exploiting prior experiences for goal-directed learning with denser rewards.
Multimodal Anomaly Detection based on Deep Auto-Encoder for Object Slip Perception of Mobile Manipulation Robots
The experimental results verified that the proposed framework reliably detects anomalies in object slip situations despite various object types and robot behaviors, and visual and auditory noise in the environment.
DUEL: Duplicate Elimination on Active Memory for Self-Supervised Class-Imbalanced Learning
Recent machine learning algorithms have been developed using well-curated datasets, which often require substantial cost and resources.
Visual Hindsight Self-Imitation Learning for Interactive Navigation
Interactive visual navigation tasks, which involve following instructions to reach and interact with specific targets, are challenging not only because successful experiences are very rare but also because the complex visual inputs require a substantial number of samples.
Neural Collage Transfer: Artistic Reconstruction via Material Manipulation
Collage is a creative art form that uses diverse material scraps as a base unit to compose a single image.
PGA: Personalizing Grasping Agents with Single Human-Robot Interaction
Based on the acquired information, PGA pseudo-labels objects in the Reminiscence by our proposed label propagation algorithm.
PROGrasp: Pragmatic Human-Robot Communication for Object Grasping
The robot should then identify the target object by interacting with a human user.
GVCCI: Lifelong Learning of Visual Grounding for Language-Guided Robotic Manipulation
Furthermore, the qualitative analysis shows that the unadapted VG model often fails to find correct objects due to a strong bias learned from the pre-training data.
EXOT: Exit-aware Object Tracker for Safe Robotic Manipulation of Moving Object
We evaluate our method on the first-person video benchmark dataset, TREK-150, and on the custom dataset, RMOT-223, that we collect from the UR5e robot.
Overcoming Weak Visual-Textual Alignment for Video Moment Retrieval
Video moment retrieval (VMR) identifies a specific moment in an untrimmed video for a given natural language query.
Ranked #6 on
Moment Retrieval
on Charades-STA
L-SA: Learning Under-Explored Targets in Multi-Target Reinforcement Learning
Tasks that involve interaction with various targets are called multi-target tasks.
Learning Geometry-aware Representations by Sketching
Our method, coined Learning by Sketching (LBS), learns to convert an image into a set of colored strokes that explicitly incorporate the geometric information of the scene in a single inference step without requiring a sketch dataset.
SelecMix: Debiased Learning by Contradicting-pair Sampling
Experiments on standard benchmarks demonstrate the effectiveness of the method, in particular when label noise complicates the identification of bias-conflicting examples.
DUEL: Adaptive Duplicate Elimination on Working Memory for Self-Supervised Learning
In Self-Supervised Learning (SSL), it is known that frequent occurrences of the collision in which target data and its negative samples share the same class can decrease performance.
Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval
Video corpus moment retrieval (VCMR) is the task to retrieve the most relevant video moment from a large video corpus using a natural language query.
Ranked #2 on
Video Corpus Moment Retrieval
on TVR
Robust Imitation via Mirror Descent Inverse Reinforcement Learning
Recently, adversarial imitation learning has shown a scalable reward acquisition method for inverse reinforcement learning (IRL) problems.
SGRAM: Improving Scene Graph Parsing via Abstract Meaning Representation
To this end, we design a simple yet effective two-stage scene graph parsing framework utilizing abstract meaning representation, SGRAM (Scene GRaph parsing via Abstract Meaning representation): 1) transforming a textual description of an image into an AMR graph (Text-to-AMR) and 2) encoding the AMR graph into a Transformer-based language model to generate a scene graph (AMR-to-SG).
Learning to Write with Coherence From Negative Examples
We compare our approach with Unlikelihood (UL) training in a text continuation task on commonsense natural language inference (NLI) corpora to show which method better models the coherence by avoiding unlikely continuations.
On the Importance of Critical Period in Multi-stage Reinforcement Learning
The initial years of an infant's life are known as the critical period, during which the overall development of learning performance is significantly impacted due to neural plasticity.
From Scratch to Sketch: Deep Decoupled Hierarchical Reinforcement Learning for Robotic Sketching Agent
We present an automated learning framework for a robotic sketching agent that is capable of learning stroke-based rendering and motor control simultaneously.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Cross-Modal Alignment Learning of Vision-Language Conceptual Systems
Additionally, we also propose an aligned cross-modal representation learning method that learns semantic representations of visual objects and words in a self-supervised manner based on the cross-modal relational graph networks.
The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training
As a result, GST scales the amount of training data up to an order of magnitude that of VisDial (1. 2M to 12. 9M QA data).
Conditional Text Generation
Out-of-Distribution Detection
+1
Hypergraph Transformer: Weakly-supervised Multi-hop Reasoning for Knowledge-based Visual Question Answering
Knowledge-based visual question answering (QA) aims to answer a question which requires visually-grounded external knowledge beyond image content itself.
Toddler-Guidance Learning: Impacts of Critical Period on Multimodal AI Agents
To validate this hypothesis, we adapt this notion of critical periods to learning in AI agents and investigate the critical period in the virtual environment for AI agents.
Smooth-Swap: A Simple Enhancement for Face-Swapping with Smoothness
Face-swapping models have been drawing attention for their compelling generation quality, but their complex architectures and loss functions often require careful tuning for successful training.
Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning
Learning in a multi-target environment without prior knowledge about the targets requires a large amount of samples and makes generalization difficult.
Toward a Human-Level Video Understanding Intelligence
In this paper, we propose the Video Turing Test to provide effective and practical assessments of video understanding intelligence as well as human-likeness evaluation of AI agents.
Mounting Video Metadata on Transformer-based Language Model for Open-ended Video Question Answering
In this paper, we challenge the existing multiple-choice video question answering by changing it to open-ended video question answering.
CogME: A Novel Evaluation Metric for Video Understanding Intelligence
Then we propose a top-down evaluation system for VideoQA, based on the cognitive process of humans and story elements: Cognitive Modules for Evaluation (CogME).
Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering
MASN consists of a motion module, an appearance module, and a motion-appearance fusion module.
M2FN: Multi-step Modality Fusion for Advertisement Image Assessment
Assessing advertisements, specifically on the basis of user preferences and ad quality, is crucial to the marketing industry.
Learning task-agnostic representation via toddler-inspired learning
One of the inherent limitations of current AI systems, stemming from the passive learning mechanisms (e. g., supervised learning), is that they perform well on labeled datasets but cannot deduce knowledge on their own.
Unbiased learning with State-Conditioned Rewards in Adversarial Imitation Learning
The formulation draws a strong connection between adversarial learning and energy-based reinforcement learning; thus, the architecture is capable of recovering a reward function that induces a multi-modal policy.
Ruminating Word Representations with Random Noise Masking
Through the re-training process, some of noises can be compensated and other noises can be utilized to learn better representations.
Deep Quotient Manifold Modeling
One of the difficulties in modeling real-world data is their complex multi-manifold structure due to discrete features.
ColdExpand: Semi-Supervised Graph Learning in Cold Start
Most real-world graphs are dynamic and eventually face the cold start problem.
Spectrally Similar Graph Pooling
To further investigate the effectiveness of our proposed method, we evaluate our approach on a real-world problem, image retrieval with visual scene graphs.
Message Passing Adaptive Resonance Theory for Online Active Semi-supervised Learning
Active learning is widely used to reduce labeling effort and training time by repeatedly querying only the most beneficial samples from unlabeled data.
Human-Like Active Learning: Machines Simulating the Human Learning Process
In the end, we showed not only that we can make build better machine training framework through the human experiment result, but also empirically confirm the result of human experiment through imitated machine experiments; human-like active learning have crucial effect on learning performance.
Co-attentional Transformers for Story-Based Video Understanding
Inspired by recent trends in vision and language learning, we explore applications of attention mechanisms for visio-lingual fusion within an application to story-based video understanding.
Toward General Scene Graph: Integration of Visual Semantic Knowledge with Entity Synset Alignment
In experiment, the integrated scene graph is applied to the image-caption retrieval task as a down-stream task.
Pattern Denoising in Molecular Associative Memory using Pairwise Markov Random Field Models
We propose an in silico molecular associative memory model for pattern learning, storage and denoising using Pairwise Markov Random Field (PMRF) model.
DramaQA: Character-Centered Video Story Understanding with Hierarchical QA
Despite recent progress on computer vision and natural language processing, developing a machine that can understand video story is still hard to achieve due to the intrinsic difficulty of video story.
Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer
Visual dialog is a task of answering a sequence of questions grounded in an image using the previous dialog history as context.
Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data
Here, we propose Cut-Based Graph Learning Networks (CB-GLNs) for learning video data by discovering these complex structures of the video.
Ruminating Word Representations with Random Noised Masker
Next, we gradually add random noises to the word representations and repeat the training process from scratch, but initialize with the noised word representations.
Which Ads to Show? Advertisement Image Assessment with Auxiliary Information via Multi-step Modality Fusion
Despite crucial influences of image quality, auxiliary information of ad images such as tags and target subjects can also determine image preference.
Manifold Learning and Alignment with Generative Adversarial Networks
We present a generative adversarial network (GAN) that conducts manifold learning and alignment (MLA): A task to learn the multi-manifold structure underlying data and to align those manifolds without any correspondence information.
Discriminative Variational Autoencoder for Continual Learning with Generative Replay
Generative replay (GR) is a method to alleviate catastrophic forgetting in continual learning (CL) by generating previous task data and learning them together with the data from new tasks.
Compositional Structure Learning for Sequential Video Data
However, most of sequential data, as seen with videos, have complex temporal dependencies that imply variable-length semantic flows and their compositions, and those are hard to be captured by conventional methods.
Encoder-Powered Generative Adversarial Networks
We present an encoder-powered generative adversarial network (EncGAN) that is able to learn both the multi-manifold structure and the abstract features of data.
Simulating Problem Difficulty in Arithmetic Cognition Through Dynamic Connectionist Models
Problem difficulty was operationalized by the number of carries involved in solving a given problem.
Constructing Hierarchical Q&A Datasets for Video Story Understanding
Video understanding is emerging as a new paradigm for studying human-like AI.
Dual Attention Networks for Visual Reference Resolution in Visual Dialog
Specifically, REFER module learns latent relationships between a given question and a dialog history by employing a self-attention mechanism.
Ranked #2 on
Visual Dialog
on VisDial v0.9 val
Visualizing Semantic Structures of Sequential Data by Learning Temporal Dependencies
While conventional methods for sequential learning focus on interaction between consecutive inputs, we suggest a new method which captures composite semantic flows with variable-length dependencies.
Data Interpolations in Deep Generative Models under Non-Simply-Connected Manifold Topology
Exploiting the deep generative model's remarkable ability of learning the data-manifold structure, some recent researches proposed a geometric data interpolation method based on the geodesic curves on the learned data-manifold.
Multimodal Dual Attention Memory for Video Story Question Answering
We confirm the best performance of the dual attention mechanism combined with late fusion by ablation studies.
GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation
The task of multi-image cued story generation, such as visual storytelling dataset (VIST) challenge, is to compose multiple coherent sentences from a given sequence of images.
Ranked #30 on
Visual Storytelling
on VIST
(METEOR metric)
Bilinear Attention Networks
In this paper, we propose bilinear attention networks (BAN) that find bilinear attention distributions to utilize given vision-language information seamlessly.
Ranked #11 on
Phrase Grounding
on Flickr30k Entities Test
Answerer in Questioner's Mind: Information Theoretic Approach to Goal-Oriented Visual Dialog
Goal-oriented dialogue tasks occur when a questioner asks an action-oriented question and an answerer responds with the intent of letting the questioner know a correct action to take.
Understanding Local Minima in Neural Networks by Loss Surface Decomposition
The parameter domain of the loss surface can be decomposed into regions in which activation values (zero or one for rectified linear units) are consistent.
Visual Explanations from Hadamard Product in Multimodal Deep Networks
Kim et al. (2016) show that the Hadamard product in multimodal deep networks, which is well-known for the joint function of visual question answering tasks, implicitly performs an attentional mechanism for visual inputs.
CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
The game involves two players: a Teller and a Drawer.
Multi-focus Attention Network for Efficient Deep Reinforcement Learning
In multi-agent cooperative task experiments, our model shows 20% faster learning than existing state-of-the-art model.
DeepStory: Video Story QA by Deep Embedded Memory Networks
This is mainly due to 1) the reconstruction of video stories in a scene-dialogue combined form that utilize the latent embedding and 2) attention.
Overcoming Catastrophic Forgetting by Incremental Moment Matching
Catastrophic forgetting is a problem of neural networks that loses the information of the first task after training the second task.
Micro-Objective Learning : Accelerating Deep Reinforcement Learning through the Discovery of Continuous Subgoals
To solve this issue, the subgoal and option framework have been proposed.
Ways of Conditioning Generative Adversarial Networks
The GANs are generative models whose random samples realistically reflect natural images.
Hadamard Product for Low-rank Bilinear Pooling
Bilinear models provide rich representations compared with linear models.
Human Body Orientation Estimation using Convolutional Neural Network
However, in most of the service robot applications, the user needs to move himself/herself to allow the robot to see him/her face to face.
Generating Images Part by Part with Composite Generative Adversarial Networks
We propose a model called composite generative adversarial network, that reveals the complex structure of images with multiple generators in which each generator generates some part of the image.
Multimodal Residual Learning for Visual QA
We present Multimodal Residual Networks (MRN) for the multimodal residual learning of visual question-answering, which extends the idea of the deep residual learning.
Dual Memory Architectures for Fast Deep Learning of Stream Data via an Online-Incremental-Transfer Strategy
The proposed architecture consists of deep representation learners and fast learnable shallow kernel networks, both of which synergize to track the information of new data.
Generative Local Metric Learning for Nearest Neighbor Classification
We consider the problem of learning a local metric to enhance the performance of nearest neighbor classification.
