Search Results for author:
Found 78 papers, 44 papers with code
Model-Agnostic Boundary-Adversarial Sampling for Test-Time Generalization in Few-Shot learning
Few-shot learning is an important research problem that tackles one of the greatest challenges of machine learning: learning a new task from a limited amount of labeled data.
Rethinking Class Activation Mapping for Weakly Supervised Object Localization
Weakly supervised object localization (WSOL) is a task of localizing an object in an image only using image-level labels.
Character Grounding and Re-Identification in Story of Videos and Text Descriptions
We address character grounding and re-identification in multiple story-based videos like movies and associated text descriptions.
Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning
Our COCOA seeks both in-distribution anchors and differences by utilizing the learned reverse dynamics model, encouraging conservatism in the compositional input space for the policy or value function.
When Meta-Learning Meets Online and Continual Learning: A Survey
Over the past decade, deep neural networks have demonstrated significant success using the training scheme that involves mini-batch stochastic gradient descent on extensive datasets.
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Theory of mind (ToM) evaluations currently focus on testing models using passive narratives that inherently lack interactivity.
Can Language Models Laugh at YouTube Short-form Videos?
Our ExFunTube is unique over existing datasets in that our videos cover a wide range of domains with various types of humor that necessitate a multimodal understanding of the content.
Recasting Continual Learning as Sequence Modeling
That is, we propose to formulate continual learning as a sequence modeling problem, allowing advanced sequence models to be utilized for continual learning.
Dense 2D-3D Indoor Prediction with Sound via Aligned Cross-Modal Distillation
Sound can convey significant information for spatial reasoning in our daily lives.
EP2P-Loc: End-to-End 3D Point to 2D Pixel Localization for Large-Scale Visual Localization
Visual localization is the task of estimating a 6-DoF camera pose of a query image within a provided 3D reference map.
Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Models
Generating intermediate steps, or Chain of Thought (CoT), is an effective way to significantly improve language models' (LM) multi-step reasoning capability.
SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaboration
The potential social harms that large language models pose, such as generating offensive content and reinforcing biases, are steeply rising.
KoSBi: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Application
Large language models (LLMs) learn not only natural text generation abilities but also social biases against different demographic groups from real-world data.
MPCHAT: Towards Multimodal Persona-Grounded Conversation
In order to build self-consistent personalized dialogue agents, previous research has mostly focused on textual persona that delivers personal facts or personalities.
Who Wrote this Code? Watermarking for Code Generation
Based on \citet{Kirchenbauer2023watermark}, we propose a new watermarking method, Selective WatErmarking via Entropy Thresholding (SWEET), that promotes "green" tokens only at the position with high entropy of the token distribution during generation, thereby preserving the correctness of the generated code.
Fusing Pre-Trained Language Models With Multimodal Prompts Through Reinforcement Learning
Language models are capable of commonsense reasoning: while domain-specific models can learn from explicit knowledge (e. g. commonsense graphs [6], ethical norms [25]), and larger models like GPT-3 manifest broad commonsense reasoning capacity.
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Data scarcity has been a long standing issue in the field of open-domain social dialogue.
Variational Laplace Autoencoders
Based on the Laplace approximation of the latent variable posterior, VLAEs enhance the expressiveness of the posterior while reducing the amortization error.
Panoramic Vision Transformer for Saliency Detection in 360° Videos
360$^\circ$ video saliency detection is one of the challenging benchmarks for 360$^\circ$ video understanding since non-negligible distortion and discontinuity occur in the projection of any format of 360$^\circ$ videos, and capture-worthy viewpoint in the omnidirectional sphere is ambiguous by nature.
LAVOLUTION: Measurement of Non-target Structural Displacement Calibrated by Structured Light
A jig for the four beams of structured light is designed and a corresponding alignment process is proposed.
ProsocialDialog: A Prosocial Backbone for Conversational Agents
With this dataset, we introduce a dialogue safety detection module, Canary, capable of generating RoTs given conversational context, and a socially-informed dialogue agent, Prost.
Ranked #1 on
Dialogue Safety Prediction
on ProsocialDialog
Multimodal Knowledge Alignment with Reinforcement Learning
Large language models readily adapt to novel settings, even without task-specific training data.
Lipschitz-constrained Unsupervised Skill Discovery
To address this issue, we propose Lipschitz-constrained Skill Discovery (LSD), which encourages the agent to discover more diverse, dynamic, and far-reaching skills.
Unsupervised Representation Learning via Neural Activation Coding
We argue that the deep encoder should maximize its nonlinear expressivity on the data for downstream predictors to take full advantage of its representation power.
Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods
SAR can handle the stochasticity of environments by adaptively reacting to changes in states during action repetition.
Continual Learning on Noisy Data Streams via Self-Purified Replay
Continually learning in the real world must overcome many challenges, among which noisy labels are a common and inevitable issue.
Pano-AVQA: Grounded Audio-Visual Question Answering on 360$^\circ$ Videos
However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings.
Neural Variational Dropout Processes
Learning to infer the conditional posterior model is a key step for robust meta-learning.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Empathy is a complex cognitive ability based on the reasoning of others' affective states.
Unsupervised Skill Discovery with Bottleneck Option Learning
Having the ability to acquire inherent skills from environments without any external rewards or supervision like humans is an important problem.
StyleMix: Separating Content and Style for Enhanced Data Augmentation
In spite of the great success of deep neural networks for many challenging classification tasks, the learned networks are vulnerable to overfitting and adversarial attacks.
Transitional Adaptation of Pretrained Models for Visual Storytelling
In this work, we claim that a transitional adaptation task is required between pretraining and finetuning to harmonize the visual encoder and the language model for challenging downstream target tasks like visual storytelling.
Ranked #1 on
Visual Storytelling
on VIST
(ROUGE-L metric, using extra
training data)
How Robust are Fact Checking Systems on Colloquial Claims?
In this work, we ask: How robust are fact checking systems on claims in colloquial style?
Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration
We propose a novel information bottleneck (IB) method named Drop-Bottleneck, which discretely drops features that are irrelevant to the target variable.
ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning
We demonstrate that our approach finds videos with high audio-visual correspondence and show that self-supervised models trained on our data achieve competitive performances compared to models trained on existing manually curated datasets.
Self-Supervised Learning of Compressed Video Representations
We show that our approach achieves competitive performance on self-supervised learning of video representations with a considerable improvement in speed compared to the traditional methods.
Pano-AVQA: Grounded Audio-Visual Question Answering on 360deg Videos
However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings.
SEDONA: Search for Decoupled Neural Networks toward Greedy Block-wise Learning
Backward locking and update locking are well-known sources of inefficiency in backpropagation that prevent from concurrently updating layers.
Ranked #25 on
Neural Architecture Search
on ImageNet
Characterizing Lookahead Dynamics of Smooth Games
As multi-agent systems proliferate in machine learning research, games have attracted much attention as a framework to understand optimization of multiple interacting objectives.
Viewpoint-Agnostic Change Captioning With Cycle Consistency
In addition, we propose a cycle consistency module that can potentially improve the performance of any change captioning networks in general by matching the composite feature of the generated caption and before image with the after image feature.
Parameter Efficient Multimodal Transformers for Video Representation Learning
The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model.
LID 2020: The Learning from Imperfect Data Challenge Results
The purpose of the Learning from Imperfect Data (LID) workshop is to inspire and facilitate the research in developing novel approaches that would harness the imperfect data and improve the data-efficiency during training.
Imbalanced Continual Learning with Partitioning Reservoir Sampling
Continual learning from a sequential stream of data is a crucial challenge for machine learning research.
Augmenting Data for Sarcasm Detection with Unlabeled Conversation Context
We present a novel data augmentation technique, CRA (Contextual Response Augmentation), which utilizes conversational context to generate meaningful samples for training.
Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness
Results on Dialogue NLI (Welleck et al., 2019) and PersonaChat (Zhang et al., 2018) dataset show that our approach reduces contradiction and improves consistency of existing dialogue models.
CurlingNet: Compositional Learning between Images and Text for Fashion IQ Data
In order to learn an effective image-text composition for the data in the fashion domain, our model proposes two key components as follows.
Ranked #16 on
Image Retrieval
on Fashion IQ
Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue
Knowledge-grounded dialogue is a task of generating an informative response based on both discourse context and external knowledge.
A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
Despite the growing interest in continual learning, most of its contemporary works have been studied in a rather restricted setting where tasks are clearly distinguishable, and task boundaries are known during training.
Self-Routing Capsule Networks
Capsule networks have recently gained a great deal of interest as a new architecture of neural networks that can be more robust to input perturbations than similar-sized CNNs.
AudioCaps: Generating Captions for Audios in The Wild
We explore the problem of Audio Captioning: generating natural language description for any kind of audio in the wild, which has been surprisingly unexplored in previous research.
Ranked #9 on
Audio captioning
on AudioCaps
IB-GAN: Disentangled Representation Learning with Information Bottleneck GAN
IB-GAN objective is similar to that of InfoGAN but has a crucial difference; a capacity regularization for mutual information is adopted, thanks to which the generator of IB-GAN can harness a latent representation in disentangled and interpretable manner.
Harmonizing Maximum Likelihood with GANs for Multimodal Conditional Generation
Recent advances in conditional image generation tasks, such as image-to-image translation and image inpainting, are largely accounted to the success of conditional GAN models, which are often optimized by the joint use of the GAN loss with the reconstruction loss.
Discovery of Natural Language Concepts in Individual Units of CNNs
Although deep convolutional networks have achieved improved performance in many natural language tasks, they have been treated as black boxes because they are difficult to interpret.
Abstractive Summarization of Reddit Posts with Multi-level Memory Networks
We address the problem of abstractive summarization in two directions: proposing a novel dataset and a new model.
A Joint Sequence Fusion Model for Video Question Answering and Retrieval
We present an approach named JSFusion (Joint Sequence Fusion) that can measure semantic similarity between any pairs of multimodal sequence data (e. g. a video clip and a language sentence).
Ranked #34 on
Video Retrieval
on LSMDC
Video Prediction with Appearance and Motion Conditions
Video prediction aims to generate realistic future frames by learning dynamic visual patterns.
A Memory Network Approach for Story-Based Temporal Summarization of 360° Videos
Second, we evaluate the temporal summarization with a newly collected 360° video dataset.
Improving Occlusion and Hard Negative Handling for Single-Stage Pedestrian Detectors
We propose methods of addressing two critical issues of pedestrian detection: (i) occlusion of target objects as false negative failure, and (ii) confusion with hard negative examples like vertical structures as false positive failure.
A Memory Network Approach for Story-based Temporal Summarization of 360° Videos
Second, we evaluate the temporal summarization with a newly collected 360{\deg} video dataset.
A Hierarchical Latent Structure for Variational Conversation Modeling
To solve the degeneration problem, we propose a novel model named Variational Hierarchical Conversation RNNs (VHCR), involving two key ideas of (1) using a hierarchical structure of latent variables, and (2) exploiting an utterance drop regularization.
Memorization Precedes Generation: Learning Unsupervised GANs with Memory Networks
We propose an approach to address two issues that commonly occur during training of unsupervised GANs.
A Deep Ranking Model for Spatio-Temporal Highlight Detection from a 360 Video
We address the problem of highlight detection from a 360 degree video by summarizing it both spatially and temporally.
A Read-Write Memory Network for Movie Story Understanding
We propose a novel memory network model named Read-Write Memory Network (RWMN) to perform question and answering tasks for large-scale, multimodal movie story understanding.
Ranked #3 on
Video Story QA
on MovieQA
SplitNet: Learning to Semantically Split Deep Networks for Parameter Reduction and Model Parallelization
We propose a novel deep neural network that is both lightweight and effectively structured for model parallelization.
Supervising Neural Attention Models for Video Captioning by Human Gaze Data
The attention mechanisms in deep neural networks are inspired by human's attention that sequentially focuses on the most relevant parts of the information over time to generate prediction output.
Encoding Video and Label Priors for Multi-label Video Classification on YouTube-8M dataset
YouTube-8M is the largest video dataset for multi-label video classification.
Attend to You: Personalized Image Captioning with Context Sequence Memory Networks
We address personalization issues of image captioning, which have not been discussed yet in previous research.
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
In this paper, we focus on extending VQA to the video domain and contribute to the literature in three important ways.
Ranked #30 on
Visual Question Answering (VQA)
on MSRVTT-QA
End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering
We propose a high-level concept word detector that can be integrated with any video-to-language models.
Ranked #37 on
Video Retrieval
on LSMDC
Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
To investigate how to adapt existing frameworks to efficiently support distributed GPUs, we propose Poseidon, a scalable system architecture for distributed inter-machine communication in existing DL frameworks.
Storyline Representation of Egocentric Videos With an Applications to Story-Based Search
To address this, we propose a storyline representation that expresses an egocentric video as a set of jointly inferred, through MRF inference, story elements comprising of actors, locations, supporting objects and events, depicted on a timeline.
Expressing an Image Stream with a Sequence of Natural Sentences
We propose an approach for generating a sequence of natural sentences for an image stream.
Ranking and Retrieval of Image Sequences From Multiple Paragraph Queries
While most previous work has dealt with the relations between a natural language sentence and an image or a video, our work extends to the relations between paragraphs and image sequences.
Joint Photo Stream and Blog Post Summarization and Exploration
We alternate between solving the two coupled latent SVM problems, by first fixing the summarization and solving for the alignment from blog images to photo streams and vice versa.
Joint Summarization of Large-scale Collections of Web Images and Videos for Storyline Reconstruction
The reconstruction of storyline graphs is formulated as the inference of sparse time-varying directed graphs from a set of photo streams with assistance of videos.
Reconstructing Storyline Graphs for Image Recommendation from Web Community Photos
In this paper, we investigate an approach for reconstructing storyline graphs from large-scale collections of Internet images, and optionally other side information such as friendship graphs.
Jointly Aligning and Segmenting Multiple Web Photo Streams for the Inference of Collective Photo Storylines
To this end, we design a scalable message-passing based optimization framework to jointly achieve both tasks for the whole input image set at once.
Unsupervised Detection of Regions of Interest Using Iterative Link Analysis
This paper proposes a fast and scalable alternating optimization technique to detect regions of interest (ROIs) in cluttered Web images without labels.
