Search Results for author:
Found 61 papers, 25 papers with code
LAVIS: A Library for Language-Vision Intelligence
We introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications.
Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions
This work presents Moonshot, a new video generation model that conditions simultaneously on multimodal inputs of image and text.
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
To address the limitations, we propose "CodeRL", a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning (RL).
Ranked #1 on
Code Generation
on APPS
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
To address these limitations, we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks.
Ranked #1 on
Code Search
on CodeXGLUE - AdvTest
CodeTF: One-stop Transformer Library for State-of-the-art Code LLM
In this paper, we present CodeTF, an open-source Transformer-based library for state-of-the-art Code LLMs and code intelligence.
OmniXAI: A Library for Explainable AI
We introduce OmniXAI (short for Omni eXplainable AI), an open-source Python library of eXplainable AI (XAI), which offers omni-way explainable AI capabilities and various interpretable machine learning techniques to address the pain points of understanding and interpreting the decisions made by machine learning (ML) in practice.
URLNet: Learning a URL Representation with Deep Learning for Malicious URL Detection
This approach allows the model to capture several types of semantic information, which was not possible by the existing models.
Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems
Developing Video-Grounded Dialogue Systems (VGDS), where a dialogue is conducted based on visual and audio aspects of a given video, is significantly more challenging than traditional image or text-grounded dialogue systems because (1) feature space of videos span across multiple picture frames, making it difficult to obtain semantic information; and (2) a dialogue agent must perceive and process information from different modalities (audio, video, caption, etc.)
Ranked #4 on
Response Generation
on SIMMC2.0
Self-Attentive Associative Memory
Heretofore, neural networks with external memory are restricted to single memory with lossy representations of memory interactions.
Ranked #1 on
Question Answering
on bAbi
Non-Autoregressive Dialog State Tracking
Recent efforts in Dialogue State Tracking (DST) for task-oriented dialogues have progressed toward open-vocabulary or generation-based approaches where the models can generate slot value candidates from the dialogue history itself.
Ranked #13 on
Multi-domain Dialogue State Tracking
on MULTIWOZ 2.0
Variational Memory Encoder-Decoder
Introducing variability while maintaining coherence is a core task in learning to generate utterances in conversation.
CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules
We find that by naturally encouraging the LLM to reuse the previously developed and verified sub-modules, CodeChain can significantly boost both modularity as well as correctness of the generated solutions, achieving relative pass@1 improvements of 35% on APPS and 76% on CodeContests.
Ranked #2 on
Code Generation
on CodeContests
(Test Set pass@1 metric)
Learning to Remember More with Less Memorization
Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning.
Ranked #5 on
Text Classification
on Yahoo! Answers
Memory and attention in deep learning
Artificial neural networks model neurons and synapses in the brain by interconnecting computational units via weights, which is a typical class of machine learning algorithms that resembles memory structure.
Neural Stored-program Memory
Neural networks powered with external memory simulate computer behaviors.
Ranked #5 on
Question Answering
on bAbi
(Mean Error Rate metric)
Dual Memory Neural Computer for Asynchronous Two-view Sequential Learning
One of the core tasks in multi-view learning is to capture relations among views.
DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue
A video-grounded dialogue system is required to understand both dialogue, which contains semantic dependencies from turn to turn, and video, which contains visual cues of spatial and temporal scene variations.
BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues
Video-grounded dialogues are very challenging due to (i) the complexity of videos which contain both spatial and temporal variations, and (ii) the complexity of user utterances which query different segments and/or different objects in videos over multiple dialogue turns.
Multimodal Dialogue State Tracking
Specifically, we introduce a novel dialogue state tracking task to track the information of visual objects that are mentioned in video-grounded dialogues.
UniConv: A Unified Conversational Neural Architecture for Multi-domain Task-oriented Dialogues
Building an end-to-end conversational agent for multi-domain task-oriented dialogues has been an open challenge for two main reasons.
Episodic Policy Gradient Training
We introduce a novel training procedure for policy gradient methods wherein episodic memory is used to optimize the hyperparameters of reinforcement learning algorithms on-the-fly.
DeepProcess: Supporting business process execution using a MANN-based recommender system
Process-aware Recommender systems can provide critical decision support functionality to aid business process execution by recommending what actions to take next.
Beyond Surprise: Improving Exploration Through Surprise Novelty
We present a new computing model for intrinsic rewards in reinforcement learning that addresses the limitations of existing surprise-driven explorations.
LaGR-SEQ: Language-Guided Reinforcement Learning with Sample-Efficient Querying
To address this issue, we introduce SEQ (sample efficient querying), where we simultaneously train a secondary RL agent to decide when the LLM should be queried for solutions.
SurvTimeSurvival: Survival Analysis On The Patient With Multiple Visits/Records
This study introduces "SurvTimeSurvival: Survival Analysis On Patients With Multiple Visits/Records", utilizing the Transformer model to not only handle the complexities of time-varying covariates but also covariates data.
What are the Receptive, Effective Receptive, and Projective Fields of Neurons in Convolutional Neural Networks?
In this work, we explain in detail how receptive fields, effective receptive fields, and projective fields of neurons in different layers, convolution or pooling, of a Convolutional Neural Network (CNN) are calculated.
Dual Control Memory Augmented Neural Networks for Treatment Recommendations
The decoding controller generates a treatment sequence, one treatment option at a time.
Meta-Learning with Domain Adaptation for Few-Shot Learning under Domain Shift
Most existing work assumes that both training and test tasks are drawn from the same distribution, and a large amount of labeled data is available in the training tasks.
FoodAI: Food Image Recognition via Deep Learning for Smart Food Logging
FoodAI has made food logging convenient, aiding smart consumption and a healthy lifestyle.
Improving Long Handwritten Text Line Recognition with Convolutional Multi-way Associative Memory
Convolutional Recurrent Neural Networks (CRNNs) excel at scene text recognition.
Optical Character Recognition
Optical Character Recognition (OCR)
+1
Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challenge
Audio-Visual Scene-Aware Dialog (AVSD) is an extension from Video Question Answering (QA) whereby the dialogue agent is required to generate natural language responses to address user queries and carry on conversations.
Video-Grounded Dialogues with Pretrained Generation Language Models
Pre-trained language models have shown remarkable success in improving various downstream NLP tasks due to their ability to capture dependencies in textual data and generate natural responses.
Neurocoder: Learning General-Purpose Computation Using Stored Neural Programs
For the first time a Neural Program is treated as a datum in memory, paving the ways for modular, recursive and procedural neural programming.
VilNMN: A Neural Module Network approach to Video-Grounded Language Tasks
Neural module networks (NMN) have achieved success in image-grounded tasks such as question answering (QA) on synthetic images.
Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues
PDC model then learns to predict reasoning paths over this semantic graph.
VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks
Neural module networks (NMN) have achieved success in image-grounded tasks such as Visual Question Answering (VQA) on synthetic images.
$C^3$: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues
Video-grounded dialogue systems aim to integrate video understanding and dialogue understanding to generate responses that are relevant to both the dialogue and video context.
A New Representation of Successor Features for Transfer across Dissimilar Environments
Transfer in reinforcement learning is usually achieved through generalisation across tasks.
Plug and Play, Model-Based Reinforcement Learning
This is achieved by representing the global transition dynamics as a union of local transition functions, each with respect to one active object in the scene.
Reachability Traces for Curriculum Design in Reinforcement Learning
The objective in goal-based reinforcement learning is to learn a policy to reach a particular goal state within the environment.
Generative Pseudo-Inverse Memory
We propose Generative Pseudo-Inverse Memory (GPM), a class of deep generative memory models that are fast to write in and read out.
Neural Latent Traversal with Semantic Constraints
Whilst Generative Adversarial Networks (GANs) generate visually appealing high resolution images, the latent representations (or codes) of these models do not allow controllable changes on the semantic attributes of the generated images.
Model-Based Episodic Memory Induces Dynamic Hybrid Controls
Episodic control enables sample efficiency in reinforcement learning by recalling past experiences from an episodic memory.
Balanced Q-learning: Combining the Influence of Optimistic and Pessimistic Targets
The optimistic nature of the Q-learning target leads to an overestimation bias, which is an inherent problem associated with standard $Q-$learning.
Robust Deep Reinforcement Learning for Extractive Legal Summarization
Automatic summarization of legal texts is an important and still a challenging task since legal documents are often long and complicated with unusual structures and styles.
Towards Effective and Robust Neural Trojan Defenses via Input Filtering
Trojan attacks on deep neural networks are both dangerous and surreptitious.
Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback
For summarization, human preference is critical to tame outputs of the summarizer in favor of human interests, as ground-truth summaries are scarce and ambiguous.
Learning Theory of Mind via Dynamic Traits Attribution
Inspired by the observation that humans often infer the character traits of others, then use it to explain behaviour, we propose a new neural ToM architecture that learns to generate a latent trait vector of an actor from the past trajectories.
Learning to Constrain Policy Optimization with Virtual Trust Region
We introduce a constrained optimization method for policy gradient reinforcement learning, which uses a virtual trust region to regulate each policy update.
HYCEDIS: HYbrid Confidence Engine for Deep Document Intelligence System
Measuring the confidence of AI models is critical for safely deploying AI in real-world industrial systems.
VGNMN: Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems
Neural module networks (NMN) have achieved success in image-grounded tasks such as Visual Question Answering (VQA) on synthetic images.
Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation
Since the EMA generator can be considered as an ensemble of the generator's old versions and often undergoes a smaller change in updates compared to the generator, training on its synthetic samples can help the student recall the past knowledge and prevent the student from adapting too quickly to new updates of the generator.
Improving Document Image Understanding with Reinforcement Finetuning
In this paper, we investigate the problem of improving the performance of Artificial Intelligence systems in understanding document images, especially in cases where training data is limited.
Functional Indirection Neural Estimator for Better Out-of-distribution Generalization
FINE consists of a backbone network and a trainable semantic memory of basis weight matrices.
Memory-Augmented Theory of Mind Network
Social reasoning necessitates the capacity of theory of mind (ToM), the ability to contextualise and attribute mental states to others without having access to their internal cognitive structure.
BO-Muse: A human expert and AI teaming framework for accelerated experimental design
In this paper we introduce BO-Muse, a new approach to human-AI teaming for the optimization of expensive black-box functions.
When Giant Language Brains Just Aren't Enough! Domain Pizzazz with Knowledge Sparkle Dust
Based on the task we design a new model relied on LLMs which are empowered by additional knowledge extracted from insurance policy rulebooks and DBpedia.
Universal Graph Continual Learning
We address catastrophic forgetting issues in graph learning as incoming data transits from one to another graph distribution.
Variational Flow Models: Flowing in Your Style
We introduce "posterior flows" - generalizations of "probability flows" to a broader class of stochastic processes not necessarily diffusion processes - and propose a systematic training-free method to transform the posterior flow of a "linear" stochastic process characterized by the equation Xt = at * X0 + st * X1 into a straight constant-speed (SC) flow, reminiscent of Rectified Flow.
Revisiting the Dataset Bias Problem from a Statistical Perspective
To overcome this challenge, we propose to approximate \frac{1}{p(u|b)} using a biased classifier trained with "bias amplification" losses.
Automatic Prompt Selection for Large Language Models
Our approach consists of three steps: (1) clustering the training data and generating candidate prompts for each cluster using an LLM-based prompt generator; (2) synthesizing a dataset of input-prompt-output tuples for training a prompt evaluator to rank the prompts based on their relevance to the input; (3) using the prompt evaluator to select the best prompt for a new input at test time.
