Search Results for author:
Found 69 papers, 27 papers with code
VGNMN: Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems
Neural module networks (NMN) have achieved success in image-grounded tasks such as Visual Question Answering (VQA) on synthetic images.
ALIAS: DAG Learning with Efficient Unconstrained Policies
Recently, reinforcement learning (RL) has proved a promising alternative for conventional local heuristics in score-based approaches to learning directed acyclic causal graphs (DAGs) from observational data.
Large Language Models Prompting With Episodic Memory
We approach prompt optimization as a Reinforcement Learning (RL) challenge, using episodic memory to archive combinations of input data, permutations of few-shot examples, and the rewards observed during training.
Variable-Agnostic Causal Exploration for Reinforcement Learning
In this paper, we introduce a novel framework, Variable-Agnostic Causal Exploration for Reinforcement Learning (VACERL), incorporating causal relationships to drive exploration in RL without specifying environmental causal variables.
VRDSynth: Synthesizing Programs for Multilingual Visually Rich Document Information Extraction
VRDSynth(Table) outperforms these baselines in 4 out of 8 languages and in average F1 score.
INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulness
In this work, we introduce INDICT: a new framework that empowers LLMs with Internal Dialogues of Critiques for both safety and helpfulness guidance.
SimSMoE: Solving Representational Collapse via Similarity Measure
Sparse mixture of experts (SMoE) have emerged as an effective approach for scaling large language models while keeping a constant computational cost.
Multi-Reference Preference Optimization for Large Language Models
Recent approaches, such as direct preference optimization (DPO), have eliminated the need for unstable and sluggish reinforcement learning optimization by introducing close-formed supervised losses.
Enhancing Length Extrapolation in Sequential Models with Pointer-Augmented Neural Memory
We propose Pointer-Augmented Neural Memory (PANM) to help neural networks understand and apply symbol processing to new, longer sequences of data.
Automatic Prompt Selection for Large Language Models
Our approach consists of three steps: (1) clustering the training data and generating candidate prompts for each cluster using an LLM-based prompt generator; (2) synthesizing a dataset of input-prompt-output tuples for training a prompt evaluator to rank the prompts based on their relevance to the input; (3) using the prompt evaluator to select the best prompt for a new input at test time.
Variational Flow Models: Flowing in Your Style
We propose a systematic training-free method to transform the probability flow of a "linear" stochastic process characterized by the equation X_{t}=a_{t}X_{0}+\sigma_{t}X_{1} into a straight constant-speed (SC) flow, reminiscent of Rectified Flow.
Revisiting the Dataset Bias Problem from a Statistical Perspective
To overcome this challenge, we propose to approximate \frac{1}{p(u|b)} using a biased classifier trained with "bias amplification" losses.
Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions
This work presents Moonshot, a new video generation model that conditions simultaneously on multimodal inputs of image and text.
SurvTimeSurvival: Survival Analysis On The Patient With Multiple Visits/Records
This study introduces "SurvTimeSurvival: Survival Analysis On Patients With Multiple Visits/Records", utilizing the Transformer model to not only handle the complexities of time-varying covariates but also covariates data.
CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules
We find that by naturally encouraging the LLM to reuse the previously developed and verified sub-modules, CodeChain can significantly boost both modularity as well as correctness of the generated solutions, achieving relative pass@1 improvements of 35% on APPS and 76% on CodeContests.
Ranked #3 on
Code Generation
on CodeContests
Universal Graph Continual Learning
We address catastrophic forgetting issues in graph learning as incoming data transits from one to another graph distribution.
LaGR-SEQ: Language-Guided Reinforcement Learning with Sample-Efficient Querying
To address this issue, we introduce SEQ (sample efficient querying), where we simultaneously train a secondary RL agent to decide when the LLM should be queried for solutions.
Beyond Surprise: Improving Exploration Through Surprise Novelty
We present a new computing model for intrinsic rewards in reinforcement learning that addresses the limitations of existing surprise-driven explorations.
CodeTF: One-stop Transformer Library for State-of-the-art Code LLM
In this paper, we present CodeTF, an open-source Transformer-based library for state-of-the-art Code LLMs and code intelligence.
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
To address these limitations, we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks.
Ranked #1 on
Code Search
on CodeXGLUE - AdvTest
When Giant Language Brains Just Aren't Enough! Domain Pizzazz with Knowledge Sparkle Dust
Based on the task we design a new model relied on LLMs which are empowered by additional knowledge extracted from insurance policy rulebooks and DBpedia.
BO-Muse: A human expert and AI teaming framework for accelerated experimental design
In this paper we introduce BO-Muse, a new approach to human-AI teaming for the optimization of expensive black-box functions.
Memory-Augmented Theory of Mind Network
Social reasoning necessitates the capacity of theory of mind (ToM), the ability to contextualise and attribute mental states to others without having access to their internal cognitive structure.
Functional Indirection Neural Estimator for Better Out-of-distribution Generalization
FINE consists of a backbone network and a trainable semantic memory of basis weight matrices.
Improving Document Image Understanding with Reinforcement Finetuning
In this paper, we investigate the problem of improving the performance of Artificial Intelligence systems in understanding document images, especially in cases where training data is limited.
Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation
Since the EMA generator can be considered as an ensemble of the generator's old versions and often undergoes a smaller change in updates compared to the generator, training on its synthetic samples can help the student recall the past knowledge and prevent the student from adapting too quickly to new updates of the generator.
LAVIS: A Library for Language-Vision Intelligence
We introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications.
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
To address the limitations, we propose "CodeRL", a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning (RL).
Ranked #2 on
Code Generation
on APPS
Multimodal Dialogue State Tracking
Specifically, we introduce a novel dialogue state tracking task to track the information of visual objects that are mentioned in video-grounded dialogues.
HYCEDIS: HYbrid Confidence Engine for Deep Document Intelligence System
Measuring the confidence of AI models is critical for safely deploying AI in real-world industrial systems.
OmniXAI: A Library for Explainable AI
We introduce OmniXAI (short for Omni eXplainable AI), an open-source Python library of eXplainable AI (XAI), which offers omni-way explainable AI capabilities and various interpretable machine learning techniques to address the pain points of understanding and interpreting the decisions made by machine learning (ML) in practice.
Learning to Constrain Policy Optimization with Virtual Trust Region
We introduce a constrained optimization method for policy gradient reinforcement learning, which uses a virtual trust region to regulate each policy update.
Learning Theory of Mind via Dynamic Traits Attribution
Inspired by the observation that humans often infer the character traits of others, then use it to explain behaviour, we propose a new neural ToM architecture that learns to generate a latent trait vector of an actor from the past trajectories.
Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback
For summarization, human preference is critical to tame outputs of the summarizer in favor of human interests, as ground-truth summaries are scarce and ambiguous.
Towards Effective and Robust Neural Trojan Defenses via Input Filtering
Trojan attacks on deep neural networks are both dangerous and surreptitious.
Episodic Policy Gradient Training
We introduce a novel training procedure for policy gradient methods wherein episodic memory is used to optimize the hyperparameters of reinforcement learning algorithms on-the-fly.
Robust Deep Reinforcement Learning for Extractive Legal Summarization
Automatic summarization of legal texts is an important and still a challenging task since legal documents are often long and complicated with unusual structures and styles.
Balanced Q-learning: Combining the Influence of Optimistic and Pessimistic Targets
The optimistic nature of the Q-learning target leads to an overestimation bias, which is an inherent problem associated with standard $Q-$learning.
Model-Based Episodic Memory Induces Dynamic Hybrid Controls
Episodic control enables sample efficiency in reinforcement learning by recalling past experiences from an episodic memory.
Generative Pseudo-Inverse Memory
We propose Generative Pseudo-Inverse Memory (GPM), a class of deep generative memory models that are fast to write in and read out.
Reachability Traces for Curriculum Design in Reinforcement Learning
The objective in goal-based reinforcement learning is to learn a policy to reach a particular goal state within the environment.
Neural Latent Traversal with Semantic Constraints
Whilst Generative Adversarial Networks (GANs) generate visually appealing high resolution images, the latent representations (or codes) of these models do not allow controllable changes on the semantic attributes of the generated images.
Plug and Play, Model-Based Reinforcement Learning
This is achieved by representing the global transition dynamics as a union of local transition functions, each with respect to one active object in the scene.
A New Representation of Successor Features for Transfer across Dissimilar Environments
Transfer in reinforcement learning is usually achieved through generalisation across tasks.
Memory and attention in deep learning
Artificial neural networks model neurons and synapses in the brain by interconnecting computational units via weights, which is a typical class of machine learning algorithms that resembles memory structure.
$C^3$: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues
Video-grounded dialogue systems aim to integrate video understanding and dialogue understanding to generate responses that are relevant to both the dialogue and video context.
VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks
Neural module networks (NMN) have achieved success in image-grounded tasks such as Visual Question Answering (VQA) on synthetic images.
Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues
PDC model then learns to predict reasoning paths over this semantic graph.
DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue
A video-grounded dialogue system is required to understand both dialogue, which contains semantic dependencies from turn to turn, and video, which contains visual cues of spatial and temporal scene variations.
VilNMN: A Neural Module Network approach to Video-Grounded Language Tasks
Neural module networks (NMN) have achieved success in image-grounded tasks such as question answering (QA) on synthetic images.
BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues
Video-grounded dialogues are very challenging due to (i) the complexity of videos which contain both spatial and temporal variations, and (ii) the complexity of user utterances which query different segments and/or different objects in videos over multiple dialogue turns.
Neurocoder: Learning General-Purpose Computation Using Stored Neural Programs
For the first time a Neural Program is treated as a datum in memory, paving the ways for modular, recursive and procedural neural programming.
Video-Grounded Dialogues with Pretrained Generation Language Models
Pre-trained language models have shown remarkable success in improving various downstream NLP tasks due to their ability to capture dependencies in textual data and generate natural responses.
UniConv: A Unified Conversational Neural Architecture for Multi-domain Task-oriented Dialogues
Building an end-to-end conversational agent for multi-domain task-oriented dialogues has been an open challenge for two main reasons.
Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challenge
Audio-Visual Scene-Aware Dialog (AVSD) is an extension from Video Question Answering (QA) whereby the dialogue agent is required to generate natural language responses to address user queries and carry on conversations.
Non-Autoregressive Dialog State Tracking
Recent efforts in Dialogue State Tracking (DST) for task-oriented dialogues have progressed toward open-vocabulary or generation-based approaches where the models can generate slot value candidates from the dialogue history itself.
Ranked #13 on
Multi-domain Dialogue State Tracking
on MULTIWOZ 2.0
Self-Attentive Associative Memory
Heretofore, neural networks with external memory are restricted to single memory with lossy representations of memory interactions.
Ranked #1 on
Question Answering
on bAbi
Improving Long Handwritten Text Line Recognition with Convolutional Multi-way Associative Memory
Convolutional Recurrent Neural Networks (CRNNs) excel at scene text recognition.
Optical Character Recognition
Optical Character Recognition (OCR)
+1
FoodAI: Food Image Recognition via Deep Learning for Smart Food Logging
FoodAI has made food logging convenient, aiding smart consumption and a healthy lifestyle.
Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems
Developing Video-Grounded Dialogue Systems (VGDS), where a dialogue is conducted based on visual and audio aspects of a given video, is significantly more challenging than traditional image or text-grounded dialogue systems because (1) feature space of videos span across multiple picture frames, making it difficult to obtain semantic information; and (2) a dialogue agent must perceive and process information from different modalities (audio, video, caption, etc.)
Ranked #4 on
Response Generation
on SIMMC2.0
Neural Stored-program Memory
Neural networks powered with external memory simulate computer behaviors.
Ranked #5 on
Question Answering
on bAbi
(Mean Error Rate metric)
Meta-Learning with Domain Adaptation for Few-Shot Learning under Domain Shift
Most existing work assumes that both training and test tasks are drawn from the same distribution, and a large amount of labeled data is available in the training tasks.
Learning to Remember More with Less Memorization
Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning.
Ranked #5 on
Text Classification
on Yahoo! Answers
Variational Memory Encoder-Decoder
Introducing variability while maintaining coherence is a core task in learning to generate utterances in conversation.
Dual Control Memory Augmented Neural Networks for Treatment Recommendations
The decoding controller generates a treatment sequence, one treatment option at a time.
URLNet: Learning a URL Representation with Deep Learning for Malicious URL Detection
This approach allows the model to capture several types of semantic information, which was not possible by the existing models.
DeepProcess: Supporting business process execution using a MANN-based recommender system
Process-aware Recommender systems can provide critical decision support functionality to aid business process execution by recommending what actions to take next.
Dual Memory Neural Computer for Asynchronous Two-view Sequential Learning
One of the core tasks in multi-view learning is to capture relations among views.
What are the Receptive, Effective Receptive, and Projective Fields of Neurons in Convolutional Neural Networks?
In this work, we explain in detail how receptive fields, effective receptive fields, and projective fields of neurons in different layers, convolution or pooling, of a Convolutional Neural Network (CNN) are calculated.
