Search Results for author:
Found 50 papers, 18 papers with code
GO4Align: Group Optimization for Multi-Task Alignment
This paper proposes \textit{GO4Align}, a multi-task optimization approach that tackles task imbalance by explicitly aligning the optimization across tasks.
Conditional Modeling Based Automatic Video Summarization
The aim of video summarization is to shorten videos automatically while retaining the key information necessary to convey the overall story.
Episodic Multi-Task Learning with Heterogeneous Neural Processes
This paper focuses on the data-insufficiency problem in multi-task learning within an episodic training setup.
Self-Supervised Open-Ended Classification with Small Visual Language Models
We present Self-Context Adaptation (SeCAt), a self-supervised approach that unlocks few-shot abilities for open-ended classification with small visual language models.
Prototype-Enhanced Hypergraph Learning for Heterogeneous Information Networks
The variety and complexity of relations in multimedia data lead to Heterogeneous Information Networks (HINs).
ProtoExplorer: Interpretable Forensic Analysis of Deepfake Videos using Prototype Exploration and Refinement
This paper proposes a Visual Analytics process model for prototype learning, and, based on this, presents ProtoExplorer, a Visual Analytics system for the exploration and refinement of prototype-based deepfake detection models.
Knowledge Graph Embeddings for Multi-Lingual Structured Representations of Radiology Reports
We show the use of this embedding on two tasks namely disease classification of X-ray reports and image classification.
Multimodal Temporal Fusion Transformers Are Good Product Demand Forecasters
Traditional approaches to demand forecasting rely on historical demand, product categories, and additional contextual information such as seasonality and events.
Query-based Video Summarization with Pseudo Label Supervision
Existing datasets for manually labelled query-based video summarization are costly and thus small, limiting the performance of supervised deep video summarization models.
Causal Video Summarizer for Video Exploration
Multi-modal video summarization has a video input and a text-based query input.
Causalainer: Causal Explainer for Automatic Video Summarization
In this work, a Causal Explainer, dubbed Causalainer, is proposed to address this issue.
Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questions
This work proposes a new method that utilizes semantically related questions, referred to as basic questions, acting as noise to evaluate the robustness of VQA models.
Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models
Most existing methods approach it as a multi-class classification problem, which restricts the outcome to a predefined closed-set of curated answers.
Ranked #1 on
Medical Visual Question Answering
on OVQA
Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learning
Existing methods are trying to communicate visual concepts as prompts to frozen language models, but rely on hand-engineered task induction to reduce the hypothesis space.
X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentation
In this paper we mimic this ability by using multi-modal retrieval augmentation and apply it to several tasks in chest X-ray analysis.
An Analytics of Culture: Modeling Subjectivity, Scalability, Contextuality, and Temporality
We believe that possible implementations of these aspects into AI research leads to AI that better captures the complexities of culture.
Probabilistic Integration of Object Level Annotations in Chest X-ray Classification
This shows how two-stage learning of labels from coarse to fine-grained, in particular with object level annotations, is an effective method for more optimal annotation usage.
Association Graph Learning for Multi-Task Classification with Category Shifts
To generalize to such test data, it is crucial for individual tasks to leverage knowledge from related tasks.
PanorAMS: Automatic Annotation for Detecting Objects in Urban Context
With the PanorAMS framework we introduce a method to automatically generate bounding box annotations for panoramic images based on urban context information.
LifeLonger: A Benchmark for Continual Disease Classification
Task and class incremental learning of diseases address the issue of classifying new samples without re-training the models from scratch, while cross-domain incremental learning addresses the issue of dealing with datasets originating from different institutions while retaining the previously obtained knowledge.
Inside Out Visual Place Recognition
Additionally, we introduce a new training protocol Inside Out Data Augmentation to adapt Visual Place Recognition methods for localizing indoor images, demonstrating the potential of Inside Out Visual Place Recognition.
Multi-Task Neural Processes
Our multi-task neural processes methodologically expand the scope of vanilla neural processes and provide a new way of exploring task relatedness in function spaces for multi-task learning.
Variational Multi-Task Learning with Gumbel-Softmax Priors
Multi-task learning aims to explore task relatedness to improve individual tasks, which is of particular significance in the challenging scenario that only limited data is available for each task.
The Dawn of Quantum Natural Language Processing
In this paper, we discuss the initial attempts at boosting understanding human language based on deep-learning models with quantum computing.
Adaptive Neural Message Passing for Inductive Learning on Hypergraphs
Most hypergraph learning approaches convert the hypergraph structure to that of a graph and then deploy existing geometric deep learning methods.
Variational Topic Inference for Chest X-Ray Report Generation
The topics are inferred in a conditional variational inference framework, with each topic governing the generation of a sentence in the report.
Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"
Automatically generating medical reports for retinal images is one of the promising ways to help ophthalmologists reduce their workload and improve work efficiency.
Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintings
We propose ArtSAGENet, a novel multimodal architecture that integrates Graph Neural Networks (GNNs) and Convolutional Neural Networks (CNNs), to jointly learn visual and semantic-based artistic representations.
GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarization
Traditional video summarization methods generate fixed video representations regardless of user interest.
Contextualized Keyword Representations for Multi-modal Retinal Image Captioning
A traditional medical image captioning model creates a medical description only based on a single medical image input.
NLQuAD: A Non-Factoid Long Question Answering Data Set
We introduce NLQuAD, the first data set with baseline methods for non-factoid long question answering, a task requiring document-level language understanding.
Variational Knowledge Distillation for Disease Classification in Chest X-Rays
It is challenging to incorporate this information into disease classification due to the high reliance on clinician input in EHRs, limiting the possibility for automated diagnosis.
Variational Multi-Task Learning
Multi-task learning aims to improve the overall performance of a set of tasks by leveraging their relatedness.
DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation
To train and validate the effectiveness of our DNN-based module, we propose a large-scale retinal disease image dataset.
HyperSAGE: Generalizing Inductive Representation Learning on Hypergraphs
To model such complex relations, hypergraphs have proven to be a natural representation.
Detecting CNN-Generated Facial Images in Real-World Scenarios
Artificial, CNN-generated images are now of such high quality that humans have trouble distinguishing them from real images.
II-20: Intelligent and pragmatic analytic categorization of image collections
Analytic categorization, however, is not machine classification (the difference between the two is called the pragmatic gap): a human adds/redefines/deletes categories of relevance on the fly to build insight, whereas the machine classifier is rigid and non-adaptive.
Query-controllable Video Summarization
In this work, we introduce a method which takes a text-based query as input and generates a video summary corresponding to it.
Fusing Structural and Functional MRIs using Graph Convolutional Networks for Autism Classification
We alleviate the use of such non-imaging metadata and propose a fully imaging-based approach where information from structural and functional Magnetic Resonance Imaging (MRI) data are fused to construct the edges and nodes of the graph.
Assessing the Robustness of Visual Question Answering Models
In this work, we propose a new method that uses semantically related questions, dubbed basic questions, acting as noise to evaluate the robustness of VQA models.
4-Connected Shift Residual Networks
Interestingly, when incorporating shifts to all point-wise convolutions in residual networks, 4-connected shifts outperform 8-connected shifts.
BERT for Evidence Retrieval and Claim Verification
Motivated by the promising performance of pre-trained language models, we investigate BERT in an evidence retrieval and claim verification pipeline for the FEVER fact extraction and verification challenge.
HyperLearn: A Distributed Approach for Representation Learning in Datasets With Many Modalities
Encoding multimedia items into a continuous low-dimensional semantic space such that both types of relations are captured and preserved is extremely challenging, especially if the goal is a unified end-to-end learning framework.
Interactive Search and Exploration in Online Discussion Forums Using Multimodal Embeddings
In this paper we present a novel interactive multimodal learning system, which facilitates search and exploration in large networks of social multimedia users.
Multimodal Classification of Urban Micro-Events
In this paper we explore several methods of creating such a classifier, including early, late, hybrid fusion and representation learning using multimodal graphs.
Exquisitor: Interactive Learning at Large
Increasing scale is a dominant trend in today's multimedia collections, which especially impacts interactive applications.
Learning Task Relatedness in Multi-Task Learning for Images in Context
When task relations are explicitly defined based on domain knowledge multi-task learning (MTL) offers such concurrent solutions, while exploiting relatedness between multiple tasks performed over the same dataset.
Many Task Learning with Task Routing
Typical multi-task learning (MTL) methods rely on architectural adjustments and a large trainable parameter set to jointly optimize over several tasks.
OmniArt: Multi-task Deep Learning for Artistic Data Analysis
Vast amounts of artistic data is scattered on-line from both museums and art applications.
Unsupervised, Efficient and Semantic Expertise Retrieval
We compare our model to state-of-the-art unsupervised statistical vector space and probabilistic generative approaches.
