Search Results for author:
Found 161 papers, 74 papers with code
G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images
Novel view synthesis aims to generate new view images of a given view image collection.
PairAug: What Can Augmented Image-Text Pairs Do for Radiology?
Acknowledging this limitation, our objective is to devise a framework capable of concurrently augmenting medical image and text data.
VL-Mamba: Exploring State Space Models for Multimodal Learning
The extensive experiments on diverse multimodal benchmarks with competitive performance show the effectiveness of our proposed VL-Mamba and demonstrate the great potential of applying state space models for multimodal learning tasks.
Ranked #61 on
Visual Question Answering
on MM-Vet
Thermal-NeRF: Neural Radiance Fields from an Infrared Camera
In recent years, Neural Radiance Fields (NeRFs) have demonstrated significant potential in encoding highly-detailed 3D geometry and environmental appearance, positioning themselves as a promising alternative to traditional explicit representation for 3D scene reconstruction.
Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework
Medical vision language pre-training (VLP) has emerged as a frontier of research, enabling zero-shot pathological recognition by comparing the query image with the textual descriptions for each disease.
Unveiling the Potential of Robustness in Evaluating Causal Inference Models
The growing demand for personalized decision-making has led to a surge of interest in estimating the Conditional Average Treatment Effect (CATE).
Explicit Interaction for Fusion-Based Place Recognition
Fusion-based place recognition is an emerging technique jointly utilizing multi-modal perception data, to recognize previously visited places in GPS-denied scenarios for robots and autonomous vehicles.
NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) stands as a key research problem of Embodied AI, aiming at enabling agents to navigate in unseen environments following linguistic instructions.
ModaVerse: Efficiently Transforming Modalities with LLMs
This alignment process, which synchronizes a language model trained on textual data with encoders and decoders trained on multi-modal data, often necessitates extensive training of several projection layers in multiple stages.
SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization
In this study, we address this gap by introducing SpecFormer, specifically designed to enhance ViTs' resilience against adversarial attacks, with support from carefully derived theoretical guarantees.
WebVLN: Vision-and-Language Navigation on Websites
Vision-and-Language Navigation (VLN) task aims to enable AI agents to accurately understand and follow natural language instructions to navigate through real-world environments, ultimately reaching specific target locations.
Subject-Oriented Video Captioning
To address this problem, we propose a new video captioning task, subject-oriented video captioning, which allows users to specify the describing target via a bounding box.
MMBaT: A Multi-task Framework for mmWave-based Human Body Reconstruction and Translation Prediction
Human body reconstruction with Millimeter Wave (mmWave) radar point clouds has gained significant interest due to its ability to work in adverse environments and its capacity to mitigate privacy concerns associated with traditional camera-based solutions.
The Causal Impact of Credit Lines on Spending Distributions
Consumer credit services offered by e-commerce platforms provide customers with convenient loan access during shopping and have the potential to stimulate sales.
Invariant Random Forest: Tree-Based Model Solution for OOD Generalization
This paper introduces a novel and effective solution for OOD generalization of decision tree models, named Invariant Decision Tree (IDT).
Dynamic Inertial Poser (DynaIP): Part-Based Motion Dynamics Learning for Enhanced Human Pose Estimation with Sparse Inertial Sensors
This paper introduces a novel human pose estimation approach using sparse inertial sensors, addressing the shortcomings of previous methods reliant on synthetic data.
Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning
In this paper, we reconsider versatile self-supervised learning from the perspective of continual learning and propose MedCoSS, a continuous self-supervised learning approach for multi-modal medical data.
Watermarking Vision-Language Pre-trained Models for Multi-modal Embedding as a Service
Our extensive experiments on various datasets indicate that the proposed watermarking approach is effective and safe for verifying the copyright of VLPs for multi-modal EaaS and robust against model extraction attacks.
Improving Online Source-free Domain Adaptation for Object Detection by Unsupervised Data Acquisition
Effective object detection in mobile robots is challenged by deployment in diverse and unfamiliar environments.
Align before Search: Aligning Ads Image to Text for Accurate Cross-Modal Sponsored Search
Conventional research mainly studies from the view of modeling the implicit correlations between images and texts for query-ads matching, ignoring the alignment of detailed product information and resulting in suboptimal search performance. In this work, we propose a simple alignment network for explicitly mapping fine-grained visual parts in ads images to the corresponding text, which leverages the co-occurrence structure consistency between vision and language spaces without requiring expensive labeled training data.
Ranked #1 on
Image-text matching
on CommercialAdsDataset
Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrieval
Different from Composed Image Retrieval task that requires expensive labels for training task-specific models, Zero-Shot Composed Image Retrieval (ZS-CIR) involves diverse tasks with a broad range of visual content manipulation intent that could be related to domain, scene, object, and attribute.
SwitchGPT: Adapting Large Language Models for Non-Text Outputs
To bridge this gap, we propose a novel approach, \methodname, from a modality conversion perspective that evolves a text-based LLM into a multi-modal one.
S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning
VQA Natural Language Explanation (VQA-NLE) task aims to explain the decision-making process of VQA models in natural language.
DeLELSTM: Decomposition-based Linear Explainable LSTM to Capture Instantaneous and Long-term Effects in Time Series
The tensorized LSTM assigns each variable with a unique hidden state making up a matrix $\mathbf{h}_t$, and the standard LSTM models all the variables with a shared hidden state $\mathbf{H}_t$.
BHSD: A 3D Multi-Class Brain Hemorrhage Segmentation Dataset
Intracranial hemorrhage (ICH) is a pathological condition characterized by bleeding inside the skull or brain, which can be attributed to various factors.
March in Chat: Interactive Prompting for Remote Embodied Referring Expression
Nevertheless, this poses more challenges than other VLN tasks since it requires agents to infer a navigation plan only based on a short instruction.
VLN-PETL: Parameter-Efficient Transfer Learning for Vision-and-Language Navigation
The performance of the Vision-and-Language Navigation~(VLN) tasks has witnessed rapid progress recently thanks to the use of large pre-trained vision-and-language models.
Likelihood-Based Text-to-Image Evaluation with Patch-Level Perceptual and Semantic Credit Assignment
In this paper, we propose to evaluate text-to-image generation performance by directly estimating the likelihood of the generated images using a pre-trained likelihood-based text-to-image generative model, i. e., a higher likelihood indicates better perceptual quality and better text-image alignment.
Identity-Consistent Aggregation for Video Object Detection
In Video Object Detection (VID), a common practice is to leverage the rich temporal contexts from the video to enhance the object representations in each frame.
Ranked #1 on
Video Object Detection
on ImageNet VID
(MAP metric)
Self-Prompting Large Vision Models for Few-Shot Medical Image Segmentation
Recent advancements in large foundation models have shown promising potential in the medical industry due to their flexible prompting capability.
Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval
In text-video retrieval, recent works have benefited from the powerful learning capabilities of pre-trained text-image foundation models (e. g., CLIP) by adapting them to the video domain.
AerialVLN: Vision-and-Language Navigation for UAVs
Navigating in the sky is more complicated than on the ground because agents need to consider the flying height and more complex spatial relationship reasoning.
Mind the Gap: Improving Success Rate of Vision-and-Language Navigation by Revisiting Oracle Success Routes
Vision-and-Language Navigation (VLN) aims to navigate to the target location by following a given instruction.
Scaling Data Generation in Vision-and-Language Navigation
Recent research in language-guided visual navigation has demonstrated a significant demand for the diversity of traversable environments and the quantity of supervision for training generalizable agents.
Probabilistic Learning of Multivariate Time Series with Temporal Irregularity
Multivariate sequential data collected in practice often exhibit temporal irregularities, including nonuniform time intervals and component misalignment.
Deep into The Domain Shift: Transfer Learning through Dependence Regularization
This paper proposes a new domain adaptation approach in which one can measure the differences in the internal dependence structure separately from those in the marginals.
Attention Mechanisms in Medical Image Segmentation: A Survey
This paper systematically reviews the basic principles of attention mechanisms and their applications in medical image segmentation.
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models
Trained with an unprecedented scale of data, large language models (LLMs) like ChatGPT and GPT-4 exhibit the emergence of significant reasoning abilities from model scaling.
S4M: Generating Radiology Reports by A Single Model for Multiple Body Parts
In this paper, we seek to design a report generation model that is able to generate reasonable reports even given different images of various body parts.
Realistic Noise Synthesis with Diffusion Models
Deep image denoising models often rely on large amount of training data for the high quality performance.
Photon Field Networks for Dynamic Real-Time Volumetric Global Illumination
In this paper, we present a novel method to enable real-time global illumination for volume data visualization.
DIPNet: Efficiency Distillation and Iterative Pruning for Image Super-Resolution
To address this issue, we propose a novel multi-stage lightweight network boosting method, which can enable lightweight networks to achieve outstanding performance.
HyperINR: A Fast and Predictive Hypernetwork for Implicit Neural Representations via Knowledge Distillation
Implicit Neural Representations (INRs) have recently exhibited immense potential in the field of scientific visualization for both data generation and visualization tasks.
Distributed Neural Representation for Reactive in situ Visualization
In this work, we develop an implicit neural representation for distributed volume data and incorporate it into the DIVA reactive programming system.
Program Generation from Diverse Video Demonstrations
The ability to use inductive reasoning to extract general rules from multiple observations is a vital indicator of intelligence.
Dynamic CVaR Portfolio Construction with Attention-Powered Generative Factor Learning
The dynamic portfolio construction problem requires dynamic modeling of the joint distribution of multivariate stock returns.
ShapeScaffolder: Structure-Aware 3D Shape Generation from Text
However, humans tend to understand both shape and text as being structure-based.
Learning to Dub Movies via Hierarchical Prosody Models
Given a piece of text, a video clip and a reference audio, the movie dubbing (also known as visual voice clone V2C) task aims to generate speeches that match the speaker's emotion presented in the video using the desired speaker voice as reference.
Decorr: Environment Partitioning for Invariant Learning and OOD Generalization
Invariant learning methods try to find an invariant predictor across several environments and have become popular in OOD generalization.
Efficient and Accurate Quantized Image Super-Resolution on Mobile NPUs, Mobile AI & AIM 2022 challenge: Report
While numerous solutions have been proposed for this problem in the past, they are usually not compatible with low-power mobile NPUs having many computational and memory constraints.
A Simple and Robust Correlation Filtering Method for Text-based Person Search
Text-based person search aims to associate pedestrian images with natural language descriptions.
Ranked #8 on
Text based Person Retrieval
on ICFG-PEDES
Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answering
Under this setting, these 2D spatial reasoning approaches cannot distinguish the fine-grain spatial relations between visual objects and scene texts on the same image plane, thereby impairing the interpretability and performance of TextVQA models.
FoVolNet: Fast Volume Rendering using Foveated Deep Neural Networks
We introduce FoVolNet -- a method to significantly increase the performance of volume data visualization.
Learning Distinct and Representative Styles for Image Captioning
Our innovative idea is to explore the rich modes in the training caption corpus to learn a set of "mode embeddings", and further use them to control the mode of the generated captions for existing image captioning models.
Moderately-Balanced Representation Learning for Treatment Effects with Orthogonality Information
In this paper, we propose a moderately-balanced representation learning (MBRL) framework based on recent covariates balanced representation learning methods and orthogonal machine learning theory.
Robust Causal Learning for the Estimation of Average Treatment Effects
Theoretically, the RCL estimators i) are as consistent and doubly robust as the DML estimators, and ii) can get rid of the error-compounding issue.
ClusTR: Exploring Efficient Self-attention via Clustering for Vision Transformers
Besides, we further extend the clustering-guided attention from single-scale to multi-scale, which is conducive to dense prediction tasks.
Fast Nearest Convolution for Real-Time Efficient Image Super-Resolution
The proposed nearest convolution has the same performance as the nearest upsampling but is much faster and more suitable for Android NNAPI.
Interactive Volume Visualization via Multi-Resolution Hash Encoding based Neural Representation
Neural networks have shown great potential in compressing volume data for visualization.
Optical Field Recovery in Jones Space
For cost-sensitive short-reach optical networks, some advanced single-polarization (SP) optical field recovery schemes are recently proposed to avoid chromatic dispersion-induced power fading effect, and improve the spectral efficiency for larger potential capacity.
Attract me to Buy: Advertisement Copywriting Generation with Multimodal Multi-structured Information
Therefore, we present a dataset, E-MMAD (e-commercial multimodal multi-structured advertisement copywriting), which requires, and supports much more detailed information in text generation.
BSRT: Improving Burst Super-Resolution with Swin Transformer and Flow-Guided Deformable Alignment
To overcome the challenges in BurstSR, we propose a Burst Super-Resolution Transformer (BSRT), which can significantly improve the capability of extracting inter-frame information and reconstruction.
Ranked #1 on
Burst Image Super-Resolution
on BurstSR
HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigation
Pre-training has been adopted in a few of recent works for Vision-and-Language Navigation (VLN).
Ranked #4 on
Visual Navigation
on R2R
Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
A long-term goal of AI research is to build intelligent agents that can communicate with humans in natural language, perceive the environment, and perform real-world tasks.
MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering
Knowledge-based visual question answering requires the ability of associating external knowledge for open-ended cross-modal scene understanding.
Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation
To bridge the discrete-to-continuous gap, we propose a predictor to generate a set of candidate waypoints during navigation, so that agents designed with high-level actions can be transferred to and trained in continuous environments.
OpenKBP-Opt: An international and reproducible evaluation of 76 knowledge-based planning pipelines
The dose predictions were input to four optimization models to form 76 unique KBP pipelines that generated 7600 plans.
Maintaining Reasoning Consistency in Compositional Visual Question Answering
Existing VQA models can answer a compositional question well, but cannot work well in terms of reasoning consistency in answering the compositional question and its sub-questions.
LocFormer: Enabling Transformers to Perform Temporal Moment Localization on Long Untrimmed Videos With a Feature Sampling Approach
We propose LocFormer, a Transformer-based model for video grounding which operates at a constant memory footprint regardless of the video length, i. e. number of frames.
UniMiSS: Universal Medical Self-Supervised Learning via Breaking Dimensionality Barrier
In this paper, we advocate bringing a wealth of 2D images like chest X-rays as compensation for the lack of 3D data, aiming to build a universal medical self-supervised representation learning framework, called UniMiSS.
Debiased Visual Question Answering from Feature and Sample Perspectives
From the sample perspective, we construct two types of negative samples to assist the training of the models, without introducing additional annotations.
Landmark-RxR: Solving Vision-and-Language Navigation with Fine-Grained Alignment Supervision
In Vision-and-Language Navigation (VLN) task, an agent is asked to navigate inside 3D indoor environments following given instructions.
V2C: Visual Voice Cloning
Existing Voice Cloning (VC) tasks aim to convert a paragraph text to a speech with desired voice specified by a reference audio.
Medical Visual Question Answering: A Survey
Medical Visual Question Answering~(VQA) is a combination of medical artificial intelligence and popular VQA challenges.
Risk and return prediction for pricing portfolios of non-performing consumer credit
We design a system for risk-analyzing and pricing portfolios of non-performing consumer credit loans.
Memory Regulation and Alignment toward Generalizer RGB-Infrared Person
The domain shift, coming from unneglectable modality gap and non-overlapped identity classes between training and test sets, is a major issue of RGB-Infrared person re-identification.
Data-driven advice for interpreting local and global model predictions in bioinformatics problems
For random forests, we find extremely high similarities and correlations of both local and global SHAP values and CFC scores, leading to very similar rankings and interpretations.
Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene
In a series of experiments, we demonstrate that human gesture cues, even without predefined semantics, improve the object-goal navigation for an embodied agent, outperforming various state-of-the-art methods.
Data Hiding with Deep Learning: A Survey Unifying Digital Watermarking and Steganography
The advancement of secure communication and identity verification fields has significantly increased through the use of deep learning techniques for data hiding.
Neighbor-view Enhanced Model for Vision and Language Navigation
Specifically, our NvEM utilizes a subject module and a reference module to collect contexts from neighbor views.
Ranked #82 on
Vision and Language Navigation
on VLN Challenge
Sketch, Ground, and Refine: Top-Down Dense Video Captioning
The dense video captioning task aims to detect and describe a sequence of events in a video for detailed and coherent storytelling.
VLN BERT: A Recurrent Vision-and-Language BERT for Navigation
In this paper we propose a recurrent BERT model that is time-aware for use in VLN.
Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression
The Remote Embodied Referring Expression (REVERIE) is a recently raised task that requires an agent to navigate to and localise a referred remote object according to a high-level language instruction.
Proposal-free One-stage Referring Expression via Grid-Word Cross-Attention
Referring Expression Comprehension (REC) has become one of the most important tasks in visual reasoning, since it is an essential step for many vision-and-language tasks such as visual question answering.
Chop Chop BERT: Visual Question Answering by Chopping VisualBERT's Heads
Based on this observation, we design a dynamic chopping module that can automatically remove heads and layers of the VisualBERT at an instance level when dealing with different questions.
Towards Accurate Text-based Image Captioning with Content Diversity Exploration
This task, however, is very challenging because an image often contains complex texts and visual information that is hard to be described comprehensively.
The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigation
Vision-and-Language Navigation (VLN) requires an agent to find a path to a remote location on the basis of natural-language instructions and a set of photo-realistic panoramas.
Diagnosing Vision-and-Language Navigation: What Really Matters
Results show that indoor navigation agents refer to both object and direction tokens when making decisions.
Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation
To further mine the non-salient region objects, we propose to exert the segmentation network's self-correction ability.
Jo-SRC: A Contrastive Approach for Combating Noisy Labels
Due to the memorization effect in Deep Neural Networks (DNNs), training with noisy labels usually results in inferior model performance.
Robust Orthogonal Machine Learning of Treatment Effects
Theoretically, the RCL estimators i) satisfy the (higher-order) orthogonal condition and are as \textit{consistent and doubly robust} as the DML estimators, and ii) get rid of the error-compounding issue.
Learning for Visual Navigation by Imagining the Success
ForeSIT is trained to imagine the recurrent latent representation of a future state that leads to success, e. g. either a sub-goal state that is important to reach before the target, or the goal state itself.
Multi-intersection Traffic Optimisation: A Benchmark Dataset and a Strong Baseline
To address these issues, in this work we carefully design our settings and propose a new dataset including both synthetic and real traffic data in more complex scenarios.
How to Train Your Agent to Read and Write
Typically, this requires an agent to fully understand the knowledge from the given text materials and generate correct and fluent novel paragraphs, which is very challenging in practice.
Ranked #3 on
KG-to-Text Generation
on AGENDA
Semantics for Robotic Mapping, Perception and Interaction: A Survey
In robotics and related research fields, the study of understanding is often referred to as semantics, which dictates what does the world "mean" to a robot, and is strongly tied to the question of how to represent that meaning.
Memory-Gated Recurrent Networks
The essence of multivariate sequential learning is all about how to extract dependencies in data.
The Causal Learning of Retail Delinquency
Classical estimators overlook the confounding effects and hence the estimation error can be magnificent.
MeisterMorxrc at SemEval-2020 Task 9: Fine-Tune Bert and Multitask Learning for Sentiment Analysis of Code-Mixed Tweets
Natural language processing (NLP) has been applied to various fields including text classification and sentiment analysis.
Simple is not Easy: A Simple Strong Baseline for TextVQA and TextCaps
Texts appearing in daily scenes that can be recognized by OCR (Optical Character Recognition) tools contain significant information, such as street name, product brand and prices.
Confidence-aware Non-repetitive Multimodal Transformers for TextCaps
Our CNMT consists of a reading, a reasoning and a generation modules, in which Reading Module employs better OCR systems to enhance text reading ability and a confidence embedding to select the most noteworthy tokens.
Generative Learning of Heterogeneous Tail Dependence
We propose a multivariate generative model to capture the complex dependence structure often encountered in business and financial data.
A Recurrent Vision-and-Language BERT for Navigation
In this paper we propose a recurrent BERT model that is time-aware for use in VLN.
Ranked #7 on
Visual Navigation
on R2R
Modular Graph Attention Network for Complex Visual Relational Reasoning
Moreover, to capture the complex logic in a query, we construct a relational graph to represent the visual objects and their relationships, and propose a multi-step reasoning method to progressively understand the complex logic.
Ranked #2 on
Referring Expression Comprehension
on CLEVR-Ref+
Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning
We then propose to recursively alternate the learning schemes of imitation and exploration to narrow the discrepancy between training and inference.
Language and Visual Entity Relationship Graph for Agent Navigation
From both the textual and visual perspectives, we find that the relationships among the scene, its objects, and directional clues are essential for the agent to interpret complex instructions and correctly perceive the environment.
Parsimonious Quantile Regression of Financial Asset Tail Dynamics via Sequential Learning
We propose a parsimonious quantile regression framework to learn the dynamic tail behaviors of financial asset returns.
MARS: Mixed Virtual and Real Wearable Sensors for Human Activity Recognition with Multi-Domain Deep Learning Model
Together with the rapid development of the Internet of Things (IoT), human activity recognition (HAR) using wearable Inertial Measurement Units (IMUs) becomes a promising technology for many research areas.
CogTree: Cognition Tree Loss for Unbiased Scene Graph Generation
We first build a cognitive structure CogTree to organize the relationships based on the prediction of a biased SGG model.
Ranked #2 on
Scene Graph Generation
on Visual Genome
(mean Recall @20 metric)
Attention-SLAM: A Visual Monocular SLAM Learning from Human Gaze
This paper proposes a novel simultaneous localization and mapping (SLAM) approach, namely Attention-SLAM, which simulates human navigation mode by combining a visual saliency model (SalNavNet) with traditional monocular visual SLAM.
Data-driven Meta-set Based Fine-Grained Visual Classification
To this end, we propose a data-driven meta-set based approach to deal with noisy web images for fine-grained recognition.
Object-and-Action Aware Model for Visual Language Navigation
The first is object description (e. g., 'table', 'door'), each presenting as a tip for the agent to determine the next action by finding the item visible in the environment, and the second is action specification (e. g., 'go straight', 'turn left') which allows the robot to directly predict the next movements without relying on visual perceptions.
Soft Expert Reward Learning for Vision-and-Language Navigation
In this paper, we introduce a Soft Expert Reward Learning (SERL) model to overcome the reward engineering designing and generalisation problems of the VLN task.
Referring Expression Comprehension: A Survey of Methods and Datasets
In this survey, we first examine the state of the art by comparing modern approaches to the problem.
Length-Controllable Image Captioning
We verify the merit of the proposed length level embedding on three models: two state-of-the-art (SOTA) autoregressive models with different types of decoder, as well as our proposed non-autoregressive model, to show its generalization ability.
Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering
However, there are few works studying the data augmentation problem for VQA and none of the existing image based augmentation schemes (such as rotation and flipping) can be directly applied to VQA due to its semantic structure -- an $\langle image, question, answer\rangle$ triplet needs to be maintained correctly.
DAM: Deliberation, Abandon and Memory Networks for Generating Detailed and Non-repetitive Responses in Visual Dialogue
The ability of generating detailed and non-repetitive responses is crucial for the agent to achieve human-like conversation.
Foreground-Background Imbalance Problem in Deep Object Detectors: A Review
Recent years have witnessed the remarkable developments made by deep learning techniques for object detection, a fundamentally challenging problem of computer vision.
Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering
In this paper, we depict an image by a multi-modal heterogeneous graph, which contains multiple layers of information corresponding to the visual, semantic and factual features.
Give Me Something to Eat: Referring Expression Comprehension with Commonsense Knowledge
In this case, we need to use commonsense knowledge to identify the objects in the image.
Structured Multimodal Attentions for TextVQA
In this paper, we propose an end-to-end structured multimodal attention (SMA) neural network to mainly solve the first two issues above.
Optimistic Agent: Accurate Graph-Based Value Estimation for More Successful Visual Navigation
Through empirical studies, we show that our agent, dubbed as the optimistic agent, has a more realistic estimate of the state value during a navigation episode which leads to a higher success rate.
Sub-Instruction Aware Vision-and-Language Navigation
Vision-and-language navigation requires an agent to navigate through a real 3D environment following natural language instructions.
Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning
To improve fine-grained video-text retrieval, we propose a Hierarchical Graph Reasoning (HGR) model, which decomposes video-text matching into global-to-local levels.
Intelligent Home 3D: Automatic 3D-House Design from Linguistic Descriptions Only
To this end, we propose a House Plan Generative Model (HPGM) that first translates the language input to a structural graph representation and then predicts the layout of rooms with a Graph Conditioned Layout Prediction Network (GC LPN) and generates the interior texture with a Language Conditioned Texture GAN (LCT-GAN).
Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs
From the ASG, we propose a novel ASG2Caption model, which is able to recognise user intentions and semantics in the graph, and therefore generate desired captions according to the graph structure.
Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehension
To bridge the gap, we propose a new dataset for visual reasoning in context of referring expression comprehension with two main features.
DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue
More importantly, we can tell which modality (visual or semantic) has more contribution in answering the current question by visualizing the gate values.
Ranked #6 on
Visual Dialog
on VisDial v0.9 val
OpenArray v1.0: a simple operator library for the decoupling of ocean modeling and parallel computing
In this work, we design a simple computing library to bridge the gap and decouple the work of ocean modeling from parallel computing.
Integrating Temporal and Spatial Attentions for VATEX Video Captioning Challenge 2019
This notebook paper presents our model in the VATEX video captioning challenge.
BERT-based Financial Sentiment Index and LSTM-based Stock Return Predictability
On the other hand, by combining with the other two commonly-used methods when it comes to building the sentiment index in the financial literature, i. e., the option-implied and the market-implied approaches, we propose a more general and comprehensive framework for the financial sentiment analysis, and further provide convincing outcomes for the predictability of individual stock return by combining LSTM (with a feature of a nonlinear mapping).
Neural Learning of Online Consumer Credit Risk
This paper takes a deep learning approach to understand consumer credit risk when e-commerce platforms issue unsecured credit to finance customers' purchase.
Understanding Distributional Ambiguity via Non-robust Chance Constraint
Without the boundedness constraint, the CCO problem is shown to perform uniformly better than the DRO problem, irrespective of the radius of the ambiguity set, the choice of the divergence measure, or the tail heaviness of the center distribution.
Show, Price and Negotiate: A Negotiator with Online Value Look-Ahead
Negotiation, as an essential and complicated aspect of online shopping, is still challenging for an intelligent agent.
REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
One of the long-term challenges of robotics is to enable robots to interact with humans in the visual world via natural language, as humans are visual animals that communicate through language.
You Only Look & Listen Once: Towards Fast and Accurate Visual Grounding
Most state-of-the-art methods in VG operate in a two-stage manner, wherein the first stage an object detector is adopted to generate a set of object proposals from the input image and the second stage is simply formulated as a cross-modal matching problem that finds the best match between the language query and all region proposals.
Gold Seeker: Information Gain from Policy Distributions for Goal-oriented Vision-and-Langauge Reasoning
For each potential action a distribution of the expected outcomes is calculated, and the value of the potential information gain assessed.
What's to know? Uncertainty as a Guide to Asking Goal-oriented Questions
We propose a solution to this problem based on a Bayesian model of the uncertainty in the implicit model maintained by the visual dialogue agent, and in the function used to select an appropriate output.
Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks
Being composed of node attention component and edge attention component, the proposed graph attention mechanism explicitly represents inter-object relationships, and properties with a flexibility and power impossible with competing approaches.
Deep Template Matching for Offline Handwritten Chinese Character Recognition
Just like its remarkable achievements in many computer vision tasks, the convolutional neural networks (CNN) provide an end-to-end solution in handwritten Chinese character recognition (HCCR) with great success.
Binary Classification
Offline Handwritten Chinese Character Recognition
+1
Goal-Oriented Visual Question Generation via Intermediate Rewards
Despite significant progress in a variety of vision-and-language problems, developing a method capable of asking intelligent, goal-oriented questions about images is proven to be an inscrutable challenge.
Connecting Language and Vision to Actions
A long-term goal of AI research is to build intelligent agents that can see the rich visual environment around us, communicate this understanding in natural language to humans and other agents, and act in a physical or embodied environment.
Topological Data Analysis Made Easy with the Topology ToolKit
This tutorial presents topological methods for the analysis and visualization of scientific data from a user's perspective, with the Topology ToolKit (TTK), a recently released open-source library for topological data analysis.
Visual Grounding via Accumulated Attention
There are three main challenges in VG: 1) what is the main focus in a query; 2) how to understand an image; 3) how to locate an object.
Learning Semantic Concepts and Order for Image and Sentence Matching
This mainly arises from that the representation of pixel-level image usually lacks of high-level semantic information as in its matched sentence.
Ranked #11 on
Image Retrieval
on Flickr30K 1K test
Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning
The Visual Dialogue task requires an agent to engage in a conversation about an image with a human.
Ranked #4 on
Visual Dialog
on VisDial v0.9 val
Asking the Difficult Questions: Goal-Oriented Visual Question Generation via Intermediate Rewards
Despite significant progress in a variety of vision-and-language problems, developing a method capable of asking intelligent, goal-oriented questions about images is proven to be an inscrutable challenge.
Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments
This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering.
Ranked #10 on
Visual Navigation
on R2R
Kill Two Birds with One Stone: Weakly-Supervised Neural Network for Image Annotation and Tag Refinement
These comments can be a description of the image, or some objects, attributes, scenes in it, which are normally used as the user-provided tags.
Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
To this end we propose a unified framework, the ParalleL AttentioN (PLAN) network, to discover the object in an image that is being referred to in variable length natural expression descriptions, from short phrases query to long multi-round dialogs.
Visual Question Answering with Memory-Augmented Networks
In this paper, we exploit a memory-augmented neural network to predict accurate answers to visual questions, even when those answers occur rarely in the training set.
Classification of Medical Images and Illustrations in the Biomedical Literature Using Synergic Deep Learning
The Classification of medical images and illustrations in the literature aims to label a medical image according to the modality it was produced or label an illustration according to its production attributes.
Care about you: towards large-scale human-centric visual relationship detection
In addressing this problem we first construct a large-scale human-centric visual relationship detection dataset (HCVRD), which provides many more types of relationship annotation (nearly 10K categories) than the previous released datasets.
Human-Object Interaction Detection
Relationship Detection
+1
The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
To train a method to perform even one of these operations accurately from {image, question, answer} tuples would be challenging, but to aim to achieve them all with a limited set of such training data seems ambitious at best.
Multi-Label Image Classification with Regional Latent Semantic Dependencies
Recent state-of-the-art approaches to multi-label image classification exploit the label dependencies in an image, at global level, largely improving the labeling capacity.
Visual Question Answering: A Survey of Methods and Datasets
Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities.
FVQA: Fact-based Visual Question Answering
We evaluate several baseline models on the FVQA dataset, and describe a novel model which is capable of reasoning about an image on the basis of supporting facts.
Ranked #2 on
Visual Question Answering (VQA)
on F-VQA
Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
Much recent progress in Vision-to-Language problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).
Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
Priming a recurrent neural network with this combined information, and the submitted question, leads to a very flexible visual question answering approach.
Explicit Knowledge-based Reasoning for Visual Question Answering
We describe a method for visual question answering which is capable of reasoning about contents of an image on the basis of information extracted from a large-scale knowledge base.
What value do explicit high level concepts have in vision to language problems?
Much of the recent progress in Vision-to-Language (V2L) problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).
The Cross-Depiction Problem: Computer Vision Algorithms for Recognising Objects in Artwork and in Photographs
The cross-depiction problem is that of recognising visual objects regardless of whether they are photographed, painted, drawn, etc.
