Search Results for author:
Found 252 papers, 90 papers with code
Example-Guided Image Synthesis using Masked Spatial-Channel Attention and Self-Supervision
Example-guided image synthesis has recently been attempted to synthesize an image from a semantic label map and an exemplary image.
Harnessing GPT-4V(ision) for Insurance: A Preliminary Exploration
The emergence of Large Multimodal Models (LMMs) marks a significant milestone in the development of artificial intelligence.
MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators
Recent advances in Text-to-Video generation (T2V) have achieved remarkable success in synthesizing high-quality general videos from textual descriptions.
Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution
Technically, SATeCo freezes all the parameters of the pre-trained UNet and VAE, and only optimizes two deliberately-designed spatial feature adaptation (SFA) and temporal feature alignment (TFA) modules, in the decoder of UNet and VAE.
DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance
However, camera movement synthesis with music and dance remains an unsolved challenging problem due to the scarcity of paired data.
SoMeLVLM: A Large Vision Language Model for Social Media Processing
The powerful Large Vision Language Models make it possible to handle a variety of tasks simultaneously, but even with carefully designed prompting methods, the general domain models often fall short in aligning with the unique speaking style and context of social media tasks.
Emo-Avatar: Efficient Monocular Video Style Avatar through Texture Rendering
To address the above problems, we propose the Efficient Monotonic Video Style Avatar (Emo-Avatar) through deferred neural rendering that enhances StyleGAN's capacity for producing dynamic, drivable portrait videos.
Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach
At inference, we generate images with arbitrary expansion multiples by inputting an anchor image and its corresponding positional embeddings.
Human vs. LMMs: Exploring the Discrepancy in Emoji Interpretation and Usage in Digital Communication
Leveraging Large Multimodal Models (LMMs) to simulate human behaviors when processing multimodal information, especially in the context of social media, has garnered immense interest due to its broad potential and far-reaching implications.
CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs
When exploring the development of Artificial General Intelligence (AGI), a critical task for these models involves interpreting and processing information from multiple image inputs.
Ranked #3 on
Visual Reasoning
on Winoground
Bring Metric Functions into Diffusion Models
The first module, similar to a standard DDPM, learns to predict the added noise and is unaffected by the metric function.
Video Understanding with Large Language Models: A Survey
With the burgeoning growth of online video platforms and the escalating volume of video content, the demand for proficient video understanding tools has intensified markedly.
SurgicalPart-SAM: Part-to-Whole Collaborative Prompting for Surgical Instrument Segmentation
Specifically, we achieve this by proposing (1) Collaborative Prompts that describe instrument structures via collaborating category-level and part-level texts; (2) Cross-Modal Prompt Encoder that encodes text prompts jointly with visual embeddings into discriminative part-level representations; and (3) Part-to-Whole Adaptive Fusion and Hierarchical Decoding that adaptively fuse the part-level representations into a whole for accurate instrument segmentation in surgical scenarios.
GPT-4V(ision) as A Social Media Analysis Engine
Our investigation begins with a preliminary quantitative analysis for each task using existing benchmark datasets, followed by a careful review of the results and a selection of qualitative samples that illustrate GPT-4V's potential in understanding multimodal social media content.
A Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Therefore, this review aims to provide a detailed overview of the development and deployment of LLMs in medicine, including the challenges and opportunities they face.
Mixture of Weak & Strong Experts on Graphs
We propose to decouple the two modalities by mixture of weak and strong experts (Mowst), where the weak expert is a light-weight Multi-layer Perceptron (MLP), and the strong expert is an off-the-shelf Graph Neural Network (GNN).
Deceptive Fairness Attacks on Graphs via Meta Learning
We study deceptive fairness attacks on graphs to answer the following question: How can we achieve poisoning attacks on a graph learning model to exacerbate the bias deceptively?
OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation
We hope our proposed framework, benchmark, and LMM evaluation could help establish the intriguing interleaved image-text generation task.
Understanding Divergent Framing of the Supreme Court Controversies: Social Media vs. News Outlets
Understanding the framing of political issues is of paramount importance as it significantly shapes how individuals perceive, interpret, and engage with these matters.
SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation
However, we observe two problems with this naive pipeline: (1) the domain gap between natural objects and surgical instruments leads to inferior generalisation of SAM; and (2) SAM relies on precise point or box locations for accurate segmentation, requiring either extensive manual guidance or a well-performing specialist detector for prompt preparation, which leads to a complex multi-stage pipeline.
Jurassic World Remake: Bringing Ancient Fossils Back to Life via Zero-Shot Long Image-to-Image Translation
In this work, we use text-guided latent diffusion models for zero-shot image-to-image translation (I2I) across large domain gaps (longI2I), where large amounts of new visual features and new geometry need to be generated to enter the target domain.
User-Controllable Recommendation via Counterfactual Retrospective and Prospective Explanations
Acknowledging the recent advancements in explainable recommender systems that enhance users' understanding of recommendation mechanisms, we propose leveraging these advancements to improve user controllability.
MobileVidFactory: Automatic Diffusion-Based Social Media Video Generation for Mobile Devices from Text
In the basic generation, we take advantage of the pretrained image diffusion model, and adapt it to a high-quality open-domain vertical video generator for mobile devices.
LLM-Rec: Personalized Recommendation via Prompting Large Language Models
Notably, the success of LLM-Rec lies in its prompting strategies, which effectively tap into the language model's comprehension of both general and specific item characteristics.
Domain-Scalable Unpaired Image Translation via Latent Space Anchoring
However, given a UNIT model trained on certain domains, it is difficult for current methods to incorporate new domains because they often need to train the full model on both existing and new domains.
Improving Video Colorization by Test-Time Tuning
With the advancements in deep learning, video colorization by propagating color information from a colorized reference frame to a monochrome video sequence has been well explored.
ICDAR 2023 Competition on Structured Text Extraction from Visually-Rich Document Images
It is hoped that this competition will attract many researchers in the field of CV and NLP, and bring some new thoughts to the field of Document AI.
Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA
In this paper, we first model a confounding effect that causes language and vision bias simultaneously, then propose a counterfactual inference to remove the influence of this effect.
Learning to Evaluate the Artness of AI-generated Images
Assessing the artness of AI-generated images continues to be a challenge within the realm of image generation.
Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation
In contrast, we propose a parameter-free temporal shift module that can leverage the spatial U-Net as is for video generation.
Meta-causal Learning for Single Domain Generalization
Under this paradigm, we propose a meta-causal learning method to learn meta-knowledge, that is, how to infer the causes of domain shift between the auxiliary and source domains during training.
Predicting Adverse Neonatal Outcomes for Preterm Neonates with Multi-Task Learning
In this work, we first analyze the correlations between three adverse neonatal outcomes and then formulate the diagnosis of multiple neonatal outcomes as a multi-task learning (MTL) problem.
Bias or Diversity? Unraveling Fine-Grained Thematic Discrepancy in U.S. News Headlines
There is a broad consensus that news media outlets incorporate ideological biases in their news articles.
Human Behavior in the Time of COVID-19: Learning from Big Data
On one hand, human behavior is found to shape the spread of the disease.
VideoXum: Cross-modal Visual and Textural Summarization of Videos
We propose a new joint video and text summarization task.
Ranked #1 on
Video Summarization
on videoxum
Grounding 3D Object Affordance from 2D Interactions in Images
Comprehensive experiments on PIAD demonstrate the reliability of the proposed task and the superiority of our method.
Spatial-Aware Token for Weakly Supervised Object Localization
Specifically, a spatial token is first introduced in the input space to aggregate representations for localization task.
SegPrompt: Using Segmentation Map as a Better Prompt to Finetune Deep Models for Kidney Stone Classification
However, the shortage of annotated training data poses a severe problem in improving the performance and generalization ability of the trained model.
Adaptive Siamese Tracking with a Compact Latent Network
Under this viewing, we perform an in-depth analysis for them through visual simulations and real tracking examples, and find that the failure cases in some challenging situations can be regarded as the issue of missing decisive samples in offline training.
Computational Assessment of Hyperpartisanship in News Titles
Through an analysis of the topic distribution, we find that societal issues gradually receive more attention from all media groups.
Event-Guided Person Re-Identification via Sparse-Dense Complementary Learning
Video-based person re-identification (Re-ID) is a prominent computer vision topic due to its wide range of video surveillance applications.
Representation Learning
Video-Based Person Re-Identification
PromptCap: Prompt-Guided Image Captioning for VQA with GPT-3
PromptCap outperforms generic captions by a large margin and achieves state-of-the-art accuracy on knowledge-based VQA tasks (60. 4% on OK-VQA and 59. 6% on A-OKVQA).
QuantArt: Quantizing Image Style Transfer Towards High Visual Fidelity
The mechanism of existing style transfer algorithms is by minimizing a hybrid loss function to push the generated image toward high similarities in both content and style.
3D Point Cloud Pre-training with Knowledge Distillation from 2D Images
The recent success of pre-trained 2D vision models is mostly attributable to learning from large-scale datasets.
Structure-Guided Image Completion with Image-level and Object-level Semantic Discriminators
Moreover, the object-level discriminators take aligned instances as inputs to enforce the realism of individual objects.
A Unified Framework for Contrastive Learning from a Perspective of Affinity Matrix
In this study, we present a new unified contrastive learning representation framework (named UniCLR) suitable for all the above four kinds of methods from a novel perspective of basic affinity matrix.
Improving Visual-textual Sentiment Analysis by Fusing Expert Features
Visual-textual sentiment analysis aims to predict sentiment with the input of a pair of image and text.
Stare at What You See: Masked Image Modeling without Reconstruction
However, unlike the low-level features such as pixel values, we argue the features extracted by powerful teacher models already encode rich semantic correlation across regions in an intact image. This raises one question: is reconstruction necessary in Masked Image Modeling (MIM) with a teacher model?
PromptCap: Prompt-Guided Task-Aware Image Captioning
PromptCap outperforms generic captions by a large margin and achieves state-of-the-art accuracy on knowledge-based VQA tasks (60. 4% on OK-VQA and 59. 6% on A-OKVQA).
Ranked #1 on
Visual Question Answering
on TextVQA test-standard
FeDXL: Provable Federated Learning for Deep X-Risk Optimization
To this end, we propose an active-passive decomposition framework that decouples the gradient's components with two types, namely active parts and passive parts, where the active parts depend on local data that are computed with the local model and the passive parts depend on other machines that are communicated/computed based on historical models and samples.
Learning a Grammar Inducer from Massive Uncurated Instructional Videos
While previous work focuses on building systems for inducing grammars on text that are well-aligned with video content, we investigate the scenario, in which text and video are only in loose correspondence.
TLDW: Extreme Multimodal Summarisation of News Videos
Multimodal summarisation with multimodal output is drawing increasing attention due to the rapid growth of multimedia data.
Contextual Modeling for 3D Dense Captioning on Point Clouds
With such global and local contextual modeling strategies, our proposed model can effectively characterize the object representations and contextual information and thereby generate comprehensive and detailed descriptions of the located objects.
Causal Inference via Nonlinear Variable Decorrelation for Healthcare Applications
Causal inference and model interpretability research are gaining increasing attention, especially in the domains of healthcare and bioinformatics.
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment
and 2) how to mitigate the impact of these factors?
Ranked #2 on
Video Retrieval
on MSR-VTT-1kA
(using extra training data)
Remote Medication Status Prediction for Individuals with Parkinson's Disease using Time-series Data from Smartphones
Medication for neurological diseases such as the Parkinson's disease usually happens remotely away from hospitals.
Bi-Calibration Networks for Weakly-Supervised Video Representation Learning
The video-to-text/video-to-query projections over text prototypes/query vocabulary then start the text-to-query or query-to-text calibration to estimate the amendment to query or text.
Stand-Alone Inter-Frame Attention in Video Models
In this paper, we present a new recipe of inter-frame attention block, namely Stand-alone Inter-Frame Attention (SIFA), that novelly delves into the deformation across frames to estimate local self-attention on each spatial location.
Ranked #13 on
Action Recognition
on Something-Something V1
Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
The advent of large-scale pre-trained language models has contributed greatly to the recent progress in natural language processing.
Automatic Relation-aware Graph Network Proliferation
However, the currently used graph search space overemphasizes learning node features and neglects mining hierarchical relational information.
Ranked #2 on
Link Prediction
on TSP/HCP Benchmark set
Localized Adversarial Domain Generalization
Adversarial domain generalization is a popular approach to DG, but conventional approaches (1) struggle to sufficiently align features so that local neighborhoods are mixed across domains; and (2) can suffer from feature space over collapse which can threaten generalization performance.
Deep Federated Anomaly Detection for Multivariate Time Series Data
Specifically, we first design an Exemplar-based Deep Neural network (ExDNN) to learn local time series representations based on their compatibility with an exemplar module which consists of hidden parameters learned to capture varieties of normal patterns on each edge device.
Explainable Fairness in Recommendation
In this paper, we study the problem of explainable fairness, which helps to gain insights about why a system is fair or unfair, and guides the design of fair recommender systems with a more informed and unified methodology.
CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training
We propose cascaded modulation GAN (CM-GAN), a new network design consisting of an encoder with Fourier convolution blocks that extract multi-scale feature representations from the input image with holes and a dual-stream decoder with a novel cascaded global-spatial modulation block at each scale level.
Ranked #1 on
Image Inpainting
on Places2
Breast Cancer Induced Bone Osteolysis Prediction Using Temporal Variational Auto-Encoders
Its ability of predicting the development of bone lesions in cancer-invading bones can assist in assessing the risk of impending fractures and choosing proper treatments in breast cancer bone metastasis.
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning
Non-exemplar class-incremental learning is to recognize both the old and new classes when old class samples cannot be saved.
RawlsGCN: Towards Rawlsian Difference Principle on Graph Convolutional Network
Graph Convolutional Network (GCN) plays pivotal roles in many real-world applications.
Point Cloud Denoising via Momentum Ascent in Gradient Fields
To achieve point cloud denoising, traditional methods heavily rely on geometric priors, and most learning-based approaches suffer from outliers and loss of details.
MANet: Improving Video Denoising with a Multi-Alignment Network
In video denoising, the adjacent frames often provide very useful information, but accurate alignment is needed before such information can be harnassed.
Cross-modal Contrastive Distillation for Instructional Activity Anticipation
In this study, we aim to predict the plausible future action steps given an observation of the past and study the task of instructional activity anticipation.
SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Color Editing
Recently, large pretrained models (e. g., BERT, StyleGAN, CLIP) show great knowledge transfer and generalization capability on various downstream tasks within their domains.
Multi-modal Dependency Tree for Video Captioning
To this end, we propose a novel video captioning method that generates a sentence by first constructing a multi-modal dependency tree and then traversing the constructed tree, where the syntactic structure and semantic relationship in the sentence are represented by the tree topology.
SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing
Recently, large pretrained models (e. g., BERT, StyleGAN, CLIP) have shown great knowledge transfer and generalization capability on various downstream tasks within their domains.
Music Sentiment Transfer
Results and literature suggest that the task of music sentiment transfer is more difficult than image sentiment transfer because of the temporal characteristics of music and lack of existing datasets.
Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning
In this work, we leverage instructional videos to study humans' decision-making processes, focusing on learning a model to plan goal-directed actions in real-life videos.
CoSeg: Cognitively Inspired Unsupervised Generic Event Segmentation
Unlike the mainstream clustering-based methods, our framework exploits a transformer-based feature reconstruction scheme to detect event boundary by reconstruction errors.
Learning to Aggregate and Refine Noisy Labels for Visual Sentiment Analysis
The learned prototypes and their labels can be regarded as denoising features and labels for the local regions and can guide the training process to prevent the model from overfitting the noisy cases.
Federated Learning of Molecular Properties with Graph Neural Networks in a Heterogeneous Setting
FLIT(+) can align the local training across heterogeneous clients by improving the performance for uncertain samples.
Stock Price Prediction Under Anomalous Circumstances
The stock market is volatile and complicated, especially in 2020.
LibFewShot: A Comprehensive Library for Few-shot Learning
Furthermore, based on LibFewShot, we provide comprehensive evaluations on multiple benchmarks with various backbone architectures to evaluate common pitfalls and effects of different training tricks.
Learning Fine-Grained Motion Embedding for Landscape Animation
To tackle this problem, we propose a model named FGLA to generate high-quality and realistic videos by learning Fine-Grained motion embedding for Landscape Animation.
LUAI Challenge 2021 on Learning to Understand Aerial Images
This report summarizes the results of Learning to Understand Aerial Images (LUAI) 2021 challenge held on ICCV 2021, which focuses on object detection and semantic segmentation in aerial images.
Multi-Modulation Network for Audio-Visual Event Localization
We propose a novel MultiModulation Network (M2N) to learn the above correlation and leverage it as semantic guidance to modulate the related auditory, visual, and fused features.
Learning Conditional Knowledge Distillation for Degraded-Reference Image Quality Assessment
And the reference space is optimized to capture deep image priors that are useful for quality assessment.
UniFaceGAN: A Unified Framework for Temporally Consistent Facial Video Editing
Compared with the state-of-the-art facial image editing methods, our framework generates video portraits that are more photo-realistic and temporally smooth.
Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization
We propose to remove the bias information misused by the target task with a cross-sample adversarial debiasing (CSAD) method.
Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph
Entity-aware image captioning aims to describe named entities and events related to the image by utilizing the background knowledge in the associated article.
Adaptive Recursive Circle Framework for Fine-grained Action Recognition
It inherits the operators and parameters of the original layer but is slightly different in the use of those operators and parameters.
Trip-ROMA: Self-Supervised Learning with Triplets and Random Mappings
However, in small data regimes, we can not obtain a sufficient number of negative pairs or effectively avoid the over-fitting problem when negatives are not used at all.
Noise Stability Regularization for Improving BERT Fine-tuning
The brittleness of this process is often reflected by the sensitivity to random seeds.
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
To tackle this, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment.
Improving OCR-Based Image Captioning by Incorporating Geometrical Relationship
Under the guidance of the geometrical relationship between OCR tokens, our LSTM-R capitalizes on a newly-devised relation-aware pointer network to select OCR tokens from the scene text for OCR-based image captioning.
Structured Multi-Level Interaction Network for Video Moment Localization via Language Query
In particular, for cross-modal interaction, we interact the sentence-level query with the whole moment while interact the word-level query with content and boundary, as in a coarse-to-fine manner.
How COVID-19 Has Changed Crowdfunding: Evidence From GoFundMe
More importantly, sentiment analysis and the paired sample t-test are performed to examine the differences in crowdfunding campaigns before and after the COVID-19 outbreak that started in March 2020.
SAT: 2D Semantics Assisted Training for 3D Visual Grounding
3D visual grounding aims at grounding a natural language description about a 3D scene, usually represented in the form of 3D point clouds, to the targeted object region.
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-and-Language Pre-training
To tackle this, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment.
Few-shot Partial Multi-view Learning
The challenges of this task are twofold: (i) it is difficult to overcome the impact of data scarcity under the interference of missing views; (ii) the limited number of data exacerbates information scarcity, thus making it harder to address the view-missing issue in turn.
Video-aided Unsupervised Grammar Induction
We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video.
Facial Attribute Transformers for Precise and Robust Makeup Transfer
In this paper, we address the problem of makeup transfer, which aims at transplanting the makeup from the reference face to the source face while preserving the identity of the source.
ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows
The forward inference projects input images into deep features, while the backward inference remaps deep features back to input images in a lossless and unbiased way.
Memory Enhanced Embedding Learning for Cross-Modal Video-Text Retrieval
The cross-modal memory module is employed to record the instance embeddings of all the datasets for global negative mining.
When Few-Shot Learning Meets Video Object Detection
We employ a transfer-learning framework to effectively train the video object detector on a large number of base-class objects and a few video clips of novel-class objects.
Group-aware Label Transfer for Domain Adaptive Person Re-identification
In this paper, we propose a Group-aware Label Transfer (GLT) algorithm, which enables the online interaction and mutual promotion of pseudo-label prediction and representation learning.
From Static to Dynamic Prediction: Wildfire Risk Assessment Based on Multiple Environmental Factors
Wildfire is one of the biggest disasters that frequently occurs on the west coast of the United States.
Enhanced Aspect-Based Sentiment Analysis Models with Progressive Self-supervised Attention Learning
In aspect-based sentiment analysis (ABSA), many neural models are equipped with an attention mechanism to quantify the contribution of each context word to sentiment prediction.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
Object Detection in Aerial Images: A Large-Scale Benchmark and Challenges
In this paper, we present a large-scale Dataset of Object deTection in Aerial images (DOTA) and comprehensive baselines for ODAI.
Are Top School Students More Critical of Their Professors? Mining Comments on RateMyProfessor.com
Next, we perform exploratory data analysis to delve into the data.
DAIL: Dataset-Aware and Invariant Learning for Face Recognition
To solve the first issue of identity overlapping, we propose a dataset-aware loss for multi-dataset training by reducing the penalty when the same person appears in multiple datasets.
Semantic Layout Manipulation with High-Resolution Sparse Attention
A core problem of this task is how to transfer visual details from the input images to the new semantic layout while making the resulting image visually realistic.
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
Due to this aligned representation learning, even pre-trained on the same downstream task dataset, TAP already boosts the absolute accuracy on the TextVQA dataset by +5. 4%, compared with a non-TAP baseline.
Multi-Scale 2D Temporal Adjacent Networks for Moment Localization with Natural Language
It is a challenging problem because a target moment may take place in the context of other temporal moments in the untrimmed video.
XraySyn: Realistic View Synthesis From a Single Radiograph Through CT Priors
A radiograph visualizes the internal anatomy of a patient through the use of X-ray, which projects 3D information onto a 2D plane.
Social Media Study of Public Opinions on Potential COVID-19 Vaccines: Informing Dissent, Disparities, and Dissemination
People who have the worst personal pandemic experience are more likely to hold the anti-vaccine opinion.
Social and Information Networks
Learning Semantic-aware Normalization for Generative Adversarial Networks
Such a model disentangles latent factors according to the semantic of feature channels by channel-/group- wise fusion of latent codes and feature channels.
Slender Object Detection: Diagnoses and Improvements
In this paper, we are concerned with the detection of a particular type of objects with extreme aspect ratios, namely \textbf{slender objects}.
Ranked #1 on
Object Detection
on COCO+
Content-based Analysis of the Cultural Differences between TikTok and Douyin
Short-form video social media shifts away from the traditional media paradigm by telling the audience a dynamic story to attract their attention.
Pose-based Body Language Recognition for Emotion and Psychiatric Symptom Interpretation
We then evaluate the framework on a proposed URMC dataset, which consists of conversations between a standardized patient and a behavioral health professional, along with expert annotations of body language, emotions, and potential psychiatric symptoms.
Predicting Parkinson's Disease with Multimodal Irregularly Collected Longitudinal Smartphone Data
Parkinsons Disease is a neurological disorder and prevalent in elderly people.
Region Comparison Network for Interpretable Few-shot Image Classification
For that, in this work, we propose a metric learning based method named Region Comparison Network (RCN), which is able to reveal how few-shot learning works as in a neural network as well as to find out specific regions that are related to each other in images coming from the query and support sets.
Ranked #33 on
Few-Shot Image Classification
on CIFAR-FS 5-way (5-shot)
Dynamic Context-guided Capsule Network for Multimodal Machine Translation
Particularly, we represent the input image with global and regional visual features, we introduce two parallel DCCNs to model multimodal context vectors with visual features at different granularities.
Ranked #3 on
Multimodal Machine Translation
on Multi30K
Learning to Localize Actions from Moments
In this paper, we introduce a new design of transfer learning type to learn action localization for a large set of action categories, but only on action moments from the categories of interest and temporal annotations of untrimmed videos from a small set of action classes.
A Smartphone-based System for Real-time Early Childhood Caries Diagnosis
The identification of ECC in an early stage usually requires expertise in the field, and hence is often ignored by parents.
Improving One-stage Visual Grounding by Recursive Sub-query Construction
We improve one-stage visual grounding by addressing current limitations on grounding long and complex queries.
Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification
In this paper, we propose a novel dynamic dual-attentive aggregation (DDAG) learning method by mining both intra-modality part-level and cross-modality graph-level contextual cues for VI-ReID.
A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
Multi-modal neural machine translation (NMT) aims to translate source sentences into a target language paired with images.
Universal Model for Multi-Domain Medical Image Retrieval
Usually, one image retrieval model is only trained to handle images from one modality or one source.
Task-agnostic Temporally Consistent Facial Video Editing
Compared with the state-of-the-art facial image editing methods, our framework generates video portraits that are more photo-realistic and temporally smooth.
Monitoring Depression Trend on Twitter during the COVID-19 Pandemic
The COVID-19 pandemic has severely affected people's daily lives and caused tremendous economic loss worldwide.
Global Image Sentiment Transfer
This work proposes a novel framework consisting of a reference image retrieval step and a global sentiment transfer step to transfer sentiments of images according to a given sentiment tag.
Image Sentiment Transfer
In this paper, we propose an effective and flexible framework that performs image sentiment transfer at the object level.
Real-time Universal Style Transfer on High-resolution Images via Zero-channel Pruning
Extracting effective deep features to represent content and style information is the key to universal style transfer.
Personalized Fashion Recommendation from Personal Social Media Data: An Item-to-Set Metric Learning Approach
In this work, we study the problem of personalized fashion recommendation from social media data, i. e. recommending new outfits to social media users that fit their fashion preferences.
On Vocabulary Reliance in Scene Text Recognition
This remedy alleviates the problem of vocabulary reliance and improves the overall scene text recognition performance.
Unsupervised Low-light Image Enhancement with Decoupled Networks
In this paper, we tackle the problem of enhancing real-world low-light images with significant noise in an unsupervised fashion.
Alleviating the Incompatibility between Cross Entropy Loss and Episode Training for Few-shot Skin Disease Classification
Inspired by the recent success of Few-Shot Learning (FSL) in natural image classification, we propose to apply FSL to skin disease identification to address the extreme scarcity of training sample problem.
The Ivory Tower Lost: How College Students Respond Differently than the General Public to the COVID-19 Pandemic
Recently, the pandemic of the novel Coronavirus Disease-2019 (COVID-19) has presented governments with ultimate challenges.
In the Eyes of the Beholder: Analyzing Social Media Use of Neutral and Controversial Terms for COVID-19
To model the substantive difference of tweets with controversial terms and those with non-controversial terms, we apply topic modeling and LIWC-based sentiment analysis.
Example-Guided Image Synthesis across Arbitrary Scenes using Masked Spatial-Channel Attention and Self-Supervision
Example-guided image synthesis has recently been attempted to synthesize an image from a semantic label map and an exemplary image.
Structured Landmark Detection via Topology-Adapting Deep Graph Learning
Image landmark detection aims to automatically identify the locations of predefined fiducial points.
Ranked #4 on
Face Alignment
on COFW-68
Unsupervised Learning of Landmarks based on Inter-Intra Subject Consistencies
To recover from the transformed images back to the original subject, the landmark detector is forced to learn spatial locations that contain the consistent semantic meanings both for the paired intra-subject images and between the paired inter-subject images.
TuiGAN: Learning Versatile Image-to-Image Translation with Two Unpaired Images
With TuiGAN, an image is translated in a coarse-to-fine manner where the generated image is gradually refined from global structures to local details.
Learning a Weakly-Supervised Video Actor-Action Segmentation Model with a Wise Selection
Instead of blindly trusting quality-inconsistent PAs, WS^2 employs a learning-based selection to select effective PAs and a novel region integrity criterion as a stopping condition for weakly-supervised training.
Video-based Person Re-Identification using Gated Convolutional Recurrent Neural Networks
In this paper, we introduce a novel gating mechanism to deep neural networks.
Optical Flow Estimation
Video-Based Person Re-Identification
Unifying Specialist Image Embedding into Universal Image Embedding
Therefore, we propose to distill the knowledge in multiple specialists into a universal embedding to solve this problem.
Adaptive Offline Quintuplet Loss for Image-Text Matching
For each image or text anchor in a training mini-batch, the model is trained to distinguish between a positive and the most confusing negative of the anchor mined from the mini-batch (i. e. online hard negative).
Anatomy-aware 3D Human Pose Estimation with Bone-based Pose Decomposition
In this work, we propose a new solution to 3D human pose estimation in videos.
Ranked #12 on
Monocular 3D Human Pose Estimation
on Human3.6M
Expressing Objects just like Words: Recurrent Visual Embedding for Image-Text Matching
Existing image-text matching approaches typically infer the similarity of an image-text pair by capturing and aggregating the affinities between the text and each independent object of the image.
Asymmetric Distribution Measure for Few-shot Learning
Given the natural asymmetric relation between a query image and a support class, we argue that an asymmetric measure is more suitable for metric-based few-shot learning.
Mi YouTube es Su YouTube? Analyzing the Cultures using YouTube Thumbnails of Popular Videos
In this study, we explore culture preferences among countries using the thumbnails of YouTube trending videos.
#MeToo on Campus: Studying College Sexual Assault at Scale Using Data Reported on Social Media
Recently, the emergence of the #MeToo trend on social media has empowered thousands of people to share their own sexual harassment experiences.
Fine-grained Image-to-Image Transformation towards Visual Recognition
In this paper, we aim at transforming an image with a fine-grained category to synthesize new images that preserve the identity of the input image, which can thereby benefit the subsequent fine-grained image recognition and few-shot learning tasks.
TransMatch: A Transfer-Learning Scheme for Semi-Supervised Few-Shot Learning
Under the proposed framework, we develop a novel method for semi-supervised few-shot learning called TransMatch by instantiating the three components with Imprinting and MixMatch.
Neural Simile Recognition with Cyclic Multitask Learning and Local Attention
Simile recognition is to detect simile sentences and to extract simile components, i. e., tenors and vehicles.
Graph-based Neural Sentence Ordering
Sentence ordering is to restore the original paragraph from a set of sentences.
Iterative Dual Domain Adaptation for Neural Machine Translation
Previous studies on the domain adaptation for neural machine translation (NMT) mainly focus on the one-pass transferring out-of-domain translation knowledge to in-domain NMT model.
Grounding-Tracking-Integration
In this paper, we study Tracking by Language that localizes the target box sequence in a video based on a language query.
Learning Sparse 2D Temporal Adjacent Networks for Temporal Action Localization
In this report, we introduce the Winner method for HACS Temporal Action Localization Challenge 2019.
Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
We address the problem of retrieving a specific moment from an untrimmed video by a query sentence.
Ultrafast Photorealistic Style Transfer via Neural Architecture Search
Our method consists of a construction step (C-step) to build a photorealistic stylization network and a pruning step (P-step) for acceleration.
Example-Guided Scene Image Synthesis using Masked Spatial-Channel Attention and Patch-Based Self-Supervision
Example-guided image synthesis has been recently attempted to synthesize an image from a semantic label map and an exemplary image.
EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformations
In this study, we propose a new EnAET framework to further improve existing semi-supervised methods with self-supervised information.
Ranked #1 on
Semi-Supervised Image Classification
on STL-10
Defensive Few-shot Learning
(2) how to narrow the distribution gap between clean and adversarial examples under the few-shot setting?
Open-Ended Visual Question Answering by Multi-Modal Domain Adaptation
A straightforward solution is to fine-tune a pre-trained source model by using those limited labeled target data, but it usually cannot work well due to the considerable difference between the data distributions of the source and target domains.
Learning Deep Bilinear Transformation for Fine-grained Image Representation
However, the computational cost to learn pairwise interactions between deep feature channels is prohibitively expensive, which restricts this powerful transformation to be used in deep neural networks.
SMP Challenge: An Overview of Social Media Prediction Challenge 2019
In the SMP Challenge at ACM Multimedia 2019, we introduce a novel prediction task Temporal Popularity Prediction, which focuses on predicting future interaction or attractiveness (in terms of clicks, views or likes etc.)
Unsupervised Pose Flow Learning for Pose Guided Synthesis
Pose guided synthesis aims to generate a new image in an arbitrary target pose while preserving the appearance details from the source image.
Gaussian Temporal Awareness Networks for Action Localization
Temporally localizing actions in a video is a fundamental challenge in video understanding.
Large-scale Tag-based Font Retrieval with Generative Feature Learning
In this paper, we address the problem of large-scale tag-based font retrieval which aims to bring semantics to the font selection process and enable people without expert knowledge to use fonts effectively.
AI for Earth: Rainforest Conservation by Acoustic Surveillance
Saving rainforests is a key to halting adverse climate changes.
BIG-bench Machine Learning
Environmental Sound Classification
+2
A Fast and Accurate One-Stage Approach to Visual Grounding
We propose a simple, fast, and accurate one-stage approach to visual grounding, inspired by the following insight.
Exploiting Temporal Relationships in Video Moment Localization with Natural Language
We address the problem of video moment localization with natural language, i. e. localizing a video segment described by a natural language sentence.
Semi-Supervised Adversarial Monocular Depth Estimation
In this paper, we address the problem of monocular depth estimation when only a limited number of training image-depth pairs are available.
ADN: Artifact Disentanglement Network for Unsupervised Metal Artifact Reduction
Current deep neural network based approaches to computed tomography (CT) metal artifact reduction (MAR) are supervised methods that rely on synthesized metal artifacts for training.
Weakly Supervised Body Part Segmentation with Pose based Part Priors
The core idea is first converting the sparse weak labels such as keypoints to the initial estimate of body part masks, and then iteratively refine the part mask predictions.
Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment
In addition, in order to enrich the decoder with descriptive semantics and enforce the correctness of the deterministic medical-related contents such as mentions of organs or diagnoses, we extract medical concepts based on the radiology reports in the training data and fine-tune the encoder to extract the most frequent medical concepts from the x-ray images.
Fast Universal Style Transfer for Artistic and Photorealistic Rendering
Given a pair of images as the source of content and the reference of style, existing solutions usually first train an auto-encoder (AE) to reconstruct the image using deep features and then embeds pre-defined style transfer modules into the AE reconstruction procedure to transfer the style of the reconstructed image through modifying the deep features.
Uncovering Download Fraud Activities in Mobile App Markets
To the best of our knowledge, this is the first work that investigates the download fraud problem in mobile App markets.
DuDoNet: Dual Domain Network for CT Metal Artifact Reduction
The linkage between the sigogram and image domains is a novel Radon inversion layer that allows the gradients to back-propagate from the image domain to the sinogram domain during training.
Generative Mask Pyramid Network for CT/CBCT Metal Artifact Reduction with Joint Projection-Sinogram Correction
A conventional approach to computed tomography (CT) or cone beam CT (CBCT) metal artifact reduction is to replace the X-ray projection data within the metal trace with synthesized data.
Patch Transformer for Multi-tagging Whole Slide Histopathology Images
Automated whole slide image (WSI) tagging has become a growing demand due to the increasing volume and diversity of WSIs collected nowadays in histopathology.
StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks
Finally compared to existing universal style transfer networks for photorealistic rendering such as PhotoWCT that stacks multiple well-trained auto-encoders and WCT transforms in a non-end-to-end manner, the architectures designed by StyleNAS produce better style-transferred images with details preserving, using a tiny number of operators/parameters, and enjoying around 500x inference time speed-up.
Artifact Disentanglement Network for Unsupervised Metal Artifact Reduction
Extensive experiments show that our method significantly outperforms the existing unsupervised models for image-to-image translation problems, and achieves comparable performance to existing supervised models on a synthesized dataset.
Progressive Self-Supervised Attention Learning for Aspect-Level Sentiment Analysis
In aspect-level sentiment classification (ASC), it is prevalent to equip dominant neural models with attention mechanisms, for the sake of acquiring the importance of each context word on the given aspect.
Aspect-Based Sentiment Analysis (ABSA)
Sentiment Classification
Relational Reasoning using Prior Knowledge for Visual Captioning
Extensive experiments on the MS-COCO image captioning benchmark and the MSVD video captioning benchmark validate the superiority of our method on leveraging prior commonsense knowledge to enhance relational reasoning for visual captioning.
Spatio-temporal Video Re-localization by Warp LSTM
The need for efficiently finding the video content a user wants is increasing because of the erupting of user-generated videos on the Web.
Human-Centered Emotion Recognition in Animated GIFs
The framework consists of a facial attention module and a hierarchical segment temporal module.
Revisiting Local Descriptor based Image-to-Class Measure for Few-shot Learning
Its key difference from the literature is the replacement of the image-level feature based measure in the final layer by a local descriptor based image-to-class measure.
Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods
Representation learning with small labeled data have emerged in many problems, since the success of deep neural networks often relies on the availability of a huge amount of labeled data that is expensive to collect.
Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition
Learning subtle yet discriminative features (e. g., beak and eyes for a bird) plays a significant role in fine-grained image recognition.
Ranked #1 on
Fine-Grained Image Classification
on iNaturalist
Fine-Grained Image Classification
Fine-Grained Image Recognition
Multiview 2D/3D Rigid Registration via a Point-Of-Interest Network for Tracking and Triangulation ($\text{POINT}^2$)
As the POI tracker is shift-invariant, $\text{POINT}^2$ is more robust to the initial pose of the 3D pre-intervention image.
Exploring Explicit Domain Supervision for Latent Space Disentanglement in Unpaired Image-to-Image Translation
After pre-training, this network is used to extract the domain-specific features of each image.
Foreground-aware Image Inpainting
We show that by such disentanglement, the contour completion model predicts reasonable contours of objects, and further substantially improves the performance of image inpainting.
AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data
The success of deep neural networks often relies on a large amount of labeled examples, which can be difficult to obtain in many real scenarios.
Joint Vertebrae Identification and Localization in Spinal CT Images by Combining Short- and Long-Range Contextual Information
For the long-range contextual information, we propose a multi-task bidirectional recurrent neural network (Bi-RNN) to encode the spatial and contextual information among the vertebrae of the visible spine column.
Joint Vertebrae Identification And Localization In Spinal Ct Images
Skin Disease Classification versus Skin Lesion Characterization: Achieving Robust Diagnosis using Multi-label Deep Neural Networks
In contrast, for lesion-targeted classification, we can achieve a much higher mAP of 0. 70.
A Deep Multi-task Learning Approach to Skin Lesion Classification
Skin lesion identification is a key step toward dermatological diagnosis.
More Knowledge is Better: Cross-Modality Volume Completion and 3D+2D Segmentation for Intracardiac Echocardiography Contouring
Using catheter ablation to treat atrial fibrillation increasingly relies on intracardiac echocardiography (ICE) for an anatomical delineation of the left atrium and the pulmonary veins that enter the atrium.
Adversarial Sparse-View CBCT Artifact Reduction
We present an effective post-processing method to reduce the artifacts from sparsely reconstructed cone-beam CT (CBCT) images.
Real-Time Referring Expression Comprehension by Single-Stage Grounding Network
Experiments on RefCOCO, RefCOCO+, and RefCOCOg datasets demonstrate that our proposed SSG without relying on any region proposals can achieve comparable performance with other advanced models.
