cross-modal knowledge enhancement mechanism for few-shot learning

1 Jan 2021  ·  Haiyang Zhang, Jiaming Duan, Liang Liu ·

Few-shot learning problems require models to recognize novel classes with only a few supported samples. However, it remains challenging for the model to generalize novel classes with such limited samples. Driven by human behavior, researchers introduced semantic information (e.g. novel categories descriptions, label names, etc.) onto existing methods as prior knowledge to generalize more precise class representations. Despite the promising performance, these methods are under the assumption that users are able to provide precise semantic information for all target categories and this is hard to be satisfied in a real scenario. To address this problem, we proposed a novel Cross-modality Knowledge Enhancement Mechanism(CKEM) to discover task-relevant information in external semantic knowledge automatically. CKEM first utilizes Cross-modality Graph Builder(CGB) to align two unitary modality information (support labeled images and external semantic knowledge) into a cross-modality knowledge graph. After that, with the message-passing mechanism, CKEM selects and transfers relevant knowledge from external semantic knowledge bank to original visual-based class representations in Knowledge Fusion Model(KFM). Through a series of experiments, we show that our method improves the existing metric-based meta-learning methods with 1\% - 5\% for 1-shot and 5-shot settings on both mini-ImageNet and tiered-ImageNet datasets.

PDF Abstract

Results from the Paper


  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods


No methods listed for this paper. Add relevant methods here