Deep Multimodal Semantic Embeddings for Speech and Images

11 Nov 2015 • David Harwath • James Glass

In this paper, we present a model which takes as input a corpus of images with relevant spoken captions and finds a correspondence between the two modalities. We employ a pair of convolutional neural networks to model visual objects and speech signals at the word level, and tie the networks together with an embedding and alignment model which learns a joint semantic space over both modalities... (read more)

PDF Abstract

Results from the Paper

Edit

Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

METHOD	TYPE
🤖 No Methods Found	Help the community by adding them if they're not listed; e.g. Deep Residual Learning for Image Recognition uses ResNet

Deep Multimodal Semantic Embeddings for Speech and Images

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods used in the Paper

Add Remove

Edit Social Preview

Deep Multimodal Semantic Embeddings for Speech and Images Edit social preview

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods used in the Paper Edit Add Remove

Deep Multimodal Semantic Embeddings for Speech and Images

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods used in the Paper

Add Remove