Natural Language Informs the Interpretation of Iconic Gestures: A Computational Approach

IJCNLP 2017  ·  Ting Han, Julian Hough, David Schlangen ·

When giving descriptions, speakers often signify object shape or size with hand gestures. Such so-called {`}iconic{'} gestures represent their meaning through their relevance to referents in the verbal content, rather than having a conventional form. The gesture form on its own is often ambiguous, and the aspect of the referent that it highlights is constrained by what the language makes salient. We show how the verbal content guides gesture interpretation through a computational model that frames the task as a multi-label classification task that maps multimodal utterances to semantic categories, using annotated human-human data.

PDF Abstract IJCNLP 2017 PDF IJCNLP 2017 Abstract
No code implementations yet. Submit your code now

Datasets


  Add Datasets introduced or used in this paper

Results from the Paper


  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods


No methods listed for this paper. Add relevant methods here