Where is Misty? Interpreting Spatial Descriptors by Modeling Regions in Space

EMNLP 2017  ·  Nikita Kitaev, Dan Klein ·

We present a model for locating regions in space based on natural language descriptions. Starting with a 3D scene and a sentence, our model is able to associate words in the sentence with regions in the scene, interpret relations such as {`}on top of{'} or {`}next to,{'} and finally locate the region described in the sentence. All components form a single neural network that is trained end-to-end without prior knowledge of object segmentation. To evaluate our model, we construct and release a new dataset consisting of Minecraft scenes with crowdsourced natural language descriptions. We achieve a 32{\%} relative error reduction compared to a strong neural baseline.

PDF Abstract


Results from the Paper

  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.


No methods listed for this paper. Add relevant methods here