Teaching Machines to Code: Neural Markup Generation with Visual Attention

15 Feb 2018  ·  Sumeet S. Singh ·

We present a neural transducer model with visual attention that learns to generate LaTeX markup of a real-world math formula given its image. Applying sequence modeling and transduction techniques that have been very successful across modalities such as natural language, image, handwriting, speech and audio; we construct an image-to-markup model that learns to produce syntactically and semantically correct LaTeX markup code over 150 words long and achieves a BLEU score of 89%; improving upon the previous state-of-art for the Im2Latex problem. We also demonstrate with heat-map visualization how attention helps in interpreting the model and can pinpoint (detect and localize) symbols on the image accurately despite having been trained without any bounding box data.

PDF Abstract

Datasets


Introduced in the Paper:

I2L-140K Im2latex-90k

Used in the Paper:

im2latex-100k
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Optical Character Recognition I2L-140K I2L-NOPOOL BLEU 89.09% # 1
Optical Character Recognition I2L-140K I2L-STRIPS BLEU 89% # 2
Optical Character Recognition im2latex-100k I2L-STRIPS BLEU 88.86% # 1

Methods


No methods listed for this paper. Add relevant methods here