no code implementations • 29 Sep 2022 • Tao Li, Gang Li, Jingjie Zheng, Purple Wang, Yang Li
To investigate the problem, we create a new dataset that consists of 77, 820 sequences of human user-agent interaction on mobile interfaces in which 20% involves multiple rounds of interactions.
1 code implementation • EMNLP 2020 • Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, Zhiwei Guan
We propose widget captioning, a novel task for automatically generating language descriptions for UI elements from multimodal input including both the image and the structural representations of user interfaces.
