Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning

1 code implementation • 1 Mar 2020 • Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, Jinshui Wang

However, the prerequisite of using screen readers is that developers have to add natural-language labels to the image-based components when they are developing the app.

Missing Labels

Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?

2 code implementations • 12 Aug 2020 • Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang Li

We conduct the first large-scale empirical study of seven representative GUI element detection methods on over 50k GUI images to understand the capabilities, limitations and effective designs of these methods.

Code Generation object-detection +1

Enhancing Virtual Assistant Intelligence: Precise Area Targeting for Instance-level User Intents beyond Metadata

no code implementations • 7 Jun 2023 • Mengyu Chen, Zhenchang Xing, Jieshan Chen, Chunyang Chen, Qinghua Lu

Although their capabilities of processing user intents have been developed rapidly, virtual assistants in most platforms are only capable of handling pre-defined high-level tasks supported by extra manual efforts of developers.

Towards Real Smart Apps: Investigating Human-AI Interactions in Smartphone On-Device AI Apps

no code implementations • 3 Jul 2023 • Jason Ching Yuen Siu, Jieshan Chen, Yujin Huang, Zhenchang Xing, Chunyang Chen

We further implemented our findings into a multi-faceted search-enabled gallery.