Search Results for author:
Found 7 papers, 1 papers with code
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks.
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data.
AXNav: Replaying Accessibility Tests from Natural Language
Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs, however to our knowledge no one has yet explored their use in controlling assistive technologies for the purposes of supporting accessibility testing.
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
We use a deep learning based approach to predict whether a selected element in a mobile UI screenshot will be perceived by users as tappable, based on pixels only instead of view hierarchies required by previous work.
IMACS: Image Model Attribution Comparison Summaries
Developing a suitable Deep Neural Network (DNN) often requires significant iteration, where different model versions are evaluated and compared.
Sketch-based Creativity Support Tools using Deep Learning
Sketching is a natural and effective visual communication medium commonly used in creative processes.
Scones: Towards Conversational Authoring of Sketches
Iteratively refining and critiquing sketches are crucial steps to developing effective designs.
