Search Results for author:
Found 11 papers, 4 papers with code
LITE: Intent-based Task Representation Learning Using Weak Supervision
Users write to-dos as personal notes to themselves, about things they need to complete, remember or organize.
On the Effects of Data Scale on UI Control Agents
Moreover, AndroidControl is the most diverse computer control dataset to date, including 14, 548 unique tasks over 833 Android apps, thus allowing us to conduct in-depth analysis of the model performance in and out of the domain of the training data.
VQA Training Sets are Self-play Environments for Generating Few-shot Pools
The use of external tools, whether other ML models, search engines, or APIs, can significantly improve performance by breaking down high-level reasoning questions into sub-questions that are answerable by individual tools, but this approach has similar dataset construction costs to teach fine-tuned models how to use the available tools.
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
AndroidWorld and the experiments in this paper are available at github. com/google-research/android_world.
Latent State Estimation Helps UI Agents to Reason
In the context of autonomous UI agents we then show that LLMs used in this manner are more than $76\%$ accurate at inferring various aspects of latent state, such as performed (vs. commanded) actions and task progression.
UINav: A Practical Approach to Train On-Device Automation Agents
Automation systems that can autonomously drive application user interfaces to complete user tasks are of great benefit, especially when users are situationally or permanently impaired.
Android in the Wild: A Large-Scale Dataset for Android Device Control
The dataset contains human demonstrations of device interactions, including the screens and actions, and corresponding natural language instructions.
Lexi: Self-Supervised Learning of the UI Language
Along with text, these resources include visual content such as UI screenshots and images of application icons referenced in the text.
Inducing a hierarchy for multi-class classification problems
In applications where categorical labels follow a natural hierarchy, classification methods that exploit the label structure often outperform those that do not.
Bew: Towards Answering Business-Entity-Related Web Questions
We present a practical approach, called BewQA, that can answer Bew queries by mining a template of the business-related webpages and using the template to guide the search.
FLIN: A Flexible Natural Language Interface for Web Navigation
AI assistants can now carry out tasks for users by directly interacting with website UIs.
