Search Results for author:
Found 14 papers, 9 papers with code
Attacking Vision-Language Computer Agents via Pop-ups
Autonomous agents powered by large vision and language models (VLM) have demonstrated significant potential in completing daily computer tasks, such as browsing the web to book travel and operating desktop software, which requires agents to understand these interfaces.
Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping
Sketches are a natural and accessible medium for UI designers to conceptualize early-stage ideas.
Distilling an End-to-End Voice Assistant Without Instruction Training Data
We show that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question Answering, Classification, and Translation.
TRINS: Towards Multimodal Language Models that Can Read
In this work, we introduce TRINS: a Text-Rich image INStruction dataset, with the objective of enhancing the reading ability of the multimodal large language model.
Best Practices and Lessons Learned on Synthetic Data
The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs.
Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering
Specifically, we manually curate 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input.
A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration
On specific subjects in MMLU, selecting a team of agents in the team optimization stage improves accuracy by up to 25. 0% in DyLAN.
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding
Instruction tuning unlocks the superior capability of Large Language Models (LLM) to interact with humans.
Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints
The limits of open-ended generative models are unclear, yet increasingly important.
Auditing Gender Presentation Differences in Text-to-Image Models
Text-to-image models, which can generate high-quality images based on textual input, have recently enabled various content-creation tools.
Robustness of Demonstration-based Learning Under Limited Data Scenario
Demonstration-based learning has shown great potential in stimulating pretrained language models' ability under limited data scenario.
Leveraging Expert Guided Adversarial Augmentation For Improving Generalization in Named Entity Recognition
We found that state-of-the-art NER systems trained on CoNLL 2003 training data drop performance dramatically on our challenging set.
Continual Sequence Generation with Adaptive Compositional Modules
Continual learning is essential for real-world deployment when there is a need to quickly adapt the model to new tasks without forgetting knowledge of old tasks.
Continual Learning for Text Classification with Information Disentanglement Based Regularization
Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time.
