Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms

no code implementations • 24 Oct 2024 • Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, Zhe Gan

Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation.

Diversity Language Modelling +3

Contrastive Localized Language-Image Pre-Training

no code implementations • 3 Oct 2024 • Hong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, BoWen Zhang, Yinfei Yang, Zhe Gan

Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations facilitating various applications.

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

no code implementations • 30 Sep 2024 • Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, BoWen Zhang, Yanghao Li, Sam Dodge, Keen You, Zhen Yang, Aleksei Timofeev, Mingze Xu, Hong-You Chen, Jean-Philippe Fauconnier, Zhengfeng Lai, Haoxuan You, ZiRui Wang, Afshin Dehghan, Peter Grasch, Yinfei Yang

We present MM1. 5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning.

Optical Character Recognition (OCR) Video Understanding +1

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

no code implementations • 8 Apr 2024 • Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, Zhe Gan

For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks.

Instruction Following

Ascle: A Python Natural Language Processing Toolkit for Medical Text Generation

3 code implementations • 28 Nov 2023 • Rui Yang, Qingcheng Zeng, Keen You, Yujie Qiao, Lucas Huang, Chia-Chun Hsieh, Benjamin Rosand, Jeremy Goldwasser, Amisha D Dave, Tiarnan D. L. Keenan, Emily Y Chew, Dragomir Radev, Zhiyong Lu, Hua Xu, Qingyu Chen, Irene Li

This study introduces Ascle, a pioneering natural language processing (NLP) toolkit designed for medical text generation.

Machine Translation Question Answering +5

EHRKit: A Python Natural Language Processing Toolkit for Electronic Health Record Texts

no code implementations • 13 Apr 2022 • Irene Li, Keen You, Yujie Qiao, Lucas Huang, Chia-Chun Hsieh, Benjamin Rosand, Jeremy Goldwasser, Dragomir Radev

The Electronic Health Record (EHR) is an essential part of the modern medical system and impacts healthcare delivery, operations, and research.

Information Retrieval Machine Translation +5

``where is this relationship going?'': Understanding Relationship Trajectories in Narrative Text

no code implementations • Joint Conference on Lexical and Computational Semantics 2020 • Keen You, Dan Goldwasser

We examine a new commonsense reasoning task: given a narrative describing a social interaction that centers on two protagonists, systems make inferences about the underlying relationship trajectory.

Navigate

"where is this relationship going?": Understanding Relationship Trajectories in Narrative Text

1 code implementation • 29 Oct 2020 • Keen You, Dan Goldwasser

We examine a new commonsense reasoning task: given a narrative describing a social interaction that centers on two protagonists, systems make inferences about the underlying relationship trajectory.

Navigate