research area Human-Computer Interaction, research area Tools, Platforms, Frameworks
content type paperpublished September 2024

Misty: UI Prototyping Through Interactive Conceptual Blending

AuthorsYuwen Lu, Alan Leung, Amanda Swearngin, Jeffrey Nichols, Titus Barik

View publication

UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples. Inspired by the cognitive process of conceptual blending, we introduce a novel UI workflow that allows developers to rapidly incorporate diverse aspects from design examples into work-in-progress UIs. We prototyped this workflow as Misty. Through an exploratory first-use study with 14 frontend developers, we assessed Misty's effectiveness and gathered feedback on this workflow. Our findings suggest that Misty's conceptual blending workflow helps developers kickstart creative explorations, flexibly specify intent in different stages of prototyping, and inspires developers through serendipitous UI blends. Misty demonstrates the potential for tools that blur the boundaries between developers and designers.

Related readings and updates.

ILuvUI: Instruction-Tuned Language-Vision Modeling of UIs from Machine Conversations

July 14, 2025research area Human-Computer Interactionconference IUI

Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining existing pixel-based methods with a Large Language Model (LLM). Unlike prior art, our method requires no human-provided annotations…
Read more

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

September 10, 2024research area Computer Vision, research area Human-Computer Interactionconference ECCV

Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced understanding of mobile UI screens, equipped with referring, grounding, and reasoning capabilities. Given that UI screens typically exhibit a more…
Read more