research area Human-Computer Interaction, research area Tools, Platforms, Frameworks
content type paperpublished September 2024

Misty: UI Prototyping Through Interactive Conceptual Blending

AuthorsYuwen Lu, Alan Leung, Amanda Swearngin, Jeffrey Nichols, Titus Barik

View publication

UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples. Inspired by the cognitive process of conceptual blending, we introduce a novel UI workflow that allows developers to rapidly incorporate diverse aspects from design examples into work-in-progress UIs. We prototyped this workflow as Misty. Through an exploratory first-use study with 14 frontend developers, we assessed Misty's effectiveness and gathered feedback on this workflow. Our findings suggest that Misty's conceptual blending workflow helps developers kickstart creative explorations, flexibly specify intent in different stages of prototyping, and inspires developers through serendipitous UI blends. Misty demonstrates the potential for tools that blur the boundaries between developers and designers.

Related readings and updates.

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

September 10, 2024research area Computer Vision, research area Human-Computer Interactionconference ECCV

Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced understanding of mobile UI screens, equipped with referring, grounding, and reasoning capabilities. Given that UI screens typically exhibit a more…
Read more

UI-JEPA: Towards Active Perception of User Intent Through Onscreen User Activity

September 9, 2024research area Human-Computer Interaction, research area Methods and Algorithms

Generating user intent from a sequence of user interface (UI) actions is a core challenge in comprehensive UI understanding. Recent advancements in multimodal large language models (MLLMs) have led to substantial progress in this area, but their demands for extensive model parameters, computing power, and high latency makes them impractical for scenarios requiring lightweight, on-device solutions with low latency or heightened privacy…
Read more