Athena: Intermediate Representations for Iterative Scaffolded App Generation with an LLM
AuthorsJazbo Beason, Ruijia Cheng, Eldon Schoop, Jeffrey Nichols
Athena: Intermediate Representations for Iterative Scaffolded App Generation with an LLM
AuthorsJazbo Beason, Ruijia Cheng, Eldon Schoop, Jeffrey Nichols
It is challenging to generate the code for a complete user interface using a Large Language Model (LLM). User interfaces are complex and their implementations often consist of multiple, inter-related files that together specify the contents of each screen, the navigation flows between the screens, and the data model used throughout the application. It is challenging to craft a single prompt for an LLM that contains enough detail to generate a complete user interface, and even then the result is frequently a single large and difficult to understand file that contains all of the generated screens. In this paper, we introduce Athena, a prototype application generation environment that demonstrates how the use of shared intermediate representations, including an app storyboard, data model, and GUI skeletons, can help a developer work with an LLM in an iterative fashion to craft a complete user interface. These intermediate representations also scaffold the LLM’s code generation process, producing organized and structured code in multiple files while limiting errors. We evaluated Athena with a user study that found 75% of participants preferred our prototype over a typical chatbot-style baseline for prototyping apps.
BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks
August 3, 2024research area Human-Computer Interaction, research area Tools, Platforms, Frameworksconference IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)
This paper was accepted at IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) 2024.
Programmers frequently engage with machine learning tutorials in computational notebooks and have been adopting code generation technologies based on large language models (LLMs). However, they encounter difficulties in understanding and working with code produced by LLMs. To mitigate these challenges, we introduce a novel workflow into…
Understanding Screen Relationships from Screenshots of Smartphone Applications
March 30, 2022research area Human-Computer Interactionconference IUI
All graphical user interfaces are comprised of one or more screens that may be shown to the user depending on their interactions. Identifying different screens of an app and understanding the type of changes that happen on the screens is a challenging task that can be applied in many areas including automatic app crawling, playback of app automation macros and large scale app dataset analysis. For example, an automated app crawler needs to…