research area Human-Computer Interaction, research area Tools, Platforms, Frameworksconference UIST
content type paperpublished October 2020

mage: Fluid Moves Between Code and Graphical Work in Computational Notebooks

In collaboration with Carnegie Mellon University, Georgia Institute of Technology

AuthorsMary Beth Kery, Donghao Ren, Fred Hohman, Dominik Moritz, Kanit Wongsuphasawat, Kayur Patel

View publication

We aim to increase the flexibility at which a data worker can choose the right tool for the job, regardless of whether the tool is a code library or an interactive graphical user interface (GUI). To achieve this flexibility, we extend computational notebooks with a new API mage, which supports tools that can represent themselves as both code and GUI as needed. We discuss the design of mage as well as design opportunities in the space of flexible code/GUI tools for data work. To understand tooling needs, we conduct a study with nine professional practitioners and elicit their feedback on mage and potential areas for flexible code/GUI tooling. We then implement six client tools for mage that illustrate the main themes of our study findings. Finally, we discuss open challenges in providing flexible code/GUI interactions for data workers.

Related readings and updates.

ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities

March 26, 2025research area Speech and Natural Language Processing, research area Tools, Platforms, Frameworksconference NAACL

Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies…
Read more

BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks

August 3, 2024research area Human-Computer Interaction, research area Tools, Platforms, Frameworksconference IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)

This paper was accepted at IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) 2024. Programmers frequently engage with machine learning tutorials in computational notebooks and have been adopting code generation technologies based on large language models (LLMs). However, they encounter difficulties in understanding and working with code produced by LLMs. To mitigate these challenges, we introduce a novel workflow into…
Read more