Despite increasing awareness of the need to support accessibility in mobile apps, many still lack support for key accessibility features. Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecyle. However, manual testing can be tedious, often has an overwhelming scope, and test passes can be difficult to time amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs; however, none have yet explored their use in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of natural language based accessibility testing workflow through a formative study. Based on this, we present a system that takes as input a manual accessibility test (e.g., "Search for a show in VoiceOver") and uses an LLM combined with pixel-based UI Understanding models to convert the test into a chaptered, navigable video that a QA tester can use to pinpoint issues. In each video, we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops) to help QA testers more easily pinpoint issues. We evaluate this system through a 10 participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and gave us several promising directions for future work.
Numerous accessibility features have been developed and included in consumer operating systems to provide people with a variety of disabilities additional ways to access computing devices. Unfortunately, many users, especially older adults who are more likely to experience ability changes, are not aware of these features or do not know which combination to use. In this paper, we first quantify this problem via a survey with 100 participants…
Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app's full functionality. We…