A11yExtensions: Accessibility Extensions to Augment Mobile AI Assistive Technology In-Situ

Existing visual AI assistive technologies have usability gaps, and may need additional adaptations and features to serve users’ needs. We propose A11yExtensions, in-situ interventions that augment existing mobile AI assistive technology with add-on services. Add-ons include features that have been researched but are not yet deployed (e.g., cross-checking AI results), or that are only available in certain applications (e.g., camera aiming assistance). Through co-design sessions with two blind accessibility professionals, we designed and implemented three exemplar extensions, leveraging mobile automation tools to invoke add-ons, enabling just-in-time interventions for adaptability. We found that A11yExtensions provide opportunities to test new features and a new degree of flexibility and customization, though they introduce additional onboarding and communication challenges. We also derived a design space of accessibility extensions as a basis for future extension designs. Overall, A11yExtensions is a demonstration of the effectiveness of deploying new features in-situ via automation, with the technologies people actually use in their day-to-day lives.