Web Science and Digital Libraries Research Group

  Figure 1: Example of AccessBolt with File Explorer application. (Figure 1 from Utku et al.) Introduction User interface (UI) automation is the process of programmatically controlling and testing the UI of applications. In our last blog on UI automation, we talk about the ideas of using it and walkthrough of these interesting tools using the Python programming language. The main purposes of UI automation are to simulate user interactions for testing, increase test coverage, and reduce repetitive manual testing. In this blog, we are going to talk about a recently published paper named “ Taming User-Interface Heterogeneity with Uniform Overlays for Blind Users, ” published at the UMAP 2022 conference, where Utku et al. explored how to improve the accessibility of desktop application interfaces for blind individuals who rely on screen readers . The paper introduces a novel system called AccessBolt that aims to make interacting with desktop software easier and more efficient using...

Fig: a demonstration of UI Automation in Notepad Essential criteria for accessibility assistance in the application are programmatic access and keyboard access. To test accessibility for people with different disabilities and limitations or those who prefer to use a keyboard, it is important that you test the accessibility of your Windows applications, assistive technology (AT) tools, and user interface (UI) frameworks.  You will not be able to use your application for users with visual, learning, dexterity/mobility, and with language/commune impairments or disabilities, without appropriate access through AT such as screen readers and on-screen keyboards. In the last blog , I talked about some high-level concepts on UI Automation (UIA). In this walkthrough, I’ll give a snippet on how to use UIA on Notepad.  You can write your script using C++, C#, or Python.   Microsoft uses C++ and C# for their UI Automation and all kinds of testing.  However, most of the develope...

Fig: Pipeline for Accessibility API UI Automation (UIA)  is an ordered tree of automation elements with a single interface for navigating the tree. This framework enables a portal to the user by providing and interpreting programmatic information from windows applications about user interfaces (UI). It helps to make products more  accessible  for everyone including people with disabilities all over the world. This not only offers programmatic access to most of the UI components but also allows assistive technology-based applications, such as screen readers, to provide users with knowledge about the UI and to control the UI. For the assistive technologies, UI Automation ensures that technology has access to important information about what is taking place in a particular user interface such as how a system will react if A button is pressed A list is opened A menu selected A combo box expanded. Assistive technology-based applications, such as screen readers or speech recogn...