Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 1 | # Chrome Accessibility on Android |
| 2 | |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 3 | This document covers some of the technical details of how Chrome |
| 4 | implements its accessibility support on Android. |
| 5 | |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 6 | Chrome plays an important role on Android - not only is it the default |
| 7 | browser, but Chrome powers WebView, which is used by many built-in and |
| 8 | third-party apps to display all sorts of content. Android includes a lightweight |
| 9 | implementation called Chrome Custom Tabs, which is also powered by Chrome. |
| 10 | All of these implementations must be accessible, and the Chrome & Chrome OS Accessibility |
| 11 | team provides the support to make these accessibility through the Android API. |
| 12 | |
| 13 | Accessibility on Android is heavily used. There are many apps that hijack the |
| 14 | Android accessibility API to act on the user's behalf (e.g. password managers, |
| 15 | screen clickers, anti-virus software, etc). Because of this, roughly **16%** of all |
| 16 | Android users are running the accessibility code (even if they do not realize it). |
| 17 | |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 18 | As background reading, you should be familiar with |
Kevin Babbitt | 805b781 | 2021-06-14 18:18:16 | [diff] [blame] | 19 | [Android Accessibility](https://developer.android.com/guide/topics/ui/accessibility) |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 20 | and in particular |
Kevin Babbitt | 805b781 | 2021-06-14 18:18:16 | [diff] [blame] | 21 | [AccessibilityNodeInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo) |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 22 | objects, [AccessibilityEvent](https://developer.android.com/reference/android/view/accessibility/AccessibilityEvent) and |
Kevin Babbitt | 805b781 | 2021-06-14 18:18:16 | [diff] [blame] | 23 | [AccessibilityNodeProvider](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeProvider). |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 24 | |
| 25 | ## WebContentsAccessibility |
| 26 | |
| 27 | The main Java class that implements the accessibility protocol in Chrome is |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 28 | [WebContentsAccessibilityImpl.java](https://cs.chromium.org/chromium/src/content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java). This |
| 29 | class acts as the AccessibilityNodeProvider (see above) for a given tab, and will |
| 30 | provide the virtual tree hierarchy, preform actions on the user's behalf, and |
| 31 | send events to downstream services for changes in the web contents. |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 32 | |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 33 | This class differs in a few key ways from other platforms. First, it represents |
| 34 | and entire web page, including all frames. The ids in the java code are unique IDs, |
| 35 | not frame-local IDs. They are typically referred to as `virtualViewId` in the code |
| 36 | and Android framework/documentation. Another key difference is the construction of |
| 37 | the native objects for nodes. On most platforms, we create a native object for every |
| 38 | AXNode in a web page, and we implement a bunch of methods on that object that assistive |
| 39 | technology can query. Android is different - it's more lightweight in one way, in that we only |
| 40 | create a native AccessibilityNodeInfo object when specifically requested, when |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 41 | an Android accessibility service is exploring the virtual tree. In another |
| 42 | sense it's more heavyweight, though, because every time a virtual view is |
| 43 | requested we have to populate it with every possible accessibility attribute, |
| 44 | and there are quite a few. |
| 45 | |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 46 | ### WebContentsAccessibilityImpl is "lazy" and "on demand" |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 47 | |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 48 | Every Tab in the Chrome Android browser will have its own instance of |
| 49 | WebContentsAccessibilityImpl. The WebContentsAccessibilityImpl object is created |
| 50 | using a static Factory method with a parameter of the WebContents object for |
| 51 | that tab. See [constructor](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22protected%20WebContentsAccessibilityImpl%22). |
| 52 | (Note: There are a few exceptions to this pattern, for example when constructing |
| 53 | the object without access to a WebContents instance, such as in the case of PaintPreview.) |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 54 | |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 55 | Although the WebContentsAccessibilityImpl object has been constructed (and |
| 56 | technically instantiated), it will not do anything until an accessibility service |
| 57 | is active and queries the system. The base class of Java widgets [View.java](https://developer.android.com/reference/android/view/View), |
| 58 | has a method [getAccessibilityNodeProvider](https://developer.android.com/reference/android/view/View#getAccessibilityNodeProvider\(\)). Custom views, such as the [ContentView.java](https://source.chromium.org/chromium/chromium/src/+/main:components/embedder_support/android/java/src/org/chromium/components/embedder_support/view/ContentView.java) |
| 59 | class in Chrome (which holds all the web contents for a single Tab), can override |
| 60 | this method to return an instance of AccessibilityNodeProvider. If an app returns |
| 61 | its own instance of AccessibilityNodeProvider, then AT will leverage this instance |
| 62 | when querying the view hierarchy. The WebContentsAccessibilityImpl acts as Chrome's |
| 63 | custom instance of AccessibilityNodeProvider so that it can serve the virtual view |
| 64 | hierarchy of the web to the native accessibility framework. |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 65 | |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 66 | The first time that `getAccessibilityNodeProvider` is called by the Android system, |
| 67 | the WebContentsAccessibilityImpl will be initialized. This is why we consider it |
| 68 | "lazy" and "on demand", because although it has technically been constructed and |
| 69 | instantiated, it does not perform any actions until AT triggered its initialization. |
| 70 | See [WebContentsAccessibilityImpl#getAccessibilityNodeProvider](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22public%20AccessibilityNodeProvider%20getAccessibilityNodeProvider%22) and the |
| 71 | associated [onNativeInit](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22protected%20void%20onNativeInit%22) methods. The getAccessibilityNodeProvider method |
| 72 | will only be called when an accessibility service is enabled, and so by lazily |
| 73 | constructing only after this call, we ensure that the accessibility code is not |
| 74 | being leveraged for users without any services enabled. |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 75 | |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 76 | Once initialized, the WebContentAccessibilityImpl plays a part in handling nearly |
| 77 | all accessibility related code on Android. This object will be where AT sends |
| 78 | actions performed by users, it constructs and serves the virtual view hierarchy, |
| 79 | and it dispatches events to AT for changes in the web contents. The |
| 80 | WebContentsAccessibilityImpl object has the same lifecycle as the Tab for which |
| 81 | it was created, and although it won't fire events or serve anything to downstream |
| 82 | AT if the tab is backgrounded/hidden, the object will continue to exist and will |
| 83 | not be destroyed until the Tab is destroyed/closed. |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 84 | |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 85 | ## AccessibilityNodeInfo |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 86 | |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 87 | The [AccessibilityNodeInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo) |
| 88 | object is at the core of Android accessibility. This is a rather heavy object |
| 89 | which holds every attribute for a given node (a virtual view element) as defined |
| 90 | by the Android API. (Note: The Android accessibiltiy API has different attributes/standards |
| 91 | than the web or other platforms, so there are many special cases and considerations, |
| 92 | more on that below). |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 93 | |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 94 | As an AccessibilityNodeProvider, the WebContentsAccessibilityImpl class must |
| 95 | override/implement the [createAccessibilityNodeInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeProvider#createAccessibilityNodeInfo\(int\)) method. This is the |
| 96 | method that the accessibility framework will query on behalf of AT to understand |
| 97 | the current virtual view hierarchy. On other platforms, the native-side code may |
| 98 | contain the entire structure of the web contents in native objects, but on Android |
| 99 | the objects are created "on demand" as requested by the framework, and so they are |
| 100 | typically generated synchronously on-the-fly. |
Dominic Mazzoni | 0712ad5 | 2019-10-23 19:33:42 | [diff] [blame] | 101 | |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 102 | The information to populate the AccessibilityNodeInfo objects is contained in |
| 103 | the accessibility tree in the C++ code in the shared BrowserAccessibilityManager. |
| 104 | For Android there is the usual BrowserAccessibilityManagerAndroid, and the |
| 105 | BrowserAccessibilityAndroid classes, as expected, but there is also an additional |
| 106 | [web\_contents\_accessibility\_android.cc](https://cs.chromium.org/chromium/src/content/browser/accessibility/web_contents_accessibility_android.cc) class. |
| 107 | This class is what allows us to connect the Java-side WebContentsAccessibilityImpl |
| 108 | with the C++ side manager, through the Java Native Interface (JNI). |
| 109 | |
| 110 | When [WebContentsAccessibilityImpl#createAccessibilityNodeInfo](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22public%20AccessibilityNodeInfoCompat%20createAccessibilityNodeInfo%22) is called for |
| 111 | a given virtual view (web node), the WebContentsAccessibilityImpl object calls into the native |
| 112 | C++ code through JNI, connecting to `web_contents_accessibility_android.cc`. The |
| 113 | web\_contents\_accessibility\_android object in turn compiles information about the |
| 114 | requested node from BrowserAccessibilityAndroid and BrowserAccessibilityManagerAndroid |
| 115 | and then calls back into WebContentsAccessibilityImpl, again through the JNI, to |
| 116 | send this information back to the Java-side to be populated into the |
| 117 | `AccessibilityNodeInfo` object that is being constructed. |
| 118 | |
| 119 | These roundtrips across the JNI come with an inherent cost. It is minuscule, but for |
| 120 | thousands of nodes on a page, each with 25+ attributes, it would be too costly to |
| 121 | make so many trips. However, passing all the attributes in one giant function call |
| 122 | is also not ideal. We try to strike a balance by grouping like attributes together |
| 123 | (e.g. all boolean attributes) into a single JNI trip, and make just a few JNI |
| 124 | trips per AccessibilityNodeInfo object. These trips can be found in the |
| 125 | [WebContentsAccessibilityAndroid::PopulateAccessibilityNodeInfo](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/web_contents_accessibility_android.cc?q=%22WebContentsAccessibilityAndroid::PopulateAccessibilityNodeInfo%22) method, |
| 126 | which is called from WebContentsAccessibilityImpl#createAccessibilityNodeInfo, and |
| 127 | is the core method for compiling a node's attributes and passing them to the Java-side. |
| 128 | |
| 129 | ### Java-side caching mechanism |
| 130 | |
| 131 | One of the most significant performance optimizations in the Android accessibility |
| 132 | code is the addition of a caching mechanism for the Java-side AccessibilityNodeInfo |
| 133 | objects. The cache is built as a simple `SparseArray` of AccessibilityNodeInfo objects. |
| 134 | We use a SparseArray instead of a HashMap because on Java the HashMap requires |
| 135 | Objects for both the key and value, and ideally we would use the `virtualViewId` |
| 136 | of any given node as the key, and this ID is an int (primitive type) in Java. So the |
| 137 | SparseArray is more light weight and is as efficient as using the HashMap in this |
| 138 | case. The array contains AccessibilityNodeInfo objects at the index of the node's |
| 139 | corresponding `virtualViewId`. If an invalid ID is requested, `null` is returned. |
| 140 | |
| 141 | In WebContentsAccessibilityImpl's implementation of createAccessibilityNodeInfo, |
| 142 | the cache is queried first, and if it contains a cached version of an object for |
| 143 | this node, then we update that reference and return it. Otherwise the object is |
| 144 | created anew and added to the cache before returning. The rough outline of the |
| 145 | code is: |
| 146 | |
| 147 | ``` |
| 148 | private SparseArray<AccessibilityNodeInfoCompat> mNodeInfoCache = new SparseArray<>(); |
| 149 | |
| 150 | @Override |
| 151 | public AccessibilityNodeInfoCompat createAccessibilityNodeInfo(int virtualViewId) { |
| 152 | if (mNodeInfoCache.get(virtualViewId) != null) { |
| 153 | cachedNode = mNodeInfoCache.get(virtualViewId); |
| 154 | // ... update cached node through JNI call ... |
| 155 | return cachedNode; |
| 156 | } else { |
| 157 | // Create new node from scratch |
| 158 | AccessibilityNodeInfo freshNode = // ... construct node through JNI call. |
| 159 | mNodeInfoCache.put(virtualViewId, freshNode); |
| 160 | return freshNode; |
| 161 | } |
| 162 | } |
| 163 | ``` |
| 164 | |
| 165 | When returning a cached node, there are some fields that we always update. This |
| 166 | requires a call through the JNI, but it still much more efficient than constructing |
| 167 | a full node from scratch. Rather than calling `PopulateAccessibilityNodeInfo`, we |
| 168 | call [WebContentsAccessibilityAndroid::UpdateCachedAccessibilityNodeInfo](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/web_contents_accessibility_android.cc?q=%22WebContentsAccessibilityAndroid::UpdateCachedAccessibilityNodeInfo%22). |
| 169 | This method updates the bounding box for the node so that AT knows where to draw |
| 170 | outlines if needed. (Note: it also technically updates RangeInfo on some nodes to |
| 171 | get around a bug in the Android framework, more on that below.) |
| 172 | |
| 173 | We clear nodes from the cache in [BrowserAccessibilityManager::OnNodeWillBeDeleted](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_manager_android.cc?q=%22BrowserAccessibilityManagerAndroid::OnNodeWillBeDeleted%22). |
| 174 | We also clear the parent node of any deleted node so that the AccessibilityNodeInfo |
| 175 | object will receive an updated list of children. We also clear any node that has a focus |
| 176 | change during [FireFocusEvent](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_manager_android.cc?q=%22BrowserAccessibilityManagerAndroid::FireFocusEvent%22). |
| 177 | |
| 178 | ### Bundle extras / API gaps |
| 179 | |
| 180 | Much of the richness of the web cannot be captured by the Android accessibility API, |
| 181 | which is designed from a native (Java) widget perspective. When there is a piece of |
| 182 | information that an AT would like to have access to, but there is no way to include |
| 183 | it through the standard API, we put that info in the AccessibilityNodeInfo's |
| 184 | [Bundle](https://developer.android.com/reference/android/os/Bundle). |
| 185 | |
| 186 | The Bundle, accessed through `getExtras()` is a map of key\-value pairs which can |
| 187 | hold any arbitrary data we would like. Some examples of extra information for a node: |
| 188 | |
| 189 | - Chrome role |
| 190 | - roleDescription |
| 191 | - targetUrl |
| 192 | |
| 193 | We also include information unique to Android, such as a "clickableScore", which is |
| 194 | a rating of how likely an object is to be clickable. The boolean "offscreen" is |
| 195 | used to denote if an element is "visible" to the user, but off screen (see more below). |
| 196 | We include unclipped bounds to give the true bounding boxes of a node if we were |
| 197 | not clipping them to be only onscreen. The full list can be seen in the |
| 198 | [list of constants](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22Constants%20defined%20for%20AccessibilityNodeInfo%20Bundle%20extras%20keys%22) |
| 199 | at the top of WebContentsAccessibilityImpl.java. |
| 200 | |
| 201 | ### Asynchronously adding "heavy" data |
| 202 | |
| 203 | Sometimes apps and downstream services will request we add additional information |
| 204 | to the AccessibilityNodeInfo objects that is too computationally heavy to compute |
| 205 | and include for every node. For these cases, the Android API has a method that |
| 206 | can be called by AT, [addExtraDataToAccessibilityNodeInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeProvider#addExtraDataToAccessibilityNodeInfo\(int,%20android.view.accessibility.AccessibilityNodeInfo,%20java.lang.String,%20android.os.Bundle\)). The method is |
| 207 | part of the AccessibilityNodeProvider, and so WebContentsAccessibilityImpl has |
| 208 | its [own implementation](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22public%20void%20addExtraDataToAccessibilityNodeInfo%22) of this for Chrome. When called with valid arguments, |
| 209 | this will start an asynchronous process to add this extra data to the given |
| 210 | AccessibilityNodeInfo object. The two current implementations of this are to add |
| 211 | text character locations, and raw image data. |
| 212 | |
| 213 | ### AccessibilityNodeInfoCompat |
| 214 | |
| 215 | The Android framework includes a [support library](https://developer.android.com/topic/libraries/support-library) for backwards compatibility. |
| 216 | This is a mechanism that allows Android to add new attributes or methods to their |
| 217 | API while also including backwards compatibility across previously released versions |
| 218 | of Android. The WebContentsAccessibilityImpl class makes heavy use of this to |
| 219 | ensure all features are supported on older versions of Android. This was a recent |
| 220 | change, and the rest of the Chrome code base still uses the non-Compat version of |
| 221 | the accessibility code because there is no use-case yet to switch. To make the |
| 222 | change as minimal as possible, the WebContentsAccessibilityImpl uses the Compat |
| 223 | version internally, but when communicating with other parts of Chrome, it will |
| 224 | unwrap the non-Compat object instead. For this entire document, whenever |
| 225 | AccessibilityNodeInfo is mentioned, technically speaking we are using an |
| 226 | [AccessibilityNodeInfoCompat](https://developer.android.com/reference/androidx/core/view/accessibility/AccessibilityNodeInfoCompat) object, and we use the |
| 227 | [AccessibilityNodeProviderCompat](https://developer.android.com/reference/androidx/core/view/accessibility/AccessibilityNodeProviderCompat) version as well. This library is designed to |
| 228 | be transparent to the end-user though, and for simplicity we generally do not |
| 229 | include the word 'Compat' in documentation, conversation, etc. |
| 230 | |
| 231 | ## Responding to user actions |
| 232 | |
| 233 | As the AccessibilityNodeProvider, the WebContentsAccessibilityImpl is responsible for |
| 234 | responding to user actions that come from downstream AT. It is also responsible |
| 235 | for telling downstream AT of Events coming from the web contents, to ensure that |
| 236 | AT is aware of any changes to the web contents state. |
| 237 | |
| 238 | ### performAction |
| 239 | |
| 240 | One of the most important methods in WebContentsAccessibilityImpl is the |
| 241 | implementation of [performAction](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeProvider#performAction\(int,%20int,%20android.os.Bundle\)). |
| 242 | This method is called by the Android framework on behalf of downstream AT for |
| 243 | any user actions, such as focus changes, clicks, scrolls, etc. For most actions, |
| 244 | we call through the JNI to web\_contents\_accessibility\_android, which will call a |
| 245 | corresponding method in BrowserAccessibilityManager to send this request to the |
| 246 | underlying accessibility code. The performAction method parameters include a |
| 247 | virtualViewId, the action, and a Bundle of args/extras. The Bundle may be null, |
| 248 | but sometimes carries necessary information, such as text that a user is trying |
| 249 | to paste into a field. If the accessibility code is able to successfully act on |
| 250 | this performAction call, it will return `true` to the framework, otherwise `false`. |
| 251 | |
| 252 | ### AccessibilityEventDispatcher |
| 253 | |
| 254 | The other direction of communication is from the accessibility code into the framework |
| 255 | and downstream AT. For this we dispatch an [AccessibilityEvent](https://developer.android.com/reference/android/view/accessibility/AccessibilityEvent). |
| 256 | Often times a call to performAction is paired with one or more AccessibilityEvents |
| 257 | being dispatched in response, but AccessibilityEvents can also be sent without |
| 258 | any user interaction, but instead from updates in the web contents. The AccessibilityEvents |
| 259 | are relatively lightweight, and they are constructed following the same model as |
| 260 | the AccessibilityNodeInfo objects (i.e. calling into web\_contents\_accessibility\_android and |
| 261 | being populated through a series of JNI calls). However, the events can put significant |
| 262 | strain on downstream AT, and this is where another important performance optimization |
| 263 | was added. |
| 264 | |
| 265 | Traditionally an app would realize it needs to generate and send an AccessibilityEvent, |
| 266 | it would generate it synchronously and send it on the main thread. The web is more |
| 267 | complicated though, and at times could be generating so many events that downstream |
| 268 | AT is strained. To alleviate this, we have implemented a throttling mechanism, |
| 269 | the [AccessibilityEventDispatcher](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/AccessibilityEventDispatcher.java). |
| 270 | Whenever WebContentsAccessibilityImpl requests to send an AccessibilityEvent, it |
| 271 | first goes through this dispatcher. For most events, they are immediately passed |
| 272 | along to the Android framework (e.g. user clicks, focus changes). Some events are |
| 273 | instead put in a queue and dispatched or dropped based on future user actions. |
| 274 | |
| 275 | For events that we wish to throttle, we will immediately send the first event received |
| 276 | of that type. We record the system time of this event. If another event request of |
| 277 | the same type is received within a set time limit, which we call the `throttleDelay`, |
| 278 | then we will wait until to send that event until after the delay. |
| 279 | |
| 280 | For example, scroll events can only be sent at most once every 100ms, otherwise we would attempt to send |
| 281 | them on every frame. Consider we have sent a scroll event and recorded the system |
| 282 | time to start a throttle delay. If we try to dispatch another scroll event in that delay |
| 283 | window, it will be queued to release after the delay. If during that throttleDelay period another scroll event |
| 284 | is added to the queue, it will replace the previous event and only the last one added |
| 285 | in the 100ms window is dispatched. The timer would then restart for another 100ms. |
| 286 | |
| 287 | The delay times can be specific to a view, or specific to an event. That is, we could |
| 288 | say "only dispatch scroll events every 100ms for any given view", meaning two different nodes |
| 289 | could send scroll events in close succession. Or we can say "only dispatch scroll |
| 290 | events every 100ms, regardless of view", in which case all views trying to send that |
| 291 | event type will enter the same queue. |
| 292 | |
| 293 | The event types and their delays can be found [in the constructor](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22Define%20our%20delays%20on%20a%20per%20event%20type%20basis%22) |
| 294 | of the WebContentsAccessibilityImpl, which also includes the construction of the |
| 295 | AccessibilityEventDispatcher. |
| 296 | |
| 297 | ### Delayed AccessibilityEvent construction |
| 298 | |
| 299 | The final performance optimization related to events (that is released on 100% stable), |
| 300 | was to delay the construction of the AccessibilityEvent until it is about to be |
| 301 | dispatched. Implementing the Dispatcher helps a significant amount, but there are |
| 302 | many events that can be dropped, and there is no reason to construct an event until |
| 303 | we are sure it will be dispatched. So we do not construct the AccessibilityEvents |
| 304 | until the moment the Dispatcher has started the request to send the event to the |
| 305 | Android framework. |
| 306 | |
| 307 | ## Testing on Android |
| 308 | |
| 309 | Testing on Android happens through a couple of build targets depending on what it |
| 310 | is that we want to test. Android has tests present in the **content\_browsertests** |
| 311 | target, same as the other platforms, which tests the BrowserAccessibilityManagerAndroid |
| 312 | and BrowserAccessibilityAndroid through the various DumpAccessibilityTreeTests |
| 313 | and DumpAccessibilityEventsTests. However, these tests do not cover the |
| 314 | web\_contents\_accessibility\_android layer, or any of the Java-side code. The |
| 315 | web\_contents\_accessibility\_android object and the associated WebContentsAccessibilityImpl |
| 316 | object are not created for content\_browsertests and require a full browser instance |
| 317 | to be available (or at least the content shell). To handle these types of tests |
| 318 | we must use the **content\_shell\_test\_apk** target, which will run an instance of a |
| 319 | web contents and allow the creation/execution of WebContentsAccessibilityImpl and |
| 320 | the corresponding native object. And finally there is the **chrome\_public\_test\_apk**, |
| 321 | which is used to test the Chrome Android UI, outside the web contents, which is |
| 322 | necessary for testing accessibility features that have a user-facing Android UI, such as |
| 323 | image descriptions, the accessibility settings pages, and page zoom. |
| 324 | |
| 325 | ### Testing the "missing layer" |
| 326 | |
| 327 | The "missing layer" in testing refers to the gap in testing for WebContentsAccessibilityImpl, |
| 328 | and namely web\_contents\_accessibility\_android mentioned above. There are three main |
| 329 | classes we use to test these. They are: |
| 330 | |
| 331 | - WebContentsAccessibilityTest |
| 332 | |
| 333 | This test suite is used to test the methods of WebContentsAccessibilityImpl.java. It tests |
| 334 | the various actions of performAction, construction of AccessibilityEvents, and |
| 335 | various helper methods we use throughout the code. |
| 336 | |
| 337 | - WebContentsAccessibilityTreeTest |
| 338 | |
| 339 | This class is the Java-side equivalent of the DumpAccessibilityTreeTests. This test suite |
| 340 | opens a given html file (shared with the content\_browsertests), generates an |
| 341 | AccessibilityNodeInfo tree for the page, and then dumps this tree and compares with |
| 342 | an expectation file (denoted with the `...-expected-android-external.txt` suffix). We continue |
| 343 | to keep around the content\_browsertests because a failure in one and not the other |
| 344 | would provide insight into a potential bug location. |
| 345 | |
| 346 | - WebContentsAccessibilityEventsTest |
| 347 | |
| 348 | This class is the Java-side equivalent of the DumpAccessibilityEventsTests. Same as the |
| 349 | suite above, it shares the same html files as the content\_browsertests, opens them, |
| 350 | runs the Javascript, and records the AccessibilityEvents that are dispatched to |
| 351 | downstream AT. There is no Android version of the DumpAccessibilityEventsTests though, |
| 352 | so these expectation files are suffixed with the usual `...-expected-android.txt`. |
| 353 | |
| 354 | When new tests are added for content_browsertests, the associated test should also |
| 355 | be added in WebContentsAccessibility\*Test, and there are PRESUBMIT warnings to |
| 356 | remind developers of this (although they are non-blocking). |
| 357 | |
| 358 | ### Writing new tests |
| 359 | |
| 360 | Adding tests is as easy on Android as it is on the other platforms because the |
| 361 | mechanism is in place and only a single new method needs to be added for the test. |
| 362 | |
| 363 | If you are adding a new events test, "example-test.html", you would |
| 364 | first create the html file as normal (content/test/data/accessibility/event/example-test.html), |
| 365 | and add the test to the existing `dump_accessibility_events_browsertests.cc`: |
| 366 | |
| 367 | ``` |
| 368 | IN_PROC_BROWSER_TEST_P(DumpAccessibilityEventsTest, AccessibilityEventsExampleTest) { |
| 369 | RunEventTest(FILE_PATH_LITERAL("example-test.html")); |
| 370 | } |
| 371 | ``` |
| 372 | |
| 373 | To include this test on Android, you would add a similar block to the |
| 374 | `WebContentsAccessibilityEventsTest.java` class: |
| 375 | |
| 376 | ``` |
| 377 | @Test |
| 378 | @SmallTest |
| 379 | public void test_exampleTest() { |
| 380 | performTest("example-test.html", "example-test-expected-android.txt"); |
| 381 | } |
| 382 | ``` |
| 383 | |
| 384 | Some tests on Android won't produce any events. For these you do not need to |
| 385 | create an empty file, but can instead make the test line: |
| 386 | |
| 387 | ``` |
| 388 | performTest("example-test.html", EMPTY_EXPECTATIONS_FILE); |
| 389 | ``` |
| 390 | |
| 391 | The easiest approach is to use the above line, run the tests, and if it fails, |
| 392 | the error message will give you the exact text to add to the |
| 393 | `-expected-android.txt` file. The `-expected-android.txt` file should go in the |
| 394 | same directory as the others (content/test/data/accessibility/event). |
| 395 | |
| 396 | For adding a new WebContentsAccessibilityTreeTest, you follow the same method but |
| 397 | include the function in the corresponding Java file. |
| 398 | |
| 399 | Note: Writing WebContentsAccessibilityTests is much more involved and there is no |
| 400 | general approach that can be encapsulated here, same with the UI tests. For those |
| 401 | there are many existing examples to reference, or you can reach out to an Android developer. |
| 402 | |
| 403 | ### Running tests and tracking down flakiness |
| 404 | |
| 405 | Running tests for Android can seem a bit daunting because it requires new build |
| 406 | targets, an emulator, and different command-line arguments and syntax. But Android |
| 407 | has a few nifty tricks that don't exist on every platform. For example, Android |
| 408 | tests can be run on repeat indefinitely, but set to break on their first failure. This |
| 409 | is great for tracking down flakiness. It is also possible to use a local repository |
| 410 | to test directly on the build bots, which is great when a test works locally but flakes |
| 411 | or fails during builds. Let's look at some basic examples. |
| 412 | |
| 413 | First ensure that you have followed the basic [Android setup](https://chromium.googlesource.com/chromium/src/+/master/docs/android_build_instructions.md) guides and can |
| 414 | successfully build the code. You should not proceed further until you can |
| 415 | successfully run the command: |
| 416 | |
| 417 | ``` |
| 418 | autoninja -C out/Debug chrome_apk |
| 419 | ``` |
| 420 | |
| 421 | One of the most important things to remember when building for unit tests is to use |
| 422 | the `x86` architecture, because most emulators use this. (Note: For running on try |
| 423 | bots however, you'll want `arm64`, more on that below). Your gn args should contain |
| 424 | at least: |
| 425 | |
| 426 | ``` |
| 427 | target_os = "android" |
| 428 | target_cpu = "x86" |
| 429 | ``` |
| 430 | |
| 431 | To run the types of tests mentioned above, you'll build with a command similar to: |
| 432 | |
| 433 | ``` |
| 434 | autoninja -C out/Debug content_shell_test_apk |
| 435 | ``` |
| 436 | |
| 437 | The filtering argument for tests is `-f` rather than the `--gtest_filter` that it is |
| 438 | used with content\_browsertests. So to run an example WebContentsAccessibilityTreeTest test, |
| 439 | you may use a command such as: |
| 440 | |
| 441 | ``` |
| 442 | out/Debug/bin/run_content_shell_test_apk --num_retries=0 -f "*WebContentsAccessibilityTreeTest*testExample" |
| 443 | ``` |
| 444 | |
| 445 | This would look for an x86 phone to deploy to, which should be your emulator. You can |
| 446 | choose to setup an emulator in Android Studio, or you can use some of the emulators |
| 447 | that come pre-built in the repo. [More information here](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/android_emulator.md). In general it is best to run on one of the |
| 448 | newer Android versions, but sometimes the newest is unstable. To specify an |
| 449 | emulator to use, you include the `--avd-config` argument, along with the desired |
| 450 | emulator (see link above for full list). This will run the test without opening a |
| 451 | window, but if you'd like to see an emulator window you can add the `--emulator-window` |
| 452 | argument. The `--repeat=#` argument allows repeats, and if set to `-1` along with |
| 453 | the `--break-on-failure` argument, the test will run repeatedly until it fails once. |
| 454 | |
| 455 | Putting this all together, to run the example test with no retries per run, run |
| 456 | repeatedly until failure, on the Android 11 emulator, with a window available, you would |
| 457 | use the command: |
| 458 | |
| 459 | ``` |
| 460 | out/Debug/bin/run_content_shell_test_apk \ |
| 461 | --num_retries=0 \ |
| 462 | --repeat=-1 \ |
| 463 | --break-on-failure \ |
| 464 | --emulator window \ |
| 465 | --avd-config tools/android/avd/proto/generic_android30.textpb \ |
| 466 | -f "*WebContentsAccessibilityTreeTest*testExample" |
| 467 | ``` |
| 468 | |
| 469 | All of this information also applies to the UI tests, which use the target: |
| 470 | |
| 471 | ``` |
| 472 | autoninja -C out/Debug chrome_public_test_apk |
| 473 | ``` |
| 474 | |
| 475 | In this case we would have a similar command to run an ImageDescriptions or Settings |
| 476 | related test: |
| 477 | |
| 478 | ``` |
| 479 | out/Debug/bin/run_chrome_public_test_apk \ |
| 480 | --num_retries=0 \ |
| 481 | --repeat=-1 \ |
| 482 | --break-on-failure \ |
| 483 | --emulator window \ |
| 484 | --avd-config tools/android/avd/proto/generic_android30.textpb \ |
| 485 | -f "*ImageDescriptions*" |
| 486 | ``` |
| 487 | |
| 488 | #### Running on try bots |
| 489 | |
| 490 | For more information you should reference the `mb.py` [user guide](https://source.chromium.org/chromium/chromium/src/+/main:tools/mb/docs/user_guide.md). |
| 491 | |
| 492 | Note: When running on the trybots, you often need to use `target_cpu = "arm64"`, since |
| 493 | these are actual devices and not emulators. |
| 494 | |
| 495 | It is not uncommon when working on the Android accessibility code to have a test that |
| 496 | works fine locally, but consistently fails on the try bots. These can be difficult |
| 497 | to debug, but if you run the test directly on the bots it is easier to gain insights. |
| 498 | This is done using the `mb.py` script. You should first build the target exactly as |
| 499 | outlined above (or mb.py will build it for you), and then you use the `mb.py run` command |
| 500 | to start a test. You provide a series of arguments to specify which properties you |
| 501 | want the try bot to have (e.g. which OS, architecture, etc), and you also can include |
| 502 | arguments for the test apk, same as above. Note: It is recommended to at least provide |
| 503 | the argument for the test filter to save time. |
| 504 | |
| 505 | With `mb.py`, you use `-s` to specify swarming, and `-d` to specify dimensions, which |
| 506 | narrow down the choice in try bot. The dimensions are added in the form: `-d dimension_name dimension_value`. |
| 507 | You should specify the `pool` as `chromium.tests`, the `device_os_type` as `userdebug`, |
| 508 | and the `device_os` for whichever Android version you're interested in (e.g. `M`, `N`, `O`, etc). |
| 509 | After specifying all your arguments to `mb.py`, include a `--`, and after this `--` |
| 510 | all further arguments are what is passed to the build target (e.g. content\_shell\_test\_apk). |
| 511 | |
| 512 | Putting this all together, to run the same tests as above, in the same way, but |
| 513 | on the Android M try bots, you would use the command: |
| 514 | |
| 515 | ``` |
| 516 | tools/mb/mb.py run -s --no-default-dimensions \ |
| 517 | -d pool chromium.tests \ |
| 518 | -d device_os_type userdebug \ |
| 519 | -d device_os M \ |
| 520 | out/Debug \ |
| 521 | content_shell_test_apk \ |
| 522 | -- \ |
| 523 | --num_retries=0 \ |
| 524 | --repeat=-1 \ |
| 525 | --break-on-failure \ |
| 526 | -f "*WebContentsAccessibilityTreeTest*testExample" |
| 527 | ``` |
| 528 | |
| 529 | Piece of cake! |
| 530 | |
| 531 | ## Common Android accessibility "gotchas" |
| 532 | |
| 533 | - "name" vs. "text" |
| 534 | |
| 535 | On other platforms, there is a concept of "name", and throughout the accessibility |
| 536 | code there are references to name. In the Android API, this is referred to as |
| 537 | "text" and is an attribute in the AccessibilityNodeInfo object. In another platform |
| 538 | you may `setName()` for a node, the equivalent on Android is `info.setText(text)`. |
| 539 | In the BrowserAccessibilityAndroid class, the relevant method that provides |
| 540 | this information is [GetTextContentUTF16](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=BrowserAccessibilityAndroid::GetTextContentUTF16). |
| 541 | |
| 542 | - ShouldExposeValueAsName |
| 543 | |
| 544 | The AccessibilityNodeInfo objects of Android do not have a concept of "value". |
| 545 | This makes some strange cases for nodes that have both a value and text or |
| 546 | label, and a challenge for how exactly to expose this data through the API. |
| 547 | The [ShouldExposeValueAsName](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=BrowserAccessibilityAndroid::ShouldExposeValueAsName) |
| 548 | method of BrowserAccessibilityAndroid returns a boolean of whether or not the |
| 549 | node's value should be returned for its name (i.e. text, see above). If this |
| 550 | value is false, then we concatenate the value with the node's text and |
| 551 | return this from GetTextContentUTF16. In the cases where ShouldExposeValueAsName |
| 552 | is true, we expose only the value in the text attribute, and use the "hint" |
| 553 | attribute of AccessibilityNodeInfo to expose the rest of the information (text, |
| 554 | label, description, placeholder). |
| 555 | |
| 556 | - stateDescription |
| 557 | |
| 558 | The [stateDescription](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo#setStateDescription\(java.lang.CharSequence\)) |
| 559 | attribute of the AccessibilityNodeInfo objects is a recent addition to the API |
| 560 | which allows custom text to be added for any node. This text is usually read at |
| 561 | the end of a node's announcement, and does not replace any other content and is |
| 562 | purely additional information. We make heavy use of the state description |
| 563 | to try and capture the richness of the web. For example, the Android API has a |
| 564 | concept of checkboxes being checked or unchecked, but it does not have the concept |
| 565 | of 'partially checked' as we have on the web (kMixed). When a checkbox is partially checked, |
| 566 | we include that information in the stateDescription attribute. For some nodes like |
| 567 | lists we include a stateDescription of the form "in list, item x of y". The full |
| 568 | list of stateDescriptions can be found in the BrowserAccessibilityAndroid method |
| 569 | [GetStateDescription](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=BrowserAccessibilityAndroid::GetStateDescription). |
| 570 | |
| 571 | - CollectionInfo and CollectionItemInfo |
| 572 | |
| 573 | The AccessibilityNodeInfo object has some child objects that do not always |
| 574 | need to be populated, for example [CollectionInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo.CollectionInfo) |
| 575 | and [CollectionItemInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo.CollectionItemInfo). |
| 576 | Collections work differently on Android than other platforms, namely that a given |
| 577 | node does not carry all necessary information to make a determination about the |
| 578 | overall collection. As the names might suggest, an item in a collection will have |
| 579 | the CollectionItemInfo populated, but not CollectionInfo, whereas the container |
| 580 | that holds all the items of the collection will have a CollectionInfo object but |
| 581 | not a CollectionItemInfo. When a CollectionItemInfo object is present, it is up |
| 582 | to the downstream AT to walk up the tree and gather information about the full |
| 583 | Collection. This information is not included on every node. |
| 584 | |
| 585 | These collections are used for any table-like node on |
| 586 | Android, such as lists, tables, grids, trees, etc. If a node is not table like or |
| 587 | an item of a table, then these child objects would be `null`. For this example |
| 588 | tree, the objects present for each node would be: |
| 589 | |
| 590 | ``` |
| 591 | kGenericContainer - CollectionInfo=null; CollectionItemInfo=null |
| 592 | kList - CollectionInfo=populated; CollectionItemInfo=null |
| 593 | ++kListItem - CollectionInfo=null; CollectionItemInfo=populated |
| 594 | ++kListItem - CollectionInfo=null; CollectionItemInfo=populated |
| 595 | ++kListItem - CollectionInfo=null; CollectionItemInfo=populated |
| 596 | ``` |
| 597 | |
| 598 | - contentInvalid |
| 599 | |
| 600 | The Android accessibility API has a boolean field isContentInvalid, however this |
| 601 | does not play well with downstream AT, so the Chrome code has some special |
| 602 | implementation details. The accessibility code reports a page exactly as it is, |
| 603 | so if a text field is labeled as contentInvalid, we report the same to all platforms. |
| 604 | There are use-cases where a field may be contentInvalid for each character typed |
| 605 | until a certain regex is met, e.g. when typing an email, empty or a few characters |
| 606 | would be reported as contentInvalid. When isContentInvalid is true on a node's |
| 607 | AccessibilityNodeInfo object, then AT (e.g. TalkBack) will proactively announce |
| 608 | "Error" or "Content Invalid", which can be jarring and unexpected for the user. |
| 609 | This announcement happens on any change, so every character typed would make |
| 610 | this announcement and give a bad user experience. It is the opinion of the Chrome |
| 611 | accessibility team that this ought to be fixed by TalkBack, and reporting an invalid |
| 612 | node as invalid is the pedantically correct approach. However, in the spirit of |
| 613 | giving the best user experience possible, we added two workarounds: |
| 614 | - The contentInvalid boolean is always false if the number of characters in |
| 615 | the field is less than [kMinimumCharacterCountForInvalid](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=kMinimumCharacterCountForInvalid), currently set to 7. This is done at the BrowserAccessibilityAndroid level. |
| 616 | - The WebContentsAccessibilityImpl includes a further workaround. contentInvalid |
| 617 | will only be reported for a currently focused node, and it will be reported |
| 618 | at most once every [CONTENT\_INVALID\_THROTTLE\_DELAY](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=CONTENT_INVALID_THROTTLE_DELAY) seconds, currently set to 4.5s. |
| 619 | See the [setAccessibilityNodeInfoBooleanAttributes](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=setAccessibilityNodeInfoBooleanAttributes) |
| 620 | method for the full implementation. |
| 621 | |
| 622 | - isVisibleToUser vs. "offscreen" |
| 623 | |
| 624 | The Android accessibility API includes only one boolean for setting whether or |
| 625 | not a node is "visible", the isVisibleToUser attribute. In general this conflicts with |
| 626 | the way the accessibility code treats [offscreen](https://source.chromium.org/chromium/chromium/src/+/main:docs/accessibility/browser/offscreen.md). |
| 627 | The name isVisibleToUser may suggest that it reflects whether or not the node |
| 628 | is currently visible to the user, but a more apt name would be: isPotentiallyVisibleToUser, |
| 629 | or isNotProgrammaticallyHidden. Nodes that are scrolled off the screen, and thus |
| 630 | not visible to the user, must still report true for isVisibleToUser. The main use-case |
| 631 | for this is for AT to allow navigation by element type. For example, if a user wants to |
| 632 | navigate by Headings, then an app like TalkBack will only navigate through nodes with a |
| 633 | true value for isVisibleToUser. If any node offscreen has isVisibleToUser as false, |
| 634 | then it would effectively remove this navigation option. So, the Chrome Android |
| 635 | accessibility code reports most nodes as isVisibleToUser, and if the node is actually |
| 636 | offscreen (not programmatically hidden but scrolled offscreen), then we include a |
| 637 | Bundle extra boolean, "offscreen" so that downstream AT can differentiate between |
| 638 | the nodes truly on/off screen. |
| 639 | |
| 640 | - RangeInfo, aria-valuetext, and caching |
| 641 | |
| 642 | The [RangeInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo.RangeInfo) |
| 643 | object is another child object of AccessibilityNodeInfo. Unfortunately this |
| 644 | object is rather limited in its options, and can only provide float values |
| 645 | for a min, max, and current value. There is no concept of a text description, or |
| 646 | steps or step size. This clashes with nodes such as sliders with an aria-valuetext, |
| 647 | or an indeterminate progress bar, for which we have to add special treatment. |
| 648 | As a further complication, AccessibilityEvents also require information on range |
| 649 | values when there is a change in value, however the event only allows integer |
| 650 | values between 0 and 100 (an integer percentage of the sliders position). |
| 651 | BrowserAccessibilityAndroid has a method [IsRangeControlWithoutAriaValueText](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=BrowserAccessibilityAndroid::IsRangeControlWithoutAriaValueText) |
| 652 | which we use to separate these cases when populating AccessibilityNodeInfo |
| 653 | objects and AccessibilityEvents (see web\_contents\_accessibility\_android.cc). |
| 654 | Similar to the Collection related objects above, RangeInfo is `null` for any |
| 655 | non-range related nodes. |
| 656 | |
| 657 | This RangeInfo object plays a small role in updating the cached AccessibilityNodeInfo |
| 658 | objects above. There is a small bug in the Android framework (which has been fixed |
| 659 | on newer versions) which breaks our caching mechanism for range objects. So the |
| 660 | `UpdateCachedAccessibilityNodeInfo` method also updates the RangeInfo object of a node |
| 661 | if it has one. |
| 662 | |
| 663 | - Leaf nodes and links |
| 664 | |
| 665 | Android has slightly different IsLeaf logic than other platforms, and this can |
| 666 | cause confusion, especially around links. On Android, links are **never** leafs. |
| 667 | See [IsLeaf](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=BrowserAccessibilityAndroid::IsLeaf). |
| 668 | This is for similar reasons to the isVisibleToUser section above. If a link |
| 669 | were a leaf node, and it were to contain something like a Heading, then AT would |
| 670 | not be able to traverse that link when navigating by headings because it would only |
| 671 | see it is a link. For this reason we always expose the entire child structure |
| 672 | of links. |
| 673 | |
| 674 | - Refocusing a node Java-side |
| 675 | |
| 676 | There is a strange bug in Android where objects that are accessibility focused |
| 677 | sometimes do not visually update their outline. This does not really block any |
| 678 | user flows per se, but we would ideally have the outlines drawn by AT like TalkBack |
| 679 | to reflect the correct bounds of the node. There is a simple way to get around |
| 680 | this bug, which is to remove focus from the node and refocus it again, which |
| 681 | triggers the underlying Android code necessary to update the bounds. In |
| 682 | WebContentsAccessibilityImpl we have a method |
| 683 | [moveAccessibilityFocusToIdAndRefocusIfNeeded](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22void%20moveAccessibilityFocusToIdAndRefocusIfNeeded%22) |
| 684 | which handles this. |
| 685 | |
| 686 | - liveRegions and forced announcements |
| 687 | |
| 688 | There is a boolean for liveRegion in the AccessibilityNodeInfo object that the |
| 689 | Chrome accessibility code will never set. This is because TalkBack will read |
| 690 | the entirety of the liveRegion node when there is a change. Instead we force |
| 691 | a custom announcement with an AccessibilityEvent in the WebContentsAccessibilityImpl's |
| 692 | [announceLiveRegionText](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22private%20void%20announceLiveRegionText%22) |
| 693 | method. |
| 694 | |
| 695 | - sendAccessibilityEvent vs. requestSendAccessibilityEvent |
| 696 | |
| 697 | In the Android framework, there are two methods for sending AccessibiltiyEvents, |
| 698 | [sendAccessibilityEvent](https://developer.android.com/reference/android/view/View#sendAccessibilityEvent\(int\)) and |
| 699 | [requestSendAccessibilityEvent](https://developer.android.com/reference/android/view/ViewParent#requestSendAccessibilityEvent\(android.view.View,%20android.view.accessibility.AccessibilityEvent\)). |
| 700 | Technically speaking, requestSendAccessibilityEvent will ask the system to |
| 701 | send an event, but it doesn't have to send it. For all intents and purposes, |
| 702 | we assume that it is always sent, but as a small detail to keep in mind, this |
| 703 | is not a guarantee. |
| 704 | |
| 705 | - TYPE\_WINDOW\_CONTENT\_CHANGED events |
| 706 | |
| 707 | The TYPE\_WINDOW\_CONTENT\_CHANGED type AccessibilityEvent is used to tell the |
| 708 | framework that something has changed for the given node. We send these events |
| 709 | for any change in the web, including scrolling. As a result, this is generally the |
| 710 | most frequently sent event, and we can often send too many and put strain on |
| 711 | downstream AT. We have included this event as part of our event throttling in |
| 712 | the AccessibilityEventDispatcher. We also include a small optimization for a |
| 713 | given atomic update. If an atomic update sends more than |
| 714 | [kMaxContentChangedEventsToFire](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/web_contents_accessibility_android.h?q=%22kMaxContentChangedEventsToFire%22) |
| 715 | events (currently set to 5), then any further events are dropped and a single |
| 716 | event on the root node is sent instead. This has proven useful for situations such |
| 717 | as many nodes being toggled visible at once. See [HandleContentChanged](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/web_contents_accessibility_android.cc?q=%22WebContentsAccessibilityAndroid::HandleContentChanged%22) |
| 718 | in web\_contents\_accessibility\_android.cc. |
| 719 | |
| 720 | - Text selection |
| 721 | |
| 722 | Static text selection exists in Android, but not when a service like TalkBack |
| 723 | is enabled. TalkBack allows for text selection inside an editable text field, |
| 724 | but not static text inside the web contents. Text selection is also tied to a |
| 725 | specific node with a start and end index. This means that we cannot select |
| 726 | text across multiple nodes. Ideally the implementation would allow a start |
| 727 | and end index on separate nodes, but this work is still in development. |
| 728 | |
| 729 | - Touch exploration |
| 730 | |
| 731 | The way touch exploration works on Android is complicated. The process that happens |
| 732 | during any hover event is: |
| 733 | |
Mark Schillaci | 03ce1bb | 2022-03-07 22:53:49 | [diff] [blame] | 734 | 1. User begins dragging their finger |
| 735 | 2. Java-side View receives a hover event and passes this through to C++ |
| 736 | 3. Accessibility code sends a hit test action to the renderer process |
| 737 | 4. The renderer process fires a HOVER accessibility event on the accessibility |
| 738 | node at that coordinate |
| 739 | 5. [WebContentsAccessibilityImpl#handleHover](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22private%20void%20handleHover%22) is called for that node. |
| 740 | 6. We fire a TYPE\_VIEW\_HOVER\_ENTER AccessibilityEvent on that node. |
| 741 | 7. We fire a TYPE\_VIEW\_HOVER\_EXIT AccessibilityEvent on the previous node. |
| 742 | 8. TalkBack sets accessibility focus to the targeted node. |
Mark Schillaci | ac3cdbd | 2022-03-04 23:40:08 | [diff] [blame] | 743 | |
| 744 | - WebView and Custom Tabs |
| 745 | |
| 746 | As mentioned at the start of this document, Chrome Android also plays an important |
| 747 | role by providing the WebView, which is used by any third party apps |
| 748 | that want to include web content in their app. Android also has a unique feature |
| 749 | of Custom Tabs, which is a lightweight implementation of Chrome that is somewhere |
| 750 | between a WebView and a full browser. The WebView and custom tabs must also be accessible, |
| 751 | and so most of this document applies to them as well. Occasionally there |
| 752 | will be an edge case or small bug that only happens in WebView, or a feature that |
| 753 | needs to be turned off only for WebView/Custom tab (e.g. image descriptions). There is a |
| 754 | WebView and Chrome Custom Tabs test app on the chrome-accessibility appspot page, |
| 755 | and there are methods on the Java-side that can give signals of whether the current |
| 756 | instance is a WebView or Chrome custom tab. [Example: isCustomTab](https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/tab/java/src/org/chromium/chrome/browser/tab/Tab.java?q=isCustomTab). |
| 757 | |
| 758 | ## Recent Progress and Features |
| 759 | |
| 760 | The Chrome Android accessibility code continues to evolve each quarter. We have |
| 761 | strengthened our testing and the stability of the code, but we also continue to |
| 762 | add new features and improvements. Beyond the usual bug fixes, below is a quick |
| 763 | summary of some features in the pipeline. |
| 764 | |
| 765 | ### OnDemand AT |
| 766 | |
| 767 | We have recently implemented a feature we refer to as "OnDemand AT" for short. |
| 768 | This feature is still rolling out and we intend to eventually have it enabled |
| 769 | on 100% stable by default. The feature modifies the AccessibilityEventDispatcher |
| 770 | that is explained above. If the feature is enabled, then WebContentsAccessibilityImpl |
| 771 | will query the Android system to determine the currently enabled accessibility |
| 772 | services, as well as the types of data they are interested in, namely the types of |
| 773 | AccessibilityEvents they want to know about. When the AccessibilityEventDispatcher |
| 774 | is sent an event to add to its queue or dispatch, if that event type is not in |
| 775 | the list of AccessibilityEvents relevant to currently enabled accessibility services, |
| 776 | the Dispatcher simply drops/ignores the request. Preliminary data shows that this |
| 777 | has created a noticeable improvement for accessibility services that do not require |
| 778 | the entire suite to function. |
| 779 | |
| 780 | ### ComputeAXMode |
| 781 | |
| 782 | Loosely related to the OnDemand feature above, the "ComputeAXMode" feature is also |
| 783 | a recent addition to improve overall performance. This feature uses the same |
| 784 | mechanism as OnDemand to query the currently enabled services and the information |
| 785 | they are interested in. ComputeAXMode then takes this information and uses a different |
| 786 | [AXMode](https://source.chromium.org/chromium/chromium/src/+/main:ui/accessibility/ax_mode.h) |
| 787 | based on the situation. This effectively does the same thing as |
| 788 | OnDemand, but further left/up-the-chain, giving a more significant performance |
| 789 | improvement. This feature is still rolling out, and it currently only has two |
| 790 | AXModes (full or basic). As it rolls out and we gather more data we will potentially |
| 791 | add more AXModes in the future. |
| 792 | |
| 793 | ### AutoDisableAccessibility |
| 794 | |
| 795 | The "AutoDisable" accessibility feature has also been ported to Android. This feature |
| 796 | tracks timing between user inputs and accessibility actions to make a determination |
| 797 | of whether or not accessibility services are still required. If they are no longer |
| 798 | needed by the user, then the accessibility code is disabled. Before this feature, |
| 799 | once the code was enabled it would continue to run for the life of the current |
| 800 | browser session. This feature is still being rolled out to stable. |
| 801 | |
| 802 | |
| 803 | ## Accessibility code in the ClankUI |
| 804 | |
| 805 | Most of this document is focused on the accessibility code and work as it relates |
| 806 | to the web contents, which is where the Chrome & Chrome OS Accessibility team |
| 807 | focuses most of its work. However, some features require a native UI in the browser |
| 808 | app, outside the web contents. When these features are added, the line between |
| 809 | the accessibility team and the Clank UI team becomes blurred. We traditionally |
| 810 | are the owners of this code, but seek regular guidance and approvals from the |
| 811 | Clank team as the front-end code must conform to the Clank standards. |
| 812 | |
| 813 | ### AccessibilitySettings |
| 814 | |
| 815 | The AccessibilitySettings page is found under the overflow/3-dot menu (Menu\>Settings\>Accessibility). |
| 816 | The page currently contains a slider to change the font scaling of the web contents, |
| 817 | options to force enable zoom, show simplified pages, enable image descriptions (see below), |
| 818 | and live captions. |
| 819 | |
| 820 | The main entry point for this code is [here](https://source.chromium.org/chromium/chromium/src/+/main:components/browser_ui/accessibility/android/java/src/org/chromium/components/browser_ui/accessibility/AccessibilitySettings.java). |
| 821 | The code leverages the PreferenceFragment of Android, and so much of the UI and |
| 822 | navigation is available out of the box, and the code is relatively simple in that |
| 823 | it only needs to respond to user actions/changes and pass this information to the |
| 824 | native C++ code. |
| 825 | |
| 826 | The settings code is heavily unit tested and stable, so it is rare to have to |
| 827 | work in this area. |
| 828 | |
| 829 | ### Image Descriptions |
| 830 | |
| 831 | The Clank-side code for the image descriptions feature is a bit more involved. |
| 832 | The image descriptions has to track state, determine whether or not to display the |
| 833 | option in the overflow menu, show dialogs, and provide toasts to the user. This |
| 834 | code is mostly controlled by the [ImageDescriptionsController](https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/image_descriptions/android/java/src/org/chromium/chrome/browser/image_descriptions/ImageDescriptionsController.java). |
| 835 | The image descriptions feature is written using Clank's 'component' model, and so |
| 836 | almost all the code exists in the directory: |
| 837 | [chrome/browser/image_descriptions](https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/image_descriptions/) |
| 838 | |
| 839 | (The exception being the few hooks that connect this code to the other parts of the |
| 840 | Clank UI). |
| 841 | |
| 842 | The image descriptions code is heavily unit tested and stable, so it is rare to |
| 843 | have to work in this area. |
| 844 | |
| 845 | ### (Upcoming) Page Zoom |
| 846 | |
| 847 | An upcoming feature is the page zoom feature, which will allow a more robust way |
| 848 | to zoom web contents than the currently existing text scaling of AccessibilitySettings |
| 849 | (which will be replaced). |
| 850 | |
| 851 | The Clank UI code for this feature has not been developed. More to come. |