blob: 225a295fdd08489d9617ae1d12bd5529dd8a5388 [file] [log] [blame] [view]
Dominic Mazzoni0712ad52019-10-23 19:33:421# Chrome Accessibility on Android
2
Dominic Mazzoni0712ad52019-10-23 19:33:423This document covers some of the technical details of how Chrome
4implements its accessibility support on Android.
5
Mark Schillaciac3cdbd2022-03-04 23:40:086Chrome plays an important role on Android - not only is it the default
7browser, but Chrome powers WebView, which is used by many built-in and
8third-party apps to display all sorts of content. Android includes a lightweight
9implementation called Chrome Custom Tabs, which is also powered by Chrome.
10All of these implementations must be accessible, and the Chrome & Chrome OS Accessibility
11team provides the support to make these accessibility through the Android API.
12
13Accessibility on Android is heavily used. There are many apps that hijack the
14Android accessibility API to act on the user's behalf (e.g. password managers,
15screen clickers, anti-virus software, etc). Because of this, roughly **16%** of all
16Android users are running the accessibility code (even if they do not realize it).
17
Dominic Mazzoni0712ad52019-10-23 19:33:4218As background reading, you should be familiar with
Kevin Babbitt805b7812021-06-14 18:18:1619[Android Accessibility](https://developer.android.com/guide/topics/ui/accessibility)
Dominic Mazzoni0712ad52019-10-23 19:33:4220and in particular
Kevin Babbitt805b7812021-06-14 18:18:1621[AccessibilityNodeInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo)
Mark Schillaciac3cdbd2022-03-04 23:40:0822objects, [AccessibilityEvent](https://developer.android.com/reference/android/view/accessibility/AccessibilityEvent) and
Kevin Babbitt805b7812021-06-14 18:18:1623[AccessibilityNodeProvider](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeProvider).
Dominic Mazzoni0712ad52019-10-23 19:33:4224
25## WebContentsAccessibility
26
27The main Java class that implements the accessibility protocol in Chrome is
Mark Schillaciac3cdbd2022-03-04 23:40:0828[WebContentsAccessibilityImpl.java](https://cs.chromium.org/chromium/src/content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java). This
29class acts as the AccessibilityNodeProvider (see above) for a given tab, and will
30provide the virtual tree hierarchy, preform actions on the user's behalf, and
31send events to downstream services for changes in the web contents.
Dominic Mazzoni0712ad52019-10-23 19:33:4232
Mark Schillaciac3cdbd2022-03-04 23:40:0833This class differs in a few key ways from other platforms. First, it represents
34and entire web page, including all frames. The ids in the java code are unique IDs,
35not frame-local IDs. They are typically referred to as `virtualViewId` in the code
36and Android framework/documentation. Another key difference is the construction of
37the native objects for nodes. On most platforms, we create a native object for every
38AXNode in a web page, and we implement a bunch of methods on that object that assistive
39technology can query. Android is different - it's more lightweight in one way, in that we only
40create a native AccessibilityNodeInfo object when specifically requested, when
Dominic Mazzoni0712ad52019-10-23 19:33:4241an Android accessibility service is exploring the virtual tree. In another
42sense it's more heavyweight, though, because every time a virtual view is
43requested we have to populate it with every possible accessibility attribute,
44and there are quite a few.
45
Mark Schillaciac3cdbd2022-03-04 23:40:0846### WebContentsAccessibilityImpl is "lazy" and "on demand"
Dominic Mazzoni0712ad52019-10-23 19:33:4247
Mark Schillaciac3cdbd2022-03-04 23:40:0848Every Tab in the Chrome Android browser will have its own instance of
49WebContentsAccessibilityImpl. The WebContentsAccessibilityImpl object is created
50using a static Factory method with a parameter of the WebContents object for
51that tab. See [constructor](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22protected%20WebContentsAccessibilityImpl%22).
52(Note: There are a few exceptions to this pattern, for example when constructing
53the object without access to a WebContents instance, such as in the case of PaintPreview.)
Dominic Mazzoni0712ad52019-10-23 19:33:4254
Mark Schillaciac3cdbd2022-03-04 23:40:0855Although the WebContentsAccessibilityImpl object has been constructed (and
56technically instantiated), it will not do anything until an accessibility service
57is active and queries the system. The base class of Java widgets [View.java](https://developer.android.com/reference/android/view/View),
58has a method [getAccessibilityNodeProvider](https://developer.android.com/reference/android/view/View#getAccessibilityNodeProvider\(\)). Custom views, such as the [ContentView.java](https://source.chromium.org/chromium/chromium/src/+/main:components/embedder_support/android/java/src/org/chromium/components/embedder_support/view/ContentView.java)
59class in Chrome (which holds all the web contents for a single Tab), can override
60this method to return an instance of AccessibilityNodeProvider. If an app returns
61its own instance of AccessibilityNodeProvider, then AT will leverage this instance
62when querying the view hierarchy. The WebContentsAccessibilityImpl acts as Chrome's
63custom instance of AccessibilityNodeProvider so that it can serve the virtual view
64hierarchy of the web to the native accessibility framework.
Dominic Mazzoni0712ad52019-10-23 19:33:4265
Mark Schillaciac3cdbd2022-03-04 23:40:0866The first time that `getAccessibilityNodeProvider` is called by the Android system,
67the WebContentsAccessibilityImpl will be initialized. This is why we consider it
68"lazy" and "on demand", because although it has technically been constructed and
69instantiated, it does not perform any actions until AT triggered its initialization.
70See [WebContentsAccessibilityImpl#getAccessibilityNodeProvider](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22public%20AccessibilityNodeProvider%20getAccessibilityNodeProvider%22) and the
71associated [onNativeInit](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22protected%20void%20onNativeInit%22) methods. The getAccessibilityNodeProvider method
72will only be called when an accessibility service is enabled, and so by lazily
73constructing only after this call, we ensure that the accessibility code is not
74being leveraged for users without any services enabled.
Dominic Mazzoni0712ad52019-10-23 19:33:4275
Mark Schillaciac3cdbd2022-03-04 23:40:0876Once initialized, the WebContentAccessibilityImpl plays a part in handling nearly
77all accessibility related code on Android. This object will be where AT sends
78actions performed by users, it constructs and serves the virtual view hierarchy,
79and it dispatches events to AT for changes in the web contents. The
80WebContentsAccessibilityImpl object has the same lifecycle as the Tab for which
81it was created, and although it won't fire events or serve anything to downstream
82AT if the tab is backgrounded/hidden, the object will continue to exist and will
83not be destroyed until the Tab is destroyed/closed.
Dominic Mazzoni0712ad52019-10-23 19:33:4284
Mark Schillaciac3cdbd2022-03-04 23:40:0885## AccessibilityNodeInfo
Dominic Mazzoni0712ad52019-10-23 19:33:4286
Mark Schillaciac3cdbd2022-03-04 23:40:0887The [AccessibilityNodeInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo)
88object is at the core of Android accessibility. This is a rather heavy object
89which holds every attribute for a given node (a virtual view element) as defined
90by the Android API. (Note: The Android accessibiltiy API has different attributes/standards
91than the web or other platforms, so there are many special cases and considerations,
92more on that below).
Dominic Mazzoni0712ad52019-10-23 19:33:4293
Mark Schillaciac3cdbd2022-03-04 23:40:0894As an AccessibilityNodeProvider, the WebContentsAccessibilityImpl class must
95override/implement the [createAccessibilityNodeInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeProvider#createAccessibilityNodeInfo\(int\)) method. This is the
96method that the accessibility framework will query on behalf of AT to understand
97the current virtual view hierarchy. On other platforms, the native-side code may
98contain the entire structure of the web contents in native objects, but on Android
99the objects are created "on demand" as requested by the framework, and so they are
100typically generated synchronously on-the-fly.
Dominic Mazzoni0712ad52019-10-23 19:33:42101
Mark Schillaciac3cdbd2022-03-04 23:40:08102The information to populate the AccessibilityNodeInfo objects is contained in
103the accessibility tree in the C++ code in the shared BrowserAccessibilityManager.
104For Android there is the usual BrowserAccessibilityManagerAndroid, and the
105BrowserAccessibilityAndroid classes, as expected, but there is also an additional
106[web\_contents\_accessibility\_android.cc](https://cs.chromium.org/chromium/src/content/browser/accessibility/web_contents_accessibility_android.cc) class.
107This class is what allows us to connect the Java-side WebContentsAccessibilityImpl
108with the C++ side manager, through the Java Native Interface (JNI).
109
110When [WebContentsAccessibilityImpl#createAccessibilityNodeInfo](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22public%20AccessibilityNodeInfoCompat%20createAccessibilityNodeInfo%22) is called for
111a given virtual view (web node), the WebContentsAccessibilityImpl object calls into the native
112C++ code through JNI, connecting to `web_contents_accessibility_android.cc`. The
113web\_contents\_accessibility\_android object in turn compiles information about the
114requested node from BrowserAccessibilityAndroid and BrowserAccessibilityManagerAndroid
115and then calls back into WebContentsAccessibilityImpl, again through the JNI, to
116send this information back to the Java-side to be populated into the
117`AccessibilityNodeInfo` object that is being constructed.
118
119These roundtrips across the JNI come with an inherent cost. It is minuscule, but for
120thousands of nodes on a page, each with 25+ attributes, it would be too costly to
121make so many trips. However, passing all the attributes in one giant function call
122is also not ideal. We try to strike a balance by grouping like attributes together
123(e.g. all boolean attributes) into a single JNI trip, and make just a few JNI
124trips per AccessibilityNodeInfo object. These trips can be found in the
125[WebContentsAccessibilityAndroid::PopulateAccessibilityNodeInfo](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/web_contents_accessibility_android.cc?q=%22WebContentsAccessibilityAndroid::PopulateAccessibilityNodeInfo%22) method,
126which is called from WebContentsAccessibilityImpl#createAccessibilityNodeInfo, and
127is the core method for compiling a node's attributes and passing them to the Java-side.
128
129### Java-side caching mechanism
130
131One of the most significant performance optimizations in the Android accessibility
132code is the addition of a caching mechanism for the Java-side AccessibilityNodeInfo
133objects. The cache is built as a simple `SparseArray` of AccessibilityNodeInfo objects.
134We use a SparseArray instead of a HashMap because on Java the HashMap requires
135Objects for both the key and value, and ideally we would use the `virtualViewId`
136of any given node as the key, and this ID is an int (primitive type) in Java. So the
137SparseArray is more light weight and is as efficient as using the HashMap in this
138case. The array contains AccessibilityNodeInfo objects at the index of the node's
139corresponding `virtualViewId`. If an invalid ID is requested, `null` is returned.
140
141In WebContentsAccessibilityImpl's implementation of createAccessibilityNodeInfo,
142the cache is queried first, and if it contains a cached version of an object for
143this node, then we update that reference and return it. Otherwise the object is
144created anew and added to the cache before returning. The rough outline of the
145code is:
146
147```
148private SparseArray<AccessibilityNodeInfoCompat> mNodeInfoCache = new SparseArray<>();
149
150@Override
151public AccessibilityNodeInfoCompat createAccessibilityNodeInfo(int virtualViewId) {
152 if (mNodeInfoCache.get(virtualViewId) != null) {
153 cachedNode = mNodeInfoCache.get(virtualViewId);
154 // ... update cached node through JNI call ...
155 return cachedNode;
156 } else {
157 // Create new node from scratch
158 AccessibilityNodeInfo freshNode = // ... construct node through JNI call.
159 mNodeInfoCache.put(virtualViewId, freshNode);
160 return freshNode;
161 }
162}
163```
164
165When returning a cached node, there are some fields that we always update. This
166requires a call through the JNI, but it still much more efficient than constructing
167a full node from scratch. Rather than calling `PopulateAccessibilityNodeInfo`, we
168call [WebContentsAccessibilityAndroid::UpdateCachedAccessibilityNodeInfo](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/web_contents_accessibility_android.cc?q=%22WebContentsAccessibilityAndroid::UpdateCachedAccessibilityNodeInfo%22).
169This method updates the bounding box for the node so that AT knows where to draw
170outlines if needed. (Note: it also technically updates RangeInfo on some nodes to
171get around a bug in the Android framework, more on that below.)
172
173We clear nodes from the cache in [BrowserAccessibilityManager::OnNodeWillBeDeleted](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_manager_android.cc?q=%22BrowserAccessibilityManagerAndroid::OnNodeWillBeDeleted%22).
174We also clear the parent node of any deleted node so that the AccessibilityNodeInfo
175object will receive an updated list of children. We also clear any node that has a focus
176change during [FireFocusEvent](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_manager_android.cc?q=%22BrowserAccessibilityManagerAndroid::FireFocusEvent%22).
177
178### Bundle extras / API gaps
179
180Much of the richness of the web cannot be captured by the Android accessibility API,
181which is designed from a native (Java) widget perspective. When there is a piece of
182information that an AT would like to have access to, but there is no way to include
183it through the standard API, we put that info in the AccessibilityNodeInfo's
184[Bundle](https://developer.android.com/reference/android/os/Bundle).
185
186The Bundle, accessed through `getExtras()` is a map of key\-value pairs which can
187hold any arbitrary data we would like. Some examples of extra information for a node:
188
189- Chrome role
190- roleDescription
191- targetUrl
192
193We also include information unique to Android, such as a "clickableScore", which is
194a rating of how likely an object is to be clickable. The boolean "offscreen" is
195used to denote if an element is "visible" to the user, but off screen (see more below).
196We include unclipped bounds to give the true bounding boxes of a node if we were
197not clipping them to be only onscreen. The full list can be seen in the
198[list of constants](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22Constants%20defined%20for%20AccessibilityNodeInfo%20Bundle%20extras%20keys%22)
199at the top of WebContentsAccessibilityImpl.java.
200
201### Asynchronously adding "heavy" data
202
203Sometimes apps and downstream services will request we add additional information
204to the AccessibilityNodeInfo objects that is too computationally heavy to compute
205and include for every node. For these cases, the Android API has a method that
206can be called by AT, [addExtraDataToAccessibilityNodeInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeProvider#addExtraDataToAccessibilityNodeInfo\(int,%20android.view.accessibility.AccessibilityNodeInfo,%20java.lang.String,%20android.os.Bundle\)). The method is
207part of the AccessibilityNodeProvider, and so WebContentsAccessibilityImpl has
208its [own implementation](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22public%20void%20addExtraDataToAccessibilityNodeInfo%22) of this for Chrome. When called with valid arguments,
209this will start an asynchronous process to add this extra data to the given
210AccessibilityNodeInfo object. The two current implementations of this are to add
211text character locations, and raw image data.
212
213### AccessibilityNodeInfoCompat
214
215The Android framework includes a [support library](https://developer.android.com/topic/libraries/support-library) for backwards compatibility.
216This is a mechanism that allows Android to add new attributes or methods to their
217API while also including backwards compatibility across previously released versions
218of Android. The WebContentsAccessibilityImpl class makes heavy use of this to
219ensure all features are supported on older versions of Android. This was a recent
220change, and the rest of the Chrome code base still uses the non-Compat version of
221the accessibility code because there is no use-case yet to switch. To make the
222change as minimal as possible, the WebContentsAccessibilityImpl uses the Compat
223version internally, but when communicating with other parts of Chrome, it will
224unwrap the non-Compat object instead. For this entire document, whenever
225AccessibilityNodeInfo is mentioned, technically speaking we are using an
226[AccessibilityNodeInfoCompat](https://developer.android.com/reference/androidx/core/view/accessibility/AccessibilityNodeInfoCompat) object, and we use the
227[AccessibilityNodeProviderCompat](https://developer.android.com/reference/androidx/core/view/accessibility/AccessibilityNodeProviderCompat) version as well. This library is designed to
228be transparent to the end-user though, and for simplicity we generally do not
229include the word 'Compat' in documentation, conversation, etc.
230
231## Responding to user actions
232
233As the AccessibilityNodeProvider, the WebContentsAccessibilityImpl is responsible for
234responding to user actions that come from downstream AT. It is also responsible
235for telling downstream AT of Events coming from the web contents, to ensure that
236AT is aware of any changes to the web contents state.
237
238### performAction
239
240One of the most important methods in WebContentsAccessibilityImpl is the
241implementation of [performAction](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeProvider#performAction\(int,%20int,%20android.os.Bundle\)).
242This method is called by the Android framework on behalf of downstream AT for
243any user actions, such as focus changes, clicks, scrolls, etc. For most actions,
244we call through the JNI to web\_contents\_accessibility\_android, which will call a
245corresponding method in BrowserAccessibilityManager to send this request to the
246underlying accessibility code. The performAction method parameters include a
247virtualViewId, the action, and a Bundle of args/extras. The Bundle may be null,
248but sometimes carries necessary information, such as text that a user is trying
249to paste into a field. If the accessibility code is able to successfully act on
250this performAction call, it will return `true` to the framework, otherwise `false`.
251
252### AccessibilityEventDispatcher
253
254The other direction of communication is from the accessibility code into the framework
255and downstream AT. For this we dispatch an [AccessibilityEvent](https://developer.android.com/reference/android/view/accessibility/AccessibilityEvent).
256Often times a call to performAction is paired with one or more AccessibilityEvents
257being dispatched in response, but AccessibilityEvents can also be sent without
258any user interaction, but instead from updates in the web contents. The AccessibilityEvents
259are relatively lightweight, and they are constructed following the same model as
260the AccessibilityNodeInfo objects (i.e. calling into web\_contents\_accessibility\_android and
261being populated through a series of JNI calls). However, the events can put significant
262strain on downstream AT, and this is where another important performance optimization
263was added.
264
265Traditionally an app would realize it needs to generate and send an AccessibilityEvent,
266it would generate it synchronously and send it on the main thread. The web is more
267complicated though, and at times could be generating so many events that downstream
268AT is strained. To alleviate this, we have implemented a throttling mechanism,
269the [AccessibilityEventDispatcher](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/AccessibilityEventDispatcher.java).
270Whenever WebContentsAccessibilityImpl requests to send an AccessibilityEvent, it
271first goes through this dispatcher. For most events, they are immediately passed
272along to the Android framework (e.g. user clicks, focus changes). Some events are
273instead put in a queue and dispatched or dropped based on future user actions.
274
275For events that we wish to throttle, we will immediately send the first event received
276of that type. We record the system time of this event. If another event request of
277the same type is received within a set time limit, which we call the `throttleDelay`,
278then we will wait until to send that event until after the delay.
279
280For example, scroll events can only be sent at most once every 100ms, otherwise we would attempt to send
281them on every frame. Consider we have sent a scroll event and recorded the system
282time to start a throttle delay. If we try to dispatch another scroll event in that delay
283window, it will be queued to release after the delay. If during that throttleDelay period another scroll event
284is added to the queue, it will replace the previous event and only the last one added
285in the 100ms window is dispatched. The timer would then restart for another 100ms.
286
287The delay times can be specific to a view, or specific to an event. That is, we could
288say "only dispatch scroll events every 100ms for any given view", meaning two different nodes
289could send scroll events in close succession. Or we can say "only dispatch scroll
290events every 100ms, regardless of view", in which case all views trying to send that
291event type will enter the same queue.
292
293The event types and their delays can be found [in the constructor](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22Define%20our%20delays%20on%20a%20per%20event%20type%20basis%22)
294of the WebContentsAccessibilityImpl, which also includes the construction of the
295AccessibilityEventDispatcher.
296
297### Delayed AccessibilityEvent construction
298
299The final performance optimization related to events (that is released on 100% stable),
300was to delay the construction of the AccessibilityEvent until it is about to be
301dispatched. Implementing the Dispatcher helps a significant amount, but there are
302many events that can be dropped, and there is no reason to construct an event until
303we are sure it will be dispatched. So we do not construct the AccessibilityEvents
304until the moment the Dispatcher has started the request to send the event to the
305Android framework.
306
307## Testing on Android
308
309Testing on Android happens through a couple of build targets depending on what it
310is that we want to test. Android has tests present in the **content\_browsertests**
311target, same as the other platforms, which tests the BrowserAccessibilityManagerAndroid
312and BrowserAccessibilityAndroid through the various DumpAccessibilityTreeTests
313and DumpAccessibilityEventsTests. However, these tests do not cover the
314web\_contents\_accessibility\_android layer, or any of the Java-side code. The
315web\_contents\_accessibility\_android object and the associated WebContentsAccessibilityImpl
316object are not created for content\_browsertests and require a full browser instance
317to be available (or at least the content shell). To handle these types of tests
318we must use the **content\_shell\_test\_apk** target, which will run an instance of a
319web contents and allow the creation/execution of WebContentsAccessibilityImpl and
320the corresponding native object. And finally there is the **chrome\_public\_test\_apk**,
321which is used to test the Chrome Android UI, outside the web contents, which is
322necessary for testing accessibility features that have a user-facing Android UI, such as
323image descriptions, the accessibility settings pages, and page zoom.
324
325### Testing the "missing layer"
326
327The "missing layer" in testing refers to the gap in testing for WebContentsAccessibilityImpl,
328and namely web\_contents\_accessibility\_android mentioned above. There are three main
329classes we use to test these. They are:
330
331- WebContentsAccessibilityTest
332
333 This test suite is used to test the methods of WebContentsAccessibilityImpl.java. It tests
334 the various actions of performAction, construction of AccessibilityEvents, and
335 various helper methods we use throughout the code.
336
337- WebContentsAccessibilityTreeTest
338
339 This class is the Java-side equivalent of the DumpAccessibilityTreeTests. This test suite
340 opens a given html file (shared with the content\_browsertests), generates an
341 AccessibilityNodeInfo tree for the page, and then dumps this tree and compares with
342 an expectation file (denoted with the `...-expected-android-external.txt` suffix). We continue
343 to keep around the content\_browsertests because a failure in one and not the other
344 would provide insight into a potential bug location.
345
346- WebContentsAccessibilityEventsTest
347
348 This class is the Java-side equivalent of the DumpAccessibilityEventsTests. Same as the
349 suite above, it shares the same html files as the content\_browsertests, opens them,
350 runs the Javascript, and records the AccessibilityEvents that are dispatched to
351 downstream AT. There is no Android version of the DumpAccessibilityEventsTests though,
352 so these expectation files are suffixed with the usual `...-expected-android.txt`.
353
354When new tests are added for content_browsertests, the associated test should also
355be added in WebContentsAccessibility\*Test, and there are PRESUBMIT warnings to
356remind developers of this (although they are non-blocking).
357
358### Writing new tests
359
360Adding tests is as easy on Android as it is on the other platforms because the
361mechanism is in place and only a single new method needs to be added for the test.
362
363If you are adding a new events test, "example-test.html", you would
364first create the html file as normal (content/test/data/accessibility/event/example-test.html),
365and add the test to the existing `dump_accessibility_events_browsertests.cc`:
366
367```
368IN_PROC_BROWSER_TEST_P(DumpAccessibilityEventsTest, AccessibilityEventsExampleTest) {
369 RunEventTest(FILE_PATH_LITERAL("example-test.html"));
370}
371```
372
373To include this test on Android, you would add a similar block to the
374`WebContentsAccessibilityEventsTest.java` class:
375
376```
377@Test
378@SmallTest
379public void test_exampleTest() {
380 performTest("example-test.html", "example-test-expected-android.txt");
381}
382```
383
384Some tests on Android won't produce any events. For these you do not need to
385create an empty file, but can instead make the test line:
386
387```
388 performTest("example-test.html", EMPTY_EXPECTATIONS_FILE);
389```
390
391The easiest approach is to use the above line, run the tests, and if it fails,
392the error message will give you the exact text to add to the
393`-expected-android.txt` file. The `-expected-android.txt` file should go in the
394same directory as the others (content/test/data/accessibility/event).
395
396For adding a new WebContentsAccessibilityTreeTest, you follow the same method but
397include the function in the corresponding Java file.
398
399Note: Writing WebContentsAccessibilityTests is much more involved and there is no
400general approach that can be encapsulated here, same with the UI tests. For those
401there are many existing examples to reference, or you can reach out to an Android developer.
402
403### Running tests and tracking down flakiness
404
405Running tests for Android can seem a bit daunting because it requires new build
406targets, an emulator, and different command-line arguments and syntax. But Android
407has a few nifty tricks that don't exist on every platform. For example, Android
408tests can be run on repeat indefinitely, but set to break on their first failure. This
409is great for tracking down flakiness. It is also possible to use a local repository
410to test directly on the build bots, which is great when a test works locally but flakes
411or fails during builds. Let's look at some basic examples.
412
413First ensure that you have followed the basic [Android setup](https://chromium.googlesource.com/chromium/src/+/master/docs/android_build_instructions.md) guides and can
414successfully build the code. You should not proceed further until you can
415successfully run the command:
416
417```
418autoninja -C out/Debug chrome_apk
419```
420
421One of the most important things to remember when building for unit tests is to use
422the `x86` architecture, because most emulators use this. (Note: For running on try
423bots however, you'll want `arm64`, more on that below). Your gn args should contain
424at least:
425
426```
427target_os = "android"
428target_cpu = "x86"
429```
430
431To run the types of tests mentioned above, you'll build with a command similar to:
432
433```
434autoninja -C out/Debug content_shell_test_apk
435```
436
437The filtering argument for tests is `-f` rather than the `--gtest_filter` that it is
438used with content\_browsertests. So to run an example WebContentsAccessibilityTreeTest test,
439you may use a command such as:
440
441```
442out/Debug/bin/run_content_shell_test_apk --num_retries=0 -f "*WebContentsAccessibilityTreeTest*testExample"
443```
444
445This would look for an x86 phone to deploy to, which should be your emulator. You can
446choose to setup an emulator in Android Studio, or you can use some of the emulators
447that come pre-built in the repo. [More information here](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/android_emulator.md). In general it is best to run on one of the
448newer Android versions, but sometimes the newest is unstable. To specify an
449emulator to use, you include the `--avd-config` argument, along with the desired
450emulator (see link above for full list). This will run the test without opening a
451window, but if you'd like to see an emulator window you can add the `--emulator-window`
452argument. The `--repeat=#` argument allows repeats, and if set to `-1` along with
453the `--break-on-failure` argument, the test will run repeatedly until it fails once.
454
455Putting this all together, to run the example test with no retries per run, run
456repeatedly until failure, on the Android 11 emulator, with a window available, you would
457use the command:
458
459```
460out/Debug/bin/run_content_shell_test_apk \
461 --num_retries=0 \
462 --repeat=-1 \
463 --break-on-failure \
464 --emulator window \
465 --avd-config tools/android/avd/proto/generic_android30.textpb \
466 -f "*WebContentsAccessibilityTreeTest*testExample"
467```
468
469All of this information also applies to the UI tests, which use the target:
470
471```
472autoninja -C out/Debug chrome_public_test_apk
473```
474
475In this case we would have a similar command to run an ImageDescriptions or Settings
476related test:
477
478```
479out/Debug/bin/run_chrome_public_test_apk \
480 --num_retries=0 \
481 --repeat=-1 \
482 --break-on-failure \
483 --emulator window \
484 --avd-config tools/android/avd/proto/generic_android30.textpb \
485 -f "*ImageDescriptions*"
486```
487
488#### Running on try bots
489
490For more information you should reference the `mb.py` [user guide](https://source.chromium.org/chromium/chromium/src/+/main:tools/mb/docs/user_guide.md).
491
492Note: When running on the trybots, you often need to use `target_cpu = "arm64"`, since
493these are actual devices and not emulators.
494
495It is not uncommon when working on the Android accessibility code to have a test that
496works fine locally, but consistently fails on the try bots. These can be difficult
497to debug, but if you run the test directly on the bots it is easier to gain insights.
498This is done using the `mb.py` script. You should first build the target exactly as
499outlined above (or mb.py will build it for you), and then you use the `mb.py run` command
500to start a test. You provide a series of arguments to specify which properties you
501want the try bot to have (e.g. which OS, architecture, etc), and you also can include
502arguments for the test apk, same as above. Note: It is recommended to at least provide
503the argument for the test filter to save time.
504
505With `mb.py`, you use `-s` to specify swarming, and `-d` to specify dimensions, which
506narrow down the choice in try bot. The dimensions are added in the form: `-d dimension_name dimension_value`.
507You should specify the `pool` as `chromium.tests`, the `device_os_type` as `userdebug`,
508and the `device_os` for whichever Android version you're interested in (e.g. `M`, `N`, `O`, etc).
509After specifying all your arguments to `mb.py`, include a `--`, and after this `--`
510all further arguments are what is passed to the build target (e.g. content\_shell\_test\_apk).
511
512Putting this all together, to run the same tests as above, in the same way, but
513on the Android M try bots, you would use the command:
514
515```
516tools/mb/mb.py run -s --no-default-dimensions \
517 -d pool chromium.tests \
518 -d device_os_type userdebug \
519 -d device_os M \
520 out/Debug \
521 content_shell_test_apk \
522 -- \
523 --num_retries=0 \
524 --repeat=-1 \
525 --break-on-failure \
526 -f "*WebContentsAccessibilityTreeTest*testExample"
527```
528
529Piece of cake!
530
531## Common Android accessibility "gotchas"
532
533- "name" vs. "text"
534
535 On other platforms, there is a concept of "name", and throughout the accessibility
536 code there are references to name. In the Android API, this is referred to as
537 "text" and is an attribute in the AccessibilityNodeInfo object. In another platform
538 you may `setName()` for a node, the equivalent on Android is `info.setText(text)`.
539 In the BrowserAccessibilityAndroid class, the relevant method that provides
540 this information is [GetTextContentUTF16](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=BrowserAccessibilityAndroid::GetTextContentUTF16).
541
542- ShouldExposeValueAsName
543
544 The AccessibilityNodeInfo objects of Android do not have a concept of "value".
545 This makes some strange cases for nodes that have both a value and text or
546 label, and a challenge for how exactly to expose this data through the API.
547 The [ShouldExposeValueAsName](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=BrowserAccessibilityAndroid::ShouldExposeValueAsName)
548 method of BrowserAccessibilityAndroid returns a boolean of whether or not the
549 node's value should be returned for its name (i.e. text, see above). If this
550 value is false, then we concatenate the value with the node's text and
551 return this from GetTextContentUTF16. In the cases where ShouldExposeValueAsName
552 is true, we expose only the value in the text attribute, and use the "hint"
553 attribute of AccessibilityNodeInfo to expose the rest of the information (text,
554 label, description, placeholder).
555
556- stateDescription
557
558 The [stateDescription](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo#setStateDescription\(java.lang.CharSequence\))
559 attribute of the AccessibilityNodeInfo objects is a recent addition to the API
560 which allows custom text to be added for any node. This text is usually read at
561 the end of a node's announcement, and does not replace any other content and is
562 purely additional information. We make heavy use of the state description
563 to try and capture the richness of the web. For example, the Android API has a
564 concept of checkboxes being checked or unchecked, but it does not have the concept
565 of 'partially checked' as we have on the web (kMixed). When a checkbox is partially checked,
566 we include that information in the stateDescription attribute. For some nodes like
567 lists we include a stateDescription of the form "in list, item x of y". The full
568 list of stateDescriptions can be found in the BrowserAccessibilityAndroid method
569 [GetStateDescription](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=BrowserAccessibilityAndroid::GetStateDescription).
570
571- CollectionInfo and CollectionItemInfo
572
573 The AccessibilityNodeInfo object has some child objects that do not always
574 need to be populated, for example [CollectionInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo.CollectionInfo)
575 and [CollectionItemInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo.CollectionItemInfo).
576 Collections work differently on Android than other platforms, namely that a given
577 node does not carry all necessary information to make a determination about the
578 overall collection. As the names might suggest, an item in a collection will have
579 the CollectionItemInfo populated, but not CollectionInfo, whereas the container
580 that holds all the items of the collection will have a CollectionInfo object but
581 not a CollectionItemInfo. When a CollectionItemInfo object is present, it is up
582 to the downstream AT to walk up the tree and gather information about the full
583 Collection. This information is not included on every node.
584
585 These collections are used for any table-like node on
586 Android, such as lists, tables, grids, trees, etc. If a node is not table like or
587 an item of a table, then these child objects would be `null`. For this example
588 tree, the objects present for each node would be:
589
590 ```
591 kGenericContainer - CollectionInfo=null; CollectionItemInfo=null
592 kList - CollectionInfo=populated; CollectionItemInfo=null
593 ++kListItem - CollectionInfo=null; CollectionItemInfo=populated
594 ++kListItem - CollectionInfo=null; CollectionItemInfo=populated
595 ++kListItem - CollectionInfo=null; CollectionItemInfo=populated
596 ```
597
598- contentInvalid
599
600 The Android accessibility API has a boolean field isContentInvalid, however this
601 does not play well with downstream AT, so the Chrome code has some special
602 implementation details. The accessibility code reports a page exactly as it is,
603 so if a text field is labeled as contentInvalid, we report the same to all platforms.
604 There are use-cases where a field may be contentInvalid for each character typed
605 until a certain regex is met, e.g. when typing an email, empty or a few characters
606 would be reported as contentInvalid. When isContentInvalid is true on a node's
607 AccessibilityNodeInfo object, then AT (e.g. TalkBack) will proactively announce
608 "Error" or "Content Invalid", which can be jarring and unexpected for the user.
609 This announcement happens on any change, so every character typed would make
610 this announcement and give a bad user experience. It is the opinion of the Chrome
611 accessibility team that this ought to be fixed by TalkBack, and reporting an invalid
612 node as invalid is the pedantically correct approach. However, in the spirit of
613 giving the best user experience possible, we added two workarounds:
614 - The contentInvalid boolean is always false if the number of characters in
615 the field is less than [kMinimumCharacterCountForInvalid](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=kMinimumCharacterCountForInvalid), currently set to 7. This is done at the BrowserAccessibilityAndroid level.
616 - The WebContentsAccessibilityImpl includes a further workaround. contentInvalid
617 will only be reported for a currently focused node, and it will be reported
618 at most once every [CONTENT\_INVALID\_THROTTLE\_DELAY](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=CONTENT_INVALID_THROTTLE_DELAY) seconds, currently set to 4.5s.
619 See the [setAccessibilityNodeInfoBooleanAttributes](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=setAccessibilityNodeInfoBooleanAttributes)
620 method for the full implementation.
621
622- isVisibleToUser vs. "offscreen"
623
624 The Android accessibility API includes only one boolean for setting whether or
625 not a node is "visible", the isVisibleToUser attribute. In general this conflicts with
626 the way the accessibility code treats [offscreen](https://source.chromium.org/chromium/chromium/src/+/main:docs/accessibility/browser/offscreen.md).
627 The name isVisibleToUser may suggest that it reflects whether or not the node
628 is currently visible to the user, but a more apt name would be: isPotentiallyVisibleToUser,
629 or isNotProgrammaticallyHidden. Nodes that are scrolled off the screen, and thus
630 not visible to the user, must still report true for isVisibleToUser. The main use-case
631 for this is for AT to allow navigation by element type. For example, if a user wants to
632 navigate by Headings, then an app like TalkBack will only navigate through nodes with a
633 true value for isVisibleToUser. If any node offscreen has isVisibleToUser as false,
634 then it would effectively remove this navigation option. So, the Chrome Android
635 accessibility code reports most nodes as isVisibleToUser, and if the node is actually
636 offscreen (not programmatically hidden but scrolled offscreen), then we include a
637 Bundle extra boolean, "offscreen" so that downstream AT can differentiate between
638 the nodes truly on/off screen.
639
640- RangeInfo, aria-valuetext, and caching
641
642 The [RangeInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo.RangeInfo)
643 object is another child object of AccessibilityNodeInfo. Unfortunately this
644 object is rather limited in its options, and can only provide float values
645 for a min, max, and current value. There is no concept of a text description, or
646 steps or step size. This clashes with nodes such as sliders with an aria-valuetext,
647 or an indeterminate progress bar, for which we have to add special treatment.
648 As a further complication, AccessibilityEvents also require information on range
649 values when there is a change in value, however the event only allows integer
650 values between 0 and 100 (an integer percentage of the sliders position).
651 BrowserAccessibilityAndroid has a method [IsRangeControlWithoutAriaValueText](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=BrowserAccessibilityAndroid::IsRangeControlWithoutAriaValueText)
652 which we use to separate these cases when populating AccessibilityNodeInfo
653 objects and AccessibilityEvents (see web\_contents\_accessibility\_android.cc).
654 Similar to the Collection related objects above, RangeInfo is `null` for any
655 non-range related nodes.
656
657 This RangeInfo object plays a small role in updating the cached AccessibilityNodeInfo
658 objects above. There is a small bug in the Android framework (which has been fixed
659 on newer versions) which breaks our caching mechanism for range objects. So the
660 `UpdateCachedAccessibilityNodeInfo` method also updates the RangeInfo object of a node
661 if it has one.
662
663- Leaf nodes and links
664
665 Android has slightly different IsLeaf logic than other platforms, and this can
666 cause confusion, especially around links. On Android, links are **never** leafs.
667 See [IsLeaf](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/browser_accessibility_android.cc?q=BrowserAccessibilityAndroid::IsLeaf).
668 This is for similar reasons to the isVisibleToUser section above. If a link
669 were a leaf node, and it were to contain something like a Heading, then AT would
670 not be able to traverse that link when navigating by headings because it would only
671 see it is a link. For this reason we always expose the entire child structure
672 of links.
673
674- Refocusing a node Java-side
675
676 There is a strange bug in Android where objects that are accessibility focused
677 sometimes do not visually update their outline. This does not really block any
678 user flows per se, but we would ideally have the outlines drawn by AT like TalkBack
679 to reflect the correct bounds of the node. There is a simple way to get around
680 this bug, which is to remove focus from the node and refocus it again, which
681 triggers the underlying Android code necessary to update the bounds. In
682 WebContentsAccessibilityImpl we have a method
683 [moveAccessibilityFocusToIdAndRefocusIfNeeded](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22void%20moveAccessibilityFocusToIdAndRefocusIfNeeded%22)
684 which handles this.
685
686- liveRegions and forced announcements
687
688 There is a boolean for liveRegion in the AccessibilityNodeInfo object that the
689 Chrome accessibility code will never set. This is because TalkBack will read
690 the entirety of the liveRegion node when there is a change. Instead we force
691 a custom announcement with an AccessibilityEvent in the WebContentsAccessibilityImpl's
692 [announceLiveRegionText](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22private%20void%20announceLiveRegionText%22)
693 method.
694
695- sendAccessibilityEvent vs. requestSendAccessibilityEvent
696
697 In the Android framework, there are two methods for sending AccessibiltiyEvents,
698 [sendAccessibilityEvent](https://developer.android.com/reference/android/view/View#sendAccessibilityEvent\(int\)) and
699 [requestSendAccessibilityEvent](https://developer.android.com/reference/android/view/ViewParent#requestSendAccessibilityEvent\(android.view.View,%20android.view.accessibility.AccessibilityEvent\)).
700 Technically speaking, requestSendAccessibilityEvent will ask the system to
701 send an event, but it doesn't have to send it. For all intents and purposes,
702 we assume that it is always sent, but as a small detail to keep in mind, this
703 is not a guarantee.
704
705- TYPE\_WINDOW\_CONTENT\_CHANGED events
706
707 The TYPE\_WINDOW\_CONTENT\_CHANGED type AccessibilityEvent is used to tell the
708 framework that something has changed for the given node. We send these events
709 for any change in the web, including scrolling. As a result, this is generally the
710 most frequently sent event, and we can often send too many and put strain on
711 downstream AT. We have included this event as part of our event throttling in
712 the AccessibilityEventDispatcher. We also include a small optimization for a
713 given atomic update. If an atomic update sends more than
714 [kMaxContentChangedEventsToFire](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/web_contents_accessibility_android.h?q=%22kMaxContentChangedEventsToFire%22)
715 events (currently set to 5), then any further events are dropped and a single
716 event on the root node is sent instead. This has proven useful for situations such
717 as many nodes being toggled visible at once. See [HandleContentChanged](https://source.chromium.org/chromium/chromium/src/+/main:content/browser/accessibility/web_contents_accessibility_android.cc?q=%22WebContentsAccessibilityAndroid::HandleContentChanged%22)
718 in web\_contents\_accessibility\_android.cc.
719
720- Text selection
721
722 Static text selection exists in Android, but not when a service like TalkBack
723 is enabled. TalkBack allows for text selection inside an editable text field,
724 but not static text inside the web contents. Text selection is also tied to a
725 specific node with a start and end index. This means that we cannot select
726 text across multiple nodes. Ideally the implementation would allow a start
727 and end index on separate nodes, but this work is still in development.
728
729- Touch exploration
730
731 The way touch exploration works on Android is complicated. The process that happens
732 during any hover event is:
733
Mark Schillaci03ce1bb2022-03-07 22:53:49734 1. User begins dragging their finger
735 2. Java-side View receives a hover event and passes this through to C++
736 3. Accessibility code sends a hit test action to the renderer process
737 4. The renderer process fires a HOVER accessibility event on the accessibility
738 node at that coordinate
739 5. [WebContentsAccessibilityImpl#handleHover](https://source.chromium.org/chromium/chromium/src/+/main:content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java?q=%22private%20void%20handleHover%22) is called for that node.
740 6. We fire a TYPE\_VIEW\_HOVER\_ENTER AccessibilityEvent on that node.
741 7. We fire a TYPE\_VIEW\_HOVER\_EXIT AccessibilityEvent on the previous node.
742 8. TalkBack sets accessibility focus to the targeted node.
Mark Schillaciac3cdbd2022-03-04 23:40:08743
744- WebView and Custom Tabs
745
746 As mentioned at the start of this document, Chrome Android also plays an important
747 role by providing the WebView, which is used by any third party apps
748 that want to include web content in their app. Android also has a unique feature
749 of Custom Tabs, which is a lightweight implementation of Chrome that is somewhere
750 between a WebView and a full browser. The WebView and custom tabs must also be accessible,
751 and so most of this document applies to them as well. Occasionally there
752 will be an edge case or small bug that only happens in WebView, or a feature that
753 needs to be turned off only for WebView/Custom tab (e.g. image descriptions). There is a
754 WebView and Chrome Custom Tabs test app on the chrome-accessibility appspot page,
755 and there are methods on the Java-side that can give signals of whether the current
756 instance is a WebView or Chrome custom tab. [Example: isCustomTab](https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/tab/java/src/org/chromium/chrome/browser/tab/Tab.java?q=isCustomTab).
757
758## Recent Progress and Features
759
760The Chrome Android accessibility code continues to evolve each quarter. We have
761strengthened our testing and the stability of the code, but we also continue to
762add new features and improvements. Beyond the usual bug fixes, below is a quick
763summary of some features in the pipeline.
764
765### OnDemand AT
766
767We have recently implemented a feature we refer to as "OnDemand AT" for short.
768This feature is still rolling out and we intend to eventually have it enabled
769on 100% stable by default. The feature modifies the AccessibilityEventDispatcher
770that is explained above. If the feature is enabled, then WebContentsAccessibilityImpl
771will query the Android system to determine the currently enabled accessibility
772services, as well as the types of data they are interested in, namely the types of
773AccessibilityEvents they want to know about. When the AccessibilityEventDispatcher
774is sent an event to add to its queue or dispatch, if that event type is not in
775the list of AccessibilityEvents relevant to currently enabled accessibility services,
776the Dispatcher simply drops/ignores the request. Preliminary data shows that this
777has created a noticeable improvement for accessibility services that do not require
778the entire suite to function.
779
780### ComputeAXMode
781
782Loosely related to the OnDemand feature above, the "ComputeAXMode" feature is also
783a recent addition to improve overall performance. This feature uses the same
784mechanism as OnDemand to query the currently enabled services and the information
785they are interested in. ComputeAXMode then takes this information and uses a different
786[AXMode](https://source.chromium.org/chromium/chromium/src/+/main:ui/accessibility/ax_mode.h)
787based on the situation. This effectively does the same thing as
788OnDemand, but further left/up-the-chain, giving a more significant performance
789improvement. This feature is still rolling out, and it currently only has two
790AXModes (full or basic). As it rolls out and we gather more data we will potentially
791add more AXModes in the future.
792
793### AutoDisableAccessibility
794
795The "AutoDisable" accessibility feature has also been ported to Android. This feature
796tracks timing between user inputs and accessibility actions to make a determination
797of whether or not accessibility services are still required. If they are no longer
798needed by the user, then the accessibility code is disabled. Before this feature,
799once the code was enabled it would continue to run for the life of the current
800browser session. This feature is still being rolled out to stable.
801
802
803## Accessibility code in the ClankUI
804
805Most of this document is focused on the accessibility code and work as it relates
806to the web contents, which is where the Chrome & Chrome OS Accessibility team
807focuses most of its work. However, some features require a native UI in the browser
808app, outside the web contents. When these features are added, the line between
809the accessibility team and the Clank UI team becomes blurred. We traditionally
810are the owners of this code, but seek regular guidance and approvals from the
811Clank team as the front-end code must conform to the Clank standards.
812
813### AccessibilitySettings
814
815The AccessibilitySettings page is found under the overflow/3-dot menu (Menu\>Settings\>Accessibility).
816The page currently contains a slider to change the font scaling of the web contents,
817options to force enable zoom, show simplified pages, enable image descriptions (see below),
818and live captions.
819
820The main entry point for this code is [here](https://source.chromium.org/chromium/chromium/src/+/main:components/browser_ui/accessibility/android/java/src/org/chromium/components/browser_ui/accessibility/AccessibilitySettings.java).
821The code leverages the PreferenceFragment of Android, and so much of the UI and
822navigation is available out of the box, and the code is relatively simple in that
823it only needs to respond to user actions/changes and pass this information to the
824native C++ code.
825
826The settings code is heavily unit tested and stable, so it is rare to have to
827work in this area.
828
829### Image Descriptions
830
831The Clank-side code for the image descriptions feature is a bit more involved.
832The image descriptions has to track state, determine whether or not to display the
833option in the overflow menu, show dialogs, and provide toasts to the user. This
834code is mostly controlled by the [ImageDescriptionsController](https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/image_descriptions/android/java/src/org/chromium/chrome/browser/image_descriptions/ImageDescriptionsController.java).
835The image descriptions feature is written using Clank's 'component' model, and so
836almost all the code exists in the directory:
837[chrome/browser/image_descriptions](https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/image_descriptions/)
838
839(The exception being the few hooks that connect this code to the other parts of the
840Clank UI).
841
842The image descriptions code is heavily unit tested and stable, so it is rare to
843have to work in this area.
844
845### (Upcoming) Page Zoom
846
847An upcoming feature is the page zoom feature, which will allow a more robust way
848to zoom web contents than the currently existing text scaling of AccessibilitySettings
849(which will be replaced).
850
851The Clank UI code for this feature has not been developed. More to come.