blob: e3fc5c5360c9b729d23668bd2763de715d6c912b [file] [log] [blame] [view]
Staphany Park384b99a2019-12-18 03:23:341# AddressSanitizer (ASan)
2
3[AddressSanitizer](https://github.com/google/sanitizers) (ASan) is a fast memory
4error detector based on compiler instrumentation (LLVM). It is fully usable for
5Chrome on Linux and Mac. There's a mostly-functional Windows port in progress
6too. Additional info on the tool itself is available at
Darwin Huang7d3b5f052019-12-23 19:25:527https://clang.llvm.org/docs/AddressSanitizer.html.
Staphany Park384b99a2019-12-18 03:23:348
9For the memory leak detector built into ASan, see
10[LeakSanitizer](https://sites.google.com/a/chromium.org/dev/developers/testing/leaksanitizer).
11If you want to debug memory leaks, please refer to the instructions on that page
12instead.
13
14## Buildbots and trybots
15
16The [Chromium Memory
17waterfall](https://ci.chromium.org/p/chromium/g/chromium.memory/console) (not to
18be confused with the Memory FYI waterfall) contains buildbots running Chromium
19tests under ASan on Linux (Linux ASan/LSan bots for the regular Linux build,
20Linux Chromium OS ASan for the chromeos=1 build running on Linux), OS X (both 32
21and 64 bits), Chromium OS (x86 and amd64 builds running inside VMs). Linux and
22Linux Chromium OS bots run with --no-sandbox, but there's an extra Linux bot
23that enables the sandbox (but disables LeakSanitizer).
24
Darwin Huang7d3b5f052019-12-23 19:25:5225The trybots running Chromium tests on Linux and macOS are:
26- linux_asan (everything except browser_tests and content_browsertests)
27- linux_browser_asan (browser_tests and content_browsertests),
28- mac_asan (many tests including browser_tests and content_browsertests)
29- linux_chromeos_asan (the chromeos=1 build running on a Linux machine, many
30tests including browser_tests and content_browsertests).
Staphany Park384b99a2019-12-18 03:23:3431
32## Pre-built Chrome binaries
33
34You can grab fresh Chrome binaries built with ASan
35[here](https://commondatastorage.googleapis.com/chromium-browser-asan/index.html).
36
37## Build tests with ASan
38
Darwin Huang7d3b5f052019-12-23 19:25:5239If you're on MacOS or linux64, building with ASan is easy. Start by compiling
40`base_unittests` to verify the build is working for you (see below). Then, you
Staphany Park384b99a2019-12-18 03:23:3441can compile `chrome`, `browser_tests`, etc.. Make sure to compile release
42builds.
43
44Make sure you've run `tools/clang/scripts/update.py` (see
45https://chromium.googlesource.com/chromium/src/+/master/docs/clang.md for
46details).
47
48### Configuring the build
49
50Create an asan build directory by running:
Darwin Huang7d3b5f052019-12-23 19:25:5251```shell
Staphany Park384b99a2019-12-18 03:23:3452gn args out/asan
53```
54
55Enter the following build variables in the editor that will pop up:
Darwin Huang7d3b5f052019-12-23 19:25:5256```python
Staphany Park384b99a2019-12-18 03:23:3457is_asan = true
58is_debug = false # Release build.
59```
60
61Build with:
Darwin Huang7d3b5f052019-12-23 19:25:5262```shell
Staphany Park384b99a2019-12-18 03:23:3463ninja -C out/asan base_unittests
64```
65
66### Goma build
67
68ASan builds should work seamlessly with Goma (except for Windows); just add
69`use_goma=1` to your `GYP_DEFINES` or `use_goma=true` in your "gn args" Don't
70forget to use ninja -j <jobs> to take advantage of goma.
71
72### Build options
73
74If you want your stack traces to be precise, you will have to disable inlining
75by setting the GN arg:
Darwin Huang7d3b5f052019-12-23 19:25:5276```shell
Staphany Park384b99a2019-12-18 03:23:3477enable_full_stack_frames_for_profiling = true
78```
79
80Note that this incurs a significant performance hit. Please do not do this on
81buildbots.
82
83If you're working on reproducing ClusterFuzz reports, you might want to add:
Darwin Huang7d3b5f052019-12-23 19:25:5284```shell
Staphany Park384b99a2019-12-18 03:23:3485v8_enable_verify_heap = true
86```
Darwin Huang7d3b5f052019-12-23 19:25:5287in order to enable the `--verify-heap` command line flag for v8 in Release builds.
Staphany Park384b99a2019-12-18 03:23:3488
89## Verify the ASan tool works
90
91**ATTENTION (Linux only)**: These instructions are for running ASan in a way
92that is compatible with the sandbox. However, this is not compatible with
93LeakSanitizer. If you want to debug memory leaks, please use the instructions on
94the
95[LeakSanitizer](https://sites.google.com/a/chromium.org/dev/developers/testing/leaksanitizer)
96page instead.
97
98Now, check that the tool works. Run the following:
Darwin Huang7d3b5f052019-12-23 19:25:5299```shell
100out/asan/base_unittests \
101 --gtest_filter=ToolsSanityTest.DISABLED_AddressSanitizerLocalOOBCrashTest \
102 --gtest_also_run_disabled_tests 2>&1 | tools/valgrind/asan/asan_symbolize.py
Staphany Park384b99a2019-12-18 03:23:34103```
104
105The test will crash with the following error report:
Darwin Huang7d3b5f052019-12-23 19:25:52106```shell
107==26552== ERROR: AddressSanitizer stack-buffer-overflow on address \
1080x7fff338adb14 at pc 0xac20a7 bp 0x7fff338adad0 sp 0x7fff338adac8
Staphany Park384b99a2019-12-18 03:23:34109WRITE of size 4 at 0x7fff338adb14 thread T0
110 #0 0xac20a7 in base::ToolsSanityTest_DISABLED_AddressSanitizerLocalOOBCrashTest_Test::TestBody() ???:0
111 #1 0xcddbd6 in testing::Test::Run() testing/gtest/src/gtest.cc:2161
112 #2 0xcdf63b in testing::TestInfo::Run() testing/gtest/src/gtest.cc:2338
113... lots more stuff
Darwin Huang7d3b5f052019-12-23 19:25:52114Address 0x7fff338adb14 is located at offset 52 in frame \
115base::ToolsSanityTest_DISABLED_AddressSanitizerLocalOOBCrashTest_Test::TestBody()> of T0's stack:
Staphany Park384b99a2019-12-18 03:23:34116 This frame has 2 object(s):
117 [32, 52) 'array'
118 [96, 104) 'access'
119==26552== ABORTING
120... lots more stuff
121```
122
Samuel Huange9a7bff9d2020-03-04 16:16:03123Congrats, you have a working ASan build! &#x1F64C;
Staphany Park384b99a2019-12-18 03:23:34124
125## Run chrome under ASan
126
Darwin Huang7d3b5f052019-12-23 19:25:52127And finally, have fun with the `out/Release/chrome` binary. The filter script
Staphany Park384b99a2019-12-18 03:23:34128`tools/valgrind/asan/asan_symbolize.py` should be used to symbolize the output.
129(Note that `asan_symbolize.py` is absolutely necessary if you need the symbols -
130there is no built-in symbolizer for ASan in Chrome).
131
132ASan should perfectly work with Chrome's sandbox. You should only need to run
133with `--no-sandbox` on Linux if you're debugging ASan.
134Note: you have to disable the sandbox on Windows until it is supported.
135
136You may need to run with `--disable-gpu` on Linux with NVIDIA driver older than
137295.20.
138
139You will likely need to define environment variable
140[`G_SLICE=always-malloc`](https://developer.gnome.org/glib/unstable/glib-running.html)
141to avoid crashes inside gtk.
142NSS_DISABLE_ARENA_FREE_LIST=1 and NSS_DISABLE_UNLOAD=1 are required as well.
143
144When filing a bug found by AddressSanitizer, please add a label
145`Stability-AddressSanitizer`.
146
147## ASan runtime options
148
149ASan's behavior can be changed by exporting the `ASAN_OPTIONS` env var. Some of
150the useful options are listed on this page, others can be obtained from running
151an ASanified binary with `ASAN_OPTIONS=help=1`. Note that Chromium sets its own
152defaults for some options, so the default behavior may be different from that
153observed in other projects.
154See `base/debug/sanitizer_options.cc` for more details.
155
156## NaCl support under ASan
157
Darwin Huang7d3b5f052019-12-23 19:25:52158On Linux (and soon on MacOS) you can build and run Chromium with NaCl under ASan.
Staphany Park384b99a2019-12-18 03:23:34159Untrusted code (nexe) itself is not instrumented with ASan in this mode, but
160everything else is.
161
162To do this, remove `disable_nacl=1` from `GYP_DEFINES`, and define
163`NACL_DANGEROUS_SKIP_QUALIFICATION_TEST=1` in your environment at run time.
164
165Pipe chromium output (stderr) through ``tools/valgrind/asan/asan_symbolize.py
166`pwd`/`` to get function names and line numbers in ASan reports.
Darwin Huang7d3b5f052019-12-23 19:25:52167If you're seeing crashes within `nacl_helper_bootstrap`, try deleting
168`out/Release/nacl_helper`.
Staphany Park384b99a2019-12-18 03:23:34169
170## Building on iOS
171
172It's possible to build and run Chrome tests for iOS simulator (which are x86
173binaries essentially) under ASan. Note that you'll need a Chrome iOS checkout
174for that. It isn't currently possible to build iOS binaries targeting ARM.
175
176Configure your build with `is_asan = true` as described above. Replace your
177build directory as needed:
Darwin Huang7d3b5f052019-12-23 19:25:52178```shell
Staphany Park384b99a2019-12-18 03:23:34179ninja -C out/Release-iphonesimulator base_unittests
Darwin Huang7d3b5f052019-12-23 19:25:52180out/Release-iphonesimulator/iossim -d "iPhone" -s 7.0 \
181 out/Release-iphonesimulator/base_unittests.app/ \
182 --gtest_filter=ToolsSanityTest.DISABLED_AddressSanitizerLocalOOBCrashTest \
183 --gtest_also_run_disabled_tests 2>&1 |
Staphany Park384b99a2019-12-18 03:23:34184tools/valgrind/asan/asan_symbolize.py
185```
186
187You'll see the same report as shown above (see the "Verify the ASan tool works"
188section), with a number of iOS-specific frames.
189
190## Building on Android
191
192Follow [AndroidBuildInstructions](android_build_instructions.md) with minor
193changes:
194
Darwin Huang7d3b5f052019-12-23 19:25:52195```python
196target_os="android"
197is_clang=true
198is_asan=true
199is_debug=false
200```
Staphany Park384b99a2019-12-18 03:23:34201
202Running ASan applications on Android requires additional device setup. Chromium
203testing scripts take care of this, so testing works as expected:
Darwin Huang7d3b5f052019-12-23 19:25:52204```shell
205build/android/test_runner.py instrumentation --test-apk ContentShellTest \
206 --test_data content:content/test/data/android/device_files -v -v -v \
207 --tool=asan --release
Staphany Park384b99a2019-12-18 03:23:34208```
209
210To run stuff without Chromium testing script (ex. ContentShell.apk, or any third
211party apk or binary), device setup is needed:
Darwin Huang7d3b5f052019-12-23 19:25:52212```shell
Samuel Huange9a7bff9d2020-03-04 16:16:03213tools/android/asan/third_party/asan_device_setup.sh \
214 --lib third_party/llvm-build/Release+Asserts/lib/clang/*/lib/linux/libclang_rt.asan-arm-android.so
Staphany Park384b99a2019-12-18 03:23:34215# wait a few seconds for the device to reload
216```
217
218It only needs to be run once per device. It is safe to run it multiple times.
Samuel Huange9a7bff9d2020-03-04 16:16:03219Examine the output to ensure that setup was successful (you may need to run
220`adb disable-verity` and restart the device first). When this is done, the
221device will run ASan apks as well as normal apks without any further setup.
Staphany Park384b99a2019-12-18 03:23:34222
223To run command-line tools (i.e. binaries), prefix them with `asanwrapper`:
Darwin Huang7d3b5f052019-12-23 19:25:52224```shell
Staphany Park384b99a2019-12-18 03:23:34225adb shell /system/bin/asanwrapper /path/to/binary
226```
227
Darwin Huang7d3b5f052019-12-23 19:25:52228Use `build/android/asan_symbolize.py` to symbolize stack from `adb logcat`. It
229needs the `--output-directory` argument and takes care of translating the device
230path to the unstripped binary in the output directory.
Staphany Park384b99a2019-12-18 03:23:34231
232## Building with v8_target_arch=arm
233
234This is needed to detect addressability bugs in the ARM code emitted by V8 and
235running on an instrumented ARM emulator in a 32-bit x86 Linux Chromium. **You
236probably don't want this, and these instructions have bitrotted because they
237still reference GYP. If you do this successfully, please update!** See
Darwin Huang7d3b5f052019-12-23 19:25:52238https://crbug.com/324207 for some context.
Staphany Park384b99a2019-12-18 03:23:34239
240First, you need to install the 32-bit chroot environment using the
241`build/install-chroot.sh` script (as described in
242https://code.google.com/p/chromium/wiki/LinuxBuild32On64). Second, install the
243build deps:
Darwin Huang7d3b5f052019-12-23 19:25:52244```shell
245precise32 build/install-build-deps.sh \
246 # assuming your schroot wrapper is called 'precise32'
Staphany Park384b99a2019-12-18 03:23:34247```
248
249You'll need to make two symlinks to avoid linking errors:
Darwin Huang7d3b5f052019-12-23 19:25:52250```shell
251sudo ln -s $CHROOT/usr/lib/i386-linux-gnu/libc_nonshared.a \
252 /usr/lib/i386-linux-gnu/libc_nonshared.a
253sudo ln -s $CHROOT/usr/lib/i386-linux-gnu/libpthread_nonshared.a \
254 /usr/lib/i386-linux-gnu/libpthread_nonshared.a
Staphany Park384b99a2019-12-18 03:23:34255```
256
257Now configure and build your Chrome:
Darwin Huang7d3b5f052019-12-23 19:25:52258```shell
259GYP_GENERATOR_FLAGS="output_dir=out_asan_chroot" GYP_DEFINES="asan=1 \
260 disable_nacl=1 v8_target_arch=arm sysroot=/var/lib/chroot/precise32bit/ \
261 chroot_cmd=precise32 host_arch=x86_64 target_arch=ia32" gclient runhooks
Staphany Park384b99a2019-12-18 03:23:34262ninja -C out_asan_chroot/Release chrome
263```
264
265**Note**: `disable_nacl=1` is needed for now.
266
267## AsanCoverage
268
269AsanCoverage is a minimalistic code coverage implementation built into ASan. For
270general information see
271[https://code.google.com/p/address-sanitizer/wiki/AsanCoverage](https://github.com/google/sanitizers)
272To use AsanCoverage in Chromium, add `use_sanitizer_coverage = true` to your GN
273args. See also the `sanitizer_coverage_flags` variable for configuring it.
274
275Chrome must be terminated gracefully in order for coverage to work. Either close
276the browser, or SIGTERM the browser process. Do not do `killall chrome` or send
277SIGKILL.
Darwin Huang7d3b5f052019-12-23 19:25:52278```shell
279kill <browser_process_pid>
280ls
Staphany Park384b99a2019-12-18 03:23:34281...
282chrome.22575.sancov
283gpu.6916123572022919124.sancov.packed
284zygote.13651804083035800069.sancov.packed
285...
286```
287
288The `gpu.*.sancov.packed` file contains coverage data for the GPU process,
289whereas the `zygote.*.sancov.packed` file contains coverage data for the
290renderers (but not the zygote process). Unpack them to regular `.sancov` files
291like so:
Darwin Huang7d3b5f052019-12-23 19:25:52292```shell
293$ $LLVM/projects/compiler-rt/lib/sanitizer_common/scripts/sancov.py unpack \
294 *.sancov.packed
Staphany Park384b99a2019-12-18 03:23:34295sancov.py: unpacking gpu.6916123572022919124.sancov.packed
296sancov.py: extracting chrome.22610.sancov
297sancov.py: unpacking zygote.13651804083035800069.sancov.packed
298sancov.py: extracting libpdf.so.12.sancov
299sancov.py: extracting chrome.12.sancov
300sancov.py: extracting libpdf.so.10.sancov
301sancov.py: extracting chrome.10.sancov
302```
303
304Now, e.g., to list the offsets of covered functions in the libpdf.so binary in
305renderer with pid 10:
Darwin Huang7d3b5f052019-12-23 19:25:52306```shell
307$ $LLVM/projects/compiler-rt/lib/sanitizer_common/scripts/sancov.py print \
308 libpdf.so.10.sancov
Staphany Park384b99a2019-12-18 03:23:34309```