David Brazdil | dd56510 | 2020-10-23 16:33:30 +0000 | [diff] [blame] | 1 | # Getting started with Protected Virtual Machines |
| 2 | |
| 3 | ## Prepare a device |
| 4 | |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 5 | First you will need a device that is capable of running virtual machines. On arm64, this means a |
| 6 | device which boots the kernel in EL2 and the kernel was built with KVM enabled. Unfortunately at the |
| 7 | moment, we don't have an arm64 device in AOSP which does that. Instead, use cuttlefish which |
| 8 | provides the same functionalities except that the virtual machines are not protected from the host |
| 9 | (i.e. Android). This however should be enough for functional testing. |
David Brazdil | dd56510 | 2020-10-23 16:33:30 +0000 | [diff] [blame] | 10 | |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 11 | We support the following device: |
David Brazdil | dd56510 | 2020-10-23 16:33:30 +0000 | [diff] [blame] | 12 | |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 13 | * aosp_cf_x86_64_phone (Cuttlefish a.k.a. Cloud Android) |
Jiyong Park | 60c8aef | 2022-06-10 15:51:22 +0900 | [diff] [blame] | 14 | * oriole (Pixel 6) |
| 15 | * raven (Pixel 6 Pro) |
| 16 | |
| 17 | ### Cuttlefish |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 18 | |
| 19 | Building Cuttlefish |
| 20 | |
| 21 | ```shell |
| 22 | source build/envsetup.sh |
| 23 | lunch aosp_cf_x86_64_phone-userdebug |
| 24 | m |
| 25 | ``` |
| 26 | |
| 27 | Run Cuttlefish locally by |
| 28 | |
| 29 | ```shell |
| 30 | acloud create --local-instance --local-image |
| 31 | ``` |
| 32 | |
Jiyong Park | 60c8aef | 2022-06-10 15:51:22 +0900 | [diff] [blame] | 33 | ### Pixel 6 and 6 Pro |
| 34 | |
| 35 | If the device is running Android 12, join the [Android Beta |
Pierre-Clément Tosi | c1c6983 | 2022-10-14 18:53:31 +0100 | [diff] [blame] | 36 | Program](https://www.google.com/android/beta) to upgrade to Android 13 Beta. |
Jiyong Park | 60c8aef | 2022-06-10 15:51:22 +0900 | [diff] [blame] | 37 | |
| 38 | Once upgraded to Android 13, execute the following command to enable pKVM. |
| 39 | |
| 40 | ```shell |
| 41 | adb reboot bootloader |
Jiyong Park | a3953d1 | 2022-06-28 20:44:23 +0900 | [diff] [blame] | 42 | fastboot flashing unlock |
Jiyong Park | 60c8aef | 2022-06-10 15:51:22 +0900 | [diff] [blame] | 43 | fastboot oem pkvm enable |
| 44 | fastboot reboot |
| 45 | ``` |
| 46 | |
Pierre-Clément Tosi | 072c11f | 2022-06-10 15:09:09 +0100 | [diff] [blame] | 47 | Due to a bug in Android 13 for these devices, pKVM may stop working after an |
| 48 | [OTA update](https://source.android.com/devices/tech/ota). To prevent this, it |
Jiyong Park | e21a24b | 2022-06-29 09:20:23 +0900 | [diff] [blame] | 49 | is necessary to manually replicate the `pvmfw` partition to the other slot: |
Pierre-Clément Tosi | 072c11f | 2022-06-10 15:09:09 +0100 | [diff] [blame] | 50 | |
| 51 | ```shell |
Jiyong Park | e21a24b | 2022-06-29 09:20:23 +0900 | [diff] [blame] | 52 | git -C <android_root>/packages/modules/Virtualization |
| 53 | show de6b0b2ecf6225a0a7b43241de27e74fc3e6ceb2:pvmfw/pvmfw.img > /tmp/pvmfw.img |
Pierre-Clément Tosi | 072c11f | 2022-06-10 15:09:09 +0100 | [diff] [blame] | 54 | adb reboot bootloader |
Jiyong Park | e21a24b | 2022-06-29 09:20:23 +0900 | [diff] [blame] | 55 | fastboot --slot other flash pvmfw /tmp/pvmfw.img |
Pierre-Clément Tosi | 072c11f | 2022-06-10 15:09:09 +0100 | [diff] [blame] | 56 | fastboot reboot |
| 57 | ``` |
| 58 | |
| 59 | Otherwise, if an OTA has already made pKVM unusable, the working partition |
Jiyong Park | e21a24b | 2022-06-29 09:20:23 +0900 | [diff] [blame] | 60 | should be copied to the "current" slot: |
Pierre-Clément Tosi | 072c11f | 2022-06-10 15:09:09 +0100 | [diff] [blame] | 61 | |
| 62 | ```shell |
Pierre-Clément Tosi | 072c11f | 2022-06-10 15:09:09 +0100 | [diff] [blame] | 63 | adb reboot bootloader |
Jiyong Park | e21a24b | 2022-06-29 09:20:23 +0900 | [diff] [blame] | 64 | fastboot flash pvmfw /tmp/pvmfw.img |
Pierre-Clément Tosi | 072c11f | 2022-06-10 15:09:09 +0100 | [diff] [blame] | 65 | fastboot reboot |
| 66 | ``` |
| 67 | |
| 68 | Finally, if the `pvmfw` partition has been corrupted, both slots may be flashed |
Pierre-Clément Tosi | 39bce67 | 2022-10-14 18:54:04 +0100 | [diff] [blame] | 69 | using the [`pvmfw.img` pre-built](https://android.googlesource.com/platform/packages/modules/Virtualization/+/08deac98acefd62e222edfa374d5292458cf97eb%5E/pvmfw/pvmfw.img) |
Pierre-Clément Tosi | 072c11f | 2022-06-10 15:09:09 +0100 | [diff] [blame] | 70 | as long as the bootloader remains unlocked. Otherwise, a fresh install of |
| 71 | Android 13 followed by the manual steps above for flashing the `other` slot |
| 72 | should be used as a last resort. |
| 73 | |
Pierre-Clément Tosi | 4e0fed7 | 2022-10-14 19:01:56 +0100 | [diff] [blame] | 74 | Starting in Android 14, `pvmfw.img` can be built using the Android Build system: |
| 75 | ``` |
| 76 | lunch <target> # where PRODUCT_BUILD_PVMFW_IMAGE=true |
| 77 | m pvmfwimage # partition image under ${ANDROID_PRODUCT_OUT}/pvmfw.img |
| 78 | ``` |
| 79 | Note that the result is not intended to be used in Android 13 as not |
| 80 | backward-compatibility is guaranteed. |
| 81 | |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 82 | ## Running demo app |
| 83 | |
| 84 | The instruction is [here](../../demo/README.md). |
David Brazdil | dd56510 | 2020-10-23 16:33:30 +0000 | [diff] [blame] | 85 | |
| 86 | ## Running tests |
| 87 | |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 88 | There are various tests that spawn guest VMs and check different aspects of the architecture. They |
| 89 | all can run via `atest`. |
Andrew Walbran | 0479a65 | 2021-04-12 11:17:13 +0000 | [diff] [blame] | 90 | |
| 91 | ```shell |
Jiyong Park | 29de517 | 2022-02-08 00:37:05 +0900 | [diff] [blame] | 92 | atest VirtualizationTestCases.64 |
Kalesh Singh | b507098 | 2021-12-14 23:21:39 -0800 | [diff] [blame] | 93 | atest MicrodroidHostTestCases |
| 94 | atest MicrodroidTestApp |
David Brazdil | dd56510 | 2020-10-23 16:33:30 +0000 | [diff] [blame] | 95 | ``` |
| 96 | |
| 97 | If you run into problems, inspect the logs produced by `atest`. Their location is printed at the |
| 98 | end. The `host_log_*.zip` file should contain the output of individual commands as well as VM logs. |
| 99 | |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 100 | ## Spawning your own VMs with custom kernel |
David Brazdil | dd56510 | 2020-10-23 16:33:30 +0000 | [diff] [blame] | 101 | |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 102 | You can spawn your own VMs by passing a JSON config file to the VirtualizationService via the `vm` |
| 103 | tool on a rooted KVM-enabled device. If your device is attached over ADB, you can run: |
Andrew Walbran | 0479a65 | 2021-04-12 11:17:13 +0000 | [diff] [blame] | 104 | |
| 105 | ```shell |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 106 | cat > vm_config.json |
Andrew Walbran | 0479a65 | 2021-04-12 11:17:13 +0000 | [diff] [blame] | 107 | { |
| 108 | "kernel": "/data/local/tmp/kernel", |
| 109 | "initrd": "/data/local/tmp/ramdisk", |
| 110 | "params": "rdinit=/bin/init" |
| 111 | } |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 112 | adb root |
| 113 | adb push <kernel> /data/local/tmp/kernel |
| 114 | adb push <ramdisk> /data/local/tmp/ramdisk |
| 115 | adb push vm_config.json /data/local/tmp/vm_config.json |
| 116 | adb shell "start virtualizationservice" |
| 117 | adb shell "/apex/com.android.virt/bin/vm run /data/local/tmp/vm_config.json" |
David Brazdil | dd56510 | 2020-10-23 16:33:30 +0000 | [diff] [blame] | 118 | ``` |
| 119 | |
Andrew Walbran | 0479a65 | 2021-04-12 11:17:13 +0000 | [diff] [blame] | 120 | The `vm` command also has other subcommands for debugging; run `/apex/com.android.virt/bin/vm help` |
| 121 | for details. |
David Brazdil | dd56510 | 2020-10-23 16:33:30 +0000 | [diff] [blame] | 122 | |
Jaewan Kim | 4642281 | 2022-11-25 10:59:39 +0900 | [diff] [blame] | 123 | ## Spawning your own VMs with custom pvmfw |
| 124 | |
| 125 | Set system property `hypervisor.pvmfw.path` to custom `pvmfw` on the device before using `vm` tool. |
| 126 | `virtualizationservice` will pass the specified `pvmfw` to `crosvm` for protected VMs. |
| 127 | |
| 128 | ```shell |
| 129 | adb push pvmfw.img /data/local/tmp/pvmfw.img |
| 130 | adb root # required for setprop |
| 131 | adb shell setprop hypervisor.pvmfw.path /data/local/tmp/pvmfw.img |
| 132 | ``` |
| 133 | |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 134 | ## Spawning your own VMs with Microdroid |
David Brazdil | dd56510 | 2020-10-23 16:33:30 +0000 | [diff] [blame] | 135 | |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 136 | [Microdroid](../../microdroid/README.md) is a lightweight version of Android that is intended to run |
Nikita Ioffe | b0b6756 | 2022-11-22 15:48:06 +0000 | [diff] [blame] | 137 | on pVM. You can run a Microdroid with empty payload using the following command: |
| 138 | |
| 139 | ```shell |
| 140 | adb shell /apex/com.android.virt/bin/vm run-microdroid --debug full |
| 141 | ``` |
| 142 | |
| 143 | The `instance.img` and `apk.idsig` files will be stored in a subdirectory under |
| 144 | `/data/local/tmp/microdroid`, that `vm` will create. |
| 145 | |
| 146 | Atlernatively, you can manually run the demo app on top of Microdroid as follows: |
Andrew Walbran | 0479a65 | 2021-04-12 11:17:13 +0000 | [diff] [blame] | 147 | |
| 148 | ```shell |
Nikita Ioffe | 519deab | 2022-09-30 17:36:02 +0100 | [diff] [blame] | 149 | UNBUNDLED_BUILD_SDKS_FROM_SOURCE=true TARGET_BUILD_APPS=MicrodroidDemoApp m apps_only dist |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 150 | adb shell mkdir -p /data/local/tmp/virt |
| 151 | adb push out/dist/MicrodroidDemoApp.apk /data/local/tmp/virt/ |
| 152 | adb shell /apex/com.android.virt/bin/vm run-app \ |
| 153 | --debug full \ |
| 154 | /data/local/tmp/virt/MicrodroidDemoApp.apk \ |
| 155 | /data/local/tmp/virt/MicrodroidDemoApp.apk.idsig \ |
Inseob Kim | 7b5f65c | 2022-11-15 14:27:04 +0900 | [diff] [blame] | 156 | /data/local/tmp/virt/instance.img \ |
| 157 | --payload-path MicrodroidTestNativeLib.so |
Jiyong Park | 978b1e3 | 2021-02-04 20:23:40 +0900 | [diff] [blame] | 158 | ``` |
David Brazdil | dd56510 | 2020-10-23 16:33:30 +0000 | [diff] [blame] | 159 | |
Jiyong Park | b5d04d8 | 2022-03-18 16:11:03 +0900 | [diff] [blame] | 160 | ## Building and updating CrosVM and VirtualizationService {#building-and-updating} |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 161 | |
| 162 | You can update CrosVM and the VirtualizationService by updating the `com.android.virt` APEX instead |
| 163 | of rebuilding the entire image. |
Jiyong Park | 978b1e3 | 2021-02-04 20:23:40 +0900 | [diff] [blame] | 164 | |
Andrew Walbran | 0479a65 | 2021-04-12 11:17:13 +0000 | [diff] [blame] | 165 | ```shell |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 166 | banchan com.android.virt aosp_arm64 // or aosp_x86_64 if the device is cuttlefish |
Jiyong Park | 5cd312a | 2022-03-11 21:37:53 +0900 | [diff] [blame] | 167 | UNBUNDLED_BUILD_SDKS_FROM_SOURCE=true m apps_only dist |
Jiyong Park | 10b354a | 2021-11-17 17:46:53 +0900 | [diff] [blame] | 168 | adb install out/dist/com.android.virt.apex |
| 169 | adb reboot |
David Brazdil | dd56510 | 2020-10-23 16:33:30 +0000 | [diff] [blame] | 170 | ``` |
Jiyong Park | b5d04d8 | 2022-03-18 16:11:03 +0900 | [diff] [blame] | 171 | |
Jiyong Park | b8e05b6 | 2022-06-24 16:40:35 +0900 | [diff] [blame] | 172 | ## Building and updating kernel inside Microdroid |
Jiyong Park | b5d04d8 | 2022-03-18 16:11:03 +0900 | [diff] [blame] | 173 | |
Jiyong Park | 1dabd56 | 2022-11-04 15:33:18 +0900 | [diff] [blame] | 174 | The instruction is [here](../../microdroid/kernel/README.md). |