SlideShare a Scribd company logo

Memory Management in TIZEN - Samsung SW Platform Team

•
Ryo Jin, profile picture
Ryo Jin

This document discusses memory management in the Tizen kernel. It provides an overview of key components used for memory management including the Graphics Execution Manager (GEM) for graphics memory allocation and sharing, the Video for Linux Two (V4L2) and Video Buffer API Version 2 (VB2) for multimedia devices, and the Universal Memory Management (UMM) subsystem for buffer sharing across components without copying. The document also presents a simplified scenario demonstrating how these components could work together to share buffers for merging and displaying a camera video stream and overlay graphics.

Technology
1 of 68
Downloaded 288 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
Memory Management in Tizen SW Platform Team, SW R&D Center
tizen.org2 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel • Memory Size Optimization
tizen.org3 Tizen Kernel Overview
tizen.org4 Tizen Kernel Overview Kernel Kernel OAL (OEM Adaptation Layer) Telephony Plug-ins GStreamer Plug-ins Sensor Plug-ins System Plug-ins OpenGL ES/EGL Graphics Driver Graphics Memory Management IOMMU DMA BUF Multimedia Power Management CPUfre q Devf req Ther mal … DRM Link1 Link2 V4L2 Storage Block Layer MMC/Flash Input, Sensor, … Peripheral CMA OpenGL ES 1.0/2.0 X11 GStreamerEFL . . . Multimedia FW Telephony FW . . .Sensor FW System FW . . . More at http://source.tizen.org/documentation/porting-guide Core DRM Link1: http:// elinux.org/images/7/71/Elce11_dae.pdf DRM Link2: http://download.tizen.org/misc/media/conference2012/wednesday/ballroom-c/2012-05-09-1330-1410- the_drm_(direct_rendering_manager)_of_tizen_kernel.pdf
tizen.org5 Tizen Kernel Overview Kernel Kernel OAL (OEM Adaptation Layer) Telephony Plug-ins GStreamer Plug-ins Sensor Plug-ins System Plug-ins OpenGL ES/EGL Graphics Driver Graphics Memory Management IOMM U DMA BUF Multimedia Power Management CPUfre q Devf req Ther mal …DRM Link1 Link2 V4L2 Storage Block Layer MMC/Flash Input, Sensor, … Peripheral CM A OpenGL ES 1.0/2.0 X11 GStreamerEFL . . . Multimedia FW Telephony FW . . .Sensor FW System FW . . . More at http://source.tizen.org/documentation/porting-guide Core Service Layer DRM Link1: http:// elinux.org/images/7/71/Elce11_dae.pdf DRM Link2: http://download.tizen.org/misc/media/conference2012/wednesday/ballroom-c/2012-05-09-1330-1410- the_drm_(direct_rendering_manager)_of_tizen_kernel.pdf • Memory Management in Tizen – Coupled with Graphics & Multimedia devices. • Graphics & Multimedia devices = DMA devices with HUGE buffers
tizen.org6 Tizen Kernel Overview Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF
tizen.org7 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel – To Run Tizen OS. • Basic Memory Management Components • Buffer Sharing (UMM/DMABUF) • Buffer Allocation (UMM/DMAAPI) • Buffer Synchronization • Memory Size Optimization
tizen.org8 Tizen Kernel MM, Graphics (DRM) Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF
tizen.org9 Graphics: DRM/GEM (Graphics Execution Manager) drm_fb_helperdrm_mode_config Linux DRM Common Framework GEM drm_framebuffer fb_infoCrtc Connector Encoder GEM Allocator Plane Common Specific drm_framebufferCrtc Connector Encoder Plane
tizen.org10 Graphics: DRM/GEM (Graphics Execution Manager) drm_fb_helperdrm_mode_config Linux DRM Common Framework GEM drm_framebuffer fb_infoCrtc Connector Encoder GEM Allocator Plane Common Specific drm_framebufferCrtc Connector Encoder Plane Graphics Execution Manager • Framework developed by Intel • To manage graphics memory • Framework for buffer management. • Allocation and sharing.
tizen.org11 Graphics: DRM/GEM Allocation • GEM Allocation steps @ Tizen (Generic) 1. DRM_IOCTL_MODE_CREATE_DUMB • Create GEM object(global) & user GEM handle(per process) • dumb_create() of struct drm_driver • No physical memory allocated. 2. DRM_IOCTL_MODE_MAP_DUMB • Create fake mmap offset of a gem object and relay the object to user • A hash key of the gem object. • dumb_map_offset() of struct drm_driver 3. MMAP • Request mmap based on the hash key as the offset • Create user address space • Setup cache attribute. • Not mapped to physical memory, yet
tizen.org12 Graphics: DRM/GEM Allocation • GEM Allocation steps @ Tizen (Generic) 4. On-demand Paging – Implement & Register a fault handler that • With a page fault, allocate a page and map the page. – vma->vm_ops->fault = xxx_drm_gem_fault 5. Use! 6. DRM_IOCTL_MODE_DESTROY_DUMB – Remove GEM handle & object – Free memory – Implement dumb_destroy() of struct drm_driver
tizen.org13 Graphics: DRM/GEM Allocation • GEM Allocation steps @ Tizen (Exynos Only) 1. DRM_IOCTL_EXYNOS_GEM_CREATE • Only use user-desired size and buffer types. • Create gem object(global) & user gem handle(per process) • physical memory allocated. 2. DRM_IOCTL_EXYNOS_GEM_MMAP • Create user address space • Map the user address space to physical memory • LIBDRM of Exynos uses these APIs, not the generic.
tizen.org14 Graphics: DRM/GEM Sharing • GEM Sharing @ Tizen • DRM_IOCTL_GEM_FLINK – “I will share this GEM to others.” – Create GEM object name for the given GEM handle • Global key vaue for sharing • DRM_IOCTL_GEM_OPEN – “I want to use the shared GEM.” – Create GEM handle based on the given GEM object name • DRM_IOCTL_GEM_CLOSE • You don’t need to implement. It’s already there with DRM. GEM Create GEM FLINK GEM OPEN gem handle gem object name gem handle Process 1 Process 2
tizen.org15 Tizen Kernel MM, Multimedia (V4L2/VB2) Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF
tizen.org16 Multimedia: V4L2/VB2 • Tizen recommends to use V4L2 at Tizen kernel for Multimedia devices • Video input (codec & camera) & Radio • However, as long as the kernel has: – Gstreamer/OpenMAX plugins – A method to share with other F/W via DMABUF of UMM, • Tizen multimedia works. • If V4L2/VB2 is used, things get easier.
tizen.org17 Tizen Kernel MM, OpenGL/G3D-GPU Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF
tizen.org18 OpenGL / G3D-GPU • Most ARM SoC GPUs (MALI, SGX, …) use their own memory manager • E.g., Exynos4412/4210 Tizen Reference Kernel has Mali w/ UMP. – Mali DDK modified to be compatible with UMM-DMABUF. • If GPU drivers use DRM, it would be great. • (and make them GPL)
tizen.org19 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel – To Run Tizen OS. • Basic Memory Management Components • Buffer Sharing (UMM/DMABUF) • Buffer Allocation (UMM/DMAAPI) • Buffer Synchronization • Memory Size Optimization
tizen.org20 NEED FOR SOMETHING, A Scenario Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc Camera fetches a video stream This scenario is simplified for presentation. Not an actual example of Tizen
tizen.org21 NEED FOR SOMETHING, A Scenario Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc Camera fetches a video stream Draws its own images This scenario is simplified for presentation. Not an actual example of Tizen
tizen.org22 NEED FOR SOMETHING, A Scenario Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc Camera fetches a video stream Draws its own images GPU Merges the two / Puts the merged image This scenario is simplified for presentation. Not an actual example of Tizen
tizen.org23 NEED FOR SOMETHING, A Scenario Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc Camera fetches a video stream Draws its own images DRM-FIMD(display) Output This scenario is simplified for presentation. Not an actual example of Tizen GPU Merges the two / Puts the merged image
tizen.org24 NEED FOR SOMETHING, A Scenario Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc Camera fetches a video stream Draws its own images DRM-FIMD(display) Output At a VB2 buffer At a GEM buffer GEM buffer This scenario is simplified for presentation. Not an actual example of Tizen GPU Merges the two / Puts the merged image From GEM + VB2 into its own (e.g., UMP)
tizen.org25 Tizen Kernel MM, UMM Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF A mechanism to share between DRM, V4L2, and others w/o memcpy
tizen.org26 Tizen Kernel MM, UMM Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF Requirement from Tizen platform and hardware • Different Memory Managers: GEM, VB2, GPU-adhoc, … • Share buffers • w/o memcpy • From and to users • Never expose directly to users (e.g., physical address)  UMM DMABUF!
tizen.org27 Tizen Kernel Memory Management Unified Memory Management (UMM) • Introduced by Jesse Barker, 2011 • Includes – DMABUF (sharing buffer) – DMA Mapping API for Allocation. – CMA (Contiguous Memory Allocator) IOMMU (MMU for I/O devs)
tizen.org28 Tizen Kernel MM, UMM DMA Buffer Sharing: DMABUF • Export – GEM/VB2/… object  DMABUF • Import – DMABUF  GEM/VB2/… object • Userspace sees DMABUF as a File Descriptor
tizen.org29 DMA Buffer Sharing: DMABUF, Example Camera  Display X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org30 DMA Buffer Sharing: DMABUF, Example Conceptual Data Flow X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org31 DMABUF Usage Example 1/4 Camera App 1) Request V4L2 camera buffer (U) 2) Allocate CMA buffer (K) 3) Request a camera frame at the V4L2 buffer (U) 4) Store the camera frame & Notify user (K) 5) Request DMABUF export for the V4L2 camera buffer (U) 6) dma_buf_exporter() (K) - Create DMABUF from V4L2 buffer 7) dma_buf_fd() (K) - Provide FD of the DMABUF to user X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org32 DMABUF Usage Example 1/4 Camera App 1) Request V4L2 camera buffer (U) 2) Allocate CMA buffer (K) 3) Request a camera frame at the V4L2 buffer (U) 4) Store the camera frame & Notify user (K) 5) Request DMABUF export for the V4L2 camera buffer (U) 6) dma_buf_exporter() (K) - Create DMABUF from V4L2 buffer 7) dma_buf_fd() (K) - Provide FD of the DMABUF to user X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org33 DMABUF Usage Example 1/4 Camera App 1) Request V4L2 camera buffer (U) 2) Allocate CMA buffer (K) 3) Request a camera frame at the V4L2 buffer (U) 4) Store the camera frame & Notify user (K) 5) Request DMABUF export for the V4L2 camera buffer (U) 6) dma_buf_exporter() (K) - Create DMABUF from V4L2 buffer 7) dma_buf_fd() (K) - Provide FD of the DMABUF to user X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org34 DMABUF Usage Example 2/4 Camera App 8) Request FD->GEM conversion (U) 9) dma_buf_get(fd) (K) - Get DMABUF from FD 10)dma_buf_attach(dma-buf) / dma_buf_map_attachment() (K) - Get Buffer from DMABUF 11) Import as GEM, send user (K) 12) Request GEM object name (U) 13) Return GEM object name (K) 14) Send GEM object name to X (U) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org35 DMABUF Usage Example 2/4 Camera App 8) Request FD->GEM conversion (U) 9) dma_buf_get(fd) (K) - Get DMABUF from FD 10) dma_buf_attach(dma-buf) / dma_buf_map_attachment() (K) - Get Buffer from DMABUF 11) Import as GEM, send user (K) 12) Request GEM object name (U) 13) Return GEM object name (K) 14) Send GEM object name to X (U) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org36 DMABUF Usage Example 3/4 X server 15) Convert given GEM object name to GEM. Display its content. (U & K) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org37 DMABUF Usage Example 3/4 X server 15) Convert given GEM object name to GEM. Display its content. (U & K) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org38 DMABUF Usage Example 4/4 Close after usage 1) “Free” the GEM object (U) 2) Remove reference from the DMABUF (K) 3) Close(DMABUF-FD) at cam app (U) 4) (No more reference to DMABUF) Release callback executed (K) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org39 DMABUF Usage Example 4/4 Close after usage 1) “Free” the GEM object (U) 2) Remove reference from the DMABUF 3) Close(DMABUF-FD) at cam app (U) 4) (No more reference to DMABUF) Release callback executed (K) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org40 DMABUF Usage Example 4/4 Close after usage 1) “Free” the GEM object (U) 2) Remove reference from the DMABUF 3) Close(DMABUF-FD) at cam app (U) 4) (No more reference to DMABUF) Release callback executed (K) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
tizen.org41 Sorry, Back to DRM/GEM… • DMABUF Import / Export needs to be added • DRM_IOCTL_PRIME_HANDLE_TO_FD • Export gem handle into dmabuf fd • DRM_IOCTL_PRIME_FD_TO_HANDLE • Import dmabuf fd into gem handle
tizen.org42 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel – To Run Tizen OS. • Basic Memory Management Components • Buffer Sharing (UMM/DMABUF) • Buffer Allocation (UMM/DMAAPI) • Buffer Synchronization • Memory Size Optimization
tizen.org43 Tizen Kernel Memory Management Unified Memory Management (UMM) • Introduced by Jesse Barker, 2011 • Includes – DMABUF (sharing buffer) – DMA Mapping API for Allocation. – CMA (Contiguous Memory Allocator) IOMMU (MMU for I/O devs)
tizen.org44 Tizen Kernel MM, UMM DMA Mapping API @ Tizen Reference DRM / GEM DMA Mapping Framework IOMMUBuddyCMA IOMMU Common Driver System Memory V4L2 / VB2 1 2 1: “Exynos-DRM” provides unified device address space for all DRM devices 2: Each V4L2 device has its own device address space 2 2
tizen.org45 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel – To Run Tizen OS. • Basic Memory Management Components • Buffer Sharing (UMM/DMABUF) • Buffer Allocation (UMM/DMAAPI) • Buffer Synchronization • Memory Size Optimization
tizen.org46 Tizen Kernel Memory Management We Still Have A HUGE Problem!
tizen.org47 Tizen Kernel MM, Buffer Sync. Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF Synchronization?
tizen.org48 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU GPU Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1
tizen.org49 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU GPU Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1
tizen.org50 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU GPU Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1
tizen.org51 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU GPU Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1
tizen.org52 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU GPU Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1
tizen.org53 Tizen Kernel MM, Buffer Sync. What’s the problem? Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1 How to ensure CPU won’t use buffer 1 until step 4? (esp., a user process)
tizen.org54 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU, Thread1 GPU • What if there are two threads using GPU? – And if the two look at the same buffer? • Not even aware of it? CPU, Thread2
tizen.org55 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU, Thread1 GPU • What if the two DMA devices (GPU, FIMC) share buffer? – But hard to know it at drivers or user threads. – FIMC never knows when GPU finishes – FIMC never knows when Threads1 stops using “buffer1” – Threads2 never know when GPU stops using “buffer1” CPU, Thread2 FIMC-IPP (Image Post Processing)
tizen.org56 Tizen Kernel Memory Management Buffer Synchronization • TGL (Tizen Global Lock) @ Tizen 2.0 – … Let userspace handle the issue … – Kernel patch required. • Sync Framework (Google) – Jun, 2012. Resources w/o DMABUF (similar with TGL) • KDS (Kernel Dependency System, ARM) – May 2012 / DMABUF-compatible • DMA Fence Framework (Canonical/TI) – Aug 2012 / DMABUF-compatible – Work-In-Progress
tizen.org57 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel • Memory Size Optimization – To Run w/ More Devices
tizen.org58 Memory Size Optimization: Challenge • A device with 512MB RAM • Graphics/Multimedia devices want 400MB reserved – No IOMMU. • The userspace wants to run a web browser and HTML5 app! •REALLY POSSIBLE?
tizen.org59 Memory Size Optimization: Challenge • A device with 512MB RAM • Graphics/Multimedia devices want 400MB reserved – No IOMMU • The userspace wants to run a web browser and HTML5 app! •REALLY POSSIBLE? •WHY NOT?
tizen.org60 Minimize H/W Reserved Memory CMA (Contiguous Memory Allocator) • Camera wants to reserve 400MB. – No IOMMU: too primitive to use paging. • You don’t use cameras while web browsing. – Free the 400MB reserved to camera, let userspace use it. • CMA can do.
tizen.org61 Minimize H/W Reserved Memory • (Obsolete/Low-cost) ARM SoC’s DMA devices – No IOMMU – Use physically continuous memory chunk. • No paging • Potentially, a lot of memory is wasted. BUS CPU MMU L2 Cache DRAM Controller DRAMDRAMDRAM CPUCPU GPU
tizen.org62 Minimize H/W Reserved Memory IOMMU • (Modern/Mid-High-cost) ARM SoC’s DMA devices – IOMMU for DMA devices! – Use paging! • Dynamically allocate & free BUS CPU MMU L2 Cache DRAM Controller DRAMDRAMDRAM CPUCPU GPU IOMMU
tizen.org63 Minimize H/W Reserved Memory IOMMU • Exynos4210/4412-based Tizen mobile device – Achieved near-zero reserved memory. – At the “menuscreen”, only around ~40MB allocated for display and GPU. – Unfortunately, not included in the Tizen reference kernel. • But easily supported by Vanilla kernel for Tizen reference boards
tizen.org64 Minimize H/W Reserved Memory Results with 1GB RAM Device -sh-4.1# free total used free shared buffers cached Mem: 1029428 302020 727408 0 16164 133084 -/+ buffers/cache: 152772 876656 Swap: 0 0 0 -sh-4.1# uname -a Linux localhost 3.8.3-00841-gc1981b1 #34 SMP PREEMPT Wed Apr 10 10:24:04 KST 2013 armv7l GNU/Linux Mostly available to kernel (1005.3 of 1024MB) Even with Tizen loaded, mostly free to users (856MB of 1GB)
tizen.org65 Memory Size Optimization: Still Hungry? • Further optimization required for low-budget devices – Running full features with 512MB? – Or even < 512MB? • What about multi-tasking or “background apps”?
tizen.org66 Memory Size Optimization: SWAP • Swap – zRAM • Used in some Samsung mobile devices successfully – Or even swap-to-flash?
tizen.org67 Memory Size Optimization: Multitasking / Background Apps • Limit multitasking & background-app support • Pseudo multitasking by freezing background apps
Thank you!

More Related Content

PPT
NVIDIA OpenGL 4.6 in 2017
Mark Kilgard
 
PDF
Design and Concepts of Android Graphics
National Cheng Kung University
 
PDF
Tizen Window System
Eun Young Lee
 
PDF
Camera2 API, SHIM, and HAL 3.2 in Android 5.1
Cheng Hsien Chen
 
PDF
The Technology of Uncharted: Drake’s Fortune
Naughty Dog
 
PDF
BKK16-315 Graphics Stack Update
Linaro
 
PPTX
Optimizing the Graphics Pipeline with Compute, GDC 2016
Graham Wihlidal
 
PPTX
OpenGL 4.5 Update for NVIDIA GPUs
Mark Kilgard
 
NVIDIA OpenGL 4.6 in 2017
Mark Kilgard
 
Design and Concepts of Android Graphics
National Cheng Kung University
 
Tizen Window System
Eun Young Lee
 
Camera2 API, SHIM, and HAL 3.2 in Android 5.1
Cheng Hsien Chen
 
The Technology of Uncharted: Drake’s Fortune
Naughty Dog
 
BKK16-315 Graphics Stack Update
Linaro
 
Optimizing the Graphics Pipeline with Compute, GDC 2016
Graham Wihlidal
 
OpenGL 4.5 Update for NVIDIA GPUs
Mark Kilgard
 

What's hot (20)

PPT
Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)
repii
 
PDF
The Production and Visual FX of Killzone Shadow Fall
Guerrilla
 
PPT
How we optimized our Game - Jake & Tess' Finding Monsters Adventure
Felipe Lira
 
PDF
The Android graphics path, in depth
Chris Simmonds
 
PDF
Book of the Dead: Environmental Design, Tools, and Techniques for Photo-Real ...
Unity Technologies
 
PDF
Executable Bloat - How it happens and how we can fight it
Electronic Arts / DICE
 
PDF
Understanding binder in android
Haifeng Li
 
PPTX
Audio compression
Madhawa Gunasekara
 
PDF
Physically Based Lighting in Unreal Engine 4
Lukas Lang
 
PPTX
Approaching zero driver overhead
Cass Everitt
 
PDF
Advanced Scenegraph Rendering Pipeline
Narann29
 
PDF
Multimedia software hardware
gopinathselvi
 
PPTX
Android graphic system (SurfaceFlinger) : Design Pattern's perspective
Bin Chen
 
PPTX
Multimedia chapter 5
PrathimaBaliga
 
PPTX
Unity3D Programming
Michael Ivanov
 
PPTX
RISC-V Boot Process: One Step at a Time
Atish Patra
 
PDF
Xen and the art of embedded virtualization (ELC 2017)
Stefano Stabellini
 
PDF
Tizen 3.0's Window System Integration Layer of OpenGLES/EGL & Vulkan Driver
Ryo Jin
 
PDF
OpenGL 4.4 - Scene Rendering Techniques
Narann29
 
PDF
Hardware Acceleration for Machine Learning
CastLabKAIST
 
Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)
repii
 
The Production and Visual FX of Killzone Shadow Fall
Guerrilla
 
How we optimized our Game - Jake & Tess' Finding Monsters Adventure
Felipe Lira
 
The Android graphics path, in depth
Chris Simmonds
 
Book of the Dead: Environmental Design, Tools, and Techniques for Photo-Real ...
Unity Technologies
 
Executable Bloat - How it happens and how we can fight it
Electronic Arts / DICE
 
Understanding binder in android
Haifeng Li
 
Audio compression
Madhawa Gunasekara
 
Physically Based Lighting in Unreal Engine 4
Lukas Lang
 
Approaching zero driver overhead
Cass Everitt
 
Advanced Scenegraph Rendering Pipeline
Narann29
 
Multimedia software hardware
gopinathselvi
 
Android graphic system (SurfaceFlinger) : Design Pattern's perspective
Bin Chen
 
Multimedia chapter 5
PrathimaBaliga
 
Unity3D Programming
Michael Ivanov
 
RISC-V Boot Process: One Step at a Time
Atish Patra
 
Xen and the art of embedded virtualization (ELC 2017)
Stefano Stabellini
 
Tizen 3.0's Window System Integration Layer of OpenGLES/EGL & Vulkan Driver
Ryo Jin
 
OpenGL 4.4 - Scene Rendering Techniques
Narann29
 
Hardware Acceleration for Machine Learning
CastLabKAIST
 
Ad

Similar to Memory Management in TIZEN - Samsung SW Platform Team (20)

PDF
Kernel Recipes 2014 - The Linux graphics stack and Nouveau driver
Anne Nicolas
 
PPTX
Add sale davinci
Akash Sahoo
 
PDF
0xdroid osdc-2010-100426084937-phpapp02
chon2010
 
PDF
Porting Android
Opersys inc.
 
PDF
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
The Linux Foundation
 
PDF
Accelerate graphics performance with ozone-gbm on Intel based Linux desktop s...
Joone Hur
 
PDF
Porting Tizen-IVI 3.0 to an ARM based SoC Platform
Ryo Jin
 
PDF
Memory Management in Android
Opersys inc.
 
PDF
There is more to C
Juraj Michálek
 
PDF
Android Development Tools
Dominik Helleberg
 
PDF
Kernel Recipes 2016 - Upstream Kernel Graphics is (Finally) Winning
Anne Nicolas
 
PDF
XPDDS17: Keynote: Shared Coprocessor Framework on ARM - Oleksandr Andrushchen...
The Linux Foundation
 
PDF
Flowframes
Artmiker Studios
 
PDF
Porting Android
Opersys inc.
 
PDF
Porting Android ABS 2011
Opersys inc.
 
PDF
Memory Management in Android
Opersys inc.
 
PDF
“Programming Vision Pipelines on AMD’s AI Engines,” a Presentation from AMD
Edge AI and Vision Alliance
 
PDF
Memory Management in Android
Opersys inc.
 
PDF
Android Internals
Opersys inc.
 
PDF
The abcs of gpu
Liang Yan
 
Kernel Recipes 2014 - The Linux graphics stack and Nouveau driver
Anne Nicolas
 
Add sale davinci
Akash Sahoo
 
0xdroid osdc-2010-100426084937-phpapp02
chon2010
 
Porting Android
Opersys inc.
 
XPDDS18: Design and Implementation of Automotive: Virtualization Based on Xen...
The Linux Foundation
 
Accelerate graphics performance with ozone-gbm on Intel based Linux desktop s...
Joone Hur
 
Porting Tizen-IVI 3.0 to an ARM based SoC Platform
Ryo Jin
 
Memory Management in Android
Opersys inc.
 
There is more to C
Juraj Michálek
 
Android Development Tools
Dominik Helleberg
 
Kernel Recipes 2016 - Upstream Kernel Graphics is (Finally) Winning
Anne Nicolas
 
XPDDS17: Keynote: Shared Coprocessor Framework on ARM - Oleksandr Andrushchen...
The Linux Foundation
 
Flowframes
Artmiker Studios
 
Porting Android
Opersys inc.
 
Porting Android ABS 2011
Opersys inc.
 
Memory Management in Android
Opersys inc.
 
“Programming Vision Pipelines on AMD’s AI Engines,” a Presentation from AMD
Edge AI and Vision Alliance
 
Memory Management in Android
Opersys inc.
 
Android Internals
Opersys inc.
 
The abcs of gpu
Liang Yan
 
Ad

More from Ryo Jin (20)

PDF
Why is EFL used on Tizen?
Ryo Jin
 
PDF
Samsung Z4 User Manual
Ryo Jin
 
PDF
Samsung ARTIK 050 (ARTIK ZERO) Modules Data Sheet
Ryo Jin
 
PDF
Introduction to Watch Face Development with Tizen Studio
Ryo Jin
 
PDF
Panduan Penggunaan Perangkat Wearable Tizen
Ryo Jin
 
PDF
Cara Menggunakan Smartphone Tizen
Ryo Jin
 
PDF
Gear Fit2 Watchface Design Guide
Ryo Jin
 
PDF
Samsung Indonesia: Tizen Store
Ryo Jin
 
PDF
Samsung Indonesia: Tizen Wearables
Ryo Jin
 
PDF
Samsung Indonesia: Tizen Web Apps
Ryo Jin
 
PDF
Samsung Indonesia: Tizen Native App
Ryo Jin
 
PDF
Samsung Indonesia: Tizen Platform Overview and IoT
Ryo Jin
 
PDF
Russian Tizen Project
Ryo Jin
 
PDF
Samsung SM-R360 Tizen User Manual
Ryo Jin
 
PDF
Tizen Micro Profile for IoT device
Ryo Jin
 
PDF
Panduan Dasar Pemrograman Tizen
Ryo Jin
 
PDF
The Story of Enlightenment, EFL, Tizen and Wayland
Ryo Jin
 
PDF
Tizen PASS
Ryo Jin
 
PDF
Tizen PASS
Ryo Jin
 
PDF
Samsung Gear UI Design Guidelines
Ryo Jin
 
Why is EFL used on Tizen?
Ryo Jin
 
Samsung Z4 User Manual
Ryo Jin
 
Samsung ARTIK 050 (ARTIK ZERO) Modules Data Sheet
Ryo Jin
 
Introduction to Watch Face Development with Tizen Studio
Ryo Jin
 
Panduan Penggunaan Perangkat Wearable Tizen
Ryo Jin
 
Cara Menggunakan Smartphone Tizen
Ryo Jin
 
Gear Fit2 Watchface Design Guide
Ryo Jin
 
Samsung Indonesia: Tizen Store
Ryo Jin
 
Samsung Indonesia: Tizen Wearables
Ryo Jin
 
Samsung Indonesia: Tizen Web Apps
Ryo Jin
 
Samsung Indonesia: Tizen Native App
Ryo Jin
 
Samsung Indonesia: Tizen Platform Overview and IoT
Ryo Jin
 
Russian Tizen Project
Ryo Jin
 
Samsung SM-R360 Tizen User Manual
Ryo Jin
 
Tizen Micro Profile for IoT device
Ryo Jin
 
Panduan Dasar Pemrograman Tizen
Ryo Jin
 
The Story of Enlightenment, EFL, Tizen and Wayland
Ryo Jin
 
Tizen PASS
Ryo Jin
 
Tizen PASS
Ryo Jin
 
Samsung Gear UI Design Guidelines
Ryo Jin
 

Recently uploaded (20)

PDF
The Future of Mobile Is Context-Aware—Are You Ready?
iProgrammer Solutions Private Limited
 
PDF
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
PPTX
What-is-the-World-Wide-Web -- Introduction
tonifi9488
 
PDF
SparkLabs Primer on Artificial Intelligence 2025
SparkLabs Group
 
PDF
Oracle AI Vector Search- Getting Started and what's new in 2025- AIOUG Yatra ...
Sandesh Rao
 
PDF
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
PDF
Software Development Methodologies in 2025
KodekX
 
PPTX
AI and Robotics for Human Well-being.pptx
JAYMIN SUTHAR
 
PDF
NewMind AI Weekly Chronicles - July'25 - Week IV
NewMind AI
 
PDF
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
PPTX
The Future of AI & Machine Learning.pptx
pritsen4700
 
PDF
Security features in Dell, HP, and Lenovo PC systems: A research-based compar...
Principled Technologies
 
PPTX
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
PPTX
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
PDF
Responsible AI and AI Ethics - By Sylvester Ebhonu
Sylvester Ebhonu
 
PDF
How ETL Control Logic Keeps Your Pipelines Safe and Reliable.pdf
Stryv Solutions Pvt. Ltd.
 
PPTX
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
PPTX
Simple and concise overview about Quantum computing..pptx
mughal641
 
PDF
Economic Impact of Data Centres to the Malaysian Economy
flintglobalapac
 
PDF
MASTERDECK GRAPHSUMMIT SYDNEY (Public).pdf
Neo4j
 
The Future of Mobile Is Context-Aware—Are You Ready?
iProgrammer Solutions Private Limited
 
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
What-is-the-World-Wide-Web -- Introduction
tonifi9488
 
SparkLabs Primer on Artificial Intelligence 2025
SparkLabs Group
 
Oracle AI Vector Search- Getting Started and what's new in 2025- AIOUG Yatra ...
Sandesh Rao
 
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
Software Development Methodologies in 2025
KodekX
 
AI and Robotics for Human Well-being.pptx
JAYMIN SUTHAR
 
NewMind AI Weekly Chronicles - July'25 - Week IV
NewMind AI
 
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
The Future of AI & Machine Learning.pptx
pritsen4700
 
Security features in Dell, HP, and Lenovo PC systems: A research-based compar...
Principled Technologies
 
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
Responsible AI and AI Ethics - By Sylvester Ebhonu
Sylvester Ebhonu
 
How ETL Control Logic Keeps Your Pipelines Safe and Reliable.pdf
Stryv Solutions Pvt. Ltd.
 
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
Simple and concise overview about Quantum computing..pptx
mughal641
 
Economic Impact of Data Centres to the Malaysian Economy
flintglobalapac
 
MASTERDECK GRAPHSUMMIT SYDNEY (Public).pdf
Neo4j
 

Memory Management in TIZEN - Samsung SW Platform Team

  • 1. Memory Management in Tizen SW Platform Team, SW R&D Center
  • 2. tizen.org2 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel • Memory Size Optimization
  • 4. tizen.org4 Tizen Kernel Overview Kernel Kernel OAL (OEM Adaptation Layer) Telephony Plug-ins GStreamer Plug-ins Sensor Plug-ins System Plug-ins OpenGL ES/EGL Graphics Driver Graphics Memory Management IOMMU DMA BUF Multimedia Power Management CPUfre q Devf req Ther mal … DRM Link1 Link2 V4L2 Storage Block Layer MMC/Flash Input, Sensor, … Peripheral CMA OpenGL ES 1.0/2.0 X11 GStreamerEFL . . . Multimedia FW Telephony FW . . .Sensor FW System FW . . . More at http://source.tizen.org/documentation/porting-guide Core DRM Link1: http:// elinux.org/images/7/71/Elce11_dae.pdf DRM Link2: http://download.tizen.org/misc/media/conference2012/wednesday/ballroom-c/2012-05-09-1330-1410- the_drm_(direct_rendering_manager)_of_tizen_kernel.pdf
  • 5. tizen.org5 Tizen Kernel Overview Kernel Kernel OAL (OEM Adaptation Layer) Telephony Plug-ins GStreamer Plug-ins Sensor Plug-ins System Plug-ins OpenGL ES/EGL Graphics Driver Graphics Memory Management IOMM U DMA BUF Multimedia Power Management CPUfre q Devf req Ther mal …DRM Link1 Link2 V4L2 Storage Block Layer MMC/Flash Input, Sensor, … Peripheral CM A OpenGL ES 1.0/2.0 X11 GStreamerEFL . . . Multimedia FW Telephony FW . . .Sensor FW System FW . . . More at http://source.tizen.org/documentation/porting-guide Core Service Layer DRM Link1: http:// elinux.org/images/7/71/Elce11_dae.pdf DRM Link2: http://download.tizen.org/misc/media/conference2012/wednesday/ballroom-c/2012-05-09-1330-1410- the_drm_(direct_rendering_manager)_of_tizen_kernel.pdf • Memory Management in Tizen – Coupled with Graphics & Multimedia devices. • Graphics & Multimedia devices = DMA devices with HUGE buffers
  • 6. tizen.org6 Tizen Kernel Overview Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF
  • 7. tizen.org7 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel – To Run Tizen OS. • Basic Memory Management Components • Buffer Sharing (UMM/DMABUF) • Buffer Allocation (UMM/DMAAPI) • Buffer Synchronization • Memory Size Optimization
  • 8. tizen.org8 Tizen Kernel MM, Graphics (DRM) Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF
  • 9. tizen.org9 Graphics: DRM/GEM (Graphics Execution Manager) drm_fb_helperdrm_mode_config Linux DRM Common Framework GEM drm_framebuffer fb_infoCrtc Connector Encoder GEM Allocator Plane Common Specific drm_framebufferCrtc Connector Encoder Plane
  • 10. tizen.org10 Graphics: DRM/GEM (Graphics Execution Manager) drm_fb_helperdrm_mode_config Linux DRM Common Framework GEM drm_framebuffer fb_infoCrtc Connector Encoder GEM Allocator Plane Common Specific drm_framebufferCrtc Connector Encoder Plane Graphics Execution Manager • Framework developed by Intel • To manage graphics memory • Framework for buffer management. • Allocation and sharing.
  • 11. tizen.org11 Graphics: DRM/GEM Allocation • GEM Allocation steps @ Tizen (Generic) 1. DRM_IOCTL_MODE_CREATE_DUMB • Create GEM object(global) & user GEM handle(per process) • dumb_create() of struct drm_driver • No physical memory allocated. 2. DRM_IOCTL_MODE_MAP_DUMB • Create fake mmap offset of a gem object and relay the object to user • A hash key of the gem object. • dumb_map_offset() of struct drm_driver 3. MMAP • Request mmap based on the hash key as the offset • Create user address space • Setup cache attribute. • Not mapped to physical memory, yet
  • 12. tizen.org12 Graphics: DRM/GEM Allocation • GEM Allocation steps @ Tizen (Generic) 4. On-demand Paging – Implement & Register a fault handler that • With a page fault, allocate a page and map the page. – vma->vm_ops->fault = xxx_drm_gem_fault 5. Use! 6. DRM_IOCTL_MODE_DESTROY_DUMB – Remove GEM handle & object – Free memory – Implement dumb_destroy() of struct drm_driver
  • 13. tizen.org13 Graphics: DRM/GEM Allocation • GEM Allocation steps @ Tizen (Exynos Only) 1. DRM_IOCTL_EXYNOS_GEM_CREATE • Only use user-desired size and buffer types. • Create gem object(global) & user gem handle(per process) • physical memory allocated. 2. DRM_IOCTL_EXYNOS_GEM_MMAP • Create user address space • Map the user address space to physical memory • LIBDRM of Exynos uses these APIs, not the generic.
  • 14. tizen.org14 Graphics: DRM/GEM Sharing • GEM Sharing @ Tizen • DRM_IOCTL_GEM_FLINK – “I will share this GEM to others.” – Create GEM object name for the given GEM handle • Global key vaue for sharing • DRM_IOCTL_GEM_OPEN – “I want to use the shared GEM.” – Create GEM handle based on the given GEM object name • DRM_IOCTL_GEM_CLOSE • You don’t need to implement. It’s already there with DRM. GEM Create GEM FLINK GEM OPEN gem handle gem object name gem handle Process 1 Process 2
  • 15. tizen.org15 Tizen Kernel MM, Multimedia (V4L2/VB2) Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF
  • 16. tizen.org16 Multimedia: V4L2/VB2 • Tizen recommends to use V4L2 at Tizen kernel for Multimedia devices • Video input (codec & camera) & Radio • However, as long as the kernel has: – Gstreamer/OpenMAX plugins – A method to share with other F/W via DMABUF of UMM, • Tizen multimedia works. • If V4L2/VB2 is used, things get easier.
  • 17. tizen.org17 Tizen Kernel MM, OpenGL/G3D-GPU Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF
  • 18. tizen.org18 OpenGL / G3D-GPU • Most ARM SoC GPUs (MALI, SGX, …) use their own memory manager • E.g., Exynos4412/4210 Tizen Reference Kernel has Mali w/ UMP. – Mali DDK modified to be compatible with UMM-DMABUF. • If GPU drivers use DRM, it would be great. • (and make them GPL)
  • 19. tizen.org19 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel – To Run Tizen OS. • Basic Memory Management Components • Buffer Sharing (UMM/DMABUF) • Buffer Allocation (UMM/DMAAPI) • Buffer Synchronization • Memory Size Optimization
  • 20. tizen.org20 NEED FOR SOMETHING, A Scenario Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc Camera fetches a video stream This scenario is simplified for presentation. Not an actual example of Tizen
  • 21. tizen.org21 NEED FOR SOMETHING, A Scenario Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc Camera fetches a video stream Draws its own images This scenario is simplified for presentation. Not an actual example of Tizen
  • 22. tizen.org22 NEED FOR SOMETHING, A Scenario Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc Camera fetches a video stream Draws its own images GPU Merges the two / Puts the merged image This scenario is simplified for presentation. Not an actual example of Tizen
  • 23. tizen.org23 NEED FOR SOMETHING, A Scenario Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc Camera fetches a video stream Draws its own images DRM-FIMD(display) Output This scenario is simplified for presentation. Not an actual example of Tizen GPU Merges the two / Puts the merged image
  • 24. tizen.org24 NEED FOR SOMETHING, A Scenario Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc Camera fetches a video stream Draws its own images DRM-FIMD(display) Output At a VB2 buffer At a GEM buffer GEM buffer This scenario is simplified for presentation. Not an actual example of Tizen GPU Merges the two / Puts the merged image From GEM + VB2 into its own (e.g., UMP)
  • 25. tizen.org25 Tizen Kernel MM, UMM Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF A mechanism to share between DRM, V4L2, and others w/o memcpy
  • 26. tizen.org26 Tizen Kernel MM, UMM Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF Requirement from Tizen platform and hardware • Different Memory Managers: GEM, VB2, GPU-adhoc, … • Share buffers • w/o memcpy • From and to users • Never expose directly to users (e.g., physical address)  UMM DMABUF!
  • 27. tizen.org27 Tizen Kernel Memory Management Unified Memory Management (UMM) • Introduced by Jesse Barker, 2011 • Includes – DMABUF (sharing buffer) – DMA Mapping API for Allocation. – CMA (Contiguous Memory Allocator) IOMMU (MMU for I/O devs)
  • 28. tizen.org28 Tizen Kernel MM, UMM DMA Buffer Sharing: DMABUF • Export – GEM/VB2/… object  DMABUF • Import – DMABUF  GEM/VB2/… object • Userspace sees DMABUF as a File Descriptor
  • 29. tizen.org29 DMA Buffer Sharing: DMABUF, Example Camera  Display X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 30. tizen.org30 DMA Buffer Sharing: DMABUF, Example Conceptual Data Flow X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 31. tizen.org31 DMABUF Usage Example 1/4 Camera App 1) Request V4L2 camera buffer (U) 2) Allocate CMA buffer (K) 3) Request a camera frame at the V4L2 buffer (U) 4) Store the camera frame & Notify user (K) 5) Request DMABUF export for the V4L2 camera buffer (U) 6) dma_buf_exporter() (K) - Create DMABUF from V4L2 buffer 7) dma_buf_fd() (K) - Provide FD of the DMABUF to user X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 32. tizen.org32 DMABUF Usage Example 1/4 Camera App 1) Request V4L2 camera buffer (U) 2) Allocate CMA buffer (K) 3) Request a camera frame at the V4L2 buffer (U) 4) Store the camera frame & Notify user (K) 5) Request DMABUF export for the V4L2 camera buffer (U) 6) dma_buf_exporter() (K) - Create DMABUF from V4L2 buffer 7) dma_buf_fd() (K) - Provide FD of the DMABUF to user X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 33. tizen.org33 DMABUF Usage Example 1/4 Camera App 1) Request V4L2 camera buffer (U) 2) Allocate CMA buffer (K) 3) Request a camera frame at the V4L2 buffer (U) 4) Store the camera frame & Notify user (K) 5) Request DMABUF export for the V4L2 camera buffer (U) 6) dma_buf_exporter() (K) - Create DMABUF from V4L2 buffer 7) dma_buf_fd() (K) - Provide FD of the DMABUF to user X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 34. tizen.org34 DMABUF Usage Example 2/4 Camera App 8) Request FD->GEM conversion (U) 9) dma_buf_get(fd) (K) - Get DMABUF from FD 10)dma_buf_attach(dma-buf) / dma_buf_map_attachment() (K) - Get Buffer from DMABUF 11) Import as GEM, send user (K) 12) Request GEM object name (U) 13) Return GEM object name (K) 14) Send GEM object name to X (U) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 35. tizen.org35 DMABUF Usage Example 2/4 Camera App 8) Request FD->GEM conversion (U) 9) dma_buf_get(fd) (K) - Get DMABUF from FD 10) dma_buf_attach(dma-buf) / dma_buf_map_attachment() (K) - Get Buffer from DMABUF 11) Import as GEM, send user (K) 12) Request GEM object name (U) 13) Return GEM object name (K) 14) Send GEM object name to X (U) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 36. tizen.org36 DMABUF Usage Example 3/4 X server 15) Convert given GEM object name to GEM. Display its content. (U & K) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 37. tizen.org37 DMABUF Usage Example 3/4 X server 15) Convert given GEM object name to GEM. Display its content. (U & K) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 38. tizen.org38 DMABUF Usage Example 4/4 Close after usage 1) “Free” the GEM object (U) 2) Remove reference from the DMABUF (K) 3) Close(DMABUF-FD) at cam app (U) 4) (No more reference to DMABUF) Release callback executed (K) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 39. tizen.org39 DMABUF Usage Example 4/4 Close after usage 1) “Free” the GEM object (U) 2) Remove reference from the DMABUF 3) Close(DMABUF-FD) at cam app (U) 4) (No more reference to DMABUF) Release callback executed (K) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 40. tizen.org40 DMABUF Usage Example 4/4 Close after usage 1) “Free” the GEM object (U) 2) Remove reference from the DMABUF 3) Close(DMABUF-FD) at cam app (U) 4) (No more reference to DMABUF) Release callback executed (K) X server Video encoder Xvsink Camera src User space Kernel space DRM KMS GEM Codec CameraV4L2 V4L2 struct dma_buf Display Codec Camera Memory Camera App FD HW
  • 41. tizen.org41 Sorry, Back to DRM/GEM… • DMABUF Import / Export needs to be added • DRM_IOCTL_PRIME_HANDLE_TO_FD • Export gem handle into dmabuf fd • DRM_IOCTL_PRIME_FD_TO_HANDLE • Import dmabuf fd into gem handle
  • 42. tizen.org42 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel – To Run Tizen OS. • Basic Memory Management Components • Buffer Sharing (UMM/DMABUF) • Buffer Allocation (UMM/DMAAPI) • Buffer Synchronization • Memory Size Optimization
  • 43. tizen.org43 Tizen Kernel Memory Management Unified Memory Management (UMM) • Introduced by Jesse Barker, 2011 • Includes – DMABUF (sharing buffer) – DMA Mapping API for Allocation. – CMA (Contiguous Memory Allocator) IOMMU (MMU for I/O devs)
  • 44. tizen.org44 Tizen Kernel MM, UMM DMA Mapping API @ Tizen Reference DRM / GEM DMA Mapping Framework IOMMUBuddyCMA IOMMU Common Driver System Memory V4L2 / VB2 1 2 1: “Exynos-DRM” provides unified device address space for all DRM devices 2: Each V4L2 device has its own device address space 2 2
  • 45. tizen.org45 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel – To Run Tizen OS. • Basic Memory Management Components • Buffer Sharing (UMM/DMABUF) • Buffer Allocation (UMM/DMAAPI) • Buffer Synchronization • Memory Size Optimization
  • 46. tizen.org46 Tizen Kernel Memory Management We Still Have A HUGE Problem!
  • 47. tizen.org47 Tizen Kernel MM, Buffer Sync. Kernel / Device Drivers (BSP) Kernel / Core & Subsystem X Server DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL Evas 2D GPU backend X Video Drivers Userptr NEON 2D GPU backend PIXMAN EXA V4L2 VB2 Codec Camera GStreamer OpenMAX Libdrm XV Extension UMM Open GL Multimedia FW XvImageSink Core External Library Core Framework OEM Adaptation Kernel Kernel GPU DDK (G3D) Radio * Images from MS Office clipart and Samsung Gstreamer- CameraSrc DMABUF Synchronization?
  • 48. tizen.org48 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU GPU Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1
  • 49. tizen.org49 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU GPU Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1
  • 50. tizen.org50 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU GPU Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1
  • 51. tizen.org51 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU GPU Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1
  • 52. tizen.org52 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU GPU Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1
  • 53. tizen.org53 Tizen Kernel MM, Buffer Sync. What’s the problem? Simple Usage Scenario 1. CPU writes to buffer1. 2. CPU tells GPU to do something on buffer1. 3. GPU does something on buffer1. 4. GPU finishes. 5. CPU reads the buffer1 How to ensure CPU won’t use buffer 1 until step 4? (esp., a user process)
  • 54. tizen.org54 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU, Thread1 GPU • What if there are two threads using GPU? – And if the two look at the same buffer? • Not even aware of it? CPU, Thread2
  • 55. tizen.org55 Tizen Kernel MM, Buffer Sync. What’s the problem? BUS DRAMDRAMDRAM CPU, Thread1 GPU • What if the two DMA devices (GPU, FIMC) share buffer? – But hard to know it at drivers or user threads. – FIMC never knows when GPU finishes – FIMC never knows when Threads1 stops using “buffer1” – Threads2 never know when GPU stops using “buffer1” CPU, Thread2 FIMC-IPP (Image Post Processing)
  • 56. tizen.org56 Tizen Kernel Memory Management Buffer Synchronization • TGL (Tizen Global Lock) @ Tizen 2.0 – … Let userspace handle the issue … – Kernel patch required. • Sync Framework (Google) – Jun, 2012. Resources w/o DMABUF (similar with TGL) • KDS (Kernel Dependency System, ARM) – May 2012 / DMABUF-compatible • DMA Fence Framework (Canonical/TI) – Aug 2012 / DMABUF-compatible – Work-In-Progress
  • 57. tizen.org57 Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel • Memory Size Optimization – To Run w/ More Devices
  • 58. tizen.org58 Memory Size Optimization: Challenge • A device with 512MB RAM • Graphics/Multimedia devices want 400MB reserved – No IOMMU. • The userspace wants to run a web browser and HTML5 app! •REALLY POSSIBLE?
  • 59. tizen.org59 Memory Size Optimization: Challenge • A device with 512MB RAM • Graphics/Multimedia devices want 400MB reserved – No IOMMU • The userspace wants to run a web browser and HTML5 app! •REALLY POSSIBLE? •WHY NOT?
  • 60. tizen.org60 Minimize H/W Reserved Memory CMA (Contiguous Memory Allocator) • Camera wants to reserve 400MB. – No IOMMU: too primitive to use paging. • You don’t use cameras while web browsing. – Free the 400MB reserved to camera, let userspace use it. • CMA can do.
  • 61. tizen.org61 Minimize H/W Reserved Memory • (Obsolete/Low-cost) ARM SoC’s DMA devices – No IOMMU – Use physically continuous memory chunk. • No paging • Potentially, a lot of memory is wasted. BUS CPU MMU L2 Cache DRAM Controller DRAMDRAMDRAM CPUCPU GPU
  • 62. tizen.org62 Minimize H/W Reserved Memory IOMMU • (Modern/Mid-High-cost) ARM SoC’s DMA devices – IOMMU for DMA devices! – Use paging! • Dynamically allocate & free BUS CPU MMU L2 Cache DRAM Controller DRAMDRAMDRAM CPUCPU GPU IOMMU
  • 63. tizen.org63 Minimize H/W Reserved Memory IOMMU • Exynos4210/4412-based Tizen mobile device – Achieved near-zero reserved memory. – At the “menuscreen”, only around ~40MB allocated for display and GPU. – Unfortunately, not included in the Tizen reference kernel. • But easily supported by Vanilla kernel for Tizen reference boards
  • 64. tizen.org64 Minimize H/W Reserved Memory Results with 1GB RAM Device -sh-4.1# free total used free shared buffers cached Mem: 1029428 302020 727408 0 16164 133084 -/+ buffers/cache: 152772 876656 Swap: 0 0 0 -sh-4.1# uname -a Linux localhost 3.8.3-00841-gc1981b1 #34 SMP PREEMPT Wed Apr 10 10:24:04 KST 2013 armv7l GNU/Linux Mostly available to kernel (1005.3 of 1024MB) Even with Tizen loaded, mostly free to users (856MB of 1GB)
  • 65. tizen.org65 Memory Size Optimization: Still Hungry? • Further optimization required for low-budget devices – Running full features with 512MB? – Or even < 512MB? • What about multi-tasking or “background apps”?
  • 66. tizen.org66 Memory Size Optimization: SWAP • Swap – zRAM • Used in some Samsung mobile devices successfully – Or even swap-to-flash?
  • 67. tizen.org67 Memory Size Optimization: Multitasking / Background Apps • Limit multitasking & background-app support • Pseudo multitasking by freezing background apps