"Building Complete Embedded Vision Systems on Linux—From Camera to Display," a Presentation from Montgomery One

"Building Complete Embedded Vision Systems on Linux—From Camera to Display," a Presentation from Montgomery One

• 1.
  © 2019 MontgomeryOne Building Complete Embedded Vision Systems on Linux - Camera to Display Clay D. Montgomery Montgomery One May 2019
• 2.
  © 2019 MontgomeryOne The Big Picture - Overview • Start on a Ubuntu Linux PC to develop and prototype new CV algorithms and applications - Install, build and explore the OpenCV sample programs on a Ubuntu system • Select your Open-Hardware ARM SoC (System on Chip) board and cameras carefully - A good BSP with camera drivers and Yocto support are critical! • Build a custom OS with Yocto Open Embedded Linux for your target board - Use the same open-source libraries on Ubuntu and target systems, if at all possible • Build your application with the Yocto toolchain for your ARM target system • Select Open-Source component libraries to use - Which do you actually need and which are the best for your application? - V4L2, FFmpeg, GStreamer, OpenCV, OpenGL ES, OpenCL and OpenVX? • Explore the options for acceleration on your ARM SoC - Acceleration is required for most CV applications, so plan for it 2
• 3.
  © 2019 MontgomeryOne Popular ARM SoC Boards for Vision 3 RaspberryPi 3 TI BeagleBoard NXP i.MX8 Nvidia Jetson TX2 i.MX6 WandBoard with MIPI Camera
• 4.
  © 2019 MontgomeryOne Exploring Open Source Video Components for Linux
• 5.
  © 2019 MontgomeryOne An Example - Vision Components for Lightwing 5
• 6.
  © 2019 MontgomeryOne V4L2 – Video for Linux (Version 2) 6 • Negotiates compatible video formats and manages buffers and events between apps and drivers • The de-facto standard low-level video API for Linux • The most widely used API for video input on Linux systems • Most Linux distros and BSPs provide drivers for cameras and TV tuners for V4L2 • Version 1 is obsolete, but still used by the OpenCV VideoCapture class • Well supported by most SoC vendors that have integrated MIPI camera support • The UVC standard (USB Video Class for web cams) is newer and still uses V4L2 on Linux
• 7.
  © 2019 MontgomeryOne GStreamer • Negotiates compatible video formats and builds pipelines and filter-graphs to connect plug-in components and apps to the V4L2 API • The de-facto standard high-level audio/video API for Linux • The most widely used API for video pipelines on Linux systems • Used by OpenCV's VideoCapture class to work around video format issues, such as color space and RGB/BGR mismatch issues gst-launch-1.0 autovideosrc ! videoconvert ! autovideosink ./videocapture_basic "imxv4l2videosrc device=/dev/video0 ! videoconvert ! appsink" • Well supported by some SoC vendors that have integrated VPUs and codecs - VPUs can be very difficult to exploit without good drivers for GStreamer - This is a major differentiator among SoC vendors and should be evaluated carefully 7
• 8.
  © 2019 MontgomeryOne OpenCV • 2D and 3D feature toolkits • Egomotion estimation • Facial recognition system • Gesture recognition • Human–computer interaction (HCI) • Motion tracking and understanding • Object segmentation and recognition • Depth perception from 2 cameras • Structure from motion (SFM) • Statistical machine learning and DNN • Support for OpenCL and CUDA accelerators 8 • Not a Khronos standard, but mostly open-source • Began as Intel Research open initiative in 1999 • Intel donated OpenCV to non-profit OpenCV.org • Most comprehensive and widely used CV library • Available on most platforms and languages today • Lots of sample programs in C/C++ and Python • Great place to start to prototype a new design • Latest 4.x available, but not much help on ARM FeaturesOrigin and Status Issues • Typically too slow for camera video (on ARM) • Samples require X11 on Linux • Bugs in VideoCapture class on Linux • Requires video in BGR format
• 9.
  © 2019 MontgomeryOne FFmpeg • A set of libraries for encoding, decoding and converting audio and video files and streams, such as JPEG, MPEG and H.264, etc. • Supported on a very wide range of platforms • Very limited support for accelerators on ARM SoCs (NEON) - Most ARM SoC vendors do not provide drivers to accelerate FFmpeg - FFmpeg codecs are typically too slow for camera video on ARM SoCs - But, still useful for de-muxing, removing containers, etc. 9
• 10.
  © 2019 MontgomeryOne OpenGL ES • The original Khronos consortium standard for 3D graphics on SoCs • One of the most successful and widely adopted APIs for embedded multimedia ever • Version 1.1 is obsolete • Versions 2.0 - 3.0 are widely supported today by most SoC vendors with GPUs • Shader code can be used to accelerate some CV algorithms • Version 3.1 added more abilities to do general-purpose compute, including CV • Driver support is more mature and stable than for OpenCL and OpenVX • GLSL coding is familiar to more developers than OpenCL, NEON or CUDA • Lightwing uses version 2.0 GLSL for faster motion tracking than is possible with OpenCV 10
• 11.
  © 2019 MontgomeryOne OpenCL 11 • Khronos consortium standard for general-purpose compute acceleration on SoCs and FPGAs • Framework for general-purpose compute acceleration across heterogeneous processors including CPUs, GPUs, DSPs and FPGAs • Based on C99 and C++11 languages • Provides a standard interface for parallel computing using task- and data-based parallelism • Some mid to high-end SoC and FPGA vendors support OpenCL for CV applications • Open alternative to Nvidia's CUDA • Not as mature on most platforms as OpenGL ES or CUDA • Will accommodate a wider range of algorithms than OpenGL ES shader code
• 12.
  © 2019 MontgomeryOne OpenVX 12 • Khronos standard for cross-platform acceleration of computer vision applications • A higher level of abstraction for programming CV than OpenCL • Based on a connected graph of vision nodes that execute a preferred chain of operations • Complementary to OpenCV, but can offer more optimized graph management • Many SoC vendors are quickly developing support for OpenVX • Available now on NVIDIA • Supports face, body and gesture tracking, smart video surveillance, advanced driver assistance systems (ADAS), object and scene reconstruction, augmented reality, visual inspection, robotics, etc.
• 13.
  © 2019 MontgomeryOne Basic Camera Video Pipelines 13
• 14.
  © 2019 MontgomeryOne Your First Camera Video Pipeline (V4L2 and GStreamer) • Attach a USB Camera to a Ubuntu PC (or your target system) and try: dmesg | grep camera - Displays kernel error messages about your camera driver's initialization ls /dev/video0 cat /dev/video0 - Shows if your camera driver is installed and working (producing data) lsmod - Lists all installed kernel drivers gst-launch-1.0 videotestsrc ! autovideosink - Test to see if gstreamer is installed and working (should display a color bars test pattern) gst-launch-1.0 autovideosrc ! autovideosink - Initialize and run a complete video pipeline (camera to display) using V4L2 and gstreamer 14
• 15.
  © 2019 MontgomeryOne OpenCV Sample Programs
• 16.
  © 2019 MontgomeryOne Explore the OpenCV Sample Programs • Install OpenCV 3.2 on Ubuntu (14 - 16) and build the sample apps: sudo apt-get install opencv sudo apt-get install build-essential sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev cd ~/opencv-3.2.0/opencv mkdir release cd release cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local .. make sudo make install cp -r /usr/local/lib/libopencv* /usr/lib/. • Run some sample apps on Ubuntu: cd /usr/share/OpenCV/samples/bin ./cpp-example-videocapture_basic • Combine sample source codes with 'videocapture_starter' to test an algorithm with camera video - cpp-example-videocapture_starter.cpp + cpp-example-edge.cpp 16
• 17.
  © 2019 MontgomeryOne Yocto Open Embedded Linux
• 18.
  © 2019 MontgomeryOne Build a Yocto Linux System for Your Target Board • Choose a board with a Yocto BSP and cameras with recipes for Yocto • Test the camera drivers provided in your BSP well - You will likely have to create recipes and/or a new V4L2 driver for your camera - An example of a working driver from your camera vendor is a minimum requirement • Install Yocto (Pyro, Rocko or Sumo) on Ubuntu and build a bootable OS: repo init -u https://github.com/Freescale/fsl-community-bsp-platform -b pyro repo sync MACHINE=wandboard DISTRO=fslc-x11 source setup-environment build bitbake fsl-image-machine-test-full (or, core-image-basic, fsl-image-x11-qt5, etc.) • Build and install cross-development toolchain to build your app and any additional packages: bitbake fsl-image-machine-test-full -c populate_sdk ./build/tmp/deploy/sdk/fslc-x11-glibc-x86_64-fsl-image-multimedia-armv7at2hf-neon-toolchain-2.3.4.sh bitbake -s 18
• 19.
  © 2019 MontgomeryOne Integrating OpenCV with Yocto Linux
• 20.
  © 2019 MontgomeryOne An Example - Vision Components for Lightwing 20
• 21.
  © 2019 MontgomeryOne Build and Run OpenCV Samples for Your Target Board • Add packages for OpenCV, X11, gstreamer, etc. to Yocto's config file: buildconflocal.conf CORE_IMAGE_EXTRA_INSTALL += "opencv x11 gstreamer" • Build and test the OpenCV 3.2 samples using Yocto toolchain, sysroot and CMake in your Yocto tree (mostly the same as on Ubuntu): source /opt/fslc-x11/2.3.4/environment-setup-arm7at2hf-neon-fslc-linux-gnueabi cd release cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local .. make • Running the sample apps on your target requires initialization of an X window (X11): export DISPLAY=:0.0 xhost +SI:localuser:root cd /usr/share/OpenCV/samples/bin ./cpp-example-videocapture_basic 21
• 22.
  © 2019 MontgomeryOne VideoCapture Class for Linux • The camera interface for OpenCV • About 40 implementations are provided for all major OS platforms • Three implementations are provided for Linux: cap_v4l.cpp - Original, for obsolete V4L version 1 cap_libv4l.cpp – Updated replacement for V4L2, with many patches cap_gstreamer.cpp – Optional extension convert formats and accepts GStreamer pipeline syntax: cpp-example-videocapture_basic 'videotestsrc ! videoconvert ! appsink' • Required some debugging to work on newer (4.x) Linux kernels - If V4L2 fails to initialize, it then tries V4L (version 1), which produces very confusing error messages. - Removed the xioctl() call macro because it was failing when calling to inquire the video format (VIDIOC_S_FMT) • Some improvements are available in OpenCV 4.1.0 for the Android Media NDK 22
• 23.
  © 2019 MontgomeryOne What Else Can (and Did) Go Wrong? • OmniVision 5640 camera discontinued –Was a popular camera supported by the FSL Community BSP –Use new OV5645 driver from Rocko version instead –Only older versions of Yocto (Pyro) actually work on most i.MX6/8 boards • Yocto only supports OpenCV 3.2 because of CMake compatibility issues –CMake installed by Yocto will not build OpenCV 3.4 • OpenCV samples require X Windows on Linux –Either use X11 or eliminate the use of the High-GUI module in samples • OpenCV VideoCapture class requires GStreamer on Linux –Unnecessarily complex and broken on the i.MX6/8 platform –Fixed the bugs in V4L2 implementation, so it works without GStreamer 23
• 24.
  © 2019 MontgomeryOne Working Around Component Limitations • OpenCV (VideoCapture class) requires the antiquated 24 bit BGR pixel format ! –Many cameras only provide 16 bit RGB 565, YUV or RAW formats –RAW format is 10 bit RGB (Bayer), but requires software conversion (very slow) • Official solution is to use gstreamer (with an appsink) element to convert YUV to BGR: videocapture_starter 'imxv4l2videosrc ! videoconvert ! appsink' - Can be accelerated by the IPU on i.MX6/8 by using NXP's plugins for gstreamer - libgstapp.so must be installed manually for this to work due to a bug in some builds for i.MX6/8 • Other ways to adapt the V4L2 VideoCapture class and get better performance: –Use the i.MX6/8 IPU without GStreamer. Red and Blue are swapped, but for many CV apps, that is fine –Use luminance data directly (dropping color data), without conversion to RGB • i.MX6 IPU will not scale video beyond 1024 x 1024 pixels (fixed on the i.MX8) • Lightwing solution - Use the GPU (OpenGL ES) to convert and scale video, instead of the IPU or GStreamer - Basic motion tracking algorithm is done in shader code, instead of OpenCV 24
• 25.
  © 2019 MontgomeryOne Lightwing Motion Tracking Demo
• 26.
  © 2019 MontgomeryOne Lightwing Motion Tracking 3D GUI Demo Motion tracking on the NXP i.MX6 GPU enables interactive control of 3D assets for mixed-reality video walls, without VR headsets or hand controllers 26
• 27.
  © 2019 MontgomeryOne Conclusions
• 28.
  © 2019 MontgomeryOne Lessons Learned • Start on a desktop PC with Ubuntu and the OpenCV samples - Develop your vision algorithms there, first • Use Yocto Open Embedded Linux to build your custom platform • Use the same open source libraries on desktop and target systems - Hacking OpenCV's VideoCapture class will likely be required • Understand which vision components you actually need and why • Select cameras carefully, considering software support - You really need a good V4L2 driver with Yocto recipes - Use a MIPI camera if possible, instead of USB • Explore the acceleration options for your SoC (GPU, IPU, NEON, etc) - Acceleration is required for most CV applications of camera video on ARM 28
• 29.
  © 2019 MontgomeryOne Resources 29 OpenCV Installation Tutorial https://www.docs.opencv.org/2.4.13/doc/tutorials/intr oduction/linux_install/linux_install.html#linux- installation Yocto Open Embedded Linux https://www.yoctoproject.org/ GStreamer Documentation https://gstreamer.freedesktop.org/documentation/ V4L2 Documentation https://www.kernel.org/doc/html/v4.9/media/uapi/v4l /v4l2.html Khronos OpenGL ES, OpenCL and OpenVX Documentation https://www.khronos.org/ Lightwing Open Mixed-Reality Platform https://montgomery1.com/lightwing/ NXP i.MX6/8 Development Boards https://www.wandboard.org/ OpenCV Status on Github https://github.com/opencv/opencv/wiki/ChangeLog
• 30.
  © 2019 MontgomeryOne Appendix
• 31.
  © 2019 MontgomeryOne OpenCV C++ Example - Edge Detection on Camera Video int iEdgeThreshSobel = 1; Mat mImageFrame, mImageGray, mImageGrayBlur, mImageEdgeMask, mImageEdgeResult; const char* WindowName = "Canny edge map with Sobel gradient"; static void onTrackbar(int, void*) { cvtColor(mImageFrame, mImageGray, COLOR_BGR2GRAY); // Create blurred gray scale image for edge detection. blur(mImageGray, mImageGrayBlur, Size(3, 3)); Canny(mImageGrayBlur, mImageEdgeMask, iEdgeThreshSobel, iEdgeThreshSobel * 3, 3); // Canny detector with sobel filter. mImageEdgeResult = Scalar::all(0); // Clear to black. mImageFrame.copyTo(mImageEdgeResult, mImageEdgeMask); imshow(WindowName, mImageEdgeResult); // Display image frame in window. } int main(int argc, char** argv) { VideoCapture capture; capture.open(0); // Open camera device through V4L2. namedWindow(WindowName, WINDOW_KEEPRATIO); // Create window and slider control. createTrackbar("Canny threshold Sobel", WindowName, &iEdgeThreshSobel, 100, onTrackbar); char key = 0; while (key != ‘q’) // Capture frames from the camera and display them. { capture >> mImageFrame; // Capture another image frame from camera. if (mImageFrame.empty()) break; onTrackbar(0, 0); // Show the image. key = (char)waitKey(30); // Wait 30 milliseconds for a key press. } return 0; } 31
• 32.
  © 2019 MontgomeryOne 32 Lightwing – Open Mixed-Reality Video Wall Platform • Build custom digital signs and video walls with interactive 3D content • Camera motion tracking controls 3D assets without VR headsets or hand controllers • Supports both touch screens and camera motion tracking input • Multipanel video wall dimensions of any size for public spaces • Built-in 3D animations, fonts, effects, audio, video, images and web RSS feeds • GPU and IPU acceleration of camera video on the NXP i.MX6/8 series SoCs • Scriptable, browserless architecture built on open tools for Windows and Linux https://montgomery1.com/lightwing/
• 33.
  © 2019 MontgomeryOne Lightwing on Linux verses Android and Windows 33
• 34.
  © 2019 MontgomeryOne About Clay D. Montgomery • 30+ Years of Embedded Multimedia Software Development • C/C++, OpenGL 3D, Audio/Video, Linux, Windows and Android • Previously worked at STB (3Dfx), VLSI (NXP), Nokia, TI and AMX (Harmon) • Authored on Embedded OpenGL ES for Intel Developer Zone, TI, Montgomery One • Freelance Embedded Linux Developer since 2010 • Now working on Open Embedded Linux (Yocto) on ARM almost Exclusively • Active in Tech Start Up Community in Texas • Created Lightwing Mixed-Reality Engine for Interactive Digital Signage, Video Walls and Touch-Kiosks on NXP i.MX6/8 SoCs • Currently adding CV Object Detection and Motion Tracking Features to Lightwing 34