Camera Architecture from Failure to Success

Camera Architecture – Options for Success

Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 2
Tizen-IVI offers the whole world of HTML5/CSS/JS solutions from a UI point of view, however
many of the features require developers to step out of the WRT for native support.
The Camera related use case is a good example where JS alone may not be enough to build
whole SW stack.
Considering available technologies, and the specifics of the use cases, there might be several
ways to implement Camera support.
Within STC we did some exploration of the options, and would like to present details of this
investigation. We hope this knowledge sharing will be helpful to other developers.
We will briefly go through possible HW configurations and related use cases before really
stepping into the SW side. Then we will introduce SW stack details and the possible solutions as
they relate to the configurations and use cases.
Introduction

As with any camera related project it is important to pay attention to the following key
requirements:
• Desired FPS
• Video Resolution
• Video Lag
• Distance of camera(s) from the processing unit
• Number of cameras to connect
• Number of cameras to display simultaneously
• Time to switch camera view if user allowed to select camera from the list of inactive devices
• Performance of CPU or assisting en(de)coding modules
Considering the level of abstraction of Tizen-IVI applications with respect to Native solutions,
it is extremely important to fix these requirements before the HW selection step. We will review
some of the implications in the next slides.
Typical requirements to consider

It is important to introduce possible types of HW first, as use cases may split in its
requirements to particular HW combinations.
We could recommend first to focus on these three groups of HW in selecting particular
combinations of HW for the setup.
• Camera HW which could be USB, Analog and IP ones.
• Host system main board offering USB root hub (one or many, USB2.0 or USB3.0), a means of
connecting an Analog video capture board, and one or more Network interfaces.
• Analog video capture board offering single or multiple channel support, together with a
video driver which may or may not perform channel multiplexing ( mapping of multiple
channels onto a single Linux video device ).
Types of HW

At a low level, some of available options will limit possible use cases. Typical points to
consider:
• IP cameras to support required quality of the video (frame size/fps) with MJPEG over TCP/IP
and/or H.264 over RTSP/RTP depending on particular needs.
 There is not too much HW related specific in this option if you have TCP connection to
the camera working along with UDP packets going well and you satisfied with the quality
versus performance of the network and host system.
Why it is important to care about particular HW set up?

• Analog video capture board which supports non-multiplexed Linux video devices.
 Many cheap and small miniPCI Express cards support many input channels, but their
drivers expose only one Linux video device, and only one channel sends frames to the
interface at one time. Hence multichannel support requires rapid switching between
channels. To minimize problems on the SW side, it is preferable to use a capture board
whose driver supports as many Linux video devices as required, ideally one per camera.
Another option would be several video capture boards, if allowed by the number of
MiniPCI Express ports on the mainboard.
 Cards with higher total FPS should be preferred. If channel multiplexing is required, then
higher FPS would help to minimize time lost ( about 3-4 frames ) due to tuning phase
after each channel switch. It’s important to distinguish between per-channel FPS, and
the total FPS supported by the video chip/card.
 Card manufacturers have a habit of not keeping their drivers up to date. Check that you
can build your card’s driver using the particular kernel used on the target system.
( cont. )

• USB cameras to support required quality of the video with acceptable USB bandwidth
considering other devices connected to same USB Root Hub at the host system mainboard.
 Feasibility of multiple USB camera solutions is determined by single camera USB
bandwidth allocation for acceptable quality video, the number of cameras needed, and
the number of USB Root Hubs.
 The main issue here is the difficulty in knowing how much USB bandwidth will be
allocated by a Hub for a given camera – it is not a simple function of video bandwidth,
but depends on options presented to the USB driver by the device.
 Typical USB Bandwidth allocations to expect for different camera profiles are from 20%
up to 60%+. 20% of bus bandwidth is reserved for non-isochronous devices. Hence this
might be a limiting factor in multiple camera projects.
 Another complication is that you would never find this USB bandwidth usage written on
the box. Even more, it might depend to particular Firmware version. So, you would need
to be in touch with manufacturer or perform hands-on experiments with sample
cameras.
( cont. )

• Mitigations for USB bandwidth issues
 Consider requiring USB 3.0 support in Root Hub and Camera as this might relax some of
the limits around USB bandwidth.
 Consider H.264 support in the camera and availability of the SW options to deal with
such interfaces.
( cont. )

In context of Tizen-IVI, let’s assume we are always talking about Web Applications presenting
a video stream in WebKit or Crosswalk browser. This is the ‘Tizen IVI way’. Typical use cases and
the most important requirements to consider might be:
• Slow motion camera - Intermediate quality of the video, no requirements regarding lag
• Rear view on the display - Minimal lag, high resolution, flipped frames
• 360 degree stitched view to present road around the car - Multiple cameras setup, correction
of lens distortion, stitching of camera outputs into single panoramic video, minimal lag
• Visualization of the driving details and some camera view merged on windshield - Merge of
the dynamic/static data with actual camera view, minimal lag, medium FPS.
• In car camera to view rear passenger area - Multiple cameras setup, medium or low FPS,
medium lag.
• One by one frame processing before display in various recognition cases such as faces, road
signs or other objects recognition – low lag is preferred, but may not be critical ( depends on
circumstance ), medium FPS as the recognition may take longer than single frame release
time, no en(de)coding artifacts to achieve best recognition results
Typical use cases

General architecture and types of SW
Source
Weston
Underlying Middleware
System/Kernel Services
Kernel Card Drivers
WebKit
WRT Application
WRT Plug-in
JS API
Wayland client
D-BUS API
Variety of
Weston
plug-ins,
shell,
compositor
ProcessingPresentation
A general architecture can be presented in the
following way, where some of the layers can be
excluded for particular cases.

• WebKit in this view is a sandbox where the app is running
 WRT Application is a combination of HTML/CSS/JS technologies, hence it can deal with the
video frames with help of JS or leave it to the functions available in WebKit core. So, it can
be Video tag, Pure Canvas, WebGL or JS MPEG-4 decoder.
 WRT Plug-in primary usage is control (over D-BUS or similar interface ) and secondary is
bypass interface to the source of the video data (Underlying Middleware implementing
access to particular System/Kernel services)
With respect to webkit there are few more options to deal with particular camera options such
as:
 Combination of webkitGetUserMedia and createObjectURL to get blob object referencing
the stream taken from V4l2, hence applicable to USB and Video Capture Board options
 Make your own Blob object with help of JS and WRT Plug-in and feed it to video tag like the
one from the above option
• Weston plug-ins might be one of the options if you want to make overlay client where the WRT
Application only controls the way and time video is presented. Video frame processing is then
done outside WRT sandbox.
Presentation layer

• Underlying Middleware ( Processing layer ) is a combination of several components:
 Tizen components such as GStreamer (or FFMPEG) to unify interface to the various video
streaming devices. NB This is distinct from GStreamer as used directly inside Webkit.
 Custom SW components to control parameters of System components
 Or Custom SW components to deal with the device interfaces directly, considering specific
details of each video streaming device
• What is the purpose of Processing layer group? Essentially it is an abstraction layer that hides
details of the various types of processing required by different video sources before that source
is exposed to the WRT.
• GStreamer can perform most such processing tasks, even if some of the options required are
really complicated. Where little or no processing is required, then of course you may avoid any
GStreamer overhead by creating Custom SW to deal with V4l2 or IP camera interface directly.
However, if it is simply that GStreamer does not provide the processing you need, you should
probably consider creating a GStreamer plug-in, which would then make all the GStreamer
facilities available too.
Processing layer (could include WRT or Weston Plug-in)

• Could be a Linux Video device in combination with V4l2 interfaces. Most applicable to USB and
Video Capture board types of HW. Functionality and parameters available thru ioctl interface
should be also considered as part of this layer group
• Could be custom interfaces made to support specific features such as video in h.264
• Network interface to IP cameras or non-standard interfaces to the devices where you would
need to use some intermediate layers to get access to the camera even thru IP or other type of
connection.
Source layer

• Webkit: video tag. Does buffering of the video if used with network stream set directly to SRC.
Lag depends to video type and properties, but typical values to expect are from 3 to 5 seconds.
Another complication is list of supported video formats. Webkit used in Tizen-IVI supports OGG
and H.264/MPEG-4 Part 10. On the plus side, it’s possible to run up to 6-8 streams at the same
time with good FPS and quality of decoding.
• Webkit: pure canvas. As fast as can be expected for single camera set up and can offer single
camera view with FPS close to 25-30, but strictly depends on implementation in JS and below,
which provides byte array with frame. Typical lag is about 1 second or less.
• Webkit: WebGL could give you some fancy features with additional processing of the video
frames, but still might be comparable to the pure canvas solution.
• Webkit: JS MPEG-4 decoder is fast enough to get smooth play of the video with very minimal
lag, but might not work out of the box on Tizen IVI.
• WRT plug-in: D-BUS control interface. Can be Corba or any other interface to the system or
custom service running to serve the webkit with the video streams and control the cameras
over ioctl and V4L2 interfaces.
Specific of particular selections of SW and HW

• WRT plug-in: Combination with underlying MW to control and run Gstreamer, or to read the
frames directly from the Linux video interface such as /dev/videoX. Good for intermediate
processing of the frames, but relatively slow because can work only with Canvas or other direct
draw method.
• Webkit: webkitGetUserMedia and createObjectURL. Doesn’t work out of the box with Tizen-IVI
as Webkit is using GStreamer 1.0 library, but still sends the parameters in the GStreamer 0.10
format. Hence some modifications would be required to bypass this issue. Meanwhile, this way
does work with Crosswalk and result is very smooth.
• Webkit: own blob object in combination with WRT plug-in to serve video tag source. Could be
comparable to webkitGetUserMedia and createObjectURL method, but performance would
depend entirely on the WRT Plug-in implementation.
• Weston: shell or compositor. Fast and flexible as you are managing merge of the layers and
surfaces directly. Can be helpful in complex layer merge scenarios such as Media Player with
data shown on top of the video, but constraints are in the high knowledge entrance level and
complexity of the system. Additionally it may impact whole system performance and stability,
hence shouldn’t be used without proper assessment of the needs and risks.
Specific of particular selections of SW and HW (cont.)

• GStreamer: Complex, but powerful solution which can read multiple sources (from files to IP
cameras) and give multiple types of outputs with different conversions in between input and
output. Main points to consider are:
 There will probably be more than one way to build a GStreamer pipe to get functionality
you need.
 Different versions of Tizen may include different versions of GStreamer and its plug-ins
leading to different and sometimes failing behavior in between releases
 Vaapi ( HW support ) is present, but it’s effectiveness depends on the Host system.
 SW encoders may fail under heavy system load and produce corrupted streams.
 Not everything is obvious . E.g. tcpserversink is good for streaming the data over the
network, but you would need to create your own wrapper (proxy) to add HTTP headers if
you try to use it with browser Video tag.
 Many useful plug-ins are still in “bad” and “ugly” packages meaning you have very minimal
support for most of functionality outside mainstream usage.

• FFMPEG: Simpler than GStreamer, but also limited. Upstream versions are not expected to be
available in the Tizen IVI repo and please be aware of modules which are not even built for
Tizen.
• Custom SW: Needed to deal with devices directly, or to control device status while streaming is
ongoing thru other components. Highly dependent on circumstances, but might be required for:
 Setting up Video Capture board parameters
 Controlling status and availability of the devices
 Matching devices versus some ID to identify exact cameras in multicamera setup.

Let’s try to assemble combinations of SW for particular usecases defined before
• Slow motion camera - Intermediate quality of the video, no requirements regarding lag
In short this is the case to record/present status of object over the time rather than to display
its immediate state. Simplest solution would be to use GStreamer and split the source into two
identical outputs where the one saved to the file and second displayed to user. No lag requirement
means variety of options to implement presentation layer
• Rear view on the display - Minimal lag, high resolution, flipped frames
For the USB and Analog cameras the simplest way would be a combination of
webkitGetUserMedia and createObjectURL methods. A custom solution could be implemented,
such as own blob object or Weston plug-in for overlay in combination with GStreamer. Last options
are helpful especially if there is a need for camera multiplexing or IP camera usage. Flipping could be
done by browser as layer property, hence no additional processing required.
Solutions

• 360 degree stitched view to present road around the car - Multiple cameras setup, correction of
lens distortion, stitching of camera outputs into single panoramic video, minimal lag
Architecturally correct way would be custom GStreamer plug-in implementation and usage of
OpenCV for different sources stitching, blending and lens corrections. Standard videomixer plug-in is
good example for quick start with GStreamer plug-ins. Custom SW could implement the same
functionality by reading the frames from Linux Video interfaces directly or using GStreamer as a
library. For minimal lag Weston overlay or own blob object could be used to pass video.
• Visualization of the driving details and some camera view merged on windshield - Merge of the
dynamic/static data with actual camera view, minimal lag, medium FPS
If data is primarily textual information, then browser layer capabilities could be used to merge it
with video stream. Otherwise variety of GStreamer plug-ins could do it at the cost of some MW
complication. With medium FPS browser Canvas is good option where the frames are read directly
from device (or from GStreamer for IP cameras), processed in custom SW and then passed to Canvas
thru WRT plug-in. Meanwhile webkitGetUserMedia and createObjectURL methods are still good if
there is no need to deal with IP cameras.
Solutions (cont.)

• In car camera to control taxi passenger area - Multiple cameras setup, medium or low FPS,
medium lag.
In general similar to “360 degree stitched view” case, but lower requirements for presentation
part, hence Canvas or own blob object can be implemented to handle display.
• One by one frame processing before display in various recognition cases such as faces, road
signs or other objects recognition - Better no lag, but depends to the case, medium FPS as the
recognition may take longer than single frame release time, no en(de)coding artifacts to achieve
best recognition results.
Custom SW to read frames directly from the device or GStreamer library and to process it
further can be implemented. Meanwhile, depending on the processing required, it could be a
Gstreamer plug-in doing analysis of the frames and sending the signals to upper layers. Canvas or
blob object for Medium FPS and minimal lag. Weston overlay may overcomplicate the system, but
technically possible to achieve minimal lag and best performance of the display.
Solutions (cont.)

Questions?
Last slide 

Camera Architecture from Failure to Success

More Related Content

What's hot (17)

Similar to Camera Architecture from Failure to Success (20)

More from Ryo Jin (20)

Recently uploaded (20)

Camera Architecture from Failure to Success