Towards Embedded Computer Vision - New @ 2013

Wang, Yuan-Kai(王元凱)

Electrical Engineering Department,
Fu Jen University (輔仁大學電機工程系)

Email:
ykwang@mail.fju.edu.tw,

URL:
http://www.ykwang.tw

Slideshare:
Search
"Towards
Embedded
Computer
Vision"

Towards
Embedded
Computer Vision
1

王元凱 Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Context Awareness
2
人

時

物

地

事
Situation
High-Level Context Low-Level Contexts

3
MIT Oxygen
http://oxygen.lcs.mit.edu/ 1999

4
The Aware Home
http://awarehome.imtc.gatech.edu/

Ambient Intelligence v.s.
Object Intelligence
Embedded system + Sensors
5
Wearable Computing
Smart Home, Smart …

66
Embedded Systems
"Without" Sensors
資料來源：資策會MIC
ITIS計畫整理

77
Embedded Systems
"With" Sensors
GPS
Navigation
Wii
Roomba
Exoskeleton

8
Driver assist
Front view
Night vision
Blind corner
Blind spot
Rear view
Embedded Systems
"With" Image Sensors

99
Embedded Systems
Google Glass

Smart Watch
10

Microsoft
SenseCam
Embedded Systems
11

Observation
Miniaturization
High Resolution
3D Capability
12

13
Embedded Computer Vision
  Embedded System + Camera
+ Computer Vision Algorithm
Image
Processing
Image
Recognition
Image
Capturing

14
Embedded Computer Vision
  Embedded compute vision
  Implements computer vision algorithms
on low-cost, low-power,
constrained hardware
  Constrained hardware
  Low-speed CPU
  Low capacity memory
  No floating-point (FPU)
  Low-resolution image sensor

A
B
Research Status (1/3)
15

H
K
K
16

1717
DARPA
Grand
Challenge
Google
Car
Google
Taxi

18
International Activities (1/2)
  Special conferences
  IEEE Int. Workshop on Embedded Computer
Vision
  IEEE Int. Conf. Distributed Smart Cameras
  Special journal issues
  IEEE Journal of Selected Topics in Signal
Processing, vol. 2, no. 4, Aug. 2008
  EURASIP Journal on Embedded Systems
  Short courses in important CV conferences
  CVPR07&08: Distributed vision processing in
smart camera networks
  ESC07: Embedded CV and smart cameras
  ICASSP09: Distributed processing in smart
cameras

19
International Activities (2/2)
  Research projects and Lab.
  Princeton Univ./Georgia Tech.:
Embedded Systems Lab., Wayne Wolf
  Stanford Univ.
Wireless sensor networks Lab.
  UCLA, CMU, MIT
  Delft Univ. of Technology
SmartCam Project
  Graz Univ. of Technology

20
Three Ways for ECV
  CPU (Central Processing Unit)
  ARM, PowerPC
  DSP (Digital Signal Processor)
  TI, ADI, NXP
  FPGA (Field Programmable Gate Array)
  Altera, Xilinx

21
CPU
  Stanford MeshEye
  ARM 7 (55MHz), ZigBee node
  3 image sensors
  30x30 grayscale x 2
  640x480 color x 1
Object detection
(http://wsnl.stanford.edu/smartcam.html)
"MeshEye:a hybrid-resolution
smart camera mote for applications
in distributed intelligent
surveillance", IPSN-SPOTS, 2007

22
DSP
  "Visual surveillance on DSP-based
embedded platform," Graz Univ. of
Technology, 2008(Phd. dissertation)
  TI C6414 (600MHz)
  Applications: Adaboost face detection, vehicle
detection, license plate detection

23
FPGA (2/3)
"Hardware, Design and
Implementation Issues on a
FPGA-Based Smart Camera,"
ICDSC, 2007

24
Hybrid: CPU+DSP
  "Distributed Embedded Smart Cameras
for Surveillance Applications,"
IEEE Computer, 2006.
Developed for traffic surveillance

Hybrid: CPU+DSP by SoC
  TI OMAP
25

26
Hybrid: CPU+FPGA
  UCLA Cyclops
  MCU: Atmega128
  CPLD: image processing
  ECV app.: Hand gesture recognition
http://research.cens.ucla.edu/
projects/2007/
Multiscaled_Actuated_Sensing/
Cyclops/

Hybrid: CPU+FPGA by SoC
  Xilinx Zynq
27

28
Challenges for ECV
  Algorithm refinement
  Parallel computation
  Function partition, Multi-threading
  Stream processing
  Memory hierarchy optimization
  Hardware design
  Pipeline, SIMD, board design
  Optimized programming skills
  Fixed-point arithmetic
  Memory management
  Intrinsic commands

29
Our Experiences
Entertainment Robot (CPU)
Vision Sensor Network (CPU)
Smart Video Surveillance (DSP)
Wearable Camera (DSP)

30
Case 1
Entertainment
Robot
CPU

31
Sony AIBO

Rolling Dice
32

33
Face Detection

34
Face Recognition

35
Facial Expression Recognition

36
Case 2
Embedded Vision
Sensor
CPU

37
FJUCam
• Power
consumption:
1W
• Power sources:
• 5V DC current
• 5V Cell Battery
• Weight: 35gm
• Size:
6 x 4.5 x 5 (cm)
(W x H x D)
Three Modules
1. Main board, 2. Lens module
3. Storage module

38
Face Detection
FJUCam
Face Detection
Image
Display

39
Cyclops Robot
  Color tracking

40
Case 3
Video Surveillance
FPGA

41
Mobile Video Surveillance
FPGA
行動通訊網路
(3G/GPRS)
網際網路
網路攝影機
監控伺服器
行動裝置
(3G行動電話)GSM
數據機
Wap Push
傳遞視訊串流
PCBackground subtraction, ...
•  2.8 GHz Intel CPU
•  Software: C/C++
•  Frame rate: 10 fps

42
Background Subtraction
Current
Frame
-
Background
Image Update
Post Processing
Motion Object Image
Background Image
1+kB
kB
1+kP
1+kM
),(),(
),(
1
1
yxByxP
yxM
kk
k
−= +
+
),()1(),(
),(
1
1
yxPyxB
yxB
kk
k
+
+
−+= αα
Speed up by (1) Circuit design, (2) Paralization

43
FPGA Speed Up
  Comparisons
  PC: 2.8GHz CPU, C implementation
  FPGA can speed up 500 times
Clock(Hz) FPS
CPU
FPGA
2.8G
25M
51
10

44
Case 4
Wearable Camera
X-EYE
ARM + DSP

45
Goal
45

Demo Video
46

47
Development
滑鼠

鍵盤

SD卡
USB-WIFI卡

讀卡機

USB-RS232
控制線

微投影機

自製

外殼

USB

連接線

顯示器

觸控面板

筆電

移動電源

BeagleBoard

Camera

USB

Hub

48
Hardware
4848
Peripheral I/O
• USB Host
• JTAG
• DVI-D video out
• S-Video out
• SD/MMC+
• Stereo in/out
• RS-232 serial1
• Alternate power
• USB 2.0 HS OTG
USER
OMAP3530 Processor
• 600MHz Cortex-A8
• NEON+VFPv3
• 16KB/16KB L1
• 256KB L2
• 430MHz C64x+ DSP
• 32K/32K L1
• 48K L1D
• 32K L2
• Power VR SGX GPU
• 64K on-chip RAM
POP Memory
• 256MB LPDDR RAM
• 256MB NAND flash
RESET
7.6 cm

49
System
2010.04.25
49

50
Software

Next : UAV
51

Next : Sleep Monitoring
52

Next : Vehicular
Object Detection
53

54
Future of
Embedded Vision

55
Summary
  Embedded CPU is not appropriate for
ECV
  Pros
  Great flexibility
  Fast development
  Cons
  No float-point arithmatics
  Interrupt-based architecture
  Smart phone with 4/8 cores is OK?

56
I Think …
  Embedded compute vision
  Low-cost, low-power,
minimal hardware
  High-Performance
computer vision
  Fast speed without cost,
power, and hardware
constraints
From contradiction to convergence ?!
Constrained
Resource
Abundant
Resource
Yes by multicore
Contradiction

57
Challenges (1/2)
  Algorithm decomposition
  Function decomposition
  Partition serial part and parallel part
  Data flow analysis and
data dependency analysis
  Parallelism
  Loop unrolling
  Multithreading
  Pipeline

58
Challenges (2/2)
  Performance analysis method
  For efficiency improvement
  Implementation efforts
  Choose a good embedded platform for
computer vision
  Software issues
  Hardware issues
  Programming skills
  Multi-threading

59
Future Research Directions
Heterogeneous
Multicore
ARM
+ DSP
ARM +
FPGA
ARM +
GPGPU

I Think Again …
  Is there a development platform for
embedded computer vision with
  Built-in camera
  AE, AF, AWB
  High-performance processor
  Heterogeneous multicore
  Software development environment:
  C, Matlab
  OpenCV
  Sample codes
  Labs
60

Embedded Vision Alliance
61

  Ramesh Raskar
MIT Platform
62

  Andrew Adams
Stanford Platform
63

Taiwan
Platform
for
Embedded
Vision
?
64

Towards Embedded Computer Vision - New @ 2013

本簡報授權聲明

 此簡報內容採用

Creative Commons 「姓名標示
- 非商業性台灣
3.0 版」
授權條款

 歡迎非商業目的的重製、散布或修改本簡報的內容，但
請標明：
(1)原作者姓名：王元凱；
(2)圖標示：

 簡報中所取用的部份圖形創作乃截取自網際網路，僅供
演講者於自由軟體推廣演講時主張合理使用，請讀者不
得對其再行取用，除非您本身自忖亦符合主張合理使用
之情狀，且自負相關法律責任。
66

Towards Embedded Computer Vision - New @ 2013

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to Towards Embedded Computer Vision - New @ 2013 (20)

More from IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing (14)

Recently uploaded (20)

Towards Embedded Computer Vision - New @ 2013