Sensor fusion of LiDAR and Camera for real time object detection - talk version
0 likes534 views
The document discusses sensor fusion methods for improving perception in autonomous driving, focusing on real-time 3D detection using LIDAR and camera sensors. It outlines the goals of achieving high performance and precision, and details the calibration techniques involved in aligning sensor outputs. The paper also highlights a deep learning approach for object detection and presents the challenges and results of the fusion process.
Sensor fusion of LiDAR and Camera for real time object detection - talk version
- 1. Sensor fusion for autonomous driving perception Bagni Fabio [email protected] .it San Francisco - 15 May 2019 1
- 2. Fabio Bagni San Francisco - 15 May 2019 Autonomous driving perception Precision and reactivity Risks of accidents, traffic laws respect. 2
- 3. Fabio Bagni San Francisco - 15 May 2019 Embedded platforms 3
- 4. Fabio Bagni San Francisco - 15 May 2019 Goals Real time 3D detection ● High performances ○ required 10 Hz frequency ● High precision ● Real time on embedded platforms 4
- 5. Fabio Bagni San Francisco - 15 May 2019 UNIMORE HiPeRT prototype 5
- 6. Fabio Bagni San Francisco - 15 May 2019 Sensors ● LiDAR ○ 3D point cloud ● Cameras ○ images Heterogeneous outputs 6
- 7. Fabio Bagni San Francisco - 15 May 2019 Sensor fusion What is sensor fusion: ● Matching of different sensors views Why sensor fusion: ● Heterogeneous information of environment 7
- 8. Fabio Bagni San Francisco - 15 May 2019 Fields of view LiDAR: 360 degrees around the vehicle Camera: wide angle, disposed do cover 360 degrees. 8 Cameras LiDAR
- 9. Fabio Bagni San Francisco - 15 May 2019 Deep learning approach Camera: ● Bird’s Eye View estimation ● Object detection LiDAR ● Bird’s Eye View calculation ● Object detection 9 Image Point cloud 3D detection 3D detection Fusion
- 10. Fabio Bagni San Francisco - 15 May 2019 Deep learning approach time Image stitching for multiple camera fusion 10 ➔ Under10 Hz Notenoughresponsive 1. Ming Liang, Bin Yang, Shenlong Wang and Raquel Urtasun: Deep Continuous Fusion for Multi-Sensor3D ObjectDetection.In: ECCV (2018).
- 11. Fabio Bagni San Francisco - 15 May 2019 Static alignment sensor fusion Alignment of LiDAR with each camera individually. 11 Calibration Preprocessing
- 12. Fabio Bagni San Francisco - 15 May 2019 Features extraction Matching is made by a calibration with a perforated panel. Holes are detected by all sensors. Holes’ centers are used as features to be matched. 12
- 13. Fabio Bagni San Francisco - 15 May 2019 Cylindrical projection Fields of view matching is made projecting sensors’ outputs on the surface of a cylinder. 13
- 14. Fabio Bagni San Francisco - 15 May 2019 Projections complexity LiDAR cylindrical projection: Depthmap O(n) : one operation per point 14 Camera cylindrical projection O(n) : one operation per pixel PARALLELIZABLE
- 15. Fabio Bagni San Francisco - 15 May 2019 Calibration results 15
- 16. Fabio Bagni San Francisco - 15 May 2019 Colored points Alignment of each point with the corresponding pixel on the image. 16
- 17. Fabio Bagni San Francisco - 15 May 2019 Object detection Neural network for object detection on camera frame Clustering of LiDAR points Bounding boxes matching 17
- 18. Fabio Bagni San Francisco - 15 May 2019 Object detection results 18
- 19. Fabio Bagni San Francisco - 15 May 2019 Object detection average time 19 LiDAR output : 10 Hz Time limit : 100 ms
- 20. Thanks for your attention Fabio Bagni - [email protected] 20 Fabio Bagni San Francisco - 15 May 2019
- 21. 21
- 22. Fabio Bagni San Francisco - 15 May 2019 Homography Homography allow to translate camera projection plane on the LiDAR projection plane. This method make cylindrical surfaces match. 22