![/18
Compare the computation time [μs]
18
Using newly received event, PointNet need to process the all events again.](https://image.slidesharecdn.com/study20200918-200919080957/85/eventnet-asynchronous-recursive-event-processing-18-320.jpg)
eventnet asynchronous recursive event processing
- 1. /18 EventNet: Asynchronous Recursive Event Processing Study 2020/09/18 2011136 Ryuta Shitomi 1 Yusuke Sekikawa, Kosuke Hara, and Hideo Saito, CVPR2019
- 2. /18 The event camera 2 High dynamic range Do not suffer from motion blur No latency Recently, frame-based approach have achieved impressive results in scenarios.
- 3. /18 The event data from event-based camera 3 When the new event occurred: 𝑒𝑖 = 𝑥𝑖, 𝑦𝑖, 𝑝𝑖, 𝑡 − 𝑡′ 𝑡′ : time of the last event Asynchronous Sparse Many events (1 mega event per sec) ↓ Need a new learning approach to differ from frame-based approach
- 4. /18 Frame-based approach for an event streams 4 Integration 𝛥𝑡 to create a frame and feed a neural net. Disadvantages: Cannot use of the sparseness of the data Redundant computation A. I. Maqueda, A. Loquercio, G. Gallego, N. N. Garcia, and D. Scaramuzza. Event-based vision meets deep learning on steering prediction for self-driving cars. CoRR, abs/1804.01310, 2018. 𝜟𝒕
- 5. /18 Purpose and Contribution 5 Develop a novel neural network: can process an extremely high-rate and variable length. for real-time processing of an asynchronous event stream. to output recursively using temporal information. 𝜏′ 𝜏 𝑡 𝑝𝑜𝑙𝑎𝑟𝑖𝑡𝑦
- 6. /18 Problem Statement 6 Consider a sequence of events within a 𝜏 ms interval based on the latest time stamp 𝑡𝑗 of the event: 𝒆𝑗 = 𝑒𝑖 𝑖 = 𝑗 − 𝑛 𝑗 + 1, … , 𝑗} 𝑒𝑖 = 𝑥𝑖, 𝑦𝑖, 𝑝𝑖, Δ𝑡𝑗,𝑖 Δ𝑡𝑗,𝑖 = 𝑡𝑗 − 𝑡𝑖 Consider a neural network 𝑓 to estimate 𝑦𝑗 given event stream 𝒆𝒋 𝑦𝑗 = 𝑓 𝒆𝑗 𝑦𝑗: Task dependent 𝜏 𝑡 𝒆𝒋 𝜏 𝒆𝒋+𝟏
- 7. /18 Problem Statement 7 To realize a trainable neural network 𝑓 that satisfies following conditions: 1. End-to-End Trainable 2. Event-Wise Processing 3. Efficient Recursive Processing 4. Local Permutation Invariance 𝜏 𝑡 𝒆𝒋 𝜏 𝒆𝒋+𝟏
- 8. /18 Using PointNet architecture 8 PointNet solves the problem by approximating the function 𝑓: 𝑦𝑗 = 𝑓 𝒆𝑗 = 𝑔 max ℎ 𝑒𝑗−𝑛 𝑗 +1 , … , ℎ 𝑒𝑗 where ℎ: ℝ4 → ℝ 𝐾 , 𝑔: ℝ 𝐾 → ℝ Because of the max(∙) , the permutation of events does not change the output 𝑦𝑗. MLP Problem: A huge amount of computation of ℎ ∙ and max ∙ . Should run on the event rate. (could be more than 1MEPS) When a new j + 1 -th event arrives, Δtj,i = tj − ti changes to tj+1 − ti . Many previous event has already been processed by h(∙).
- 9. /18 EventNet 9 Introduce the ”Temporal Coding” and “Recursive Processing” to overcome the difficulty. The Δ𝑡 only changes as the new event is being received. Split the ℎ(𝑒): ℎ 𝑒𝑖 = 𝑐(ℎ 𝑒𝑖 − , Δ𝑡𝑗,𝑖), 𝑒− : (𝑥, 𝑦, 𝑝) 𝑐 ∙ :temporal coding function, ℎ: ℝ3 → ℂ 𝐾 𝑦𝑗 become: 𝑓 𝒆𝑗 = 𝑔 max 𝑐 𝑧𝑗−𝑛 𝑗 +1, Δ𝑡𝑗,𝑗−𝑛 𝑗 +1 , … , 𝑐 𝑧𝑗, 0 where 𝑧𝑖 = ℎ 𝑒− ∈ ℂ 𝐾 𝑐 ∙ , max ∙ need to be computed for all events. Temporal Coding
- 10. /18 EventNet 10 • Assume the norm of each element of 𝑧𝑖 is less than 1 (by tanh). • max ∙ , 𝑐 ∙ need to be recursive. Example) 𝑠𝑗 = max 𝑠𝑗−1, 𝑐 𝑧𝑗, 0 𝑠𝑗+1 = max 𝑠𝑗, 𝑐 𝑧𝑗+1, 0 • Propose temporal coding function to satisfy this condition: 𝑎𝑗,𝑖 = 𝑐 𝑧𝑖, Δ𝑡𝑗,𝑖 = 𝑧𝑖 − Δ𝑡𝑗,𝑖 𝜏 + exp −i 2𝜋Δ𝑡𝑗,𝑖 𝜏 Recursive Processing Encode the elapsed time
- 11. /18 Temporal Coding Function 11 𝑎𝑗,𝑖 = 𝑐 𝑧𝑖, Δ𝑡𝑗,𝑖 = 𝑧𝑖 − Δ𝑡𝑗,𝑖 𝜏 + exp −i 2𝜋Δ𝑡𝑗,𝑖 𝜏 When the new event is received: 𝑎𝑗+1,𝑖 = 𝑐 𝑧𝑖, Δ𝑡𝑗+1,𝑖 = 𝑧𝑖 − Δ𝑡𝑗,𝑖 𝜏 − 𝑡𝑗+1 − 𝑡𝑗 𝜏 + exp −i 2𝜋(Δ𝑡𝑗,𝑖 − 𝑡𝑗+1 + 𝑡𝑗 𝜏 Recursive Processing 𝑡 |𝑎 𝑗,𝑖| 𝑒𝑗 New event is received 𝑡 |𝑎 𝑗,𝑖| 𝑒𝑗 𝑒𝑗+1 − 𝑡𝑗+1 − 𝑡𝑗
- 12. /18 Temporal Coding Function 12 𝑡 |𝑎 𝑗,𝑖| 𝑒𝑗 New event is received 𝑡 |𝑎 𝑗+1,𝑖| 𝑒𝑗 𝑒𝑗+1 − 𝑡𝑗+1 − 𝑡𝑗 𝑎𝑗,𝑗−2 = max 𝑎𝑗,𝑗−5, … , 𝑎𝑗,𝑗 when you new event is received. 𝑎𝑗+1,𝑗−2 = max 𝑐 𝑎𝑗,𝑗−2, 𝑡𝑗+1 − 𝑡𝑗 , 𝑎𝑗+1,𝑗+1 Therefore, 𝑦𝑗+1: 𝑦𝑗+1 = 𝑓 𝒆𝒋+𝟏 = 𝑔 max 𝑐 𝑠𝑗, 𝑡𝑗+1 − 𝑡𝑗 , ℎ 𝑒𝑗+1 − 𝑒𝑗−2 𝑒𝑗−5 |𝑎 𝑗,𝑗−2|
- 13. /18 Overview of EventNet 13 First event is arrived: 𝒆0 = {𝑒0} 𝑧0 = ℎ(𝑒0 − ) 𝑦0 = 𝑓 𝒆0 = 𝑔 max 𝑐 𝑧0, 0 𝑡 𝑐(𝑧0, 0) 𝑗 = 0 𝑗 = 1 Next event is arrived: 𝒆1 = {𝑒0, 𝑒1} 𝑠1 = max(𝑐 𝑧0, 𝑡1 − 𝑡0 , ℎ(𝑒1 − ) 𝑦1 = 𝑓 𝒆1 = 𝑔(𝑠1) Next event is arrived: 𝒆2 = {𝑒0, 𝑒1, 𝑒2} 𝑠2 = max 𝑐 𝑠1, 𝑡2 − 𝑡1 , ℎ 𝑒2 − 𝑦3 = 𝑔 𝑠2 𝑗 = 2
- 14. /18 EventNet Architecture 14 Create Look-up table(LUT) for decreasing inference time. (There are only 𝑊 × 𝐻 × 2 patterns in inputs 𝑒− ) ℎ(∙) 𝑔(∙)
- 15. /18 Experiments 15 ETHTED+ dataset for target motion estimation and semantic segmentation MVSEC dataset for ego-motion estimation
- 16. /18 Results for target motion estimation 16
- 17. /18 Results for ego-motion 17 Estimated ego motions are shown as yellow. Estimated at the rate of 1000 Hz.
- 18. /18 Compare the computation time [μs] 18 Using newly received event, PointNet need to process the all events again.