E2SL: Efficient Depth Sensing from Event-based Abstract
Structured light (SL) is a popular approach for 3D reconstruction. Most SL techniques rely on frame-based cameras and are often not robust in high-speed dynamic scenes. Recently, event cameras have sparked growing interest in high-speed SL imaging, due to their high temporal resolution. The event-based SL enjoys the high-speed data acquisition, however, most existing methods tend to pursue the reconstruction accuracy but sacrificing the computational efficiency, limiting the applicability in real-world scenarios. To this end, we propose E2SL, an Efficient deep network tailored for monocular Event-based SL. Specifically, E2SL comprises three key components: binary embedding lookup table (BE-LUT), spatial context enhancement (SCE), and geometric-prior regression (GPR). Given the input event frame, BE-LUT, which is precomputed and stored, first retrieves the features efficiently. Then, SCE extends the receptive field of the features and captures the spatial context. Finally, GPR conducts the geometric-prior-based tree classification for fast and robust depth estimation. To support training and evaluation, we contribute an event-based SL simulator, which generates a large-scale and diverse synthetic dataset. Besides, we develop an event-based SL prototype and collect a dataset with accurate ground truth for real-world evaluation. Extensive experiments demonstrate that our method achieves state-of-the-art accuracy while maintaining a per-frame reconstruction time of 7.7 ms, meeting the demands of high-speed depth sensing.
Overview
Overview of our framework. (1) Event-based structured light dataset synthesis and proposed E2SL training. (2) Real-time reconstructed point clouds from the dynamic real-world scenarios.
Synthetic Dataset & Real-world System
The pipeline for constructing the synthetic event-based SL dataset. (1) Based on the frame-based SL renderer, the rendered frames are rendered by controlling the on and off of the projector. (2) The logarithmic intensity difference (log-diff) frames between adjacent frames are then calculated. (3) The event stream is triggered based on log-diff frames. (4) Finally, the positive events are accumulated to generate the event binary frames.
The pipeline for the alignment between event-based SL real-world and simulated systems. (1) We set up the event-based SL system. (2) The calibration event frames are captured using the circular pattern. (3) Calibration is performed, followed by epipolar rectification to obtain geometric parameters. (4) Speckle pattern. (5) The speckle pattern is rectified and padded. (6) Rectified geometric parameters and the processed speckle pattern are fed into the SL simulator.
Method
The overview of efficient event-based SL (E
Results
Qualitative results in real-world test set. Ground truth and scene intensity are shown in 1st column. Disparity and error maps are shown in 2nd-6th columns. Color coding is applied for error maps of 0–2 pixels, with outliers (>2 pixels) being black.
Point cloud qualitative results of SGBM and our method on real-world dynamic scenes with comparisons of the completeness. Scenes are shown in 1st row. Point clouds reconstructed from different methods are shown in 2nd-3rd rows.