Light-Constrained Structure-from-Motion

(bottom) red: Finding conventional features like Scale Invariant Feature Transform (SIFT) in light-constrained scenes can yield spurious features and break 3D vision algorithms like Structure from Motion (SfM); yellow: We propose burst imaging for light-constrained SfM that shows more stable set of true features and less spurious features, enabling SfM to operate in extreme low light.

We establish the viability of using burst imaging to improve robotic vision in low light, and provide a set of recommendations for adopting this approach in reconstruction tasks such as SfM
We evaluate different approaches to burst imaging in robotics applications and show that burst capture with merge offers significant advantages in both computational requirements and performance
We demonstrate more precise 3D points, more true features, fewer spurious features, more precise camera trajectories and an ability to operate in lower light than was previously possible

This work opens the way for a broad range of applications in which low light commonly complicates vision such as autonomous driving and delivery drones.

Publications

• A. Ravendran, M. Bryson, and D. G. Dansereau, “Burst imaging for light-constrained structure-from-motion,” IEEE Robotics and Automation Letters (RA-L, ICRA), vol. 7, no. 2, pp. 1040–1047, Apr. 2022. Available here.

Citing

If you find this work useful please cite

@article{ravendran2022burst,
  title={Burst Imaging for Light-Constrained Structure-From-Motion}, 
  author={Ahalya Ravendran and Mitch Bryson and Donald G. Dansereau},
  journal = {IEEE Robotics and Automation Letters ({RA-L, ICRA})},
  year = {2022},
  volume = {7}, 
  issue = {2},
  pages = {1040--1047},
  month = apr
}

This work was carried out within the Robotic Imaging Group at the Sydney Institute for Robotics and Intelligent Systems, Australian Centre for Field Robotics, University of Sydney.

Themes

Robust Vision, Algorithms and Architectures

Downloads

The code for Light-Constrained SfM is on GitHub here.

The dataset used in the Light-Constrained SfM paper is here (13 GB or 32 GB download).

The dataset was captured with a UR5e robotic arm-mounted monocular camera, and contains 22 bursts of 7 images each, for 20 challenging scenes with a variety of objects in an environment with controlled lighting.

Gallery

(click to enlarge)

We improve the noise in the captured images by aligning every image in a burst with the chosen reference image of the burst. We merge the aligned images in temporal direction with a voting scheme to avoid misalignment and perform filtering on temporally merged images.

Comparison of alternative exposure schemes:

Single frame captured over an exposure time of t ms.
Multiple frames captured over short exposure times and merged naïvely. This is representative of a long exposure with motion blur.
Aligning and merging frames.

Detected features on synthetic images using SIFT, presented at noise level σ = 0.03. Our method performs well with no spurious features; conventional noisy images and using multiple images from the burst both yield many spurious features.

At higher noise, σ = 0.1, while our method detects all true features, conventional noisy approach detects more spurious features and less true features than ours and burst without merge registers overwhelming spurious features.

Noise performance: (top row) Sweeping noise level σ for fixed detection thresholds, our method has the best true positive (TP) rate.

(bottom row): Sweeping detection threshold, our method delivers a much higher TP rate and lower false positive (FP) count in high noise for appropriately set threshold.

Our method matches or outperforms conventional method in feature performance in noise.

Camera trajectory estimation: The proposed method employed all input images and yielded a more accurate trajectory estimate.

Mean rotational and translation error in camera pose estimation for different reconstruction methods. Our approach outperforms conventional image capture and is competitive with burst without merge, while requiring a fraction the overall compute. (red): Best results from conventional approaches; (green): Best results from our proposed approach.

Average SfM reconstruction performance for light-constrained scenes. With COLMAP's default settings, our approach outperforms conventional imaging in all measures in both well-lit (1ms exposure) and poorly lit (0.1ms exposure) scenes. By tuning COLMAP parameters ("Peak Threshold"), our approach performs successful reconstruction in extremely low light.