LBurst Learned Burst Feature Finder

Feature matching at night: A commercial drone (top-left) captures imagery at millilux conditions at night that is too noisy for conventional 3D reconstruction in low light (top-right). This is because of the high rate of spurious features offered by conventional features like scale invariant feature transform (SIFT) (blue) and less reliable matches offered by learned features like repeatable and reliable detector and descriptor (R2D2) (red) on noisy images. The proposed LBurst yields higher-quality feature matches from reliable regions of the images (yellow) for reconstruction.

We introduce a learning-based robotic burst feature finder, LBurst, a joint feature detector and descriptor that finds learned features with well defined scale within a low-light robotic burst
We enable robots to detect noise-tolerant features in low light by employing consistent motion within bursts and leveraging uncorrelated noise between them
We demonstrate overall improved 3D reconstruction from drone images in low-light (millilux) conditions, outperforming single image-based feature extractors and robotic burst feature finder in low-SNR scenes.

This work aims to identify learning-based features in low-light robotic bursts that enhance low-light reconstruction, benefiting a wide range of drone applications, including nighttime drone delivery and understanding nocturnal behaviors of animals.

Publications

• A. Ravendran, M. Bryson, and D. G. Dansereau, “LBurst: Learning-based robotic burst feature extraction for 3D reconstruction in low light,” 2024. Preprint available here.

This work was carried out within the Robotic Imaging Group at the Australian Centre for Robotics, University of Sydney in collaboration with CSIRO.

Themes

Robust Vision, Algorithms and Architectures, Learning to See

Downloads

The code is on GitHub here.

The drone imagery dataset used for validation is available here (11 GB download).

We capture robotic bursts along drone trajectories using DJI Phantom Pro 4 and DJI Mini Pro 3, each contains 15 bursts of 7 images each.

Gallery

(click to enlarge)

We locate learned features with well-defined scale within low-light robotic burst. By employing a flow map using known transformation between common images of the ground truth bursts during training, the network learns to find noise-tolerant features and describe using 128-dimensional descriptors.

We generate a pair of robotic bursts using a single image by enabling random homographic translations between bursts and introducing consistent uniform 2D translations within individual bursts. For every image in a robotic burst, we also introduce random Gaussian noise to simulate thermal noise present in low-light conditions.

During inference, we only pass a single burst to find multi-scale features.

We generate robotic bursts from the HPatches dataset consisting of progressively challenging images in terms of viewpoint and illumination. We compare the mean matching accuracy (MMA) for various pixel error thresholds using our feature finder on generated low-light robotic bursts and single-image based R2D2.

We compute ground truth by operating R2D2 on original high SNR images (black) and demonstrate improved overall matching performance.

We evaluate the accuracy of our camera trajectory estimation by validating the camera poses estimated using COLMAP. We align and scale the camera poses estimated through COLMAP with the ground truth poses as there is an inherent scale ambiguity in monocular SfM. The arbitrary scale factor is established based on the distance between the initial pair of registered images.

Our method converges for 60% of diverse viewpoint scenes, outperforming R2D2 (12.5%) with accurate pose estimates on the generated subset of HPatches robotic burst dataset with varying viewpoints.

We demonstrate LBurst features and their confidence for both detection and descriptors on captured scenes using DJI Phantom 4 Pro (top) and DJI Mini 3 Pro (bottom).

By selecting features with top-scored confidence for both detection and descriptor, we find high quality true features and avoid spurious features for reconstruction.

For a captured scene by DJI Mini 3 Pro, burst-based methods demonstrate an improved capability to locate more true features in both foreground and background areas.

Our feature finder avoids most of the spurious features and regions with low contrast, successfully locating high quality true features within the burst.

For a trajectory captured as multiple bursts by DJI Mini 3 Pro we demonstrate reconstruction evaluations using COLMAP.

State-of-the-art feature extractors, both classical and learning-based methods demonstrate poor performance on captured single images.

We operate SIFT, SuperPoint and R2D2 on burst-merged images which benefit from improved SNR. We also directly employ robotic bursts with BuFF and our learning-based feature finder.

Overall, our proposed approach, LBurst reconstructs the scene with all input images, with improved 3D points and complete structure.

Our method outperforms state-of-the-art feature detection methods on both conventional and burst-merged imagery by reconstructing most of the scenes with the highest convergence rate and an overall inproved 3D points.

Learned burst features, LBurst, also show improved performance compared to physics-based BuFF feature finder.

Citing

If you find this work useful please cite

@article{ravendran2024lburst,
  title={{LBurst}: Learning-based Robotic Burst Feature Extraction for {3D} Reconstruction in Low Light},
  author={Ravendran, Ahalya and Bryson, Mitch and Dansereau, Donald G.},
  journal={arXiv preprint arXiv:2410.23522},
  year={2024}
}