Learning to See with Sparse Light Field Video Cameras

Learning to use new kinds of cameras in robotics applications is challenging and time-consuming. This work represents a first step toward streamlining the integration of new kinds of imaging devices into robotics applications. We use unsupervised learning to simultaneously learn metric depth and odometry using an emerging sensing technology, the sparse light field (LF) camera.

We generalize unsupervised odometry and depth estimation to operate on sparse 4D light fields
We introduce an encoding scheme for sparse LFs appropriate for odometry and shape estimation
We outperform the monocular approach, yielding more accurate trajectories and depth maps with known scale

We expect our method to work well with other cameras with regular overlapping views: regularly spaced 1D and 2D camera arrays, sparse cameras like the EPIModule, and lenslet-based plenoptic cameras like the Lytro and Raytrix devices.

Publications

• S. T. Digumarti, J. Daniel, A. Ravendran, R. Griffiths, and D. G. Dansereau, “Unsupervised learning of depth estimation and visual odometry for sparse light field cameras,” in Intelligent Robots and Systems (IROS), 2021. Preprint here.

This work was carried out within the Robotic Imaging Group at the Sydney Institute for Robotics and Intelligent Systems, Australian Centre for Field Robotics, University of Sydney.

Acknowledgments

We would like to thank EPIImaging LLC and the University of Sydney Aerospace, Mechanical and Mechatronic Engineering FabLab for their support, and the University of Sydney HPC service for providing HPC resources that contributed to the research results reported in this work.

Themes

Novel Cameras, Learning to See

Downloads

The code is on GitHub here.

Data: please request instant access via return email here.

The EPIModule captures 17 overlapping views in the configuration shown at left. We mounted the module on a UR5e robotic arm and captured video over 46 trajectories in a variety of indoor scenes, yielding a total of 8298 LFs. After downsampling as described in the paper, the dataset occupies 13 GBytes.

See the dataset readme file here for further details.

Gallery

(click to enlarge)

The proposed network architecture uses prediction as a signal for unsupervised learning of a depth map D and pose T. We introduce the encoding layers (purple) that allow a sparse LF input to drive 2D pose and depth networks.

We estimate depth over all LF pixels and perform a differentiable 4D warp to relate adjacent LF frames, such that the resulting photometric loss makes use of all measured pixels.

The encoding layer (shown above, purple) is achieved using tiled EPI stacks (left) and a single-layer CNN encoder, to produce a stack of 2D images (right).

We demonstrate the proposed approach outperforming monocular and competing LF encoding schemes, yielding more accurate depth maps and camera trajectories.

Depth from monocular imagery misses important scene detail and, as seen in the quantitative evaluation below, lacks scale.

Depth from LF using single-warp evaluation considers photometric loss only on the central view of the predicted LF.

The proposed encoding yields more scene detail compared with monocular imaging, and outperforms LF encoding via focal and volumetric stacking.

Depth from LF using multi-warp evaluation considers all predicted LF pixels and yields improved results.

Again the proposed encoding scheme outperforms focal and volumetric stacking.

Stereo imagery improves on the monocular result, but because geometric information is less rich and less readily available than in LF imagery, the results are not as strong as in the LF examples. Further comparisons below...

A comparison of depth maps using a range of methods and different scenes.

The depth results are delivered with metric scale. The proposed method using multi-warp evaluation (violet) is the only approach which follows the ground truth trend.

Evaluating odometry accuracy, the proposed approach outperforms monocular and stereo imaging and alternative LF encoding methods.

Citing

If you find this work useful please cite

@inproceedings{digumarti2021unsupervised,
  title = {Unsupervised Learning of Depth Estimation and Visual Odometry for Sparse Light Field Cameras},
  author = {Sundara Tejaswi Digumarti and Joseph Daniel and Ahalya Ravendran and Ryan Griffiths and Donald G. Dansereau},
  booktitle = {Intelligent Robots and Systems ({IROS})},
  year = {2021},
  publisher = {IEEE}
}