NOCaL: Calibration-Free Odometry

Given a pair of input frames NOCaL can render novel synthetic views of the captured scene. As NOCaL can produce novel views for multiple cameras geometries, it allows NOCaL to use this as supervision of odometry for a range of cameras.

There are a multitude of emerging imaging technologies that could benefit robotics. However the need for bespoke models, calibration and low-level processing represents a key barrier to their adoption. In this work we present NOCaL, Neural Odometry and Calibration using Light fields, a semi-supervised learning architecture capable of interpreting cameras without calibration.

NOCaL automatically interprets previously unseen cameras by jointly learning to estimate novel views, odometry, and camera parameters.
A rendering hypernetwork is leveraged to allow us to benefit from a large pool of existing data.
We propose using a ray based light field rendering network for supervision of odometry, which can generates views in a camera agnostic way. This allows NOCaL to be used for multiple cameras, even ones that haven't been created yet.

This allows use of a broad range of monocular cameras without calibration. This work is a key step toward automatically interpreting more general camera geometries and emerging camera technologies.

Publications

• R. Griffiths, J. Naylor, and D. G. Dansereau, “NOCaL: Calibration-free semi-supervised learning of odometry and camera intrinsics,” Robotics and Automation (ICRA), 2023. Preprint here.

This work was carried out within the Robotic Imaging Group at the Sydney Institute for Robotics and Intelligent Systems, Australian Centre for Field Robotics, University of Sydney.

Themes

Novel Cameras, Learning to See

Code and Dataset

The code is available on GitHub here.
The dataset used is from the LearnLFOdo project and can be requested via email here.

Gallery

(click to enlarge)

The proposed NOCaL network architecture has 4 separate sub-modules. The camera interpreter (green) encodes input frames into a latent space z. The scene rendering hypernetwork (orange) produces weights for the light field network (blue). The camera model (yellow) has learnable parameters to be able to estimate the camera intrinsics.

The intrinsics K are first used to calculate the distorted points on the image plane. A distortion network determines the distortion values for individual pixels. These corrections are applied to the image plane producing undistorted points, which are converted to rays in world space.

View synthesis from the learnt LFN on a test scene. A pair of input images at times i, i + 1 are shown on the left, with the corresponding predicted views on the right closely matching in visual appearance. The temporal difference frames show the motion through time, it can be seen that the predicted and measured motion are very similar.

Calibration results. It can be seen here that NOCaL can refine the camera parameters from some initilisation to better reflect the real geometry that the camera has.

We compared NOCaL to supervised and unsupervised alternatives and showed that our approach is able to leverage the rendering network to perform some supervision on the unlabelled data. NOCaL outperforms the alternative methods, while also not requiring camera intrinsics.

Using a ray based rendering method allows us to model and predict views for many camera geometries. Here the focal length and distortion parameters have been varied. This shows that our method can render views from previously unseen cameras, which is a major advantage on this method.

Citing

If you find this work useful please cite

@article{griffiths2023nocal,
  title = {{NOCaL}: Calibration-Free Semi-Supervised Learning of Odometry and Camera Intrinsics},
  author = {Ryan Griffiths and Jack Naylor and Donald G. Dansereau},
  journal = {Robotics and Automation ({ICRA})},
  year = {2023}
}