Volumetric Focus
Light field cameras, both array and lenslet-based, gather more light for a given depth of field than conventional cameras. The extra light manifests itself as redundancy in the light field, which must be processed in order to be exploited. Conventional approaches attempt depth estimation directly from the noisy data, or focus on a plane in the scene to reduce noise, trading off depth of field.
We introduce volumetric focus, a linear noise-rejecting filter that maintains focus over a controllable range of depths, rather than focusing on a single plane. It rejects noise even when keeping the whole scene in focus.
- We derive the 4D hyperfan-shaped frequency-domain region of support of the light field, a subset of a 4D hypercone
- We design linear hyperfan filters and show them outperforming competing methods including planar focus, 2D fan filters, and a range of nonlinear image and video denoising techniques
- We include results in low light and through murky water and particulate matter
- We demonstrate the inclusion of aliased components for high-quality rendering
Publications
• D. G. Dansereau, O. Pizarro, and S. B. Williams, “Linear volumetric focus for light field cameras,” ACM Transactions on Graphics (TOG), Presented at SIGGRAPH 2015, vol. 34, no. 2, p. 15, Feb. 2015. Available here, or uncompressed here.
• D. G. Dansereau, D. L. Bongiorno, O. Pizarro, and S. B. Williams, “Light field image denoising using a linear 4D frequency-hyperfan all-in-focus filter,” in Proceedings SPIE Computational Imaging XI, 2013, p. 86570P. Available here.
Conventional focus gathers more light, increasing the signal-to-noise ratio (SNR), but decreasing depth of field. This video shows how conventional cameras pick out a plane of sharp focus, with sharpness falling off away from this plane.
Volumetric focus lets us define a range of depths to keep in focus. Sharpness falls off away from this volume, but the scene is always sharp within the volume. This kind of focus is impossible with conventional cameras, and is only enabled by virtue of the redundant information in light field imaging.
With volumetric focus, noise selectivity is partially independent from depth selectivity. Here we see the noise selectivity being adjusted without changing the range of depths that's in focus. By placing the entire scene in focus, we make an all-in-focus noise-rejecting filter.
A low-light scene captured using a Lytro Illum. Here we see the input, a gain-adjusted version, the fan-filtered version showing appreciable improvement, and finally a median-filtered version showing still more improvement.
Another low-light example captured using the Lytro Illum.
A low-light example using a Lytro F01.
In underwater imaging the volumetric focus filter serves two purposes: first, we can improve the SNR in regions that show low contrast due to backscatter and attenuation in the water column. This requires only a hypercone filter, leaving the whole scene in focus, and rejecting noise while maintaining depth of field as in the low light examples.
Second, we can focus behind occluding particulate matter. This scene shows significant depth variation, as the smiling target is about halfway between the camera and the checkerboard, but there is still a volume between the camera and the scene which we can focus around. Here the particulate in that volume is significantly attenuated by the hyperfan filter, while keeping the rest of the scene in focus.
Another underwater scene, showing improvement in SNR and removal of particulate.
A light field of a Lorikeet taken through a grubby window.
The same with a planar focus filter. Note the grubby window is attenuated, but detail in the background is also lost.
A fan filter set to pass a volume containing the Lorikeet and the background, but not the window. The grubby glass is significantly attenuated, without loss of detail in the background. Depth information is also maintained, not the case for planar filters.
The so-called "dimensionality gap" has long been recognized: the light field lies on a 3D manifold in the frequency domain. Here we show this manifold as it rotates in 4D space.
By applying a specific set of rotations, we see how the "dimensionality gap" manifold is actually a hypercone, a circle that grows larger further from the origin. This is a saddle hypercone, not a spherical hypercone. Please see Sect. 4.3 of the
TOG paper for the mathematical derivation of this shape.
The same rotating hypercone as above, visualized as a row of 3D slices in 3D space.
As above, focused on the near half of the board.
As above, focused on the near half of the scene.
More complex shapes can be constructed as the superposition of linear hyperfan filters. Here a volume coincident with the crystal ball is left as a stopband, and the rest of the scene is kept in focus. This single filter was constructed as the superposition of two hyperfan filters in the frequency domain.
