profile photo

NeLF-Pro

Neural Light Field Probes for Multi-Scale Novel View Synthesis


CVPR 2024


Zinuo You1     Andreas Geiger2,3     Anpei Chen1,2,3
1ETH Zürich     2University of Tübingen     3Tübingen AI Center    

TL;DR: NeLF-Pro represents and reconstructs diverse natural scenes using spatially distributed, learnable light field probes.

Abstract

We present NeLF-Pro, a novel representation for modeling and reconstructing light fields in diverse natural scenes that vary in extend and spatial granularity. In contrast to previous fast reconstruction methods that represent the 3D scene globally, we model the light field of a scene as a set of local light field feature probes, parameterized with position and multi-channel 2D feature maps. Our central idea is to bake the scene's light field into spatially varying learnable representations and to query point features by weighted blending of probes close to the camera - allowing for mipmap representation and rendering. We introduce a novel vector-matrix-matrix (VMM) factorization technique that effectively represents the light field feature probes as products of core factors (i.e., VM) shared among local feature probes, and a basis factor (i.e., M) - efficiently encoding internal relationships and patterns within the scene. Experimentally, we demonstrate that NeLF-Pro significantly boosts the performance of feature grid-based representations, and achieves fast reconstruction with better rendering quality while maintaining compact modeling.

Method

Our approach models a scene using light field probes represented by a set of core vectors, core matrices as well as a set of basis matrices. We project the sample points onto local probes to obtain local spherical coordinates and then query factors from probes near the camera. These factors are then aggregated using a self-attention mechanism and combined using Hadamard products. The resulting aggregated feature vector and the viewing direction are used to calculate the volume density and the light field radiance.

Video

References

BibTeX


    @inproceedings{You2024CVPR,
      author = {Zinuo You and Andreas Geiger and Anpei Chen},
      title = {NeLF-Pro: Neural Light Field Probes for Multi-Scale Novel View Synthesis},
      booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
      year = {2024}
    }
    

Acknowledgements

Andreas Geiger and Anpei Chen are supported by the ERC Starting Grant LEGO-3D (850533) and DFG EXC number 2064/1 - project number 390727645. We thank Zehao Yu and Binbin Huang for proofreading and their insightful suggestions. We also appreciate Siyu Tang and Kaifeng Zhao for their valuable discussions and help. Special thanks to The Flying Dutchman for the video of city walk.