Neural Ray-Tracing: Learning Surfaces and Reflectance for Relighting and View Synthesis

Abstract

Recent neural rendering methods have demonstrated accurate view interpolation by predicting volumetric density and color with a neural network. Although such volumetric representations can be supervised on static and dynamic scenes, existing methods implicitly bake the scene’s light transport into a single neural network, which includes surface modeling, bidirectional scattering distribution functions (BSDFs), and indirect lighting effects. In contrast to traditional rendering pipelines, this prohibits changing surface reflectance, illumination, or composing objects to produce new scenes.

In this work, we explicitly model the light transport between scene surfaces and rely on traditional integration schemes and the rendering equation to reconstruct a scene. The proposed method allows BSDF recovery with unknown lighting conditions and classic light integration approaches such as path tracing. By learning decomposed transport with surface representations which are well established in conventional rendering methods, our method naturally facilitates editing shape, reflectance, lighting and scene composition. The method outperforms NeRV for relighting under known lighting conditions, and produces realistic reconstructions for relit and edited scenes. We validate the proposed approach for scene editing, relighting and reflectance estimation learned from synthetic and captured views on a subset of NeRV’s datasets, and perform qualitative comparisons on the DTU and NeRF dataset.