Conditional visibility aware view synthesis via parallel light fields

Yu Shen

doi:10.1016/j.neucom.2024.127644

2024-07-01

PRE

DOI：10.1016/j.neucom.2024.127644

原文链接

原文求助

摘要

En 中文

In the area of neural rendering -based novel view synthesis, illumination is important since shadows cast by objects under various light sources provide indications about their geometries and materials. However, due to high physical device complexity and simulation distortion, large-scale photorealistic multiple illumination multi -view datasets are difficult to obtain. In order to address this problem, a physical -virtual interactive parallel light fields based collection method is proposed in this paper. The physical part of parallel light fields is firstly used to capture 3D models and 2D images of objects under different lights. Then a Reakto-Sim adaptation module was proposed to enhance realism by estimating material characteristic. Instead of manually setting, the learned resulting material parameters are then utilized to initialize virtual engine blender for subsequent rendering and data collection. Besides, to better handle self -occlusion problem in the acquired parallel light fields dataset, a conditional visibility module is designed in modeling visibility of each sampling point along a sampling ray. Compared with the Neuray, by introducing Conditional Normalizing Flow, visibility are assumed as samples from some distribution due to the fact that visibilities along the ray should be monotonically decreasing and are within the range of [0 , 1] . The visibility are calculated in a data driven manner, which brings more flexibility. By pretraining the conditional visibility network in parallel light field dataset, experiments demonstrate that more photorealistic inputs improve Peak -Signal -Noise Ratio (PSNR) and Structure Similarity Index Measure (SSIM) by 0.11% and 0.68% in validation dataset NeRF synthesis and LLFF. Besides, compared to Neuray, the proposed conditional visibility module is more flexible and get a PSNR improvement of 0.55 and 0.5 in NeRF synthesis and LLFF dataset, respectively.

Keyword:

Parallel theory

Light fields

Neural rendering

View synthesis

Conditional visibility

Normalizing Flow