Neural LightRig

Unlocking Accurate Object Normal and Material Estimation
with Multi-Light Diffusion

Zexin He1*, Tengfei Wang2*, Xin Huang2, Xingang Pan3, Ziwei Liu3
1The Chinese University of Hong Kong, 2Shanghai AI Lab, 3Nanyang Technological University
Paper arXiv Code Data

TL;DR: Neural LightRig addresses the ill-posed nature of estimating surface properties
from a single image by generating multi-light images with a diffusion model, providing
auxiliary information for accurately predicting object normals and PBR materials.

Abstract

Recovering the geometry and materials of objects from a single image is challenging due to its under-constrained nature. In this paper, we present Neural LightRig, a novel framework that boosts intrinsic estimation by leveraging auxiliary multi-lighting conditions from 2D diffusion priors.

1) We first leverage illumination priors from large-scale diffusion models to build our Multi-Light Diffusion Model on a synthetic relighting dataset with dedicated designs. This diffusion model generates multiple consistent images, each illuminated by point light sources in different directions.
2) By using these varied lighting images to reduce estimation uncertainty, we train a Large G-Buffer Model with a U-Net backbone to accurately predict surface normals and materials.

Extensive experiments validate that our approach significantly outperforms state-of-the-art methods, enabling accurate surface normal and PBR material estimation with vivid relighting effects.

Method

Method Overview

Framework Overview. Multi-light diffusion generates multi-light images from an input image. These images with corresponding lighting orientations are then used to predict surface normals and PBR materials with a regression U-Net.

Selected Light: 1

Light Source

Point Light Source

Light Image

Light Image

Comparisons

Surface Normal Estimation

Normal Comparison

Qualitative comparison on surface normal estimation. Ground truth normals (G.T.) are provided for input images rendered from available 3D objects and are omitted for in-the-wild images.

PBR Material Estimation

PBR Comparison

Qualitative comparison on PBR material estimation. Ground truth materials (G.T.) are provided for input images rendered from available 3D objects and are omitted for in-the-wild images.

Single-Image Relighting

Relighting Comparison

Qualitative comparison on single-image relighting. The input image is relit with the estimated normal and material under different lighting conditions.

RGB-X

DiLightNet

IC-Light

Yi. et al

IntrinsicAnything

Ours

G.T.

Demo Video

BibTeX

@misc{neural_lightrig,
    title={Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion}, 
    author={Zexin He and Tengfei Wang and Xin Huang and Xingang Pan and Ziwei Liu},
    year={2024},
    eprint={2412.09593},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2412.09593},
}