Abstract

• Recent neural representations require using dense multi-view inputs and costly training
• HumanNeRF
  • A neural representation with efficient generalization ability for high-fidelity free-view synthesis of dynamic humans
    • Aggregated pixel-alignment feature across multiple view inputs
    • Pose embedded non-rigid deformation field
    • In-hour scene-specific fine-tuning
    • Appearance blending module

Motivations and Contributions

Motivation

• NeRF’s shortcomings
  • Requiring expensive dense capture views
  • Suffer from tedious time-consuming per-scene training

Contributions

• High quality performance rendering approach via efficient radiance field generation for arbitrary performers from sparse RGB streams.
• Dynamic and light-weight setting trough implicit feature aggregation and hybrid deformation
• Implicit blending scheme to preserve texture detail from the input images

HumanNeRF Approach

1. Generalizable Dynamic Neural Radiance Field

• Two main changes to NeRF for human dynamics and generalization

  • Aggregated Pixel Alignment Feature

  

  input image $I^k \in$ $\mathbb{R}^{H \times W \times 4}$

  output

  

  out put $f^k$ is fed to mlp

  

  which are the blending weights

  • Pose Embedded Non-rigid Human Deformation

    • pose embedded non-rigid deformation field can be formulated as:

      

    • The final generalizable dynamic neural radiance fied outputs the following:

      

  

2. Neural Appearance Blending

• Textures produced by NeRF lack high-frequency due to the sparsity of input images
• Texture information can be recovered by adjacent input views

  • Appearance blending weight gained by appearance blending network MLP

    

  • The final color for q in v is: