1. Introduction

• Deformation based nerfs struggle to models changes in topology.
• HyperNeRF adapts the level set framework for Deformable NeRF(Nerfies) to generate photorealistic, free-viewpoint renderings of objects undergoing topological changes

2. Related Work

2-1. Neural Rendering

• Paradigm shift from “Image to image translation” to “Neural scene representation”, using the weights of a NN to directly model some aspect of the scene itself
• Deformation-based NeRFs
  • Use MLP to parameterize the deformation field of a scene
  • Observation coordinates > canonical coordinates > template NeRF
  • Nerfies, D-NeRF, NR-NeRF…
• Modulation-based(latent-conditioned) NeRFs
  • Conditioning NN with a latent code to modulate its output

2-2. Modeling Time-Varying Shapes

2-2-1. Level Set Methods

• Modeling a surface that varies topologically with respect to some additional dimensions (”ambient” space )

2-2-2. Deformable Slicing Surfaces

• Axis-aligned Plane(AP)
  • Need copy of each AP separately
• Deformable Slicing Surface(DS)
  • Can share information resulting in simpler ambient surfaces
  • Defined as MLP
  • $H : (\mathbf{x},\omega_i) \rightarrow \mathbf{w}$
    • $\mathbf{x}$ : spatial position
    • $\mathbf{w}$ : position along ambient axes
    • $\omega_i$ per-input latent embedding

3. Method

3-1. Hyper-space Template

• Deformable NeRFs represent a scene in a canonical-space template NeRF indexed by spatial deformation field
• Hyper-space template NeRF takes template NeRF in to a high dimension
  • Slice from high-dimension surface yields a full 3D NeRF