https://arxiv.org/pdf/1912.08804.pdf

Abstract

• View synthesis
  • Generating new views of a scene given one or more images
  • Challenging as it requires comprehensively understanding the 3D scene from images
• Synsis
  • A novel differentiable point cloud renderer(latent 3D feature point cloud → target view)
  • Refinement network

Introduction

View Synthesis

Given a view of a scene, view synthesis aims to generate images of the scene from new viewpoints

• Requires understanding both 3D structure and semantics of the input image

Methods

• Using multiple views to reconstruct 3D scene geometry
  • [-] Not practical
    • Vast majority of images are not accompanied by views from other angles
• Training CNN to estimate depth from image
  • [-] Requires GT depth
  • [-] Depth prediction are not generalized
    • e.g. indoor, outdoor
• Synsin
  • End-to-End without GT 3D supervision
  • Only need a single image at test time

Method

• Input image $I$
• Relative pose $T$
• Feature space $F$
  • Spatial feature predictor $f$