We present a method to estimate an HDR environment map from a narrow field-of-view LDR camera image in real-time. This enables perceptually appealing reflections and shading on virtual objects of any material finish, from mirror to diffuse, rendered into a real environment using augmented reality. Our method is based on our efficient convolutional neural network, EnvMapNet, trained end-to-end with two novel losses, ProjectionLoss for the generated image, and ClusterLoss for adversarial training. Through qualitative and quantitative comparison to state-of-the-art methods, we demonstrate that our algorithm reduces the directional error of estimated light sources by more than 50 percent, and achieves 3.7 times lower Frechet Inception Distance (FID). We further showcase a mobile application that is able to run our neural network model in under 9 ms on an iPhone XS, and render in real-time, visually coherent virtual objects in previously unseen real-world environments.

Related readings and updates.

CVPR 2021

Apple sponsored the annual conference of Computer Vision and Pattern Recognition (CVPR). The conference focuses on computer vision and its applications and took place virtually from June 19 to 25.

See event details

ASSETS 2020

Apple sponsored the twenty-second ACM SIGACCESS Conference on Computers and Accessibility (ASSETS), which was held virtually from October 26 to 28. ASSETS is a global conference focused on education surrounding computing for older adults and people with disabilities.

See event details