3D Face Reconstruction from Light Field Images: A Model-free Approach

Mingtao Feng, Syed Zulqarnain Gilani, Yaonan Wang, Ajmal Mian

Research output: Chapter in Book/Conference paperConference paper

Abstract

Reconstructing 3D facial geometry from a single RGB image has recently instigated wide research interest. However, it is still an ill-posed problem and most methods rely on prior models hence undermining the accuracy of the recovered 3D faces. In this paper, we exploit the Epipolar Plane Images (EPI) obtained from light field cameras and learn CNN models that recover horizontal and vertical 3D facial curves from the respective horizontal and vertical EPIs. Our 3D face reconstruction network (FaceLFnet) comprises a densely connected architecture to learn accurate 3D facial curves from low resolution EPIs. To train the proposed FaceLFnets from scratch, we synthesize photo-realistic light field images from 3D facial scans. The curve by curve 3D face estimation approach allows the networks to learn from only 14K images of 80 identities, which still comprises over 11 Million EPIs/curves. The estimated facial curves are merged into a single pointcloud to which a surface is fitted to get the final 3D face. Our method is model-free, requires only a few training samples to learn FaceLFnet and can reconstruct 3D faces with high accuracy from single light field images under varying poses, expressions and lighting conditions. Comparison on the BU-3DFE and BU-4DFE datasets show that our method reduces reconstruction errors by over 20% compared to recent state of the art.
Original languageEnglish
Title of host publicationEuropean Conference on Computer Vision
EditorsV Ferrari, M Hebert, C Sminchisescu, Y Weiss
PublisherSpringer
Pages508-526
Volume11214
ISBN (Electronic)9783030012496
ISBN (Print)9783030012489
DOIs
Publication statusPublished - 8 Sep 2018
Event15th European Conference on Computer Vision - Munich, Germany
Duration: 8 Sep 201814 Sep 2018

Conference

Conference15th European Conference on Computer Vision
Abbreviated titleECCV 2018
CountryGermany
CityMunich
Period8/09/1814/09/18

Fingerprint

Lighting
Cameras
Geometry

Cite this

Feng, M., Gilani, S. Z., Wang, Y., & Mian, A. (2018). 3D Face Reconstruction from Light Field Images: A Model-free Approach. In V. Ferrari, M. Hebert, C. Sminchisescu, & Y. Weiss (Eds.), European Conference on Computer Vision (Vol. 11214, pp. 508-526). Springer. https://doi.org/10.1007/978-3-030-01249-6_31
Feng, Mingtao ; Gilani, Syed Zulqarnain ; Wang, Yaonan ; Mian, Ajmal. / 3D Face Reconstruction from Light Field Images: A Model-free Approach. European Conference on Computer Vision. editor / V Ferrari ; M Hebert ; C Sminchisescu ; Y Weiss. Vol. 11214 Springer, 2018. pp. 508-526
@inproceedings{23d5aad9c1c84547a700fc77192eeaf5,
title = "3D Face Reconstruction from Light Field Images: A Model-free Approach",
abstract = "Reconstructing 3D facial geometry from a single RGB image has recently instigated wide research interest. However, it is still an ill-posed problem and most methods rely on prior models hence undermining the accuracy of the recovered 3D faces. In this paper, we exploit the Epipolar Plane Images (EPI) obtained from light field cameras and learn CNN models that recover horizontal and vertical 3D facial curves from the respective horizontal and vertical EPIs. Our 3D face reconstruction network (FaceLFnet) comprises a densely connected architecture to learn accurate 3D facial curves from low resolution EPIs. To train the proposed FaceLFnets from scratch, we synthesize photo-realistic light field images from 3D facial scans. The curve by curve 3D face estimation approach allows the networks to learn from only 14K images of 80 identities, which still comprises over 11 Million EPIs/curves. The estimated facial curves are merged into a single pointcloud to which a surface is fitted to get the final 3D face. Our method is model-free, requires only a few training samples to learn FaceLFnet and can reconstruct 3D faces with high accuracy from single light field images under varying poses, expressions and lighting conditions. Comparison on the BU-3DFE and BU-4DFE datasets show that our method reduces reconstruction errors by over 20{\%} compared to recent state of the art.",
author = "Mingtao Feng and Gilani, {Syed Zulqarnain} and Yaonan Wang and Ajmal Mian",
year = "2018",
month = "9",
day = "8",
doi = "10.1007/978-3-030-01249-6_31",
language = "English",
isbn = "9783030012489",
volume = "11214",
pages = "508--526",
editor = "V Ferrari and M Hebert and C Sminchisescu and Y Weiss",
booktitle = "European Conference on Computer Vision",
publisher = "Springer",

}

Feng, M, Gilani, SZ, Wang, Y & Mian, A 2018, 3D Face Reconstruction from Light Field Images: A Model-free Approach. in V Ferrari, M Hebert, C Sminchisescu & Y Weiss (eds), European Conference on Computer Vision. vol. 11214, Springer, pp. 508-526, 15th European Conference on Computer Vision, Munich, Germany, 8/09/18. https://doi.org/10.1007/978-3-030-01249-6_31

3D Face Reconstruction from Light Field Images: A Model-free Approach. / Feng, Mingtao; Gilani, Syed Zulqarnain; Wang, Yaonan; Mian, Ajmal.

European Conference on Computer Vision. ed. / V Ferrari; M Hebert; C Sminchisescu; Y Weiss. Vol. 11214 Springer, 2018. p. 508-526.

Research output: Chapter in Book/Conference paperConference paper

TY - GEN

T1 - 3D Face Reconstruction from Light Field Images: A Model-free Approach

AU - Feng, Mingtao

AU - Gilani, Syed Zulqarnain

AU - Wang, Yaonan

AU - Mian, Ajmal

PY - 2018/9/8

Y1 - 2018/9/8

N2 - Reconstructing 3D facial geometry from a single RGB image has recently instigated wide research interest. However, it is still an ill-posed problem and most methods rely on prior models hence undermining the accuracy of the recovered 3D faces. In this paper, we exploit the Epipolar Plane Images (EPI) obtained from light field cameras and learn CNN models that recover horizontal and vertical 3D facial curves from the respective horizontal and vertical EPIs. Our 3D face reconstruction network (FaceLFnet) comprises a densely connected architecture to learn accurate 3D facial curves from low resolution EPIs. To train the proposed FaceLFnets from scratch, we synthesize photo-realistic light field images from 3D facial scans. The curve by curve 3D face estimation approach allows the networks to learn from only 14K images of 80 identities, which still comprises over 11 Million EPIs/curves. The estimated facial curves are merged into a single pointcloud to which a surface is fitted to get the final 3D face. Our method is model-free, requires only a few training samples to learn FaceLFnet and can reconstruct 3D faces with high accuracy from single light field images under varying poses, expressions and lighting conditions. Comparison on the BU-3DFE and BU-4DFE datasets show that our method reduces reconstruction errors by over 20% compared to recent state of the art.

AB - Reconstructing 3D facial geometry from a single RGB image has recently instigated wide research interest. However, it is still an ill-posed problem and most methods rely on prior models hence undermining the accuracy of the recovered 3D faces. In this paper, we exploit the Epipolar Plane Images (EPI) obtained from light field cameras and learn CNN models that recover horizontal and vertical 3D facial curves from the respective horizontal and vertical EPIs. Our 3D face reconstruction network (FaceLFnet) comprises a densely connected architecture to learn accurate 3D facial curves from low resolution EPIs. To train the proposed FaceLFnets from scratch, we synthesize photo-realistic light field images from 3D facial scans. The curve by curve 3D face estimation approach allows the networks to learn from only 14K images of 80 identities, which still comprises over 11 Million EPIs/curves. The estimated facial curves are merged into a single pointcloud to which a surface is fitted to get the final 3D face. Our method is model-free, requires only a few training samples to learn FaceLFnet and can reconstruct 3D faces with high accuracy from single light field images under varying poses, expressions and lighting conditions. Comparison on the BU-3DFE and BU-4DFE datasets show that our method reduces reconstruction errors by over 20% compared to recent state of the art.

U2 - 10.1007/978-3-030-01249-6_31

DO - 10.1007/978-3-030-01249-6_31

M3 - Conference paper

SN - 9783030012489

VL - 11214

SP - 508

EP - 526

BT - European Conference on Computer Vision

A2 - Ferrari, V

A2 - Hebert, M

A2 - Sminchisescu, C

A2 - Weiss, Y

PB - Springer

ER -

Feng M, Gilani SZ, Wang Y, Mian A. 3D Face Reconstruction from Light Field Images: A Model-free Approach. In Ferrari V, Hebert M, Sminchisescu C, Weiss Y, editors, European Conference on Computer Vision. Vol. 11214. Springer. 2018. p. 508-526 https://doi.org/10.1007/978-3-030-01249-6_31