Researchers from Carnegie Mellon University have developed a system that can create an accurate 3D reconstruction of a face with just a smartphone. Typically this process is done with pricey equipment, but the new method has achieved sub-millimeter accuracy and outperformed other camera-based processes.
In a 3D face reconstruction process developed at Carnegie Mellon University, smartphone video of a person, left, is analyzed to produce an imperfect model of the face, middle. Deep learning is then combined with conventional computer vision techniques to complete the reconstruction, right. Source: Carnegie Mellon University
The team used an iPhone X smartphone to shoot 15 to 20 seconds of continuous slow-motion footage of the front and sides of a person’s face. The high frame rate of slow-motion video generated a dense point cloud. Visual simultaneous localization and mapping (SLAM) was then used to triangulate points on the surface to calculate shape and determine camera position. This created an initial geometry of the face, but missing data left gaps in the model. As such, deep learning algorithms were used to identify the profile and facial landmarks and computer vision filled in those gaps.
The whole process can take 30 to 40 minutes and is done completely on the smartphone. The team said that this method could be used to capture the geometry of any object.
Digital faces are often used for avatars in gaming, virtual and augmented reality, animation, biometric identification, and medical processes. They can also be helpful in building customized ventilators and masks. People are sensitive to the look of facial features. If a rendering is off, it can make the entire reconstruction look unrealistic, which is why it is so important for the reconstruction to be accurate.
This research was presented at IEEE Winter Conference on Applications of Computer Vision.
