Computer scientists at Tel-Aviv University collaborated with researchers at Facebook to develop a new computational technique that makes it possible for users to animate still images in a highly realistic way. People can bring out a range of human emotions in any still photo, automatically.
This method enables subjects in a still photo to come to life and express various emotions. For example, when a user “likes” a photo on Facebook, or tweets a thumbs-up, the facial expression of their profile photo could automatically react with a smile or express and emotion of approval or happiness. The researchers demonstrate their technique on varying facial images, including selfies, old portraits and facial avatars. The results were able to show portraits brought to life, depicting a person in the photo as if they are actually smiling or frowning.
"Facial expressions in humans convey not just major emotions, but through subtle variations, a rather nuanced view into the emotional state of a person," said Hadar Averbuch-Elor, computer science Ph.D. at Tel-Aviv University and a lead author of the study. "This expressiveness is what we attempt to capture and animate in our work."
Coauthors of this research were Daniel Cohen-Or, professor of computer science at Tel-Aviv University, and Facebook research scientists, Michael F. Cohen and Johannes Kopf.
Given a single image, the method automatically generates photo-realistic video clips that express various emotions. Previous facial animation techniques typically assume the availability of a video of the target face, which exhibits variation in pose expression. In this work, the researchers use as input only single image of a target face to animate it.
"This makes our method more widely applicable, to the nearly endless supply of portraits or selfies on the Internet," explained Cohen. "We animate the single target face image from a driving video, allowing the target image--the still photo--to come alive and mimic the expressiveness of the subject in the driving video."
The target image in the still photo is animated by a series of geometric warps that intimidate the facial transformations in the driving video. Other work has been restricted to animating only the face region, but this new method can animate the full head and upper body. The researchers manipulate the face by lightweight 2D warps and are able to create a moderate head movement that is depicted realistically without having to convert or project the image to 3D. Technically the main challenge was to preserve the identity of the person’s face in the still photo while manipulating it with warps and features taken and transferred from frames of a driving video.
The new method includes a continuous transfer of fine-scale details, like wrinkles and creases, and avoiding “outlier” wrinkles caused by cast shadows or misalignment between the warped video frames. When needed, the computational technique can also automatically fill in “hidden” facial details.
In a user study, participants were presented with 24 randomly selected videos, eight of which were real. They were asked to rate them as real or not real based on how the animation appeared meaning 50 percent would be a perfect score. The researchers’ animated videos were identified as real 46 percent of the time. The ‘happy’ animations were perceived as the most real and the ‘surprise’ ones were perceived as less real.
In the future, the team intends to build on this work by combining their technique with 3D methods or coupling it with artificial intelligence to create an interactive avatar starting from a single photo.
The team will present its innovative work at SIGGRAPH Asia 2017 in Bangkok, November 27-30.