Computer generated image and scenes are common place in modern films and TV shows. These images are computed by rendering systems that simulate the flow of light in a three-dimensional scene and convert the information into a two-dimensional image.
Computing the thousands of light rays per frame to achieve accurate color, shadows, reflectivity and other light-based characteristics is very labor-intensive, time-consuming and expensive undertaking. An alternative to this process is rendering the images using only a few light rays. While that saves time and labor, it can result in inaccuracies that show up as “noise” in the final image.
Researchers at UC Santa Barbra are working on a solution to this problem. Steve Bako, an electrical and computer engineering Ph.D. student, and Pradeep Sen, Bako’s advisor, have worked with a team at Disney Research and Pixar Animation Studios to develop a technology based on artificial intelligence (AI) and deep learning. This technology eliminates the noise and enables production-quality rendering at higher speeds.
Bako worked with Pixar for a year. His team tested the software by using millions of examples from the Pixar film “Finding Dory” to train a deep-learning model called a convolutional neural network. The system learned to transform noisy images into noise-free versions that resemble the ones computed with more light rays through this process. When the system was properly trained, it successfully removed the noise on test images from different films, like Pixar’s latest film “Cars 3” and an upcoming film “Coco” even though they had completely different styles and color palettes.
"Noise is a really big problem for production rendering," said Tony DeRose, head of research at Pixar. "This new technology allows us to automatically remove the noise while preserving the detail in our scenes."
This research is a big step forward over previous state-of-the-art denoising methods. Those methods often left artifacts or residual noise that requires artists to render more light rays or tweak the denoising filter to improve the quality of an image. Disney and Pixar are planning to incorporate the technology in their production pipelines to accelerate the movie-making process.
"Other approaches for removing image noise have grown increasingly complex, with diminishing returns," said Markus Gross, vice president for research at Disney Research. "By leveraging deep learning, this work presents an important step forward for removing undesirable artifacts from animated films."
To further explore this, the team will make their code and the deep learning model’s trained weights are available to the entire research community.