From phone camera snapshots to lifesaving medical scans, digital images play an important role in the way humans communicate information. But digital images are subject to a range of imperfections like blurriness, grainy noise, missing pixels and color corruption.
A group led by a University of Maryland computer scientist has designed a new algorithm that incorporates artificial neural networks to simultaneously apply a wide range of fixes to corrupted digital images. This algorithm can be “trained” to recognize what an ideal, uncorrupted should look like, so it can address multiple flaws in a single image.
The research team included members from the University of Bern in Switzerland. They tested their algorithm by taking high-quality, uncorrupted images, and inducing severe issues into the images. Then, they used the algorithms to repair the damage. In many cases, the algorithm outperformed competitors’ techniques, very nearly returning the images to their original state.
The researchers presented their findings on December 5, 2017 at the 31st Conference on Neural Information Processing Systems in Long Beach, California.
"Traditionally, there have been tools that address each problem with an image separately. Each of these uses intuitive assumptions of what a good image looks like, but these assumptions have to be hand-coded into the algorithms," said Matthias Zwicker, the Reginald Allan Hahne Endowed E-Novate Professor in Computer Science at UMD and senior author of the research presentation. "Recently, artificial neural networks have been applied to address problems one by one. But our algorithm goes a step further--it can address a wide variety of problems at the same time."
Artificial neural networks are a type of artificial intelligence algorithm inspired by the structure of the human brain. They can assemble patterns of behavior based on input data, in a process that resembles the way a human brain learns new information. For example, human brains can learn a new language through repeated exposure to words and sentences in specific contexts.
Zwicker and his colleagues can “train” their algorithm by exposing it to a large database of high-quality, uncorrupted images widely used for research with artificial neural networks. Because the algorithm can take in a large amount of data and extrapolate the complex parameters that define images and it is able to predict what an ideal, uncorrupted image should look like. Then it can recognize and fix deviations from these ideal parameters in a new image.
"This is the key element. The algorithm needs to be able to recognize a good image without degradations. But for an image that is already degraded, we can't know what this would look like," said Zwicker, who also has an appointment at the University of Maryland Institute for Advanced Computer Studies (UMIACS). "So instead, we first train the algorithm on a database of high-quality images. Then we can give it an image and the algorithm will modify the imperfections."
There are several other research groups working on the same idea and achieving similar results. Many of the research groups have designed algorithms were tasked with only removing noise from an image, but Zwicker’s algorithm would automatically address many of the other imperfections as well. His group proposed a new theoretical explanation for this effect that leads to a simple and effective algorithm.
"When you have a noisy image, it is randomly shifted or jittered away from a high-quality image in all possible dimensions. Other degradations, such as blurring, for example, diverge from the ideal only in a subset of dimensions," Zwicker explained. "Our work revealed how fixing noise will bring all dimensions back in line, allowing us to address several types of other degradations, like blurring, at the same time."
Zwicker also said the new algorithm still has room for improvement. The algorithm currently works well for fixing easily recognizable “low-level” structures in images, like sharp edges. The researchers hope to push the algorithm to recognize and repair “high-level” features, including complex textures like hair and water.
"To recognize high-level features, the algorithm needs context to understand what is in the image. For example, if there is a face in an image, it's likely that the pixels near the top are probably hair," Zwicker said. "It's like assembling a jigsaw puzzle. If you're only looking at one piece, it's hard to place that part of the image in context. But once you find where the piece belongs, it's much easier to recognize what the pixels represent. It's quite clear that this approach can be pushed much further still."
The research will be presented at the 31st Neural Information Processing Systems in Long Beach, California. The paper can be accessed here.