Researchers from the Division of Sustainable Energy and Environmental Engineering at Osaka University created a prototype for a real time augmented reality (AR) software that can virtually remove static structures and moving objects. This technology would be useful in the construction industry to visualize future landscapes and urban renovations.
Proposed "diminished reality" (DR) system. Left: Input image captured by a web camera showing the current situation. Right: DR output image of the future landscape. A moving object (pedestrian) has been virtually removed in real- time along with the immobile building. Source: Osaka University
With AR systems, characters or buildings are added to the user’s screen to create an immersive environment. It is much harder to virtually remove objects and show what is behind them, which is a computationally intensive task. But to accurately plan demolitions or landscapes, city planners need to visualize their plans.
The research team set out to overcome this issue and they created an AR system that can virtually remove objects. Their prototype combines deep learning with a video game graphics engine to create real-time landscapes and virtually remove buildings, trees and moving objects. The system reveals a background that was previously obscured.
The AR system dynamically blends photos of the background behind buildings based on the current viewpoint of the camera. The machine learning algorithm can automatically detect and mask the region being obscured by a moving object and replace it with a current or future scene.
The system was tested on Osaka University’s campus. Testing proves that it could automatically detect and remove cars and pedestrians with ease. There is more development to be done before it becomes commercially available, but the team says that one day it could be used by the construction industry for planning purposes.
A paper on this technology was published in Environmental Modeling and Software.
