Electronics and Semiconductors

Using AI to create 3D spaces from 2D images

14 June 2024

The potential to improve navigation for autonomous vehicles has emerged with research from NC State University (NCSU) that uses artificial intelligence (AI) programs to map 3D spaces using 2D images captured by multiple cameras.

Called Multi-View Attentive Contextualization (MvACon), the method is a plug-and-play supplement that can be used in conjunction with existing vision transformer AIs to improve how automated vehicles map 3D spaces.

“Most autonomous vehicles use powerful AI programs called vision transformers to take 2D images from multiple cameras and create a representation of the 3D space around the vehicle,” said Tianfu Wu, an associate professor of electrical and computer engineering at NCSU. “However, while each of these AI programs takes a different approach, there is still substantial room for improvement.

Wu added the vision transformers don’t get additional data from the cameras but just make better use of the data.

How it works

The technology modifies an approach called Patch-to-Cluster attention (PaCa) that allows transformer AIs to work more efficiently and effectively in identifying objects in an image.

Researchers use the MvACon in conjunction with three leading vision transformers — BEVFormer, the BEVFormer DFA3D variant and PETR. When testing the technology with these transformers, MvACon improved the performance of all three when collecting 2D images from six different cameras.

“Performance was particularly improved when it came to locating objects, as well as the speed and orientation of those objects,” Wu said. “And the increase in computational demand of adding MvACon to the vision transformers was almost negligible.”

The next steps are to test the technology against additional benchmark datasets and testing it against actual video input from autonomous vehicles to see if the method could be used for widespread use.

The full research will be presented at the upcoming IEEE/CVF Conference on Computer Vision and Patter Recognition taking place next week.

To contact the author of this article, email PBrown@globalspec.com

Powered by CR4, the Engineering Community

Discussion – 0 comments

By posting a comment you confirm that you have read and accept our Posting Rules and Terms of Use.
Engineering Newsletter Signup
Get the GlobalSpec
Stay up to date on:
Features the top stories, latest news, charts, insights and more on the end-to-end electronics value chain.
Weekly Newsletter
Get news, research, and analysis
on the Electronics industry in your
inbox every week - for FREE
Sign up for our FREE eNewsletter
Find Free Electronics Datasheets