3D Imaging Technique to Aid Robot Navigation and Display Systems

Researchers have created a camera that records 3D data concerning objects in a scene with a single exposure using a tiny microlens array and a new image processing technique. The camera can be used for various tasks, including inspecting industrial parts, recognizing gestures, and gathering information for 3D display systems.

We consider our camera lens less because it replaces the bulk lenses used in conventional cameras with a thin, lightweight microlens array made of flexible polymer. Because each microlens can observe objects from different viewing angles, it can accomplish complex imaging tasks such as acquiring 3D information from objects partially obscured by objects closer to the camera.

Weijian Yang, Research Team Leader, University of California, Davis

Yang and first author Feng Tian, a Ph.D. student in Yang’s lab, reveal the new 3D camera in the Optica Publishing Group’s journal Optics Express. The camera is able to create real-time 3D images as it learns how to digitally recreate a 3D scene from previous data .

Yang stated, “This 3D camera could be used to give robots 3D vision, which could help them navigate 3D space or enable complex tasks such as manipulation of fine objects. It could also be used to acquire rich 3D information that could provide content for 3D displays used in gaming, entertainment or many other applications.”

A Camera That Learns

The development of a portable microscope that can capture 3D images of microscopic structures for use in biomedical applications has led to the creation of the new camera.

“We built the microscope using a microlens array and thought that a similar concept could be applied for imaging macroscopic objects,” Yang further added.

The new camera’s separate lenses enable it to view objects from various angles or perspectives, providing depth information. Although other research teams have created cameras based on single-layer microlens arrays, their practical application has been hampered by long reconstruction times and lengthy calibration procedures.

Yang further added, “Many existing neural networks can perform designated tasks, but the underlying mechanism is difficult to explain and understand. Our neural network is based on a physical model of image reconstruction. This makes the learning process much easier and results in high quality reconstructions.”

Once the learning process is complete, it can reconstruct images containing objects at different distances from the camera at immense speed. The new camera does not need calibration and can be used to map the 3D locations and spatial profiles — or outlines — of objects.

Seeing Through Objects

The researchers initially performed numerical simulations to confirm the camera’s functionality. They then carried out 2D imaging, which allowed them to obtain perceptually acceptable results. The scientists then tested the camera’s capacity to capture 3D images of objects at various depths.

The resulting 3D reconstruction could be adequately refocussed to different distances or depths. Additionally, the camera produced a depth map matching the actual objects' placement.

Yang continued, “In a final demonstration we showed that our camera could image objects behind the opaque obstacles. To the best of our knowledge, this is the first demonstration of imaging objects behind opaque obstacles using a lensless camera.”

The researchers are currently focusing on minimizing artifacts, or errors, perceptible in 3D reconstructions and enhancing the algorithms to increase both quality and speed. Additionally, they hope to reduce the size of the entire device so that it can fit within a cellphone, which would increase portability and open up new application possibilities.

“Our lensless 3D camera uses computational imaging, an emerging approach that jointly optimizes imaging hardware and object reconstruction algorithms to achieve desired imaging tasks and quality. With the recent development of low-cost, advanced micro-optics manufacturing techniques as well as advancements in machine learning and computational resources, computational imaging will enable many new imaging systems with advanced functionality,” concluded Yang.

Journal Reference:

Tian, F., et al. (2022) Learned lensless 3D camera. Optics Express. doi:10.1364/OE.465933