New Non-Line-of-Sight Imaging Method to Reconstruct Images in Great Detail

Computer vision scientists have shown that specialized light sources and sensors can be used to view around corners or through gauzy filters, thereby allowing them to rebuild the shapes of objects that are not seen.

According to researchers at Carnegie Mellon University, the University of Toronto, and University College London, this method allows them to rebuild images with in-depth detail, such as the relief of George Washington’s profile on a U.S. quarter.

Ioannis Gkioulekas, an assistant professor in Carnegie Mellon’s Robotics Institute, stated that for the first time, scientists could calculate micrometer- and millimeter-scale shapes of curved objects, offering a significant new component to a wider range of non-line-of-sight (NLOS) imaging methods currently being devised by computer vision researchers.

It is exciting to see the quality of reconstructions of hidden objects get closer to the scans we’re used to seeing for objects that are in the line of sight. Thus far, we can achieve this level of detail for only relatively small areas, but this capability will complement other NLOS techniques.

Srinivasa Narasimhan, Professor, Robotics Institute, Carnegie Mellon University

This study was supported by the Defense Advanced Research Project Agency’s REVEAL program, which has been creating NLOS capabilities. The study will be presented at the 2019 Conference on Computer Vision and Pattern Recognition (CVPR2019) in Long Beach, California, on June 20^th, 2019, where it has obtained a Best Paper award.

“This paper makes significant advances in non-line-of-sight reconstruction—in essence, the ability to see around corners,” states the award citation. “It is both a beautiful paper theoretically as well as inspiring. It continues to push the boundaries of what is possible in computer vision.”

Much of what people view—and what is detected by cameras—is the result of reflection of light off an object and direct bouncing of the light to the eye or the lens. However, light is also reflected off the objects in other directions, bouncing off objects and walls.

Although a weak portion of this scattered light might eventually reach the lens or the eye, it is washed out by more powerful, direct light sources. NLOS methods make attempts to derive information from scattered light, naturally occurring or otherwise, and generate images of objects, scenes, or parts of objects not otherwise visible.

Other NLOS researchers have already demonstrated NLOS imaging systems that can understand room-size scenes, or even extract information using only naturally occurring light. We’re doing something that’s complementary to those approaches—enabling NLOS systems to capture fine detail over a small area.

Ioannis Gkioulekas, Assistant Professor, Robotics Institute, Carnegie Mellon University

In this scenario, the scientists employed an ultrafast laser to make light to bounce off a wall and illuminate a concealed object. By being aware when the light pulses were fired by the laser, the researchers could compute the time taken by the light to be reflected off the object, bounce off the wall on its trip back, and reach a sensor.

“This time-of-flight technique is similar to that of the lidars often used by self-driving cars to build a 3D map of the car’s surroundings,” stated Shumian Xin, a PhD student in robotics.

Earlier efforts to use these time-of-flight computations to rebuild the object’s image have relied on the brightness of the reflections from it. However, in this research, Gkioulekas stated that the scientists devised a new technique solely based on the geometry of the object, which in turn allowed them to develop an algorithm to measure its curvature.

The scientists employed an imaging system that is essentially a LiDAR with the potential to sense single light particles to test the method on objects like a glass bowl, a plastic jug, a ball bearing, and a plastic bowl. They also integrated this method with an imaging technique known as optical coherence tomography to rebuild the images of U.S. quarters.

Apart from seeing around corners, the method proved efficient in viewing through diffusing filters, like thick paper.

So far, the method has been demonstrated only at short distances—up to a maximum of 1 m. However, the researchers propose that the method, which is based on geometric measurements of objects, might be integrated with other, complementary strategies to enhance NLOS imaging. It could also be used in other applications like acoustic and ultrasound imaging and seismic imaging.

Apart from Narasimhan, Gkioulekas, and Xin, the research group included Aswin Sankaranarayanan, assistant professor in CMU’s Department of Electrical and Computer Engineering; Sotiris Nousias, a PhD student in medical physics and bioengineering at University College London; and Kiriakos N. Kutulakos, a professor of computer science at the University of Toronto.

The scientists are part of a larger collaborative group, including researchers from Stanford University, the University of Wisconsin Madison, the University of Zaragosa, Politecnico di Milano, and the French-German Research Institute of Saint-Louis, which is creating a series of complementary methods for NLOS imaging.

This study was supported by DARPA, as well as the National Science Foundation, the Office of Naval Research, and the Natural Sciences and Engineering Research Council of Canada.

Images from the research of objects seen around corners. (Video credit: Carnegie Mellon University)