Lensless Optoelectronic Neural Network Developed for Computer Vision Operations

In the last few years, thanks to the progress in the vast processing capacity and parallelism of contemporary graphics processing units (GPUs), deep learning reliant on convolutional neural networks (CNN) has advanced quickly, resulting in operative solutions for a range of problems in artificial intelligence (AI) applications.

However, the enormous volumes of data required in vision processing restrict the application of CNNs to those portable, energy-efficient, computation-efficient hardware to evaluate data on site.

Numerous studies have been carried out in the field of optical computing to surpass the difficulties in electrical neural networks. Optical computing has numerous attractive benefits, like optical parallelism, which can significantly enhance computing speed, and optical passivity can reduce latency and decrease energy costs.

Optical neural networks (ONNs) offer a way to boost computing speed and surpass the bandwidth bottlenecks of electrical units. However, ONNs necessitate a coherent laser as the light source for computation and can barely be integrated with a mature machine vision platform in natural light areas.

Therefore, optoelectronic hybrid neural networks, wherein the back end is electrical and the front end is optical, have been suggested. These lens-based platforms intensify the difficulty of application in edge devices, like autonomous vehicles.

A group of scientists, directed by Professor Hongwei Chen from Beijing National Research Center for Information Science and Technology (BNRist), Department of Electronic Engineering, Tsinghua University, China, have created a lensless optoelectronic neural network (LOEN) architecture for computer vision operations that uses a passive mask introduced into the imaging light path to carry out convolution procedures in the optical field and addresses the difficulty of processing incoherent light signals and broadband light signals in natural scenes.

The researchers describe their findings in a paper published in the journal Light Science & Application.

Moreover, the optical link, back-end network, and image signal processing are smoothly integrated to accomplish joint optimization for particular operations to decrease calculation effort and power consumption across the whole pipeline.

In contrast to the hardware architecture in traditional machine vision, an optical mask closing to the imaging sensor is suggested to substitute the lenses in this study.

Based on the geometrical optics theory that light spreads in a straight line, the scenes can be considered as sets of point light sources, and the optical signal is spatially controlled by the mask to achieve the convolution operation of transfer and superposition on the image sensor. It has been confirmed that optical masks can substitute the convolutional layers of neural networks for feature extraction in the optical field.

For object classification operations like handwritten digit recognition, a lightweight network for instant recognition is developed to confirm the performance of the optical convolution in the architecture.

While employing a single convolution kernel, the accuracy of recognition can go up to 93.47%. When the multi-channel convolution operation is executed by arranging numerous kernels in parallel on the mask, the classification precision can be enhanced to 97.21%. In contrast to conventional machine vision links, it can save around 50% of power consumption.

Furthermore, by increasing the dimension of the optical mask, the image is convolved in the optical field, and the sensor records an aliased image that is distorted to the human eye, which can innately encode private data without computational consumption. The performance of optical encryption was confirmed on the face recognition function.

In contrast to the random MLS pattern, the recognition precision of the mask equally optimized by an end-to-end network was enhanced by over 6%. Together with privacy protection encryption, it essentially accomplished the same recognition accuracy performance as no-encryption approaches.

This study suggests a highly simplified system for machine vision operations, which not only achieves the optoelectronic neural network calculation in natural scenes but also makes way for the whole optoelectronic link to fulfill joint optimization to attain the best results for particular vision tasks.

Integrated with the nonlinear materials, the all-natural-light neural network will be realized. The unique architecture will have several prospective applications in numerous real scenarios, such as smart homes, autonomous driving and smart security.

Journal Reference:

Shi, W., et al. (2022) LOEN: Lensless opto-electronic neural network empowered machine vision. Light Science & Application. doi.org/10.1038/s41377-022-00809-5.