Researchers from Tsinghua University have unveiled a groundbreaking intelligent photonic sensing-computing chip capable of processing, transmitting, and reconstructing images of a scene in nanoseconds. Thanks to its remarkably swift picture processing capabilities, this development promises to significantly enhance edge intelligence for machine vision applications such as robotic vision, industrial inspection, and autonomous driving. This study was published in the journal Optica.
Researchers have developed a new intelligent photonic sensing-computing chip that can process, transmit, and reconstruct images of a scene within nanoseconds. Image Credit: Wei Wu, Tsinghua University
Edge computing, known for conducting intensive computing tasks such as image processing and analysis on local devices, is transitioning into edge intelligence. This evolution incorporates AI to enable local analysis and decision-making, enhancing the responsiveness and efficiency of systems by reducing the need to send data back and forth to central servers.
Capturing, processing, and analyzing images for edge-based tasks such as autonomous driving is currently limited to millisecond-level speeds due to the necessity of optical-to-electronic conversions. Our new chip can perform all these processes in just nanoseconds by keeping them all in the optical domain. This could be used to significantly enhance, or even replace, the traditional architecture of sensor acquisition followed by AI post-processing.
Lu Fang, Research Team Leader, Tsinghua University
The researchers describe their new innovation as an Optical Parallel Computational Array (OPCA) chip. They demonstrate that the OPCA can process up to 100 billion pixels with a reaction time of only six nanoseconds, which is roughly six orders of magnitude quicker than existing techniques. An optical neural network that combines visual perception, processing, and reconstruction was also developed using the technology.
The chip and optical neural network could boost the efficiency of processing complex scenes in industrial inspection and help advance intelligent robot technology to a higher level of cognitive intelligence. We think it could also revolutionize edge intelligence.
Wei Wu, Study Co-First Author, Tsinghua University
Eliminating Optical to Electrical Conversions
Traditionally, machine vision has involved using sensors to translate optical information into digital electrical signals. Machine vision uses cameras, image sensors, illumination, and computer algorithms to capture, process, and analyze images for specialized tasks.
After that, these signals are sent over optical fibers to be used for downstream activities and long-distance data transmission. Unfortunately, regular conversions between optical and electrical data and the slow progress of electronic processors have become a significant barrier to further increasing machine vision's processing speed and capacity.
The world is entering an AI era, but AI is very time- and energy-exhaustive, meanwhile, the growth of edge devices, such as smartphones, intelligent cars, and laptops has resulted in explosive growth of image data to be processed, transmitted, and displayed. We are working to advance machine vision by integrating sensing and computing in the optical domain, which is particularly important for edge computing and for enabling more sustainable AI applications.
Lu Fang, Research Team Leader, Tsinghua University
The difficulty lies in converting the free-space spatial light utilized for imaging into an on-chip guided light wave such that both image acquisition and analysis may be done on the same chip in the optical domain.
To do this, the researchers created a chip that comprises a sensing-computing array of specifically built ring resonators. These resonators transform a free-space optical intensity image, which is a 2D depiction of the light intensity of a scene, into a coherent light signal that can be steered on the platform. By concentrating the scene onto the OPCA chip, a micro-lens array improves the procedure.
Creating an All-Optical Input-Output Connection
The chip's architecture enabled researchers to develop an end-to-end multi-wavelength optical neural network. This network couples the on-chip modulated light into a large-bandwidth optical waveguide, where the modulated light is spectrally combined. The multispectral optical outputs produced can be utilized for classification tasks or to construct an all-optical reconstruction of the image.
Fang said, “Because each sensing-computing element of this chip is reconfigurable, they can each operate as a programmable neuron that generates light modulation output based on the input and weight, the neural network connects all the sensing-computing neurons with a single waveguide, facilitating an all-optical full connection between the input information and the output.”
To demonstrate the capabilities of the OPCA chip, the researchers used it to classify a handwritten image and perform image convolution, a process that applies a filter to an image to extract features. Their findings revealed that the chip architecture can effectively complete information compression and scene reconstruction, highlighting its potential for widespread applications.
The researchers are now working to enhance the sensing-computing OPCA chip to further boost its computational performance and align it more closely with real-world scenarios, optimizing it for edge computing applications. They note that for practical use, the optical neural network's processing capacity needs to be increased to handle increasingly complex and realistic intelligent tasks effectively. Additionally, the form factor of the OPCA chip and its overall size need to be minimized for broader application.
Fang said, “We hope that machine vision will be gradually improved to be faster and more energy-efficient by using light to perform both sensing and computing, even though today’s approach will not likely be completely replaced, we expect the sensing-computing method to find its niche in edge computing where it can drive a wide range of promising applications.”
Journal Reference:
Wu, W., et al. (2024) A parallel photonic chip for nano-second end-to-end image processing, transmission, and reconstruction. Optica. doi.org/10.1364/optica.516241