AI Applications of Inference Cameras

insights from industryRob WebbProduct Sales SpecialistIDS Imaging Development Systems GmbH

In this interview, AZoOptics interviews Rob Webb, Product Sales Specialist for IDS Imaging Development Systems GmbH, about inference cameras, and their role in AI applications.

To begin, can you give our readers an introduction to inference cameras? 

I should probably start with a little background. You will have no doubt heard about artificial intelligence as it has become so prominent in everyday life, processes like bank loan approval systems, email spam filtering and social media content recommendations are common examples of where AI has been employed. Without going too deep into the specifics there have been a number of key advances in the last decade that has really pushed forward the use of AI in image processing, primarily greater access to better processing hardware, more user friendly software development platforms, major investment by large tech companies and also access to significantly more image data through the internet supporting research and development.The use of AI-based processing can be industrial or non-industrial and it generally lends itself to applications where a qualitative decision must be made and the target object or scene is variable either because it is organic by nature or because the conditions are unpredictable.

Another buzz term at the moment is edge processing. When we talk about inference cameras we are referring to cameras that use AI processing onboard, or on the edge, to make a classification of a scene or object or to detect or segment something in an image. Rather than having a passive camera connected to a central image processing unit within a facility or even to a cloud-based solution, inference cameras provide results directly at the point where the work is being done. This has the benefit of minimizing latency, simplifying the system architecture, and using lower-power hardware.

Can you tell us about the IDS NXT concept, which seems tailor-made for bringing AI and image processing together?

IDS NXT ocean is the umbrella product for all of the components we offer to provide an end-to-end AI image processing solution. Broadly we split this into the IDS NXT industrial cameras, the rio and rome, and IDS NXT lighthouse which is our easy-to-use cloud-based service for labeling image data and generating convolutional neural networks that provide the AI processing on the camera.

The IDS NXT rio and NXT rome cameras are available with a number of different sensors up to 6 megapixels and the NXT rome also offers IP65/67 protection which is ideal for challenging working conditions.

What are some of the main benefits of IDS NXT ocean?

We aimed to address both ends of the user base with IDS NXT ocean. The primary focus was to make AI-based image processing accessible to anybody. With IDS NXT lighthouse you do not have to understand anything about AI and deep learning, you only need to know about your application. We often talk about the steps between the user or developer and the end goal or system deployment and how with many AI-based solutions you have to manage the data collection, the data set labeling, the AI knowledge, the programming of the neural network, the training, and validation and finally the hardware platform for the AI processing to run on.

Even for experienced users, this can be time-consuming and error-prone so our goal is to flatten these steps and take care of everything between the user knowledge and physical deployment.

But even advanced users of deep learning, they can still leverage the hardware platform as we also offer a utility to convert existing convolutional neural networks to a format that can run our cameras. And one final way that users can benefit from the hardware platform is through our Vision App Creator SDK where developers can write their own programs to run on the camera and this functionality is not limited only to AI-based processing but extends also to rules-based processing like barcode reading, optical character recognition, pattern matching and really anything that can realistically run on the IDS NXT camera platform.

What would you say the IDS NXT ocean offers the food processing industry that they are currently not being offered by other companies?

Food being an organic product by nature can be difficult to characterize by a fixed set of rules. So on one side the AI-based image processing can handle product variation and it can be used to identify products, or defects in products, for sorting applications and quality control. Where the IDS NXT ocean really benefits the user is the ease of use. Most food processing facilities will likely have some automation experts who are great at machine control and programming PLCs but image processing is a specialist field. IDS NXT ocean allows automation engineers to quickly and easily set up a solution without having to learn about image processing or bring in a third party to handle this element. They in essence become self-sufficient in the complete processing chain.

Costs are of course an important aspect of any process and the cost of implementation and use for an IDS NXT solution being relatively low means the return on investment can be very fast. If a human operative working on a quality assurance task can be re-assigned to another area where automation is not so applicable the ROI can be measured in months.

How important is it that industries such as agritech and food processing incorporate technology such as AI and edge processing?

We hear a lot of reports about access to labor becoming extremely challenging following the UK’s withdrawal from the EU and during the current Covid pandemic and even beyond these significant challenges the cost of labor is increasing so with low margin industries like agriculture and food processing automation can address these challenges. And whilst artificial intelligence is not applicable to every application the variable nature of products in these industries and challenging working environments clearly benefit from AI-based image processing and edge devices. 

During the development of your IDS NXT cameras, what were some of the biggest challenges you came up against, and how were these overcome? 

The main challenge is being able to handle AI-based image processing in an edge device. AI processing can be pretty intensive since neural networks can sometimes have millions of parameters and this means processing and memory resources must be carefully managed. Also, a typical embedded CPU doesn’t really have the necessary oomph to handle AI processing at a usable rate. To combat this IDS has developed an FPGA AI accelerator core and worked hard to optimize the neural networks created with IDS NXT lighthouse to ensure that AI processing runs efficiently on the IDS NXT camera platform.

What are some of the advantages of IDS NXT taking an app-based approach? 

The original concept of these intelligent cameras was based on how we use smartphones. When we buy a new smartphone it is has a number of default apps provided by the manufacturer and then we customize it to our needs by downloading other apps. With the IDS NXT cameras, we provide a number of default apps, or Vision Apps, which cover AI-based image processing and interfacing to wider systems via the means of OPC UA, REST api, serial interface, and general-purpose digital outputs. But with the Vision App Creator, the scope for what is possible is wide open so if a customer develops their own Vision App it can turn the IDS NXT camera into a very specific solution for a vertical market. We have a customer in Germany for example that has written their own pose estimation algorithm that runs on the cameras to provide co-ordinates for picking applications. On top of this, they have also written an interface which means the camera will communicate directly with a robot from Universal Robots. 

Where can readers find more information?

The usual resources such as datasheets, user manuals, and software with source code examples can be found on our website. We have also recently launched our IDS Vision Channel which regularly hosts free live sessions and also offers a number of video presentations providing an introduction to the IDS NXT product range as well as related topics such as the use of the OPC UA interface for automation purposes. It also provides information about other IDS products such as our USB3 and GigE industrial camera range and our Ensenso 3D cameras. All sessions are also hosted in the media library should you wish to watch them again or at a later date.

About Rob Webb

Rob Webb joined IDS in 2018 but has been working with IDS products within the world of machine vision and imaging for over 20 years. With experience in both technical and sales areas, Rob’s focus is on understanding the latest technologies and coherently communicating the benefits to customers as well as within IDS. All that experience is not just with cameras but in all areas of vision systems including lighting, optics, and image processing so he is well placed to advise on customer applications and is very motivated by finding solutions to challenges.