Intel Distro of OpenVINO Helps Devs with Inferencing Beyond Computer Vision

Developers continue to adapt to the advent in nearly everything of Artificial Intelligence (AI), or more precisely, machine learning (ML) and deep learning (DL). And the tools are evolving with them and this growing demand.

One example is the OpenVINO toolkit. OpenVINO stands for Open Visual Inference and Neural Network Optimization. It's a toolkit developed and open sourced by Intel to facilitate faster inferencing in deep learning models on Intel hardware. It's based on convolutional neural networks (CNN), a deep learning algorithm designed for working with two-dimensional image data, although it can also be used with one-dimensional and three-dimensional data. The open-source version is licensed under and Apache License v2.

I talked with Bill Pearson, VP of Intel's IoT group, about his company's distro of OpenVINO.

"I suppose the key thing this tool does is to simplify the process of helping developers with their high performance inferencing needs," Pearson told me. "We're seeing a number of different use cases and requirements for doing inference--which silicon do I need, and what programming models on top of it. This can be quite confusing, so with OpenVINO, we created an API-based programming model that allows you to write once and deploy anywhere. What that means practically is, we take all the complexity of writing for FPGA, GPU, CPU, or whatever. We hide that complexity, so the developers have a consistent interface that lets them literally write once and then deploy to any of those different architecture types, depending on their needs and their requirements."

The first set of use cases Intel focused on were all computer vision related, Pearson said, because that's where most developers needed with inferencing. But with the most recent release of the toolkit, Intel is looking at inference beyond computer vision.

OpenVINO 2021.1, announced in October, is designed to enable end-to-end capabilities that leverage the toolkit for workloads beyond computer vision. These capabilities include audio, speech, language, and recommendation with new pretrained models; support for public models, code samples, and demos; and support for non-vision workloads in the DL Streamer component.

This release also introduces official support for models trained in the TensorFlow 2.2.x framework; support for 11th generation Intel Core processors (formerly code named Tiger Lake); and new inference performance enhancements with Intel Iris Xe graphics, Intel Deep Learning Boost (Intel DL Boost) instructions, and Intel Gaussian & Neural Accelerator 2.0 for low-power speech processing acceleration.

It comes with the OpenVINO model server, which is an add-on to the Intel distro, and a scalable microservice. "This add-on provides a gRPC or HTTP/REST endpoint for inference, making it easier to deploy models in cloud or edge server environments," the company said in a statement. Also, it's now implemented in C++, which enables reduced container footprint (for example, less than 500 MB), and delivers higher throughput and lower latency.

There's also a beta release due this quarter that integrates the Deep Learning Workbench with the Intel DevCloud for the Edge. The result: developers can now graphically analyze models using the Deep Learning Workbench on Intel DevCloud for the Edge, instead of being stuck on a local machine only, to compare, visualize, and fine-tune a solution against multiple remote hardware configurations. And there's another add-on: the OpenVINO model server, which provides a gRPC or HTTP/REST endpoint for inference, making it easier to deploy models in cloud or edge server environments. It's now implemented in C++, to enable reduced container footprint (for example, less than 500 MB), and deliver higher throughput and lower latency.

"We have to deliver features, of course, but we also need to deliver on performance," Pearson said. "With this release, that's what we've done. Applying CNN is helping us to take advantage of modern AI, to be able to get models that do a much better job at delivering what we'd expect from inference, and to be able to detect that defect or notice that object."

Pearson offered a customer example: "There are some great you know customer examples where we've seen this be the true. The earliest things we've seen are in finding manufacturing defects. There's an aluminum engine provider, who was doing inspections the old way--by hand. They had to wait for these objects to cool so that a person could turn it and look at it. You can imagine that doing that with a computer and a camera is going to be much more accurate and much quicker. And we've just seen this expand way beyond that to use cases in health care for medical screenings, and things like sewer pipe inspections--which basic, but essential--and in businesses like retail, where we're going through these frictionless checkout systems and trying to be able to see what objects are going through the scanner and track tell what what's being processed"

Intel OpenVINO is available today on its DevCloud.

Posted by John K. Waters on December 15, 2020