OpenVINO Model Server is distributed as a Docker image with minimal dependencies. Indeed, it introduces a significantly smaller overhead than REST. For use cases where low latency and high throughput are needed, you’ll probably want to interact with the model server via the gRPC API.
Both APIs are compatible with TensorFlow Serving and expose endpoints for prediction, checking model metadata, and monitoring model status. The server offers two sets of APIs to interface with it: REST and gRPC. It also offers centralised model management to serve multiple different models or different versions of the same model and model pipelines. The OpenVINO Model Server provides inference as a service via HTTP/REST and gRPC endpoints for serving models in OpenVINO IR or ONNX format.
#Black appstore icon software#
These components all use the Ubuntu base image for a consistent software ecosystem and containerised environment.Ī few reads if you’re not familiar with this type of microservices architecture: The Model Server component hosts two different demonstration neural networks to compare their results (V1 and V2). Our architecture consists of three microservices: a backend, a frontend, and the OpenVINO Model Server (OVMS) to serve the neural network predictions. More than 280 pre-trained models are available to download, from speech recognition to natural language processing and computer vision.įor this blog series, we will use the pre-trained colourisation models from Open Model Zoo and serve them with Model Server.Īrchitecture diagram of the colouriser demo app running on MicroK8s
#Black appstore icon how to#
Additionally, Python and C++ sample codes demonstrate how to interact with the model. Open Model Zoo provides pre-trained models that work for real-world use cases to get you started quickly. As a result, Intel devices’ throughput increases on CPUs, integrated GPUs, VPUs, and other accelerators. The Post-training Optimisation Tool (POT) and Neural Network Compression Framework (NNCF) provide quantisation, binarisation, filter pruning, and sparsity algorithms. You can unlock additional performance by using low-precision tools from OpenVINO. In fact, it cuts the model’s memory usage in half by converting it from FP32 to FP16 precision. The first step is to convert a deep learning model (trained with TensorFlow, PyTorch,…) to an Intermediate Representation (IR) using the Model Optimizer. OpenVINO includes open-source developer tools to improve model inference performance. It also provides a Model Server for serving models at scale and managing deployments. OpenVINO provides a lightweight Inference Engine with a binary size of just over 40MB for CPU-based inference. When you’re ready to deploy deep learning inference in production, binary size and memory footprint are key considerations – especially when deploying at the edge. Additionally, it comes with hardware optimisation on clouds and on-premises, including Intel hardware. We even used a prebuilt & preconfigured container image for the NGINX web server from the LTS images portfolio maintained by Canonical for up to 10 years.īeyond providing a secure, stable, and consistent experience across container images, Ubuntu is a safe choice from bare metal servers to containers. In the next and final blog (coming soon, keep posted…), you’ll see that using Ubuntu Docker images greatly simplifies components containerisation. More than 30.000 packages are available in one “install” command, with the option to subscribe to enterprise support from Canonical.
One of the main reasons for adopting Ubuntu-based container images is the software ecosystem. From public cloud hosts to IoT devices, the Ubuntu experience is consistent and loved by developers. Why Ubuntu Docker images?Īs the most popular container image in its category, the Ubuntu base image provides a seamless, easy-to-set-up experience. As AI/ML development often requires complex dependencies, it’s the perfect proof point for secure and stable container images. Removing toil and friction from your app development, containerisation, and deployment processes avoids encouraging developers to use untrusted sources or bad practices in the name of getting things done. More specifically, the developer experience. In that case, this blog is an excellent read for you too.ĭocker image security isn’t only about provenance and supply chains it’s also about the user experience. OpenVINO on Ubuntu containers: making developers’ lives easierĪlso, suppose you’re curious about AI/ML and what you can do with OpenVINO on Ubuntu containers.
0 kommentar(er)
