{"id":15766,"date":"2019-10-21T17:00:40","date_gmt":"2019-10-22T00:00:40","guid":{"rendered":"https:\/\/developer.nvidia.com\/blog\/?p=15766"},"modified":"2022-08-21T16:39:38","modified_gmt":"2022-08-21T23:39:38","slug":"nvidia-gpu-operator-simplifying-gpu-management-in-kubernetes","status":"publish","type":"post","link":"https:\/\/developer.nvidia.com\/blog\/nvidia-gpu-operator-simplifying-gpu-management-in-kubernetes\/","title":{"rendered":"NVIDIA GPU Operator: Simplifying GPU Management in Kubernetes"},"content":{"rendered":"

Editor’s note: Interested in GPU Operator? Register for our upcoming webinar on January 20th, “How to Easily use GPUs with Kubernetes”<\/a><\/em>.<\/p>\n

Over the last few years, NVIDIA has leveraged GPU containers in a variety of ways for testing, development and running AI workloads in production at scale. Containers optimized for NVIDIA GPUs and systems such as the DGX and OEM NGC-Ready servers\u00a0 are available as part of NGC<\/a>.<\/p>\n

But provisioning servers with GPUs reliably and scaling AI applications can be tricky. Kubernetes has quickly become the platform of choice for deploying complex applications built on top of numerous microservices due to its rich set of APIs, reliability, scalability and performance features.<\/p>\n

Kubernetes provides access to special hardware resources such as NVIDIA GPUs, NICs, Infiniband adapters and other devices through the device plugin framework<\/a>. However, configuring and managing nodes with these hardware resources requires configuration of multiple software components such as drivers, container runtimes or other libraries which\u00a0 are difficult and prone to errors.<\/p>\n

The Operator Framework<\/a> within Kubernetes takes operational business logic and allows the creation of an automated framework for the deployment of applications within Kubernetes using standard Kubernetes APIs and kubectl.\u00a0 The NVIDIA GPU Operator introduced here is based on the operator framework and automates the management of all NVIDIA software components needed to provision GPUs within Kubernetes. NVIDIA, Red Hat, and others in the community have collaborated on creating the GPU Operator. The GPU Operator is an important component of the NVIDIA EGX<\/a> software-defined platform that is designed to make large-scale hybrid-cloud and edge operations possible and efficient.<\/p>\n

NVIDIA GPU Operator<\/i><\/a><\/h2>\n

To provision GPU worker nodes in a Kubernetes cluster, the following NVIDIA software components are required – the driver, container runtime, device plugin and monitoring. As shown in Figure 1, these components need to be manually provisioned before GPU resources are available to the cluster and also need to be managed during the operation of the cluster. The GPU Operator simplifies both the initial deployment and management of the components by containerizing all the components and using standard Kubernetes APIs for automating and managing these components including versioning and upgrades. The GPU operator is fully open-source and is available on our GitHub repo<\/a>.<\/p>\n

\"\"<\/a>
Figure 1: Manual Install (some components need to be installed on bare-metal) vs. Automation by GPU Operator with fully containerized components<\/figcaption><\/figure>\n

Operator State Machine<\/i><\/a><\/h3>\n

The GPU Operator is based on the Operator Framework<\/a> in Kubernetes. The operator is built as a new Custom Resource Definition (CRD) API with a corresponding controller. The operator runs in its own namespace (called \u201cgpu-operator\u201d) with the underlying NVIDIA components orchestrated in a separate namespace (called \u201cgpu-operator-resources\u201d). As with any standard operator in Kubernetes, the controller watches the namespace for changes and uses a reconcile loop (via the Reconcile() function) to implement a simple state machine for starting each of the NVIDIA components. The state machine includes a validation step at each state and on failure, the reconcile loop exits with an error. This is shown in Figure 2.<\/p>\n

\"\"<\/a>
Figure 2: GPU Operator State Machine<\/figcaption><\/figure>\n

The GPU operator should run on nodes that are equipped with GPUs. To determine which nodes have GPUs, the operator relies on Node Feature Discovery<\/a> (NFD) within Kubernetes. The NFD worker detects various hardware features on the node – for example, PCIe device ids, kernel versions, memory and other attributes. It then advertises these features to Kubernetes using node labels. The GPU operator then uses these node labels (by checking the PCIe device id) to determine if NVIDIA software components should be provisioned on the node. In this initial release, the GPU operator currently deploys the NVIDIA container runtime<\/a>, NVIDIA containerized driver<\/a> and the NVIDIA Kubernetes Device Plugin<\/a>. In the future, the operator will also manage other components such as DCGM-based<\/a> monitoring.<\/p>\n

Let\u2019s briefly look at the different states.<\/p>\n

State Container Toolkit<\/h4>\n

This state deploys a DaemonSet that installs the NVIDIA container runtime on the host system via a container<\/a>. The DaemonSet uses the PCIe device id from the NFD label to only install the runtime on nodes that have GPU resources. The PCIe device id 0x10DE<\/code> is the vendor id for NVIDIA.<\/p>\n

nodeSelector:\n        feature.node.kubernetes.io/pci-10de.present: "true"<\/pre>\n
<\/h4>\n
State Driver<\/h4>\n
This state deploys a DaemonSet with the NVIDIA driver that is containerized. You can read more about driver containers here<\/a>. On startup, the driver container may build the final NVIDIA kernel modules and load them into the Linux kernel on the host in preparation to run CUDA applications and runs in the background. The driver container includes the user-mode components of the driver required for applications. Again, the DaemonSet uses the NFD label to select nodes on which to deploy the driver container.<\/p>\n
State Driver Validation<\/h4>\n
As mentioned above, the operator state machine includes validation steps to ensure that components have been started successfully. The operator schedules a simple CUDA workload (in this case a vectorAdd sample). The container state is \u201cSuccess\u201d if the application ran without any errors.<\/p>\n
State Device Plugin<\/h4>\n
This state deploys a DaemonSet for the NVIDIA Kubernetes device plugin. It registers the list of GPUs on the node with the kubelet so that GPUs can be allocated to CUDA workloads.<\/p>\n
State Device Plugin Validation<\/h4>\n
At this state, the validation container requests for a GPU to be allocated by Kubernetes and runs a simple CUDA workload (as described above) to check if the device plugin registered the list of resources and the workload ran successfully (i.e. the container status was \u201cSuccess\u201d).<\/p>\n
To simplify the deployment of the GPU operator itself, NVIDIA provides a Helm chart. The versions of the software components that are deployed by the operator (e.g. driver, device plugin) can be customized by the user with templates (values.yaml) in the Helm chart. The operator then uses the template values to provision the desired versions of the software on the node. This provides a level of parameterization to the user.<\/p>\n
Running the GPU Operator<\/i><\/a><\/h2>\n
Let\u2019s take a quick look at how to deploy the GPU operator and run a CUDA workload. At this point, we assume that you have a Kubernetes cluster operational (i.e. the master control plane is available and worker nodes have joined the cluster). To keep things simple for this blog post, we will use a single node Kubernetes cluster with an NVIDIA Tesla T4 GPU running Ubuntu 18.04.3 LTS.<\/p>\n
The GPU Operator does not address the setting up of a Kubernetes cluster itself – there are many solutions available<\/a> today for this purpose. NVIDIA is working with different partners on integrating the GPU Operator into their solutions for managing GPUs.<\/p>\n
Let\u2019s verify that our Kubernetes cluster (along with a Helm setup with Tiller) is operational. Note that while the node has a GPU, there are no NVIDIA software components deployed on the node – we will be using the GPU operator to provision the components.<\/p>\n
$ sudo kubectl get pods --all-namespaces                                                                                                          NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE\nkube-system   calico-kube-controllers-6fcc7d5fd6-n2dnt   1/1     Running   0          6m45s\nkube-system   calico-node-77hjv                          1/1     Running   0          6m45s\nkube-system   coredns-5c98db65d4-cg6st                   1/1     Running   0          7m10s\nkube-system   coredns-5c98db65d4-kfl6v                   1/1     Running   0          7m10s\nkube-system   etcd-ip-172-31-5-174                       1/1     Running   0          6m5s\nkube-system   kube-apiserver-ip-172-31-5-174             1/1     Running   0          6m11s\nkube-system   kube-controller-manager-ip-172-31-5-174    1/1     Running   0          6m26s\nkube-system   kube-proxy-mbnsg                           1/1     Running   0          7m10s\nkube-system   kube-scheduler-ip-172-31-5-174             1/1     Running   0          6m18s\nkube-system   tiller-deploy-8557598fbc-hrrhd             1/1     Running   0          21s<\/pre>\n
A single node Kubernetes cluster (the master has been untainted, so it can run workloads)<\/p>\n
$ kubectl get nodes\nNAME              STATUS   ROLES    AGE    VERSION\nip-172-31-5-174   Ready    master   3m2s   v1.15.3<\/pre>\n
We can see that the node has an NVIDIA GPU but no drivers or other software tools installed.<\/p>\n
$ lspci | grep -i nvidia\n00:1e.0 3D controller: NVIDIA Corporation Device 1eb8 (rev a1)\n\n$ nvidia-smi\nnvidia-smi: command not found<\/pre>\n
As a prerequisite, let’s ensure that some kernel modules are setup on the system. The NVIDIA driver has some dependencies on these modules for symbol resolution.<\/p>\n
$ sudo modprobe -a i2c_core ipmi_msghandler<\/pre>\n
Now, let\u2019s go ahead and deploy the GPU operator. We will use a Helm chart for this purpose that is available from NGC. First, add the Helm repo:<\/p>\n
$ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia\n"nvidia" has been added to your repositories\n\n$ helm repo update\nHang tight while we grab the latest from your chart repositories...\n...Skip local chart repository\n...Successfully got an update from the "nvidia" chart repository\n...Successfully got an update from the "stable" chart repository\nUpdate Complete.<\/pre>\n
And then deploy the operator with the chart<\/p>\n
$ helm install --devel nvidia/gpu-operator -n test-operator --wait\n$ kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/master/manifests/cr/sro_cr_sched_none.yaml\nspecialresource.sro.openshift.io/gpu created<\/pre>\n
We can verify that the GPU operator is running in its own namespace and is watching the components in another namespace.<\/p>\n
$ kubectl get pods -n gpu-operator\nNAME                                         READY   STATUS    RESTARTS   AGE\nspecial-resource-operator-7654cd5d88-w5jbf   1/1     Running   0          98s<\/pre>\n
After a few minutes, the GPU operator would have deployed all the NVIDIA software components. The output also shows the validation containers run as part of the GPU operator state machine. The sample CUDA containers (vectorAdd) have completed successfully as part of the state machine.<\/p>\n
$ kubectl get pods -n gpu-operator-resources\nNAME                                       READY   STATUS      RESTARTS   AGE\nnvidia-container-toolkit-daemonset-wwzfn   1/1     Running     0          3m36s\nnvidia-device-plugin-daemonset-pwfq7       1/1     Running     0          101s\nnvidia-device-plugin-validation            0/1     Completed   0          92s\nnvidia-driver-daemonset-skpn7              1/1     Running     0          3m27s\nnvidia-driver-validation                   0/1     Completed   0          3m\n\n$ kubectl -n gpu-operator-resources logs -f nvidia-device-plugin-validation\n[Vector addition of 50000 elements]\nCopy input data from the host memory to the CUDA device\nCUDA kernel launch with 196 blocks of 256 threads\nCopy output data from the CUDA device to the host memory\nTest PASSED\nDone<\/pre>\n
We can also see that the NFD has labeled the node with different attributes. A node label with the PCIe device id 0x10DE has been set for the NVIDIA GPU.<\/p>\n
$ kubectl -n node-feature-discovery logs -f nfd-worker-zsjsp\n2019/10/21 00:46:25 cpu-cpuid.AVX512F = true\n2019/10/21 00:46:25 cpu-hardware_multithreading = true\n2019/10/21 00:46:25 cpu-cpuid.AVX = true\n2019/10/21 00:46:25 cpu-cpuid.AVX512VL = true\n2019/10/21 00:46:25 cpu-cpuid.AVX512CD = true\n2019/10/21 00:46:25 cpu-cpuid.AVX2 = true\n2019/10/21 00:46:25 cpu-cpuid.FMA3 = true\n2019/10/21 00:46:25 cpu-cpuid.ADX = true\n2019/10/21 00:46:25 cpu-cpuid.AVX512DQ = true\n2019/10/21 00:46:25 cpu-cpuid.AESNI = true\n2019/10/21 00:46:25 cpu-cpuid.AVX512BW = true\n2019/10/21 00:46:25 cpu-cpuid.MPX = true\n2019/10/21 00:46:25 kernel-config.NO_HZ = true\n2019/10/21 00:46:25 kernel-config.NO_HZ_IDLE = true\n2019/10/21 00:46:25 kernel-version.full = 4.15.0-1051-aws\n2019/10/21 00:46:25 kernel-version.major = 4\n2019/10/21 00:46:25 kernel-version.minor = 15\n2019/10/21 00:46:25 kernel-version.revision = 0\n2019/10/21 00:46:25 pci-10de.present = true\n2019/10/21 00:46:25 pci-1d0f.present = true\n2019/10/21 00:46:25 storage-nonrotationaldisk = true\n2019/10/21 00:46:25 system-os_release.ID = ubuntu\n2019/10/21 00:46:25 system-os_release.VERSION_ID = 18.04\n2019/10/21 00:46:25 system-os_release.VERSION_ID.major = 18\n2019/10/21 00:46:25 system-os_release.VERSION_ID.minor = 04<\/pre>\n
Let\u2019s launch a TensorFlow notebook. An example manifest is available on our GitHub repo, so let\u2019s use that<\/p>\n
$ kubectl apply -f https://nvidia.github.io/gpu-operator/notebook-example.yml<\/pre>\n
Once the pod is created, we can use the token to view the notebook in a browser window.<\/p>\n
$ kubectl logs -f tf-notebook\n[C 02:52:44.849 NotebookApp]\n    Copy/paste this URL into your browser when you connect for the first time,\n    to login with a token:\n        http://localhost:8888/?token=b7881f90dfb6c8c5892cff7e8232684f201c846c48da81c9<\/pre>\n
We can either use port forwarding or use the node port 30001 to reach the container. Use the URL from the logs above to open the Jupyter notebook in the browser.<\/p>\n
$ kubectl port-forward tf-notebook 8888:8888<\/pre>\n
\"\"<\/a><\/p>\n
You can now see the Jupyter homepage and continue with your workflows — all running within Kubernetes and accelerated with GPUs!<\/p>\n
Conclusion<\/i><\/a><\/h2>\n
This post covers the NVIDIA GPU Operator and how it can be used to provision and manage nodes with NVIDIA GPUs into a Kubernetes cluster. Get started<\/a> with the GPU Operator via a Helm chart on NGC today or get the source<\/a> from our GitHub repo. The future is exciting and includes features like support for advanced labelling, monitoring, update management and more.<\/p>\n
If you have questions or comments please leave them below in the comments section. For technical questions about installation and usage, we recommend filing an issue on the GitHub repo<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
Editor’s note: Interested in GPU Operator? Register for our upcoming webinar on January 20th, “How to Easily use GPUs with Kubernetes”. Over the last few years, NVIDIA has leveraged GPU containers in a variety of ways for testing, development and running AI workloads in production at scale. Containers optimized for NVIDIA GPUs and systems such … Continued<\/a><\/p>\n","protected":false},"author":468,"featured_media":15768,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"publish_to_discourse":"","publish_post_category":"318","wpdc_auto_publish_overridden":"","wpdc_topic_tags":"","wpdc_pin_topic":"","wpdc_pin_until":"","discourse_post_id":"602299","discourse_permalink":"https:\/\/forums.developer.nvidia.com\/t\/nvidia-gpu-operator-simplifying-gpu-management-in-kubernetes\/148729","wpdc_publishing_response":"success","wpdc_publishing_error":"","nv_subtitle":"","ai_post_summary":"