{"id":13325,"date":"2019-01-22T06:00:21","date_gmt":"2019-01-22T14:00:21","guid":{"rendered":"https:\/\/developer.nvidia.com\/blog\/?p=13325"},"modified":"2023-04-03T12:40:00","modified_gmt":"2023-04-03T19:40:00","slug":"gpu-telemetry-nvidia-dcgm","status":"publish","type":"post","link":"https:\/\/developer.nvidia.com\/blog\/gpu-telemetry-nvidia-dcgm\/","title":{"rendered":"Setting Up GPU Telemetry with NVIDIA Data Center GPU Manager"},"content":{"rendered":"

\"\"Understanding GPU usage provides important insights for IT administrators managing a data center. Trends in GPU metrics correlate with workload behavior and make it possible to optimize resource allocation, diagnose anomalies, and increase overall data center efficiency.\u00a0NVIDIA Data Center GPU Manager<\/a>\u00a0(DCGM) offers a comprehensive tool suite to simplify administration and monitoring of NVIDIA Tesla-accelerated data centers.<\/p>\n

One key capability provided by DCGM is GPU telemetry. DCGM includes sample code for integrating GPU metrics with open source telemetry frameworks such as collectd<\/a>\u00a0and Prometheus<\/a>. The DCGM API can also be used to write custom code that can integrate with site specific telemetry frameworks.<\/p>\n

Let\u2019s look at\u00a0how to integrate DCGM with collectd on a CentOS system, making GPU telemetry data available alongside your existing telemetry data.<\/p>\n

Integrating DCGM with collectd<\/i><\/a><\/h2>\n

Prerequisites<\/i><\/a><\/h3>\n

First you need to install and configure collectd and DCGM.<\/p>\n

If collectd is not already present on the system, you\u00a0can install it from the EPEL repository. (Unless otherwise specified, all command line examples need to be run as a superuser.)<\/p>\n

# yum install -y epel-release\r\n# yum install -y collectd<\/pre>\n
DCGM is available free-of-charge from the NVIDIA website<\/a>. Download the x86_64 RPM package and install it.<\/p>\n
# rpm --install datacenter-gpu-manager-1.5.6-1.x86_64.rpm<\/pre>\n
The DCGM host engine service (nv-hostengine) needs to be running in order to collect the GPU telemetry data.<\/p>\n
# nv-hostengine<\/pre>\n
Verify the DCGM host engine service is running by using it to query the current temperature of the GPUs. Note, this command can be run as a non-superuser.<\/p>\n
$ dcgmi dmon -e 150 -c 1<\/pre>\n
If you want to\u00a0automatically start the host engine when the system starts, configure a DCGM systemd service. Otherwise the host engine will need to be started manually whenever the system restarts.<\/p>\n
[Unit]\r\nDescription=DCGM service\r\n\r\n[Service]\r\nUser=root\r\nPrivateTmp=false\r\nExecStart=/usr/bin/nv-hostengine -n\r\nRestart=on-abort\r\n\r\n[Install]\r\nWantedBy=multi-user.target<\/pre>\n
Setting up the DCGM collectd plugin<\/i><\/a><\/h2>\n
Now that you\u2019ve successfully installed\u00a0collectd and DCGM, the real work to integrate the two begins. The DCGM package includes a sample collectd plugin\u00a0implemented\u00a0using the DCGM Python binding. The plugin needs to be installed and configured to use it with collectd.<\/p>\n
First, copy the DCGM Python binding and collectd plugin to the collectd plugin directory. The DCGM collectd plugin installs into\u00a0a subdirectory to separate\u00a0it from other collectd plugins.<\/p>\n
# mkdir /usr/lib64/collectd/dcgm\r\n# cp /usr/src/dcgm/bindings/*.py /usr/lib64/collectd/dcgm\r\n# cp /usr/src/dcgm/samples/scripts/dcgm_collectd_plugin.py /usr/lib64/collectd/dcgm<\/pre>\n
Next, verify that the plugin is configured with the correct location of the DCGM library (libdcgm.so<\/code>) on this system. The DCGM library is installed in \/usr\/lib64<\/code>\u00a0on CentOS systems\u00a0by default. Edit \/usr\/lib64\/collectd\/dcgm\/dcgm_collectd_plugin.py<\/code>\u00a0so that the variable g_dcgmLibPath<\/code>\u00a0is set to \/usr\/lib64<\/code>.<\/p>\n
# sed -i -e 's|\\(g_dcgmLibPath =\\) '\"'\"'\/usr\/lib'\"'\"'|\\1 '\"'\"'\/usr\/lib64'\"'\"'|g' \/usr\/lib64\/collectd\/dcgm\/dcgm_collectd_plugin.py<\/pre>\n
The DCGM plugin is initially configured to collect a number of generally useful GPU metrics. \u00a0You can customize the list of metrics by modifying the g_publishFieldIds<\/code>\u00a0variable. You\u2019ll find the names and meaning of the available fields in \/usr\/src\/dcgm\/bindings\/dcgm_fields.py<\/code>.<\/p>\n
Configuring collectd<\/i><\/a><\/h2>\n
Once the DCGM collect plugin has been set\u00a0up, collectd still needs to be configured to recognize the new metrics.<\/p>\n
First, configure collectd to recognize the DCGM plugin by adding dcgm.conf<\/code>\u00a0to \/etc\/collectd.d<\/code>.<\/p>\n
LoadPlugin python\r\n<Plugin python>\r\n\u00a0 \u00a0 \u00a0 ModulePath \"\/usr\/lib64\/collectd\/dcgm\"\r\n\u00a0 \u00a0 \u00a0 LogTraces true\r\n\u00a0 \u00a0 \u00a0 Interactive false\r\n\u00a0 \u00a0 \u00a0 Import \"dcgm_collectd_plugin\"\r\n<\/Plugin><\/pre>\n
Second, add a corresponding collectd type<\/a>\u00a0for each of the GPU fields defined in \/usr\/lib64\/collectd\/dcgm\/dcgm_collectd_plugin.py<\/code>. Assuming no additional fields were defined, append the following type information to \/usr\/share\/collectd\/types.db<\/code>.<\/p>\n
### DCGM types\r\necc_dbe_aggregate_total\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0          value:GAUGE:0:U\r\necc_sbe_aggregate_total\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0          value:GAUGE:0:U\r\necc_dbe_volatile_total\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0           value:GAUGE:0:U\r\necc_sbe_volatile_total\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0           value:GAUGE:0:U\r\nfb_free\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0          value:GAUGE:0:U\r\nfb_total\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0         value:GAUGE:0:U\r\nfb_used\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0          value:GAUGE:0:U\r\ngpu_temp \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0         value:GAUGE:U:U\r\ngpu_utilization\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0          value:GAUGE:0:100\r\nmem_copy_utilization\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0             value:GAUGE:0:100\r\nmemory_clock\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0     value:GAUGE:0:U\r\nmemory_temp\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0      value:GAUGE:U:U\r\nnvlink_bandwidth_total\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0           value:GAUGE:0:U\r\nnvlink_recovery_error_count_total\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0value:GAUGE:0:U\r\nnvlink_replay_error_count_total\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0  value:GAUGE:0:U\r\npcie_replay_counter\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0              value:GAUGE:0:U\r\npcie_rx_throughput\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0       value:GAUGE:0:U\r\npcie_tx_throughput\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0       value:GAUGE:0:U\r\npower_usage\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0      value:GAUGE:0:U\r\npower_violation\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0          value:GAUGE:0:U\r\nretired_pages_dbe\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0        value:GAUGE:0:U\r\nretired_pages_pending\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0            value:GAUGE:0:U\r\nretired_pages_sbe\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0        value:GAUGE:0:U\r\nsm_clock\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0         value:GAUGE:0:U\r\nthermal_violation\u00a0        \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0value:GAUGE:0:U\r\ntotal_energy_consumption\u00a0\u00a0         \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0value:GAUGE:0:U\r\nxid_errors\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0       value:GAUGE:0:U<\/pre>\n
If you defined additional GPU fields when installing the DCGM collectd plugin, then a corresponding collectd type needs to be manually added to the list above. The Python field name in \/usr\/lib64\/collectd\/dcgm\/dcgm_collectd_plugin.py<\/code>\u00a0and the collectd type in \/usr\/share\/collectd\/types.db<\/code>\u00a0are related, but different. To correlate the two variants of a metric name, use the field ID defined in \/usr\/src\/dcgm\/bindings\/dcgm_fields.py<\/code>\u00a0to correlate the two variants of a metric name. For example, DCGM_FI_DEV_GPU_TEMP<\/code> represents the GPU temperature in \/usr\/lib64\/collectd\/dcgm\/dcgm_collectd_plugin.py<\/code>. Looking up this field in \/usr\/src\/dcgm\/bindings\/dcgm_fields.py<\/code>\u00a0shows that it corresponds to fieldID 150. The list of collectd visible field names can be obtained from the command dcgmi dmon -l<\/code>; the collectd type name corresponding to field ID 150 is gpu_temp<\/code>.<\/p>\n
(Re-)Start collectd<\/i><\/a><\/h2>\n
When DCGM is successfully integrated with collectd, output similar to what is shown below should be reported by collectd when it starts.<\/p>\n
collectd[25]: plugin_load: plugin \"python\" successfully loaded.\r\n\u2026\r\ncollectd[25]: uc_update: Value too old: name = f707be0c326d\/dcgm_collectd-GPU-ace28880-3f61-dbc4-1f8c-0dc7916f3108\/gpu_temp-0; value time = 1539719060.000; last cache update = 1539719060.000;\r\ncollectd[25]: uc_update: Value too old: name = f707be0c326d\/dcgm_collectd-GPU-ace28880-3f61-dbc4-1f8c-0dc7916f3108\/power_usage-0; value time = 1539719060.000; last cache update = 1539719060.000;\r\ncollectd[25]: uc_update: Value too old: name = f707be0c326d\/dcgm_collectd-GPU-ace28880-3f61-dbc4-1f8c-0dc7916f3108\/ecc_sbe_volatile_total-0; value time = 1539719060.000; last cache update = 1539719060.000;\r\ncollectd[25]: uc_update: Value too old: name = f707be0c326d\/dcgm_collectd-GPU-ace28880-3f61-dbc4-1f8c-0dc7916f3108\/ecc_dbe_volatile_total-0; value time = 1539719060.000; last cache update = 1539719060.000;\r\ncollectd[25]: uc_update: Value too old: name = f707be0c326d\/dcgm_collectd-GPU-ace28880-3f61-dbc4-1f8c-0dc7916f3108\/ecc_sbe_aggregate_total-0; value time = 1539719060.000; last cache update = 1539719060.000;\r\n...<\/pre>\n
The GPU data provided by DCGM can be visualized along side the rest of your monitoring data, as shown in figure 1.<\/p>\n
\"Sample
Figure 1. Example output from collectd, visualized by Grafana<\/figcaption><\/figure>\n
Summary<\/h1>\n
Integrating DCGM<\/a> with the collectd telemetry framework provides IT administrators with a comprehensive view of GPU usage. If you are already using collectd, the information in this blog post will enable you to include GPU monitoring on the same pane of glass as the rest of your telemetry data. If you are using another telemetry framework, please see Chapter 4 of the DCGM User\u2019s Guide for more information on how to integrate GPU metrics into your solution.<\/p>\n
GPU telemetry is just scratching the surface of the full feature set<\/a>\u00a0of DCGM. DCGM also includes active health checks, diagnostics, as well as management and accounting capabilities.<\/p>\n","protected":false},"excerpt":{"rendered":"
Understanding GPU usage provides important insights for IT administrators managing a data center. Trends in GPU metrics correlate with workload behavior and make it possible to optimize resource allocation, diagnose anomalies, and increase overall data center efficiency.\u00a0NVIDIA Data Center GPU Manager\u00a0(DCGM) offers a comprehensive tool suite to simplify administration and monitoring of NVIDIA Tesla-accelerated data … Continued<\/a><\/p>\n","protected":false},"author":463,"featured_media":11067,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"publish_to_discourse":"","publish_post_category":"318","wpdc_auto_publish_overridden":"","wpdc_topic_tags":"","wpdc_pin_topic":"","wpdc_pin_until":"","discourse_post_id":"602369","discourse_permalink":"https:\/\/forums.developer.nvidia.com\/t\/setting-up-gpu-telemetry-with-nvidia-data-center-gpu-manager\/148799","wpdc_publishing_response":"success","wpdc_publishing_error":"","nv_subtitle":"","ai_post_summary":"