{"id":4005,"date":"2014-11-17T06:19:32","date_gmt":"2014-11-17T14:19:32","guid":{"rendered":"https:\/\/developer.nvidia.com\/blog\/parallelforall\/?p=4005"},"modified":"2022-08-21T16:37:28","modified_gmt":"2022-08-21T23:37:28","slug":"increase-performance-gpu-boost-k80-autoboost","status":"publish","type":"post","link":"https:\/\/developer.nvidia.com\/blog\/increase-performance-gpu-boost-k80-autoboost\/","title":{"rendered":"Increase Performance with GPU Boost and K80 Autoboost"},"content":{"rendered":"

NVIDIA\u00ae GPU Boost™\u00a0is a feature available on NVIDIA\u00ae GeForce\u00ae and Tesla\u00ae GPUs that\u00a0boosts application performance by increasing GPU core and memory clock rates when\u00a0sufficient power and thermal headroom are\u00a0available (See the earlier Parallel Forall\u00a0post about GPU Boost by Mark Harris<\/a>). \u00a0In the case of Tesla GPUs, GPU Boost is customized for compute-intensive workloads running on clusters. In this post I describe GPU Boost in more detail and show you how you can take advantage of it in your applications. I also introduce Tesla K80 autoboost and demonstrate that it can automatically match the performance of explicitly controlled\u00a0application clocks.<\/p>\n

Tesla GPUs target a specific power budget, for example Tesla K40 has a TDP (Thermal Design Power) of 235W and Tesla K80 has a TDP of\u00a0300W. These TDP ratings are upper limits, and the graph in Figure 1 shows that many HPC workloads do not come close to this power limit. NVIDIA GPU Boost for Tesla allows users\u00a0to increase application performance by using available power headroom to select higher graphics clock rates.<\/p>\n

\"Figure<\/a>
Figure 1: Average GPU Power Consumption for Real Applications<\/figcaption><\/figure>\n

NVIDIA GPU Boost is exposed for Tesla accelerators\u00a0via application clock settings and on the new Tesla K80 accelerator it can also be enabled via the new autoboost feature, which is enabled by default. A user or system administrator can disable autoboost and manually set the right clocks for an application, by\u00a0either:<\/p>\n

    \n
  1. running the command line tool\u00a0nvidia-smi<\/a><\/kbd>\u00a0locally on the node, or<\/li>\n
  2. programmatically using\u00a0the\u00a0NVIDIA Management Library (NVML)<\/a>.<\/li>\n<\/ol>\n

    <\/p>\n

    Controlling GPU Boost with the NVIDIA System Management Interface<\/i><\/a><\/h2>\n

    You can\u00a0use\u00a0nvidia-smi<\/kbd>\u00a0to control application clocks without any changes to the application.<\/p>\n

    You can display the current application clock setting by passing the query option (-q<\/kbd>) to\u00a0nvidia-smi<\/kbd>. With the\u00a0-i<\/kbd>\u00a0and the display options (-d<\/kbd>) you can\u00a0filter this view to only show the clock information for a specific GPU.<\/p>\n

    $ nvidia-smi  -q -i 0 -d CLOCK\r\n\r\n==============NVSMI LOG==============\r\n[...]\r\n    Applications Clocks\r\n        Graphics                    : 745 MHz\r\n        Memory                      : 3004 MHz\r\n    Default Applications Clocks\r\n        Graphics                    : 745 MHz\r\n        Memory                      : 3004 MHz\r\n[...]\r\n<\/pre>\n
    Before you\u00a0can change the application clocks you need to put the GPU in Persistence Mode and query the available\u00a0application clock rates. Persistence mode ensures that the driver stays loaded even when no CUDA or X applications are running on the GPU. This maintains current state, including requested applications clocks.Persistence Mode is necessary to make application clock changes persistent until the application runs. Enable Persistence Mode with the following command line (for GPU 0).<\/p>\n
    $ sudo nvidia-smi -pm ENABLED -i 0\r\nEnabled persistence mode for GPU 0000:04:00.0.\r\nAll done.\r\n<\/pre>\n
    You can then query the supported\u00a0application clocks with the display option (-d SUPPORTED_CLOCKS<\/kbd>).<\/p>\n
    $ nvidia-smi  -q -i 0 -d SUPPORTED_CLOCKS\r\n\r\n==============NVSMI LOG==============\r\n\r\nTimestamp                           : Wed Oct 29 08:31:22 2014\r\nDriver Version                      : 340.32\r\n\r\nAttached GPUs                       : 6\r\nGPU 0000:04:00.0\r\n    Supported Clocks\r\n        Memory                      : 3004 MHz\r\n            Graphics                : 875 MHz\r\n            Graphics                : 810 MHz\r\n            Graphics                : 745 MHz\r\n            Graphics                : 666 MHz\r\n        Memory                      : 324 MHz\r\n            Graphics                : 324 MHz\r\n<\/pre>\n
    Please note that the supported graphics clock rates are tied to a specific\u00a0memory clock rate so when setting application clocks you must\u00a0set both the memory clock and the graphics clock. Do this using the\u00a0-ac<\/kbd>\u00a0command line option.<\/p>\n
    $ sudo nvidia-smi -ac 3004,875 -i 0\r\nApplications clocks set to "(MEM 3004, SM 875)" for GPU 0000:04:00.0\r\nAll done.\r\n<\/pre>\n
    Resetting the default is possible with the\u00a0-rac<\/kbd>\u00a0(“reset application clocks) option.<\/p>\n
    $ sudo nvidia-smi  -rac -i 0\r\nAll done.\r\n<\/pre>\n
    To avoid trouble in multi-user environments, changing application clocks requires administrative privileges. However, a system administrator can relax this requirement\u00a0to allow non-admin users to change application clocks by setting the application clock permissions to\u00a0UNRESTRICTED<\/kbd>\u00a0using the\u00a0-acp<\/kbd>\u00a0(“application clock permissions”) option to\u00a0nvidia-smi<\/kbd>.<\/p>\n
    sudo nvidia-smi -acp UNRESTRICTED -i 0\r\nApplications clocks commands have been set to UNRESTRICTED for GPU 0000:04:00.0\r\nAll done.\r\n<\/pre>\n
    Please be aware that the application clocks setting is a recommendation. If the GPU cannot safely run at the selected clocks, for example\u00a0due to thermal or power reasons, it will dynamically lower the clocks. You can query whether\u00a0this has\u00a0happened with\u00a0nvidia-smi -q -i 0 -d PERFORMANCE<\/kbd>. This behavior ensures that you always get correct results even if the application clocks are set\u00a0too high.<\/p>\n
    Controlling GPU Boost with the\u00a0NVIDIA Management Library<\/i><\/a><\/h2>\n
    The\u00a0NVIDIA Management Library (NVML) is a C-based API for monitoring and managing various states of NVIDIA GPU devices. NVML is primarily used by\u00a0cluster management tools<\/a>\u00a0to manage\u00a0NVIDIA\u00ae Tesla\u00ae GPUs, but it can be also used directly from GPU-accelerated\u00a0applications. This is interesting because end users of GPU-accelerated applications may\u00a0not be aware of GPU Boost™ and they may not know\u00a0the optimal settings for execution of applications. With NVML a GPU-accelerated application can set application clocks to the optimal value without requiring the user run\u00a0nvidia-smi<\/kbd>\u00a0to set them before starting the application. The NVML runtime ships with the CUDA driver and you can download the NVML\u00a0SDK as part of the\u00a0NVIDIA GPU Deployment Kit<\/a>\u00a0(GDK).<\/p>\n
    An application can use the full\u00a0NVML API\u00a0<\/a>to interact with the installed GPUs. Let’s\u00a0step through an\u00a0example program that uses NVML to control\u00a0application clocks.<\/p>\n
    Compiling and Linking<\/i><\/a><\/h3>\n
    To use the NVML API the application needs to include the NVML header (#include <nvml.h><\/code>) and link to the NVML runtime library (nvidia-ml<\/kbd>\u00a0on Linux and\u00a0nvml<\/kbd>\u00a0on Windows).<\/p>\n
    nvcc -I$(GPU_DEPLOYMENT_KIT_ROOT_DIR)/include/nvidia/gdk -lnvidia-ml ...<\/pre>\n
    Initializing NVML<\/i><\/a><\/h3>\n
    Before making any NVML calls need to initialize it using\u00a0nvmlInit()<\/code>.<\/p>\n
    nvmlReturn_t nvmlError = nvmlInit();\r\nif (NVML_SUCCESS != nvmlError )\r\n    fprintf (stderr, "NVML_ERROR: %s (%d) \\n", \r\n             nvmlErrorString( nvmlError ), nvmlError);\r\n<\/pre>\n
    You should always check error codes, but for simplicity we omit error checking code\u00a0in\u00a0the remainder of this post. NVML is thread-safe and initialization is reference counted so it’s safe to call\u00a0nvmlInit<\/kbd><\/a>\u00a0and\u00a0nvmlShutdown<\/kbd><\/a>\u00a0multiple times as long as there is a\u00a0nvmlShutdown()<\/kbd>\u00a0call for each call to\u00a0nvmlInit()<\/kbd>.<\/p>\n
    Obtaining a NVML device handle from PCI-E identifiers<\/i><\/a><\/h3>\n
    Before we can make queries or change any\u00a0GPU state we need an\u00a0NVML device handle. Since the numbering of NVML devices can be different than the numbering of CUDA devices (for example\u00a0due to a non-default\u00a0CUDA_VISIBLE_DEVICES<\/a>\u00a0environment variable), we need to search for the NVML devices matching the the PCIe information for the active CUDA device.<\/p>\n
    //0. Get active CUDA device\r\nint activeCUDAdevice = 0;\r\ncudaGetDevice ( &activeCUDAdevice );\r\n\r\n//1. Get device properties of active CUDA device\r\ncudaDeviceProp activeCUDAdeviceProp;\r\ncudaGetDeviceProperties ( &activeCUDAdeviceProp, activeCUDAdevice );\r\n\r\n//2. Get number of NVML devices\r\nunsigned int nvmlDeviceCount = 0;\r\nnvmlDeviceGetCount ( &nvmlDeviceCount );\r\n\r\nnvmlDevice_t nvmlDeviceId;\r\n//3. Loop over all NVML devices\r\nfor ( unsigned int nvmlDeviceIdx = 0; \r\n      nvmlDeviceIdx < nvmlDeviceCount; \r\n      ++nvmlDeviceIdx )\r\n{\r\n    //4. Obtain NVML device Id\r\n    nvmlDeviceGetHandleByIndex ( nvmlDeviceIdx, nvmlDeviceId );\r\n    \r\n    //5. Query PCIe Info of the NVML device\r\n    nvmlPciInfo_t nvmPCIInfo;\r\n    nvmlDeviceGetPciInfo ( *nvmlDeviceId, &nvmPCIInfo );\r\n    \r\n    //6. Compare NVML device PCI-E info with CUDA device properties\r\n    if ( static_cast<unsigned int>(activeCUDAdeviceProp.pciBusID)\r\n             == nvmPCIInfo.bus &&\r\n         static_cast<unsigned int>(activeCUDAdeviceProp.pciDeviceID)\r\n             == nvmPCIInfo.device && \r\n         static_cast<unsigned int>(activeCUDAdeviceProp.pciDomainID)\r\n             == nvmPCIInfo.domain )\r\n        break;\r\n}\r\n<\/pre>\n
    Controlling Application Clocks with NVML<\/i><\/a><\/h3>\n
    Now that we have a\u00a0nvmlDeviceId<\/kbd>\u00a0we can query the GPU status.<\/p>\n
    //Query current application clock setting\r\nunsigned int appSMclock = 0;\r\nunsigned int appMemclock = 0;\r\nnvmlDeviceGetApplicationsClock ( nvmlDeviceId, \r\n                                 NVML_CLOCK_SM, \r\n                                 &appSMclock );\r\nnvmlDeviceGetApplicationsClock ( nvmlDeviceId, \r\n                                 NVML_CLOCK_MEM, \r\n                                 &appMemclock );\r\n\r\n//Query maximum application clock setting\r\nunsigned int maxSMclock = 0;\r\nunsigned int maxMemclock = 0;\r\nnvmlDeviceGetMaxClockInfo ( nvmlDeviceId, \r\n                            NVML_CLOCK_SM, \r\n                            &maxSMclock );\r\nnvmlDeviceGetMaxClockInfo ( nvmlDeviceId, \r\n                            NVML_CLOCK_MEM, \r\n                            &maxMemclock );\r\n<\/pre>\n
    Before attempting to change the application clocks we should check the application clock permissions.<\/p>\n
    nvmlEnableState_t isRestricted;\r\nnvmlDeviceGetAPIRestriction ( nvmlDeviceId, \r\n                              NVML_RESTRICTED_API_SET_APPLICATION_CLOCKS, \r\n                              &isRestricted );\r\n<\/pre>\n
    If the application clock permissions allow non-admin users (or applications)\u00a0to change the application clocks, then we can go ahead and change them.<\/p>\n
    if ( NVML_FEATURE_DISABLED == isRestricted )\r\n{\r\n    nvmlDeviceSetApplicationsClocks ( nvmlDeviceId, \r\n                                      maxMemclock , \r\n                                      maxSMclock  );\r\n}\r\n<\/pre>\n
    This example only covers how to query the maximum supported application clocks. With\u00a0nvmlDeviceGetSupportedGraphicsClocks<\/kbd><\/a>\u00a0and\u00a0nvmlDeviceGetSupportedMemoryClocks<\/kbd><\/a>\u00a0you can\u00a0query all supported application clock rates, which gives you finer-grain control. The example given below actually uses\u00a0nvmlDeviceGetSupportedGraphicsClocks<\/kbd>. Also please remember that the application clocks setting is a recommendation. You can query if the GPU could not run with the specified application clocks by calling\u00a0nvmlDeviceGetCurrentClocksThrottleReasons<\/kbd><\/a>.<\/p>\n
    Shutting down NVML<\/i><\/a><\/h3>\n
    Its always nice to clean up after yourself so reset the application clocks (if they have been changed) before shutting down NVML.<\/p>\n
    nvmlDeviceResetApplicationsClocks ( nvmlDeviceId );\r\nnvmlShutdown();\r\n<\/pre>\n
    NVML Example Performance Results<\/i><\/a><\/h2>\n
    I started with\u00a0the\u00a0Matrix Multiplication example from the CUDA Code Samples<\/a>\u00a0and used NVML to execute the Matrix Multiplication with different application clocks. Complete source code for this modified example can be found in the\u00a0Parallel Forall Github repository<\/a>. Besides this the\u00a0GDK<\/a>\u00a0also contains an example which can be found at\u00a0usr\/src\/gdk\/nvml\/examples<\/kbd>\u00a0within your GDK installation. Figure 1 plots the performance of the example on Tesla K40 and Tesla K80, normalized to the performance of each GPU at the Tesla K40 base clock rate of 745 Mhz. For both GPUs increasing the clocks leads to a linear performance increase for this kernel.<\/p>\n
    \"Performance<\/a>
    Figure 2: Performance of the CUDA Samples Matrix Multiplication normalized to K40 GPU base clock rate of 745MHz. (Matrix size 1024×1024)<\/figcaption><\/figure>\n
    Tesla K80 and Autoboost<\/i><\/a><\/h2>\n
    As shown in Figure 2 above the Tesla K80 significantly extends the available\u00a0set of\u00a0application clocks from 4 levels to 25 levels. This gives more flexibility in choosing the best clocks for a given application. K80 also introduces autoboost, which\u00a0automatically selects the highest possible clock rate allowed by the thermal and power budget as visualized in Figure 3 below. With autoboost for the matrix multiplication example the highest performance can be achieved as shown in In Figure 2 above. Figure 2 shows that\u00a0Tesla K80 autoboost is able to achieve the highest performance for our matrix multiplication example\u00a0 (the same as K80’s\u00a0maximum application clock setting).<\/p>\n
    \"Tesla<\/a>
    Figure 3: Tesla K80 Auto Boost (Target application clock is shown as dotted orange line)<\/figcaption><\/figure>\n
    Figure 4 plots the performance across varying GPU clocks of the Molecular Dynamics package\u00a0GROMACS v5.0.2 for a water box with 96k atoms using PME electrostatics on Telsa K40 and Tesla K80. Performance of K80 with autoboost enabled is shown on the far right of the plots. As you can see Auto Boost delivers the best performance for Tesla K80 and with a Tesla K80 the simulation runs up to 1.9x faster than with a Tesla K40 running at default clocks and up to 1.5x faster when compared to the Tesla K40 running at 875 Mhz ([1]). To demonstrate the impact of GPU Boost in isolation these benchmarks have been run with the latest release of GROMACS which does not have any special tuning for Tesla K80. Tesla K80 specific optimization making use of the larger register file will be available with the next GROMACS release.<\/p>\n
    \"GROMACS<\/a>
    Figure 4: GROMACS Performance with GPU Boost for Tesla K40 and Tesla K80 ([1])<\/figcaption><\/figure>\n
    \nSince the autoboost feature of the Tesla K80 allows the GPU to automatically control its clocks one might think that this renders application clocks unnecessary. However, application clocks are still necessary to avoid load balancing issues in large cluster installations running multi-node, multi-GPU applications.<\/p>\n
    The autoboost feature is enabled by default. In case it has been\u00a0disabled it can be enabled with nvidia-smi by changing the default setting.<\/p>\n
    sudo nvidia-smi --auto-boost-default=ENABLED -i 0\r\n<\/pre>\n
    As with application clocks, this setting requires administrative priveleges, and the GPU should have Persistence Mode enabled. Autoboost permissions can be relaxed similarly to application clock permissions.<\/p>\n
    sudo nvidia-smi --auto-boost-permission=UNRESTRICTED -i 0<\/pre>\n
    Independent of the global default setting the autoboost behavior can be overridden by setting the environment variable CUDA_AUTO_BOOST<\/kbd> to 0<\/kbd> (to disable) or 1<\/kbd> (to enable) or via NVML for the calling process with nvmlDeviceSetAutoBoostedClocksEnabled<\/kbd>.<\/p>\n
    Conclusion<\/i><\/a><\/h2>\n
    NVIDIA GPU Boost and especially the Automatic Boost feature introduced with the new Tesla K80 accelerator is an easy path to enable more performance. Using NVML your CUDA application can choose the best GPU Boost setting without any user intervention. Even when\u00a0the applications clocks permission setting prevents\u00a0your app from\u00a0changing the application clocks, NVML can help\u00a0you inform application users about this so they\u00a0can consult with their system administrator to enable\u00a0GPU Boost. To achieve exactly this the popular GPU-accelerated Molecular Dynamic application\u00a0GROMACS<\/a>\u00a0will use NVML in its\u00a0next release to control GPUBoost on\u00a0NVIDIA\u00ae Tesla\u00ae accelerators.<\/p>\n
    Try out GPU Boost and the\u00a0NVIDIA Management Library (NVML)<\/a>\u00a0in your application today!<\/p>\n
    [1] GROMACS 5.0.2 was built with GCC 4.8.2, CUDA 6.5. The benchmarks have been executed on a Dual Socket Intel\u00ae Xeon\u00ae E5-2690 v2 @ 3.00GHz (HT on) running the CUDA Driver version 340.34.<\/p>\n","protected":false},"excerpt":{"rendered":"
    NVIDIA\u00ae GPU Boost™\u00a0is a feature available on NVIDIA\u00ae GeForce\u00ae and Tesla\u00ae GPUs that\u00a0boosts application performance by increasing GPU core and memory clock rates when\u00a0sufficient power and thermal headroom are\u00a0available (See the earlier Parallel Forall\u00a0post about GPU Boost by Mark Harris). \u00a0In the case of Tesla GPUs, GPU Boost is customized for compute-intensive workloads running on … Continued<\/a><\/p>\n","protected":false},"author":245,"featured_media":4035,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"publish_to_discourse":"","publish_post_category":"318","wpdc_auto_publish_overridden":"","wpdc_topic_tags":"","wpdc_pin_topic":"","wpdc_pin_until":"","discourse_post_id":"601971","discourse_permalink":"https:\/\/forums.developer.nvidia.com\/t\/increase-performance-with-gpu-boost-and-k80-autoboost\/148415","wpdc_publishing_response":"success","wpdc_publishing_error":"","nv_subtitle":"","ai_post_summary":"