{"id":5177,"date":"2015-05-05T19:30:13","date_gmt":"2015-05-06T02:30:13","guid":{"rendered":"https:\/\/developer.nvidia.com\/blog\/parallelforall\/?p=5177"},"modified":"2022-08-21T16:37:32","modified_gmt":"2022-08-21T23:37:32","slug":"gpu-pro-tip-track-mpi-calls-nvidia-visual-profiler","status":"publish","type":"post","link":"https:\/\/developer.nvidia.com\/blog\/gpu-pro-tip-track-mpi-calls-nvidia-visual-profiler\/","title":{"rendered":"GPU Pro Tip: Track MPI Calls In The NVIDIA Visual Profiler"},"content":{"rendered":"

Often when profiling GPU-accelerated applications that run on clusters, one needs to visualize MPI\u00a0(Message Passing Interface<\/a>) calls on the GPU timeline in the profiler. While tools like Vampir and Tau will allow programmers to see a big picture view of how a parallel application performs, sometimes all you need is a look at how MPI is affecting GPU performance on\u00a0a single node using a simple\u00a0tool like the NVIDIA Visual Profiler. With the help of the NVIDIA Tools Extensions (NVTX) and the MPI standard itself, this is pretty easy to do.<\/p>\n

The NVTX API lets you\u00a0embed information within a\u00a0GPU profile, such as marking events\u00a0or annotating ranges in the timeline with details\u00a0about\u00a0application behavior during that time. Jiri Kraus<\/a> wrote past posts about generating custom application timelines with NVTX<\/a>, and about using it to label individual MPI ranks in MPI profiles<\/a>. In this post I’ll show you how to use an NVTX range to annotate the time spent in MPI calls. To do this, we\u2019ll use the MPI profiling interface (PMPI), which is a standard part of MPI. PMPI allows tools to intercept calls to the MPI library to perform actions before or after the MPI call is executed. This means that we can insert NVTX calls into our MPI library calls to mark MPI calls on the GPU timeline.<\/p>\n

Wrapping every MPI routine in this way is a bit tedious, but fortunately there\u2019s a tool to automate the process. We’ll\u00a0use the wrap.py<\/code> script found at https:\/\/github.com\/scalability-llnl\/wrap<\/a> to generate the PMPI wrappers for a number of commonly used MPI routines. The input file for this script is the following (also available as a github gist<\/a>):<\/p>\n

#include <pthread.h>\n#include <nvToolsExt.h>\n#include <nvToolsExtCudaRt.h>\n// Setup event category name\n{{fn name MPI_Init}}\n  nvtxNameCategoryA(999, "MPI");\n  {{callfn}}\n  int rank;\n  PMPI_Comm_rank(MPI_COMM_WORLD, &rank);\n  char name[256];\n  sprintf( name, "MPI Rank %d", rank );\n \n  nvtxNameOsThread(pthread_self(), name);\n  nvtxNameCudaDeviceA(rank, name);\n{{endfn}}\n// Wrap select MPI functions with NVTX ranges\n{{fn name MPI_Send MPI_Recv MPI_Allreduce MPI_Reduce MPI_Wait MPI_Waitany\nMPI_Waitall MPI_Waitsome MPI_Gather MPI_Gatherv MPI_Scatter MPI_Scatterv\nMPI_Allgather MPI_Allgatherv MPI_Alltoall MPI_Alltoallv MPI_Alltoallw MPI_Bcast\nMPI_Sendrecv MPI_Barrier MPI_Start MPI_Test MPI_Send_init MPI_Recv_init }}\n  nvtxEventAttributes_t eventAttrib = {0};\n  eventAttrib.version = NVTX_VERSION;\n  eventAttrib.size = NVTX_EVENT_ATTRIB_STRUCT_SIZE;\n  eventAttrib.messageType = NVTX_MESSAGE_TYPE_ASCII;\n  eventAttrib.message.ascii  = "{{name}}";\n  eventAttrib.category = 999;\n \n  nvtxRangePushEx(&eventAttrib);\n  {{callfn}}\n  nvtxRangePop();\n{{endfn}}\n<\/pre>\n
So what\u2019s happening in this file? First, it includes the NVTX header file, and then loops\u00a0over a series of common MPI functions and inserts the beginning of an NVTX range (nvtxRangePushEx<\/code>) and then ends the range as we leave the MPI routine (nvtxRangePop<\/code>). For convenience, I\u2019ve named the range after the MPI routine being called. All I need to do now is call wrap.py<\/code> to generate a C file with my PMPI wrappers, which I\u2019ll then build with my MPI C compiler.<\/p>\n
$ python wrap/wrap.py -g -o nvtx_pmpi.c nvtx.w\n$ mpicc -c nvtx_pmpi.c\n<\/pre>\n
Now I just need to rerun my code with these wrappers. To do this I\u2019ll relink my application with the object file I just built and the NVTX library (libnvToolsExt). As an example, I\u2019ll use the simple Jacobi Iteration used in the GTC session Multi GPU Programming with MPI<\/a>, which you can find\u00a0on Github<\/a>. Once I\u2019ve built both the application and the wrappers generated above, I run the executable as follows.<\/p>\n
$ mpicc -fast -ta=tesla -Minfo=all $HOME/nvtx_pmpi.o laplace2d.c -L$CUDA_HOME/lib64 -lnvToolsExt -o laplace2d\n$ MV2_USE_CUDA=1 mpirun -np 2 nvprof -o laplace2d.%q{MV2_COMM_WORLD_RANK}.nvvp ./laplace2d\n<\/pre>\n
One word of caution: the linking order does matter when using tools such as PMPI, so if you run your code and are not seeing the expected results, the object file containing the wrappers may not appear early enough in the build command.<\/em><\/p>\n
In the above commands I\u2019m rebuilding my code with the necessary bits. I\u2019m also setting MV2_USE_CUDA at runtime to enable cuda-awareness in my MVAPICH library. Additionally I\u2019m informing nvprof to generate a timeline file per-MPI process by passing the MV2_COMM_WORLD_RANK environment variable to nvprof, which is\u00a0defined\u00a0to equal the MPI rank of each process. Figure 1 is the result of importing one of these resulting nvprof output files into Visual Profiler and then zooming in to an area of interest.<\/p>\n
\"NVIDIA<\/a>
Figure 1: NVIDIA Visual Profiler with MPI ranges.<\/figcaption><\/figure>\n
Looking in the \u201cMarkers and Ranges\u201d row of the GPU timeline for MPI Rank 0, we see three green boxes denoting two calls to MPI_Sendrecv and one to MPI_Allreduce. Furthermore, we can see that the MPI library is using a device-to-device memcpy operation to communicate between two GPUs on the same node. As you can see, the NVIDIA Visual Profiler, combined with PMPI and NVTX can give you interesting insights into how the MPI calls in your application interact with the GPU.<\/p>\n","protected":false},"excerpt":{"rendered":"
Often when profiling GPU-accelerated applications that run on clusters, one needs to visualize MPI\u00a0(Message Passing Interface) calls on the GPU timeline in the profiler. While tools like Vampir and Tau will allow programmers to see a big picture view of how a parallel application performs, sometimes all you need is a look at how MPI … Continued<\/a><\/p>\n","protected":false},"author":268,"featured_media":9152,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"publish_to_discourse":"","publish_post_category":"318","wpdc_auto_publish_overridden":"","wpdc_topic_tags":"","wpdc_pin_topic":"","wpdc_pin_until":"","discourse_post_id":"602162","discourse_permalink":"https:\/\/forums.developer.nvidia.com\/t\/gpu-pro-tip-track-mpi-calls-in-the-nvidia-visual-profiler\/148593","wpdc_publishing_response":"success","wpdc_publishing_error":"","nv_subtitle":"","ai_post_summary":"