Enabling Infiniband support for Gloo data channel with auto IB detection #4795

teng-li · 2018-01-23T03:08:37Z

This PR enables the proper and easy use of Infiniband support for Gloo backend of distributed training.

Now simply just building PyTorch with

python ./setup.py install

will take care of everything by automatically detecting IB devices on the system.

this helper function of gloo ::gloo::transport::ibverbs::getDeviceNames was added earlier by me to automatically find all IB interfaces in the system.

For the Gloo data channel and cache. We now use a vector to store all the devices (not used currently, but will be able to easily extend in the future to support multiple IB devices).

Also fixed the format error of the bcast gpu direct checking.

Tested for both TCP and IB, both works fine.

Snippet of build logs:

-- Found NCCL: /public/apps/NCCL/2.1.2-1/include
-- Determining NCCL version from the header file: /public/apps/NCCL/2.1.2-1/include/nccl.h
-- NCCL_MAJOR_VERSION: 2
-- Found NCCL (include: /public/apps/NCCL/2.1.2-1/include, library: /public/apps/NCCL/2.1.2-1/lib/libnccl.so)
-- Found MPI_C: /public/apps/openmpi/2.1.1/gcc.4.8.4/lib/libmpi.so
-- Found MPI_CXX: /public/apps/openmpi/2.1.1/gcc.4.8.4/lib/libmpi.so
-- Found Gloo: TRUE
-- Found CUDA: /public/apps/cuda/9.0 (found suitable version "9.0", minimum required is "7.5")
-- MPI_LIBRARIES: /public/apps/openmpi/2.1.1/gcc.4.8.4/lib/libmpi.so
-- Found Gloo, will compile with Gloo distributed backend
-- Building the gloo backend with both TCP and infiniband support
-- NCCL_LIBRARIES: /public/apps/NCCL/2.1.2-1/lib/libnccl.so
-- NCCL Version 2 or higher found, will compile with NCCL distributed backend

PLUS

Added a helper script to automatically detect IB devices in the system and enable IB build by default. The user can have the option to force IB build as well using

USE_GLOO_IBVERBS python ./setup.py install

IB detected

running install
running build_deps
-- IB_detect: 4 IB devices detected, compiling with IB support.
-- Autodetected CUDA architecture(s): 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0
-- Found CUDA with FP16 support, compiling with torch.CudaHalfTensor
-- Removing -DNDEBUG from compile flags

No IB detected

running install
running build_deps
-- IB_detect: no IB device detected, compiling with no IB support by default unless overridden by WITH_GLOO_IBVERBS
-- Autodetected CUDA architecture(s): 6.0 6.0
-- Found CUDA with FP16 support, compiling with torch.CudaHalfTensor
-- Removing -DNDEBUG from compile flags

No IB tool found

running install
running build_deps
-- IB_detect: unable to detect IB devices, compiling with no IB support by default unless overridden by WITH_GLOO_IBVERBS

torch/lib/THD/base/data_channels/DataChannelGloo.cpp

torch/lib/THD/base/data_channels/DataChannelGloo.hpp

torch/lib/THD/base/data_channels/GlooCache.hpp

setup.py

teng-li · 2018-01-24T01:51:59Z

@apaszke Now we auto-detect IB and enables the IB build when detected, user can force IB build by also using
USE_GLOO_IBVERBS python ./setup.py install

torch/lib/THD/base/data_channels/DataChannelGloo.cpp

teng-li · 2018-01-24T03:46:11Z

CI failure is unrelated, fixed in #4826

teng-li · 2018-01-24T08:09:10Z

@pytorchbot retest this please

tools/setup_helpers/ib_detect.py

+IB_DEVINFO_CMD = "ibv_devinfo"
+
+
+def get_command_path(command):


tools/setup_helpers/ib_detect.py

+        if len(res) != 1:
+            raise Exception("-- IB_detect: unexpected parsing error while "
+                            "trying to find the number of available devices.")
+        return int(res[0])


setup.py

        my_env["CUDNN_LIBRARY"] = CUDNN_LIBRARY
        my_env["CUDNN_INCLUDE_DIR"] = CUDNN_INCLUDE_DIR

+    if WITH_DISTRIBUTED and (WITH_IB_DEVICES or


tools/setup_helpers/ib_detect.py

+    WITH_IB_DEVICES = True
+
+else:
+    print("-- IB_detect: no IB device detected, compiling with no IB support "


…mments

onnxbot-worker-1 mentioned this pull request Jan 23, 2018

[auto] pytorch-pr-4795 onnxbot/onnx-fb-universe#365

Closed

teng-li force-pushed the gloo-enable-ib branch 3 times, most recently from 04a3ab9 to 8647463 Compare January 23, 2018 06:42

apaszke reviewed Jan 23, 2018

View reviewed changes

setup.py Outdated

This comment was marked as off-topic.

Sign in to view

teng-li force-pushed the gloo-enable-ib branch from 30788a8 to 97b49a6 Compare January 24, 2018 01:46

teng-li changed the title ~~Enabling Infiniband support for Gloo data channel~~ Enabling Infiniband support for Gloo data channel with auto IB detection Jan 24, 2018

teng-li force-pushed the gloo-enable-ib branch from 97b49a6 to e2c4fd2 Compare January 24, 2018 01:56

teng-li commented Jan 24, 2018

View reviewed changes

torch/lib/THD/base/data_channels/DataChannelGloo.cpp Outdated

This comment was marked as off-topic.

Sign in to view

teng-li force-pushed the gloo-enable-ib branch 5 times, most recently from 3c6939b to a67ea5a Compare January 24, 2018 03:20

teng-li added 3 commits January 23, 2018 19:34

Enabling Infiniband support for Gloo data channel

3fb4357

Automatic use IB if available or fallback to TCP

fa9ae6e

Set WITH_GLOO_IBVERBS off by default

40b9fa8

teng-li force-pushed the gloo-enable-ib branch 2 times, most recently from a82e11e to fb4eb7f Compare January 24, 2018 03:45

teng-li force-pushed the gloo-enable-ib branch from fb4eb7f to 9a143f7 Compare January 24, 2018 03:46

Added IB detecter in tools.setup_helpers

b5a4201

teng-li force-pushed the gloo-enable-ib branch from 9a143f7 to b5a4201 Compare January 24, 2018 04:55

apaszke reviewed Jan 24, 2018

View reviewed changes

teng-li force-pushed the gloo-enable-ib branch from 454ae08 to 1ef007c Compare January 24, 2018 19:26

Now use dist_check to check all distributed related and addressed com…

b45de3a

…mments

teng-li force-pushed the gloo-enable-ib branch from 1ef007c to b45de3a Compare January 24, 2018 19:30

apaszke approved these changes Jan 24, 2018

View reviewed changes

apaszke merged commit 1b3d6ab into pytorch:master Jan 24, 2018

		IB_DEVINFO_CMD = "ibv_devinfo"


		def get_command_path(command):

Enabling Infiniband support for Gloo data channel with auto IB detection #4795

Enabling Infiniband support for Gloo data channel with auto IB detection #4795

Uh oh!

Conversation

teng-li commented Jan 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

teng-li commented Jan 24, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

teng-li commented Jan 24, 2018

Uh oh!

teng-li commented Jan 24, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

teng-li commented Jan 23, 2018 •

edited

Loading