Fix nccl regression on PyTorch 2.3 upgrade#2099
Conversation
| pip install -r requirements_cuda.txt && \ | ||
| pip install ".[bnb, accelerate, quantize, peft, outlines]" --no-cache-dir | ||
| pip install ".[bnb, accelerate, quantize, peft, outlines]" --no-cache-dir && \ | ||
| pip install nvidia-nccl-cu12==2.22.3 |
There was a problem hiding this comment.
Would have liked to use pyproject.toml for that, but poetry disapproves of conflict handling python-poetry/poetry#697 (comment)
|
Thanks a lot for the find, the fix and the details. I'm more on the fence of waiting for torch to fix it (2.3.1 hasn't fixed it yet) since afaik this does NOT affect production. |
|
As you'd like. I am using this fix to benchmark. |
|
Nice fix @fxmarty ! |
OlivierDehaene
left a comment
OlivierDehaene
left a comment
There was a problem hiding this comment.
Since this affect real deployments, let's merge this.
| pip install ".[bnb, accelerate, quantize, peft, outlines]" --no-cache-dir && \ | ||
| pip install nvidia-nccl-cu12==2.22.3 | ||
|
|
||
| ENV LD_PRELOAD=/opt/conda/lib/python3.10/site-packages/nvidia/nccl/lib/libnccl.so.2 |
There was a problem hiding this comment.
Why do we need to preload?
There was a problem hiding this comment.
Otherwise, the shared object is not used. The current base docker image of TGI is nvidia/cuda:12.1.0-base-ubuntu22.04, where there is no libnccl.so anywhere and it is not loaded by pytorch either, although we have /opt/conda/lib/libcudart.so.12.1.105 etc. COPY --from=pytorch-install /opt/conda /opt/conda does not seem to copy any libnccl.so. Weird.
* fix nccl issue * add note in dockerfile * use v2.22.3 that also fixes @samsamoa's repro * poetry actually can't handle the conflict between torch and nccl * set LD_PRELOAD
4 participants
As per title, fixes NVIDIA/nccl#1251 in TGI's cuda image, regression introduced in #1730 & #1833
We hit this issue e.g. with llama 3 70B model with TP=4 or TP=8 on H100 & default cuda graphs, one can e.g. repro the hanging with
text-generation-benchmark --tokenizer-name meta-llama/Meta-Llama-3-70B-Instruct --sequence-length 128 --decode-length 10 --warmups 2 --runs 100 -b 1, where shards hang inPyTorch 2.3 has a hard requirement on nccl 2.20.5 so I am not completely sure this fix is fine. We could also choose to downgrade.
interesting read as well https://pytorch.slack.com/archives/C3PDTEV8E/p1713223950622429?thread_ts=1712807088.459829&cid=C3PDTEV8E
Will wait for the build to run to check TGI's benchmark again & any potential regression.