create_nccl_config

paddle.distributed. create_nccl_config ( nccl_config: dict[str, int | str] | None = None ) → NCCLConfig | None [source]

Function that creates nccl config.

Parameters

nccl_config (dict[str, int | str] | None) – None or a dict containing the following keys: commName (str): name of the process group. ll_buffsize (int): buffer size of ll protocol. ll128_buffsize (int): buffer size of ll128 protocol. simple_buffsize (int): buffer size of simple protocol. buffsize_align (int): alignment unit of the total buffer size. nchannels (int): max number of channels. algoStr (str): communication algorithm. protoStr (str): communication protocol.

Returns

an object containing the information, which can be used as an argument of new_group().

Return type

NCCLConfig (NCCLConfig | None)

Examples

>>> 
>>> import paddle
>>> import paddle.distributed as dist
>>> from typing import Union
>>> dist.init_parallel_env()
>>> nccl_config: dict[str, Union[int, str]] = {
...     "commName": "tp_comm",
...     "ll_buffsize": 0,
...     "ll128_buffsize": 0,
...     "simple_buffsize": 1024,
...     "buffsize_align": 1024,
...     "nchannels": 4,
...     "algoStr": "Ring",
...     "protoStr": "Simple",
... }
>>> ranks = [0, 1, 2, 3, 4, 5, 6, 7]
>>> nccl_config = dist.create_nccl_config(nccl_config)
>>> pg = dist.new_group(ranks, nccl_config=nccl_config)
>>> m, n = 4096, 8192
>>> local_rank = dist.get_rank(pg)
>>> num_local_ranks = dist.get_world_size(pg)
>>> x = paddle.ones(shape=[m, n], dtype=paddle.float32) * (local_rank + 1)
>>> dist.all_reduce(x, group=pg)