nccl
仓库
https://github.com/NVIDIA/nccl
文档
https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html
如NCCL_P2P_DISABLE=0开启NCCL p2p
如NCCL_NVLS_ENABLE=1 将 AllReduce、Broadcast 等集合通信操作卸载到交换机硬件执行，避免数据传输到GPU内存处理。
nixl
用于加速ai推理框架的点对点通信
仓库
https://github.com/ai-dynamo/nixl?tab=readme-ov-file
nvlink
gpu间高效互联,解决pcie的性能瓶颈
执行
nvidia-smi topo -m
得到如
nvidia-smi topo -m
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 NIC12 NIC13 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 0-23,96-119 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 0-23,96-119 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 0-23,96-119 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS SYS 24-47,120-143 1 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS SYS PIX NODE NODE SYS SYS SYS SYS 48-71,144-167 2 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS SYS SYS NODE PIX NODE SYS SYS SYS SYS 48-71,144-167 2 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS SYS NODE NODE PIX SYS SYS SYS SYS 48-71,144-167 2 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE PIX 72-95,168-191 3 N/A
NIC0 PIX NODE NODE SYS SYS SYS SYS SYS X NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC1 NODE PIX NODE SYS SYS SYS SYS SYS NODE X NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC2 NODE NODE PIX SYS SYS SYS SYS SYS NODE NODE X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC3 SYS SYS SYS NODE SYS SYS SYS SYS SYS SYS SYS X PIX PXB NODE SYS SYS SYS SYS SYS SYS SYS
NIC4 SYS SYS SYS NODE SYS SYS SYS SYS SYS SYS SYS PIX X PXB NODE SYS SYS SYS SYS SYS SYS SYS
NIC5 SYS SYS SYS NODE SYS SYS SYS SYS SYS SYS SYS PXB PXB X NODE SYS SYS SYS SYS SYS SYS SYS
NIC6 SYS SYS SYS PIX SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE X SYS SYS SYS SYS SYS SYS SYS
NIC7 SYS SYS SYS SYS PIX NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS X NODE NODE SYS SYS SYS SYS
NIC8 SYS SYS SYS SYS NODE PIX NODE SYS SYS SYS SYS SYS SYS SYS SYS NODE X NODE SYS SYS SYS SYS
NIC9 SYS SYS SYS SYS NODE NODE PIX SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE X SYS SYS SYS SYS
NIC10 SYS SYS SYS SYS SYS SYS SYS NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X PIX PXB NODE
NIC11 SYS SYS SYS SYS SYS SYS SYS NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX X PXB NODE
NIC12 SYS SYS SYS SYS SYS SYS SYS NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB X NODE
NIC13 SYS SYS SYS SYS SYS SYS SYS PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8
NIC9: mlx5_9
NIC10: mlx5_10
NIC11: mlx5_11
NIC12: mlx5_12
NIC13: mlx5_13
其中NV18表示gpu间通过nvlink连接
性能NV# > PIX > PXB > PHB > NODE > SYS
nvswitch
在交换机内执行 AllReduce、Broadcast 等集合操作，将 GPU 计算负载降低
动态规避故障链路，保障通信连续性
执行
nvidia-smi topo -m
得到
NVB标识通过nvswitch连接
NV1/NV2则比啊是nvlink连接
fabricmanager
自动探测 GPU/NVSwitch 的物理连接关系，生成最优通信路径。
用于nvswitch的配置
网友评论