Yandex shared practical experience on how they learned to prioritize ML traffic in InfiniBand networks of GPU clusters so that important tasks don't "suffer" in speed due to neighboring launches.
Key points:
InfiniBand uses a centralized Subnet Manager (OpenSM) that manages addressing, routing, and QoS policies through a combination of Service Level (SL) and Virtual Lanes (VL).
QoS is built as follows: traffic of different types is "colored" into different SLs, which are mapped to VLs with different priorities and weights; in the test scheme, SL1 gets 80% of the bandwidth, SL0 gets 20%.
In YATI clusters, multiple trainings from different users share one InfiniBand fabric, so without QoS, large and critical trainings are easily "drowned" by parallel tasks.
On a FatTree cluster with HDR, they initially saw no effect until they artificially created oversubscription (disabled some spine switches), after which SL1 traffic actually began to squeeze out SL0 under contention.
In DragonFly+, it's more complex: routing uses different VLs for the direct path and +1/+3 hops to avoid credit loop deadlock in a lossless network, so SL→VL mapping becomes part of the Control Plane, and the available number of "colors" is effectively reduced.
As a result, Yandex turned QoS into a product mechanism: the training scheduler marks large trainings (by GPU threshold per cluster, configurable for each cluster) as priority, the agent on the host recolors their traffic to SL1, the rest go to SL0 — even if the user tried to set their own SLs.
Next, they plan to use the same approach to separate training and multi-host inference, giving priority to real-time inference over the network.
QoS in InfiniBand is not just "queues on a port", but a tight coupling with topology and routing engine (especially in DragonFly+), otherwise you can easily get either no effect or the risk of deadlocks.
#yandex #infiniband #qos #gpu #ml #mlops #networking #dragonflyplus #cloud #infrastructure
Comments
0No comments yet.
Sign in to join the discussion.