Microsoft launches ZeRO++ technology, which can significantly reduce the training time and cost of AI large model.
CTOnews.com June 27 news, Microsoft researchers have launched a new technology called ZeRO++, used to optimize the training of large AI models, easy to encounter data transmission costs and bandwidth constraints, can significantly reduce the training time and cost of large models.
ZeRO++ builds on existing ZeRO transmission technology and provides enhanced communication strategies that improve training efficiency while reducing training time and costs.
In order to reduce parameter traffic, ZeRO++ quantizes the weights, using a block-based quantization method to maintain training accuracy, which is faster and more accurate than the original Zero transmission technology. To minimize communication overhead, ZeRO++ trades GPU memory for communication bandwidth by maintaining a complete copy of the model on each machine. In gradient communication, ZeRO++ introduces a new quantized gradient communication method called qgZ, which can reduce cross-node traffic and delay.
These improved communication technologies have greatly reduced traffic, and Microsoft researchers say ZeRO++ reduces traffic by up to four times compared to ZeRO, improving training throughput and efficiency. When small batch sizes are used on each GPU, ZeRO++ achieves throughput improvements of 28 to 36 percent over ZeRO-3 in high-bandwidth clusters. In low-bandwidth clusters, ZeRO++ achieves an average of 2x speedup compared to ZeRO-3, making large model training more feasible on a wider variety of clusters.
CTOnews.com Note: CTOnews.com notes that large models such as Turing-NLG, ChatGPT, and GPT-4 require significant memory and compute resources to train across multiple GPU devices, while ZeRO++ introduces communication optimization strategies to overcome bandwidth limitations of the original ZeRO transport technology when trained on low-bandwidth clusters. Microsoft has released relevant technical documentation, and researchers can use ZeRO++ to train models more effectively and explore new possibilities in the field of AI.
