| 李伟民* **,孙毓忠*.混合通信调度的细粒度部分分片数据并行训练方法[J].高技术通讯(中文),2026,36(8):803~811 |
| 混合通信调度的细粒度部分分片数据并行训练方法 |
| Hybrid communication scheduling based fine-grained partially sharded data parallel training |
| |
| DOI：10. 3772 / j. issn. 1002 - 0470. 2026. 08. 003 |
| 中文关键词: 深度学习； 分布式训练； 部分分片数据并行； 通信调度； 梯度裁剪 |
| 英文关键词: deep learning, distributed training, partially sharded data parallel, communication scheduling, gradient clipping |
| 基金项目: |
| 作者 | 单位 | | 李伟民* ** | （*中国科学院计算技术研究所处理器芯片全国重点实验室北京 100190）
（**中国科学院大学北京 100049） | | 孙毓忠* | |
|
| 摘要点击次数: 20 |
| 全文下载次数: 19 |
| 中文摘要: |
| 数据并行是分布式深度学习训练的主流技术之一，部分分片数据并行在不增加通信量时降低了传统数据并行的图形处理器(graphics processing unit，GPU)显存消耗，提升部分分片数据并行的训练效率成为学术和工业界研究的热点。本文首先揭示GPU之间不均衡的通信负载，交换更新后参数导致的长尾延迟制约了部分分片数据并行训练系统的可扩展性。设计了一个通信高效的新型部分分片数据并行训练系统，采用了一种新的细粒度状态分片方法和新型层次化并行通信方案，平衡了不同网络链路的通信负载的差异。还设计了一种高效两阶段混合通信调度算法，将更新参数传输与反向传播重叠进行。选取了5类典型的深度学习训练负载并在32个GPU上进行了系列实验，与目前最先进的部分分片数据并行训练系统相比，本文深度学习模型的训练时间减少了15.0%到85.0%。 |
| 英文摘要: |
| Data parallelism is a mainstream approach for distributed deep learning training. Partial sharded data parallelism reduces graphics processing unit (GPU) memory consumption without increasing communication volume. Improving its training efficiency has become a key research focus in academia and industry. This paper first reveals that the unbalanced communication overhead between GPUs and the long-tail latency caused by exchanging updated parameters limit the scalability of the partial shardeddata parallel training system. We propose a communication-efficient partial sharded data parallel training system, which employs a novel fine-grained state sharding method and a hierarchical communication scheme to balance communication loads across network links. We further design an efficient two-stage hybrid communication scheduling algorithm to overlap parameter synchronization with backpropagation by utilizing idle network resources.Experimental evaluation on five representative deep learning workloads across 32 GPUs shows that our approach reduces training time by 15.0% to 85.0% compared to the state-of-the-art partial sharded data parallel systems. |
|
查看全文
查看/发表评论 下载PDF阅读器 |
| 关闭 |
