Identificador persistente para citar o vincular este elemento:
http://hdl.handle.net/10662/21063
Títulos: | Enhancing Distributed Neural Network Training through Node-Based Communications |
Autores/as: | Moreno Álvarez, Sergio Paoletti Ávila, Mercedes Eugenia Cavallaro, Gabriele Haut Hurtado, Juan Mario |
Palabras clave: | Paralelismo de datos;Enseñanza profunda;Computación de alto rendimiento;Redes neuronales;Comunicación síncrona;Data parallelism;Deep learning;High-performance computing;Neural networks;Synchronous communications |
Fecha de publicación: | 2023 |
Resumen: | The amount of data needed to effectively train modern deep neural architectures has grown significantly, leading to increased computational requirements. These intensive computations are tackled by the combination of last generation computing resources, such as accelerators, or classic processing units. Nevertheless, gradient communication remains as the major bottleneck, hindering the efficiency notwithstanding the improvements in runtimes obtained through data parallelism strategies. Data parallelism involves all processes in a global exchange of potentially high amount of data, which may impede the achievement of the desired speedup and the elimination of noticeable delays or bottlenecks. As a result, communication latency issues pose a significant challenge that profoundly impacts the performance on distributed platforms. This research presents node-based optimization steps to significantly reduce the gradient exchange between model replicas whilst ensuring model convergence. The proposal serves as a versatile communication scheme, suitable for integration into a wide range of general-purpose deep neural network (DNN) algorithms. The optimization takes into consideration the specific location of each replica within the platform. To demonstrate the effectiveness, different neural network approaches and datasets with disjoint properties are used. In addition, multiple types of applications are considered to demonstrate the robustness and versatility of our proposal. The experimental results show a global training time reduction whilst slightly improving accuracy. Code: https://github.com/mhaut/eDNNcom. |
Descripción: | Versión aceptada del artículo publicado en IEEE Transactions on Neural Networks and Learning Systems, pp. 1-15.2023. ISSN 2162237X |
DOI: | 10.1109/TNNLS.2023.3309735 |
Colección: | DISIT - Artículos DTCYC - Artículos |
Archivos
Archivo | Descripción | Tamaño | Formato | |
---|---|---|---|---|
TNNLS_2023_3309735.pdf | 3,11 MB | Adobe PDF | Descargar |
Este elemento está sujeto a una licencia Licencia Creative Commons