AI, Vol. 4, Pages 16-27: Embarrassingly Parallel Independent Training of Multi-Layer Perceptrons with Heterogeneous Architectures

In this paper we propose a procedure to enable the training of several independent Multilayer Perceptron Neural Networks with a different number of neurons and activation functions in parallel (ParallelMLPs) by exploring the principle of locality and parallelization capabilities of modern CPUs and GPUs. The core idea of this technique is to represent several sub-networks as a single large network and use a Modified Matrix Multiplication that replaces an ordinal matrix multiplication with two simple matrix operations that allow separate and independent paths for gradient flowing. We have assessed our algorithm in simulated datasets varying the number of samples, features and batches using 10,000 different models as well as in the MNIST dataset. We achieved a training speedup from 1 to 4 orders of magnitude if compared to the sequential approach. The code is available online.

Apr 23, 2024 - 22:50
 0  114
AI, Vol. 4, Pages 16-27: Embarrassingly Parallel Independent Training of Multi-Layer Perceptrons with Heterogeneous Architectures

AI, Vol. 4, Pages 16-27: Embarrassingly Parallel Independent Training of Multi-Layer Perceptrons with Heterogeneous Architectures

AI doi: 10.3390/ai4010002

Authors: Felipe C. Farias Teresa B. Ludermir Carmelo J. A. Bastos-Filho

In this paper we propose a procedure to enable the training of several independent Multilayer Perceptron Neural Networks with a different number of neurons and activation functions in parallel (ParallelMLPs) by exploring the principle of locality and parallelization capabilities of modern CPUs and GPUs. The core idea of this technique is to represent several sub-networks as a single large network and use a Modified Matrix Multiplication that replaces an ordinal matrix multiplication with two simple matrix operations that allow separate and independent paths for gradient flowing. We have assessed our algorithm in simulated datasets varying the number of samples, features and batches using 10,000 different models as well as in the MNIST dataset. We achieved a training speedup from 1 to 4 orders of magnitude if compared to the sequential approach. The code is available online.

What's Your Reaction?

like

dislike

love

funny

angry

sad

wow

Lema Kefyalew Studying a Ph.D. in Civil Engineering (Hydraulic Engineering). Studying Master of Project Management (MPM). Graduated with MSc in Civil Engineering (Hydraulic Engineering). Followed a 12 month Software Engineering program at ALX_Africa (Holberton school). Graduated with BA degree in Management. Graduated with BSc degree in Civil Engineering.