Neural Network is a useful machine-learning model. Training up a Neural Network is however a complicated task.

Two of the existing Neural Network training algorithms are Backpropagation with Magnified Gradient Function (MGFProp) and Adaptive Particle Swarm Optimization (APSO). MGFProp is good at searching for optima but usually requires a long start up time before it can make any progress. While APSO requires almost no start up time but bad at searching for optima, i.e. usually there are better positions unexplored next to the global best-explored position.

Based on these observations, this project aims to design a hybrid algorithm of MGFProp and APSO and investigate its performance, in terms of learning speed and global optimality. In this project, we use APSO to perform global searches for good starting positions for MGFProp to perform local searches.

The searching agents, known as particles, perform global search by APSO, except one of them perform local search by MGF around the global best position to make sure that the current global best position is at least a local optimum. When the MGFProp is trapped by a local optimum, it restarts the searching at other position based on the heuristic of APSO.

Our performance investigation shows that the hybrid algorithm outperforms both algorithm in terms of the global optimality, and learning rate. This project makes good use of the benefits of both algorithms to provide a new algorithm that has high learning rate, success rate and ease of implementation to save the time and resource for training up a neural network for real-life application. This project also suggests the possibility of applying gradient method in PSO for other purpose besides neural network training.

Copyright Lui Wing On and Vanessa Ng 2012