Gradient Decent Algorithm —Minimizing Loss Function in Deep Neural Networks

Gradient Decent Algorithm —Minimizing Loss Function in Deep Neural Networks

Mar 17, 2024 - 21:23
Mar 17, 2024 - 21:25
 0  433
Gradient Decent Algorithm —Minimizing Loss Function in Deep Neural Networks

In my past blog on predicting inflow to reservoir, I have tried to show everything important step by step. If you haven’t read, please use this link .

You can also access the flow data from my github account. Two above links are important to understand what I’m writing here.

Ok, let’s start. In ANN, gradient decent is an important hyperparameter to evaluate our prediction’s accuracy. Accuracy is very crucial in prediction. Most of the time, it’s very difficult to reach at required accuracy level with foreword propagation only. Thus, finding the lowest possible error is all about a gradient decent.

What is a gradient decent? Gradient Descent is an iterative optimization algorithm widely employed in machine learning to minimize a cost function by adjusting the parameters of a model. The core idea is to move towards the minimum of the cost function by iteratively updating the model parameters in the direction opposite to the gradient of the cost function with respect to those parameters. Mathematically, the update rule for the parameters (θ) at each iteration is given by:

θ=θ−α⋅∇J(θ)

Here, α denotes the learning rate, a hyperparameter that controls the size of each update, and ∇J(θ) represents the gradient of the cost function J(θ) with respect to the parameters. The negative sign shows movement towards the minimum, and the process is repeated until convergence or a predefined number of iterations.

During each iteration, the algorithm performs forward propagation to compute the predicted output, calculates the cost using the predicted and actual values, and then performs backward propagation to obtain the gradients. The model parameters are updated using these gradients, and the process continues until the cost function reaches a minimum.

Here learning rate should be chosen carefully. Too high rate may lead to overshooting and convergence issues, while a too low rate can result in slow convergence or getting stuck in local minima. The global minima should be found in order to have high prediction accuracy.

Now Let’s visualize a gradient decent using python and see the plot in finding the global minima.

import numpy as np
import matplotlib.pyplot as plt


def cost_function(x):
    return x**2
# Now, let's do the derivative of the function (gradient)
def gradient(x):
    return 2 * x
#Here what we do finding the gradient decent with starting point, learning rate and iterations
def gradient_descent(starting_point, learning_rate, iterations):
    path = [starting_point]
    
    for _ in range(iterations):
        current_point = path[-1]
        update = -learning_rate * gradient(current_point)
        new_point = current_point + update
        path.append(new_point)
    
    return np.array(path)

# Giving the initial parameters
starting_point = 4.0
learning_rate = 0.1
iterations = 20

path = gradient_descent(starting_point, learning_rate, iterations)
#Let's see the plot for visualization
x_vals = np.linspace(-5, 5, 100)
y_vals = cost_function(x_vals)
plt.plot(x_vals, y_vals, label='Cost Function')
plt.scatter(path, cost_function(path), c='#ff00ff', label='Gradient Descent Path')
plt.title('Gradient Descent Vs Cost function')
plt.xlabel('Parameter Value')
plt.ylabel('Cost')
plt.legend()
plt.show()

What does it mean when reaching a global minimum in ANN?

The global minimum of the cost function represents the optimal set of parameters where the network achieves its lowest possible error. Reaching the global minimum implies that the model has found the most suitable configuration to accurately represent the underlying patterns in the training data. However, due to the complex and high-dimensional nature of neural network cost functions, finding the global minimum is not always guaranteed, and the algorithm might converge to a local minimum or a saddle point.

With global minimum value, our mean squared error (MSE) will be closer to 0. That means, the predicted value is almost equal to the actual value of the testing data.

J(θ) = (1/2m) * Σ (hat{y}_i - y_i)^2

Where:

J(θ) is the MSE.

m is the number of training examples.

hat{y}_i is the predicted output for the i-th example.

y_i is the actual output for the i-th example.

In the next, article, we will see how to approach to the global minimum in our prediction of Inflow in to reservoir.

What's Your Reaction?

like

dislike

love

funny

angry

sad

wow

Lema Kefyalew Studying a Ph.D. in Civil Engineering (Hydraulic Engineering). Studying Master of Project Management (MPM). Graduated with MSc in Civil Engineering (Hydraulic Engineering). Followed a 12 month Software Engineering program at ALX_Africa (Holberton school). Graduated with BA degree in Management. Graduated with BSc degree in Civil Engineering.