Regression Analysis using Deep Learning

Deep Learning [DL] is the field of Machine Learning [ML] that uses Artificial Neural Network (ANN) for data processing and inferencing.

Neural Network mimics the human brain, where we have multiple layers of Neurons connected to each other via Synapses. Its sometimes referred to as DNN i.e. Deep Neural Network

Simplest DL model is MLP i.e. Multi Layer Perceptron that can be used for Regression and Classification problem domains.

---

• Layers in Neural Networks:
  • Input Layer
    • Here we send the input data for model training
  • Hidden Layers
    • This is where the Magic happens
    • Neurons on this layer do the major computation to find hidden data patterns and learn from them
    • There can be multiple hidden layers from 1 to 100 and many more
    • More the Hidden layers, complex the DL model and more time it takes to train
  • Output Layer
    • This is where we can collect or see the results from model

• Activation Functions used in hidden layer:
  • sigmoid
  • ReLU [Rectified Linear Unit]
  • Leaky ReLU
  • softmax

---

1. DL using Python [_^via sklearn package]

#Python classes for Deep Learning

from sklearn.neural_network import MLPRegressor, MLPClassifier


#creating an object of MLPRegressor class

reg = MLPRegressor(
hidden_layer_sizes=(4,5,),
activation="relu", 
solver='adam', 
alpha=0.0001, 
batch_size=40, 
learning_rate='adaptive', 
learning_rate_init=0.03, 
random_state=1, 
max_iter=2000, 
verbose=2
)

***hidden_layer_sizes=(4,5,) means that we need 2 hidden layers,  first layer with 4 neurons and second layer with 5 neurons
***In this case the type of hidden layer will be Fully Connected [Dense] layer

#train the DL model

reg.fit(X_train, Y_train)

#test the DL model

Y_predicted = reg.predict(X_test)

#Y_predicted should be compared with Y_test to understand the error and accuracy of the model

***sklearn provides very basic Deep Learning features, so we will now explore keras which is much advanced and goto package for Deep Learning.

---

---

Keras + TensorFlow

Keras is an Open Source high-level neural network library/API that can run on top of TensorFlow [TF]

It provides an easy interface for users to implement deep learning programs to be executed on CPU and GPU

Additional Benefits with TensorFlow [TF]: 

1. TensorBoard is the TensorFlow's visualization toolkit. It is a tool for providing the measurements and visualizations   needed during the machine learning workflow

2. TensorFlow Extended (TFX) is an end-to-end platform for deploying production ML pipelines

3. TensorFlow Serving provides TensorFlow Serving framework for deploying trained models to production, so developers do not need to use Django or Flask as a back-end server

4. TensorFlow Model Analysis (TFMA) is a library for performing model evaluation

• There are other Deep Learning Frameworks similar to Keras:
  1. Theano [Universite de Montreal]
  2. PyTorch [Facebook]
  3. Cuda [Nvidia]
  4. CNTK [Microsoft]
  5. Tensorflow [Open Source by Google]
  6. Caffe, Caffe(2) [Open Source].
• We will explore them in upcoming blog posts.

---

2. DL using Python [^_via keras and tensorflow package]

#Load python package dependencies
#here keras will be using TensorFlow as backend for better user experience

from keras.models import Sequential
from keras.layers import Dense

#some places you can see below python packages for the same

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Set the input shape i.e. input layer

#shape of input layer, here its 10 neurons in input layer
input_shape = (10,)

print(f'Feature shape: {input_shape}')

# Create the sequential keras model

model = Sequential()


#adding layers to the DL model

#first hidden layer with 16 neurons, mentioning the input layer as well
model.add(Dense(16, activation='relu', input_shape=input_shape))

#Second hidden layer with 8 neurons
model.add(Dense(8, activation='relu'))

#output layer with 1 neuron for regression predicted value
model.add(Dense(1, activation='softmax'))

A Sequential model is appropriate for a plain stack of layers where each layer has exactly one input tensor and one output tensor.

A Sequential model is not appropriate when:

1. Your model has multiple inputs or multiple outputs
2. Any of your layers has multiple inputs or multiple outputs
3. You need to do layer sharing
4. You want non-linear topology (e.g. a residual connection, a multi-branch model)

Types of Hidden Layers:
1. Dense Layer 
    + Fully Connected Layer
2. Convolutional Layer
    + Conv1D
    + Conv2D
    + Conv3D
3. Pooling Layer
    + MinPooling
    + MaxPooling
    + AveragePooling
4. Flatten Layer

# Preparing the Dataset i.e. dividing into 4 parts as per classical ML approach

x_train, y_train
x_test, y_test

# Configure the model 


model.compile(
optimizer='adam', 
loss='mean_absolute_error', 
metrics=['mean_squared_error']
)

# Start model training phase

model.fit(
x_train, 
y_train, 
batch_size=442, 
validation_split=0.2,
epochs=100,  
verbose=1
)

MODEL EXPLAINABILITY

#checking model summary i.e. layers and associated weights at each synapse per layer

#model.inputs
#model.layers 
#model.outputs 

model.summary()

EXPLANATION:

The above chart depicts that the model contains 3 layers [2 hidden + 1 output]. 
Remember that Input Layer do not have any weights and hence not represented here

Param represents number of weights/coefficients learned in each connected layer.

Overall this 3 layers DNN model had learned/adjusted values for 321 weights.

So you can understand that number of weights learned by DNN model increased exponentially as you increase the number of layers. 

So properly analyze the problem statement and its complexity before increasing the number of hidden layers

---