1. Python
  2. ML | Logistic regression using Tensorflow

ML | Logistic regression using Tensorflow



Brief description of logistic regression:
Logistic regression — it is a classification algorithm commonly used in machine learning. This allows you to classify data into distinct classes by examining relationships from a given set of labeled data. It examines linear relationships from a given dataset and then introduces nonlinearity as a sigmoidal function.

In the case of logistic regression, the hypothesis is a straight line sigmoid, i.e.
where
Where is the vector w represents the weights, and the scalar b represents the offset of the model.
Let`s render the sigmoid function —

import numpy as np

import matplotlib.pyplot as plt

 

def sigmoid (z):

return 1 / ( 1 + np.exp ( - z))

 

plt.plot (np.arange ( - 5 , 5 , 0.1 ), sigmoid (np.arange ( - 5 , 5 , 0.1 )))

plt.title ( `Visualization of the Sigmoid Function` )

 
plt.show ()

Output:

Note that the range of the Sigmoid function is (0, 1), which means that the resulting values ​​are between 0 and 1. This property of the Sigmoid function makes it a really good choice of activation function for binary classification. Also for z = 0, Sigmoid (z) = 0.5 which is the midpoint of the range of the sigmoidal function.

As with linear regression, we need to find the optimal values ​​for w and b, for which the cost function J is minimal. In this case, we will use the Sigmoid Cross Entropy cost function which is defined

This cost function will then be optimized with using gradient descent.

Implementation:
Let`s start by importing the required libraries. We`ll be using Numpy along with Tensorflow for calculations, Pandas for basic data analysis, and Matplotlib for plotting. We will also use the Scikit-Learn preprocessor module for One Hot Encoding data.

# module import

import numpy as np

import pandas as pd

import tensorflow as tf

import matplotlib.pyplot as plt

from sklearn.preprocessing import OneHotEncoder

Next we will import dataset . We will use a subset of the well-known Iris dataset .

data = pd.read_csv ( `dataset. csv` , header = None )

print ( "Data Shape:" , data.shape)

 

print (data.head ())

Exit:

 Data Shape: (100, 4) 0 1 2 3 0 0 5.1 3.5 1 1 1 4.9 3.0 1 2 2 4.7 3.2 1 3 3 4.6 3.1 1 4 4 5.0 3.6 1

Now let`s semi read the matrix of objects and the corresponding labels and render.

# Feature Matrix

x_orig = data.iloc [:, 1 : - 1 ]. values ​​

 
# Data Labels

y_orig = data.iloc [:, - 1 :]. values ​​

 

print ( "Shape of Feature Matrix : " , x_orig.shape)

print ( "Shape Label Vector:" , y_orig.shape)

Output :

 Shape of Feature Matrix: (100, 2) Shape Label Vector: (100, 1)

Render the data to the data.

Amount of positive data

x_pos = np.array ([x_orig [i] for i in range ( len (x_orig))

if y_orig [i] = = 1 ])

 
# Negative data points

x_neg = np.array ([x_orig [i] for i in range ( len (x_orig))

if y_orig [i] = = 0 ])

 
# Plot positive data points

plt.scatter (x_pos [:, 0 ], x_pos [:, 1 ], color = ` blue` , label = `Positive` )

  
# Plot negative data points

plt.scatter (x_neg [:, 0 ], x_neg [:, 1 ], color = `red` , label = `Negative` )

  

plt.xlabel ( ` Feature 1` )

plt.ylabel ( `Feature 2` )

plt.title ( `Plot of given data` )

plt.legend ()

 
plt.show ()

,

Now we will be One Hot Encoding for the data to work with the algorithm. One hot coding converts categorical features into a format that works better with classification and regression algorithms. We will also set the learning rate and the number of epochs.

# Create One Hot Encoder

oneHot = OneHotEncoder ()

 
# x_orig encoding
oneHot.fit (x_orig)

x = oneHot.transform (x_orig) .toarray ()

 
# y_orig encoding
oneHot.fit (y_orig)

y = oneHot.transform (y_orig) .toarray ()

 

alpha, epochs = 0.0035 , 500

m, n = x.shape

print ( `m = ` , m)

print ( `n =` , n)

print ( `Learning Rate =` , alpha)

print ( `Number of Epochs =` , epochs)

Exit :

 m = 100 n = 7 Learning Rate = 0.0035 Number of Epochs = 500

Now we will start creating a model with a subdividing the X and Y placeholders so that we can feed our tutorials x and y to the optimizer as we learn. We will also create W and b trainable variables that can be optimized with the Gradient Descent Optimizer.

# There are n columns in the object matrix
# after one hot encoding.

X = tf.placeholder (tf.float32, [ None , n])

 
# Since this is a binary classification problem,
# Y can only take 2 values.

Y = tf.placeholder (tf.float32, [ None , 2 ])

 
# Learning weight variables

W = tf.Variable (tf.zeros ([n, 2 ]))

 
# Learning variable offset

b = tf.Variable (tf.zeros ([ 2 ]))

Now declare hypothesis, cost function, optimizer and an initializer for global variables.

# Hypothesis

Y_hat = tf.nn.sigmoid (tf .add (tf.matmul (X, W), b))

 
# Sigmoid Cross Entropy Cost Function

cost = tf.nn.sigmoid_cross_entropy_with_logits (

  logits = Y_hat, labels = Y)

 
# Gradient Descent Optimizer

optimizer = tf.train.GradientDescentOptimizer (

learning_rate = alpha) .minimize ( cost)

 
# Global Variables Initializer

init = tf.global_variables_initializer ()

Start the tutorial process within a Tensorflow session.

# Start a Tensorflow session
with tf.Session () as sess:

 

# Initializing variables

sess.run (init)

 

# Lists for storing varying value and accuracy at each epoch

cost_history, accuracy_history = [], []

 

# Loop over all eras

for epoch in range (epochs):

cost_per_epoch = 0

 

# Launching the Optimizer

sess.run (optimizer, feed_dict = {X: x, Y: y})

 

  # Calculate cost for the current era

c = sess.run (cost, feed_dict = {X: x, Y: y})

  

# Calculate accuracy for the current epoch

correct_prediction = tf.equal (tf.argmax (Y_hat , 1 ),

tf.argmax (Y, 1 ))

accuracy = tf.re duce_mean (tf.cast (correct_prediction,

tf.float32))

 

# Save cost and accuracy in history

cost_history.append ( sum ( sum (c)))

accuracy_history.append (accuracy. eval ({X: x, Y: y}) * 100 )

  

  # Display the result in the current era

if epoch % 100 = = 0 and epoch! = 0 :

print ( "Epoch" + str (epoch) + " Cost: "

  + str (cost_history [ - 1 ]))

 

Weight = sess.run (W) # Optimized weight

Bias = sess.run (b)  # Optimized slope

 

# Final precision

correct_prediction = tf.equal (tf.argmax (Y_hat, 1 ),

  tf.argmax (Y, 1 ))

accuracy = tf.reduce_mean (tf.cast (correct_prediction, 

tf.float32))

  print ( " Accuracy: " , accuracy_history [ - 1 ], "%" )

Output: 

 Epoch 100 Cost: 125.700202942 Epoch 200 Cost: 120.647117615 Epoch 300 Cost: 118.151592255 Epoch 400 Cost: 116.549999237 Accuracy: 91.0000026226%

Let`s outline the cost change during eras.

plt.plot ( list ( range (epochs)), cost_history)

plt.xlabel ( `Epochs` )

plt.ylabel ( `Cost` )

plt.title ( `Decrease in Cost with Epochs` )

 
plt.show ()

Graph precision changes by epoch.

plt.plot ( list ( range (epochs)), accuracy_history)

plt.xlabel ( `Epochs` )

plt.ylabel ( `Accuracy` )

plt.title ( `Increase in Accuracy with Epochs` )

 
plt.show ()

We will now build a Decision Boundary for our trained classifier. Decision boundary — it is a hypersurface that divides the base vector space into two sets, one for each class.

# Calculate decision boundary

decision_boundary_x = np.array ([np. min (x_orig [:, 0 ]),

np. max (x_orig [:, 0 ])])

 

decision_boundary_y = ( - 1.0 / Weight [ 0 ]) *

(decision_boundary_x * Weight + Bias)

 

decision_boundary_y = [ sum (decision_boundary_y [:, 0 ]), 

sum (decision_boundary_y [:, 1 ])]

 
Amount of positive data

x_pos = np.array ([x_orig [i] for i in range ( len (x_orig))

if y_orig [i] = = 1 ])

 
# Negative data points

x_neg = np.array ([x_orig [i] for i in range ( len (x_orig)) 

  if y_orig [i] = = 0 ])

 
# Plot positive data points

plt.scatter (x_pos [:, 0 ], x_pos [:, 1 ],

color = `blue` , label = ` Positive` )

 
# Plot negative data points

plt.scatter (x_neg [:, 0 ], x_neg [:, 1 ],

color = `red` , label = ` Negative` )

 
# Building a decision boundary
plt.plot (decision_boundary_x, decision_boundary_y)

plt.xlabel ( `Feature 1` )

plt. ylabel ( `Feature 2` )

plt.title ( `Plot of Decision Boundary` )

plt.legend ()

  
plt.show ()

code class = "keyword"> = = 0 ])

 
# Plot positive data points

plt.scatter (x_pos [:, 0 ], x_pos [:, 1 ],

color = `blue` , label =