Lesson 03 - Artificial Neural Networks

The following topics are discussed in this notebook:

  • The architecture of an artificial neural network.
  • Building neural networks in keras.

Additional Resources

  • Deep Learning with Python, Section 3.4, Pages 68 - 77
In [1]:
import numpy as np
import matplotlib.pyplot as plt

As we saw in the previous lesson, logistic regression models (which consist of a single artifial neuron) always result in a linear decision boundary. While that might be appropriate for some problems, there are many problems for which a linear decision boundary will be insufficient. For those types of problems, we will need a more complex model. This can be achieved by stacking individual neurons together into an artificial neural network.

An ANN consists of many layers of neurons, which are processed in a specific order. There are three types of layers in an ANN: The input layer, one or more hidden layers, and an output layer.

  • The first layer, called the input layer, consist of the features being fed into the model. It contains one node for each feature being used, plus a single bias node.

  • The output of each node in the input layer is sent to each (non-bias) node in the first hidden layer. Every connection between any two pair of nodes in these two layers will have its own weight. The nodes in the first hidden layer process the inputs it recieves from the input layer, and send their outputs downstream to any subsequent hidden layers.

  • The last hidden layer will send its output to an output layer. This output layer could contain one or more neurons, depending on the type of task that the network is being used for.

We can build a neural network with as many hidden layers as we wish, and each hidden layer can have as many nodes as we would like. As a general rule, the more neurons that there are in a model, the more flexible that model is.

NNet

Example 1

The first neural network that we will build will be to tackle the following classification problem.

In [2]:
np.random.seed(887)
X = np.random.uniform(0,1,[300,2])
pB = 40*np.abs(X[:,0] - X[:,1])**4
pR = np.random.uniform(0,1,300)
col = np.where(pB < pR, 'r', 'b')
y = np.where(pB < pR, 0, 1)

plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=col, edgecolor='k', s=60)
plt.show()

Network Architecture

We will use keras to build a neural network with one hidden layer, containing four (non-bias) nodes.

Net01

Import Keras Modules

We will be using the Keras package to construct neural networks in this course. Keras provides a high-level API for building neural networks. It is built on top of TensorFlow, which is a library created by Google to efficiently perform matrix calculations.

We will now import some Keras modules that we will need to build our network.

In [3]:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import Adam
from tensorflow import set_random_seed
Using TensorFlow backend.

Construct the Network

We will now specify our network architecture by creating a class of type Sequential to represent our model. We will then add the necessary layers to our model.

In [4]:
np.random.seed(1)
set_random_seed(1)

model = Sequential()
model.add(Dense(4, input_shape=(2,), activation='sigmoid'))
model.add(Dense(1, activation='sigmoid'))

We can use the summary method to view a description of our model.

In [5]:
model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 4)                 12        
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 5         
=================================================================
Total params: 17
Trainable params: 17
Non-trainable params: 0
_________________________________________________________________

Compile the Model

Before we train the model, we need to specify what loss function to minimize, as well as what optimization method we will use to (hopefully) achieve that minimization. We can do this with the compile method of the Sequential class. We can also use compile to specify useful metrics other than the loss to be reported during training.

In [6]:
opt = Adam(lr=0.1)
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])

Training the Model

We will now train the model using the fit method. We specify values for the following parameters in the fit method below:

  • batch_size refers to the number of samples to calculate the loss on at one time. The weights will be updated after each batch. We will include the entire training set as our batch, but there can be advantages to using smaller batches. A single training loop will consider every sample in the data set, even if we are using multiple batches.
  • epochs is the number of training loops to perform.
  • verbose determines the amount of information displayed while training. If verbose=0, then no ouput is displayed. If verbose=2, then loss and any requested metrics are shown after each epoch. Setting verbose=1 is similar to verbose=2, except that you will see a progress bar as well as execution time for each epoch.
In [7]:
h = model.fit(X, y, batch_size=300, epochs=200, verbose=2)
Epoch 1/200
 - 2s - loss: 0.6969 - acc: 0.3967
Epoch 2/200
 - 0s - loss: 0.6921 - acc: 0.6933
Epoch 3/200
 - 0s - loss: 0.6963 - acc: 0.5200
Epoch 4/200
 - 0s - loss: 0.6934 - acc: 0.6100
Epoch 5/200
 - 0s - loss: 0.6910 - acc: 0.6300
Epoch 6/200
 - 0s - loss: 0.6926 - acc: 0.3767
Epoch 7/200
 - 0s - loss: 0.6936 - acc: 0.3000
Epoch 8/200
 - 0s - loss: 0.6920 - acc: 0.3900
Epoch 9/200
 - 0s - loss: 0.6907 - acc: 0.6033
Epoch 10/200
 - 0s - loss: 0.6910 - acc: 0.6900
Epoch 11/200
 - 0s - loss: 0.6918 - acc: 0.6500
Epoch 12/200
 - 0s - loss: 0.6914 - acc: 0.6700
Epoch 13/200
 - 0s - loss: 0.6903 - acc: 0.6933
Epoch 14/200
 - 0s - loss: 0.6897 - acc: 0.6167
Epoch 15/200
 - 0s - loss: 0.6900 - acc: 0.4933
Epoch 16/200
 - 0s - loss: 0.6901 - acc: 0.4400
Epoch 17/200
 - 0s - loss: 0.6895 - acc: 0.4767
Epoch 18/200
 - 0s - loss: 0.6886 - acc: 0.5767
Epoch 19/200
 - 0s - loss: 0.6881 - acc: 0.6533
Epoch 20/200
 - 0s - loss: 0.6880 - acc: 0.6933
Epoch 21/200
 - 0s - loss: 0.6877 - acc: 0.7067
Epoch 22/200
 - 0s - loss: 0.6869 - acc: 0.6967
Epoch 23/200
 - 0s - loss: 0.6859 - acc: 0.6633
Epoch 24/200
 - 0s - loss: 0.6852 - acc: 0.6067
Epoch 25/200
 - 0s - loss: 0.6846 - acc: 0.5700
Epoch 26/200
 - 0s - loss: 0.6837 - acc: 0.5500
Epoch 27/200
 - 0s - loss: 0.6824 - acc: 0.6000
Epoch 28/200
 - 0s - loss: 0.6811 - acc: 0.6400
Epoch 29/200
 - 0s - loss: 0.6799 - acc: 0.6833
Epoch 30/200
 - 0s - loss: 0.6785 - acc: 0.7000
Epoch 31/200
 - 0s - loss: 0.6768 - acc: 0.7000
Epoch 32/200
 - 0s - loss: 0.6749 - acc: 0.6767
Epoch 33/200
 - 0s - loss: 0.6728 - acc: 0.6533
Epoch 34/200
 - 0s - loss: 0.6707 - acc: 0.6267
Epoch 35/200
 - 0s - loss: 0.6683 - acc: 0.6267
Epoch 36/200
 - 0s - loss: 0.6655 - acc: 0.6567
Epoch 37/200
 - 0s - loss: 0.6626 - acc: 0.6733
Epoch 38/200
 - 0s - loss: 0.6595 - acc: 0.6833
Epoch 39/200
 - 0s - loss: 0.6562 - acc: 0.6867
Epoch 40/200
 - 0s - loss: 0.6526 - acc: 0.6867
Epoch 41/200
 - 0s - loss: 0.6487 - acc: 0.6867
Epoch 42/200
 - 0s - loss: 0.6448 - acc: 0.6733
Epoch 43/200
 - 0s - loss: 0.6406 - acc: 0.6767
Epoch 44/200
 - 0s - loss: 0.6363 - acc: 0.6833
Epoch 45/200
 - 0s - loss: 0.6317 - acc: 0.6900
Epoch 46/200
 - 0s - loss: 0.6271 - acc: 0.6933
Epoch 47/200
 - 0s - loss: 0.6223 - acc: 0.6933
Epoch 48/200
 - 0s - loss: 0.6174 - acc: 0.6967
Epoch 49/200
 - 0s - loss: 0.6124 - acc: 0.6967
Epoch 50/200
 - 0s - loss: 0.6072 - acc: 0.6933
Epoch 51/200
 - 0s - loss: 0.6020 - acc: 0.6967
Epoch 52/200
 - 0s - loss: 0.5966 - acc: 0.7067
Epoch 53/200
 - 0s - loss: 0.5910 - acc: 0.7100
Epoch 54/200
 - 0s - loss: 0.5851 - acc: 0.7167
Epoch 55/200
 - 0s - loss: 0.5789 - acc: 0.7300
Epoch 56/200
 - 0s - loss: 0.5722 - acc: 0.7333
Epoch 57/200
 - 0s - loss: 0.5648 - acc: 0.7367
Epoch 58/200
 - 0s - loss: 0.5570 - acc: 0.7367
Epoch 59/200
 - 0s - loss: 0.5488 - acc: 0.7367
Epoch 60/200
 - 0s - loss: 0.5402 - acc: 0.7333
Epoch 61/200
 - 0s - loss: 0.5312 - acc: 0.7333
Epoch 62/200
 - 0s - loss: 0.5217 - acc: 0.7333
Epoch 63/200
 - 0s - loss: 0.5117 - acc: 0.7367
Epoch 64/200
 - 0s - loss: 0.5009 - acc: 0.7433
Epoch 65/200
 - 0s - loss: 0.4895 - acc: 0.7667
Epoch 66/200
 - 0s - loss: 0.4776 - acc: 0.7867
Epoch 67/200
 - 0s - loss: 0.4653 - acc: 0.7900
Epoch 68/200
 - 0s - loss: 0.4528 - acc: 0.8100
Epoch 69/200
 - 0s - loss: 0.4399 - acc: 0.8367
Epoch 70/200
 - 0s - loss: 0.4268 - acc: 0.8533
Epoch 71/200
 - 0s - loss: 0.4135 - acc: 0.8600
Epoch 72/200
 - 0s - loss: 0.4003 - acc: 0.8733
Epoch 73/200
 - 0s - loss: 0.3873 - acc: 0.8767
Epoch 74/200
 - 0s - loss: 0.3746 - acc: 0.8900
Epoch 75/200
 - 0s - loss: 0.3620 - acc: 0.9033
Epoch 76/200
 - 0s - loss: 0.3496 - acc: 0.9100
Epoch 77/200
 - 0s - loss: 0.3376 - acc: 0.9200
Epoch 78/200
 - 0s - loss: 0.3261 - acc: 0.9367
Epoch 79/200
 - 0s - loss: 0.3151 - acc: 0.9433
Epoch 80/200
 - 0s - loss: 0.3045 - acc: 0.9433
Epoch 81/200
 - 0s - loss: 0.2944 - acc: 0.9467
Epoch 82/200
 - 0s - loss: 0.2849 - acc: 0.9467
Epoch 83/200
 - 0s - loss: 0.2759 - acc: 0.9467
Epoch 84/200
 - 0s - loss: 0.2675 - acc: 0.9433
Epoch 85/200
 - 0s - loss: 0.2595 - acc: 0.9467
Epoch 86/200
 - 0s - loss: 0.2521 - acc: 0.9500
Epoch 87/200
 - 0s - loss: 0.2451 - acc: 0.9467
Epoch 88/200
 - 0s - loss: 0.2387 - acc: 0.9500
Epoch 89/200
 - 0s - loss: 0.2327 - acc: 0.9500
Epoch 90/200
 - 0s - loss: 0.2271 - acc: 0.9500
Epoch 91/200
 - 0s - loss: 0.2219 - acc: 0.9500
Epoch 92/200
 - 0s - loss: 0.2170 - acc: 0.9467
Epoch 93/200
 - 0s - loss: 0.2126 - acc: 0.9467
Epoch 94/200
 - 0s - loss: 0.2084 - acc: 0.9467
Epoch 95/200
 - 0s - loss: 0.2045 - acc: 0.9467
Epoch 96/200
 - 0s - loss: 0.2010 - acc: 0.9433
Epoch 97/200
 - 0s - loss: 0.1977 - acc: 0.9433
Epoch 98/200
 - 0s - loss: 0.1946 - acc: 0.9433
Epoch 99/200
 - 0s - loss: 0.1917 - acc: 0.9433
Epoch 100/200
 - 0s - loss: 0.1890 - acc: 0.9433
Epoch 101/200
 - 0s - loss: 0.1865 - acc: 0.9433
Epoch 102/200
 - 0s - loss: 0.1842 - acc: 0.9433
Epoch 103/200
 - 0s - loss: 0.1820 - acc: 0.9433
Epoch 104/200
 - 0s - loss: 0.1800 - acc: 0.9433
Epoch 105/200
 - 0s - loss: 0.1781 - acc: 0.9433
Epoch 106/200
 - 0s - loss: 0.1763 - acc: 0.9433
Epoch 107/200
 - 0s - loss: 0.1746 - acc: 0.9467
Epoch 108/200
 - 0s - loss: 0.1731 - acc: 0.9467
Epoch 109/200
 - 0s - loss: 0.1716 - acc: 0.9467
Epoch 110/200
 - 0s - loss: 0.1702 - acc: 0.9467
Epoch 111/200
 - 0s - loss: 0.1689 - acc: 0.9467
Epoch 112/200
 - 0s - loss: 0.1677 - acc: 0.9467
Epoch 113/200
 - 0s - loss: 0.1665 - acc: 0.9467
Epoch 114/200
 - 0s - loss: 0.1655 - acc: 0.9467
Epoch 115/200
 - 0s - loss: 0.1644 - acc: 0.9467
Epoch 116/200
 - 0s - loss: 0.1635 - acc: 0.9467
Epoch 117/200
 - 0s - loss: 0.1625 - acc: 0.9467
Epoch 118/200
 - 0s - loss: 0.1617 - acc: 0.9467
Epoch 119/200
 - 0s - loss: 0.1608 - acc: 0.9467
Epoch 120/200
 - 0s - loss: 0.1601 - acc: 0.9500
Epoch 121/200
 - 0s - loss: 0.1593 - acc: 0.9500
Epoch 122/200
 - 0s - loss: 0.1586 - acc: 0.9500
Epoch 123/200
 - 0s - loss: 0.1579 - acc: 0.9500
Epoch 124/200
 - 0s - loss: 0.1573 - acc: 0.9500
Epoch 125/200
 - 0s - loss: 0.1567 - acc: 0.9500
Epoch 126/200
 - 0s - loss: 0.1561 - acc: 0.9500
Epoch 127/200
 - 0s - loss: 0.1556 - acc: 0.9500
Epoch 128/200
 - 0s - loss: 0.1551 - acc: 0.9500
Epoch 129/200
 - 0s - loss: 0.1546 - acc: 0.9500
Epoch 130/200
 - 0s - loss: 0.1541 - acc: 0.9500
Epoch 131/200
 - 0s - loss: 0.1536 - acc: 0.9500
Epoch 132/200
 - 0s - loss: 0.1532 - acc: 0.9500
Epoch 133/200
 - 0s - loss: 0.1528 - acc: 0.9500
Epoch 134/200
 - 0s - loss: 0.1524 - acc: 0.9533
Epoch 135/200
 - 0s - loss: 0.1520 - acc: 0.9533
Epoch 136/200
 - 0s - loss: 0.1517 - acc: 0.9533
Epoch 137/200
 - 0s - loss: 0.1513 - acc: 0.9533
Epoch 138/200
 - 0s - loss: 0.1510 - acc: 0.9533
Epoch 139/200
 - 0s - loss: 0.1507 - acc: 0.9533
Epoch 140/200
 - 0s - loss: 0.1504 - acc: 0.9533
Epoch 141/200
 - 0s - loss: 0.1501 - acc: 0.9533
Epoch 142/200
 - 0s - loss: 0.1498 - acc: 0.9533
Epoch 143/200
 - 0s - loss: 0.1495 - acc: 0.9500
Epoch 144/200
 - 0s - loss: 0.1492 - acc: 0.9500
Epoch 145/200
 - 0s - loss: 0.1490 - acc: 0.9500
Epoch 146/200
 - 0s - loss: 0.1487 - acc: 0.9500
Epoch 147/200
 - 0s - loss: 0.1485 - acc: 0.9500
Epoch 148/200
 - 0s - loss: 0.1483 - acc: 0.9500
Epoch 149/200
 - 0s - loss: 0.1481 - acc: 0.9500
Epoch 150/200
 - 0s - loss: 0.1478 - acc: 0.9500
Epoch 151/200
 - 0s - loss: 0.1476 - acc: 0.9500
Epoch 152/200
 - 0s - loss: 0.1474 - acc: 0.9500
Epoch 153/200
 - 0s - loss: 0.1472 - acc: 0.9500
Epoch 154/200
 - 0s - loss: 0.1471 - acc: 0.9500
Epoch 155/200
 - 0s - loss: 0.1469 - acc: 0.9500
Epoch 156/200
 - 0s - loss: 0.1467 - acc: 0.9500
Epoch 157/200
 - 0s - loss: 0.1465 - acc: 0.9500
Epoch 158/200
 - 0s - loss: 0.1464 - acc: 0.9533
Epoch 159/200
 - 0s - loss: 0.1462 - acc: 0.9533
Epoch 160/200
 - 0s - loss: 0.1460 - acc: 0.9533
Epoch 161/200
 - 0s - loss: 0.1459 - acc: 0.9533
Epoch 162/200
 - 0s - loss: 0.1457 - acc: 0.9533
Epoch 163/200
 - 0s - loss: 0.1456 - acc: 0.9533
Epoch 164/200
 - 0s - loss: 0.1454 - acc: 0.9533
Epoch 165/200
 - 0s - loss: 0.1453 - acc: 0.9533
Epoch 166/200
 - 0s - loss: 0.1452 - acc: 0.9533
Epoch 167/200
 - 0s - loss: 0.1450 - acc: 0.9533
Epoch 168/200
 - 0s - loss: 0.1449 - acc: 0.9533
Epoch 169/200
 - 0s - loss: 0.1448 - acc: 0.9533
Epoch 170/200
 - 0s - loss: 0.1447 - acc: 0.9533
Epoch 171/200
 - 0s - loss: 0.1445 - acc: 0.9533
Epoch 172/200
 - 0s - loss: 0.1444 - acc: 0.9533
Epoch 173/200
 - 0s - loss: 0.1443 - acc: 0.9533
Epoch 174/200
 - 0s - loss: 0.1442 - acc: 0.9533
Epoch 175/200
 - 0s - loss: 0.1441 - acc: 0.9533
Epoch 176/200
 - 0s - loss: 0.1440 - acc: 0.9533
Epoch 177/200
 - 0s - loss: 0.1439 - acc: 0.9533
Epoch 178/200
 - 0s - loss: 0.1438 - acc: 0.9533
Epoch 179/200
 - 0s - loss: 0.1437 - acc: 0.9533
Epoch 180/200
 - 0s - loss: 0.1436 - acc: 0.9533
Epoch 181/200
 - 0s - loss: 0.1435 - acc: 0.9533
Epoch 182/200
 - 0s - loss: 0.1434 - acc: 0.9533
Epoch 183/200
 - 0s - loss: 0.1433 - acc: 0.9533
Epoch 184/200
 - 0s - loss: 0.1432 - acc: 0.9533
Epoch 185/200
 - 0s - loss: 0.1431 - acc: 0.9533
Epoch 186/200
 - 0s - loss: 0.1430 - acc: 0.9533
Epoch 187/200
 - 0s - loss: 0.1429 - acc: 0.9533
Epoch 188/200
 - 0s - loss: 0.1428 - acc: 0.9533
Epoch 189/200
 - 0s - loss: 0.1427 - acc: 0.9533
Epoch 190/200
 - 0s - loss: 0.1427 - acc: 0.9533
Epoch 191/200
 - 0s - loss: 0.1426 - acc: 0.9533
Epoch 192/200
 - 0s - loss: 0.1425 - acc: 0.9533
Epoch 193/200
 - 0s - loss: 0.1424 - acc: 0.9533
Epoch 194/200
 - 0s - loss: 0.1423 - acc: 0.9533
Epoch 195/200
 - 0s - loss: 0.1423 - acc: 0.9533
Epoch 196/200
 - 0s - loss: 0.1422 - acc: 0.9533
Epoch 197/200
 - 0s - loss: 0.1421 - acc: 0.9533
Epoch 198/200
 - 0s - loss: 0.1421 - acc: 0.9533
Epoch 199/200
 - 0s - loss: 0.1420 - acc: 0.9533
Epoch 200/200
 - 0s - loss: 0.1419 - acc: 0.9533

Analyzing Training Process

The output of the fit method contains a history attribute that we can use to visualize the changes in loss and accuracy during the training process.

In [8]:
plt.rcParams["figure.figsize"] = [8,4]
plt.subplot(1,2,1)
plt.plot(h.history['acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')

plt.subplot(1,2,2)
plt.plot(h.history['loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.show()

Visualiztion the Decision Region

I have written a function called plot_regions that can be used to plot the decision regions for classification problems with two features.

In [9]:
from ClassificationPlotter import plot_regions

plot_regions(model, X, y, cmap='bwr_r', fig_size=(8,6), keras=True)

The gif below shows how the classification regions changed during the process of training.

Net01

Making Predictions

We can used the predict and predict_classes methods of our model to generate predictions for new observations.

In [10]:
Xnew = np.array([[0.2, 0.4], [0.2, 0.5], [0.2,0.6]])

np.set_printoptions(precision=6, suppress=True)

print('Estimated probabilities of being blue:\n', model.predict(Xnew))
print()
print('Predicted classes:\n', model.predict_classes(Xnew))
Estimated probabilities of being blue:
 [[0.028446]
 [0.207942]
 [0.835405]]

Predicted classes:
 [[0]
 [0]
 [1]]
In [11]:
plot_regions(model, X, y, cmap='bwr', fig_size=(8,6), keras=True, display=False)
plt.scatter(Xnew[:,0], Xnew[:,1], marker="D", c='lime', edgecolor='k', s=100, zorder=3)
plt.show()

Example 2

We will now construct a neural network with a more complicated architecture to address the following classification problem.

In [12]:
from sklearn.datasets import make_circles

np.random.seed(1)
plt.figure(figsize=(8,6))
Z, w = make_circles(500, factor=0.25, noise=0.12)
w = (w+1)%2
plt.scatter(Z[:,0], Z[:,1], c=w, cmap='bwr_r', edgecolor='k')
plt.show()

Network Architecture

The network we will use for this problem will have two hidden layers, each containing 8 (non-bias) neurons.

Net01

Build, Compile, and Train the Network

We will now perform the same steps as before to build, compile, and train the network.

In [13]:
np.random.seed(1)
set_random_seed(1)

model2 = Sequential()
model2.add(Dense(8, input_shape=(2,), activation='sigmoid'))
model2.add(Dense(8, activation='sigmoid'))
model2.add(Dense(1, activation='sigmoid'))
In [14]:
model2.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_3 (Dense)              (None, 8)                 24        
_________________________________________________________________
dense_4 (Dense)              (None, 8)                 72        
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 9         
=================================================================
Total params: 105
Trainable params: 105
Non-trainable params: 0
_________________________________________________________________
In [15]:
opt = Adam(lr=0.1)
model2.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
In [16]:
h = model2.fit(Z, w, batch_size=500, epochs=50, verbose=2)
Epoch 1/50
 - 0s - loss: 0.7145 - acc: 0.5000
Epoch 2/50
 - 0s - loss: 0.6981 - acc: 0.5000
Epoch 3/50
 - 0s - loss: 0.7102 - acc: 0.5000
Epoch 4/50
 - 0s - loss: 0.6972 - acc: 0.5000
Epoch 5/50
 - 0s - loss: 0.6937 - acc: 0.5000
Epoch 6/50
 - 0s - loss: 0.6999 - acc: 0.5000
Epoch 7/50
 - 0s - loss: 0.7004 - acc: 0.5000
Epoch 8/50
 - 0s - loss: 0.6954 - acc: 0.5000
Epoch 9/50
 - 0s - loss: 0.6923 - acc: 0.6380
Epoch 10/50
 - 0s - loss: 0.6937 - acc: 0.5000
Epoch 11/50
 - 0s - loss: 0.6957 - acc: 0.5000
Epoch 12/50
 - 0s - loss: 0.6949 - acc: 0.5000
Epoch 13/50
 - 0s - loss: 0.6920 - acc: 0.5000
Epoch 14/50
 - 0s - loss: 0.6897 - acc: 0.4080
Epoch 15/50
 - 0s - loss: 0.6890 - acc: 0.6000
Epoch 16/50
 - 0s - loss: 0.6889 - acc: 0.5000
Epoch 17/50
 - 0s - loss: 0.6877 - acc: 0.5000
Epoch 18/50
 - 0s - loss: 0.6849 - acc: 0.5080
Epoch 19/50
 - 0s - loss: 0.6810 - acc: 0.6820
Epoch 20/50
 - 0s - loss: 0.6771 - acc: 0.7640
Epoch 21/50
 - 0s - loss: 0.6731 - acc: 0.6140
Epoch 22/50
 - 0s - loss: 0.6682 - acc: 0.5300
Epoch 23/50
 - 0s - loss: 0.6612 - acc: 0.6080
Epoch 24/50
 - 0s - loss: 0.6516 - acc: 0.8060
Epoch 25/50
 - 0s - loss: 0.6397 - acc: 0.8600
Epoch 26/50
 - 0s - loss: 0.6253 - acc: 0.8660
Epoch 27/50
 - 0s - loss: 0.6077 - acc: 0.8660
Epoch 28/50
 - 0s - loss: 0.5851 - acc: 0.8760
Epoch 29/50
 - 0s - loss: 0.5565 - acc: 0.8900
Epoch 30/50
 - 0s - loss: 0.5217 - acc: 0.9040
Epoch 31/50
 - 0s - loss: 0.4808 - acc: 0.9100
Epoch 32/50
 - 0s - loss: 0.4336 - acc: 0.9140
Epoch 33/50
 - 0s - loss: 0.3816 - acc: 0.9280
Epoch 34/50
 - 0s - loss: 0.3283 - acc: 0.9420
Epoch 35/50
 - 0s - loss: 0.2758 - acc: 0.9580
Epoch 36/50
 - 0s - loss: 0.2259 - acc: 0.9760
Epoch 37/50
 - 0s - loss: 0.1821 - acc: 0.9900
Epoch 38/50
 - 0s - loss: 0.1449 - acc: 0.9900
Epoch 39/50
 - 0s - loss: 0.1136 - acc: 0.9900
Epoch 40/50
 - 0s - loss: 0.0879 - acc: 0.9920
Epoch 41/50
 - 0s - loss: 0.0672 - acc: 0.9940
Epoch 42/50
 - 0s - loss: 0.0509 - acc: 0.9960
Epoch 43/50
 - 0s - loss: 0.0388 - acc: 0.9960
Epoch 44/50
 - 0s - loss: 0.0302 - acc: 0.9960
Epoch 45/50
 - 0s - loss: 0.0241 - acc: 0.9960
Epoch 46/50
 - 0s - loss: 0.0197 - acc: 0.9980
Epoch 47/50
 - 0s - loss: 0.0165 - acc: 0.9980
Epoch 48/50
 - 0s - loss: 0.0142 - acc: 0.9980
Epoch 49/50
 - 0s - loss: 0.0126 - acc: 0.9980
Epoch 50/50
 - 0s - loss: 0.0113 - acc: 0.9980

Analyzing Training Process

In [17]:
plt.rcParams["figure.figsize"] = [8,4]
plt.subplot(1,2,1)
plt.plot(h.history['acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')

plt.subplot(1,2,2)
plt.plot(h.history['loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.show()

Visualizing the Decision Region

In [18]:
plot_regions(model2, Z, w, cmap='bwr_r', fig_size=(8,6), keras=True)

Net02

Generating Predictions

In [19]:
Znew = np.array([[0.5, 0.3], [0.5, 0.4], [0.5,0.5]])

np.set_printoptions(precision=6, suppress=True)

print('Estimated probabilities of being red:\n', model2.predict(Znew))
print()
print('Predicted classes:\n', model2.predict_classes(Znew))
Estimated probabilities of being red:
 [[0.235682]
 [0.606608]
 [0.88536 ]]

Predicted classes:
 [[0]
 [1]
 [1]]
In [20]:
plot_regions(model2, Z, w, cmap='bwr_r', fig_size=(8,6), keras=True, display=False)
plt.scatter(Znew[:,0], Znew[:,1], marker="D", c='lime', edgecolor='k', s=60, zorder=3)
plt.show()