Lesson 03 - Artificial Neural Networks

The following topics are discussed in this notebook:

  • The architecture of an artificial neural network.
  • Building neural networks in keras.

Additional Resources

  • Deep Learning with Python, Section 3.4, Pages 68 - 77
In [1]:
import numpy as np
import matplotlib.pyplot as plt

As we saw in the previous lesson, logistic regression models (which consist of a single artifial neuron) always result in a linear decision boundary. While that might be appropriate for some problems, there are many problems for which a linear decision boundary will be insufficient. For those types of problems, we will need a more complex model. This can be achieved by stacking individual neurons together into an artificial neural network.

An ANN consists of many layers of neurons, which are processed in a specific order. There are three types of layers in an ANN: The input layer, one or more hidden layers, and an output layer.

  • The first layer, called the input layer, consist of the features being fed into the model. It contains one node for each feature being used, plus a single bias node.

  • The output of each node in the input layer is sent to each (non-bias) node in the first hidden layer. Every connection between any two pair of nodes in these two layers will have its own weight. The nodes in the first hidden layer process the inputs it recieves from the input layer, and send their outputs downstream to any subsequent hidden layers.

  • The last hidden layer will send its output to an output layer. This output layer could contain one or more neurons, depending on the type of task that the network is being used for.

We can build a neural network with as many hidden layers as we wish, and each hidden layer can have as many nodes as we would like. As a general rule, the more neurons that there are in a model, the more flexible that model is.

NNet

Example 1

The first neural network that we will build will be to tackle the following classification problem.

In [2]:
np.random.seed(887)
X = np.random.uniform(0,1,[300,2])
pB = 40*np.abs(X[:,0] - X[:,1])**4
pR = np.random.uniform(0,1,300)
col = np.where(pB < pR, 'r', 'b')
y = np.where(pB < pR, 0, 1)

plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=col, edgecolor='k', s=60)
plt.show()

Network Architecture

We will use keras to build a neural network with one hidden layer, containing four (non-bias) nodes.

Net01

Import Keras Modules

We will be using the Keras package to construct neural networks in this course. Keras provides a high-level API for building neural networks. It is built on top of TensorFlow, which is a library created by Google to efficiently perform matrix calculations.

We will now import some Keras modules that we will need to build our network.

In [3]:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import Adam
from tensorflow import set_random_seed
Using TensorFlow backend.

Construct the Network

We will now specify our network architecture by creating a class of type Sequential to represent our model. We will then add the necessary layers to our model.

In [4]:
np.random.seed(1)
set_random_seed(1)

model = Sequential()
model.add(Dense(4, input_shape=(2,), activation='sigmoid'))
model.add(Dense(1, activation='sigmoid'))

We can use the summary method to view a description of our model.

In [5]:
model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 4)                 12        
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 5         
=================================================================
Total params: 17
Trainable params: 17
Non-trainable params: 0
_________________________________________________________________

Compile the Model

Before we train the model, we need to specify what loss function to minimize, as well as what optimization method we will use to (hopefully) achieve that minimization. We can do this with the compile method of the Sequential class. We can also use compile to specify useful metrics other than the loss to be reported during training.

In [6]:
opt = Adam(lr=0.1)
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])

Training the Model

We will now train the model using the fit method. We specify values for the following parameters in the fit method below:

  • batch_size refers to the number of samples to calculate the loss on at one time. The weights will be updated after each batch. We will include the entire training set as our batch, but there can be advantages to using smaller batches. A single training loop will consider every sample in the data set, even if we are using multiple batches.
  • epochs is the number of training loops to perform.
  • verbose determines the amount of information displayed while training. If verbose=0, then no ouput is displayed. If verbose=2, then loss and any requested metrics are shown after each epoch. Setting verbose=1 is similar to verbose=2, except that you will see a progress bar as well as execution time for each epoch.
In [7]:
h = model.fit(X, y, batch_size=300, epochs=200, verbose=2)
Epoch 1/200
 - 1s - loss: 0.6969 - acc: 0.3967
Epoch 2/200
 - 0s - loss: 0.6953 - acc: 0.5333
Epoch 3/200
 - 0s - loss: 0.6960 - acc: 0.5200
Epoch 4/200
 - 0s - loss: 0.6917 - acc: 0.6933
Epoch 5/200
 - 0s - loss: 0.6918 - acc: 0.4333
Epoch 6/200
 - 0s - loss: 0.6939 - acc: 0.2833
Epoch 7/200
 - 0s - loss: 0.6929 - acc: 0.3333
Epoch 8/200
 - 0s - loss: 0.6910 - acc: 0.5333
Epoch 9/200
 - 0s - loss: 0.6908 - acc: 0.6967
Epoch 10/200
 - 0s - loss: 0.6918 - acc: 0.6500
Epoch 11/200
 - 0s - loss: 0.6919 - acc: 0.6500
Epoch 12/200
 - 0s - loss: 0.6908 - acc: 0.6933
Epoch 13/200
 - 0s - loss: 0.6899 - acc: 0.6533
Epoch 14/200
 - 0s - loss: 0.6900 - acc: 0.5267
Epoch 15/200
 - 0s - loss: 0.6904 - acc: 0.4267
Epoch 16/200
 - 0s - loss: 0.6900 - acc: 0.4400
Epoch 17/200
 - 0s - loss: 0.6891 - acc: 0.5333
Epoch 18/200
 - 0s - loss: 0.6885 - acc: 0.6367
Epoch 19/200
 - 0s - loss: 0.6883 - acc: 0.6933
Epoch 20/200
 - 0s - loss: 0.6882 - acc: 0.7000
Epoch 21/200
 - 0s - loss: 0.6876 - acc: 0.7133
Epoch 22/200
 - 0s - loss: 0.6867 - acc: 0.6833
Epoch 23/200
 - 0s - loss: 0.6859 - acc: 0.6300
Epoch 24/200
 - 0s - loss: 0.6854 - acc: 0.5867
Epoch 25/200
 - 0s - loss: 0.6848 - acc: 0.5433
Epoch 26/200
 - 0s - loss: 0.6838 - acc: 0.5800
Epoch 27/200
 - 0s - loss: 0.6825 - acc: 0.6200
Epoch 28/200
 - 0s - loss: 0.6814 - acc: 0.6667
Epoch 29/200
 - 0s - loss: 0.6802 - acc: 0.7000
Epoch 30/200
 - 0s - loss: 0.6789 - acc: 0.7033
Epoch 31/200
 - 0s - loss: 0.6772 - acc: 0.6867
Epoch 32/200
 - 0s - loss: 0.6753 - acc: 0.6700
Epoch 33/200
 - 0s - loss: 0.6734 - acc: 0.6400
Epoch 34/200
 - 0s - loss: 0.6714 - acc: 0.6200
Epoch 35/200
 - 0s - loss: 0.6690 - acc: 0.6367
Epoch 36/200
 - 0s - loss: 0.6664 - acc: 0.6567
Epoch 37/200
 - 0s - loss: 0.6636 - acc: 0.6800
Epoch 38/200
 - 0s - loss: 0.6606 - acc: 0.6867
Epoch 39/200
 - 0s - loss: 0.6574 - acc: 0.6900
Epoch 40/200
 - 0s - loss: 0.6539 - acc: 0.6900
Epoch 41/200
 - 0s - loss: 0.6502 - acc: 0.6833
Epoch 42/200
 - 0s - loss: 0.6463 - acc: 0.6733
Epoch 43/200
 - 0s - loss: 0.6422 - acc: 0.6733
Epoch 44/200
 - 0s - loss: 0.6379 - acc: 0.6800
Epoch 45/200
 - 0s - loss: 0.6333 - acc: 0.6933
Epoch 46/200
 - 0s - loss: 0.6286 - acc: 0.6933
Epoch 47/200
 - 0s - loss: 0.6238 - acc: 0.6967
Epoch 48/200
 - 0s - loss: 0.6187 - acc: 0.6967
Epoch 49/200
 - 0s - loss: 0.6133 - acc: 0.6967
Epoch 50/200
 - 0s - loss: 0.6077 - acc: 0.6967
Epoch 51/200
 - 0s - loss: 0.6019 - acc: 0.7000
Epoch 52/200
 - 0s - loss: 0.5957 - acc: 0.7133
Epoch 53/200
 - 0s - loss: 0.5891 - acc: 0.7200
Epoch 54/200
 - 0s - loss: 0.5822 - acc: 0.7267
Epoch 55/200
 - 0s - loss: 0.5749 - acc: 0.7300
Epoch 56/200
 - 0s - loss: 0.5669 - acc: 0.7333
Epoch 57/200
 - 0s - loss: 0.5585 - acc: 0.7333
Epoch 58/200
 - 0s - loss: 0.5496 - acc: 0.7333
Epoch 59/200
 - 0s - loss: 0.5401 - acc: 0.7333
Epoch 60/200
 - 0s - loss: 0.5301 - acc: 0.7333
Epoch 61/200
 - 0s - loss: 0.5195 - acc: 0.7333
Epoch 62/200
 - 0s - loss: 0.5082 - acc: 0.7567
Epoch 63/200
 - 0s - loss: 0.4964 - acc: 0.7667
Epoch 64/200
 - 0s - loss: 0.4842 - acc: 0.7900
Epoch 65/200
 - 0s - loss: 0.4716 - acc: 0.7867
Epoch 66/200
 - 0s - loss: 0.4587 - acc: 0.8000
Epoch 67/200
 - 0s - loss: 0.4455 - acc: 0.8133
Epoch 68/200
 - 0s - loss: 0.4321 - acc: 0.8433
Epoch 69/200
 - 0s - loss: 0.4187 - acc: 0.8633
Epoch 70/200
 - 0s - loss: 0.4054 - acc: 0.8733
Epoch 71/200
 - 0s - loss: 0.3923 - acc: 0.8833
Epoch 72/200
 - 0s - loss: 0.3793 - acc: 0.8933
Epoch 73/200
 - 0s - loss: 0.3665 - acc: 0.9133
Epoch 74/200
 - 0s - loss: 0.3541 - acc: 0.9133
Epoch 75/200
 - 0s - loss: 0.3421 - acc: 0.9200
Epoch 76/200
 - 0s - loss: 0.3305 - acc: 0.9267
Epoch 77/200
 - 0s - loss: 0.3194 - acc: 0.9433
Epoch 78/200
 - 0s - loss: 0.3087 - acc: 0.9433
Epoch 79/200
 - 0s - loss: 0.2986 - acc: 0.9533
Epoch 80/200
 - 0s - loss: 0.2890 - acc: 0.9533
Epoch 81/200
 - 0s - loss: 0.2799 - acc: 0.9533
Epoch 82/200
 - 0s - loss: 0.2714 - acc: 0.9533
Epoch 83/200
 - 0s - loss: 0.2633 - acc: 0.9533
Epoch 84/200
 - 0s - loss: 0.2558 - acc: 0.9533
Epoch 85/200
 - 0s - loss: 0.2488 - acc: 0.9567
Epoch 86/200
 - 0s - loss: 0.2422 - acc: 0.9467
Epoch 87/200
 - 0s - loss: 0.2360 - acc: 0.9467
Epoch 88/200
 - 0s - loss: 0.2303 - acc: 0.9467
Epoch 89/200
 - 0s - loss: 0.2250 - acc: 0.9467
Epoch 90/200
 - 0s - loss: 0.2200 - acc: 0.9467
Epoch 91/200
 - 0s - loss: 0.2154 - acc: 0.9467
Epoch 92/200
 - 0s - loss: 0.2111 - acc: 0.9467
Epoch 93/200
 - 0s - loss: 0.2072 - acc: 0.9467
Epoch 94/200
 - 0s - loss: 0.2035 - acc: 0.9467
Epoch 95/200
 - 0s - loss: 0.2000 - acc: 0.9467
Epoch 96/200
 - 0s - loss: 0.1968 - acc: 0.9467
Epoch 97/200
 - 0s - loss: 0.1939 - acc: 0.9500
Epoch 98/200
 - 0s - loss: 0.1911 - acc: 0.9500
Epoch 99/200
 - 0s - loss: 0.1885 - acc: 0.9500
Epoch 100/200
 - 0s - loss: 0.1861 - acc: 0.9500
Epoch 101/200
 - 0s - loss: 0.1838 - acc: 0.9467
Epoch 102/200
 - 0s - loss: 0.1817 - acc: 0.9467
Epoch 103/200
 - 0s - loss: 0.1798 - acc: 0.9467
Epoch 104/200
 - 0s - loss: 0.1779 - acc: 0.9467
Epoch 105/200
 - 0s - loss: 0.1762 - acc: 0.9467
Epoch 106/200
 - 0s - loss: 0.1746 - acc: 0.9467
Epoch 107/200
 - 0s - loss: 0.1731 - acc: 0.9467
Epoch 108/200
 - 0s - loss: 0.1716 - acc: 0.9467
Epoch 109/200
 - 0s - loss: 0.1703 - acc: 0.9467
Epoch 110/200
 - 0s - loss: 0.1690 - acc: 0.9467
Epoch 111/200
 - 0s - loss: 0.1678 - acc: 0.9500
Epoch 112/200
 - 0s - loss: 0.1667 - acc: 0.9500
Epoch 113/200
 - 0s - loss: 0.1656 - acc: 0.9500
Epoch 114/200
 - 0s - loss: 0.1646 - acc: 0.9500
Epoch 115/200
 - 0s - loss: 0.1637 - acc: 0.9500
Epoch 116/200
 - 0s - loss: 0.1628 - acc: 0.9500
Epoch 117/200
 - 0s - loss: 0.1619 - acc: 0.9500
Epoch 118/200
 - 0s - loss: 0.1611 - acc: 0.9500
Epoch 119/200
 - 0s - loss: 0.1603 - acc: 0.9500
Epoch 120/200
 - 0s - loss: 0.1596 - acc: 0.9500
Epoch 121/200
 - 0s - loss: 0.1589 - acc: 0.9500
Epoch 122/200
 - 0s - loss: 0.1582 - acc: 0.9500
Epoch 123/200
 - 0s - loss: 0.1576 - acc: 0.9500
Epoch 124/200
 - 0s - loss: 0.1570 - acc: 0.9500
Epoch 125/200
 - 0s - loss: 0.1565 - acc: 0.9500
Epoch 126/200
 - 0s - loss: 0.1559 - acc: 0.9500
Epoch 127/200
 - 0s - loss: 0.1554 - acc: 0.9500
Epoch 128/200
 - 0s - loss: 0.1549 - acc: 0.9500
Epoch 129/200
 - 0s - loss: 0.1544 - acc: 0.9500
Epoch 130/200
 - 0s - loss: 0.1540 - acc: 0.9533
Epoch 131/200
 - 0s - loss: 0.1536 - acc: 0.9533
Epoch 132/200
 - 0s - loss: 0.1531 - acc: 0.9533
Epoch 133/200
 - 0s - loss: 0.1527 - acc: 0.9533
Epoch 134/200
 - 0s - loss: 0.1524 - acc: 0.9533
Epoch 135/200
 - 0s - loss: 0.1520 - acc: 0.9533
Epoch 136/200
 - 0s - loss: 0.1517 - acc: 0.9533
Epoch 137/200
 - 0s - loss: 0.1513 - acc: 0.9533
Epoch 138/200
 - 0s - loss: 0.1510 - acc: 0.9533
Epoch 139/200
 - 0s - loss: 0.1507 - acc: 0.9500
Epoch 140/200
 - 0s - loss: 0.1504 - acc: 0.9500
Epoch 141/200
 - 0s - loss: 0.1501 - acc: 0.9500
Epoch 142/200
 - 0s - loss: 0.1498 - acc: 0.9500
Epoch 143/200
 - 0s - loss: 0.1496 - acc: 0.9500
Epoch 144/200
 - 0s - loss: 0.1493 - acc: 0.9500
Epoch 145/200
 - 0s - loss: 0.1491 - acc: 0.9500
Epoch 146/200
 - 0s - loss: 0.1488 - acc: 0.9500
Epoch 147/200
 - 0s - loss: 0.1486 - acc: 0.9500
Epoch 148/200
 - 0s - loss: 0.1484 - acc: 0.9500
Epoch 149/200
 - 0s - loss: 0.1482 - acc: 0.9500
Epoch 150/200
 - 0s - loss: 0.1479 - acc: 0.9500
Epoch 151/200
 - 0s - loss: 0.1477 - acc: 0.9533
Epoch 152/200
 - 0s - loss: 0.1475 - acc: 0.9533
Epoch 153/200
 - 0s - loss: 0.1474 - acc: 0.9533
Epoch 154/200
 - 0s - loss: 0.1472 - acc: 0.9533
Epoch 155/200
 - 0s - loss: 0.1470 - acc: 0.9533
Epoch 156/200
 - 0s - loss: 0.1468 - acc: 0.9533
Epoch 157/200
 - 0s - loss: 0.1466 - acc: 0.9533
Epoch 158/200
 - 0s - loss: 0.1465 - acc: 0.9533
Epoch 159/200
 - 0s - loss: 0.1463 - acc: 0.9533
Epoch 160/200
 - 0s - loss: 0.1461 - acc: 0.9533
Epoch 161/200
 - 0s - loss: 0.1460 - acc: 0.9533
Epoch 162/200
 - 0s - loss: 0.1458 - acc: 0.9533
Epoch 163/200
 - 0s - loss: 0.1457 - acc: 0.9533
Epoch 164/200
 - 0s - loss: 0.1456 - acc: 0.9533
Epoch 165/200
 - 0s - loss: 0.1454 - acc: 0.9533
Epoch 166/200
 - 0s - loss: 0.1453 - acc: 0.9533
Epoch 167/200
 - 0s - loss: 0.1451 - acc: 0.9533
Epoch 168/200
 - 0s - loss: 0.1450 - acc: 0.9533
Epoch 169/200
 - 0s - loss: 0.1449 - acc: 0.9533
Epoch 170/200
 - 0s - loss: 0.1448 - acc: 0.9533
Epoch 171/200
 - 0s - loss: 0.1446 - acc: 0.9533
Epoch 172/200
 - 0s - loss: 0.1445 - acc: 0.9533
Epoch 173/200
 - 0s - loss: 0.1444 - acc: 0.9533
Epoch 174/200
 - 0s - loss: 0.1443 - acc: 0.9533
Epoch 175/200
 - 0s - loss: 0.1442 - acc: 0.9533
Epoch 176/200
 - 0s - loss: 0.1441 - acc: 0.9533
Epoch 177/200
 - 0s - loss: 0.1440 - acc: 0.9533
Epoch 178/200
 - 0s - loss: 0.1439 - acc: 0.9533
Epoch 179/200
 - 0s - loss: 0.1438 - acc: 0.9533
Epoch 180/200
 - 0s - loss: 0.1437 - acc: 0.9533
Epoch 181/200
 - 0s - loss: 0.1436 - acc: 0.9533
Epoch 182/200
 - 0s - loss: 0.1435 - acc: 0.9533
Epoch 183/200
 - 0s - loss: 0.1434 - acc: 0.9533
Epoch 184/200
 - 0s - loss: 0.1433 - acc: 0.9533
Epoch 185/200
 - 0s - loss: 0.1432 - acc: 0.9533
Epoch 186/200
 - 0s - loss: 0.1431 - acc: 0.9533
Epoch 187/200
 - 0s - loss: 0.1430 - acc: 0.9533
Epoch 188/200
 - 0s - loss: 0.1429 - acc: 0.9533
Epoch 189/200
 - 0s - loss: 0.1428 - acc: 0.9533
Epoch 190/200
 - 0s - loss: 0.1428 - acc: 0.9533
Epoch 191/200
 - 0s - loss: 0.1427 - acc: 0.9533
Epoch 192/200
 - 0s - loss: 0.1426 - acc: 0.9533
Epoch 193/200
 - 0s - loss: 0.1425 - acc: 0.9533
Epoch 194/200
 - 0s - loss: 0.1424 - acc: 0.9533
Epoch 195/200
 - 0s - loss: 0.1424 - acc: 0.9533
Epoch 196/200
 - 0s - loss: 0.1423 - acc: 0.9533
Epoch 197/200
 - 0s - loss: 0.1422 - acc: 0.9533
Epoch 198/200
 - 0s - loss: 0.1421 - acc: 0.9533
Epoch 199/200
 - 0s - loss: 0.1421 - acc: 0.9533
Epoch 200/200
 - 0s - loss: 0.1420 - acc: 0.9533

Analyzing Training Process

The output of the fit method contains a history attribute that we can use to visualize the changes in loss and accuracy during the training process.

In [8]:
plt.rcParams["figure.figsize"] = [8,4]
plt.subplot(1,2,1)
plt.plot(h.history['acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')

plt.subplot(1,2,2)
plt.plot(h.history['loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.show()

Visualiztion the Decision Region

I have written a function called plot_regions that can be used to plot the decision regions for classification problems with two features.

In [9]:
from ClassificationPlotter import plot_regions

plot_regions(model, X, y, cmap='bwr_r', fig_size=(8,6), keras=True)

The gif below shows how the classification regions changed during the process of training.

Net01

Making Predictions

We can used the predict and predict_classes methods of our model to generate predictions for new observations.

In [10]:
Xnew = np.array([[0.2, 0.4], [0.2, 0.5], [0.2,0.6]])

np.set_printoptions(precision=6, suppress=True)

print('Estimated probabilities of being blue:\n', model.predict(Xnew))
print()
print('Predicted classes:\n', model.predict_classes(Xnew))
Estimated probabilities of being blue:
 [[0.028326]
 [0.204441]
 [0.835694]]

Predicted classes:
 [[0]
 [0]
 [1]]
In [11]:
plot_regions(model, X, y, cmap='bwr', fig_size=(8,6), keras=True, display=False)
plt.scatter(Xnew[:,0], Xnew[:,1], marker="D", c='lime', edgecolor='k', s=100, zorder=3)
plt.show()

Example 2

We will now construct a neural network with a more complicated architecture to address the following classification problem.

In [12]:
from sklearn.datasets import make_circles

np.random.seed(1)
plt.figure(figsize=(8,6))
Z, w = make_circles(500, factor=0.25, noise=0.12)
w = (w+1)%2
plt.scatter(Z[:,0], Z[:,1], c=w, cmap='bwr_r', edgecolor='k')
plt.show()

Network Architecture

The network we will use for this problem will have two hidden layers, each containing 8 (non-bias) neurons.

Net01

Build, Compile, and Train the Network

We will now perform the same steps as before to build, compile, and train the network.

In [13]:
np.random.seed(1)
set_random_seed(1)

model2 = Sequential()
model2.add(Dense(8, input_shape=(2,), activation='sigmoid'))
model2.add(Dense(8, activation='sigmoid'))
model2.add(Dense(1, activation='sigmoid'))
In [14]:
model2.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_3 (Dense)              (None, 8)                 24        
_________________________________________________________________
dense_4 (Dense)              (None, 8)                 72        
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 9         
=================================================================
Total params: 105
Trainable params: 105
Non-trainable params: 0
_________________________________________________________________
In [15]:
opt = Adam(lr=0.1)
model2.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
In [16]:
h = model2.fit(Z, w, batch_size=500, epochs=50, verbose=2)
Epoch 1/50
 - 1s - loss: 0.7145 - acc: 0.5000
Epoch 2/50
 - 0s - loss: 0.7140 - acc: 0.5000
Epoch 3/50
 - 0s - loss: 0.7072 - acc: 0.5000
Epoch 4/50
 - 0s - loss: 0.6933 - acc: 0.5000
Epoch 5/50
 - 0s - loss: 0.6975 - acc: 0.5000
Epoch 6/50
 - 0s - loss: 0.7026 - acc: 0.5000
Epoch 7/50
 - 0s - loss: 0.6993 - acc: 0.5000
Epoch 8/50
 - 0s - loss: 0.6939 - acc: 0.5000
Epoch 9/50
 - 0s - loss: 0.6924 - acc: 0.3740
Epoch 10/50
 - 0s - loss: 0.6945 - acc: 0.5000
Epoch 11/50
 - 0s - loss: 0.6961 - acc: 0.5000
Epoch 12/50
 - 0s - loss: 0.6949 - acc: 0.5000
Epoch 13/50
 - 0s - loss: 0.6919 - acc: 0.5000
Epoch 14/50
 - 0s - loss: 0.6896 - acc: 0.3820
Epoch 15/50
 - 0s - loss: 0.6888 - acc: 0.6360
Epoch 16/50
 - 0s - loss: 0.6886 - acc: 0.5000
Epoch 17/50
 - 0s - loss: 0.6878 - acc: 0.5000
Epoch 18/50
 - 0s - loss: 0.6854 - acc: 0.5000
Epoch 19/50
 - 0s - loss: 0.6818 - acc: 0.5760
Epoch 20/50
 - 0s - loss: 0.6778 - acc: 0.7340
Epoch 21/50
 - 0s - loss: 0.6737 - acc: 0.7860
Epoch 22/50
 - 0s - loss: 0.6692 - acc: 0.6180
Epoch 23/50
 - 0s - loss: 0.6632 - acc: 0.5960
Epoch 24/50
 - 0s - loss: 0.6550 - acc: 0.6980
Epoch 25/50
 - 0s - loss: 0.6442 - acc: 0.8500
Epoch 26/50
 - 0s - loss: 0.6308 - acc: 0.8700
Epoch 27/50
 - 0s - loss: 0.6147 - acc: 0.8720
Epoch 28/50
 - 0s - loss: 0.5949 - acc: 0.8760
Epoch 29/50
 - 0s - loss: 0.5699 - acc: 0.8800
Epoch 30/50
 - 0s - loss: 0.5390 - acc: 0.8980
Epoch 31/50
 - 0s - loss: 0.5021 - acc: 0.9080
Epoch 32/50
 - 0s - loss: 0.4597 - acc: 0.9120
Epoch 33/50
 - 0s - loss: 0.4119 - acc: 0.9240
Epoch 34/50
 - 0s - loss: 0.3605 - acc: 0.9360
Epoch 35/50
 - 0s - loss: 0.3086 - acc: 0.9480
Epoch 36/50
 - 0s - loss: 0.2584 - acc: 0.9720
Epoch 37/50
 - 0s - loss: 0.2113 - acc: 0.9780
Epoch 38/50
 - 0s - loss: 0.1701 - acc: 0.9900
Epoch 39/50
 - 0s - loss: 0.1351 - acc: 0.9900
Epoch 40/50
 - 0s - loss: 0.1055 - acc: 0.9920
Epoch 41/50
 - 0s - loss: 0.0814 - acc: 0.9940
Epoch 42/50
 - 0s - loss: 0.0623 - acc: 0.9960
Epoch 43/50
 - 0s - loss: 0.0475 - acc: 0.9960
Epoch 44/50
 - 0s - loss: 0.0365 - acc: 0.9960
Epoch 45/50
 - 0s - loss: 0.0286 - acc: 0.9960
Epoch 46/50
 - 0s - loss: 0.0231 - acc: 0.9980
Epoch 47/50
 - 0s - loss: 0.0191 - acc: 0.9980
Epoch 48/50
 - 0s - loss: 0.0163 - acc: 0.9980
Epoch 49/50
 - 0s - loss: 0.0142 - acc: 0.9980
Epoch 50/50
 - 0s - loss: 0.0127 - acc: 0.9980

Analyzing Training Process

In [17]:
plt.rcParams["figure.figsize"] = [8,4]
plt.subplot(1,2,1)
plt.plot(h.history['acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')

plt.subplot(1,2,2)
plt.plot(h.history['loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.show()

Visualizing the Decision Region

In [18]:
plot_regions(model2, Z, w, cmap='bwr_r', fig_size=(8,6), keras=True)

Net02

Generating Predictions

In [19]:
Znew = np.array([[0.5, 0.3], [0.5, 0.4], [0.5,0.5]])

np.set_printoptions(precision=6, suppress=True)

print('Estimated probabilities of being red:\n', model2.predict(Znew))
print()
print('Predicted classes:\n', model2.predict_classes(Znew))
Estimated probabilities of being red:
 [[0.24103 ]
 [0.590467]
 [0.865612]]

Predicted classes:
 [[0]
 [1]
 [1]]
In [20]:
plot_regions(model2, Z, w, cmap='bwr_r', fig_size=(8,6), keras=True, display=False)
plt.scatter(Znew[:,0], Znew[:,1], marker="D", c='lime', edgecolor='k', s=60, zorder=3)
plt.show()