02. Neural Network Classification with TensorFlow¶
Okay, we've seen how to deal with a regression problem in TensorFlow, let's look at how we can approach a classification problem.
A classification problem involves predicting whether something is one thing or another.
For example, you might want to:
- Predict whether or not someone has heart disease based on their health parameters. This is called binary classification since there are only two options.
- Decide whether a photo of is of food, a person or a dog. This is called multi-class classification since there are more than two options.
- Predict what categories should be assigned to a Wikipedia article. This is called multi-label classification since a single article could have more than one category assigned.
In this notebook, we're going to work through a number of different classification problems with TensorFlow. In other words, taking a set of inputs and predicting what class those set of inputs belong to.
What we're going to cover¶
Specifically, we're going to go through doing the following with TensorFlow:
- Architecture of a classification model
- Input shapes and output shapes
X
: features/data (inputs)y
: labels (outputs)- "What class do the inputs belong to?"
- Creating custom data to view and fit
- Steps in modelling for binary and mutliclass classification
- Creating a model
- Compiling a model
- Defining a loss function
- Setting up an optimizer
- Finding the best learning rate
- Creating evaluation metrics
- Fitting a model (getting it to find patterns in our data)
- Improving a model
- The power of non-linearity
- Evaluating classification models
- Visualizng the model ("visualize, visualize, visualize")
- Looking at training curves
- Compare predictions to ground truth (using our evaluation metrics)
How you can use this notebook¶
You can read through the descriptions and the code (it should all run, except for the cells which error on purpose), but there's a better option.
Write all of the code yourself.
Yes. I'm serious. Create a new notebook, and rewrite each line by yourself. Investigate it, see if you can break it, why does it break?
You don't have to write the text descriptions but writing the code yourself is a great way to get hands-on experience.
Don't worry if you make mistakes, we all do. The way to get better and make less mistakes is to write more code.
Typical architecture of a classification neural network¶
The word typical is on purpose.
Because the architecture of a classification neural network can widely vary depending on the problem you're working on.
However, there are some fundamentals all deep neural networks contain:
- An input layer.
- Some hidden layers.
- An output layer.
Much of the rest is up to the data analyst creating the model.
The following are some standard values you'll often use in your classification neural networks.
Hyperparameter | Binary Classification | Multiclass classification |
---|---|---|
Input layer shape | Same as number of features (e.g. 5 for age, sex, height, weight, smoking status in heart disease prediction) | Same as binary classification |
Hidden layer(s) | Problem specific, minimum = 1, maximum = unlimited | Same as binary classification |
Neurons per hidden layer | Problem specific, generally 10 to 100 | Same as binary classification |
Output layer shape | 1 (one class or the other) | 1 per class (e.g. 3 for food, person or dog photo) |
Hidden activation | Usually ReLU (rectified linear unit) | Same as binary classification |
Output activation | Sigmoid | Softmax |
Loss function | Cross entropy (tf.keras.losses.BinaryCrossentropy in TensorFlow) |
Cross entropy (tf.keras.losses.CategoricalCrossentropy in TensorFlow) |
Optimizer | SGD (stochastic gradient descent), Adam | Same as binary classification |
Table 1: Typical architecture of a classification network. Source: Adapted from page 295 of Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow Book by Aurélien Géron
Don't worry if not much of the above makes sense right now, we'll get plenty of experience as we go through this notebook.
Let's start by importing TensorFlow as the common alias tf
. For this notebook, make sure you're using version 2.x+.
import tensorflow as tf
print(tf.__version__)
2.8.0
Creating data to view and fit¶
We could start by importing a classification dataset but let's practice making some of our own classification data.
🔑 Note: It's a common practice to get you and model you build working on a toy (or simple) dataset before moving to your actual problem. Treat it as a rehersal experiment before the actual experiment(s).
Since classification is predicting whether something is one thing or another, let's make some data to reflect that.
To do so, we'll use Scikit-Learn's make_circles()
function.
from sklearn.datasets import make_circles
# Make 1000 examples
n_samples = 1000
# Create circles
X, y = make_circles(n_samples,
noise=0.03,
random_state=42)
Wonderful, now we've created some data, let's look at the features (X
) and labels (y
).
# Check out the features
X
array([[ 0.75424625, 0.23148074], [-0.75615888, 0.15325888], [-0.81539193, 0.17328203], ..., [-0.13690036, -0.81001183], [ 0.67036156, -0.76750154], [ 0.28105665, 0.96382443]])
# See the first 10 labels
y[:10]
array([1, 1, 1, 1, 0, 1, 1, 1, 1, 0])
Okay, we've seen some of our data and labels, how about we move towards visualizing?
🔑 Note: One important step of starting any kind of machine learning project is to become one with the data. And one of the best ways to do this is to visualize the data you're working with as much as possible. The data explorer's motto is "visualize, visualize, visualize".
We'll start with a DataFrame.
# Make dataframe of features and labels
import pandas as pd
circles = pd.DataFrame({"X0":X[:, 0], "X1":X[:, 1], "label":y})
circles.head()
X0 | X1 | label | |
---|---|---|---|
0 | 0.754246 | 0.231481 | 1 |
1 | -0.756159 | 0.153259 | 1 |
2 | -0.815392 | 0.173282 | 1 |
3 | -0.393731 | 0.692883 | 1 |
4 | 0.442208 | -0.896723 | 0 |
What kind of labels are we dealing with?
# Check out the different labels
circles.label.value_counts()
1 500 0 500 Name: label, dtype: int64
Alright, looks like we're dealing with a binary classification problem. It's binary because there are only two labels (0 or 1).
If there were more label options (e.g. 0, 1, 2, 3 or 4), it would be called multiclass classification.
Let's take our visualization a step further and plot our data.
# Visualize with a plot
import matplotlib.pyplot as plt
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.RdYlBu);
Nice! From the plot, can you guess what kind of model we might want to build?
How about we try and build one to classify blue or red dots? As in, a model which is able to distinguish blue from red dots.
🛠 Practice: Before pushing forward, you might want to spend 10 minutes playing around with the TensorFlow Playground. Try adjusting the different hyperparameters you see and click play to see a neural network train. I think you'll find the data very similar to what we've just created.
Input and output shapes¶
One of the most common issues you'll run into when building neural networks is shape mismatches.
More specifically, the shape of the input data and the shape of the output data.
In our case, we want to input X
and get our model to predict y
.
So let's check out the shapes of X
and y
.
# Check the shapes of our features and labels
X.shape, y.shape
((1000, 2), (1000,))
Hmm, where do these numbers come from?
# Check how many samples we have
len(X), len(y)
(1000, 1000)
So we've got as many X
values as we do y
values, that makes sense.
Let's check out one example of each.
# View the first example of features and labels
X[0], y[0]
(array([0.75424625, 0.23148074]), 1)
Alright, so we've got two X
features which lead to one y
value.
This means our neural network input shape will has to accept a tensor with at least one dimension being two and output a tensor with at least one value.
🤔 Note:
y
having a shape of (1000,) can seem confusing. However, this is because ally
values are actually scalars (single values) and therefore don't have a dimension. For now, think of your output shape as being at least the same value as one example ofy
(in our case, the output from our neural network has to be at least one value).
Steps in modelling¶
Now we know what data we have as well as the input and output shapes, let's see how we'd build a neural network to model it.
In TensorFlow, there are typically 3 fundamental steps to creating and training a model.
- Creating a model - piece together the layers of a neural network yourself (using the functional or sequential API) or import a previously built model (known as transfer learning).
- Compiling a model - defining how a model's performance should be measured (loss/metrics) as well as defining how it should improve (optimizer).
- Fitting a model - letting the model try to find patterns in the data (how does
X
get toy
).
Let's see these in action using the Sequential API to build a model for our regression data. And then we'll step through each.
# Set random seed
tf.random.set_seed(42)
# 1. Create the model using the Sequential API
model_1 = tf.keras.Sequential([
tf.keras.layers.Dense(1)
])
# 2. Compile the model
model_1.compile(loss=tf.keras.losses.BinaryCrossentropy(), # binary since we are working with 2 clases (0 & 1)
optimizer=tf.keras.optimizers.SGD(),
metrics=['accuracy'])
# 3. Fit the model
model_1.fit(X, y, epochs=5)
Epoch 1/5 32/32 [==============================] - 0s 1ms/step - loss: 2.8544 - accuracy: 0.4600 Epoch 2/5 32/32 [==============================] - 0s 1ms/step - loss: 0.7131 - accuracy: 0.5430 Epoch 3/5 32/32 [==============================] - 0s 1ms/step - loss: 0.6973 - accuracy: 0.5090 Epoch 4/5 32/32 [==============================] - 0s 2ms/step - loss: 0.6950 - accuracy: 0.5010 Epoch 5/5 32/32 [==============================] - 0s 2ms/step - loss: 0.6942 - accuracy: 0.4830
<keras.callbacks.History at 0x7fc521b6edd0>
Looking at the accuracy metric, our model performs poorly (50% accuracy on a binary classification problem is the equivalent of guessing), but what if we trained it for longer?
# Train our model for longer (more chances to look at the data)
model_1.fit(X, y, epochs=200, verbose=0) # set verbose=0 to remove training updates
model_1.evaluate(X, y)
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5000
[0.6934831142425537, 0.5]
Even after 200 passes of the data, it's still performing as if it's guessing.
What if we added an extra layer and trained for a little longer?
# Set random seed
tf.random.set_seed(42)
# 1. Create the model (same as model_1 but with an extra layer)
model_2 = tf.keras.Sequential([
tf.keras.layers.Dense(1), # add an extra layer
tf.keras.layers.Dense(1)
])
# 2. Compile the model
model_2.compile(loss=tf.keras.losses.BinaryCrossentropy(),
optimizer=tf.keras.optimizers.SGD(),
metrics=['accuracy'])
# 3. Fit the model
model_2.fit(X, y, epochs=100, verbose=0) # set verbose=0 to make the output print less
<keras.callbacks.History at 0x7fc521a02d90>
# Evaluate the model
model_2.evaluate(X, y)
32/32 [==============================] - 0s 6ms/step - loss: 0.6933 - accuracy: 0.5000
[0.6933314800262451, 0.5]
Still not even as good as guessing (~50% accuracy)... hmm...?
Let's remind ourselves of a couple more ways we can use to improve our models.
Improving a model¶
To improve our model, we can alter almost every part of the 3 steps we went through before.
- Creating a model - here you might want to add more layers, increase the number of hidden units (also called neurons) within each layer, change the activation functions of each layer.
- Compiling a model - you might want to choose a different optimization function (such as the Adam optimizer, which is usually pretty good for many problems) or perhaps change the learning rate of the optimization function.
- Fitting a model - perhaps you could fit a model for more epochs (leave it training for longer).
There are many different ways to potentially improve a neural network. Some of the most common include: increasing the number of layers (making the network deeper), increasing the number of hidden units (making the network wider) and changing the learning rate. Because these values are all human-changeable, they're referred to as hyperparameters) and the practice of trying to find the best hyperparameters is referred to as hyperparameter tuning.
How about we try adding more neurons, an extra layer and our friend the Adam optimizer?
Surely doing this will result in predictions better than guessing...
Note: The following message (below this one) can be ignored if you're running TensorFlow 2.8.0+, the error seems to have been fixed.
Note: If you're using TensorFlow 2.7.0+ (but not 2.8.0+) the original code from the following cells may have caused some errors. They've since been updated to fix those errors. You can see explanations on what happened at the following resources:
# Set random seed
tf.random.set_seed(42)
# 1. Create the model (this time 3 layers)
model_3 = tf.keras.Sequential([
# Before TensorFlow 2.7.0
# tf.keras.layers.Dense(100), # add 100 dense neurons
# With TensorFlow 2.7.0
# tf.keras.layers.Dense(100, input_shape=(None, 1)), # add 100 dense neurons
## After TensorFlow 2.8.0 ##
tf.keras.layers.Dense(100), # add 100 dense neurons
tf.keras.layers.Dense(10), # add another layer with 10 neurons
tf.keras.layers.Dense(1)
])
# 2. Compile the model
model_3.compile(loss=tf.keras.losses.BinaryCrossentropy(),
optimizer=tf.keras.optimizers.Adam(), # use Adam instead of SGD
metrics=['accuracy'])
# 3. Fit the model
model_3.fit(X, y, epochs=100, verbose=1) # fit for 100 passes of the data
Epoch 1/100 32/32 [==============================] - 2s 4ms/step - loss: 2.9658 - accuracy: 0.4490 Epoch 2/100 32/32 [==============================] - 0s 3ms/step - loss: 0.7229 - accuracy: 0.4390 Epoch 3/100 32/32 [==============================] - 0s 8ms/step - loss: 0.6971 - accuracy: 0.4700 Epoch 4/100 32/32 [==============================] - 0s 8ms/step - loss: 0.6953 - accuracy: 0.4620 Epoch 5/100 32/32 [==============================] - 0s 6ms/step - loss: 0.6943 - accuracy: 0.4750 Epoch 6/100 32/32 [==============================] - 0s 5ms/step - loss: 0.6941 - accuracy: 0.4750 Epoch 7/100 32/32 [==============================] - 0s 6ms/step - loss: 0.6939 - accuracy: 0.5040 Epoch 8/100 32/32 [==============================] - 0s 6ms/step - loss: 0.6946 - accuracy: 0.4980 Epoch 9/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6943 - accuracy: 0.4880 Epoch 10/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6944 - accuracy: 0.4480 Epoch 11/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6952 - accuracy: 0.4450 Epoch 12/100 32/32 [==============================] - 0s 6ms/step - loss: 0.6946 - accuracy: 0.4890 Epoch 13/100 32/32 [==============================] - 0s 5ms/step - loss: 0.6956 - accuracy: 0.4920 Epoch 14/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6950 - accuracy: 0.5140 Epoch 15/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6956 - accuracy: 0.4590 Epoch 16/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6943 - accuracy: 0.4820 Epoch 17/100 32/32 [==============================] - 0s 5ms/step - loss: 0.6975 - accuracy: 0.4950 Epoch 18/100 32/32 [==============================] - 0s 5ms/step - loss: 0.6945 - accuracy: 0.4550 Epoch 19/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6953 - accuracy: 0.5330 Epoch 20/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6956 - accuracy: 0.4560 Epoch 21/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6955 - accuracy: 0.4920 Epoch 22/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6948 - accuracy: 0.4350 Epoch 23/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6944 - accuracy: 0.4990 Epoch 24/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6963 - accuracy: 0.4690 Epoch 25/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6973 - accuracy: 0.4970 Epoch 26/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6973 - accuracy: 0.4830 Epoch 27/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6963 - accuracy: 0.5040 Epoch 28/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6946 - accuracy: 0.4680 Epoch 29/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6973 - accuracy: 0.4790 Epoch 30/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6998 - accuracy: 0.4800 Epoch 31/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6950 - accuracy: 0.5030 Epoch 32/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6965 - accuracy: 0.4760 Epoch 33/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6957 - accuracy: 0.4570 Epoch 34/100 32/32 [==============================] - 0s 7ms/step - loss: 0.6973 - accuracy: 0.4800 Epoch 35/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6960 - accuracy: 0.5050 Epoch 36/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6946 - accuracy: 0.4680 Epoch 37/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6952 - accuracy: 0.4870 Epoch 38/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6975 - accuracy: 0.4980 Epoch 39/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6949 - accuracy: 0.4690 Epoch 40/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6965 - accuracy: 0.4990 Epoch 41/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6979 - accuracy: 0.4890 Epoch 42/100 32/32 [==============================] - 0s 5ms/step - loss: 0.6995 - accuracy: 0.4700 Epoch 43/100 32/32 [==============================] - 0s 5ms/step - loss: 0.6957 - accuracy: 0.5030 Epoch 44/100 32/32 [==============================] - 0s 5ms/step - loss: 0.6981 - accuracy: 0.4830 Epoch 45/100 32/32 [==============================] - 0s 5ms/step - loss: 0.6964 - accuracy: 0.5120 Epoch 46/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6949 - accuracy: 0.4730 Epoch 47/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6974 - accuracy: 0.5170 Epoch 48/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6966 - accuracy: 0.4840 Epoch 49/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6976 - accuracy: 0.4870 Epoch 50/100 32/32 [==============================] - 0s 4ms/step - loss: 0.6972 - accuracy: 0.4610 Epoch 51/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6980 - accuracy: 0.4800 Epoch 52/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6962 - accuracy: 0.5020 Epoch 53/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6972 - accuracy: 0.4940 Epoch 54/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6993 - accuracy: 0.5020 Epoch 55/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6953 - accuracy: 0.5220 Epoch 56/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6966 - accuracy: 0.4930 Epoch 57/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6981 - accuracy: 0.4720 Epoch 58/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6968 - accuracy: 0.4980 Epoch 59/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7004 - accuracy: 0.4890 Epoch 60/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7011 - accuracy: 0.4870 Epoch 61/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6996 - accuracy: 0.4680 Epoch 62/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6965 - accuracy: 0.4980 Epoch 63/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6948 - accuracy: 0.4860 Epoch 64/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6960 - accuracy: 0.4940 Epoch 65/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6996 - accuracy: 0.4810 Epoch 66/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6965 - accuracy: 0.5010 Epoch 67/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6977 - accuracy: 0.4500 Epoch 68/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6958 - accuracy: 0.5020 Epoch 69/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6953 - accuracy: 0.5230 Epoch 70/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6988 - accuracy: 0.4680 Epoch 71/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6975 - accuracy: 0.4650 Epoch 72/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6966 - accuracy: 0.5010 Epoch 73/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6963 - accuracy: 0.4930 Epoch 74/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6960 - accuracy: 0.5130 Epoch 75/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6963 - accuracy: 0.4980 Epoch 76/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6980 - accuracy: 0.4850 Epoch 77/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6971 - accuracy: 0.5020 Epoch 78/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6972 - accuracy: 0.4720 Epoch 79/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7009 - accuracy: 0.5110 Epoch 80/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6980 - accuracy: 0.4930 Epoch 81/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6980 - accuracy: 0.5030 Epoch 82/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6968 - accuracy: 0.4830 Epoch 83/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7002 - accuracy: 0.4480 Epoch 84/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6963 - accuracy: 0.4970 Epoch 85/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6967 - accuracy: 0.4590 Epoch 86/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7002 - accuracy: 0.4900 Epoch 87/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6977 - accuracy: 0.4550 Epoch 88/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6981 - accuracy: 0.4590 Epoch 89/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6979 - accuracy: 0.4750 Epoch 90/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6981 - accuracy: 0.4620 Epoch 91/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6984 - accuracy: 0.4980 Epoch 92/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6988 - accuracy: 0.4760 Epoch 93/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6982 - accuracy: 0.4690 Epoch 94/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7005 - accuracy: 0.4800 Epoch 95/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6974 - accuracy: 0.4520 Epoch 96/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6980 - accuracy: 0.4790 Epoch 97/100 32/32 [==============================] - 0s 3ms/step - loss: 0.6973 - accuracy: 0.4580 Epoch 98/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6989 - accuracy: 0.4800 Epoch 99/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6976 - accuracy: 0.5090 Epoch 100/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6966 - accuracy: 0.4870
<keras.callbacks.History at 0x7fc5208e3cd0>
Still!
We've pulled out a few tricks but our model isn't even doing better than guessing.
Let's make some visualizations to see what's happening.
🔑 Note: Whenever your model is performing strangely or there's something going on with your data you're not quite sure of, remember these three words: visualize, visualize, visualize. Inspect your data, inspect your model, inpsect your model's predictions.
To visualize our model's predictions we're going to create a function plot_decision_boundary()
which:
- Takes in a trained model, features (
X
) and labels (y
). - Creates a meshgrid of the different
X
values. - Makes predictions across the meshgrid.
- Plots the predictions as well as a line between the different zones (where each unique class falls).
If this sounds confusing, let's see it in code and then see the output.
🔑 Note: If you're ever unsure of what a function does, try unraveling it and writing it line by line for yourself to see what it does. Break it into small parts and see what each part outputs.
import numpy as np
def plot_decision_boundary(model, X, y):
"""
Plots the decision boundary created by a model predicting on X.
This function has been adapted from two phenomenal resources:
1. CS231n - https://cs231n.github.io/neural-networks-case-study/
2. Made with ML basics - https://github.com/GokuMohandas/MadeWithML/blob/main/notebooks/08_Neural_Networks.ipynb
"""
# Define the axis boundaries of the plot and create a meshgrid
x_min, x_max = X[:, 0].min() - 0.1, X[:, 0].max() + 0.1
y_min, y_max = X[:, 1].min() - 0.1, X[:, 1].max() + 0.1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 100),
np.linspace(y_min, y_max, 100))
# Create X values (we're going to predict on all of these)
x_in = np.c_[xx.ravel(), yy.ravel()] # stack 2D arrays together: https://numpy.org/devdocs/reference/generated/numpy.c_.html
# Make predictions using the trained model
y_pred = model.predict(x_in)
# Check for multi-class
if model.output_shape[-1] > 1: # checks the final dimension of the model's output shape, if this is > (greater than) 1, it's multi-class
print("doing multiclass classification...")
# We have to reshape our predictions to get them ready for plotting
y_pred = np.argmax(y_pred, axis=1).reshape(xx.shape)
else:
print("doing binary classifcation...")
y_pred = np.round(np.max(y_pred, axis=1)).reshape(xx.shape)
# Plot decision boundary
plt.contourf(xx, yy, y_pred, cmap=plt.cm.RdYlBu, alpha=0.7)
plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.RdYlBu)
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
Now we've got a function to plot our model's decision boundary (the cut off point its making between red and blue dots), let's try it out.
# Check out the predictions our model is making
plot_decision_boundary(model_3, X, y)
doing binary classifcation...
Looks like our model is trying to draw a straight line through the data.
What's wrong with doing this?
The main issue is our data isn't separable by a straight line.
In a regression problem, our model might work. In fact, let's try it.
# Set random seed
tf.random.set_seed(42)
# Create some regression data
X_regression = np.arange(0, 1000, 5)
y_regression = np.arange(100, 1100, 5)
# Split it into training and test sets
X_reg_train = X_regression[:150]
X_reg_test = X_regression[150:]
y_reg_train = y_regression[:150]
y_reg_test = y_regression[150:]
# Fit our model to the data
# Note: Before TensorFlow 2.7.0, this line would work
# model_3.fit(X_reg_train, y_reg_train, epochs=100)
# After TensorFlow 2.7.0, see here for more: https://github.com/mrdbourke/tensorflow-deep-learning/discussions/278
model_3.fit(tf.expand_dims(X_reg_train, axis=-1),
y_reg_train,
epochs=100)
Epoch 1/100
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-19-5dd5867236b4> in <module>() 19 model_3.fit(tf.expand_dims(X_reg_train, axis=-1), 20 y_reg_train, ---> 21 epochs=100) /usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs) 65 except Exception as e: # pylint: disable=broad-except 66 filtered_tb = _process_traceback_frames(e.__traceback__) ---> 67 raise e.with_traceback(filtered_tb) from None 68 finally: 69 del filtered_tb /usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py in autograph_handler(*args, **kwargs) 1145 except Exception as e: # pylint:disable=broad-except 1146 if hasattr(e, "ag_error_metadata"): -> 1147 raise e.ag_error_metadata.to_exception(e) 1148 else: 1149 raise ValueError: in user code: File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1021, in train_function * return step_function(self, iterator) File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1010, in step_function ** outputs = model.distribute_strategy.run(run_step, args=(data,)) File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1000, in run_step ** outputs = model.train_step(data) File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 859, in train_step y_pred = self(x, training=True) File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler raise e.with_traceback(filtered_tb) from None File "/usr/local/lib/python3.7/dist-packages/keras/engine/input_spec.py", line 249, in assert_input_compatibility f'Input {input_index} of layer "{layer_name}" is ' ValueError: Exception encountered when calling layer "sequential_2" (type Sequential). Input 0 of layer "dense_3" is incompatible with the layer: expected axis -1 of input shape to have value 2, but received input with shape (None, 1) Call arguments received: • inputs=tf.Tensor(shape=(None, 1), dtype=int64) • training=True • mask=None
model_3.summary()
Model: "sequential_2" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_3 (Dense) (None, 100) 300 dense_4 (Dense) (None, 10) 1010 dense_5 (Dense) (None, 1) 11 ================================================================= Total params: 1,321 Trainable params: 1,321 Non-trainable params: 0 _________________________________________________________________
Oh wait... we compiled our model for a binary classification problem.
No trouble, we can recreate it for a regression problem.
# Setup random seed
tf.random.set_seed(42)
# Recreate the model
model_3 = tf.keras.Sequential([
tf.keras.layers.Dense(100),
tf.keras.layers.Dense(10),
tf.keras.layers.Dense(1)
])
# Change the loss and metrics of our compiled model
model_3.compile(loss=tf.keras.losses.mae, # change the loss function to be regression-specific
optimizer=tf.keras.optimizers.Adam(),
metrics=['mae']) # change the metric to be regression-specific
# Fit the recompiled model
model_3.fit(tf.expand_dims(X_reg_train, axis=-1),
y_reg_train,
epochs=100)
Epoch 1/100 5/5 [==============================] - 0s 3ms/step - loss: 248.2155 - mae: 248.2155 Epoch 2/100 5/5 [==============================] - 0s 3ms/step - loss: 138.9005 - mae: 138.9005 Epoch 3/100 5/5 [==============================] - 0s 3ms/step - loss: 53.1039 - mae: 53.1039 Epoch 4/100 5/5 [==============================] - 0s 4ms/step - loss: 73.5170 - mae: 73.5170 Epoch 5/100 5/5 [==============================] - 0s 4ms/step - loss: 71.2358 - mae: 71.2358 Epoch 6/100 5/5 [==============================] - 0s 3ms/step - loss: 47.0040 - mae: 47.0040 Epoch 7/100 5/5 [==============================] - 0s 3ms/step - loss: 45.9386 - mae: 45.9386 Epoch 8/100 5/5 [==============================] - 0s 3ms/step - loss: 42.3638 - mae: 42.3638 Epoch 9/100 5/5 [==============================] - 0s 3ms/step - loss: 43.6831 - mae: 43.6831 Epoch 10/100 5/5 [==============================] - 0s 3ms/step - loss: 42.6198 - mae: 42.6198 Epoch 11/100 5/5 [==============================] - 0s 5ms/step - loss: 42.4797 - mae: 42.4797 Epoch 12/100 5/5 [==============================] - 0s 3ms/step - loss: 41.5537 - mae: 41.5537 Epoch 13/100 5/5 [==============================] - 0s 4ms/step - loss: 42.0972 - mae: 42.0972 Epoch 14/100 5/5 [==============================] - 0s 3ms/step - loss: 41.8647 - mae: 41.8647 Epoch 15/100 5/5 [==============================] - 0s 3ms/step - loss: 41.5342 - mae: 41.5342 Epoch 16/100 5/5 [==============================] - 0s 3ms/step - loss: 41.4028 - mae: 41.4028 Epoch 17/100 5/5 [==============================] - 0s 3ms/step - loss: 41.6887 - mae: 41.6887 Epoch 18/100 5/5 [==============================] - 0s 4ms/step - loss: 41.6137 - mae: 41.6137 Epoch 19/100 5/5 [==============================] - 0s 3ms/step - loss: 41.2796 - mae: 41.2796 Epoch 20/100 5/5 [==============================] - 0s 3ms/step - loss: 41.1947 - mae: 41.1947 Epoch 21/100 5/5 [==============================] - 0s 3ms/step - loss: 41.2130 - mae: 41.2130 Epoch 22/100 5/5 [==============================] - 0s 4ms/step - loss: 41.0893 - mae: 41.0893 Epoch 23/100 5/5 [==============================] - 0s 4ms/step - loss: 41.2019 - mae: 41.2019 Epoch 24/100 5/5 [==============================] - 0s 4ms/step - loss: 40.9989 - mae: 40.9989 Epoch 25/100 5/5 [==============================] - 0s 4ms/step - loss: 41.0131 - mae: 41.0131 Epoch 26/100 5/5 [==============================] - 0s 3ms/step - loss: 41.0654 - mae: 41.0654 Epoch 27/100 5/5 [==============================] - 0s 4ms/step - loss: 40.8764 - mae: 40.8764 Epoch 28/100 5/5 [==============================] - 0s 3ms/step - loss: 41.0545 - mae: 41.0545 Epoch 29/100 5/5 [==============================] - 0s 3ms/step - loss: 41.0480 - mae: 41.0480 Epoch 30/100 5/5 [==============================] - 0s 3ms/step - loss: 40.8807 - mae: 40.8807 Epoch 31/100 5/5 [==============================] - 0s 3ms/step - loss: 41.2695 - mae: 41.2695 Epoch 32/100 5/5 [==============================] - 0s 3ms/step - loss: 40.9949 - mae: 40.9949 Epoch 33/100 5/5 [==============================] - 0s 3ms/step - loss: 41.0760 - mae: 41.0760 Epoch 34/100 5/5 [==============================] - 0s 3ms/step - loss: 41.2471 - mae: 41.2471 Epoch 35/100 5/5 [==============================] - 0s 3ms/step - loss: 40.6102 - mae: 40.6102 Epoch 36/100 5/5 [==============================] - 0s 3ms/step - loss: 41.1093 - mae: 41.1093 Epoch 37/100 5/5 [==============================] - 0s 3ms/step - loss: 40.8191 - mae: 40.8191 Epoch 38/100 5/5 [==============================] - 0s 3ms/step - loss: 40.2485 - mae: 40.2485 Epoch 39/100 5/5 [==============================] - 0s 3ms/step - loss: 41.0625 - mae: 41.0625 Epoch 40/100 5/5 [==============================] - 0s 3ms/step - loss: 40.5311 - mae: 40.5311 Epoch 41/100 5/5 [==============================] - 0s 3ms/step - loss: 40.5497 - mae: 40.5497 Epoch 42/100 5/5 [==============================] - 0s 3ms/step - loss: 40.4322 - mae: 40.4322 Epoch 43/100 5/5 [==============================] - 0s 3ms/step - loss: 40.5367 - mae: 40.5367 Epoch 44/100 5/5 [==============================] - 0s 3ms/step - loss: 40.2487 - mae: 40.2487 Epoch 45/100 5/5 [==============================] - 0s 3ms/step - loss: 40.5152 - mae: 40.5152 Epoch 46/100 5/5 [==============================] - 0s 3ms/step - loss: 40.3702 - mae: 40.3702 Epoch 47/100 5/5 [==============================] - 0s 3ms/step - loss: 40.4769 - mae: 40.4769 Epoch 48/100 5/5 [==============================] - 0s 4ms/step - loss: 40.1532 - mae: 40.1532 Epoch 49/100 5/5 [==============================] - 0s 3ms/step - loss: 40.7291 - mae: 40.7291 Epoch 50/100 5/5 [==============================] - 0s 3ms/step - loss: 40.1536 - mae: 40.1536 Epoch 51/100 5/5 [==============================] - 0s 4ms/step - loss: 40.2711 - mae: 40.2711 Epoch 52/100 5/5 [==============================] - 0s 3ms/step - loss: 40.6572 - mae: 40.6572 Epoch 53/100 5/5 [==============================] - 0s 3ms/step - loss: 40.6573 - mae: 40.6573 Epoch 54/100 5/5 [==============================] - 0s 3ms/step - loss: 40.6894 - mae: 40.6894 Epoch 55/100 5/5 [==============================] - 0s 4ms/step - loss: 41.2771 - mae: 41.2771 Epoch 56/100 5/5 [==============================] - 0s 3ms/step - loss: 41.8519 - mae: 41.8519 Epoch 57/100 5/5 [==============================] - 0s 3ms/step - loss: 40.7903 - mae: 40.7903 Epoch 58/100 5/5 [==============================] - 0s 3ms/step - loss: 40.3128 - mae: 40.3128 Epoch 59/100 5/5 [==============================] - 0s 3ms/step - loss: 40.7198 - mae: 40.7198 Epoch 60/100 5/5 [==============================] - 0s 3ms/step - loss: 40.1478 - mae: 40.1478 Epoch 61/100 5/5 [==============================] - 0s 3ms/step - loss: 40.1116 - mae: 40.1116 Epoch 62/100 5/5 [==============================] - 0s 4ms/step - loss: 40.7800 - mae: 40.7800 Epoch 63/100 5/5 [==============================] - 0s 3ms/step - loss: 39.7242 - mae: 39.7242 Epoch 64/100 5/5 [==============================] - 0s 3ms/step - loss: 40.1465 - mae: 40.1465 Epoch 65/100 5/5 [==============================] - 0s 3ms/step - loss: 39.6887 - mae: 39.6887 Epoch 66/100 5/5 [==============================] - 0s 3ms/step - loss: 40.2840 - mae: 40.2840 Epoch 67/100 5/5 [==============================] - 0s 3ms/step - loss: 39.5541 - mae: 39.5541 Epoch 68/100 5/5 [==============================] - 0s 3ms/step - loss: 39.7378 - mae: 39.7378 Epoch 69/100 5/5 [==============================] - 0s 3ms/step - loss: 39.9784 - mae: 39.9784 Epoch 70/100 5/5 [==============================] - 0s 3ms/step - loss: 40.0016 - mae: 40.0016 Epoch 71/100 5/5 [==============================] - 0s 3ms/step - loss: 40.0913 - mae: 40.0913 Epoch 72/100 5/5 [==============================] - 0s 3ms/step - loss: 39.2547 - mae: 39.2547 Epoch 73/100 5/5 [==============================] - 0s 3ms/step - loss: 39.6828 - mae: 39.6828 Epoch 74/100 5/5 [==============================] - 0s 3ms/step - loss: 39.5373 - mae: 39.5373 Epoch 75/100 5/5 [==============================] - 0s 3ms/step - loss: 39.6265 - mae: 39.6265 Epoch 76/100 5/5 [==============================] - 0s 3ms/step - loss: 39.3110 - mae: 39.3110 Epoch 77/100 5/5 [==============================] - 0s 3ms/step - loss: 39.1599 - mae: 39.1599 Epoch 78/100 5/5 [==============================] - 0s 3ms/step - loss: 39.7550 - mae: 39.7550 Epoch 79/100 5/5 [==============================] - 0s 3ms/step - loss: 39.2542 - mae: 39.2542 Epoch 80/100 5/5 [==============================] - 0s 3ms/step - loss: 38.6968 - mae: 38.6968 Epoch 81/100 5/5 [==============================] - 0s 4ms/step - loss: 39.5442 - mae: 39.5442 Epoch 82/100 5/5 [==============================] - 0s 3ms/step - loss: 39.8686 - mae: 39.8686 Epoch 83/100 5/5 [==============================] - 0s 3ms/step - loss: 39.1693 - mae: 39.1693 Epoch 84/100 5/5 [==============================] - 0s 3ms/step - loss: 38.8840 - mae: 38.8840 Epoch 85/100 5/5 [==============================] - 0s 3ms/step - loss: 38.8887 - mae: 38.8887 Epoch 86/100 5/5 [==============================] - 0s 4ms/step - loss: 38.6614 - mae: 38.6614 Epoch 87/100 5/5 [==============================] - 0s 3ms/step - loss: 38.8399 - mae: 38.8399 Epoch 88/100 5/5 [==============================] - 0s 3ms/step - loss: 38.6604 - mae: 38.6604 Epoch 89/100 5/5 [==============================] - 0s 3ms/step - loss: 38.7559 - mae: 38.7559 Epoch 90/100 5/5 [==============================] - 0s 3ms/step - loss: 38.5442 - mae: 38.5442 Epoch 91/100 5/5 [==============================] - 0s 3ms/step - loss: 38.3247 - mae: 38.3247 Epoch 92/100 5/5 [==============================] - 0s 3ms/step - loss: 38.8431 - mae: 38.8431 Epoch 93/100 5/5 [==============================] - 0s 3ms/step - loss: 39.1137 - mae: 39.1137 Epoch 94/100 5/5 [==============================] - 0s 3ms/step - loss: 38.1463 - mae: 38.1463 Epoch 95/100 5/5 [==============================] - 0s 4ms/step - loss: 38.3998 - mae: 38.3998 Epoch 96/100 5/5 [==============================] - 0s 3ms/step - loss: 38.5599 - mae: 38.5599 Epoch 97/100 5/5 [==============================] - 0s 3ms/step - loss: 38.1038 - mae: 38.1038 Epoch 98/100 5/5 [==============================] - 0s 4ms/step - loss: 39.0081 - mae: 39.0081 Epoch 99/100 5/5 [==============================] - 0s 4ms/step - loss: 38.3056 - mae: 38.3056 Epoch 100/100 5/5 [==============================] - 0s 3ms/step - loss: 37.9976 - mae: 37.9976
<keras.callbacks.History at 0x7fc520608fd0>
Okay, it seems like our model is learning something (the mae
value trends down with each epoch), let's plot its predictions.
# Make predictions with our trained model
y_reg_preds = model_3.predict(y_reg_test)
# Plot the model's predictions against our regression data
plt.figure(figsize=(10, 7))
plt.scatter(X_reg_train, y_reg_train, c='b', label='Training data')
plt.scatter(X_reg_test, y_reg_test, c='g', label='Testing data')
plt.scatter(X_reg_test, y_reg_preds.squeeze(), c='r', label='Predictions')
plt.legend();
Okay, the predictions aren't perfect (if the predictions were perfect, the red would line up with the green), but they look better than complete guessing.
So this means our model must be learning something...
There must be something we're missing out on for our classification problem.
The missing piece: Non-linearity¶
Okay, so we saw our neural network can model straight lines (with ability a little bit better than guessing).
What about non-straight (non-linear) lines?
If we're going to model our classification data (the red and clue circles), we're going to need some non-linear lines.
🔨 Practice: Before we get to the next steps, I'd encourage you to play around with the TensorFlow Playground (check out what the data has in common with our own classification data) for 10-minutes. In particular the tab which says "activation". Once you're done, come back.
Did you try out the activation options? If so, what did you find?
If you didn't, don't worry, let's see it in code.
We're going to replicate the neural network you can see at this link: TensorFlow Playground.
The neural network we're going to recreate with TensorFlow code. See it live at TensorFlow Playground.
The main change we'll add to models we've built before is the use of the activation
keyword.
# Set the random seed
tf.random.set_seed(42)
# Create the model
model_4 = tf.keras.Sequential([
tf.keras.layers.Dense(1, activation=tf.keras.activations.linear), # 1 hidden layer with linear activation
tf.keras.layers.Dense(1) # output layer
])
# Compile the model
model_4.compile(loss=tf.keras.losses.binary_crossentropy,
optimizer=tf.keras.optimizers.Adam(learning_rate=0.001), # note: "lr" used to be what was used, now "learning_rate" is favoured
metrics=["accuracy"])
# Fit the model
history = model_4.fit(X, y, epochs=100)
Epoch 1/100 32/32 [==============================] - 1s 2ms/step - loss: 4.2380 - accuracy: 0.5000 Epoch 2/100 32/32 [==============================] - 0s 2ms/step - loss: 4.0223 - accuracy: 0.5000 Epoch 3/100 32/32 [==============================] - 0s 2ms/step - loss: 3.8296 - accuracy: 0.5000 Epoch 4/100 32/32 [==============================] - 0s 2ms/step - loss: 3.7654 - accuracy: 0.5000 Epoch 5/100 32/32 [==============================] - 0s 2ms/step - loss: 3.6464 - accuracy: 0.5000 Epoch 6/100 32/32 [==============================] - 0s 2ms/step - loss: 3.4960 - accuracy: 0.5000 Epoch 7/100 32/32 [==============================] - 0s 2ms/step - loss: 3.3804 - accuracy: 0.5000 Epoch 8/100 32/32 [==============================] - 0s 2ms/step - loss: 3.2279 - accuracy: 0.5000 Epoch 9/100 32/32 [==============================] - 0s 2ms/step - loss: 2.7024 - accuracy: 0.5000 Epoch 10/100 32/32 [==============================] - 0s 1ms/step - loss: 2.4002 - accuracy: 0.5000 Epoch 11/100 32/32 [==============================] - 0s 1ms/step - loss: 2.1984 - accuracy: 0.5000 Epoch 12/100 32/32 [==============================] - 0s 2ms/step - loss: 1.3257 - accuracy: 0.5000 Epoch 13/100 32/32 [==============================] - 0s 2ms/step - loss: 1.0542 - accuracy: 0.5000 Epoch 14/100 32/32 [==============================] - 0s 2ms/step - loss: 1.0261 - accuracy: 0.5000 Epoch 15/100 32/32 [==============================] - 0s 2ms/step - loss: 1.0069 - accuracy: 0.5000 Epoch 16/100 32/32 [==============================] - 0s 2ms/step - loss: 0.9914 - accuracy: 0.5000 Epoch 17/100 32/32 [==============================] - 0s 2ms/step - loss: 0.9776 - accuracy: 0.5000 Epoch 18/100 32/32 [==============================] - 0s 1ms/step - loss: 0.9656 - accuracy: 0.5000 Epoch 19/100 32/32 [==============================] - 0s 1ms/step - loss: 0.9549 - accuracy: 0.5000 Epoch 20/100 32/32 [==============================] - 0s 1ms/step - loss: 0.9448 - accuracy: 0.5000 Epoch 21/100 32/32 [==============================] - 0s 2ms/step - loss: 0.9357 - accuracy: 0.5000 Epoch 22/100 32/32 [==============================] - 0s 1ms/step - loss: 0.9269 - accuracy: 0.5000 Epoch 23/100 32/32 [==============================] - 0s 1ms/step - loss: 0.9190 - accuracy: 0.5000 Epoch 24/100 32/32 [==============================] - 0s 2ms/step - loss: 0.9115 - accuracy: 0.5000 Epoch 25/100 32/32 [==============================] - 0s 1ms/step - loss: 0.9046 - accuracy: 0.5000 Epoch 26/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8976 - accuracy: 0.5000 Epoch 27/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8912 - accuracy: 0.4990 Epoch 28/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8850 - accuracy: 0.4960 Epoch 29/100 32/32 [==============================] - 0s 1ms/step - loss: 0.8790 - accuracy: 0.4950 Epoch 30/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8732 - accuracy: 0.4950 Epoch 31/100 32/32 [==============================] - 0s 1ms/step - loss: 0.8678 - accuracy: 0.4870 Epoch 32/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8623 - accuracy: 0.4790 Epoch 33/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8573 - accuracy: 0.4750 Epoch 34/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8525 - accuracy: 0.4730 Epoch 35/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8477 - accuracy: 0.4650 Epoch 36/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8432 - accuracy: 0.4590 Epoch 37/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8388 - accuracy: 0.4560 Epoch 38/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8347 - accuracy: 0.4470 Epoch 39/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8305 - accuracy: 0.4420 Epoch 40/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8266 - accuracy: 0.4380 Epoch 41/100 32/32 [==============================] - 0s 1ms/step - loss: 0.8227 - accuracy: 0.4340 Epoch 42/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8189 - accuracy: 0.4290 Epoch 43/100 32/32 [==============================] - 0s 1ms/step - loss: 0.8151 - accuracy: 0.4280 Epoch 44/100 32/32 [==============================] - 0s 1ms/step - loss: 0.8116 - accuracy: 0.4280 Epoch 45/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8082 - accuracy: 0.4200 Epoch 46/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8049 - accuracy: 0.4160 Epoch 47/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8018 - accuracy: 0.4150 Epoch 48/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7986 - accuracy: 0.4130 Epoch 49/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7956 - accuracy: 0.4140 Epoch 50/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7926 - accuracy: 0.4170 Epoch 51/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7897 - accuracy: 0.4200 Epoch 52/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7870 - accuracy: 0.4210 Epoch 53/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7843 - accuracy: 0.4280 Epoch 54/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7817 - accuracy: 0.4350 Epoch 55/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7791 - accuracy: 0.4480 Epoch 56/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7766 - accuracy: 0.4510 Epoch 57/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7742 - accuracy: 0.4530 Epoch 58/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7718 - accuracy: 0.4530 Epoch 59/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7696 - accuracy: 0.4550 Epoch 60/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7674 - accuracy: 0.4570 Epoch 61/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7654 - accuracy: 0.4560 Epoch 62/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7634 - accuracy: 0.4600 Epoch 63/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7613 - accuracy: 0.4610 Epoch 64/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7595 - accuracy: 0.4640 Epoch 65/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7577 - accuracy: 0.4590 Epoch 66/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7558 - accuracy: 0.4630 Epoch 67/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7541 - accuracy: 0.4620 Epoch 68/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7524 - accuracy: 0.4640 Epoch 69/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7507 - accuracy: 0.4660 Epoch 70/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7491 - accuracy: 0.4660 Epoch 71/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7475 - accuracy: 0.4680 Epoch 72/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7460 - accuracy: 0.4710 Epoch 73/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7445 - accuracy: 0.4720 Epoch 74/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7430 - accuracy: 0.4740 Epoch 75/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7416 - accuracy: 0.4730 Epoch 76/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7403 - accuracy: 0.4750 Epoch 77/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7390 - accuracy: 0.4730 Epoch 78/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7377 - accuracy: 0.4720 Epoch 79/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7365 - accuracy: 0.4730 Epoch 80/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7353 - accuracy: 0.4760 Epoch 81/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7341 - accuracy: 0.4740 Epoch 82/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7330 - accuracy: 0.4750 Epoch 83/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7320 - accuracy: 0.4790 Epoch 84/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7309 - accuracy: 0.4790 Epoch 85/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7299 - accuracy: 0.4810 Epoch 86/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7290 - accuracy: 0.4820 Epoch 87/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7280 - accuracy: 0.4850 Epoch 88/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7271 - accuracy: 0.4850 Epoch 89/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7263 - accuracy: 0.4870 Epoch 90/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7254 - accuracy: 0.4880 Epoch 91/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7246 - accuracy: 0.4900 Epoch 92/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7238 - accuracy: 0.4890 Epoch 93/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7230 - accuracy: 0.4890 Epoch 94/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7223 - accuracy: 0.4860 Epoch 95/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7215 - accuracy: 0.4860 Epoch 96/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7208 - accuracy: 0.4860 Epoch 97/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7202 - accuracy: 0.4880 Epoch 98/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7195 - accuracy: 0.4870 Epoch 99/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7188 - accuracy: 0.4870 Epoch 100/100 32/32 [==============================] - 0s 1ms/step - loss: 0.7181 - accuracy: 0.4850
Okay, our model performs a little worse than guessing.
Let's remind ourselves what our data looks like.
# Check out our data
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.RdYlBu);
And let's see how our model is making predictions on it.
# Check the deicison boundary (blue is blue class, yellow is the crossover, red is red class)
plot_decision_boundary(model_4, X, y)
doing binary classifcation...
Well, it looks like we're getting a straight (linear) line prediction again.
But our data is non-linear (not a straight line)...
What we're going to have to do is add some non-linearity to our model.
To do so, we'll use the activation
parameter in on of our layers.
# Set random seed
tf.random.set_seed(42)
# Create a model with a non-linear activation
model_5 = tf.keras.Sequential([
tf.keras.layers.Dense(1, activation=tf.keras.activations.relu), # can also do activation='relu'
tf.keras.layers.Dense(1) # output layer
])
# Compile the model
model_5.compile(loss=tf.keras.losses.binary_crossentropy,
optimizer=tf.keras.optimizers.Adam(),
metrics=["accuracy"])
# Fit the model
history = model_5.fit(X, y, epochs=100)
Epoch 1/100 32/32 [==============================] - 1s 1ms/step - loss: 1.8377 - accuracy: 0.5000 Epoch 2/100 32/32 [==============================] - 0s 1ms/step - loss: 1.4449 - accuracy: 0.5000 Epoch 3/100 32/32 [==============================] - 0s 2ms/step - loss: 1.3410 - accuracy: 0.5000 Epoch 4/100 32/32 [==============================] - 0s 2ms/step - loss: 1.2678 - accuracy: 0.4770 Epoch 5/100 32/32 [==============================] - 0s 2ms/step - loss: 1.2116 - accuracy: 0.4390 Epoch 6/100 32/32 [==============================] - 0s 2ms/step - loss: 1.1664 - accuracy: 0.4180 Epoch 7/100 32/32 [==============================] - 0s 2ms/step - loss: 1.1294 - accuracy: 0.4250 Epoch 8/100 32/32 [==============================] - 0s 2ms/step - loss: 1.0970 - accuracy: 0.4420 Epoch 9/100 32/32 [==============================] - 0s 2ms/step - loss: 1.0670 - accuracy: 0.4540 Epoch 10/100 32/32 [==============================] - 0s 2ms/step - loss: 1.0407 - accuracy: 0.4550 Epoch 11/100 32/32 [==============================] - 0s 2ms/step - loss: 1.0147 - accuracy: 0.4600 Epoch 12/100 32/32 [==============================] - 0s 2ms/step - loss: 0.9872 - accuracy: 0.4630 Epoch 13/100 32/32 [==============================] - 0s 2ms/step - loss: 0.9579 - accuracy: 0.4620 Epoch 14/100 32/32 [==============================] - 0s 2ms/step - loss: 0.9201 - accuracy: 0.4660 Epoch 15/100 32/32 [==============================] - 0s 2ms/step - loss: 0.8514 - accuracy: 0.4660 Epoch 16/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7888 - accuracy: 0.4720 Epoch 17/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7580 - accuracy: 0.4740 Epoch 18/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7392 - accuracy: 0.4760 Epoch 19/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7273 - accuracy: 0.4850 Epoch 20/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7180 - accuracy: 0.4870 Epoch 21/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7121 - accuracy: 0.4880 Epoch 22/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7070 - accuracy: 0.4880 Epoch 23/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7036 - accuracy: 0.4870 Epoch 24/100 32/32 [==============================] - 0s 2ms/step - loss: 0.7011 - accuracy: 0.4900 Epoch 25/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6993 - accuracy: 0.4860 Epoch 26/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6977 - accuracy: 0.4910 Epoch 27/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6970 - accuracy: 0.4950 Epoch 28/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6959 - accuracy: 0.4970 Epoch 29/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6955 - accuracy: 0.4950 Epoch 30/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6949 - accuracy: 0.4970 Epoch 31/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6946 - accuracy: 0.4990 Epoch 32/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6943 - accuracy: 0.4910 Epoch 33/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6940 - accuracy: 0.5000 Epoch 34/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6939 - accuracy: 0.4910 Epoch 35/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6937 - accuracy: 0.4940 Epoch 36/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4930 Epoch 37/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4950 Epoch 38/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6935 - accuracy: 0.4900 Epoch 39/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4840 Epoch 40/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5000 Epoch 41/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6936 - accuracy: 0.5060 Epoch 42/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4800 Epoch 43/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5050 Epoch 44/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6936 - accuracy: 0.4810 Epoch 45/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4550 Epoch 46/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6932 - accuracy: 0.4920 Epoch 47/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6935 - accuracy: 0.4870 Epoch 48/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.5100 Epoch 49/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6934 - accuracy: 0.4770 Epoch 50/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6934 - accuracy: 0.5000 Epoch 51/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6933 - accuracy: 0.4760 Epoch 52/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6934 - accuracy: 0.5010 Epoch 53/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4880 Epoch 54/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5530 Epoch 55/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5060 Epoch 56/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5200 Epoch 57/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4910 Epoch 58/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4990 Epoch 59/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6938 - accuracy: 0.5000 Epoch 60/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.5000 Epoch 61/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4900 Epoch 62/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4820 Epoch 63/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4700 Epoch 64/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4820 Epoch 65/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4620 Epoch 66/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4760 Epoch 67/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4830 Epoch 68/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6933 - accuracy: 0.4680 Epoch 69/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5010 Epoch 70/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4970 Epoch 71/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4680 Epoch 72/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.5070 Epoch 73/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.5320 Epoch 74/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5290 Epoch 75/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5000 Epoch 76/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4730 Epoch 77/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4870 Epoch 78/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4830 Epoch 79/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4640 Epoch 80/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5040 Epoch 81/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6936 - accuracy: 0.5000 Epoch 82/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5020 Epoch 83/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6936 - accuracy: 0.4840 Epoch 84/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6932 - accuracy: 0.5070 Epoch 85/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5000 Epoch 86/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5000 Epoch 87/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6933 - accuracy: 0.5000 Epoch 88/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4680 Epoch 89/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4590 Epoch 90/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6937 - accuracy: 0.4980 Epoch 91/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4910 Epoch 92/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4850 Epoch 93/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6938 - accuracy: 0.4970 Epoch 94/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4710 Epoch 95/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4720 Epoch 96/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6934 - accuracy: 0.4880 Epoch 97/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6934 - accuracy: 0.4510 Epoch 98/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4640 Epoch 99/100 32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4940 Epoch 100/100 32/32 [==============================] - 0s 1ms/step - loss: 0.6936 - accuracy: 0.5180
Hmm... still not learning...
What we if increased the number of neurons and layers?
Say, 2 hidden layers, with ReLU, pronounced "rel-u", (short for rectified linear unit), activation on the first one, and 4 neurons each?
To see this network in action, check out the TensorFlow Playground demo.
The neural network we're going to recreate with TensorFlow code. See it live at TensorFlow Playground.
Let's try.
# Set random seed
tf.random.set_seed(42)
# Create a model
model_6 = tf.keras.Sequential([
tf.keras.layers.Dense(4, activation=tf.keras.activations.relu), # hidden layer 1, 4 neurons, ReLU activation
tf.keras.layers.Dense(4, activation=tf.keras.activations.relu), # hidden layer 2, 4 neurons, ReLU activation
tf.keras.layers.Dense(1) # ouput layer
])
# Compile the model
model_6.compile(loss=tf.keras.losses.binary_crossentropy,
optimizer=tf.keras.optimizers.Adam(lr=0.001), # Adam's default learning rate is 0.001
metrics=['accuracy'])
# Fit the model
history = model_6.fit(X, y, epochs=100)
/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/adam.py:105: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead. super(Adam, self).__init__(name, **kwargs)
Epoch 1/100 32/32 [==============================] - 1s 1ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 2/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 3/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 4/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 5/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 6/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 7/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 8/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 9/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 10/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 11/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 12/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 13/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 14/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 15/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 16/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 17/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 18/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 19/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 20/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 21/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 22/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 23/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 24/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 25/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 26/100 32/32 [==============================] - 0s 1ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 27/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 28/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 29/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 30/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 31/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 32/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 33/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 34/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 35/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 36/100 32/32 [==============================] - 0s 1ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 37/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 38/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 39/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 40/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 41/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 42/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 43/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 44/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 45/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 46/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 47/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 48/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 49/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 50/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 51/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 52/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 53/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 54/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 55/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 56/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 57/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 58/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 59/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 60/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 61/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 62/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 63/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 64/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 65/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 66/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 67/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 68/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 69/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 70/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 71/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 72/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 73/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 74/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 75/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 76/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 77/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 78/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 79/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 80/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 81/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 82/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 83/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 84/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 85/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 86/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 87/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 88/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 89/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 90/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 91/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 92/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 93/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 94/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 95/100 32/32 [==============================] - 0s 1ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 96/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 97/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 98/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 99/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000 Epoch 100/100 32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
# Evaluate the model
model_6.evaluate(X, y)
32/32 [==============================] - 0s 1ms/step - loss: 7.7125 - accuracy: 0.5000
[7.712474346160889, 0.5]
We're still hitting 50% accuracy, our model is still practically as good as guessing.
How do the predictions look?
# Check out the predictions using 2 hidden layers
plot_decision_boundary(model_6, X, y)
doing binary classifcation...