02. Neural Network Classification with TensorFlow¶

Okay, we've seen how to deal with a regression problem in TensorFlow, let's look at how we can approach a classification problem.

A classification problem involves predicting whether something is one thing or another.

For example, you might want to:

• Predict whether or not someone has heart disease based on their health parameters. This is called binary classification since there are only two options.
• Decide whether a photo of is of food, a person or a dog. This is called multi-class classification since there are more than two options.
• Predict what categories should be assigned to a Wikipedia article. This is called multi-label classification since a single article could have more than one category assigned.

In this notebook, we're going to work through a number of different classification problems with TensorFlow. In other words, taking a set of inputs and predicting what class those set of inputs belong to.

What we're going to cover¶

Specifically, we're going to go through doing the following with TensorFlow:

• Architecture of a classification model
• Input shapes and output shapes
  • X: features/data (inputs)
  • y: labels (outputs)
    • "What class do the inputs belong to?"
• Creating custom data to view and fit
• Steps in modelling for binary and mutliclass classification
  • Creating a model
  • Compiling a model
    • Defining a loss function
    • Setting up an optimizer
      • Finding the best learning rate
    • Creating evaluation metrics
  • Fitting a model (getting it to find patterns in our data)
  • Improving a model
• The power of non-linearity
• Evaluating classification models
  • Visualizng the model ("visualize, visualize, visualize")
  • Looking at training curves
  • Compare predictions to ground truth (using our evaluation metrics)

How you can use this notebook¶

You can read through the descriptions and the code (it should all run, except for the cells which error on purpose), but there's a better option.

Write all of the code yourself.

Yes. I'm serious. Create a new notebook, and rewrite each line by yourself. Investigate it, see if you can break it, why does it break?

You don't have to write the text descriptions but writing the code yourself is a great way to get hands-on experience.

Don't worry if you make mistakes, we all do. The way to get better and make less mistakes is to write more code.

Typical architecture of a classification neural network¶

The word typical is on purpose.

Because the architecture of a classification neural network can widely vary depending on the problem you're working on.

However, there are some fundamentals all deep neural networks contain:

• An input layer.
• Some hidden layers.
• An output layer.

Much of the rest is up to the data analyst creating the model.

The following are some standard values you'll often use in your classification neural networks.

Hyperparameter	Binary Classification	Multiclass classification
Input layer shape	Same as number of features (e.g. 5 for age, sex, height, weight, smoking status in heart disease prediction)	Same as binary classification
Hidden layer(s)	Problem specific, minimum = 1, maximum = unlimited	Same as binary classification
Neurons per hidden layer	Problem specific, generally 10 to 100	Same as binary classification
Output layer shape	1 (one class or the other)	1 per class (e.g. 3 for food, person or dog photo)
Hidden activation	Usually ReLU (rectified linear unit)	Same as binary classification
Output activation	Sigmoid	Softmax
Loss function	Cross entropy (tf.keras.losses.BinaryCrossentropy in TensorFlow)	Cross entropy (tf.keras.losses.CategoricalCrossentropy in TensorFlow)
Optimizer	SGD (stochastic gradient descent), Adam	Same as binary classification

Table 1: Typical architecture of a classification network. Source: Adapted from page 295 of Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow Book by Aurélien Géron

Don't worry if not much of the above makes sense right now, we'll get plenty of experience as we go through this notebook.

Let's start by importing TensorFlow as the common alias tf. For this notebook, make sure you're using version 2.x+.

In [1]:

import tensorflow as tf
print(tf.__version__)

import tensorflow as tf print(tf.__version__)

2.8.0

Creating data to view and fit¶

We could start by importing a classification dataset but let's practice making some of our own classification data.

🔑 Note: It's a common practice to get you and model you build working on a toy (or simple) dataset before moving to your actual problem. Treat it as a rehersal experiment before the actual experiment(s).

Since classification is predicting whether something is one thing or another, let's make some data to reflect that.

To do so, we'll use Scikit-Learn's make_circles() function.

In [2]:

from sklearn.datasets import make_circles

# Make 1000 examples
n_samples = 1000

# Create circles
X, y = make_circles(n_samples, 
                    noise=0.03, 
                    random_state=42)

from sklearn.datasets import make_circles # Make 1000 examples n_samples = 1000 # Create circles X, y = make_circles(n_samples, noise=0.03, random_state=42)

Wonderful, now we've created some data, let's look at the features (X) and labels (y).

In [3]:

# Check out the features
X

# Check out the features X

Out[3]:

array([[ 0.75424625,  0.23148074],
       [-0.75615888,  0.15325888],
       [-0.81539193,  0.17328203],
       ...,
       [-0.13690036, -0.81001183],
       [ 0.67036156, -0.76750154],
       [ 0.28105665,  0.96382443]])

In [4]:

# See the first 10 labels
y[:10]

# See the first 10 labels y[:10]

Out[4]:

array([1, 1, 1, 1, 0, 1, 1, 1, 1, 0])

Okay, we've seen some of our data and labels, how about we move towards visualizing?

🔑 Note: One important step of starting any kind of machine learning project is to become one with the data. And one of the best ways to do this is to visualize the data you're working with as much as possible. The data explorer's motto is "visualize, visualize, visualize".

We'll start with a DataFrame.

In [5]:

# Make dataframe of features and labels
import pandas as pd
circles = pd.DataFrame({"X0":X[:, 0], "X1":X[:, 1], "label":y})
circles.head()

# Make dataframe of features and labels import pandas as pd circles = pd.DataFrame({"X0":X[:, 0], "X1":X[:, 1], "label":y}) circles.head()

Out[5]:

	X0	X1	label
0	0.754246	0.231481	1
1	-0.756159	0.153259	1
2	-0.815392	0.173282	1
3	-0.393731	0.692883	1
4	0.442208	-0.896723	0

What kind of labels are we dealing with?

In [6]:

# Check out the different labels
circles.label.value_counts()

# Check out the different labels circles.label.value_counts()

Out[6]:

1    500
0    500
Name: label, dtype: int64

Alright, looks like we're dealing with a binary classification problem. It's binary because there are only two labels (0 or 1).

If there were more label options (e.g. 0, 1, 2, 3 or 4), it would be called multiclass classification.

Let's take our visualization a step further and plot our data.

In [7]:

# Visualize with a plot
import matplotlib.pyplot as plt
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.RdYlBu);

# Visualize with a plot import matplotlib.pyplot as plt plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.RdYlBu);

Nice! From the plot, can you guess what kind of model we might want to build?

How about we try and build one to classify blue or red dots? As in, a model which is able to distinguish blue from red dots.

🛠 Practice: Before pushing forward, you might want to spend 10 minutes playing around with the TensorFlow Playground. Try adjusting the different hyperparameters you see and click play to see a neural network train. I think you'll find the data very similar to what we've just created.

Input and output shapes¶

One of the most common issues you'll run into when building neural networks is shape mismatches.

More specifically, the shape of the input data and the shape of the output data.

In our case, we want to input X and get our model to predict y.

So let's check out the shapes of X and y.

In [8]:

# Check the shapes of our features and labels
X.shape, y.shape

# Check the shapes of our features and labels X.shape, y.shape

Out[8]:

((1000, 2), (1000,))

Hmm, where do these numbers come from?

In [9]:

# Check how many samples we have
len(X), len(y)

# Check how many samples we have len(X), len(y)

Out[9]:

(1000, 1000)

So we've got as many X values as we do y values, that makes sense.

Let's check out one example of each.

In [10]:

# View the first example of features and labels
X[0], y[0]

# View the first example of features and labels X[0], y[0]

Out[10]:

(array([0.75424625, 0.23148074]), 1)

Alright, so we've got two X features which lead to one y value.

This means our neural network input shape will has to accept a tensor with at least one dimension being two and output a tensor with at least one value.

🤔 Note: y having a shape of (1000,) can seem confusing. However, this is because all y values are actually scalars (single values) and therefore don't have a dimension. For now, think of your output shape as being at least the same value as one example of y (in our case, the output from our neural network has to be at least one value).

Steps in modelling¶

Now we know what data we have as well as the input and output shapes, let's see how we'd build a neural network to model it.

In TensorFlow, there are typically 3 fundamental steps to creating and training a model.

1. Creating a model - piece together the layers of a neural network yourself (using the functional or sequential API) or import a previously built model (known as transfer learning).
2. Compiling a model - defining how a model's performance should be measured (loss/metrics) as well as defining how it should improve (optimizer).
3. Fitting a model - letting the model try to find patterns in the data (how does X get to y).

Let's see these in action using the Sequential API to build a model for our regression data. And then we'll step through each.

In [11]:

# Set random seed
tf.random.set_seed(42)

# 1. Create the model using the Sequential API
model_1 = tf.keras.Sequential([
  tf.keras.layers.Dense(1)
])

# 2. Compile the model
model_1.compile(loss=tf.keras.losses.BinaryCrossentropy(), # binary since we are working with 2 clases (0 & 1)
                optimizer=tf.keras.optimizers.SGD(),
                metrics=['accuracy'])

# 3. Fit the model
model_1.fit(X, y, epochs=5)

# Set random seed tf.random.set_seed(42) # 1. Create the model using the Sequential API model_1 = tf.keras.Sequential([ tf.keras.layers.Dense(1) ]) # 2. Compile the model model_1.compile(loss=tf.keras.losses.BinaryCrossentropy(), # binary since we are working with 2 clases (0 & 1) optimizer=tf.keras.optimizers.SGD(), metrics=['accuracy']) # 3. Fit the model model_1.fit(X, y, epochs=5)

Epoch 1/5
32/32 [==============================] - 0s 1ms/step - loss: 2.8544 - accuracy: 0.4600
Epoch 2/5
32/32 [==============================] - 0s 1ms/step - loss: 0.7131 - accuracy: 0.5430
Epoch 3/5
32/32 [==============================] - 0s 1ms/step - loss: 0.6973 - accuracy: 0.5090
Epoch 4/5
32/32 [==============================] - 0s 2ms/step - loss: 0.6950 - accuracy: 0.5010
Epoch 5/5
32/32 [==============================] - 0s 2ms/step - loss: 0.6942 - accuracy: 0.4830

Out[11]:

<keras.callbacks.History at 0x7fc521b6edd0>

Looking at the accuracy metric, our model performs poorly (50% accuracy on a binary classification problem is the equivalent of guessing), but what if we trained it for longer?

In [12]:

# Train our model for longer (more chances to look at the data)
model_1.fit(X, y, epochs=200, verbose=0) # set verbose=0 to remove training updates
model_1.evaluate(X, y)

# Train our model for longer (more chances to look at the data) model_1.fit(X, y, epochs=200, verbose=0) # set verbose=0 to remove training updates model_1.evaluate(X, y)

32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5000

Out[12]:

[0.6934831142425537, 0.5]

Even after 200 passes of the data, it's still performing as if it's guessing.

What if we added an extra layer and trained for a little longer?

In [13]:

# Set random seed
tf.random.set_seed(42)

# 1. Create the model (same as model_1 but with an extra layer)
model_2 = tf.keras.Sequential([
  tf.keras.layers.Dense(1), # add an extra layer
  tf.keras.layers.Dense(1) 
])

# 2. Compile the model
model_2.compile(loss=tf.keras.losses.BinaryCrossentropy(),
                optimizer=tf.keras.optimizers.SGD(),
                metrics=['accuracy'])

# 3. Fit the model
model_2.fit(X, y, epochs=100, verbose=0) # set verbose=0 to make the output print less

# Set random seed tf.random.set_seed(42) # 1. Create the model (same as model_1 but with an extra layer) model_2 = tf.keras.Sequential([ tf.keras.layers.Dense(1), # add an extra layer tf.keras.layers.Dense(1) ]) # 2. Compile the model model_2.compile(loss=tf.keras.losses.BinaryCrossentropy(), optimizer=tf.keras.optimizers.SGD(), metrics=['accuracy']) # 3. Fit the model model_2.fit(X, y, epochs=100, verbose=0) # set verbose=0 to make the output print less

Out[13]:

<keras.callbacks.History at 0x7fc521a02d90>

In [14]:

# Evaluate the model
model_2.evaluate(X, y)

# Evaluate the model model_2.evaluate(X, y)

32/32 [==============================] - 0s 6ms/step - loss: 0.6933 - accuracy: 0.5000

Out[14]:

[0.6933314800262451, 0.5]

Still not even as good as guessing (~50% accuracy)... hmm...?

Let's remind ourselves of a couple more ways we can use to improve our models.

Improving a model¶

To improve our model, we can alter almost every part of the 3 steps we went through before.

1. Creating a model - here you might want to add more layers, increase the number of hidden units (also called neurons) within each layer, change the activation functions of each layer.
2. Compiling a model - you might want to choose a different optimization function (such as the Adam optimizer, which is usually pretty good for many problems) or perhaps change the learning rate of the optimization function.
3. Fitting a model - perhaps you could fit a model for more epochs (leave it training for longer).

There are many different ways to potentially improve a neural network. Some of the most common include: increasing the number of layers (making the network deeper), increasing the number of hidden units (making the network wider) and changing the learning rate. Because these values are all human-changeable, they're referred to as hyperparameters) and the practice of trying to find the best hyperparameters is referred to as hyperparameter tuning.

How about we try adding more neurons, an extra layer and our friend the Adam optimizer?

Surely doing this will result in predictions better than guessing...

Note: The following message (below this one) can be ignored if you're running TensorFlow 2.8.0+, the error seems to have been fixed.

Note: If you're using TensorFlow 2.7.0+ (but not 2.8.0+) the original code from the following cells may have caused some errors. They've since been updated to fix those errors. You can see explanations on what happened at the following resources:

• Example Colab Notebook
• TensorFlow for Deep Learning GitHub Discussion on TensorFlow 2.7.0 breaking changes

In [15]:

# Set random seed
tf.random.set_seed(42)

# 1. Create the model (this time 3 layers)
model_3 = tf.keras.Sequential([
  # Before TensorFlow 2.7.0
  # tf.keras.layers.Dense(100), # add 100 dense neurons

  # With TensorFlow 2.7.0
  # tf.keras.layers.Dense(100, input_shape=(None, 1)), # add 100 dense neurons

  ## After TensorFlow 2.8.0 ##
  tf.keras.layers.Dense(100), # add 100 dense neurons
  tf.keras.layers.Dense(10), # add another layer with 10 neurons
  tf.keras.layers.Dense(1)
])

# 2. Compile the model
model_3.compile(loss=tf.keras.losses.BinaryCrossentropy(),
                optimizer=tf.keras.optimizers.Adam(), # use Adam instead of SGD
                metrics=['accuracy'])

# 3. Fit the model
model_3.fit(X, y, epochs=100, verbose=1) # fit for 100 passes of the data

# Set random seed tf.random.set_seed(42) # 1. Create the model (this time 3 layers) model_3 = tf.keras.Sequential([ # Before TensorFlow 2.7.0 # tf.keras.layers.Dense(100), # add 100 dense neurons # With TensorFlow 2.7.0 # tf.keras.layers.Dense(100, input_shape=(None, 1)), # add 100 dense neurons ## After TensorFlow 2.8.0 ## tf.keras.layers.Dense(100), # add 100 dense neurons tf.keras.layers.Dense(10), # add another layer with 10 neurons tf.keras.layers.Dense(1) ]) # 2. Compile the model model_3.compile(loss=tf.keras.losses.BinaryCrossentropy(), optimizer=tf.keras.optimizers.Adam(), # use Adam instead of SGD metrics=['accuracy']) # 3. Fit the model model_3.fit(X, y, epochs=100, verbose=1) # fit for 100 passes of the data

Epoch 1/100
32/32 [==============================] - 2s 4ms/step - loss: 2.9658 - accuracy: 0.4490
Epoch 2/100
32/32 [==============================] - 0s 3ms/step - loss: 0.7229 - accuracy: 0.4390
Epoch 3/100
32/32 [==============================] - 0s 8ms/step - loss: 0.6971 - accuracy: 0.4700
Epoch 4/100
32/32 [==============================] - 0s 8ms/step - loss: 0.6953 - accuracy: 0.4620
Epoch 5/100
32/32 [==============================] - 0s 6ms/step - loss: 0.6943 - accuracy: 0.4750
Epoch 6/100
32/32 [==============================] - 0s 5ms/step - loss: 0.6941 - accuracy: 0.4750
Epoch 7/100
32/32 [==============================] - 0s 6ms/step - loss: 0.6939 - accuracy: 0.5040
Epoch 8/100
32/32 [==============================] - 0s 6ms/step - loss: 0.6946 - accuracy: 0.4980
Epoch 9/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6943 - accuracy: 0.4880
Epoch 10/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6944 - accuracy: 0.4480
Epoch 11/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6952 - accuracy: 0.4450
Epoch 12/100
32/32 [==============================] - 0s 6ms/step - loss: 0.6946 - accuracy: 0.4890
Epoch 13/100
32/32 [==============================] - 0s 5ms/step - loss: 0.6956 - accuracy: 0.4920
Epoch 14/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6950 - accuracy: 0.5140
Epoch 15/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6956 - accuracy: 0.4590
Epoch 16/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6943 - accuracy: 0.4820
Epoch 17/100
32/32 [==============================] - 0s 5ms/step - loss: 0.6975 - accuracy: 0.4950
Epoch 18/100
32/32 [==============================] - 0s 5ms/step - loss: 0.6945 - accuracy: 0.4550
Epoch 19/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6953 - accuracy: 0.5330
Epoch 20/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6956 - accuracy: 0.4560
Epoch 21/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6955 - accuracy: 0.4920
Epoch 22/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6948 - accuracy: 0.4350
Epoch 23/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6944 - accuracy: 0.4990
Epoch 24/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6963 - accuracy: 0.4690
Epoch 25/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6973 - accuracy: 0.4970
Epoch 26/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6973 - accuracy: 0.4830
Epoch 27/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6963 - accuracy: 0.5040
Epoch 28/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6946 - accuracy: 0.4680
Epoch 29/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6973 - accuracy: 0.4790
Epoch 30/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6998 - accuracy: 0.4800
Epoch 31/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6950 - accuracy: 0.5030
Epoch 32/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6965 - accuracy: 0.4760
Epoch 33/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6957 - accuracy: 0.4570
Epoch 34/100
32/32 [==============================] - 0s 7ms/step - loss: 0.6973 - accuracy: 0.4800
Epoch 35/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6960 - accuracy: 0.5050
Epoch 36/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6946 - accuracy: 0.4680
Epoch 37/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6952 - accuracy: 0.4870
Epoch 38/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6975 - accuracy: 0.4980
Epoch 39/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6949 - accuracy: 0.4690
Epoch 40/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6965 - accuracy: 0.4990
Epoch 41/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6979 - accuracy: 0.4890
Epoch 42/100
32/32 [==============================] - 0s 5ms/step - loss: 0.6995 - accuracy: 0.4700
Epoch 43/100
32/32 [==============================] - 0s 5ms/step - loss: 0.6957 - accuracy: 0.5030
Epoch 44/100
32/32 [==============================] - 0s 5ms/step - loss: 0.6981 - accuracy: 0.4830
Epoch 45/100
32/32 [==============================] - 0s 5ms/step - loss: 0.6964 - accuracy: 0.5120
Epoch 46/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6949 - accuracy: 0.4730
Epoch 47/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6974 - accuracy: 0.5170
Epoch 48/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6966 - accuracy: 0.4840
Epoch 49/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6976 - accuracy: 0.4870
Epoch 50/100
32/32 [==============================] - 0s 4ms/step - loss: 0.6972 - accuracy: 0.4610
Epoch 51/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6980 - accuracy: 0.4800
Epoch 52/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6962 - accuracy: 0.5020
Epoch 53/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6972 - accuracy: 0.4940
Epoch 54/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6993 - accuracy: 0.5020
Epoch 55/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6953 - accuracy: 0.5220
Epoch 56/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6966 - accuracy: 0.4930
Epoch 57/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6981 - accuracy: 0.4720
Epoch 58/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6968 - accuracy: 0.4980
Epoch 59/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7004 - accuracy: 0.4890
Epoch 60/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7011 - accuracy: 0.4870
Epoch 61/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6996 - accuracy: 0.4680
Epoch 62/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6965 - accuracy: 0.4980
Epoch 63/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6948 - accuracy: 0.4860
Epoch 64/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6960 - accuracy: 0.4940
Epoch 65/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6996 - accuracy: 0.4810
Epoch 66/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6965 - accuracy: 0.5010
Epoch 67/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6977 - accuracy: 0.4500
Epoch 68/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6958 - accuracy: 0.5020
Epoch 69/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6953 - accuracy: 0.5230
Epoch 70/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6988 - accuracy: 0.4680
Epoch 71/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6975 - accuracy: 0.4650
Epoch 72/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6966 - accuracy: 0.5010
Epoch 73/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6963 - accuracy: 0.4930
Epoch 74/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6960 - accuracy: 0.5130
Epoch 75/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6963 - accuracy: 0.4980
Epoch 76/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6980 - accuracy: 0.4850
Epoch 77/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6971 - accuracy: 0.5020
Epoch 78/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6972 - accuracy: 0.4720
Epoch 79/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7009 - accuracy: 0.5110
Epoch 80/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6980 - accuracy: 0.4930
Epoch 81/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6980 - accuracy: 0.5030
Epoch 82/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6968 - accuracy: 0.4830
Epoch 83/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7002 - accuracy: 0.4480
Epoch 84/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6963 - accuracy: 0.4970
Epoch 85/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6967 - accuracy: 0.4590
Epoch 86/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7002 - accuracy: 0.4900
Epoch 87/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6977 - accuracy: 0.4550
Epoch 88/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6981 - accuracy: 0.4590
Epoch 89/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6979 - accuracy: 0.4750
Epoch 90/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6981 - accuracy: 0.4620
Epoch 91/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6984 - accuracy: 0.4980
Epoch 92/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6988 - accuracy: 0.4760
Epoch 93/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6982 - accuracy: 0.4690
Epoch 94/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7005 - accuracy: 0.4800
Epoch 95/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6974 - accuracy: 0.4520
Epoch 96/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6980 - accuracy: 0.4790
Epoch 97/100
32/32 [==============================] - 0s 3ms/step - loss: 0.6973 - accuracy: 0.4580
Epoch 98/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6989 - accuracy: 0.4800
Epoch 99/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6976 - accuracy: 0.5090
Epoch 100/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6966 - accuracy: 0.4870

Out[15]:

<keras.callbacks.History at 0x7fc5208e3cd0>

Still!

We've pulled out a few tricks but our model isn't even doing better than guessing.

Let's make some visualizations to see what's happening.

🔑 Note: Whenever your model is performing strangely or there's something going on with your data you're not quite sure of, remember these three words: visualize, visualize, visualize. Inspect your data, inspect your model, inpsect your model's predictions.

To visualize our model's predictions we're going to create a function plot_decision_boundary() which:

• Takes in a trained model, features (X) and labels (y).
• Creates a meshgrid of the different X values.
• Makes predictions across the meshgrid.
• Plots the predictions as well as a line between the different zones (where each unique class falls).

If this sounds confusing, let's see it in code and then see the output.

🔑 Note: If you're ever unsure of what a function does, try unraveling it and writing it line by line for yourself to see what it does. Break it into small parts and see what each part outputs.

In [16]:

import numpy as np

def plot_decision_boundary(model, X, y):
  """
  Plots the decision boundary created by a model predicting on X.
  This function has been adapted from two phenomenal resources:
   1. CS231n - https://cs231n.github.io/neural-networks-case-study/
   2. Made with ML basics - https://github.com/GokuMohandas/MadeWithML/blob/main/notebooks/08_Neural_Networks.ipynb
  """
  # Define the axis boundaries of the plot and create a meshgrid
  x_min, x_max = X[:, 0].min() - 0.1, X[:, 0].max() + 0.1
  y_min, y_max = X[:, 1].min() - 0.1, X[:, 1].max() + 0.1
  xx, yy = np.meshgrid(np.linspace(x_min, x_max, 100),
                       np.linspace(y_min, y_max, 100))
  
  # Create X values (we're going to predict on all of these)
  x_in = np.c_[xx.ravel(), yy.ravel()] # stack 2D arrays together: https://numpy.org/devdocs/reference/generated/numpy.c_.html
  
  # Make predictions using the trained model
  y_pred = model.predict(x_in)

  # Check for multi-class
  if model.output_shape[-1] > 1: # checks the final dimension of the model's output shape, if this is > (greater than) 1, it's multi-class 
    print("doing multiclass classification...")
    # We have to reshape our predictions to get them ready for plotting
    y_pred = np.argmax(y_pred, axis=1).reshape(xx.shape)
  else:
    print("doing binary classifcation...")
    y_pred = np.round(np.max(y_pred, axis=1)).reshape(xx.shape)
  
  # Plot decision boundary
  plt.contourf(xx, yy, y_pred, cmap=plt.cm.RdYlBu, alpha=0.7)
  plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.RdYlBu)
  plt.xlim(xx.min(), xx.max())
  plt.ylim(yy.min(), yy.max())

import numpy as np def plot_decision_boundary(model, X, y): """ Plots the decision boundary created by a model predicting on X. This function has been adapted from two phenomenal resources: 1. CS231n - https://cs231n.github.io/neural-networks-case-study/ 2. Made with ML basics - https://github.com/GokuMohandas/MadeWithML/blob/main/notebooks/08_Neural_Networks.ipynb """ # Define the axis boundaries of the plot and create a meshgrid x_min, x_max = X[:, 0].min() - 0.1, X[:, 0].max() + 0.1 y_min, y_max = X[:, 1].min() - 0.1, X[:, 1].max() + 0.1 xx, yy = np.meshgrid(np.linspace(x_min, x_max, 100), np.linspace(y_min, y_max, 100)) # Create X values (we're going to predict on all of these) x_in = np.c_[xx.ravel(), yy.ravel()] # stack 2D arrays together: https://numpy.org/devdocs/reference/generated/numpy.c_.html # Make predictions using the trained model y_pred = model.predict(x_in) # Check for multi-class if model.output_shape[-1] > 1: # checks the final dimension of the model's output shape, if this is > (greater than) 1, it's multi-class print("doing multiclass classification...") # We have to reshape our predictions to get them ready for plotting y_pred = np.argmax(y_pred, axis=1).reshape(xx.shape) else: print("doing binary classifcation...") y_pred = np.round(np.max(y_pred, axis=1)).reshape(xx.shape) # Plot decision boundary plt.contourf(xx, yy, y_pred, cmap=plt.cm.RdYlBu, alpha=0.7) plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.RdYlBu) plt.xlim(xx.min(), xx.max()) plt.ylim(yy.min(), yy.max())

Now we've got a function to plot our model's decision boundary (the cut off point its making between red and blue dots), let's try it out.

In [18]:

# Check out the predictions our model is making
plot_decision_boundary(model_3, X, y)

# Check out the predictions our model is making plot_decision_boundary(model_3, X, y)

doing binary classifcation...

Looks like our model is trying to draw a straight line through the data.

What's wrong with doing this?

The main issue is our data isn't separable by a straight line.

In a regression problem, our model might work. In fact, let's try it.

In [19]:

# Set random seed
tf.random.set_seed(42)

# Create some regression data
X_regression = np.arange(0, 1000, 5)
y_regression = np.arange(100, 1100, 5)

# Split it into training and test sets
X_reg_train = X_regression[:150]
X_reg_test = X_regression[150:]
y_reg_train = y_regression[:150]
y_reg_test = y_regression[150:]

# Fit our model to the data
# Note: Before TensorFlow 2.7.0, this line would work
# model_3.fit(X_reg_train, y_reg_train, epochs=100)

# After TensorFlow 2.7.0, see here for more: https://github.com/mrdbourke/tensorflow-deep-learning/discussions/278
model_3.fit(tf.expand_dims(X_reg_train, axis=-1), 
            y_reg_train,
            epochs=100)

# Set random seed tf.random.set_seed(42) # Create some regression data X_regression = np.arange(0, 1000, 5) y_regression = np.arange(100, 1100, 5) # Split it into training and test sets X_reg_train = X_regression[:150] X_reg_test = X_regression[150:] y_reg_train = y_regression[:150] y_reg_test = y_regression[150:] # Fit our model to the data # Note: Before TensorFlow 2.7.0, this line would work # model_3.fit(X_reg_train, y_reg_train, epochs=100) # After TensorFlow 2.7.0, see here for more: https://github.com/mrdbourke/tensorflow-deep-learning/discussions/278 model_3.fit(tf.expand_dims(X_reg_train, axis=-1), y_reg_train, epochs=100)

Epoch 1/100

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-19-5dd5867236b4> in <module>()
     19 model_3.fit(tf.expand_dims(X_reg_train, axis=-1), 
     20             y_reg_train,
---> 21             epochs=100)

/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py in autograph_handler(*args, **kwargs)
   1145           except Exception as e:  # pylint:disable=broad-except
   1146             if hasattr(e, "ag_error_metadata"):
-> 1147               raise e.ag_error_metadata.to_exception(e)
   1148             else:
   1149               raise

ValueError: in user code:

    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1021, in train_function  *
        return step_function(self, iterator)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1010, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1000, in run_step  **
        outputs = model.train_step(data)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 859, in train_step
        y_pred = self(x, training=True)
    File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
        raise e.with_traceback(filtered_tb) from None
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/input_spec.py", line 249, in assert_input_compatibility
        f'Input {input_index} of layer "{layer_name}" is '

    ValueError: Exception encountered when calling layer "sequential_2" (type Sequential).
    
    Input 0 of layer "dense_3" is incompatible with the layer: expected axis -1 of input shape to have value 2, but received input with shape (None, 1)
    
    Call arguments received:
      • inputs=tf.Tensor(shape=(None, 1), dtype=int64)
      • training=True
      • mask=None

In [20]:

model_3.summary()

model_3.summary()

Model: "sequential_2"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_3 (Dense)             (None, 100)               300       
                                                                 
 dense_4 (Dense)             (None, 10)                1010      
                                                                 
 dense_5 (Dense)             (None, 1)                 11        
                                                                 
=================================================================
Total params: 1,321
Trainable params: 1,321
Non-trainable params: 0
_________________________________________________________________

Oh wait... we compiled our model for a binary classification problem.

No trouble, we can recreate it for a regression problem.

In [21]:

# Setup random seed
tf.random.set_seed(42)

# Recreate the model
model_3 = tf.keras.Sequential([
  tf.keras.layers.Dense(100),
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)
])

# Change the loss and metrics of our compiled model
model_3.compile(loss=tf.keras.losses.mae, # change the loss function to be regression-specific
                optimizer=tf.keras.optimizers.Adam(),
                metrics=['mae']) # change the metric to be regression-specific

# Fit the recompiled model
model_3.fit(tf.expand_dims(X_reg_train, axis=-1), 
            y_reg_train, 
            epochs=100)

# Setup random seed tf.random.set_seed(42) # Recreate the model model_3 = tf.keras.Sequential([ tf.keras.layers.Dense(100), tf.keras.layers.Dense(10), tf.keras.layers.Dense(1) ]) # Change the loss and metrics of our compiled model model_3.compile(loss=tf.keras.losses.mae, # change the loss function to be regression-specific optimizer=tf.keras.optimizers.Adam(), metrics=['mae']) # change the metric to be regression-specific # Fit the recompiled model model_3.fit(tf.expand_dims(X_reg_train, axis=-1), y_reg_train, epochs=100)

Epoch 1/100
5/5 [==============================] - 0s 3ms/step - loss: 248.2155 - mae: 248.2155
Epoch 2/100
5/5 [==============================] - 0s 3ms/step - loss: 138.9005 - mae: 138.9005
Epoch 3/100
5/5 [==============================] - 0s 3ms/step - loss: 53.1039 - mae: 53.1039
Epoch 4/100
5/5 [==============================] - 0s 4ms/step - loss: 73.5170 - mae: 73.5170
Epoch 5/100
5/5 [==============================] - 0s 4ms/step - loss: 71.2358 - mae: 71.2358
Epoch 6/100
5/5 [==============================] - 0s 3ms/step - loss: 47.0040 - mae: 47.0040
Epoch 7/100
5/5 [==============================] - 0s 3ms/step - loss: 45.9386 - mae: 45.9386
Epoch 8/100
5/5 [==============================] - 0s 3ms/step - loss: 42.3638 - mae: 42.3638
Epoch 9/100
5/5 [==============================] - 0s 3ms/step - loss: 43.6831 - mae: 43.6831
Epoch 10/100
5/5 [==============================] - 0s 3ms/step - loss: 42.6198 - mae: 42.6198
Epoch 11/100
5/5 [==============================] - 0s 5ms/step - loss: 42.4797 - mae: 42.4797
Epoch 12/100
5/5 [==============================] - 0s 3ms/step - loss: 41.5537 - mae: 41.5537
Epoch 13/100
5/5 [==============================] - 0s 4ms/step - loss: 42.0972 - mae: 42.0972
Epoch 14/100
5/5 [==============================] - 0s 3ms/step - loss: 41.8647 - mae: 41.8647
Epoch 15/100
5/5 [==============================] - 0s 3ms/step - loss: 41.5342 - mae: 41.5342
Epoch 16/100
5/5 [==============================] - 0s 3ms/step - loss: 41.4028 - mae: 41.4028
Epoch 17/100
5/5 [==============================] - 0s 3ms/step - loss: 41.6887 - mae: 41.6887
Epoch 18/100
5/5 [==============================] - 0s 4ms/step - loss: 41.6137 - mae: 41.6137
Epoch 19/100
5/5 [==============================] - 0s 3ms/step - loss: 41.2796 - mae: 41.2796
Epoch 20/100
5/5 [==============================] - 0s 3ms/step - loss: 41.1947 - mae: 41.1947
Epoch 21/100
5/5 [==============================] - 0s 3ms/step - loss: 41.2130 - mae: 41.2130
Epoch 22/100
5/5 [==============================] - 0s 4ms/step - loss: 41.0893 - mae: 41.0893
Epoch 23/100
5/5 [==============================] - 0s 4ms/step - loss: 41.2019 - mae: 41.2019
Epoch 24/100
5/5 [==============================] - 0s 4ms/step - loss: 40.9989 - mae: 40.9989
Epoch 25/100
5/5 [==============================] - 0s 4ms/step - loss: 41.0131 - mae: 41.0131
Epoch 26/100
5/5 [==============================] - 0s 3ms/step - loss: 41.0654 - mae: 41.0654
Epoch 27/100
5/5 [==============================] - 0s 4ms/step - loss: 40.8764 - mae: 40.8764
Epoch 28/100
5/5 [==============================] - 0s 3ms/step - loss: 41.0545 - mae: 41.0545
Epoch 29/100
5/5 [==============================] - 0s 3ms/step - loss: 41.0480 - mae: 41.0480
Epoch 30/100
5/5 [==============================] - 0s 3ms/step - loss: 40.8807 - mae: 40.8807
Epoch 31/100
5/5 [==============================] - 0s 3ms/step - loss: 41.2695 - mae: 41.2695
Epoch 32/100
5/5 [==============================] - 0s 3ms/step - loss: 40.9949 - mae: 40.9949
Epoch 33/100
5/5 [==============================] - 0s 3ms/step - loss: 41.0760 - mae: 41.0760
Epoch 34/100
5/5 [==============================] - 0s 3ms/step - loss: 41.2471 - mae: 41.2471
Epoch 35/100
5/5 [==============================] - 0s 3ms/step - loss: 40.6102 - mae: 40.6102
Epoch 36/100
5/5 [==============================] - 0s 3ms/step - loss: 41.1093 - mae: 41.1093
Epoch 37/100
5/5 [==============================] - 0s 3ms/step - loss: 40.8191 - mae: 40.8191
Epoch 38/100
5/5 [==============================] - 0s 3ms/step - loss: 40.2485 - mae: 40.2485
Epoch 39/100
5/5 [==============================] - 0s 3ms/step - loss: 41.0625 - mae: 41.0625
Epoch 40/100
5/5 [==============================] - 0s 3ms/step - loss: 40.5311 - mae: 40.5311
Epoch 41/100
5/5 [==============================] - 0s 3ms/step - loss: 40.5497 - mae: 40.5497
Epoch 42/100
5/5 [==============================] - 0s 3ms/step - loss: 40.4322 - mae: 40.4322
Epoch 43/100
5/5 [==============================] - 0s 3ms/step - loss: 40.5367 - mae: 40.5367
Epoch 44/100
5/5 [==============================] - 0s 3ms/step - loss: 40.2487 - mae: 40.2487
Epoch 45/100
5/5 [==============================] - 0s 3ms/step - loss: 40.5152 - mae: 40.5152
Epoch 46/100
5/5 [==============================] - 0s 3ms/step - loss: 40.3702 - mae: 40.3702
Epoch 47/100
5/5 [==============================] - 0s 3ms/step - loss: 40.4769 - mae: 40.4769
Epoch 48/100
5/5 [==============================] - 0s 4ms/step - loss: 40.1532 - mae: 40.1532
Epoch 49/100
5/5 [==============================] - 0s 3ms/step - loss: 40.7291 - mae: 40.7291
Epoch 50/100
5/5 [==============================] - 0s 3ms/step - loss: 40.1536 - mae: 40.1536
Epoch 51/100
5/5 [==============================] - 0s 4ms/step - loss: 40.2711 - mae: 40.2711
Epoch 52/100
5/5 [==============================] - 0s 3ms/step - loss: 40.6572 - mae: 40.6572
Epoch 53/100
5/5 [==============================] - 0s 3ms/step - loss: 40.6573 - mae: 40.6573
Epoch 54/100
5/5 [==============================] - 0s 3ms/step - loss: 40.6894 - mae: 40.6894
Epoch 55/100
5/5 [==============================] - 0s 4ms/step - loss: 41.2771 - mae: 41.2771
Epoch 56/100
5/5 [==============================] - 0s 3ms/step - loss: 41.8519 - mae: 41.8519
Epoch 57/100
5/5 [==============================] - 0s 3ms/step - loss: 40.7903 - mae: 40.7903
Epoch 58/100
5/5 [==============================] - 0s 3ms/step - loss: 40.3128 - mae: 40.3128
Epoch 59/100
5/5 [==============================] - 0s 3ms/step - loss: 40.7198 - mae: 40.7198
Epoch 60/100
5/5 [==============================] - 0s 3ms/step - loss: 40.1478 - mae: 40.1478
Epoch 61/100
5/5 [==============================] - 0s 3ms/step - loss: 40.1116 - mae: 40.1116
Epoch 62/100
5/5 [==============================] - 0s 4ms/step - loss: 40.7800 - mae: 40.7800
Epoch 63/100
5/5 [==============================] - 0s 3ms/step - loss: 39.7242 - mae: 39.7242
Epoch 64/100
5/5 [==============================] - 0s 3ms/step - loss: 40.1465 - mae: 40.1465
Epoch 65/100
5/5 [==============================] - 0s 3ms/step - loss: 39.6887 - mae: 39.6887
Epoch 66/100
5/5 [==============================] - 0s 3ms/step - loss: 40.2840 - mae: 40.2840
Epoch 67/100
5/5 [==============================] - 0s 3ms/step - loss: 39.5541 - mae: 39.5541
Epoch 68/100
5/5 [==============================] - 0s 3ms/step - loss: 39.7378 - mae: 39.7378
Epoch 69/100
5/5 [==============================] - 0s 3ms/step - loss: 39.9784 - mae: 39.9784
Epoch 70/100
5/5 [==============================] - 0s 3ms/step - loss: 40.0016 - mae: 40.0016
Epoch 71/100
5/5 [==============================] - 0s 3ms/step - loss: 40.0913 - mae: 40.0913
Epoch 72/100
5/5 [==============================] - 0s 3ms/step - loss: 39.2547 - mae: 39.2547
Epoch 73/100
5/5 [==============================] - 0s 3ms/step - loss: 39.6828 - mae: 39.6828
Epoch 74/100
5/5 [==============================] - 0s 3ms/step - loss: 39.5373 - mae: 39.5373
Epoch 75/100
5/5 [==============================] - 0s 3ms/step - loss: 39.6265 - mae: 39.6265
Epoch 76/100
5/5 [==============================] - 0s 3ms/step - loss: 39.3110 - mae: 39.3110
Epoch 77/100
5/5 [==============================] - 0s 3ms/step - loss: 39.1599 - mae: 39.1599
Epoch 78/100
5/5 [==============================] - 0s 3ms/step - loss: 39.7550 - mae: 39.7550
Epoch 79/100
5/5 [==============================] - 0s 3ms/step - loss: 39.2542 - mae: 39.2542
Epoch 80/100
5/5 [==============================] - 0s 3ms/step - loss: 38.6968 - mae: 38.6968
Epoch 81/100
5/5 [==============================] - 0s 4ms/step - loss: 39.5442 - mae: 39.5442
Epoch 82/100
5/5 [==============================] - 0s 3ms/step - loss: 39.8686 - mae: 39.8686
Epoch 83/100
5/5 [==============================] - 0s 3ms/step - loss: 39.1693 - mae: 39.1693
Epoch 84/100
5/5 [==============================] - 0s 3ms/step - loss: 38.8840 - mae: 38.8840
Epoch 85/100
5/5 [==============================] - 0s 3ms/step - loss: 38.8887 - mae: 38.8887
Epoch 86/100
5/5 [==============================] - 0s 4ms/step - loss: 38.6614 - mae: 38.6614
Epoch 87/100
5/5 [==============================] - 0s 3ms/step - loss: 38.8399 - mae: 38.8399
Epoch 88/100
5/5 [==============================] - 0s 3ms/step - loss: 38.6604 - mae: 38.6604
Epoch 89/100
5/5 [==============================] - 0s 3ms/step - loss: 38.7559 - mae: 38.7559
Epoch 90/100
5/5 [==============================] - 0s 3ms/step - loss: 38.5442 - mae: 38.5442
Epoch 91/100
5/5 [==============================] - 0s 3ms/step - loss: 38.3247 - mae: 38.3247
Epoch 92/100
5/5 [==============================] - 0s 3ms/step - loss: 38.8431 - mae: 38.8431
Epoch 93/100
5/5 [==============================] - 0s 3ms/step - loss: 39.1137 - mae: 39.1137
Epoch 94/100
5/5 [==============================] - 0s 3ms/step - loss: 38.1463 - mae: 38.1463
Epoch 95/100
5/5 [==============================] - 0s 4ms/step - loss: 38.3998 - mae: 38.3998
Epoch 96/100
5/5 [==============================] - 0s 3ms/step - loss: 38.5599 - mae: 38.5599
Epoch 97/100
5/5 [==============================] - 0s 3ms/step - loss: 38.1038 - mae: 38.1038
Epoch 98/100
5/5 [==============================] - 0s 4ms/step - loss: 39.0081 - mae: 39.0081
Epoch 99/100
5/5 [==============================] - 0s 4ms/step - loss: 38.3056 - mae: 38.3056
Epoch 100/100
5/5 [==============================] - 0s 3ms/step - loss: 37.9976 - mae: 37.9976

Out[21]:

<keras.callbacks.History at 0x7fc520608fd0>

Okay, it seems like our model is learning something (the mae value trends down with each epoch), let's plot its predictions.

In [22]:

# Make predictions with our trained model
y_reg_preds = model_3.predict(y_reg_test)

# Plot the model's predictions against our regression data
plt.figure(figsize=(10, 7))
plt.scatter(X_reg_train, y_reg_train, c='b', label='Training data')
plt.scatter(X_reg_test, y_reg_test, c='g', label='Testing data')
plt.scatter(X_reg_test, y_reg_preds.squeeze(), c='r', label='Predictions')
plt.legend();

# Make predictions with our trained model y_reg_preds = model_3.predict(y_reg_test) # Plot the model's predictions against our regression data plt.figure(figsize=(10, 7)) plt.scatter(X_reg_train, y_reg_train, c='b', label='Training data') plt.scatter(X_reg_test, y_reg_test, c='g', label='Testing data') plt.scatter(X_reg_test, y_reg_preds.squeeze(), c='r', label='Predictions') plt.legend();

Okay, the predictions aren't perfect (if the predictions were perfect, the red would line up with the green), but they look better than complete guessing.

So this means our model must be learning something...

There must be something we're missing out on for our classification problem.

The missing piece: Non-linearity¶

Okay, so we saw our neural network can model straight lines (with ability a little bit better than guessing).

What about non-straight (non-linear) lines?

If we're going to model our classification data (the red and clue circles), we're going to need some non-linear lines.

🔨 Practice: Before we get to the next steps, I'd encourage you to play around with the TensorFlow Playground (check out what the data has in common with our own classification data) for 10-minutes. In particular the tab which says "activation". Once you're done, come back.

Did you try out the activation options? If so, what did you find?

If you didn't, don't worry, let's see it in code.

We're going to replicate the neural network you can see at this link: TensorFlow Playground.

The neural network we're going to recreate with TensorFlow code. See it live at TensorFlow Playground.

The main change we'll add to models we've built before is the use of the activation keyword.

In [23]:

# Set the random seed
tf.random.set_seed(42)

# Create the model
model_4 = tf.keras.Sequential([
  tf.keras.layers.Dense(1, activation=tf.keras.activations.linear), # 1 hidden layer with linear activation
  tf.keras.layers.Dense(1) # output layer
])

# Compile the model
model_4.compile(loss=tf.keras.losses.binary_crossentropy,
                optimizer=tf.keras.optimizers.Adam(learning_rate=0.001), # note: "lr" used to be what was used, now "learning_rate" is favoured
                metrics=["accuracy"])

# Fit the model
history = model_4.fit(X, y, epochs=100)

# Set the random seed tf.random.set_seed(42) # Create the model model_4 = tf.keras.Sequential([ tf.keras.layers.Dense(1, activation=tf.keras.activations.linear), # 1 hidden layer with linear activation tf.keras.layers.Dense(1) # output layer ]) # Compile the model model_4.compile(loss=tf.keras.losses.binary_crossentropy, optimizer=tf.keras.optimizers.Adam(learning_rate=0.001), # note: "lr" used to be what was used, now "learning_rate" is favoured metrics=["accuracy"]) # Fit the model history = model_4.fit(X, y, epochs=100)

Epoch 1/100
32/32 [==============================] - 1s 2ms/step - loss: 4.2380 - accuracy: 0.5000
Epoch 2/100
32/32 [==============================] - 0s 2ms/step - loss: 4.0223 - accuracy: 0.5000
Epoch 3/100
32/32 [==============================] - 0s 2ms/step - loss: 3.8296 - accuracy: 0.5000
Epoch 4/100
32/32 [==============================] - 0s 2ms/step - loss: 3.7654 - accuracy: 0.5000
Epoch 5/100
32/32 [==============================] - 0s 2ms/step - loss: 3.6464 - accuracy: 0.5000
Epoch 6/100
32/32 [==============================] - 0s 2ms/step - loss: 3.4960 - accuracy: 0.5000
Epoch 7/100
32/32 [==============================] - 0s 2ms/step - loss: 3.3804 - accuracy: 0.5000
Epoch 8/100
32/32 [==============================] - 0s 2ms/step - loss: 3.2279 - accuracy: 0.5000
Epoch 9/100
32/32 [==============================] - 0s 2ms/step - loss: 2.7024 - accuracy: 0.5000
Epoch 10/100
32/32 [==============================] - 0s 1ms/step - loss: 2.4002 - accuracy: 0.5000
Epoch 11/100
32/32 [==============================] - 0s 1ms/step - loss: 2.1984 - accuracy: 0.5000
Epoch 12/100
32/32 [==============================] - 0s 2ms/step - loss: 1.3257 - accuracy: 0.5000
Epoch 13/100
32/32 [==============================] - 0s 2ms/step - loss: 1.0542 - accuracy: 0.5000
Epoch 14/100
32/32 [==============================] - 0s 2ms/step - loss: 1.0261 - accuracy: 0.5000
Epoch 15/100
32/32 [==============================] - 0s 2ms/step - loss: 1.0069 - accuracy: 0.5000
Epoch 16/100
32/32 [==============================] - 0s 2ms/step - loss: 0.9914 - accuracy: 0.5000
Epoch 17/100
32/32 [==============================] - 0s 2ms/step - loss: 0.9776 - accuracy: 0.5000
Epoch 18/100
32/32 [==============================] - 0s 1ms/step - loss: 0.9656 - accuracy: 0.5000
Epoch 19/100
32/32 [==============================] - 0s 1ms/step - loss: 0.9549 - accuracy: 0.5000
Epoch 20/100
32/32 [==============================] - 0s 1ms/step - loss: 0.9448 - accuracy: 0.5000
Epoch 21/100
32/32 [==============================] - 0s 2ms/step - loss: 0.9357 - accuracy: 0.5000
Epoch 22/100
32/32 [==============================] - 0s 1ms/step - loss: 0.9269 - accuracy: 0.5000
Epoch 23/100
32/32 [==============================] - 0s 1ms/step - loss: 0.9190 - accuracy: 0.5000
Epoch 24/100
32/32 [==============================] - 0s 2ms/step - loss: 0.9115 - accuracy: 0.5000
Epoch 25/100
32/32 [==============================] - 0s 1ms/step - loss: 0.9046 - accuracy: 0.5000
Epoch 26/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8976 - accuracy: 0.5000
Epoch 27/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8912 - accuracy: 0.4990
Epoch 28/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8850 - accuracy: 0.4960
Epoch 29/100
32/32 [==============================] - 0s 1ms/step - loss: 0.8790 - accuracy: 0.4950
Epoch 30/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8732 - accuracy: 0.4950
Epoch 31/100
32/32 [==============================] - 0s 1ms/step - loss: 0.8678 - accuracy: 0.4870
Epoch 32/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8623 - accuracy: 0.4790
Epoch 33/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8573 - accuracy: 0.4750
Epoch 34/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8525 - accuracy: 0.4730
Epoch 35/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8477 - accuracy: 0.4650
Epoch 36/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8432 - accuracy: 0.4590
Epoch 37/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8388 - accuracy: 0.4560
Epoch 38/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8347 - accuracy: 0.4470
Epoch 39/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8305 - accuracy: 0.4420
Epoch 40/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8266 - accuracy: 0.4380
Epoch 41/100
32/32 [==============================] - 0s 1ms/step - loss: 0.8227 - accuracy: 0.4340
Epoch 42/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8189 - accuracy: 0.4290
Epoch 43/100
32/32 [==============================] - 0s 1ms/step - loss: 0.8151 - accuracy: 0.4280
Epoch 44/100
32/32 [==============================] - 0s 1ms/step - loss: 0.8116 - accuracy: 0.4280
Epoch 45/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8082 - accuracy: 0.4200
Epoch 46/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8049 - accuracy: 0.4160
Epoch 47/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8018 - accuracy: 0.4150
Epoch 48/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7986 - accuracy: 0.4130
Epoch 49/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7956 - accuracy: 0.4140
Epoch 50/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7926 - accuracy: 0.4170
Epoch 51/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7897 - accuracy: 0.4200
Epoch 52/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7870 - accuracy: 0.4210
Epoch 53/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7843 - accuracy: 0.4280
Epoch 54/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7817 - accuracy: 0.4350
Epoch 55/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7791 - accuracy: 0.4480
Epoch 56/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7766 - accuracy: 0.4510
Epoch 57/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7742 - accuracy: 0.4530
Epoch 58/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7718 - accuracy: 0.4530
Epoch 59/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7696 - accuracy: 0.4550
Epoch 60/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7674 - accuracy: 0.4570
Epoch 61/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7654 - accuracy: 0.4560
Epoch 62/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7634 - accuracy: 0.4600
Epoch 63/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7613 - accuracy: 0.4610
Epoch 64/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7595 - accuracy: 0.4640
Epoch 65/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7577 - accuracy: 0.4590
Epoch 66/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7558 - accuracy: 0.4630
Epoch 67/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7541 - accuracy: 0.4620
Epoch 68/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7524 - accuracy: 0.4640
Epoch 69/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7507 - accuracy: 0.4660
Epoch 70/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7491 - accuracy: 0.4660
Epoch 71/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7475 - accuracy: 0.4680
Epoch 72/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7460 - accuracy: 0.4710
Epoch 73/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7445 - accuracy: 0.4720
Epoch 74/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7430 - accuracy: 0.4740
Epoch 75/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7416 - accuracy: 0.4730
Epoch 76/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7403 - accuracy: 0.4750
Epoch 77/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7390 - accuracy: 0.4730
Epoch 78/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7377 - accuracy: 0.4720
Epoch 79/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7365 - accuracy: 0.4730
Epoch 80/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7353 - accuracy: 0.4760
Epoch 81/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7341 - accuracy: 0.4740
Epoch 82/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7330 - accuracy: 0.4750
Epoch 83/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7320 - accuracy: 0.4790
Epoch 84/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7309 - accuracy: 0.4790
Epoch 85/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7299 - accuracy: 0.4810
Epoch 86/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7290 - accuracy: 0.4820
Epoch 87/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7280 - accuracy: 0.4850
Epoch 88/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7271 - accuracy: 0.4850
Epoch 89/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7263 - accuracy: 0.4870
Epoch 90/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7254 - accuracy: 0.4880
Epoch 91/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7246 - accuracy: 0.4900
Epoch 92/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7238 - accuracy: 0.4890
Epoch 93/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7230 - accuracy: 0.4890
Epoch 94/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7223 - accuracy: 0.4860
Epoch 95/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7215 - accuracy: 0.4860
Epoch 96/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7208 - accuracy: 0.4860
Epoch 97/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7202 - accuracy: 0.4880
Epoch 98/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7195 - accuracy: 0.4870
Epoch 99/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7188 - accuracy: 0.4870
Epoch 100/100
32/32 [==============================] - 0s 1ms/step - loss: 0.7181 - accuracy: 0.4850

Okay, our model performs a little worse than guessing.

Let's remind ourselves what our data looks like.

In [24]:

# Check out our data
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.RdYlBu);

# Check out our data plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.RdYlBu);

And let's see how our model is making predictions on it.

In [25]:

# Check the deicison boundary (blue is blue class, yellow is the crossover, red is red class)
plot_decision_boundary(model_4, X, y)

# Check the deicison boundary (blue is blue class, yellow is the crossover, red is red class) plot_decision_boundary(model_4, X, y)

doing binary classifcation...

Well, it looks like we're getting a straight (linear) line prediction again.

But our data is non-linear (not a straight line)...

What we're going to have to do is add some non-linearity to our model.

To do so, we'll use the activation parameter in on of our layers.

In [26]:

# Set random seed
tf.random.set_seed(42)

# Create a model with a non-linear activation
model_5 = tf.keras.Sequential([
  tf.keras.layers.Dense(1, activation=tf.keras.activations.relu), # can also do activation='relu'
  tf.keras.layers.Dense(1) # output layer 
])

# Compile the model
model_5.compile(loss=tf.keras.losses.binary_crossentropy,
              optimizer=tf.keras.optimizers.Adam(),
              metrics=["accuracy"])

# Fit the model
history = model_5.fit(X, y, epochs=100)

# Set random seed tf.random.set_seed(42) # Create a model with a non-linear activation model_5 = tf.keras.Sequential([ tf.keras.layers.Dense(1, activation=tf.keras.activations.relu), # can also do activation='relu' tf.keras.layers.Dense(1) # output layer ]) # Compile the model model_5.compile(loss=tf.keras.losses.binary_crossentropy, optimizer=tf.keras.optimizers.Adam(), metrics=["accuracy"]) # Fit the model history = model_5.fit(X, y, epochs=100)

Epoch 1/100
32/32 [==============================] - 1s 1ms/step - loss: 1.8377 - accuracy: 0.5000
Epoch 2/100
32/32 [==============================] - 0s 1ms/step - loss: 1.4449 - accuracy: 0.5000
Epoch 3/100
32/32 [==============================] - 0s 2ms/step - loss: 1.3410 - accuracy: 0.5000
Epoch 4/100
32/32 [==============================] - 0s 2ms/step - loss: 1.2678 - accuracy: 0.4770
Epoch 5/100
32/32 [==============================] - 0s 2ms/step - loss: 1.2116 - accuracy: 0.4390
Epoch 6/100
32/32 [==============================] - 0s 2ms/step - loss: 1.1664 - accuracy: 0.4180
Epoch 7/100
32/32 [==============================] - 0s 2ms/step - loss: 1.1294 - accuracy: 0.4250
Epoch 8/100
32/32 [==============================] - 0s 2ms/step - loss: 1.0970 - accuracy: 0.4420
Epoch 9/100
32/32 [==============================] - 0s 2ms/step - loss: 1.0670 - accuracy: 0.4540
Epoch 10/100
32/32 [==============================] - 0s 2ms/step - loss: 1.0407 - accuracy: 0.4550
Epoch 11/100
32/32 [==============================] - 0s 2ms/step - loss: 1.0147 - accuracy: 0.4600
Epoch 12/100
32/32 [==============================] - 0s 2ms/step - loss: 0.9872 - accuracy: 0.4630
Epoch 13/100
32/32 [==============================] - 0s 2ms/step - loss: 0.9579 - accuracy: 0.4620
Epoch 14/100
32/32 [==============================] - 0s 2ms/step - loss: 0.9201 - accuracy: 0.4660
Epoch 15/100
32/32 [==============================] - 0s 2ms/step - loss: 0.8514 - accuracy: 0.4660
Epoch 16/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7888 - accuracy: 0.4720
Epoch 17/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7580 - accuracy: 0.4740
Epoch 18/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7392 - accuracy: 0.4760
Epoch 19/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7273 - accuracy: 0.4850
Epoch 20/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7180 - accuracy: 0.4870
Epoch 21/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7121 - accuracy: 0.4880
Epoch 22/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7070 - accuracy: 0.4880
Epoch 23/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7036 - accuracy: 0.4870
Epoch 24/100
32/32 [==============================] - 0s 2ms/step - loss: 0.7011 - accuracy: 0.4900
Epoch 25/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6993 - accuracy: 0.4860
Epoch 26/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6977 - accuracy: 0.4910
Epoch 27/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6970 - accuracy: 0.4950
Epoch 28/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6959 - accuracy: 0.4970
Epoch 29/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6955 - accuracy: 0.4950
Epoch 30/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6949 - accuracy: 0.4970
Epoch 31/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6946 - accuracy: 0.4990
Epoch 32/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6943 - accuracy: 0.4910
Epoch 33/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6940 - accuracy: 0.5000
Epoch 34/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6939 - accuracy: 0.4910
Epoch 35/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6937 - accuracy: 0.4940
Epoch 36/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4930
Epoch 37/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4950
Epoch 38/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6935 - accuracy: 0.4900
Epoch 39/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4840
Epoch 40/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5000
Epoch 41/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6936 - accuracy: 0.5060
Epoch 42/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4800
Epoch 43/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5050
Epoch 44/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6936 - accuracy: 0.4810
Epoch 45/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4550
Epoch 46/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6932 - accuracy: 0.4920
Epoch 47/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6935 - accuracy: 0.4870
Epoch 48/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.5100
Epoch 49/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6934 - accuracy: 0.4770
Epoch 50/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6934 - accuracy: 0.5000
Epoch 51/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6933 - accuracy: 0.4760
Epoch 52/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6934 - accuracy: 0.5010
Epoch 53/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4880
Epoch 54/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5530
Epoch 55/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5060
Epoch 56/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5200
Epoch 57/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4910
Epoch 58/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4990
Epoch 59/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6938 - accuracy: 0.5000
Epoch 60/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.5000
Epoch 61/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4900
Epoch 62/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4820
Epoch 63/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4700
Epoch 64/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4820
Epoch 65/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4620
Epoch 66/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4760
Epoch 67/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4830
Epoch 68/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6933 - accuracy: 0.4680
Epoch 69/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5010
Epoch 70/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4970
Epoch 71/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4680
Epoch 72/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.5070
Epoch 73/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.5320
Epoch 74/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5290
Epoch 75/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5000
Epoch 76/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4730
Epoch 77/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4870
Epoch 78/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4830
Epoch 79/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4640
Epoch 80/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5040
Epoch 81/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6936 - accuracy: 0.5000
Epoch 82/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5020
Epoch 83/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6936 - accuracy: 0.4840
Epoch 84/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6932 - accuracy: 0.5070
Epoch 85/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.5000
Epoch 86/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.5000
Epoch 87/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6933 - accuracy: 0.5000
Epoch 88/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4680
Epoch 89/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4590
Epoch 90/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6937 - accuracy: 0.4980
Epoch 91/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6933 - accuracy: 0.4910
Epoch 92/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4850
Epoch 93/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6938 - accuracy: 0.4970
Epoch 94/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4710
Epoch 95/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4720
Epoch 96/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6934 - accuracy: 0.4880
Epoch 97/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6934 - accuracy: 0.4510
Epoch 98/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.4640
Epoch 99/100
32/32 [==============================] - 0s 2ms/step - loss: 0.6935 - accuracy: 0.4940
Epoch 100/100
32/32 [==============================] - 0s 1ms/step - loss: 0.6936 - accuracy: 0.5180

Hmm... still not learning...

What we if increased the number of neurons and layers?

Say, 2 hidden layers, with ReLU, pronounced "rel-u", (short for rectified linear unit), activation on the first one, and 4 neurons each?

To see this network in action, check out the TensorFlow Playground demo.

The neural network we're going to recreate with TensorFlow code. See it live at TensorFlow Playground.

Let's try.

In [27]:

# Set random seed
tf.random.set_seed(42)

# Create a model
model_6 = tf.keras.Sequential([
  tf.keras.layers.Dense(4, activation=tf.keras.activations.relu), # hidden layer 1, 4 neurons, ReLU activation
  tf.keras.layers.Dense(4, activation=tf.keras.activations.relu), # hidden layer 2, 4 neurons, ReLU activation
  tf.keras.layers.Dense(1) # ouput layer
])

# Compile the model
model_6.compile(loss=tf.keras.losses.binary_crossentropy,
                optimizer=tf.keras.optimizers.Adam(lr=0.001), # Adam's default learning rate is 0.001
                metrics=['accuracy'])

# Fit the model
history = model_6.fit(X, y, epochs=100)

# Set random seed tf.random.set_seed(42) # Create a model model_6 = tf.keras.Sequential([ tf.keras.layers.Dense(4, activation=tf.keras.activations.relu), # hidden layer 1, 4 neurons, ReLU activation tf.keras.layers.Dense(4, activation=tf.keras.activations.relu), # hidden layer 2, 4 neurons, ReLU activation tf.keras.layers.Dense(1) # ouput layer ]) # Compile the model model_6.compile(loss=tf.keras.losses.binary_crossentropy, optimizer=tf.keras.optimizers.Adam(lr=0.001), # Adam's default learning rate is 0.001 metrics=['accuracy']) # Fit the model history = model_6.fit(X, y, epochs=100)

/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/adam.py:105: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  super(Adam, self).__init__(name, **kwargs)

Epoch 1/100
32/32 [==============================] - 1s 1ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 2/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 3/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 4/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 5/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 6/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 7/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 8/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 9/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 10/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 11/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 12/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 13/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 14/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 15/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 16/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 17/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 18/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 19/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 20/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 21/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 22/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 23/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 24/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 25/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 26/100
32/32 [==============================] - 0s 1ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 27/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 28/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 29/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 30/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 31/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 32/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 33/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 34/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 35/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 36/100
32/32 [==============================] - 0s 1ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 37/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 38/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 39/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 40/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 41/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 42/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 43/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 44/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 45/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 46/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 47/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 48/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 49/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 50/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 51/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 52/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 53/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 54/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 55/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 56/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 57/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 58/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 59/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 60/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 61/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 62/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 63/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 64/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 65/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 66/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 67/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 68/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 69/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 70/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 71/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 72/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 73/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 74/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 75/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 76/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 77/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 78/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 79/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 80/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 81/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 82/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 83/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 84/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 85/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 86/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 87/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 88/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 89/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 90/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 91/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 92/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 93/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 94/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 95/100
32/32 [==============================] - 0s 1ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 96/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 97/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 98/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 99/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000
Epoch 100/100
32/32 [==============================] - 0s 2ms/step - loss: 7.7125 - accuracy: 0.5000

In [28]:

# Evaluate the model
model_6.evaluate(X, y)

# Evaluate the model model_6.evaluate(X, y)

32/32 [==============================] - 0s 1ms/step - loss: 7.7125 - accuracy: 0.5000

Out[28]:

[7.712474346160889, 0.5]

We're still hitting 50% accuracy, our model is still practically as good as guessing.

How do the predictions look?

In [29]:

# Check out the predictions using 2 hidden layers
plot_decision_boundary(model_6, X, y)

# Check out the predictions using 2 hidden layers plot_decision_boundary(model_6, X, y)

doing binary classifcation...